Exactly! A near property of the matryoshka embeddings is that you can compute a low dimension embedding similarity really fast and then refine afterwards.

Looks cool! You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`

That is neat I like that.

It would be cool if the "More Like This" had a + button that would append the arxiv id to the search query.

That's a nice idea! Might take a look this weekend!

Sadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!

Here it is:

https://chemrxiv.org/engage/chemrxiv/public-api/documentatio...

Thanks!

There is also engrXiv, which has an OAI endpoint. https://engrxiv.org/oai?verb=ListRecords&metadataPrefix=oai_...

Amazing!

Do they have a public API/dataset?

They have RSS feeds for new/updated papers: https://eprint.iacr.org/rss/

Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!

I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...

So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?

Impressive! Will you parse the papers in the future? Without citations this is not that usable for professors or scientists in general. The relevance ranking largely depends on showing these older, prominent papers. (from our lab experience building decentralised search using transformers)

One chunk embedded together

That method can break when author names and subject matter collide.

True, but similarly if your embeddings are any good they'll capture interesting associations between authors, topics and your search query. If you find any interesting author overlap results I'd be very interested!

Not exactly what I was looking for, but interesting nonetheless: https://arxivxplorer.com/?q=exotic+penis

Thank you!!

Tensorflow also created Eager - their dynamic environment

Yeah I think it's important to be completely open about it since browser mining has a bad rep. Everyone trying it should know what happens as much as possible before just clicking 'mine'

Hey HN!

Would love to hear any feedback on this project! Positive or negative