Understanding the BM25 full text search algorithm

If you use async Rust and Tokio, you are likely to run into some variant of the "future is not Send" compiler error. While transitioning some sequential async code to use streams, a friend suggested a small technique for pinning down the source of the non-Send errors. It helped a lot, so I thought it would be worth writing up in case it saves...

This was sent out to everyone who's signed up for Scour. Reposting here for anyone else that comes across it later. Hi friends, This is Evan from Scour, writing to you with the first product update. Here are some of the new features I've added over the last month or so. Enjoy -- and let me know what you think! Filtering Out Low-Quality...

Applications that run documents through LLMs or embedding models need to clean the text before feeding it into the model. I'm building a personalized content feed called Scour and was looking for a Rust crate to extract text from scraped HTML. I started off using a library that's used by a couple of LLM-related projects. However, while hunting a...

If you're developing an application and find yourself running a benchmark whose results are measured in nanoseconds... you should probably stop and get back to more important tasks. But here we are. I'm using binary vector embeddings to build Scour, a service that scours noisy feeds for content related to your interests. Scour uses the Hamming...

I wrote another post about Understanding the BM25 full text search algorithm and had initially included comparisons with two other algorithms. However, that post was already quite long so here are the brief comparisons between BM25, TF-IDF, and PostgreSQL's full text search. BM25 vs TF-IDFTF-IDF was the main model that was used prior to the...