BM25

BM25 is a ranking function that scores how well a document matches a keyword query, using only the words the two have in common. It needs no learned model and no embedding, which keeps it fast and predictable. BM25 is the default relevance scorer in Lucene and Elasticsearch, and the keyword half of hybrid retrieval in many retrieval-augmented generation (RAG) pipelines.

The score adds up one weight per query term. Each weight is the product of two factors. The inverse document frequency (IDF) factor favors rare words over common ones. The term frequency (TF) factor saturates, so a tenth occurrence of a word counts for far less than a second.

Two parameters shape that curve. The first, k1, sets how quickly saturation kicks in and usually sits between 1.2 and 2.0. The second, b, sets how much a document’s length discounts its score, running from 0 for no normalization to 1 for full, with 0.75 as the common default.

The widget below plots the weight a single term earns as it repeats. Moving k1, b, and the document’s length relative to the average document in the collection bends that curve.

BM25 grew out of the probabilistic relevance framework and the Okapi retrieval system, which is why it’s also called Okapi BM25. The BM stands for best matching, and the function debuted at the third Text REtrieval Conference (TREC) in 1994.

Matching literal terms lets BM25 surface error strings, identifiers, and rare jargon that a dense vector search blurs together, which is why retrieval stacks often run both and pass the merged candidates to a reranker.

Related Resources

Tutorial

Embeddings and Vector Databases With ChromaDB

Learn how to use ChromaDB, an open-source Python vector database, to store embeddings and run semantic search with hands-on examples.

advanced ai databases data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Build an LLM RAG Chatbot With LangChain (Tutorial)
• LlamaIndex in Python: A RAG Guide With Examples (Tutorial)
• Natural Language Processing With Python's NLTK Package (Tutorial)
• Natural Language Processing With spaCy in Python (Tutorial)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• Using LlamaIndex for RAG in Python (Course)
• LlamaIndex in Python: A RAG Guide With Examples (Quiz)
• Natural Language Processing With Python's NLTK Package (Quiz)