term frequency-inverse document frequency (TF-IDF)

Term frequency-inverse document frequency (TF-IDF) is a weighting scheme that scores how important a word is to one document by balancing how often the word appears there against how common it is across the whole collection of documents.

The score is a product of two factors. Term frequency counts how often a term shows up in a single document. Inverse document frequency is the logarithm of the total number of documents divided by the number of documents containing that term, so a term present in nearly every document contributes almost nothing.

TF-IDF therefore peaks for a word used many times in a handful of documents, which is what gives that word the power to tell those documents apart. The effect suppresses filler words like “the” and “of” statistically, without a hand-written stop-word list.

The widget below scores a single word against a collection of ten documents. Raising how often the word appears in the scored document lifts its score, and spreading the word across more of the collection collapses that score back toward zero.

The weighting traces back to a 1972 Journal of Documentation paper by Karen Spärck Jones, who argued that a term’s specificity should be treated as a function of how the term is used rather than what it means. Implementations differ in smoothing and normalization. Scikit-learn’s TfidfVectorizer, for instance, adds one to both counts inside the logarithm and one to the logarithm itself, then scales each document to unit length.

A document’s TF-IDF weights form a sparse vector over the vocabulary, with one dimension per distinct word and zeros for every word the document doesn’t use. Such vectors match wording rather than meaning, unlike a dense embedding.

That difference is why retrieval-augmented generation pipelines often run a lexical scorer, which matches exact words, and a dense one together, then fuse the two ranked lists in a setup known as hybrid search. BM25 refines the same idea with document-length normalization and saturating term frequency, where each extra repeat of a word adds less to the score. BM25, not plain TF-IDF, is the default ranking function in Elasticsearch.

Related Resources

Tutorial

Practical Text Classification With Python and Keras

Learn about Python text classification with Keras. Work your way from a bag-of-words model with logistic regression to more advanced methods leading to convolutional neural networks. See why word embeddings are useful and how you can use pretrained word embeddings. Use hyperparameter optimization to squeeze more performance out of your model.

advanced data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Natural Language Processing With Python's NLTK Package (Tutorial)
• Natural Language Processing With spaCy in Python (Tutorial)
• Use Sentiment Analysis With Python to Classify Movie Reviews (Tutorial)
• Sentiment Analysis: First Steps With Python's NLTK Library (Tutorial)
• Learn Text Classification With Python and Keras (Course)
• Natural Language Processing With Python's NLTK Package (Quiz)