Bag of Words and TF-IDF

Bag of Words (BoW) and TF-IDF are two foundational techniques for converting raw text into numerical vectors that machine learning models can process, forming the backbone of classical NLP pipelines from spam filters to search engines. They matter in modern AI because even today's large language models are often compared against or combined with these simple baselines, which are fast, interpretable, and surprisingly competitive on many tasks. The counterintuitive fact: a spam filter built with TF-IDF in the 1990s can still outperform a neural network on small datasets — more data often matters more than a fancier model.

The core insight

Every text document can be represented as a table of word counts — ignore word order, ignore grammar, just count which words appear and how often. The "aha moment" is realizing that this crude representation is actually enough information to distinguish sports articles from medical articles, spam from legitimate email, and positive reviews from negative ones. TF-IDF goes one step further by rewarding words that are distinctive to a document (high frequency there) while penalizing words that appear everywhere (like "the" and "is"). Think of it as measuring not just how often a word shows up, but how much it tells you about this specific document versus all the others.

How It Works

Step 1 — Bag of Words

Start with a corpus of documents. Build a vocabulary: the set of all unique words across every document. Each document becomes a vector whose length equals the vocabulary size, and each position holds the count of that word in the document. ```python from sklearn.feature_extraction.text import CountVectorizer corpus = [ "the cat sat on the mat", "the dog sat on the log", "cats and dogs are friends", ] vectorizer = CountVectorizer() X = vectorizer.fit_transform(corpus) print(vectorizer.get_feature_names_out())

# ['and' 'are' 'cat' 'cats' 'dog' 'dogs' 'friends' 'log' 'mat' 'on' 'sat' 'the'] print(X.toarray())

# [[0 0 1 0 0 0 0 0 1 1 1 2] <- doc 1

# [0 0 0 0 1 0 0 1 0 1 1 2] <- doc 2

# [1 1 0 1 0 1 1 0 0 0 0 0]] <- doc 3 ``` Notice: word order is completely lost. "The cat sat on the mat" and "the mat sat on the cat" would produce identical vectors.

Step 2 — Term Frequency (TF)

Raw counts work, but longer documents will naturally have larger counts for every word, making them look more "intense" than short documents. We normalise by dividing each count by the total number of words in that document:

\quad TF(t, d) = \frac{\text{count of term } t \text{ in document } d}{\text{total number of terms in document } d}

Explanation: This turns raw counts into proportions so that a 1000-word document and a 10-word document are on a comparable scale.

t — the specific term (word) you are scoring
d — the document you are looking at
The numerator counts how many times t appears in d
The denominator is the total word count of d

Step 3 — Inverse Document Frequency (IDF)

Words like "the", "is", and "a" appear in almost every document, so a high TF score for them tells us nothing useful. IDF punishes such words by measuring how rare a word is across all documents:

\quad IDF(t, D) = \log\left(\frac{N}{1 + df(t)}\right)

Explanation: Words appearing in every document get an IDF close to zero; words appearing in only one document get a high IDF, amplifying their importance.

N — total number of documents in the corpus
df(t) — number of documents that contain term t
The +1 in the denominator is a smoothing trick to avoid division by zero when a word appears in every document
The log compresses the scale so one very rare word doesn't dominate everything

Step 4 — TF-IDF Combined

Multiply the two scores together:

\quad TFIDF(t, d, D) = TF(t, d) \times IDF(t, D)

Explanation: A word scores highly only when it appears frequently in this specific document AND is rare across the overall corpus — the sweet spot of "distinctive relevance." ```python from sklearn.feature_extraction.text import TfidfVectorizer corpus = [ "the cat sat on the mat", "the dog sat on the log", "cats and dogs are friends", ] tfidf = TfidfVectorizer() X = tfidf.fit_transform(corpus) import pandas as pd df = pd.DataFrame( X.toarray(), columns=tfidf.get_feature_names_out() ) print(df.round(2))

# and are cat cats dog dogs friends log mat on sat the

# 0 0.00 0.00 0.47 0.00 0.00 0.00 0.00 0.00 0.47 0.35 0.35 0.57

# 1 0.00 0.00 0.00 0.00 0.47 0.00 0.00 0.47 0.00 0.35 0.35 0.57

# 2 0.44 0.44 0.00 0.44 0.00 0.44 0.44 0.00 0.00 0.00 0.00 0.00 ``` Notice "the" gets a moderate score (high TF, but also appears in 2 out of 3 documents, reducing IDF), while "cat" and "mat" get higher scores in document 1 because they are unique to it.

researcher · Karen Spärck Jones | 1935 – 2007 | Invented Inverse Document Frequency

Karen Spärck Jones was a British computer scientist at the University of Cambridge who spent her career making computers understand human language at a time when the field had almost no women in senior roles. In her landmark 1972 paper "A Statistical Interpretation of Term Specificity and Its Application in Retrieval," she introduced the idea that a term's usefulness for retrieval should be inversely proportional to how many documents it appears in — the core idea behind IDF. A surprising fact: she was also a fierce advocate for women in computing and publicly refused several honours she felt were not awarded on merit, insisting the field needed systemic change rather than tokenism.

Figure

Bag of Words to TF-IDF Pipeline

Intuition

The Mental Model

Imagine you are a librarian sorting thousands of books by topic, but you can only read a bag full of paper slips pulled randomly from each book — each slip has one word on it. You tip the bag out, sort the slips into piles by word, and count each pile. That is Bag of Words: you have completely lost the sentence structure, but you can still tell a medical textbook (lots of slips saying "patient", "dosage", "symptom") from a cookery book (lots of "flour", "bake", "sauce"). Now, some slips say "the" or "and" — they are in every book's bag and tell you nothing useful. A smart librarian would ignore those. TF-IDF is that smart librarian: it says "I'll weight each word by how surprising it is to find it here." A book with ten slips saying "mitochondria" when most other books have zero? That's a very important clue. A book with fifty slips saying "the"? Totally unremarkable. The score captures surprise — how distinctive this word is for this document specifically.

Historical Development

Origins · 1 minhow this was discovered

History

1957

Hans Peter Luhn at IBM proposed using word frequency as a measure of significance in documents — one of the earliest automatic text analysis systems.

1972

Karen Spärck Jones publishes "A Statistical Interpretation of Term Specificity and Its Application in Retrieval," introducing the concept of inverse document frequency (IDF).

1983

Salton and McGill formalize the Vector Space Model in their textbook "Introduction to Modern Information Retrieval," establishing TF-IDF as the standard for document retrieval.

1988

Salton and Buckley publish the systematic comparison of TF-IDF weighting schemes, showing that combined TF×IDF consistently outperforms either component alone.

1998

Joachims demonstrates that SVM classifiers trained on TF-IDF vectors achieve state-of-the-art text classification, cementing BoW+TF-IDF as the dominant NLP pipeline for over a decade.

2003

Blei, Ng, and Jordan publish Latent Dirichlet Allocation (LDA), which builds directly on BoW word-count matrices to discover latent topics — showing BoW representations still enable sophisticated models.

2013

Mikolov et al. release Word2Vec, which begins to challenge TF-IDF by capturing word meaning through context, though TF-IDF remains competitive on many benchmarks for years after.

2019

Researchers at Google and elsewhere continue to use TF-IDF as a fast retrieval step inside BM25-based systems, which underpins hybrid search in production retrieval-augmented generation (RAG) pipelines.

Why It Matters

Real-World Impact

Gmail Spam Filter (early versions)Google's initial spam detection pipeline relied heavily on TF-IDF-weighted word vectors fed into Naive Bayes classifiers — words like "viagra" or "lottery" had high TF-IDF scores in spam corpora, making them powerful discriminating features with minimal computation.

Elasticsearch / Apache SolrBoth of the world's most widely deployed open-source search engines use BM25, a probabilistic extension of TF-IDF, as their default relevance ranking algorithm. When you search on Wikipedia, Stack Overflow, or most enterprise intranets, TF-IDF-style scoring is ranking your results.

Scikit-learn Text PipelinesThe `TfidfVectorizer` in scikit-learn is one of the most-used classes in all of applied ML; data scientists use it daily as the feature-extraction step before training Logistic Regression or SVM classifiers for sentiment analysis, topic classification, and intent detection in chatbots.

Retrieval-Augmented Generation (RAG) SystemsModern LLM applications like LlamaIndex and LangChain use BM25 (a TF-IDF variant) as the "sparse retrieval" component in hybrid search, alongside dense vector embeddings — TF-IDF catches exact keyword matches that neural embeddings sometimes miss.

Legal and Medical Document ReviewLaw firms and clinical NLP tools use TF-IDF to surface the most distinctive terms in contracts or clinical notes, enabling rapid categorisation of thousands of documents without expensive GPU inference — interpretability and speed matter more than marginal accuracy gains.

Plagiarism Detection (Turnitin)Document similarity systems compute cosine similarity between TF-IDF vectors of submitted essays and a corpus of known documents — a direct application of the vector space model that processes millions of submissions daily.