Naive Bayes Classifier

Naive Bayes applies Bayes' Theorem to flip a conditional probability into a useful prediction. The idea is: instead of directly estimating P(class | features), which requires a huge amount of data, we estimate the simpler quantities P(features | class) and P(class) from our training set, then combine them.

Explanation: The probability of class C given all the observed features equals the prior probability of C multiplied by the product of each individual feature's likelihood under that class.

Symbol guide:

• $P(C \mid x_1,\ldots,x_n)$ — the posterior: what we ultimately want, the probability of class C given the features we observed.
• $P(C)$ — the prior: how often class C appears in the training data (e.g., 40% of emails are spam).
• $P(x_i \mid C)$ — the likelihood of feature $i$ given class $C$ (e.g., how often the word "free" appears in spam emails).
• $\prod$ — multiply together all the per-feature likelihoods (the "naive" step — we treat them as independent).

We pick the class with the highest posterior score:

Explanation: The predicted class is simply whichever class produces the largest combined score when you multiply the prior by all the feature likelihoods.

Avoiding Underflow with Log Probabilities

Multiplying many small probabilities together causes numbers to shrink toward zero and create floating-point underflow. In practice, we take the logarithm of equation [1], turning the product into a sum:

Explanation: Summing log-probabilities gives identical ranking results to multiplying raw probabilities, but stays numerically stable even with thousands of features.

Laplace Smoothing

If a word never appeared in the training set for a given class, its likelihood is zero — and that zero kills the entire product. Laplace smoothing adds a small count (usually 1) to every feature, preventing zero probabilities:

Explanation: By adding 1 to the numerator and the vocabulary size |V| to the denominator, every feature gets a non-zero probability even if it was never seen during training.

Python: Training a Naive Bayes Spam Filter from Scratch

```python from collections import defaultdict import math class NaiveBayesClassifier: def __init__(self, alpha=1.0): self.alpha = alpha # Laplace smoothing parameter self.class_log_priors = {} # log P(C) self.feature_log_likelihoods = defaultdict(dict) # log P(word|C) self.vocab = set() def fit(self, X, y): """ X: list of lists of words (tokenised documents) y: list of class labels (e.g., 'spam', 'ham') """ class_word_counts = defaultdict(lambda: defaultdict(int)) class_doc_counts = defaultdict(int) for doc, label in zip(X, y): class_doc_counts[label] += 1 for word in doc: class_word_counts[label][word] += 1 self.vocab.add(word) total_docs = len(y) V = len(self.vocab) for cls, doc_count in class_doc_counts.items():

# log prior: how common is this class? self.class_log_priors[cls] = math.log(doc_count / total_docs) total_words = sum(class_word_counts[cls].values()) for word in self.vocab: count = class_word_counts[cls].get(word, 0)

# Laplace-smoothed log likelihood self.feature_log_likelihoods[cls][word] = math.log( (count + self.alpha) / (total_words + self.alpha * V) ) return self def predict(self, X): return [self._predict_single(doc) for doc in X] def _predict_single(self, doc): scores = {} for cls in self.class_log_priors:

# Start with the log prior (equation [3]) score = self.class_log_priors[cls] for word in doc: if word in self.vocab: score += self.feature_log_likelihoods[cls][word] scores[cls] = score

# Return the class with the highest log-probability (equation [2]) return max(scores, key=scores.get)

# ── Quick demo ───────────────────────────────────────────────────────────── train_docs = [ ["free", "money", "click", "now"], # spam ["free", "prize", "winner", "claim"], # spam ["meeting", "tomorrow", "agenda"], # ham ["project", "update", "please", "review"], # ham ["win", "cash", "free", "offer"], # spam ["lunch", "tomorrow", "office"], # ham ] train_labels = ["spam","spam","ham","ham","spam","ham"] clf = NaiveBayesClassifier(alpha=1.0) clf.fit(train_docs, train_labels) test = [["free", "cash", "win"], ["meeting", "agenda"]] print(clf.predict(test)) # → ['spam', 'ham'] ```

Using scikit-learn (Production Path)

```python from sklearn.naive_bayes import MultinomialNB from sklearn.feature_extraction.text import CountVectorizer from sklearn.pipeline import Pipeline

# Raw text documents train_texts = [ "free money click now", "free prize winner claim", "meeting tomorrow agenda", "project update please review", "win cash free offer", "lunch tomorrow office", ] train_labels = ["spam","spam","ham","ham","spam","ham"]

# Pipeline: tokenise → count → Naive Bayes pipeline = Pipeline([ ("vectorizer", CountVectorizer()), ("classifier", MultinomialNB(alpha=1.0)), # alpha = Laplace smoothing ]) pipeline.fit(train_texts, train_labels)

# Predict new examples test_texts = ["free cash win now", "project meeting tomorrow"] print(pipeline.predict(test_texts))

# → ['spam', 'ham']

# Get calibrated-ish probabilities (still interpret with caution) probs = pipeline.predict_proba(test_texts) print(probs) # columns correspond to pipeline.classes_ ```

Three Flavours of Naive Bayes

```python from sklearn.naive_bayes import GaussianNB import numpy as np

# Gaussian NB for continuous features (e.g., height, weight) X_train = np.array([[170, 65], [180, 80], [160, 55], [155, 50]]) y_train = ["male","male","female","female"] gnb = GaussianNB() gnb.fit(X_train, y_train) print(gnb.predict(165, 60)) # → ['female'] ```

Logistic Regression · Decision Trees · Support Vector Machine · Text Classification · Bayesian Networks · Probabilistic Graphical Models · Feature Engineering