Naive Bayes, the game
The spam filter’s favourite algorithm. Build a spam filter word by word, learn the three flavours of Naive Bayes, and fit one in scikit-learn.
Bayes, but naive
The naive part: it assumes every feature is independent of the others given the class. That is rarely exactly true, but the classifier still works surprisingly well and trains extremely fast.
What makes Naive Bayes “naive”?
Build a spam filter
Each word you add multiplies in its evidence: words common in spam push the probability up, words common in normal mail push it down.
The word probabilities are example numbers, as a spam filter would learn them from labelled emails.
Tap the words that appear in the email:
Word likelihoods used (from a training set of labelled emails)
| word | P(word | spam) | P(word | not spam) |
|---|---|---|
free | 38% | 4% |
win | 25% | 2% |
click | 30% | 6% |
offer | 28% | 8% |
invoice | 5% | 12% |
meeting | 3% | 22% |
project | 2% | 18% |
Prior: 40% of training emails were spam.
- ✓Write a 95%+ spam email+25 XP
- ✓Write a clearly safe email (spam ≤ 5%)+25 XP
- ✓Mix spammy and normal words until the filter is unsure (40–60%)+30 XP
Three flavours
| Variant | Features | Typical use |
|---|---|---|
| Gaussian | Continuous numbers (assumed normally distributed per class) | Sensor readings, measurements |
| Multinomial | Counts / frequencies | Word counts in documents |
| Bernoulli | Binary 0/1 (present or absent) | Whether each word appears |
Code it in Python
The article predicts whether a bank client subscribes to a term deposit (y) from the bank-marketing data with GaussianNB.
Step 1Prepare the data
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score
data = pd.read_csv("bank_marketing.csv")
X = pd.get_dummies(data.drop(columns="y"), drop_first=True, dtype=int)
y = data["y"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)One-hot encoding avoids treating category codes as meaningful numbers.
Step 2Train and evaluate
gnb = GaussianNB()
y_pred = gnb.fit(X_train, y_train).predict(X_test)
print("Mislabeled:", (y_test != y_pred).sum(), "of", len(y_test))
print("Accuracy:", accuracy_score(y_test, y_pred))Which scikit-learn class suits word-count features best?
Defeat the Bayes Boss 🐲
Six questions, three hearts.
You’re a Bayes Classifier Master!
Keep levelling up
You learned how Naive Bayes applies Bayes’ theorem with an independence assumption, how evidence from each feature multiplies together, when to use the Gaussian, Multinomial and Bernoulli variants, and how to train one.