Classification Metrics, the game
Accuracy isn’t everything. Move the decision threshold on real data and watch the confusion matrix, precision, recall, F1 and the ROC curve react.
The confusion matrix
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP) | False negative (FN), a “miss” |
| Actually negative | False positive (FP), a “false alarm” | True negative (TN) |
Every classification metric is built from these four counts. For multi-class problems the matrix simply grows to one row and column per class.
The metric zoo
- MCC (Matthews correlation): a balanced score from −1 to 1 that stays honest on imbalanced data.
- Log loss (binary cross-entropy; categorical cross-entropy for many classes): punishes confident wrong probabilities.
- ROC AUC: how well the model ranks positives above negatives across all thresholds (0.5 = random, 1 = perfect).
TP = 8, FP = 2, FN = 4. What is the precision?
Threshold & ROC lab
The same real model as the logistic-regression lesson: predicting a manual gearbox from car weight on mtcars. Move the threshold and follow the red dot along the ROC curve.
| Predicted: manual | Predicted: auto | |
|---|---|---|
| Actual: manual | true positive | false negative |
| Actual: auto | false positive | true negative |
- ✓Find the threshold with the best F1 score+30 XP
- ✓Reach 100% recall+20 XP
- ✓Reach 100% specificity while still catching some manual cars+20 XP
Code it in Python
The article evaluates logistic regression on a cardiovascular-disease dataset. This version fixes two bugs in the original notebook: the scaler is properly fitted and applied (the original wrote scaler.fit=(X_train)), and plot_confusion_matrix, removed from scikit-learn, is replaced with ConfusionMatrixDisplay.
Step 1Train a model
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.linear_model import LogisticRegression
df = pd.read_csv("cardiovascular_diseases_dv3.csv", sep=";")
X, y = df.drop(columns="CARDIO_DISEASE"), df["CARDIO_DISEASE"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
scaler = MinMaxScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
y_pred, y_prob = clf.predict(X_test), clf.predict_proba(X_test)[:, 1]Step 2Every metric
from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
matthews_corrcoef, log_loss, roc_auc_score, ConfusionMatrixDisplay,
classification_report)
print(classification_report(y_test, y_pred))
print("Accuracy ", accuracy_score(y_test, y_pred))
print("Precision", precision_score(y_test, y_pred))
print("Recall ", recall_score(y_test, y_pred))
print("F1 ", f1_score(y_test, y_pred))
print("MCC ", matthews_corrcoef(y_test, y_pred))
print("Log loss ", log_loss(y_test, y_prob))
print("ROC AUC ", roc_auc_score(y_test, y_prob))
ConfusionMatrixDisplay.from_predictions(y_test, y_pred)Log loss and ROC AUC need probabilities (y_prob), not hard labels.
Why pass y_prob (not y_pred) to roc_auc_score?
Defeat the Metrics Boss 🐲
Seven questions, three hearts.
You’re a Metrics Champion!
Keep levelling up
You learned to read a confusion matrix, when to use accuracy, precision, recall, specificity, F1, MCC, log loss and ROC AUC, and how the threshold trades them off.