Logistic Regression, the game

Why not a straight line?

A straight regression line can predict values below 0 or above 1, which makes no sense as a probability, and a few outliers can drag it badly. Logistic regression passes the linear score z = β₀ + β₁x through the sigmoid function, which squashes any number into the range 0 to 1:

Odds and log-odds

Odds compare the chance something happens with the chance it doesn’t: odds = p / (1 − p). Logistic regression models the log of the odds as a straight line:

The coefficients are found by maximum likelihood: choose β so that the predicted probabilities are as close to 1 as possible for the positive examples and as close to 0 for the negative ones. There is no closed-form answer, so it is solved numerically (gradient descent, or Newton–Raphson as R’s glm does).

Threshold lab

Real data: R’s mtcars. Can a car’s weight tell us whether it has a manual gearbox? The fitted model (same as R’s glm(am ~ wt, family = binomial)) is:

The model outputs a probability; the threshold decides when to say “manual”. Move it and watch the confusion matrix.

Code it from scratch

The article builds two clusters of points and trains logistic regression with gradient descent. Here is a compact, corrected version (labels 0/1, the standard log-loss gradient).

import numpy as np

rng = np.random.default_rng(42)
X = np.vstack([rng.normal([2, 3], 1, (50, 2)), rng.normal([10, 11], 1, (50, 2))])
y = np.r_[np.zeros(50), np.ones(50)]
X = np.c_[np.ones(len(X)), X]          # add a column of 1s for the intercept

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w = np.zeros(X.shape[1])
lr = 0.01
for epoch in range(2000):
    p = sigmoid(X @ w)
    grad = X.T @ (p - y) / len(y)     # gradient of the log-loss
    w -= lr * grad

pred = (sigmoid(X @ w) >= 0.5).astype(int)
print("Accuracy:", (pred == y).mean())

# The same model in one line with scikit-learn:
from sklearn.linear_model import LogisticRegression
print(LogisticRegression().fit(X[:, 1:], y).score(X[:, 1:], y))

Defeat the Sigmoid Boss 🐲

Seven questions, three hearts.

Keep levelling up

You learned why linear regression fails for yes/no outcomes, how the sigmoid turns a linear score into a probability, what log-odds are, how maximum likelihood fits the model, and how the threshold trades precision for recall.