Stepwise Regression, the game

Why select variables?

Too many features cause overfitting, slow models, higher data-collection costs and coefficients nobody can interpret. Stepwise regression builds a model automatically by adding and removing predictors based on a statistical criterion.

• Forward selection: start empty and add the best predictor each round.
• Backward elimination: start with everything and remove the weakest predictor each round.
• Bidirectional (stepwise): add like forward, but re-check and remove predictors that stop being useful.

Play forward selection

Your turn to be the algorithm. These are the real p-values for predicting mpg on R’s mtcars data (the article uses the Framingham heart-study data; the procedure is identical). Rule: add the feature with the lowest p-value if it is below 0.05; otherwise stop.

Selection criteria

• p-value: is the coefficient significantly different from zero? Common threshold 0.05.
• AIC (Akaike Information Criterion): balances fit and complexity. Lower is better.
• BIC (Bayesian Information Criterion): like AIC but penalizes complexity more, so it prefers simpler models.
• Adjusted R²: R² with a penalty for each extra predictor. Higher is better.

Code it in Python

The article’s forward-selection function, made runnable (it defines X and y, which the original notebook left out). The target is TenYearCHD in the Framingham dataset.

import pandas as pd
import statsmodels.api as sm

df = pd.read_csv("framingham.csv").dropna()
X = df.drop(columns="TenYearCHD")
y = df["TenYearCHD"]

def forward_selection(X, y, threshold_in=0.05):
    selected = []
    remaining = list(X.columns)
    while remaining:
        pvals = pd.Series(index=remaining, dtype=float)
        for feature in remaining:
            model = sm.OLS(y, sm.add_constant(X[selected + [feature]])).fit()
            pvals[feature] = model.pvalues[feature]
        if pvals.min() < threshold_in:
            best = pvals.idxmin()
            selected.append(best)
            remaining.remove(best)
        else:
            break
    return selected

features = forward_selection(X, y)
print(features)
print(sm.OLS(y, sm.add_constant(X[features])).fit().summary())

Defeat the Selection Boss 🐲

Six questions, three hearts.

Keep levelling up

You learned why variable selection matters, how forward, backward and bidirectional selection work, how p-values, AIC, BIC and adjusted R² guide them, and the limits of stepwise methods.