Blending Classifier, the game
Stacking’s simpler cousin. Learn the hold-out trick, put the blending steps in order, and write leak-free blending code in Python.
The hold-out trick
Blending trains several base models, then trains a blender (meta-model) to combine their predictions. Instead of stacking’s cross-validation, the blender learns from predictions on a single hold-out validation set that the base models never trained on.
| Stacking | Blending | |
|---|---|---|
| Meta-model learns from | Out-of-fold predictions (cross-validation) | Predictions on one hold-out set |
| Data used | All training rows | Training rows minus the hold-out |
| Speed / simplicity | Slower, more robust | Faster, simpler |
What does the blender train on?
Blend step by step
The article’s bank example: a logistic regression and a decision tree each predict the probability that a customer will default, e.g. [0.7, 0.6]. Those probabilities become the features of the blender.
Tap the steps in the order they happen.
Code it (without the leak)
The article balances the Framingham data and blends logistic regression, a random forest and an SVM. Its original code trains the blender on the test set and then scores it on that same test set, which inflates the accuracy. This version adds a proper validation split.
Step 1Three-way split
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
df = pd.read_csv("framingham.csv").dropna()
minority = df[df.TenYearCHD == 1]
majority = df[df.TenYearCHD == 0].sample(n=len(minority), random_state=42) # undersample
df = pd.concat([majority, minority]).sample(frac=1, random_state=42)
X, y = df.drop(columns="TenYearCHD"), df["TenYearCHD"]
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=42)Step 2Blend
base_models = [LogisticRegression(max_iter=1000), RandomForestClassifier(random_state=42), SVC(probability=True)]
for m in base_models:
m.fit(X_train, y_train)
val_preds = np.column_stack([m.predict_proba(X_val)[:, 1] for m in base_models])
test_preds = np.column_stack([m.predict_proba(X_test)[:, 1] for m in base_models])
blender = LogisticRegression().fit(val_preds, y_val) # learns on the hold-out set
print("Blending accuracy:", accuracy_score(y_test, blender.predict(test_preds)))Why is training the blender on y_test a problem?
Defeat the Blend Boss 🐲
Six questions, three hearts.
You’re a Blend Master!
Keep levelling up
You learned how blending combines base models with a blender trained on a hold-out set, how it differs from stacking, why the blender must never see the test labels, and how to code it correctly.