Stacking Classifier, the game
Let a model learn how to combine other models. Build the stacking pipeline step by step, then code it in scikit-learn.
Learning to combine
Meta learning means learning from the outputs of other learners. In stacking, several different base models (level 0) make predictions, and a meta-model (level 1) learns the best way to combine them. Unlike a fixed vote, it can learn that one model is great for some cases and another for others.
Build the stack
The key detail: the meta-model must learn from predictions the base models made on data they didn’t train on. Otherwise it learns from over-confident, overfit predictions. scikit-learn’s StackingClassifier handles this with internal cross-validation (cv=5 by default).
Tap the steps in the order they happen.
Code it in Python
The article predicts a sales category for the Carseats-style Company_Data.csv with four base models and a logistic-regression meta-model.
Step 1Prepare the data
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from xgboost import XGBClassifier
from catboost import CatBoostClassifier
company = pd.read_csv("Company_Data.csv")
company["Sales_Category"] = pd.cut(company["Sales"], bins=[0, 5, 10, 15, 20, 25],
labels=False, include_lowest=True)
company = pd.get_dummies(company.dropna(), drop_first=True, dtype=int)
X = company.drop(columns=["Sales", "Sales_Category"])
y = company["Sales_Category"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)Step 2Stack and evaluate
base_estimators = [
("rf", RandomForestClassifier(n_estimators=100, random_state=42)), # bagging
("ada", AdaBoostClassifier(n_estimators=100, random_state=42)), # boosting
("xgb", XGBClassifier()), # boosting
("cat", CatBoostClassifier(verbose=0)), # boosting
]
stack = StackingClassifier(estimators=base_estimators,
final_estimator=LogisticRegression(max_iter=1000), cv=5)
stack.fit(X_train, y_train)
print("Accuracy:", stack.score(X_test, y_test))
print(classification_report(y_test, stack.predict(X_test)))labels=False gives integer classes, which XGBoost requires.
What does cv=5 in StackingClassifier control?
Defeat the Stack Boss 🐲
Six questions, three hearts.
You’re a Stack Master!
Keep levelling up
You learned what meta learning is, the roles of base models and the meta-model, why stacking uses cross-validated (out-of-fold) predictions, and how to build a StackingClassifier.