Ensemble Learning, the game
Why ask one model when you can ask a hundred? Simulate the wisdom of the crowd, then learn the four ways to combine models.
Wisdom of the crowd
An ensemble combines the predictions of several models to get a better prediction than any single one. If each model is right more often than not and their mistakes are independent, a majority vote is right far more often.
The simulator assumes the models make independent errors. Real models are correlated, so real gains are smaller; that is why ensembles try to make their members different.
- ✓Reach 90%+ ensemble accuracy using models that are each only 65% (or less) accurate+30 XP
- ✓Show the dark side: models worse than a coin flip (below 50%) make the vote worse+30 XP
Weak and strong learners
A weak learner is only slightly better than random guessing (for two classes, a bit over 50% accuracy). A decision stump (a tree with a single split) is the classic example. A strong learner achieves high accuracy on its own.
Which is the classic weak learner used in boosting?
Four ways to combine
| Method | How it trains | How it combines | Main effect |
|---|---|---|---|
| Bagging | Same algorithm on different bootstrap samples, in parallel | Vote / average | Reduces variance |
| Boosting | Models in sequence; each focuses on the previous mistakes | Weighted vote | Reduces bias |
| Stacking | Different algorithms; a meta-model learns from their cross-validated predictions | Meta-model | Learns the best combination |
| Blending | Like stacking, but the meta-model learns from a single hold-out set | Meta-model | Simpler, faster stacking |
The article’s boosting analogy: a student fails Physics, Maths and CS in test 1, so they give those subjects more weight for test 2, and so on. Each “test” is a model that focuses on what the previous one got wrong.
Defeat the Ensemble Boss 🐲
Six questions, three hearts.
You’re a Ensemble Master!
Keep levelling up
You learned why combining models works (and when it doesn’t), the difference between weak and strong learners, and how bagging, boosting, stacking and blending combine models.