Random Forest in Python, the game

Bootstrap sampling

• Sampling without replacement: once a row is picked, it can’t be picked again.
• Sampling with replacement (the bootstrap): a row goes back into the bag after it is picked, so it can appear several times.
• Out-of-bag (OOB) rows: the rows a bootstrap sample happened not to pick, roughly 37% of the data. They act as a free test set for that tree.

From bagging to a forest

Bagging trains many decision trees on different bootstrap samples and lets them vote. A random forest adds one more twist: at every split, each tree may only consider a random subset of the features. That makes the trees more different from each other, so their errors cancel out better.

• Classification: majority vote of the trees.
• Regression: average of the trees.
• Key settings: n_estimators (number of trees), max_depth, max_features.

Grow the forest

Each tree here is fully grown and sees a bootstrap sample with one random feature per split. Switch from 1 tree to 100 and watch the decision regions.

The playground uses R’s built-in iris data (150 flowers, 3 species) with just two features, sepal length and width, so you can see the decision regions. Every third flower is held out as a test set.

Code it in Python

The article classifies Fashion-MNIST clothing images (10 classes). This version fixes two bugs in the original: it trains on the class labels (not one-hot vectors) and actually calls predict().

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

train = pd.read_csv("fashion-mnist_train.csv")
test  = pd.read_csv("fashion-mnist_test.csv")
X_train, y_train = train.iloc[:, 1:] / 255, train.iloc[:, 0]
X_test,  y_test  = test.iloc[:, 1:] / 255,  test.iloc[:, 0]

clf = RandomForestClassifier(n_estimators=100, max_depth=50, criterion="entropy",
                             oob_score=True, n_jobs=-1, random_state=42)
clf.fit(X_train, y_train)
print("Out-of-bag accuracy:", clf.oob_score_)

y_pred = clf.predict(X_test)
print(accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))

Defeat the Forest Boss 🐲

Six questions, three hearts.

Keep levelling up

You learned sampling with and without replacement, bootstrap and out-of-bag samples, how bagging plus random feature selection builds a random forest, and how to train one in scikit-learn.