Gaussian Process Classifier, the game

A distribution over functions

A Gaussian process (GP) is a probability distribution over whole functions. Instead of one fitted curve, you get a mean curve and an uncertainty band. It is defined by a mean function (often zero) and a covariance function, also called the kernel, which says how similar the outputs at two inputs should be.

Kernel playground

This is the GP machinery behind GPC, shown on six example points with an RBF kernel. The length scale ℓ sets how quickly the function can wiggle; the noise σ says how much to trust each point. Notice how the band widens away from the data.

From GP to classifier

For classification, the GP models a hidden latent function f(x). A sigmoid (logistic) link squashes it into a class probability, just like logistic regression. Because the class likelihood isn’t Gaussian, the posterior can’t be computed exactly, so scikit-learn uses the Laplace approximation. For more than two classes it trains one-vs-rest GPs.

• Strengths: honest uncertainty, works well on small data, flexible kernels.
• Weaknesses: training cost grows with n³ and memory with n², so it is impractical for large datasets; results depend on the kernel choice.

Code it in Python

The article predicts 10-year heart-disease risk (TenYearCHD) in the Framingham data with scikit-learn’s GaussianProcessClassifier.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.metrics import classification_report

data = pd.read_csv("framingham.csv").dropna()
X, y = data.drop(columns="TenYearCHD"), data["TenYearCHD"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

gpc = GaussianProcessClassifier(kernel=1.0 * RBF(length_scale=1.0), random_state=42)
gpc.fit(X_train, y_train)            # the length scale is tuned automatically
print(gpc.kernel_)                   # learned kernel
proba = gpc.predict_proba(X_test)[:, 1]
print(classification_report(y_test, gpc.predict(X_test)))

Defeat the GP Boss 🐲

Six questions, three hearts.

Keep levelling up

You learned what a Gaussian process is, how the kernel and its length scale control smoothness, why GPs give honest uncertainty, how GPC squashes a latent GP through a link with the Laplace approximation, and when GPs are (and aren’t) practical.