Bayesian Linear Regression, the game
Instead of one best line, get a whole distribution of plausible lines. Watch your beliefs update live as data arrives, then fit it in Python with PyMC.
Two schools of statistics
| Frequentist | Bayesian | |
|---|---|---|
| Parameters | Fixed, unknown numbers | Random variables with distributions |
| Prior knowledge | Not used | Built in through the prior |
| Result | A point estimate (plus a confidence interval) | A full posterior distribution |
| Uncertainty | Confidence interval | Credible interval |
Bayesian methods shine with small datasets, when you have expert knowledge to use, and when you need honest uncertainty for decisions (finance, medicine, risk).
Bayes’ theorem
Flip each card to see what the pieces mean for a regression model with parameters θ (the coefficients) and data D.
Update your beliefs
We want the slope β₁ of a line. The true value is 0.8. Set a prior (your guess and how sure you are), then add data and watch the posterior move and narrow.
The data points are simulated from y = 1.8 + 0.8x + noise (σ = 1.5). The intercept is treated as known so you can focus on the slope.
- ✓Add at least 15 data points and watch the posterior narrow+30 XP
- ✓Make a stubborn prior: spread ≤ 0.15 and a guess at least 1 away from 0.8, then add 15+ points+30 XP
Code it in Python
The original article applies PyMC3 to portfolio weights. Here is a direct Bayesian linear regression with the current pymc package (PyMC3 is no longer maintained).
Step 1Make some data
import numpy as np
import pymc as pm
import arviz as az
rng = np.random.default_rng(42)
x = rng.uniform(0, 10, 50)
y = 1.8 + 0.8 * x + rng.normal(0, 1.5, 50)Step 2Priors, likelihood, sample
with pm.Model() as model:
b0 = pm.Normal("b0", mu=0, sigma=10) # priors
b1 = pm.Normal("b1", mu=0, sigma=10)
sigma = pm.HalfNormal("sigma", sigma=5)
pm.Normal("y", mu=b0 + b1 * x, sigma=sigma, observed=y) # likelihood
idata = pm.sample(2000, tune=1000, random_seed=42)
print(az.summary(idata, var_names=["b0", "b1", "sigma"]))The summary shows each coefficient’s posterior mean and a 94% highest-density interval.
Step 3Quick version with scikit-learn
from sklearn.linear_model import BayesianRidge
model = BayesianRidge().fit(x.reshape(-1, 1), y)
mean, std = model.predict([[5.0]], return_std=True) # prediction + uncertaintyIn the PyMC model, which line is the likelihood?
Defeat the Bayes Boss 🐲
Six questions, three hearts.
You’re a Bayes Master!
Keep levelling up
You learned how the Bayesian and frequentist views differ, the four parts of Bayes’ theorem, how a prior turns into a posterior as data arrives, and how to fit a Bayesian linear regression with PyMC.