Linear Regression in Python, the game

The straight-line story

Linear regression models a straight-line relationship between a dependent variable y and one independent variable x (simple linear regression) or several (multiple linear regression).

The relationship can be positive (as x rises, y rises: salary and hours worked) or negative (as x rises, y falls: a laptop’s price as it gets older).

Fit the line by hand

Linear regression picks the line with the smallest sum of squared errors (SSE): square each residual (the red lines) and add them up. This is called ordinary least squares.

These five points are the student data from the original article: x = 1, 2, 3, 4, 5 and y = 2, 4, 5, 4, 6. Move the sliders and try to make the SSE as small as possible.

Do the maths

You can find the best line exactly with two formulas. Using the same five points (n = 5):

• Σx = 1+2+3+4+5 = 15
• Σy = 2+4+5+4+6 = 21
• Σxy = 2+8+15+16+30 = 71
• Σx² = 1+4+9+16+25 = 55

Check the assumptions

Linear regression gives trustworthy results when these assumptions hold:

• Linearity: the relationship between x and y is a straight line.
• Normal residuals: the errors are roughly bell-shaped.
• Constant variance (homoscedasticity): the spread of residuals is the same at every level of x.
• Little or no multicollinearity between features.
• No autocorrelation: one error doesn’t predict the next.

Code it in Python

The original article solves linear regression three ways. Here they are, cleaned up.

def mean(values):
    return sum(values) / len(values)

def variance(values, m):
    return sum((x - m) ** 2 for x in values)

def covariance(x, mean_x, y, mean_y):
    return sum((x[i] - mean_x) * (y[i] - mean_y) for i in range(len(x)))

def coefficients(x, y):
    x_mean, y_mean = mean(x), mean(y)
    b1 = covariance(x, x_mean, y, y_mean) / variance(x, x_mean)
    b0 = y_mean - b1 * x_mean
    return b0, b1

print(coefficients([1, 2, 3, 4, 5], [2, 4, 5, 4, 6]))  # (1.8, 0.8)

import numpy as np
import pandas as pd
import statsmodels.api as sm

data_url = "http://lib.stat.cmu.edu/datasets/boston"
raw_df = pd.read_csv(data_url, sep=r"\s+", skiprows=22, header=None)
data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
target = raw_df.values[1::2, 2]          # median house value

X = sm.add_constant(data)                # adds the intercept column
model = sm.OLS(target, X).fit()
print(model.summary())                   # coefficients, p-values, R²

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(data, target)
predictions = model.predict(data)
print(model.intercept_, model.coef_)

Defeat the Line Boss 🐲

Seven questions, three hearts.

Keep levelling up

You learned the linear regression equation, fitted a line by minimizing squared errors, calculated the slope and intercept by hand, checked the assumptions and ran it in Python three ways.