Ridge Regression in R, the game
Tame overfitting with the L2 penalty. Slide λ and watch every coefficient shrink without ever hitting zero, then fit Ridge in R with glmnet.
The L2 penalty
Ridge regression is linear regression with an extra penalty on the squared size of the coefficients (L2 regularization):
- λ = 0: no penalty, so it is ordinary least squares.
- Larger λ: coefficients shrink towards zero, adding a little bias but cutting variance.
- Huge λ: coefficients get tiny and the model underfits.
Ridge adds a penalty on…
Slide the λ lever
Slide λ to the right and watch all ten coefficients shrink. Notice what never happens.
- ✓Shrink every coefficient below 0.5 (standardized)+30 XP
- ✓Max out λ: how many features are still in the model?+20 XP
- ✓Go back to λ ≈ 0 (plain least squares)+20 XP
The playground uses the same algorithm as glmnet (coordinate descent on standardized features, 100 λ values) on R’s built-in mtcars data, predicting mpg from the other 10 columns. At the smallest λ the coefficients match ordinary least squares.
Ridge vs Lasso
Both penalize large coefficients. The difference is the shape of the penalty: Lasso’s absolute value (L1) can push coefficients to exactly zero; Ridge’s square (L2) only shrinks them. That makes Ridge great for multicollinearity: correlated features share the weight instead of one being dropped at random.
Code it in R
Step 1Load the data
library(glmnet)
library(Metrics)
library(caTools)
data(mtcars)
X <- as.matrix(mtcars[, -1]) # all columns except mpg
y <- mtcars$mpgStep 2Split into train and test
set.seed(123)
split <- sample.split(y, SplitRatio = 0.7)
x_train <- X[split, ]; x_test <- X[!split, ]
y_train <- y[split]; y_test <- y[!split]Step 3Fit Ridge and pick λ by cross-validation
lambdas <- 10^seq(3, -2, by = -0.1)
ridge <- glmnet(x_train, y_train, alpha = 0, lambda = lambdas, family = "gaussian")
cv_model <- cv.glmnet(x_train, y_train, alpha = 0, lambda = lambdas, nfolds = 5)
best_lambda <- cv_model$lambda.min
cat("Best lambda:", best_lambda, "\n")With only about 22 training rows, 5 folds keeps each fold big enough.
Step 4Predict and evaluate
predictions <- predict(ridge, s = best_lambda, newx = x_test)
cat("MSE: ", mse(y_test, predictions), "\n")
cat("RMSE:", rmse(y_test, predictions), "\n")
coef(ridge, s = best_lambda) # every feature keeps a non-zero coefficientIn glmnet(), which argument makes the model Ridge?
Defeat the Ridge Boss 🐲
Seven questions, three hearts.
You’re a Ridge Master!
Keep levelling up
You learned the Ridge cost function, how λ trades variance for bias, why Ridge shrinks but never drops features, how it handles multicollinearity, and how to fit and tune it with glmnet.