Stochastic Gradient Descent, the game
Walk downhill, one noisy step at a time. Race different learning rates, watch them crawl, zig-zag or explode, and train an SGDClassifier.
Walk downhill
Training a model means minimizing a loss function: a number that says how wrong the model is (mean squared error for regression, log-loss or hinge loss for classification). Gradient descent repeatedly steps in the direction that lowers the loss fastest:
Learning-rate race
Here the loss is (w − 3)², so the best weight is 3. Start at w = −2 and take 30 steps. Change the learning rate and see what happens. Then switch on the noise to see what “stochastic” looks like.
- ✓Converge smoothly (learning rate ≤ 0.5, no noise)+25 XP
- ✓Converge while zig-zagging over the minimum (rate between 0.5 and 1)+25 XP
- ✓Make it diverge (rate above 1)+20 XP
Batch, mini-batch or stochastic?
| Variant | Gradient computed on | Trade-off |
|---|---|---|
| Batch GD | The whole dataset | Smooth but slow; needs everything in memory |
| Stochastic GD | One row at a time | Fast, noisy updates; can escape shallow minima |
| Mini-batch GD | A small batch (e.g. 32–256 rows) | The practical sweet spot; used in deep learning |
Learning-rate schedules (step decay, exponential decay, “optimal”) start with bigger steps and shrink them so the noisy updates settle down near the minimum.
Code it in Python
The article predicts drug ratings from the drug name and condition with an SGDClassifier. Note: newer scikit-learn renamed loss="log" to loss="log_loss".
Step 1Prepare text features
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.pipeline import Pipeline
data = pd.read_csv("Drug_Data.csv").dropna()
data["features"] = data["drugName"] + " " + data["Prescribed_for"]
X_train, X_test, y_train, y_test = train_test_split(data["features"], data["User_Rating"], test_size=0.2, random_state=42)Step 2Train with SGD
pipeline = Pipeline([
("vectorizer", CountVectorizer()),
("classifier", SGDClassifier(loss="log_loss", learning_rate="optimal", random_state=42)),
])
pipeline.fit(X_train, y_train)
print("Accuracy:", pipeline.score(X_test, y_test))loss="log_loss" makes it logistic regression trained by SGD; loss="hinge" would make it a linear SVM.
Which loss turns SGDClassifier into a linear SVM?
Defeat the Descent Boss 🐲
Six questions, three hearts.
You’re a Descent Master!
Keep levelling up
You learned what a loss function is, how gradient descent steps downhill, the batch / mini-batch / stochastic variants, why the learning rate matters so much, schedules that shrink it, and how to use SGDClassifier.