Beyond Linear Regression - L1 & L2 Regularization From Scratch
Adjusting Linear Regression With L1 and L2
Linear regression is often the first real machine learning model people learn. It’s simple, interpretable, and surprisingly powerful. But once you move past toy datasets, plain linear regression starts to show its limits.
Models overfit. Coefficients explode. Predictions look great on training data and fall apart everywhere else.
That’s where regularization comes in.
Linear Regression, Defined Carefully
At its core, linear regression models the relationship between input variables and a continuous output using a linear equation.
If you have:
Input features:
Output value:
Linear regression assumes:
To learn the weights, linear regression minimizes prediction error over the dataset. The most common choice is Mean Squared Error (MSE).
This loss function:
Penalizes large errors heavily
Is smooth and convex
Has a closed-form solution under ideal conditions
Plain linear regression works well only when several assumptions hold:
Features are not strongly correlated
Noise is limited
The number of features is small relative to data
In real datasets, features overlap and correlate, many features are weak or irrelevant, and noise is unavoidable.
The result is a model with large, unstable coefficients, high variance, and strong overfitting
Regularization directly addresses these problems.
What Is Regularization?
Regularization modifies the learning objective by adding a penalty for large weights.
Instead of minimizing only prediction error, we minimize:
The idea is simple: models with smaller weights tend to be simpler, more stable, and better at generalizing.
The two most important forms are:
L2 regularization (Ridge)
L1 regularization (Lasso)
Stop Guessing. Start Trading with The Signal Beyond report 🚀
COT Data • Technical charts • Equity Arbitrage • Seasonality
L2 Regularization (Ridge Regression)
L2 regularization adds the sum of squared weights to the loss function.
In compact vector notation:
L2 regularization penalizes large weights quadratically. This means:
Very large weights are heavily discouraged
Small weights are only lightly penalized
As a result, L2 regularization shrinks coefficients smoothly toward zero but rarely makes them exactly zero.
What L2 Does Well
Reduces variance
Stabilizes models with correlated features
Keeps all features in the model
L1 Regularization (Lasso Regression)
L1 regularization adds the sum of absolute values of the weights to the loss.
In vector form:
L1 regularization penalizes weights linearly instead of quadratically. This creates a strong incentive for small weights to collapse to exactly zero.
The result:
Sparse models
Built-in feature selection
Easier interpretation
Python Example From Scratch
Below is a minimal implementation showing how plain, L1-, and L2-regularized regression differ.
import numpy as np
# Generate synthetic data
np.random.seed(42)
m = 100
n = 5
X = np.random.randn(m, n)
true_w = np.array([3, 0, 0, 2, 0])
y = X @ true_w + np.random.randn(m) * 0.5
# Add bias column
X = np.hstack([np.ones((m, 1)), X])
def train_linear_regression(X, y, lr=0.01, epochs=1000, l1=0.0, l2=0.0):
w = np.zeros(X.shape[1])
for _ in range(epochs):
y_pred = X @ w
error = y_pred - y
grad = (2 / len(y)) * (X.T @ error)
grad += l1 * np.sign(w)
grad += 2 * l2 * w
w -= lr * grad
return wOptimization Objective Being Minimized
Plain regression:
With L2:
With L1:
Regularization isn’t just a mathematical trick. It encodes assumptions about the world.
L2 regularization assumes:
L1 regularization assumes:
Understanding that difference changes how you use linear models. With the right regularization, linear regression becomes a reliable, sharp tool instead of a fragile one.
Let’s try a simple comparison between the three (using synthetic data):
import numpy as np
import matplotlib.pyplot as plt
# Reproducibility
np.random.seed(0)
# Create synthetic forecasting data
n_samples = 2000
n_features = 20
# Features
X = np.random.randn(n_samples, n_features)
# True sparse relationship
true_w = np.zeros(n_features)
true_w[[1, 4, 7]] = [2.5, -1.8, 3.0]
# Target with noise
y = X @ true_w + np.random.randn(n_samples) * 0.8
# Train-test split
split = int(0.7 * n_samples)
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]
# Add bias term
X_train = np.hstack([np.ones((X_train.shape[0], 1)), X_train])
X_test = np.hstack([np.ones((X_test.shape[0], 1)), X_test])
def train_model(X, y, lr=0.01, epochs=2000, l1=0.0, l2=0.0):
w = np.zeros(X.shape[1])
for _ in range(epochs):
error = X @ w - y
grad = (2 / len(y)) * (X.T @ error)
grad += l1 * np.sign(w)
grad += 2 * l2 * w
w -= lr * grad
return w
# Train models
w_plain = train_model(X_train, y_train)
w_l1 = train_model(X_train, y_train, l1=0.05)
w_l2 = train_model(X_train, y_train, l2=0.1)
# Predictions
pred_plain = X_test @ w_plain
pred_l1 = X_test @ w_l1
pred_l2 = X_test @ w_l2
# Compute MSE
mse_plain = np.mean((y_test - pred_plain) ** 2)
mse_l1 = np.mean((y_test - pred_l1) ** 2)
mse_l2 = np.mean((y_test - pred_l2) ** 2)
# Plot comparison
models = ["Linear", "L1 (Lasso)", "L2 (Ridge)"]
mse_values = [mse_plain, mse_l1, mse_l2]
plt.figure()
plt.bar(models, mse_values)
plt.title("Forecasting Error Comparison (MSE)")
plt.ylabel("Mean Squared Error")
plt.show()
Master Deep Learning techniques tailored for time series and market analysis🔥
My book breaks it all down from basic machine learning to complex multi-period LSTM forecasting while going through concepts such as fractional differentiation and forecasting thresholds. Get your copy here 📖!
This piece really made me think about my early ML headaches. 'Coefficients explode' is truly the most accurate descrption of what happens outside of toy datasets. Plain linear regression really does show its limits fast. Agree completely with your insights here.