Ridge and Lasso Regression

Ridge and Lasso Regression are two powerful extensions of ordinary linear regression that add a penalty term to the loss function, discouraging the model from learning overly large coefficients — a technique called regularization. They matter enormously in modern ML because most real-world datasets have noisy or redundant features, and unregularized models overfit badly; Ridge and Lasso are often the first line of defense. The counterintuitive fact: Lasso can completely zero out coefficients, effectively performing automatic feature selection — something no amount of ordinary least squares can achieve.

The core insight

Ordinary linear regression finds the coefficients that minimize prediction error — but it will happily assign huge weights to noisy features if that reduces training error even slightly, leading to a model that falls apart on new data. Ridge and Lasso solve this by adding a "tax" on large coefficients: the model must balance fitting the training data and keeping its weights small. The "aha moment" is realizing that adding a penalty for complexity is not a constraint — it's a trade-off that almost always makes models generalize better. Ridge spreads this penalty across all features; Lasso applies pressure so hard that weak features get driven to exactly zero.

How It Works

Start with ordinary linear regression. We have a dataset of $n$ samples, each with $p$ features, and we want to predict a target $y$ . Ordinary least squares (OLS) minimizes the sum of squared residuals:

\quad \text{Loss}_{\text{OLS}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2

Explanation: This says "find coefficients that make the squared gap between predictions and true values as small as possible on the training data." Here $y_i$ is the true target for sample $i$ , $\hat{y}_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip}$ is the predicted value, and $\beta$ are the coefficients (weights).

Ridge Regression adds an L2 penalty — the sum of squared coefficients — multiplied by a strength hyperparameter $\alpha$ :

\quad \text{Loss}_{\text{Ridge}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^{p}\beta_j^2

Explanation: The model pays a cost for every large coefficient, so it learns to be "humble" about the weights it assigns.

Lasso Regression swaps in an L1 penalty — the sum of absolute values of coefficients:

\quad \text{Loss}_{\text{Lasso}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^{p}|\beta_j|

Explanation: Using absolute value instead of squared value creates a geometric shape (a diamond in 2D) whose sharp corners sit exactly on the axes, pulling weak coefficients all the way to zero. In both equations:

$\alpha \geq 0$ is the regularization strength — a hyperparameter you choose. $\alpha = 0$ recovers OLS; larger $\alpha$ = stronger penalty.
$\beta_j$ are the model coefficients (one per feature).
The intercept $\beta_0$ is typically not penalized.

Here is how you use both in Python with scikit-learn: ```python import numpy as np from sklearn.linear_model import Ridge, Lasso from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.datasets import make_regression

# Generate synthetic data: 100 samples, 20 features, only 5 truly matter X, y = make_regression(n_samples=100, n_features=20, n_informative=5, noise=10, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42)

# IMPORTANT: always scale features before regularization scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test)

# Ridge Regression ridge = Ridge(alpha=1.0) ridge.fit(X_train, y_train) print("Ridge R²:", ridge.score(X_test, y_test)) print("Ridge non-zero coefficients:", np.sum(ridge.coef_ != 0)) # always 20

# Lasso Regression lasso = Lasso(alpha=1.0) lasso.fit(X_train, y_train) print("Lasso R²:", lasso.score(X_test, y_test)) print("Lasso non-zero coefficients:", np.sum(lasso.coef_ != 0)) # likely ~5! ``` ```python

# Choosing alpha with cross-validation (the right way) from sklearn.linear_model import RidgeCV, LassoCV ridge_cv = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0], cv=5) ridge_cv.fit(X_train, y_train) print("Best Ridge alpha:", ridge_cv.alpha_) lasso_cv = LassoCV(alphas=[0.001, 0.01, 0.1, 1.0], cv=5, max_iter=5000) lasso_cv.fit(X_train, y_train) print("Best Lasso alpha:", lasso_cv.alpha_) ```

researcher · Robert Tibshirani | b. 1956 | Inventor of Lasso Regression

Robert Tibshirani is a Canadian statistician at Stanford University, best known for his landmark 1996 paper "Regression Shrinkage and Selection via the Lasso" in the Journal of the Royal Statistical Society. He recognized that Ridge regression's inability to zero out coefficients was a practical limitation for high-dimensional data, and formulated the L1 penalty as a mathematically elegant solution that simultaneously shrinks and selects. Surprisingly, Tibshirani originally submitted the Lasso paper to JASA (a different journal) and it was rejected — it was accepted by JRSS-B and went on to become one of the most cited papers in statistics history.

Figure

Ridge vs Lasso Constraint Regions

Intuition

The Mental Model

Imagine you are packing a suitcase and you get charged by weight. You could pack everything — but the airline charges extra for every kilogram. So you start leaving things out unless they're truly essential. That's regularization. Ridge is like a progressive weight tax: every item still makes it in, but heavier items get a bigger penalty, so you pack lighter versions of everything. Lasso is like a strict baggage allowance: items that don't justify their weight get left at home entirely — they're zeroed out. The hyperparameter $\alpha$ is the price per kilogram: a tiny $\alpha$ means you barely care about weight (close to OLS), a huge $\alpha$ means you travel almost empty. Feature scaling is like converting all your items to the same weight unit before deciding — without it, a feature measured in kilometers would get penalized far less than one measured in meters, just because of its units, which is completely arbitrary.

Historical Development

Origins · 1 minhow this was discovered

History

1943

Andrey Tikhonov begins theoretical work on regularization for ill-posed problems in mathematics — the foundation Ridge regression is built on (later called Tikhonov regularization in his honor).

1959

Norbert Wiener's work on signal processing establishes the idea that prior assumptions about smoothness can improve estimation — an early probabilistic framing of regularization.

1970

Hoerl and Kennard publish "Ridge Regression: Biased Estimation for Nonorthogonal Problems" in Technometrics, formally introducing Ridge regression to statistics and showing it can outperform OLS when predictors are correlated.

1986

Leo Breiman proposes the "Garrote" estimator, an early attempt at a regression method that can set coefficients to zero — a conceptual predecessor to Lasso that didn't fully succeed.

1996

Robert Tibshirani publishes "Regression Shrinkage and Selection via the Lasso" in JRSS-B, introducing Lasso regression and proving it performs simultaneous shrinkage and variable selection via the L1 penalty.

2004

Zou and Hastie introduce Elastic Net regression, combining L1 and L2 penalties in a single model to capture the benefits of both Ridge and Lasso — especially useful when features are correlated.

2007

The LARS (Least Angle Regression) algorithm by Efron et al. provides an efficient computational path for solving Lasso across all values of

\alpha

, making Lasso practical for large datasets.

2010

s: Regularized regression becomes a standard first-step baseline in genomics, neuroscience, and finance — fields with "wide" data (many more features than samples), where unregularized models are mathematically impossible to fit.

2020

s: Ridge and Lasso remain foundational building blocks; their penalty concepts directly inspired weight decay and L1 sparsity in deep neural networks, keeping them highly relevant in modern AI.

Why It Matters

Real-World Impact

Genomics / GWAS StudiesLasso regression is used to identify which of hundreds of thousands of genetic variants (SNPs) are associated with a disease — it zeroes out the vast majority of irrelevant genes and retains the handful that matter, making the problem tractable.

Finance — Factor ModelsRidge regression is used in portfolio optimization to estimate return covariance matrices that are stable under correlated financial features, preventing the model from making extreme bets based on spurious historical correlations.

Real Estate Pricing (Zillow-style models)Ridge regression handles the many correlated features in property data (square footage, number of rooms, lot size all move together) — it keeps all features but stabilizes coefficients so the model doesn't blow up.

Text Classification / Spam DetectionLasso logistic regression (the classification analogue) is applied to bag-of-words features in email filtering — with 50,000 possible words as features, Lasso automatically identifies the ~200 words that signal spam and zeros out the rest.

Clinical Prediction ModelsRidge regression is preferred in clinical risk scores (predicting hospital readmission, ICU mortality) when all measured variables are expected to contribute something, and dropping any feature would introduce bias.

scikit-learn's Default Pipelines`RidgeCV` and `LassoCV` are among the most commonly used estimators in scikit-learn production pipelines — they are fast, interpretable baselines that frequently outperform unregularized linear models with zero tuning effort beyond cross-validating

\alpha

Elastic Net in Coordinate Descent SolversThe coordinate descent algorithm used to solve Lasso inspired the optimization backend of scikit-learn's `SGDClassifier` with `penalty='elasticnet'`, which powers linear models at internet scale.

Ridge and Lasso Regression

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this

Where this leads