MerQuR+Statistics Glossary

Statistics Glossary → Lasso Regression

Lasso Regressionanalysis

A regression method that shrinks the coefficients with a penalty, setting some of them exactly to zero, and thereby selects the predictors needed for prediction from many candidates.

Concept

When the number of candidate predictors approaches or exceeds the number of observations, ordinary regression overfits: it explains the data at hand almost perfectly but predicts new data poorly. It also becomes hard to tell which variables really matter. The lasso (Least Absolute Shrinkage and Selection Operator) both stabilises prediction by shrinking the coefficients and drops unnecessary variables from the model.

The lasso adds the sum of the absolute values of the coefficients as a penalty to the residual sum of squares: Σ(y − ŷ)² + λ·Σ|β| (up to constants and scaling). Unlike the squared penalty in ridge, the absolute-value penalty pushes some coefficients exactly to zero once λ is large enough, and those variables leave the model. The larger λ, the fewer variables remain. Predictors are standardised beforehand and λ is usually chosen by cross-validation; the 1-SE rule picks a somewhat simpler model.

The lasso's selection shows a sparse predictive model supported by the data, not a definitive truth. Among strongly correlated variables it often keeps one and drops the other, and this choice can change from sample to sample, so selection stability should be examined separately. Because the lasso coefficients are deliberately shrunk, classical p values and confidence intervals are not valid; even in an ordinary model refitted on the selected variables, p values are optimistic because they ignore the selection step.

When to use it

  • When there are many candidate predictors and you want a model that predicts well yet contains only a few variables (e.g. predicting yield from dozens of soil and climate measures, or risk from many biomarkers).
  • If the predictors are strongly correlated and keeping them all in the model is acceptable, ridge is preferred; to select while keeping groups of correlated variables together, use the elastic net.
  • If there are few predictors chosen on theoretical grounds, ordinary multiple linear regression is better for interpreting and testing the coefficients.

An everyday example

A farm wants to predict wheat yield on 80 plots from 25 measurements such as soil pH, organic matter, nitrogen, phosphorus and potassium levels, sowing date, rainfall and temperature. With 80 plots and 25 variables, ordinary regression overfits. At the penalty chosen by cross-validation, the lasso might keep only nitrogen, rainfall, sowing date and organic matter and set the other coefficients to zero. The farm thus gets a simple predictive model that suggests which measurements to prioritise next season.

Parameters in MerQur

Independents (Features) and Target column

Concept: The predictors are the candidate variables and the target is the outcome being predicted. Because the penalty applies equally to all coefficients, the predictors are standardised, so whether a variable is selected does not depend on its unit of measurement.

Everyday example: In the farm example the target is wheat yield and the predictors are 25 soil and climate measurements. Rainfall is in millimetres and pH has no unit, yet standardisation lets both be judged by the same yardstick.

What it does: Selects the predictors and the dependent variable; text columns are dummy-coded automatically.

Effect on the result: The penalty is applied to all standardized columns.

α (regularization)

Concept: The penalty (λ); the larger it is, the more coefficients are set exactly to zero. At a large enough value every coefficient is zero and the model predicts only the mean.

Everyday example: With a small penalty, 20 of the 25 measurements may stay in the model; as it increases the weakest drop out first, until only a few of the strongest, such as nitrogen and rainfall, remain.

What it does: The L1 penalty coefficient: (1/2n)||y - X_s b||² + α|b|₁; as it grows more coefficients become exactly zero. The meaningful range is 0 to λ_max; alpha is the λ of this objective. If α >= λ_max an "all coefficients are zero" warning is written.

Effect on the result: Coefficients, selected variables and R² change. The convergence tolerance is 1e-12; a warning is written if it does not converge.

On the validation data: QuickStart: α = 0.001 -> 20/20 nonzero, R² 0.9132; 0.1 -> 9/20, R² 0.8994; 1 -> 2/20. Prostate 0.1 -> 5/8. Hitters CHmRun -0.0664293.

→ Regularization (penalty) and the penalty parameter λ/α

Auto α (cross-validation)

Concept: Choosing the penalty by fitting the model on part of the data and examining its prediction error on the rest, so that λ is set to give the best prediction on new data.

Everyday example: The plots are split into five groups; each time the model is fitted on four and yield is predicted in the fifth. This is repeated for different penalties and the one with the smallest average prediction error is chosen.

What it does: Selects alpha from a log-scale grid (from λ_max to λ_max × 1e-4, 100 points) by k-fold CV MSE; it rescales in each training fold and averages with fold-size weights.

Effect on the result: The selected alpha and all results that depend on it change.

On the validation data: QuickStart 5-fold λ_min 0.0830733 (9 variables), prostate 10-fold 0.00578245 (7/8), Hitters 10-fold 2.67438 (13/19).

→ Regularization (penalty) and the penalty parameter λ/α

CV folds

Concept: Sets how many parts the data are split into for cross-validation. The chosen penalty, and hence the selected variables, can change slightly with the number of folds, so it should be reported.

Everyday example: With 80 plots in 5 folds, 4 variables might be selected, and with 10 folds, 5; the difference usually comes from a weak variable at the margin.

What it does: Sets the number of consecutive (unshuffled) folds (3-20, default 5).

Effect on the result: Affects auto alpha, CV performance and stability; no effect when CV is off.

On the validation data: QuickStart λ_min: K = 3 -> 0.0628419 (11 variables), 5 -> 0.0830733, 10 and 20 -> 0.0756933.

→ k-fold cross-validation

λ selection rule (λ_min, λ_1SE)

Concept: λ_min is the penalty that minimises the cross-validation error. λ_1SE is the largest penalty whose error lies within one standard error of the minimum; it gives a model with statistically similar predictive performance but fewer variables.

Everyday example: λ_min might give a seven-variable model and λ_1SE a five-variable one with almost the same prediction error. If the farm wants to take fewer measurements, it prefers the second model.

What it does: λ_1SE selects the largest λ with CV MSE <= min + SE(min) (Breiman et al., 1984); a sparser model.

Effect on the result: Changes the selected alpha and the model only when auto alpha is on.

On the validation data: QuickStart 5-fold: 0.0830733 (9 variables, R² 0.9023) -> 0.145173 (8, R² 0.8901). Prostate: 0.00578 (7/8) -> 0.164686 (5/8).

→ One-standard-error (1-SE) rule

Coefficient path

Concept: Shows how each coefficient changes as the penalty grows and at which penalty each variable becomes non-zero. Variables entering early are those most strongly related to the outcome.

Everyday example: The coefficient path might show that nitrogen stays in the model even at the largest penalty, while phosphorus enters only when the penalty becomes small. This order hints at which measurement is most indispensable for predicting yield.

What it does: Gives the standardized coefficient path along the grid and the order in which variables enter the model (the largest λ at which they leave zero).

Effect on the result: An extra table and plot; for generalized linear model families the k-fold deviance curve is also given.

On the validation data: Prostate entry order: lcavol (0.8008), lweight (0.4175), svi (0.3466), pgg45 and lbph (0.1984).

→ Coefficient path and ridge trace

Target type (Continuous (linear), binary (logistic), count (Poisson))

Concept: The lasso penalty can also be used for binary (logistic) and count (Poisson) outcomes; it is added to the likelihood of these models and variables are selected in the same way.

Everyday example: If the farm wants to answer 'did disease appear on the plot?' instead of predicting yield, the outcome is binary; a penalised logistic model selects the measurements among the 25 that best indicate disease risk.

What it does: Fits a penalised logistic model (-l/n + α|b|₁) for a binary target and a penalised Poisson model for a count target.

Effect on the result: The model family, deviance and D² change.

On the validation data: Binomial α = 0.1: deviance 99.028, D² 0.2781, 8/30; Poisson α = 1: D² 0.9404, 6/20.

→ Distribution family (GLM)

Categorical predictors

Concept: Expands categorical predictors into dummy variables. Because the dummies are penalised separately, some levels of the same variable may stay in the model while others drop out.

Everyday example: Soil type ('clay / loam / sand') enters as two dummies. The lasso may keep the 'sand' dummy and zero the 'loam' dummy, which means loam gives yields similar to the reference level.

What it does: Dummy-codes the named columns (even if numeric), first level as the baseline. Dummies are penalised individually (not group lasso).

Effect on the result: The dummy design and the selection change.

On the validation data: Prostate "gleason" -> 3 dummies; 6/10 nonzero, R² 0.6594.

→ Dummy (indicator) variable coding

Extra output: CV performance

Concept: Summarises how well the model with the chosen penalty predicts data not used to fit it (mean squared error, cross-validated R²). It is a more honest measure than the fit to the training data.

Everyday example: If the model explains 80% of yield differences on the plots it was fitted on but 70% on unseen plots, the performance to expect next season is close to the second figure.

What it does: Out-of-fold prediction with the selected alpha over consecutive k folds: MSE, R²_CV; for GLMs mean unit deviance and classification error.

Effect on the result: An extra table.

On the validation data: QuickStart α = 0.1, K = 5: MSE 1.1289, R²_CV 0.8676, df 9; K = 10: MSE 1.1321.

→ Prediction error measures (RMSE, MAE, MAPE, MASE)

Extra output: OLS on selected

Concept: An ordinary, unpenalised model is refitted with the variables the lasso selected, so the coefficients can be interpreted without shrinkage. Because the variables were selected on the same data, the p values and confidence intervals of this model are optimistic.

Everyday example: If the lasso selected nitrogen, rainfall, sowing date and organic matter, an ordinary regression is fitted on these four and 'how much does each extra kilogram of nitrogen raise yield?' is answered with an unshrunk coefficient. Confirming it requires data from a new season.

What it does: Unpenalised OLS/generalized linear model refit on the variables that remain nonzero (relaxed lasso, γ = 0); p and confidence intervals ignore the selection (a warning is written).

Effect on the result: An extra table.

On the validation data: Prostate s = .36: 3 variables, R² 0.6374; lcavol B 0.51999 (SE 0.09442). QuickStart α = 0.1: 9 variables, R² 0.9087.

→ Ordinary least squares (OLS)

Extra output: Selection stability

Concept: Reruns the lasso on many random subsamples of the data and shows how often each variable is selected. Frequently selected variables are reliable, while occasionally selected ones depend on the particular sample.

Everyday example: If nitrogen is selected in 100% of subsamples, rainfall in 90% and phosphorus in 35%, the importance of the first two is solid; whether phosphorus appears depends largely on which plots happen to be in the data.

What it does: Gives how often each variable is selected with the same alpha in 100 subsamples (floor(n/2), without replacement, seed 42); >= 0.6 counts as stable. Not computed for binary/Poisson targets (a warning is written).

Effect on the result: An extra table; each subsample is rescaled.

On the validation data: QuickStart α = 0.1: 1.00 for V1, V3, V5, V6, V8, V14, V20; 0.76 for V11. Prostate s = .36: lcavol 1.00, lweight 0.92, svi 0.70.

→ Selection stability

Example on real data

Data: Stamey et al. 1989 prostate data (ESL training set)

Variables: target lpsa (log PSA), eight clinical predictors (lcavol, lweight, age, lbph, svi, lcp, gleason, pgg45)

  • Question: In the prostate data (n = 67), which of the eight predictors are needed to explain PSA level (lpsa)? The lasso L1 penalty both shrinks and selects variables; predictors were standardized with the 1/n-divisor SD, penalty α = 0.1, intercept unpenalised.
  • Assumption: α = 0.1 was entered manually; in a predictive study α should be chosen by k-fold cross-validation (λ_min or λ_1SE) and selection stability should be checked with subsamples. Classical SE/p values are not interpreted for lasso coefficients.
  • Result: Five predictors remain in the model: lcavol (0.4627), lweight (0.4833), svi (0.4102), lbph (0.0723), pgg45 (0.0022); age, lcp and gleason are shrunk exactly to zero.
  • Effect size: R² = 0.6478, adjusted R² = 0.5992 (with p = 8); slightly lower fit than ridge (R² = 0.694) in exchange for a simpler model.
  • APA: Lasso regression (α = 0.1, standardized predictors) selected 5 of 8 predictors, R² = .65.

Related concepts

Regularization (penalty) and the penalty parameter λ/αSelection stabilityCoefficient path and ridge tracek-fold cross-validationOne-standard-error (1-SE) rulePrediction error measures (RMSE, MAE, MAPE, MASE)Distribution family (GLM)Ordinary least squares (OLS)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.