MerQuR+Statistics Glossary

Statistics Glossary → Regularized Regression (LASSO / Ridge / ElasticNet)

Regularized Regression (LASSO / Ridge / ElasticNet)analysis

Regression methods (LASSO, Ridge, ElasticNet) that prevent overfitting when there are many correlated predictors by penalising the size of the coefficients, and that, in variants such as LASSO, also select predictors.

Concept

As the number of predictors grows, ordinary least squares (OLS) fits the data at hand ever better, but part of that fit reflects random noise rather than the real relation, and the model predicts poorly on new data (overfitting). When predictors are strongly correlated, the coefficients become unstable and may even change sign with small changes in the data. When the number of predictors approaches or exceeds the number of observations, OLS cannot be used at all. Regularised regression responds to these problems by deliberately shrinking the coefficients.

The method adds to the residual sum of squares a penalty that depends on the size of the coefficients and minimises the total: Σ(y − ŷ)² + λ · penalty. The Ridge penalty is the sum of squared coefficients (L2); it shrinks all coefficients towards zero without making any exactly zero, and shares the effect among correlated predictors. The LASSO penalty is the sum of absolute coefficients (L1); it sets some coefficients exactly to zero and so selects predictors. ElasticNet mixes the two and selects more stably than LASSO within groups of correlated predictors. The penalty strength λ is usually chosen by cross-validation, so as to give the best prediction on new data.

Because the penalty applies equally to all coefficients, predictors should be standardised first; otherwise variables measured in small units are penalised unfairly. Shrunken coefficients are deliberately biased: the aim is not unbiased estimates of individual coefficients but lower prediction error. Ordinary p values and confidence intervals therefore do not apply directly to these models. The set of variables LASSO selects can be sensitive to the data; looking at selection stability with the bootstrap shows which predictors are truly reliable.

When to use it

  • When you want a model that predicts well from many correlated predictors and selects which predictors matter; when the number of predictors is close to or larger than the number of observations.
  • If there are few predictors, relations are modest and the aim is to interpret and test coefficients theoretically, ordinary multiple regression is more suitable.
  • If the predictors are very highly correlated and numerous (e.g. spectral data), consider partial least squares regression; if relations are nonlinear, flexible methods such as random forests.

An everyday example

A supermarket chain wants to predict its branches' weekly sales from 40 candidate variables: price cuts, leaflet campaigns, temperature, rainfall, holidays, competitor openings and more. With only 120 weeks of data, a 40-variable OLS model fits the past very well but predicts future weeks poorly. LASSO, with a penalty chosen by cross-validation, might set most coefficients to zero and keep, say, only the discount rate, holidays and temperature. The result is a simple model that both predicts better and is easy for managers to understand.

Parameters in MerQur

Target column and Target distribution (Continuous, Binary, Count)

Concept: The target distribution sets the model suited to the outcome type: linear regression for continuous outcomes, logistic regression for binary outcomes, Poisson regression for counts. The penalty logic is the same in all three; what changes is the loss function measuring model fit.

Everyday example: Weekly sales revenue is a continuous target. If the question were 'did the branch hit its sales target this week?', a binary target would be chosen; if it were 'how many customer complaints came in this week?', a count target.

What it does: Selects the dependent variable and the distribution family. For binary and count targets a penalized-likelihood generalized linear model (binomial / Poisson family) is built; the CV criterion is deviance.

Effect on the result: All outputs.

On the validation data: Binary LASSO: lambda_min 0.025785, lambda_1se 0.049454, 17/30 variables, deviance 48.2255. Poisson lambda_min 0.080441.

→ Distribution family (GLM)

Method (LASSO, Ridge, ElasticNet)

Concept: The method sets the type of penalty applied to the coefficients. Ridge shrinks all coefficients but zeroes none; LASSO sets some exactly to zero and so selects variables; ElasticNet mixes the two and can keep groups of correlated variables together.

Everyday example: If temperature and 'season' carry almost the same information, LASSO may pick one of them almost arbitrarily and set the other to zero. Ridge gives both small coefficients; ElasticNet tends to keep the two together.

What it does: Selects the penalty type. LASSO (L1) selects variables by shrinking coefficients to zero; Ridge (L2) shrinks all coefficients without zeroing them; ElasticNet is a mixture of the two (Zou and Hastie, 2005). The Ridge alpha is not on the same scale as the LASSO / ElasticNet alpha (on the standardized scale lambda = a s_y / n).

Effect on the result: Coefficients, the number of selected variables, the alpha selected by CV.

On the validation data: QuickStart CV: LASSO alpha 0.08307, 9/20 variables, R2(CV) 0.8471; Ridge alpha 5.7263, 20/20, 0.8260; ElasticNet (L1 ratio 0.5) alpha 0.0950752.

→ Regularization (penalty) and the penalty parameter λ/α

α (alpha)

Concept: The penalty strength sets how much the coefficients are shrunk. As it approaches zero the result approaches ordinary regression; as it grows the result approaches a constant prediction. The best value is usually chosen by cross-validation.

Everyday example: If the penalty is too weak, the sales model keeps all 40 variables and memorises past weeks. If it is too strong, every coefficient is zeroed and the model predicts the same average sales for every week. A value in between predicts future weeks best.

What it does: The penalty strength. 0 = select by CV (default); a positive value applies a fixed penalty.

Effect on the result: As it grows, coefficients shrink; if alpha is above lambda_max, all coefficients become 0 and a warning is given. The unit varies by method.

On the validation data: QuickStart LASSO alpha = 0.0001: 20/20 variables; 0.1: 9/20; 0.5: 6/20; 10 and 100: 0/20 and the warning "alpha >= lambda_max = 1.63076". Hitters alpha = 1: CHmRun 0.176368.

→ Regularization (penalty) and the penalty parameter λ/α

L1 ratio (ElasticNet)

Concept: The L1 ratio sets how much of the ElasticNet penalty is of the LASSO (L1) type and how much of the Ridge (L2) type. As the ratio approaches 1 the model keeps fewer variables; as it approaches 0 it behaves like Ridge.

Everyday example: With an L1 ratio of 0.9 the model works largely like LASSO, but its tendency to drop one of a correlated pair such as temperature and season somewhat arbitrarily is softened.

What it does: The ElasticNet mixture (1 = LASSO, 0 = pure Ridge). Ignored in LASSO and Ridge.

Effect on the result: The number of selected variables and coefficients.

On the validation data: ElasticNet CV: 19/20 variables with L1 ratio 0.2; 13 with 0.5; 9 with 0.9; 1.0 = LASSO. L1 ratio 0: pure Ridge, alpha 0.16308 at the lower end of the grid (warning).

→ Elastic net and the L1 ratio

Select L1 ratio by CV (ElasticNet)

Concept: Choosing the L1 ratio by cross-validation too lets the data decide how much of the mix is LASSO and how much Ridge. The best penalty strength is found for each candidate ratio, and the ratio-strength pair with the smallest error on held-out data is chosen.

Everyday example: If the analyst does not know whether to pick 0.5 or 0.9, several ratios are tried together, and the mix that best predicts the held-out weeks is chosen.

What it does: Selects the L1 ratio by cross-validation from the grid {0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1} (if alpha is given manually, only the L1 ratio is selected). It also works for the binary target.

Effect on the result: The selected ratio, alpha and coefficients; no change when LASSO is selected or for the Poisson target.

On the validation data: In QuickStart the L1 ratio 1.0 is selected. In the binary example the selected ratio is 0.1, lambda_min 0.0483173.

→ Elastic net and the L1 ratio

CV folds

Concept: k-fold cross-validation splits the data into k parts; each time the model is built on k − 1 parts and tested on the remaining one, and the errors are averaged. This measures how well a penalty value predicts on data not used to build the model.

Everyday example: If the 120 weeks are split into 5 parts, the model is built on 96 weeks each time and predicts the remaining 24. The average error over the five runs indicates how well that penalty value will work in future weeks.

What it does: Sets the number of folds for alpha selection, the CV curve, R2(CV) and RMSE(CV) in consecutive (unshuffled) k-fold cross-validation. 5-10 is usual; shuffling should be turned on for ordered data.

Effect on the result: The selected alpha.

On the validation data: QuickStart LASSO alpha: 0.05217 for k = 2; 0.08307 for k = 5; 0.07569 for k = 10. Because the prostate data are ordered by lpsa, R2(CV) is -8.09 in consecutive folds.

→ k-fold cross-validation

Standardize (recommended)

Concept: Standardisation rescales each predictor to mean 0 and standard deviation 1. Since the penalty depends on coefficient size, without it the choice of measurement units decides which variable is penalised more.

Everyday example: If temperature is measured in degrees Celsius and advertising budget in lira, without standardisation the budget's coefficient looks tiny and is hardly penalised at all. Standardisation ensures both variables are treated equally.

What it does: Scales the predictors by 1/n sd (the penalty is applied equally to all predictors); coefficients are converted back to the original scale.

Effect on the result: The distribution of the penalty and the selection; pronounced for predictors with different scales.

On the validation data: QuickStart (X on a similar scale): CV alpha 0.07884 / 0.08116 (small effect).

→ z-score (standardization)

Chart (Coefficient bars, Coefficient path, CV error curve)

Concept: Coefficient bars show the selected model's coefficients, the coefficient path shows how each coefficient shrinks towards zero as the penalty grows, and the cross-validation curve shows prediction error against penalty strength.

Everyday example: If, on the coefficient path, the discount rate's coefficient does not reach zero until the penalty is very strong, that variable is the most durable predictor of sales. The coefficient for competitor openings, by contrast, may vanish as soon as the penalty rises a little.

What it does: Selects the chart type.

Effect on the result: Graphics only; the numerical output does not change.

On the validation data: No numerical difference.

→ Coefficient path and ridge trace

α/λ selection rule (Minimum CV error, 1-SE rule)

Concept: The minimum cross-validation error rule picks the penalty that minimises prediction error. The 1-SE rule picks the strongest penalty whose error stays within one standard error of the best; it gives a simpler model with no meaningful loss in predictive accuracy.

Everyday example: The minimum-error rule might keep 9 variables in the sales model, while the 1-SE rule achieves almost the same accuracy with 4. For a model to be explained to managers, the 4-variable version is more practical.

What it does: Selects the minimum error on the CV curve or the 1-SE rule (the largest penalty within one SE of the minimum error, a sparser model).

Effect on the result: The selected penalty and the number of variables; no effect if alpha is given manually.

On the validation data: QuickStart LASSO: minimum error 0.08307 (9 variables), 1-SE 0.14517 (8 variables); binary 0.025785 and 0.049454.

→ One-standard-error (1-SE) rule

Shuffle CV folds (seeded) and CV / bootstrap seed

Concept: Shuffling the folds assigns observations to cross-validation parts at random; if the data are sorted by some variable or by time, consecutive parts are unbalanced and the error estimates mislead. The seed makes the same random assignment reproducible.

Everyday example: If the data are sorted by sales amount, the first unshuffled part contains only the weeks with the lowest sales, and the model looks absurdly bad when predicting it. Shuffling ensures each part gets a mix of different kinds of weeks.

What it does: Assigns folds randomly with a seed; the seed also determines the bootstrap sampling. When shuffling is off, the seed affects only the bootstrap.

Effect on the result: The selected alpha.

On the validation data: QuickStart LASSO alpha: 0.08307 without shuffling; with shuffling 0.10006 for seed 42 and 0.07569 for seed 7.

→ k-fold cross-validation

CV repeats

Concept: Cross-validation can give slightly different results depending on how observations happen to be assigned to the parts. Repeating it with several different assignments and averaging reduces the dependence of the chosen penalty on this chance.

Everyday example: One assignment might give a best penalty of 0.05 and another 0.08. Repeating cross-validation 10 times and averaging gives a more stable penalty value for the sales model.

What it does: Averages the CV curves and SEs of several fold assignments; if the number of repeats is greater than 1, the folds are shuffled at each repeat.

Effect on the result: The selected alpha.

On the validation data: 3 repeats, alpha 0.09117.

→ k-fold cross-validation

Bootstrap stability (B)

Concept: Selection stability shows how often each predictor is selected when many bootstrap samples are drawn from the data and the model is refitted in each. Predictors chosen in nearly every sample are reliable; those chosen only occasionally may owe their selection to quirks of the data.

Everyday example: If the discount rate is selected in 98% of 500 bootstrap samples and competitor openings in only 35%, the discount rate can be trusted for predicting sales, whereas competitor openings entering the model is largely a matter of chance.

What it does: At the selected alpha, gives the selection proportion of each variable over B bootstrap samples (Meinshausen and Buhlmann, 2010). Meaningless for Ridge.

Effect on the result: An additional table only.

On the validation data: QuickStart B = 200: V2 selection frequency 0.42.

→ Selection stability

Example on real data

Data: Stamey et al. 1989, ESL prostate training set

Variables: target lpsa (log PSA), explanatory lcavol, lweight, age, lbph, svi, lcp, gleason, pgg45

  • Question: In prostate cancer patients, how well can log PSA (lpsa) be predicted from eight clinical predictors with LASSO, and which predictors stay in the model?
  • Penalty selection: In consecutive 5-fold cross-validation the minimum error is at α = 0.001305; this is at the lower end of the grid, so the solution is very close to unpenalized OLS and all 8 of 8 predictors stay in the model. The 1-SE rule with α = 0.164686 suggests a sparser 5-variable model.
  • Result: The strongest effects are positive for svi (B = 0.731), lweight (B = 0.613) and lcavol (B = 0.573); lcp (B = -0.200) and age (B = -0.019) are negative.
  • Model fit: Training R² = .69, RMSE = 0.6628; but the CV fold mean R² is -8.09, RMSE(CV) = 0.9412.
  • Limitation: Because the data are ordered by lpsa, each unshuffled (consecutive) fold covers a narrow lpsa range; hence the fold R² is very negative. This is a fold-design problem as much as an overfitting warning; the fold shuffling option should be turned on.
  • APA: LASSO (L1): R² = -8.09 (CV), R² = .69 (training), α = 0.0013, k = 8/8 predictors, N = 67.

Related concepts

Elastic net and the L1 ratiok-fold cross-validationRegularization (penalty) and the penalty parameter λ/αSelection stabilityCoefficient path and ridge traceOne-standard-error (1-SE) ruleDistribution family (GLM)z-score (standardization)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.