MerQuR+Statistics Glossary

Statistics Glossary → Regularization (penalty) and the penalty parameter λ/α

Regularization (penalty) and the penalty parameter λ/αconcept

A method that reduces overfitting and stabilizes estimates by adding a penalty on the size of the model coefficients; the strength of the penalty is set by λ (α in some software).

Ordinary least squares minimizes only the error on the training data; with many predictors, or strongly correlated ones, the coefficients become large and unstable and the model predicts poorly on new data. Regularization adds a penalty to the objective function. The ridge (L2) penalty is the sum of squared coefficients (λ·Σβ²): it shrinks all coefficients towards zero but never sets any exactly to zero. The lasso (L1) penalty is the sum of absolute values (λ·Σ|β|): it sets some coefficients exactly to zero and so performs variable selection. The elastic net mixes the two. Weight decay in neural networks is also an L2 penalty.

As λ grows, the coefficients shrink: bias increases but variance falls. The best λ is usually chosen by cross-validation (the 'one-standard-error rule' picks the simplest model within one standard error of the best). Because the penalty treats all coefficients on the same scale, predictors must be standardized first; the intercept is not penalized. Penalized estimates are deliberately biased, so classical standard errors and p-values do not apply, and the variables selected by the lasso can change from sample to sample when predictors are correlated.

An everyday example

A student tries to predict their real exam score from just five practice tests and builds an elaborate formula giving a separate weight to every question in every test. The formula fits the five practice tests perfectly but fails on the real exam. A simpler formula that 'penalizes' large weights fits the practice tests slightly worse but predicts the real exam better; regularization strikes this balance.

In MerQur

It appears as 'α (regularization)', 'Auto α (cross-validation)' and, in Ridge, 'α selection criterion' in Ridge Regression and Lasso Regression; as 'Method' (LASSO, Ridge, ElasticNet) and 'Alpha' in Regularized Regression; as 'Regularization (L2: decay / alpha)' in the Neural Network; as 'Penalizer (L2)' in Time-Dependent Cox Regression; and as the misclassification penalty 'C (cost)' in the Support Vector Machine. The 'Firth correction' of Logistic Regression uses penalized likelihood.

Analyses where this concept appears

Logistic RegressionRidge RegressionLasso RegressionSupport Vector Machine (SVM)Neural Network (MLP)Generalized Additive Model (GAM, Nonlinear Regression)Regularized Regression (LASSO / Ridge / ElasticNet)Time-dependent Cox Regression

Related concepts

Elastic net and the L1 ratioOne-standard-error (1-SE) ruleCoefficient path and ridge trace