MerQuR+Statistics Glossary

Statistics Glossary → Ridge Regression

Ridge Regressionanalysis

A regression method that stabilises the coefficients when predictors are strongly correlated by shrinking them towards zero with a small penalty, with the aim of better prediction on new data.

Concept

In ordinary least squares (OLS) regression, strongly correlated predictors (multicollinearity) make the coefficients very unstable: adding or removing a few observations can change the size or even the sign of a coefficient. The model fits the data at hand well but may predict poorly on new data. Ridge regression reduces this instability by adding a penalty on the size of the coefficients.

Ridge minimises the residual sum of squares together with the sum of squared coefficients: Σ(y − ŷ)² + α·Σβ². As the penalty α grows, the coefficients shrink towards zero; with α = 0 the result is the same as OLS. Because the penalty applies equally to all coefficients, the predictors are standardised first. Some bias is thereby added to the coefficients, but their variance falls markedly; this bias-variance trade-off often means better prediction. A suitable α is usually chosen by cross-validation.

Ridge coefficients shrink but never reach exactly zero, so ridge does not select variables: all predictors stay in the model. Because the coefficients are deliberately biased, classical standard errors and p values are not interpreted; the method aims at prediction and stability, not at testing the significance of individual coefficients.

When to use it

  • When there are many correlated predictors and you want stable coefficients and a good predictive model (e.g. clinical or environmental indicators built from related measurements).
  • If some predictors are thought to be irrelevant and a simpler model is wanted, use the lasso, which can set coefficients exactly to zero; if both selection and keeping groups of correlated variables together are wanted, use the elastic net.
  • If there is no multicollinearity and the aim is to interpret or test the coefficients, ordinary multiple linear regression is sufficient.

An everyday example

An estate agency wants to predict the sale price of 200 flats from net floor area, gross floor area, number of rooms, number of bathrooms, building age and distance to the centre. Net area, gross area and number of rooms are so closely related that in OLS the coefficient for gross area can come out negative, absurdly implying that bigger flats are cheaper. Ridge regression pulls the coefficients of these three variables to sensible, similar values; the results do not jump when a few flats are added or removed, and predictions for new listings become more reliable.

Parameters in MerQur

Independents (Features) and Target column

Concept: The predictors are the variables used for prediction and the target is the continuous outcome being predicted. Because the penalty applies equally to all predictors, they are standardised so that differences in scale do not affect the results.

Everyday example: In the estate example the target is the sale price and the predictors are floor area, number of rooms, building age and distance. Area is measured in square metres and distance in kilometres, so without standardisation the penalty would treat the variables differently just because of their units.

What it does: Selects the predictors and the dependent variable. Text columns are dummy-coded automatically.

Effect on the result: The penalty is applied to all standardized columns (including dummies).

α (regularization)

Concept: The penalty α determines how much the coefficients are shrunk. α = 0 equals OLS; as α grows the coefficients approach zero and the model becomes more stable but fits the data less closely.

Everyday example: With a small α, the coefficients for net and gross area may still be large and of opposite sign. As α increases they move towards similar, sensible values; if α becomes very large, every variable is suppressed so much that none explains the price.

What it does: The L2 penalty coefficient: minimises ||y - X_s b||² + α||b||² (intercept unpenalised). As it grows, coefficients shrink towards zero but never reach zero. The range is 1e-6 - 1e6. MerQur's alpha is on the scale where X is standardized with the 1/n-divisor SD.

Effect on the result: All coefficients, R² and fitted values change.

On the validation data: Longley: α = 0.001 R² 0.9926; α = 1 R² 0.9850; α = 100 R² 0.6536. For the acetylene data 10 k values × 6 coefficients are computed. Prostate α = 1: R² 0.6940.

→ Regularization (penalty) and the penalty parameter λ/α

Auto α (cross-validation)

Concept: Cross-validation splits the data into parts, fits the model on some and measures the prediction error on the rest; the α giving the lowest prediction error is chosen. So α is set by how well the model predicts new data rather than by the researcher's guess.

Everyday example: The flats are split into five groups; the model is fitted on four and the prices in the fifth are predicted, and this is repeated for each group. Several values of α are tried and the one with the smallest average prediction error is chosen.

What it does: Selects alpha from a 100-point df(α)-based grid built from the data; the rule is in the "α selection criterion" field (default LOO). A warning is written if the selected alpha is at the grid edge.

Effect on the result: The selected alpha and all results that depend on it change. LOO rescales X in every n - 1 subset and k-fold in every training fold (Hastie, Tibshirani and Friedman, ESL 7.10.2).

On the validation data: Longley LOO α = 0.0014547; QuickStart LOO 3.07232; prostate LOO 3.73253; Hitters LOO 3.53139.

→ Regularization (penalty) and the penalty parameter λ/α

α selection criterion (LOO (default), k-fold λ_min, k-fold λ_1SE, GCV, HKB, L-W)

Concept: Different rules for choosing α. Cross-validation rules measure prediction error directly; the 1-SE rule prefers the most heavily penalised (simplest) model that stays statistically close to the best result; GCV and closed-form criteria give fast approximate solutions.

Everyday example: If the lowest prediction error is reached at α = 2 but α = 5 gives almost the same error, the 1-SE rule chooses α = 5, accepting a very small loss in prediction for more stable coefficients.

What it does: LOO uses the exact leave-one-out MSE; k-fold the minimum CV MSE or the 1-SE rule; GCV the minimum of RSS/(n - df)²; HKB (Hoerl, Kennard and Baldwin, 1975) and L-W (Lawless and Wang, 1976) closed forms.

Effect on the result: Determines the selected alpha only when auto alpha is on. LOO, GCV, HKB and L-W are not defined for binary/Poisson targets: k-fold λ_min is used with a warning.

On the validation data: Longley: HKB 0.0068370, L-W 0.052672, GCV 0.0060077; 5-fold λ_1SE 1.74758. Hitters: GCV 1.42066, HKB 2.2793, L-W 15.292.

→ Regularization (penalty) and the penalty parameter λ/α

CV folds

Concept: Sets how many parts the data are split into for cross-validation. Because the number of folds can change the results somewhat, it should be reported.

Everyday example: With 200 flats in 5 folds, 40 flats are tested in each round; with 10 folds, 20 flats per round. The two choices may give slightly different values of α.

What it does: Sets the number of consecutive (unshuffled) folds for the k-fold rules and CV performance (3-20, default 5). It should be reported because it can change the results.

Effect on the result: Affects only the k-fold rules and CV performance; no effect for LOO, GCV, HKB, L-W or when CV is off. Consecutive folds are deterministic.

On the validation data: QuickStart k-fold λ_min: K = 3 -> 3.07232, 5 -> 5.72633, 10 -> 3.58979.

→ k-fold cross-validation

Target type (Continuous (linear), binary (logistic), count (Poisson))

Concept: The ridge penalty can be used not only for continuous outcomes but also for binary (logistic) and count (Poisson) outcomes; the penalty is added to the likelihood of these models.

Everyday example: If the agency wants to answer 'was the flat sold within three months?' instead of predicting price, the outcome is binary and penalised logistic regression is used; the related area variables are still shrunk together.

What it does: Fits a penalised logistic or Poisson model: -l(b) + α/2·|b|² (intercept unpenalised). Used for binary/count outcomes with many correlated predictors.

Effect on the result: The model family, deviance and D² measures change.

On the validation data: BinomialExample α = 1: deviance 25.716, D² 0.8125; PoissonExample α = 1 D² 0.9715.

→ Distribution family (GLM)

Categorical predictors

Concept: Expands categorical predictors into dummy variables. In ridge, the dummies are standardised and penalised too, so their coefficients come out smaller than in ordinary regression.

Everyday example: A variable such as heating type ('gas boiler / stove / central') enters the model as two dummies, and their coefficients are pulled towards zero like the others.

What it does: Dummy-codes the named columns (even if numeric), first level as the baseline; text columns are automatic. Dummies are also standardized and penalised (a warning is written).

Effect on the result: The dummy design and the penalised coefficients change.

On the validation data: Prostate "gleason" (6/7/8/9) -> gleason[7], [8], [9]; R² 0.6940 -> 0.7119.

→ Dummy (indicator) variable coding

Extra output: Ridge trace + CV curve

Concept: The ridge trace is a plot showing how each coefficient shrinks as α increases; the CV curve shows the prediction error for each α. Together they show at which penalty the coefficients become stable and prediction is best.

Everyday example: On the trace, the coefficients for net and gross area are large and of opposite sign at small α and quickly converge as α grows. The α at the bottom of the CV curve is where this convergence also lowers the prediction error.

What it does: Gives the standardized coefficient trace along the grid (SVD), df(α) = Σd²/(d² + α) and the CV curve.

Effect on the result: An extra table and plot.

On the validation data: Longley trace: df 0.622 (α = 100) - 5.795 (α = 0.001); df 2.937 at α = 1.

→ Coefficient path and ridge trace

Extra output: CV performance

Concept: Summarises the model's predictive performance on data not used to fit it (mean squared error, mean absolute error, cross-validated R²). It is a more honest measure of performance than R² on the training data.

Everyday example: If the model explains 85% of price variation in the flats it was fitted on but 70% in flats it has never seen, the second figure is what to expect for a new listing.

What it does: Out-of-fold prediction with the selected alpha over consecutive k folds: MSE, RMSE, MAE, R²_CV; for GLMs mean unit deviance and classification error.

Effect on the result: An extra table.

On the validation data: QuickStart α = 1, K = 5: MSE 1.2977, R²_CV 0.8478. Longley K = 5: MSE 23.277, R²_CV 0.7868.

→ Prediction error measures (RMSE, MAE, MAPE, MASE)

Extra output: VIF / condition number

Concept: The VIF shows how much each predictor overlaps with the others, and the condition number shows how collinear the predictor set is as a whole. Lower values after ridge show how far the penalty has removed the instability.

Everyday example: If the VIF of net area is 40 in OLS but falls to 3 after ridge, the inflation caused by the interdependence of the area variables has largely been removed.

What it does: OLS VIF = diag(C⁻¹); ridge VIF = diag((C + kI)⁻¹C(C + kI)⁻¹), with C the predictor correlation matrix, k = α/n (Marquardt, 1970); condition number √(λ_max/λ_min) and its value after ridge.

Effect on the result: An extra table; for GLMs only the OLS VIF.

On the validation data: Acetylene OLS VIF x1 7682.37; ridge VIF x1 11.177 at k = .002 and 1.387 at k = .02; condition number 218.3 -> 40.07.

→ Multicollinearity (VIF, tolerance, condition index)

Extra output: HKB/L-W/GCV

Concept: Reports the values of three quick, data-based criteria for α for comparison; it does not change the selected model.

Everyday example: If cross-validation picks α = 3 and these criteria give values between 2 and 5, the choice lies in a reasonable range; very different values suggest examining the result more carefully.

What it does: Writes the three criteria for information; does not change the selected alpha.

Effect on the result: Extra rows; for a binary target only a warning ("defined only for a continuous target").

On the validation data: Longley HKB 0.006837, L-W 0.052672, GCV 0.0060077; QuickStart HKB 2.4818, L-W 2.1653.

→ Generalized cross-validation (GCV)

Example on real data

Data: Stamey et al. 1989 prostate data (ESL training set)

Variables: target lpsa (log PSA), eight clinical predictors (lcavol, lweight, age, lbph, svi, lcp, gleason, pgg45)

  • Question: In 67 prostate cancer patients, when PSA level (lpsa) is predicted from eight clinical predictors, how does the ridge penalty stabilise coefficients made unstable by collinearity? Predictors were standardized with the 1/n-divisor SD, an L2 penalty α = 1 was applied (intercept unpenalised) and the coefficients were transformed back to the raw scale.
  • Assumption: α = 1 is a manually entered value; in a predictive model α should be chosen by cross-validation (LOO, k-fold, GCV), and classical SE/p values should not be interpreted for ridge coefficients.
  • Result: R² = 0.6940, adjusted R² = 0.6518. The largest standardized coefficients are lcavol (0.685), svi (0.302) and lweight (0.290); the gleason coefficient is shrunk almost to zero (-0.016).
  • Effect size: Ridge coefficients are smaller than OLS (lcavol 0.5765 -> 0.5558, lcp -0.2063 -> -0.1831) but none becomes exactly zero; no variable selection is performed.
  • APA: Ridge regression (α = 1, standardized predictors), R² = .69, adjusted R² = .65.

Related concepts

Regularization (penalty) and the penalty parameter λ/αCoefficient path and ridge traceGeneralized cross-validation (GCV)k-fold cross-validationMulticollinearity (VIF, tolerance, condition index)Prediction error measures (RMSE, MAE, MAPE, MASE)Leave-one-out (LOO)Distribution family (GLM)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.