MerQuR+Statistics Glossary

Statistics Glossary → Selection stability

Selection stabilityconcept

A method for judging how reliable a variable selection is by measuring how often a selecting method such as the lasso picks the same variables across different subsamples of the data.

When the lasso selects variables in a single data set, which of several similar predictors gets chosen can change with small changes in the data. In stability selection the data are resampled many times (e.g. 100 times, each time drawing half the observations without replacement, or drawing bootstrap samples), a model is fitted in each sample with the same penalty, and the selection rate of each variable is computed. Variables with a selection rate above a threshold (e.g. 0.6) are considered 'stable'.

This approach, proposed by Meinshausen and Bühlmann, gives a more reliable selection than relying on a single cross-validation result and offers theoretical control over the number of falsely selected variables. Low stability does not necessarily mean a variable is unimportant; it may be trading places with another strongly correlated variable. Because ridge regression never sets coefficients exactly to zero, selection stability is meaningless there.

An everyday example

A marketing team uses the lasso to find which of 40 candidate factors predict customer churn. A single analysis selects 9 variables. Repeated over 100 subsamples, 'number of complaints in the last 3 months' is selected every time (rate 1.00) and 'membership length' 85% of the time, while 'device type' is selected only 30% of the time; it may have been picked in the first analysis by chance.

In MerQur

Available as the 'Selection stability' output in lasso regression (100 half-size subsamples without replacement, 0.6 or more counted as stable) and as the 'Bootstrap stability (B)' option in regularized regression; the latter is meaningless for ridge.

Analyses where this concept appears

Lasso RegressionRegularized Regression (LASSO / Ridge / ElasticNet)

Related concepts

Regularization (penalty) and the penalty parameter λ/αBootstrap (resampling)