MerQuR+Statistics Glossary

Statistics Glossary → Bootstrap optimism correction

Bootstrap optimism correctionconcept

An internal validation method that uses the bootstrap to estimate how optimistic a prediction model's performance on its own development data is, and subtracts that optimism.

A model looks better on the data it was built on (apparent performance) than on new data, because it also fits the chance features of that data. In Harrell's method, a bootstrap sample of the data is drawn, the model is rebuilt from scratch on it, and its performance (e.g. AUC) is measured both on the bootstrap sample and on the original data. The difference is the optimism for that replicate; this is repeated B times and the average optimism is subtracted from the apparent performance: corrected performance = apparent performance - average optimism.

The method uses all the data both to build and to evaluate the model without setting aside a separate test set, so in small and medium samples it is generally more efficient than a train-test split. To be valid, every modelling step, such as variable selection or choice of transformations, must be repeated in each bootstrap sample; otherwise the optimism is underestimated. It is an internal validation; it does not show whether the model works in another population or period (external validity).

An everyday example

A hospital develops a risk score from data on 150 patients, and the score's AUC on its own data is 0.82. Over 200 bootstrap replicates the model looks on average 0.04 better on its own sample. The corrected AUC is about 0.78, closer to the performance the score can be expected to show in new patients.

In MerQur

'Optimism correction (bootstrap)' in the ROC Curve: Harrell's bootstrap optimism correction is applied to the combined logistic model; it is not applied in a single-variable analysis.

Analyses where this concept appears

ROC Curve

Related concepts

Bootstrap (resampling)Area under the ROC curve (AUC)