MerQuR+Statistics Glossary

Statistics Glossary → k-fold cross-validation

k-fold cross-validationconcept

A method that estimates how well a model will predict new data by splitting the data into k equal parts and repeatedly using one part for testing and the rest for training.

The data are divided into k folds; the model is fitted k times, each time leaving one fold out and measuring the prediction error on it. The average of the k errors estimates the model's performance on data it has not seen, and every observation is used for testing exactly once. k = 5 or 10 is common; when k = n the method is called leave-one-out (LOO). A small k gives a more pessimistic (biased) but less variable estimate, a large k a less biased but more variable one.

Cross-validation is used both to assess model performance and to choose settings such as a penalty parameter, the number of neighbours or the number of components. If the same CV is used both to choose a setting and to report performance, the result is optimistic; nested cross-validation is needed for that. Every preprocessing step, such as variable selection or scaling, must be redone inside each fold, otherwise information leaks. Classification uses stratified folds that preserve class proportions. With time series or ordered data, random shuffling can leak the future into the past, and with spatial data nearby points are similar, so performance can be overstated.

An everyday example

A property website builds a model to predict the price of 500 houses. It splits the data into 5 folds of 100 houses: the model is trained on 400 houses, predicts the prices of the remaining 100, and this is repeated five times. The average error over the five runs is 42 thousand. This is a realistic estimate of the error to expect on new listings the model has never seen.

In MerQur

Options include 'CV folds' in ridge and lasso regression (3-20, default 5), 'CV folds', 'Shuffle CV folds' and 'CV repeats' in regularized regression, 'CV folds (xval)' in the decision tree, 'CV folds' in discriminant analysis, 'CV type (LOO, 10-fold consecutive)' in partial least squares regression and 'Choose from range by CV' in k-nearest neighbours. The true skill statistic, random forest, SVM, gradient boosting and the neural network also offer cross-validation. In GWR the 'Bandwidth criterion' includes a CV option, Variogram + Kriging gives leave-one-out (LOO) cross-validation, and the KDE density map can choose its bandwidth by cross-validation.

Analyses where this concept appears

Ridge RegressionLasso RegressionTrue Skill Statistic (TSS)Random ForestSupport Vector Machine (SVM)Gradient BoostingDecision Tree (CART)Neural Network (MLP)Discriminant Analysis (LDA / QDA)Regularized Regression (LASSO / Ridge / ElasticNet)Partial Least Squares Regression (PLS)GWR (Geographically Weighted Regression)k-Nearest Neighbors (KNN)KDE Density MapVariogram + Kriging (Spatial Interpolation)

Related concepts

Leave-one-out (LOO)Train-test split (holdout)Hyperparameter tuning (grid search)