MerQuR+Statistics Glossary

Statistics Glossary → Train-test split (holdout)

Train-test split (holdout)concept

Setting aside part of the data to build the model (training) and the rest to measure how well it performs on data it has not seen (testing).

A model's performance on the data it was built on is almost always more optimistic than on new data; flexible models can also learn the noise in the data (overfitting). Because the test set is never used while building the model, it gives an honest estimate of performance. Common ratios are 70-80% training and 20-30% testing; in classification a stratified split keeps the class proportions.

A single split depends on chance: another random seed can give different performance, especially with small data. k-fold cross-validation reduces this variability; a common workflow is to tune hyperparameters by cross-validation within the training set and to look at the test set only once, at the very end. In time series the last observations are held out rather than a random subset, because the future must not be used to predict the past. Preprocessing such as scaling should also be learned from the training set alone; otherwise information leaks.

An everyday example

A teacher writes 100 practice questions for an exam. She lets the students work through 80 of them and then tests them not on those memorized questions but on the 20 they have never seen. Those 20 questions show the students' real level of learning.

In MerQur

The 'Test proportion', 'CV folds' and 'Random seed' settings in Random Forest, Support Vector Machine, Gradient Boosting, Decision Tree, Neural Network and k-Nearest Neighbours; 'Evaluation' (in-sample, independent test set, k-fold cross-validation) in the True Skill Statistic. 'Holdout set (last k observations)' in ARIMA Forecast and Exponential Smoothing.

Analyses where this concept appears

True Skill Statistic (TSS)Random ForestSupport Vector Machine (SVM)Gradient BoostingDecision Tree (CART)Neural Network (MLP)ARIMA ForecastExponential Smoothing (Holt-Winters)k-Nearest Neighbors (KNN)

Related concepts

k-fold cross-validationPrediction error measures (RMSE, MAE, MAPE, MASE)