Statistics Glossary → Early stopping
Early stoppingconcept
A regularization technique for iteratively trained models (gradient boosting, neural networks) that prevents overfitting by ending training when performance on a held-out validation set stops improving.
Gradient boosting adds a tree in each round, and a neural network updates its weights in each epoch. As training continues the error on the training data keeps falling, but past a certain point the model starts memorizing noise and the error on unseen data rises. In early stopping, part of the training data (e.g. 20%) is set aside for validation; if the validation error fails to improve for a set number of consecutive rounds (the patience), training stops and the number of rounds with the best validation error is used.
Early stopping is a cheap alternative to searching over the number of rounds as a separate hyperparameter, and the number of rounds needed grows as the learning rate shrinks. The result depends on the size and random split of the validation set; with little data the validation error is noisy, and the model may stop too early or too late. Because the validation set is used to choose the number of rounds, final performance should be measured on a separate test set. If the patience is set too small, training can be cut short by temporary plateaus.
An everyday example
A student preparing for an exam tests herself every day on a question set she has never looked at, alongside her usual practice. For ten days her score on this fresh set rises, then it stops rising and even dips slightly as she starts memorizing answers to the old questions. After three days in a row with no improvement she decides to change how she studies; the level she reached on day 11, her best score, reflects her real preparation.
In MerQur
Available as 'Early stopping (rounds)' in gradient boosting: 20% of the training data is held out for internal validation, training stops if there is no improvement for the given number of rounds, and the model is refitted with the best number of rounds. It is also among the training options of the neural network analysis.
Analyses where this concept appears
Gradient BoostingNeural Network (MLP)
Related concepts
Learning rate (shrinkage)Hyperparameter tuning (grid search)