MerQuR+Statistics Glossary

Statistics Glossary → Hyperparameter tuning (grid search)

Hyperparameter tuning (grid search)concept

Choosing the settings of a machine-learning model that are not learned from the data (such as tree depth or a penalty parameter) from a set of candidate combinations, based on cross-validated performance.

A model's parameters (regression coefficients, the splits in a tree) are learned from the data during training. Hyperparameters, by contrast, govern how the learning happens and must be set beforehand: the number of variables tried at each split in a random forest, C and γ in an SVM, or the learning rate and tree depth in gradient boosting. In a grid search a few candidate values are chosen for each hyperparameter, every combination is tried and each is evaluated with k-fold cross-validation within the training data; the combination with the best average performance is chosen.

The choice must use the training data only; if the test data leak into the tuning process, the reported performance will be optimistic. With a large grid, computing time grows quickly, and random search or sequential methods are more efficient. If the chosen value lies at the edge of the grid, the best value may be outside it. Also, when many combinations are tried, even the best cross-validation score is somewhat optimistic; an unbiased estimate of performance needs a separate test set or nested cross-validation.

An everyday example

A baker wants to tune a new cake recipe for oven temperature (170, 180, 190 °C) and baking time (30, 35, 40 minutes). She bakes each of the nine combinations a few times, has tasters score them and picks 180 °C for 35 minutes, which gets the best average score. When she finally serves the chosen recipe to guests who took no part in the tasting, she sees how well it is really liked; that last step corresponds to the test set.

In MerQur

In Random Forest (mtry and minimum leaf size) and Support Vector Machine (C x γ), the 'Hyperparameter tuning (grid search + CV)' option, and in Gradient Boosting (eta x depth x rounds) the 'Hyperparameter tuning' option, select the best setting from a grid by cross-validation within the training set.

Analyses where this concept appears

Random ForestSupport Vector Machine (SVM)Gradient Boosting

Related concepts

k-fold cross-validationEarly stopping