MerQuR+Statistics Glossary

Statistics Glossary → Learning rate (shrinkage)

Learning rate (shrinkage)concept

A coefficient that sets how large the update made at each step of an iterative learning algorithm is.

In gradient boosting, each new tree's contribution is multiplied by the learning rate (ν, usually 0.01-0.3) before it is added to the model; this is also called shrinkage. A small learning rate reduces each tree's influence, so more trees are needed to reach the same performance, but the model usually generalizes better to new data. The learning rate and the number of trees should be tuned together; early stopping helps find a suitable number of trees.

In neural networks and other methods based on gradient descent, such as t-SNE, the learning rate is the size of the step taken along the slope of the loss function. A rate that is too large causes oscillation around the solution or divergence; one that is too small causes very slow progress and getting stuck in local solutions. In t-SNE, a learning rate that is too low can squeeze the points into a tight ball, and one that is too high can produce a scattered, irregular picture.

An everyday example

Picture a hiker trying to reach the lowest point of a valley in the dark. With very large steps he overshoots the bottom and climbs the opposite slope; with very small steps it takes him hours to get there. The learning rate is the size of that step.

In MerQur

'Learning rate' in t-SNE (the size of the gradient step; an 'Automatic' setting is available); the learning rate is also tuned in Gradient Boosting and Neural Network.

Analyses where this concept appears

Gradient BoostingNeural Network (MLP)t-SNE (t-Distributed Stochastic Neighbor Embedding)

Related concepts

Number of trees (ensemble models)Early stopping