Statistics Glossary → Optimization algorithm (Adam, SGD, L-BFGS)
Optimization algorithm (Adam, SGD, L-BFGS)concept
A numerical method that updates a model's parameters step by step so as to minimize the loss function (or the residual sum of squares).
For models without a closed-form solution (neural networks, nonlinear regression), the parameters are improved iteratively from a starting point. Gradient descent moves the parameters in the direction in which the loss decreases fastest (opposite to the gradient), by an amount set by the learning rate. Stochastic gradient descent (SGD) computes the gradient at each step from a small subset of the data (a mini-batch) rather than all of it; it is cheap and noisy. Adam keeps running averages of the gradients and their variance for each parameter and adapts the step size, usually converging quickly without much tuning. BFGS and its memory-limited version L-BFGS are quasi-Newton methods that approximate the curvature of the loss (the Hessian) from gradients; they are very fast and stable on small to medium data.
In nonlinear least squares, Levenberg-Marquardt (LM) and trust-region methods (TRF) are widely used; when parameter bounds are given, trust-region methods are preferred. If the loss surface has many minima, as in neural networks, the chosen algorithm can reach different local minima, so results may depend on starting values and the random seed. Convergence warnings should not be ignored: the coefficients and standard errors of a model that has not converged are not reliable.
An everyday example
Picture a hiker looking for the lowest valley on a foggy mountain. At each step she checks the slope under her feet and walks downhill (gradient descent). A hiker using Adam remembers the slopes of her last few steps, shortening her stride on steep ground and lengthening it on flat ground. One using L-BFGS estimates how curved the terrain is and tries to jump straight to the valley floor; on smooth ground she gets there in far fewer steps.
In MerQur
Available as the 'Engine / solver' option (nnet / Adam / SGD / L-BFGS) in the neural network analysis; the nnet engine trains a single-hidden-layer network with BFGS. In nonlinear regression the solver uses the LM and trust-region reflective (TRF) algorithms; TRF is used when parameter bounds are given.
Analyses where this concept appears
Neural Network (MLP)Nonlinear Regression