Statistics Glossary → Hidden layer and activation function
Hidden layer and activation functionconcept
The intermediate layers of units in a neural network, between input and output, that create new features by passing weighted sums of the inputs through a nonlinear function, and the function those units use.
In a multilayer perceptron (MLP), each hidden unit computes a weighted sum of the values in the previous layer, adds a constant and passes the result through an activation function: h = f(Σ wᵢxᵢ + b). Common activations are ReLU (max(0, z)), the logistic (sigmoid) and tanh. If the activation were linear, the network would collapse into a single linear model however many layers it had; the nonlinearity is what lets it learn complex relationships.
The number of units and layers sets the model's flexibility. Too few units cannot capture the relationship (underfitting), while too many memorise the training data (overfitting), so the size is tuned together with regularization (weight penalty), early stopping and cross-validation. For tabular data one or two hidden layers are often enough. Scaling the inputs is important for stable training. Individual hidden units usually have no meaningful interpretation.
An everyday example
A network predicts a house's price from its floor area, age and location. Without being explicitly programmed, one hidden unit may become sensitive to the combination 'new and spacious', another to 'central but old'. The output layer combines these intermediate features to predict the price.
In MerQur
In the neural network, 'Hidden layers (units)' sets the number of units in each layer, and several layers can be entered separated by commas (Adam, SGD and L-BFGS solvers only). 'Activation, Learning rate, Early stopping' set the activation function, the Adam/SGD learning rate and early stopping on a validation fraction.
Analyses where this concept appears
Related concepts
Optimization algorithm (Adam, SGD, L-BFGS)Regularization (penalty) and the penalty parameter λ/α