Statistics Glossary → Neural Network (MLP)
Neural Network (MLP)analysis
A flexible model that predicts a class or a numeric value by passing the inputs through intermediate layers of weighted sums and non-linear transformations, learning its weights from the data so as to minimise the error.
Concept
When the relationship between the variables and the outcome is too complex to be described by a straight line or a few simple interactions, a model is needed that can learn the shape of the relationship from the data without it being specified in advance. A feed-forward neural network (multilayer perceptron, MLP) serves this purpose: with enough hidden units it can approximate almost any smooth relationship.
The network is made of layers. Each hidden unit takes a weighted sum of the inputs and passes it through a non-linear activation function: h = f(w₀ + w₁x₁ + … + w_px_p), where f may be, for example, the logistic curve or tanh. The output unit combines the hidden units' outputs in the same way; in classification the result is turned into a probability, in regression it is the numeric prediction itself. The weights start at random values and an optimisation algorithm updates them step by step in the direction that lowers the loss function (cross-entropy in classification, squared error in regression); the gradient of the error with respect to each weight is computed by backpropagation.
The network's objective function can have several local minima, so different random starts can lead to different solutions and results should be checked over several starts. The variables should be standardised; many units with little data lead to overfitting, which is limited by a weight penalty (L2) and early stopping. The weights cannot be interpreted one by one; the role of the variables is read from permutation importance and partial dependence. On small tabular datasets a neural network often does no better than tree-based methods or logistic regression; performance should always be assessed on data the model has not seen.
When to use it
- When a class or a numeric value is to be predicted from many continuous variables, the relationships are thought to be non-linear and enough data are available.
- If the sample is small and the effects must be interpreted, logistic or linear regression is preferred; for a strong model on tabular data that needs little tuning, random forest or gradient boosting.
- For unstructured data such as images, sound or text, convolutional or recurrent deep network architectures are used rather than a small fully connected network.
An everyday example
A greenhouse grower wants to predict whether tomato plants will catch a fungal disease within a week from measurements such as temperature, humidity, amount of watering, ventilation time and plant age. Disease risk is not a simple sum of these factors: high humidity may be dangerous only when the temperature is in a certain range and ventilation is poor. The network's hidden units learn such combinations on their own; one unit might, for example, fire strongly in 'warm, humid and stuffy' conditions, and the output unit combines this with the others to give a disease probability for each row of plants.
Parameters in MerQur
Engine / solver (nnet / Adam / SGD / L-BFGS)
Concept: The solver is the optimisation algorithm that updates the weights so as to reduce the loss function. BFGS and L-BFGS approximately use the curvature as well as the gradient and converge quickly on small data; SGD and Adam update the weights in small steps, Adam adapting the step size for each weight.
Everyday example: Picture a hiker looking for the lowest point of a foggy valley: BFGS looks at both the slope and the bend of the ground and takes large, well-aimed steps; SGD moves in small steps and does not reach the bottom if it does not walk long enough. On the grower's few hundred records, BFGS and L-BFGS usually reach a solution in fewer steps.
What it does: Training algorithm: nnet (BFGS, one hidden layer) or the Adam, SGD and L-BFGS solvers.
Effect on the result: The optimization path, convergence and the solution reached change; test and CV performance change.
On the validation data: iris test accuracy nnet 0.9667 (objective 6.0482, 165 iterations); L-BFGS 0.9333; Adam 0.9333 (did not converge in 500 epochs); SGD 0.8667 (did not converge). Boston test R²: nnet 0.8589, L-BFGS 0.8303, Adam 0.7589.
Hidden layers (units)
Concept: Hidden units are intermediate features the network derives from the inputs; with more units the network can represent more complex relationships. Several hidden layers let it derive new features from intermediate ones; but every extra unit means new weights and a higher risk of overfitting.
Everyday example: A network with 2 hidden units can form only two intermediate features, such as 'humidity risk' and 'temperature risk'. A network with 10 units builds many more combinations; but if there are records for only 150 rows of plants, some of these units learn chance details rather than real patterns.
What it does: Number of units in the hidden layer(s); several layers separated by commas (Adam, SGD and L-BFGS solvers only).
Effect on the result: More units make the model more flexible and raise the risk of overfitting; the objective value and performance change.
On the validation data: iris 2 units: 1.0 (objective 8.061); 5 units: 0.9667; 10 units: 0.9667 (objective 5.4674). "10,5" is rejected in nnet with an explicit error message (single layer); in Adam 0.9667.
Activation, Learning rate, Early stopping
Concept: The activation function sets how a hidden unit transforms its weighted sum (logistic, tanh, ReLU or linear); without a non-linear function the network is no different from an ordinary linear model. The learning rate is the size of each update; early stopping halts training when performance on a held-out validation share stops improving.
Everyday example: If the learning rate is too large the network keeps jumping over the best solution; if it is too small it takes very long to get there. With early stopping the grower sets some records aside; training stops when the network's predictions on them stop improving, but with a small learning rate the network may also stop before it has learned anything.
What it does: In the Adam, SGD and L-BFGS solvers: hidden-unit activation function, Adam/SGD learning rate and early stopping with a validation share.
Effect on the result: All three have no effect in the nnet engine (expected; the tooltip text states this).
On the validation data: (Adam): logistic 0.8333, tanh 0.9667, linear 0.9667; learning rate 0.01 and 1.0 (converged in 309 epochs). Adam with early stopping: stopped at 15 epochs, accuracy 0.40 (with the default learning rate 0.001 the validation score did not improve for 10 epochs). With L-BFGS the learning rate has no effect.
Regularization (L2: decay / alpha)
Concept: Regularisation adds a penalty for large weights to the loss function (the sum of squared weights). The penalty keeps the weights small, so the network learns smoother, less wiggly relationships and overfits less.
Everyday example: Without a penalty, the network might use very large weights to explain a chance cluster of disease in a few rows, building sharp rules such as 'risk is very high when humidity is exactly 83%'. With the penalty it settles for a smoother relationship in which risk rises gradually with humidity.
What it does: nnet objective = loss + decay·Σw² (including the bias).
Effect on the result: A larger decay shrinks the weights and smooths the model; the objective value, probabilities and performance change.
On the validation data: iris decay 0: 0.90 (objective 1e-4, 36 iterations); 0.01: 0.9667; 0.5: 0.9667 (objective 47.04).
Maximum iterations
Concept: The maximum number of times the optimisation algorithm updates the weights. If the limit is too low, the network stops before getting close to the best solution (it does not converge); once convergence is reached, raising the limit does not change the result.
Everyday example: If the hiker in the valley is allowed only 20 steps, they stop before reaching the bottom and their position is misleading. If the bottom is reached in 90 steps, allowing 500 or 1000 steps makes no difference.
What it does: Maximum number of iterations (epochs) of the optimizer.
Effect on the result: With few iterations the network stops before converging and the result changes; increasing it after convergence has no effect.
On the validation data: iris maxit 10: did not converge, 0.90 (objective 14.1367); 500 and 2000: converges at 165.
Seed repeats (initializations)
Concept: Because the weights are initialised at random, the optimisation can reach different local solutions from different starts. Refitting the network from several starts shows how much the result depends on this chance and increases the chance of finding a better solution.
Everyday example: If the valley has several hollows, you may end up in a different one depending on where the walk starts. The grower fits the network from five different starts; if all five give similar performance, the result can be trusted more.
What it does: Refitting with the initializations seed, seed+1, ....
Effect on the result: Different initializations can lead to different local solutions; the repeats show how sensitive the result is to the start.
On the validation data: The test and CV values depend on the initial weights. In regression, 5 initializations are tried in each repeat and the one with the lowest objective value is selected.
Test proportion, CV folds and repeats, Random seed, Probability threshold, Permutation importance, Save predictions, Additional metrics, Partial dependence, Chart
Concept: The data are split into training and test sets, cross-validation gives an additional measure of generalisation on the training set, and the seed makes these splits reproducible. The probability threshold sets the value at which probabilities become class decisions; permutation importance and partial dependence show what information the network, whose weights cannot be read directly, relies on.
Everyday example: If the grower wants to spray every suspect row, the threshold is kept low and rows with a probability above 30% are sprayed too. If permutation importance shows that performance drops clearly when the humidity readings are shuffled, the network relies heavily on humidity for its decisions.
What it does: Split proportion and seed, CV within training, probability threshold in binary classification, permutation importance on the test set, saving predictions, additional metrics, partial dependence and chart.
Effect on the result: The split, folds and seed change the test and CV scores; the threshold changes only the class predictions; the others produce additional output.
On the validation data: Test proportion 0 (all data): iris training accuracy 0.9867. Pima threshold 0.3: accuracy from 0.60 to 0.675. iris CV 0.95. Boston nnet CV R² 0.782. The additional metrics, permutation importance (the chart "Permutation importance" also opens it), PDP and prediction saving are produced.
Regression (Boston)
Concept: If the outcome is continuous, the output unit gives a numeric prediction directly rather than a probability (a linear output). Standardising the target during training makes the optimisation easier; the predictions are then converted back to the original units.
Everyday example: If, instead of disease, the grower wants to predict each row's weekly tomato yield in kilograms, the same network is built for regression. The network learns the yield on a scale adjusted by its mean and standard deviation, but still reports the results in kilograms.
What it does: Regression network for a continuous target (linear output unit).
Effect on the result: nnet z-standardizes the target (predictions are transformed back) and selects the one with the lowest objective value out of 5 initializations; the Adam, SGD and L-BFGS solvers also standardize the target.
On the validation data: The objective values of the 5 initializations (z-target) 33.638, 28.218, 25.328, 29.245, 23.974; the 5th initialization, with the lowest objective, is selected. Test R² 0.8589, CV R² 0.782 (full data 0.861; 10 units 0.708).
Example on real data
Data: Pima Indians (Pima.tr), R MASS package (n = 200)
Variables: target type (diabetes Yes/No), features number of pregnancies, glucose, blood pressure, skin thickness, BMI, pedigree function and age
- Question: How accurately does a neural network with one hidden layer built on seven clinical measurements (number of pregnancies, glucose, blood pressure, skin thickness, BMI, pedigree function, age) classify diabetes (type)?
- Model: One hidden layer with 5 units, logistic activation, BFGS optimization (nnet-compatible), L2 penalty 0.01; the network converged in 91 iterations with a final objective value of 36.337336. Training n = 160, test n = 40 (stratified, seed 42).
- Result: On the test set accuracy = .60, precision = .44, recall = .50, F1 = .47, ROC-AUC = .63; 7 of 14 diabetic women are detected and 9 of 26 healthy women are classified as false positives.
- Generalization: 5-fold CV accuracy M = .69 (SD = .08); the test accuracy being below the CV accuracy shows the sampling variability of the small test set.
- Limitation: The starting weights are random; a different seed may lead to a different local solution (the seed repeat option is recommended). In a small sample the network may not outperform simpler models such as logistic regression.
- APA: A neural network (5 hidden units, logistic, λ = 0.01) on the test set (n = 40) gave accuracy = .60, ROC-AUC = .63; 5-fold CV accuracy M = .69 (SD = .08).
Related concepts
Hidden layer and activation functionOptimization algorithm (Adam, SGD, L-BFGS)Random starts and local optimaConvergence and maximum iterationsTrain-test split (holdout)Regularization (penalty) and the penalty parameter λ/αPermutation importanceVariable importance (gain, cover, frequency)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.