Statistics Glossary → Gradient Boosting
Gradient Boostinganalysis
A machine learning method that builds a strong predictive model by adding small decision trees one after another, each new tree correcting a little more of the errors left by the previous ones.
Concept
In a random forest the trees are grown independently and averaged; in gradient boosting the trees are built in sequence and each learns from the errors of those before it. Combining trees that are weak on their own (only a few levels deep) in this way yields one of the methods with the highest predictive accuracy, especially on tabular data.
The model starts with the same simple prediction for everyone. In each round it looks at the gradient of the loss function of the current model (e.g. squared error in regression, log-loss in classification), that is, the direction in which the error can be reduced, and fits a small tree that predicts this direction. The new tree's contribution is not added in full but shrunk by the learning rate (eta): F_new(x) = F_old(x) + η · tree(x). Moving in small steps needs more rounds but usually gives a model that generalises better.
The method is as prone to overfitting as it is flexible: many rounds, deep trees or a large learning rate can memorise the training data. The number of rounds, the depth and the learning rate should therefore be tuned together, with early stopping and cross-validation. The model gives no coefficients; the role of the variables is read from importance measures, SHAP contributions and partial dependence. Performance is always assessed on data the model has not seen.
When to use it
- On tabular data (customer records, clinical measurements, environmental variables), when the highest achievable predictive accuracy for a class or a numeric value is the goal.
- If less time can be spent on tuning and a robust, easily built model is enough, random forest is preferred; if readable rules are needed, a single decision tree.
- If the direction and size of effects must be reported with coefficients and confidence intervals, logistic or linear regression is more suitable.
An everyday example
A supermarket chain wants to decide who should receive a discount coupon, using information such as shopping frequency, basket value, age and past campaign participation. The first small tree builds a crude rule such as 'frequent shoppers use the coupon' and gets some customers wrong. The second tree focuses only on these mistakes (e.g. frequent shoppers with small baskets), and the third looks at the errors that remain. After a hundred rounds, the sum of these small corrections forms a model that predicts quite accurately how likely each customer is to use the coupon.
Parameters in MerQur
Engine (Automatic (XGBoost if available) / XGBoost / scikit-learn)
Concept: These are different software implementations of the same boosting idea. XGBoost also uses the second derivative of the loss function and penalises tree complexity; scikit-learn's GradientBoosting is a more classical implementation. Because their default settings differ, their results also differ somewhat.
Everyday example: Two bakers following the same recipe still produce slightly different loaves because their ovens and habits differ. The supermarket chain can build the model with both engines and check whether the results agree.
What it does: The implementation of the boosting algorithm. "Automatic" is XGBoost if XGBoost is installed.
Effect on the result: The tree-building rules and the default eta and depth depend on the engine; test and CV results change.
On the validation data: iris XGBoost 0.9333 (CV 0.9533), GradientBoosting 0.9667 (0.9667); Boston R² 0.9058 and 0.9153. "Automatic" and "XGBoost" are the same (no effect, expected).
Number of trees, Learning rate (eta), Maximum depth
Concept: The number of trees is the number of correction rounds, the learning rate is the size of the step taken in each round, and the depth is how many variables each tree can handle together (the level of interaction). A small learning rate needs more rounds; a large rate combined with many rounds can lead to overfitting.
Everyday example: With a learning rate of 0.3 each tree applies 30% of the correction it finds; with 0.05 only 5%, so many more rounds are needed to reach the same point. A tree of depth 2 can combine two pieces of information such as 'frequent shopper with a small basket', while a tree of depth 6 builds much more detailed combinations.
What it does: Number of rounds; shrinkage (0 = 0.3 in XGBoost, 0.1 in GradientBoosting); tree depth (0 = 6 in XGBoost, 3 in GradientBoosting).
Effect on the result: Many rounds and a large eta can overfit, few rounds and a small eta can underfit; depth sets the interaction level. All metrics change.
On the validation data: iris 10 rounds 0.9667, 100 rounds 0.9333, 500 rounds 0.9333; eta 0.05: 0.9667, eta 1.0: 0.9333 (CV 0.94); depth 2: 0.9333 (AUC 0.9617); depth 10 is the same as 6 (iris trees do not reach 6; data-related).
Subsample ratio
Concept: In each round the tree is built not on all the data but on a randomly chosen part of it (stochastic boosting). This randomness reduces the chance that the trees get stuck on the same errors and can limit overfitting; the price is that the result depends on the seed.
Everyday example: With a rate of 0.8, each new tree is built by looking at a random 80% of the customers. Each tree thus learns from slightly different customers, and the model does not overfit to a few unusual ones.
What it does: Proportion of rows used in each round (stochastic boosting); when below 1 the result depends on the seed.
Effect on the result: Below 1 the variance may decrease and the result depends on the seed; at 1 XGBoost is deterministic.
On the validation data: Boston 0.5: R² 0.9159 (1.0: 0.9058). The subsampled result depends on the seed; the values are for seed 42.
min_child_weight
Concept: A leaf can form only if the total 'weight' of its observations (the sum of the second derivatives of the loss function) reaches at least this value. In regression this is roughly the number of observations in the leaf; a large value prevents splits that rest on very few observations.
Everyday example: If the value is set high, the model cannot build very specific rules resting on just three or four customers, such as 'people of this age, from this district, who shop on this day use coupons'. The rules rest on broader groups of customers.
What it does: In XGBoost, the minimum total Hessian (instance weight) required in a leaf; a large value restricts splitting.
Effect on the result: Trees become simpler; probabilities, accuracy and AUC may change. No effect in the GradientBoosting engine.
On the validation data: iris 5: accuracy 0.9667, AUC 0.9583. It is not used in the GradientBoosting engine (no effect, expected).
Early stopping (rounds)
Concept: Part of the training data is held back and the error on it is tracked after each round. If there is no improvement for a given number of rounds, training stops, so the model settles at a suitable number of rounds before it starts memorising the training data.
Everyday example: The supermarket sets some customers aside and checks the prediction success on them after each new tree. If success has not improved for 10 rounds, it stops adding trees; further trees would be learning noise rather than a real pattern.
What it does: 20% of the training set is set aside as internal validation; it stops if there is no improvement for k rounds and is refitted with the best number of rounds.
Effect on the result: The number of rounds is selected from the data; overfitting is reduced, test performance and importance values change.
On the validation data: (XGBoost): iris with 10 rounds best round 12; Pima 4; Boston 10; the training and validation loss curves are reported. In the GradientBoosting engine for iris the best round is 27 and the stopping round 37 (reported separately).
Importance type (Gain (mean per split) / Total gain (R Gain) / Cover / Frequency (number of splits))
Concept: These are different definitions of variable importance: gain measures how much the splits using a variable reduce the loss on average; total gain is the sum of these reductions; cover measures how many observations those splits affect; and frequency counts how many splits use the variable.
Everyday example: A variable used very often but bringing only a small improvement each time can rank high on frequency and low on gain. For instance, age may be used in many small distinctions while past campaign participation is used in a few highly decisive splits; looking at several measures reveals this difference.
What it does: The XGBoost importance measure; normalized to sum to 1.
Effect on the result: Changes only the measure of the importance table; the model and predictions do not change.
On the validation data: For iris all four types are reported (total gain, mean gain = total gain/frequency, Cover, Frequency). No effect in the GradientBoosting engine.
SHAP
Concept: SHAP splits each prediction into the contributions of the variables: starting from the average prediction, adding each variable's contribution leads to that observation's prediction. The mean of the absolute contributions gives the overall importance of the variables.
Everyday example: If a customer's tendency to use the coupon comes out above average, SHAP can show how much of this comes from frequent shopping and how much from past campaign participation, and that age pulls the tendency down slightly.
What it does: Mean |SHAP| variable contributions with XGBoost pred_contribs; not produced in the GradientBoosting engine.
Effect on the result: The model does not change; an additional table of variable contributions is produced.
On the validation data: Per-observation SHAP contributions and the mean |SHAP| ranking are produced for iris (multi-class), Pima and Boston.
Imbalanced classes: balanced weights
Concept: Each observation in the rare class is given a weight inversely proportional to the class size, so errors in that class take up more of the loss function and the model cannot ignore the minority class.
Everyday example: If only 15% of customers use the coupon, the model tends to say 'nobody uses it'. With balanced weights each coupon user counts for more, and the model learns to recognise these customers.
What it does: Per-class instance weight n/(K·n_k); the loss of the minority class is enlarged.
Effect on the result: Probabilities, class predictions and AUC change. No effect with balanced classes.
On the validation data: XGBoost sample weight n/(K·n_k); Pima accuracy from 0.675 to 0.65, AUC from 0.6923 to 0.7033. In CV the weights are computed from the training part. It has no effect in iris because it is balanced.
Hyperparameter tuning
Concept: Possible combinations of learning rate, depth and number of rounds are tried by cross-validation within the training set, and the one with the best average performance is selected. Because these settings interact, they must be searched together rather than one at a time.
Everyday example: The supermarket tests every combination of '300 rounds at 0.05 or 50 rounds at 0.3; depth 3 or 6?' on the training customers and picks the most consistent one.
What it does: Selects the best setting on an eta x depth x rounds grid by CV within the training set.
Effect on the result: The model with the selected setting is reported; test performance changes with the selection.
On the validation data: eta {0.05, 0.1, 0.3} x depth {2, 3, 6} x rounds {100, 200}; iris best 0.05, 2, 100 (0.95), Boston 0.1, 3, 200 (0.88).
Test proportion, CV folds and repeats, Random seed, Probability threshold, Permutation importance, Save predictions, Additional metrics, Partial dependence, Chart
Concept: The data are split into training and test sets, cross-validation gives an extra measure of generalisation on the training set, and the seed makes these random splits reproducible. The probability threshold sets the value above which a probability becomes a 'positive' decision; permutation importance, partial dependence and additional metrics show what the model relies on and how it performs with imbalanced classes.
Everyday example: If the coupon budget is tight, the supermarket raises the threshold and sends coupons only to customers with a high probability of using them; with a larger budget it lowers it. The partial dependence plot shows beyond which basket value the probability of coupon use levels off.
What it does: Split proportion and seed, CV within training, probability threshold in binary classification, permutation importance on the test set, saving predictions, additional metrics, partial dependence and plot.
Effect on the result: The split, folds and seed change the test and CV scores; the threshold changes only the class predictions; the others produce additional output.
On the validation data: Threshold 0.3 (Pima): accuracy from 0.675 to 0.725. Boston XGBoost 5-fold CV R² 0.847. The additional metrics, permutation importance, PDP and prediction saving are produced.
Example on real data
Data: Pima Indians (Pima.tr), R MASS package (n = 200)
Variables: target type (diabetes Yes/No), features number of pregnancies, glucose, blood pressure, skin thickness, BMI, pedigree function and age
- Question: How accurately does gradient boosting (XGBoost) classify diabetes (type) with seven clinical measurements, and which variables stand out?
- Model: XGBoost, 100 trees, learning rate 0.3, maximum depth 6 (library defaults); training n = 160, test n = 40 (stratified, seed 42).
- Result: On the test set accuracy = .68, precision = .53, recall = .57, F1 = .55, ROC-AUC = .69; 8 of 14 diabetic women are detected, with 7 false positives.
- Generalization: 5-fold CV accuracy M = .73 (SD = .09).
- Variable importance: In mean gain per split glucose (0.227) and age (0.212) lead, followed by BMI (0.148) and pedigree function (0.131).
- Limitation: The default depth and number of rounds are prone to overfitting in small data; the learning rate, depth and early stopping should be tuned.
- APA: XGBoost (100 trees, eta = 0.3, depth 6) on the test set (n = 40) gave accuracy = .68, F1 = .55, ROC-AUC = .69; 5-fold CV accuracy M = .73 (SD = .09).
Related concepts
SHAP valuesVariable importance (gain, cover, frequency)Early stoppingSubsample ratio (stochastic boosting)Train-test split (holdout)Number of trees (ensemble models)Minimum leaf / node sizeHyperparameter tuning (grid search)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.