Statistics Glossary → Random Forest
Random Forestanalysis
A machine learning method that predicts a class or a numeric value by growing many decision trees on different resamples of the data and random subsets of the variables, then combining the trees' votes (or averaging their predictions).
Concept
A single decision tree produces readable rules but is unstable: a small change in the data can lead to a quite different tree, and the tree tends to memorise the training data. A random forest overcomes this instability by relying on the joint decision of hundreds of slightly different trees. The goal is to predict new observations as accurately as possible and to see which variables contribute most to the prediction.
Each tree is grown on a bootstrap sample drawn from the data with replacement, and at each split only a small random subset of the variables is tried rather than all of them. These two sources of randomness make the trees less alike, and the average of dissimilar trees fluctuates far less than any single tree. In classification the result is the majority vote of the trees; in regression it is the average of their predictions. Observations left out of a tree's bootstrap sample (about one third) form a natural validation set for that tree; the OOB (out-of-bag) error computed from them estimates generalisation performance without a separate test set.
The method needs no distributional assumptions and picks up non-linear relationships and interactions on its own. On the other hand, the result cannot be summarised in a single formula; the role of the variables is read from importance measures and partial dependence plots. Importance based on impurity decrease can favour variables that take many distinct values, so it should be read alongside permutation importance. In regression a forest cannot predict outside the range seen in the training data. Performance should always be judged on data the model has not seen (a test set, cross-validation or OOB).
When to use it
- When a class or a numeric value is to be predicted from many explanatory variables and the relationships cannot be assumed to be linear (species distribution, land cover, disease risk, customer behaviour).
- If easily read rules matter more than predictive accuracy, a single decision tree is preferred; if the direction and size of effects must be reported as coefficients, logistic or linear regression is preferred.
- On tabular data, when a little extra predictive accuracy is worth the time spent on tuning, gradient boosting is a strong alternative to the forest.
An everyday example
A family health centre wants to predict diabetes risk from measurements such as blood sugar, body mass index, age and blood pressure taken at check-ups. Instead of one doctor's judgement, imagine 500 doctors voting separately: each has seen only some of the patients' files and, at each question, looked at only some of the measurements. If 340 of the 500 votes say 'diabetic' for a person, the forest classes that person as high risk; the variable importance may show that the decisions relied most on blood sugar, followed by age.
Parameters in MerQur
Number of trees
Concept: The number of trees is the number of independent 'opinions' being averaged. Adding trees does not make the forest overfit; the error levels off after a certain number and only the computing time grows.
Everyday example: A vote among 10 doctors can easily swing on a few odd decisions, whereas a vote among 500 gives almost the same result every time. If going from 500 to 1000 hardly changes anything, enough trees have been grown.
What it does: Number of trees in the forest (ntree); many trees reduce prediction variance.
Effect on the result: Test performance and OOB; with very few trees some observations have no OOB prediction.
On the validation data: (iris, test accuracy and OOB accuracy): 10 trees 0.9667 and 0.9417; 100 trees 0.90 and 0.95; 500: 0.9333 and 0.95; 1000: 0.9333 and 0.95.
mtry (variables per split)
Concept: The number of variables tried as candidates at each split. A small value makes the trees differ from one another but weakens each tree; a large value strengthens the trees but makes them more alike.
Everyday example: If every doctor saw all the measurements at every question, they would all look at blood sugar first and become copies of one another. When only two measurements are shown at each question, some doctors also weigh age or weight, and the joint decision becomes more robust.
What it does: Number of variables randomly tried at each split. 0 = default: sqrt(p) in classification, max(1, floor(p/3)) in regression.
Effect on the result: Sets the correlation between trees and the strength of a single tree; test performance, OOB and variable importance change.
On the validation data: iris mtry 1: 0.9333, mtry 4: 0.9667. Boston mtry 4 (p/3) test R² 0.8754, OOB R² 0.8544, CV 0.8437.
Minimum leaf size (nodesize)
Concept: The minimum number of observations a leaf (a final decision of the tree) must contain. A larger value stops the tree from memorising details specific to a handful of people.
Everyday example: With a minimum leaf size of 1, a tree can build a rule such as 'the one person of this age, weight and blood pressure is diabetic', specific to a single patient. Requiring at least 20 people per leaf makes the rules more general.
What it does: Each leaf contains at least this many observations (minimum leaf size); it is not a node-splitting threshold.
Effect on the result: A larger value gives shallower, smoother trees; test performance and probabilities change.
On the validation data: iris 5: 0.9667, 20: 0.9333 (CV 0.9467); Boston 5: R² 0.8347.
Maximum depth
Concept: The largest number of successive questions a tree may ask between its root and a leaf. Shallow trees are simpler and more stable (low variance) but can miss complex relationships (high bias).
Everyday example: A tree of depth 1 asks only 'is blood sugar high?' and decides. With depth 5 it can also tell apart combined situations such as 'blood sugar borderline, but older and overweight?'.
What it does: Tree depth limit (0 = unlimited).
Effect on the result: Shallow trees increase bias and reduce variance; test performance, OOB and AUC change.
On the validation data: iris depth 1 (stumps): test 0.9667, OOB 0.925, AUC 0.9417; depth 2: 0.90. Boston depth 5: R² 0.8783.
Imbalanced classes: balanced weights
Concept: When one class has very few observations, the model tends to predict the majority class. Balanced weighting counts each observation in the rare class more heavily, so that the two classes have an equal say overall.
Everyday example: If only 30 of 100 people at the check-up are diabetic, the trees lean towards saying 'everyone is healthy' and some diabetics are missed. With balanced weights each diabetic person's file counts for more; more patients are caught, though some healthy people may also wrongly look at risk.
What it does: The balanced weight n/(K·n_k) is applied as a sample weight; it enters the Gini and the leaf proportions (it is not used as a prior).
Effect on the result: The weight of the minority class increases; sensitivity, accuracy and AUC may change. No effect with balanced data.
On the validation data: Pima accuracy from 0.65 to 0.675, AUC from 0.7102 to 0.7473. It has no effect in iris and Boston (expected).
Task, Test proportion, CV folds and repeats, Random seed
Concept: To measure the model's performance honestly, the data are split into a training set on which the model is built and a test set it never sees; cross-validation (CV) divides the training set into k parts and uses each part in turn for validation. The random seed makes these random splits reproducible.
Everyday example: The centre builds the forest on 160 of its 200 files and keeps the other 40 as a final exam. Because a 40-person exam is open to chance, 5-fold CV adds another measure, like the average of five different small exams. Re-running the analysis with the same seed puts the same people in the same sets.
What it does: Classification or regression; stratified training/test split; seeded shuffled (stratified in classification) k-fold CV within the training set; number of repeats; seed for the split, CV and permutation.
Effect on the result: The task sets the model and metrics; the split, number of folds and seed change the test and CV results.
On the validation data: iris test proportion 0.1: 0.9333; 0.5: 0.8933. CV with 2, 5, 10 folds: 0.9533, 0.9667, 0.9666; 3x5 repeats 0.9556; seed 1: 0.9667, OOB 0.9333. Boston 5-fold CV R² 0.844.
Hyperparameter tuning (grid search + CV)
Concept: Each possible combination of model settings (variables per split, minimum leaf size) is tried by cross-validation and the best-performing one is selected. The selection is done on the training set only; if the test set were used for tuning, the performance would look better than it really is.
Everyday example: The centre tries every combination of 'show 2 or 3 measurements, require at least 1 or 5 people per leaf?' on the training files and picks the one with the most consistent result. The final exam files are looked at only after this choice.
What it does: Selects the best setting on a grid of mtry and minimum leaf size by CV within the training set.
Effect on the result: The forest built with the selected setting is reported; test performance and importance ranks may change.
On the validation data: In the grid mtry {sqrt, .33, .5, 1} x min_leaf {1, 3, 5}, sqrt and 3 were selected for iris (CV 0.9583), and 0.5 and 1 for Boston (0.8546).
Permutation importance (test set), Save predictions and probabilities to the dataset, Additional output (imbalanced-data metrics, partial dependence), Chart
Concept: Permutation importance looks at how much performance drops when a variable's values are randomly shuffled across people; a large drop means the model relies heavily on that variable. A partial dependence plot shows how the prediction changes as one variable changes, averaging over the others; additional metrics such as Kappa and MCC are more informative than accuracy when classes are imbalanced.
Everyday example: If shuffling blood sugar values across the test files clearly lowers the share of correct predictions, blood sugar is an important clue; if shuffling blood pressure changes nothing, the model does not rely on it. The partial dependence plot shows beyond which blood sugar level the predicted risk rises sharply.
What it does: The drop in performance when each variable is shuffled in the test set; writing the prediction and probability columns to the dataset; additional metrics (Kappa, MCC, NIR, McNemar); partial dependence (PDP/ICE); variable importance, ROC or PDP chart.
Effect on the result: The chart type changes only the drawing; no ROC chart is produced for a multi-class response.
Probability threshold (binary)
Concept: The model gives each person a probability; the threshold sets the value above which this probability becomes a 'positive' class decision. Lowering the threshold catches more positives (higher sensitivity) but also raises false alarms; the probabilities and the AUC themselves do not change.
Everyday example: With the default threshold of 50%, a person with a 40% probability of diabetes is labelled 'healthy'. If the centre does not want to miss anyone, it can lower the threshold to 30% and call this person in for a further test as well; the price is that some truly healthy people are also called in needlessly.
What it does: In binary classification, predict positive if P(positive) >= threshold.
Effect on the result: Probabilities and AUC do not change; class predictions, accuracy, precision, recall and F1 change.
On the validation data: Pima threshold 0.3: accuracy 0.625, F1 0.5946; 0.7: accuracy 0.675, F1 0.2353. It has no effect in the multi-class case (expected).
Example on real data
Data: Pima Indians (Pima.tr), R MASS package (n = 200)
Variables: target type (diabetes Yes/No), features number of pregnancies, glucose, blood pressure, skin thickness, BMI, pedigree function and age
- Question: How accurately does a random forest built on seven clinical measurements classify diabetes (type) in Pima women, and which variables stand out?
- Model: A forest of 100 trees (√p = 2 variables per split, at least 1 observation per leaf); training n = 160, test n = 40 (stratified, seed 42). OOB accuracy = .73 (OOB error .27); class-wise OOB error No = .16, Yes = .48.
- Result: On the test set accuracy = .65, precision = .50, recall = .57, F1 = .53, ROC-AUC = .71; 8 of 14 diabetic women are detected and 8 of 26 healthy women are false positives.
- Generalization: 5-fold CV accuracy M = .76 (SD = .09); the test accuracy is below the CV mean because of the small test set.
- Variable importance: The most important variable is glucose (0.244), followed by age (0.159), BMI (0.145) and number of pregnancies (0.141); blood pressure contributes least (0.081).
- Limitation: The forest is random; the metrics change when the seed or the split changes. Sensitivity for the diabetic class is low; balanced class weights or threshold tuning can be tried.
- APA: A random forest (100 trees) on the test set (n = 40) gave accuracy = .65, F1 = .53, ROC-AUC = .71; 5-fold CV accuracy M = .76 (SD = .09).
Related concepts
mtry (variables tried per split)Permutation importanceMinimum leaf / node sizeVariable importance (gain, cover, frequency)Train-test split (holdout)Class imbalance and balanced weightsNumber of trees (ensemble models)Maximum tree depth
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.