Statistics Glossary → Node impurity (Gini, entropy)
Node impurity (Gini, entropy)concept
A criterion that measures how mixed the classes are among the observations in a decision-tree node and is used to choose the best split.
Gini impurity is G = 1 − Σ pₖ² and entropy H = −Σ pₖ log pₖ, where pₖ is the proportion of class k in the node. If all observations in a node belong to one class, both measures are zero; if the classes are equally mixed, they reach their maximum (0.5 for Gini with two classes). At each step the tree chooses the variable and cut-point that most reduce the weighted impurity of the child nodes; with entropy, this reduction is called information gain.
In practice Gini and entropy usually produce very similar trees; Gini is slightly faster to compute. The misclassification rate is also an impurity measure, but because it is less sensitive to splits it is preferred for pruning rather than growing. In regression trees, the sum of squares takes the place of impurity. The entropy in latent class analysis comes from the same family of formulas but measures something else: how clearly observations are assigned to classes.
An everyday example
If 50 of the 100 customers in a node bought a product and 50 did not, Gini = 1 − (0.5² + 0.5²) = 0.5. If the question 'Is age under 30?' splits the node into one child with 40 buyers and 10 non-buyers and another with 10 buyers and 40 non-buyers, each child has Gini 1 − (0.8² + 0.2²) = 0.32; impurity falls from 0.5 to 0.32, so this split is a good candidate.
In MerQur
In Decision Tree, the 'Splitting criterion (Gini / Entropy)' chooses the node impurity measure for classification; in Random Forest, the weights of the 'Imbalanced classes: balanced weights' option enter the Gini computation.
Analyses where this concept appears
Random ForestDecision Tree (CART)