Statistics Glossary → Kullback-Leibler (KL) divergence
Kullback-Leibler (KL) divergenceconcept
A non-negative measure of divergence that quantifies how much information is lost when one probability distribution is approximated by another.
For two discrete distributions P and Q, KL(P‖Q) = Σ pᵢ·log(pᵢ/qᵢ). It is always zero or positive, and it is zero only when the two distributions are the same. In information theory it is read as the average extra information needed to encode data that actually come from P using a code built for Q. Maximum likelihood estimation is equivalent to minimising the KL divergence between the data distribution and the model, and the AIC rests on the same idea.
KL is not a distance: it is not symmetric (KL(P‖Q) ≠ KL(Q‖P)) and it does not satisfy the triangle inequality. t-SNE minimises KL(P‖Q) between the neighbourhood probabilities P in the high-dimensional space and Q in the low-dimensional embedding. This direction heavily penalises placing close neighbours far apart but only lightly penalises bringing distant points together, which is why t-SNE preserves local structure well and global structure poorly. In t-SNE the KL value can be compared only between runs on the same data with the same perplexity.
An everyday example
A weather app reports a 50% chance of rain every day, whereas the true share of rainy days in the area is 10%. The KL divergence between the true distribution (P: 10% rain, 90% dry) and the forecast distribution (Q: 50%, 50%) is about 0.37 nats. As the app's forecast approaches the true rate, the value falls to zero.
In MerQur
In t-SNE, the 'KL divergence' output reports the exact KL(P‖Q) computed from the final embedding; a low value shows that local neighbourhoods are well preserved.
Analyses where this concept appears
t-SNE (t-Distributed Stochastic Neighbor Embedding)