Statistics Glossary → Kappa statistic (Cohen's κ)
Kappa statistic (Cohen's κ)concept
A coefficient that measures agreement between two raters or two classifications on categorical decisions, after removing the agreement expected by chance.
The proportion of cases on which two raters make the same decision (observed agreement, p_o) overstates agreement, because some agreement occurs even if both decide at random. Cohen's κ = (p_o - p_e) / (1 - p_e), where p_e is the chance agreement expected from the raters' marginal proportions. κ = 1 indicates perfect agreement and κ = 0 agreement at chance level; negative values mean agreement worse than chance. The Landis and Koch labels (0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, above 0.81 almost perfect) are widely used but arbitrary.
κ is affected by how common the categories are and by rater bias: when one category is very common, κ can be low even though observed agreement is high (the kappa paradox). For ordered categories weighted kappa is used, and for more than two raters Fleiss' κ or Krippendorff's α. In classification models, κ summarizes how far the model's predictions agree with the true classes beyond chance. For agreement on continuous measurements, the intraclass correlation coefficient (ICC) or the Bland-Altman method is appropriate rather than kappa.
An everyday example
Two radiologists classify 100 X-rays as 'fracture' or 'no fracture' and agree on 85 of them. Because each says 'no fracture' for about 70% of the films, about 58% agreement would be expected even if they decided at random. κ = (0.85 - 0.58) / (1 - 0.58) ≈ 0.64, a more modest level of agreement than the raw 85% first suggests.
In MerQur
It is offered in the Cohen's Kappa analysis itself (including Fleiss κ and Krippendorff α) with the 'Category κ + symmetry tests' extra output, in the True Skill Statistic with the 'TSS CI + Kappa + AUC' option, and in Confusion Matrix Metrics with the 'Kappa SE/CI + McNemar + NIR test' output. It is also among the additional metrics of the Random Forest, Support Vector Machine, Decision Tree and k-Nearest Neighbors classifiers.
Analyses where this concept appears
Cohen's Kappa (including Fleiss κ and Krippendorff α)True Skill Statistic (TSS)Confusion Matrix MetricsRandom ForestSupport Vector Machine (SVM)Decision Tree (CART)k-Nearest Neighbors (KNN)
Related concepts
Weighted kappaFleiss' κ and Krippendorff's α (multiple raters)PABAK and κ_max