Statistics Glossary → Fleiss' κ and Krippendorff's α (multiple raters)
Fleiss' κ and Krippendorff's α (multiple raters)concept
Coefficients that measure how well more than two raters agree when classifying or scoring the same units, correcting for chance agreement.
Cohen's κ is defined only for two raters. Fleiss' κ extends it to many raters: for each unit, the proportion of rater pairs choosing the same category is computed and averaged, then compared with the chance agreement expected from the category proportions: κ = (P̄ - P_e) / (1 - P_e). Each unit must be rated by the same number of raters, but they need not be the same people.
Krippendorff's α is a more general framework: α = 1 - D_o / D_e, the observed disagreement relative to the disagreement expected by chance. It allows missing values, and a difference function suited to the level of measurement (nominal, ordinal, interval, ratio) can be chosen. For both coefficients 1 means perfect agreement and 0 means chance-level agreement; when the category distribution is very unbalanced they can be low despite high raw agreement (the kappa paradox).
An everyday example
Five radiologists classify the same 40 X-rays as 'normal', 'suspicious' or 'diseased'. Most of them seem to agree on most films, but since most films are normal anyway, high agreement would be expected by chance too. A Fleiss' κ of 0.62 shows the agreement left once this chance share is removed.
In MerQur
The 'Multiple raters' option of the Cohen's Kappa analysis (None (Cohen's κ), Fleiss' κ, Krippendorff's α nominal, ordinal, interval and ratio) and 'Additional raters'.
Analyses where this concept appears
Cohen's Kappa (including Fleiss κ and Krippendorff α)Intraclass Correlation Coefficient (ICC)