MerQuR+Statistics Glossary

Statistics Glossary → Weighted kappa

Weighted kappaconcept

A version of Cohen's kappa for ordered categories that credits not only exact agreement between two raters but also partial agreement in neighbouring categories.

Unweighted Cohen's kappa counts as agreement only the cases where two raters choose exactly the same category, and subtracts the agreement expected by chance. When categories are ordered (mild, moderate, severe), a disagreement between 'mild' and 'moderate' is smaller than one between 'mild' and 'severe'. Weighted kappa assigns an agreement weight to each pair of categories: with linear weights the weight falls in proportion to the distance, with quadratic weights in proportion to its square.

Quadratic weighted kappa penalizes large disagreements more heavily and, under certain conditions, is equivalent to the intraclass correlation coefficient. Because the choice of weights can change the result considerably, it should be fixed in advance and reported. Kappa is sensitive to how common the categories are and to the raters' tendencies: with very unbalanced distributions, kappa can be low despite high observed agreement. If the categories are unordered (nominal), unweighted kappa should be used.

An everyday example

Two radiologists grade the disease stage on 100 X-rays from 1 to 4. On one image one says 2 and the other 3; on another, one says 1 and the other 4. Unweighted kappa treats both as equal disagreements. With linear weights the first counts as 2/3 agreement and the second as none; with quadratic weights the first counts as 8/9 agreement.

In MerQur

In the Cohen's Kappa analysis, the 'Weight' option: Unweighted (nominal / binary), Linear weighted or Quadratic weighted (ordinal). The 'Custom weight matrix' field computes an extra kappa row with user-supplied weights.

Analyses where this concept appears

Cohen's Kappa (including Fleiss κ and Krippendorff α)

Related concepts

Kappa statistic (Cohen's κ)