MerQuR+Statistics Glossary

Statistics Glossary → PABAK and κ_max

PABAK and κ_maxconcept

Two companion measures that help interpret a Cohen's kappa that is misleadingly low because categories are very unevenly distributed or the raters have different tendencies.

Kappa corrects the observed agreement (p_o) for the agreement expected by chance (p_e). When one category is very common, p_e becomes very high and κ comes out low despite high observed agreement; this is known as the 'kappa paradox'. The prevalence-adjusted bias-adjusted kappa (PABAK; Byrt, Bishop and Carlin, 1993) assumes equally likely categories: PABAK = (k·p_o − 1)/(k − 1). With two categories this reduces to 2·p_o − 1.

κ_max is the highest kappa attainable given the raters' marginal distributions. The ratio of κ to κ_max shows how much of the agreement allowed by the marginal differences was actually reached. For 2x2 tables, the prevalence index |a − d|/n and the bias index |b − c|/n are also reported. PABAK does not replace kappa; it only makes the influence of prevalence visible, so κ, PABAK and these indices should be reported together.

An everyday example

Two radiologists decide whether 100 X-rays show a fracture. Both say 'no fracture' for 90 images, both say 'fracture' for 2, and they disagree on 8. Although observed agreement is 92%, fractures are so rare that κ ≈ 0.29. PABAK is 2 × 0.92 − 1 = 0.84. The gap shows that the low κ is largely due to the low prevalence.

In MerQur

In Cohen's Kappa, the 'Extra outputs: PABAK / κ_max' option gives PABAK, κ_max and, for 2x2 tables, the prevalence and bias indices, all intended to be reported alongside κ.

Analyses where this concept appears

Cohen's Kappa (including Fleiss κ and Krippendorff α)

Related concepts

Kappa statistic (Cohen's κ)