MerQuR+Statistics Glossary

Statistics Glossary → Cohen's Kappa (including Fleiss κ and Krippendorff α)

Cohen's Kappa (including Fleiss κ and Krippendorff α)analysis

A coefficient that measures how well two or more raters agree when sorting the same items into categories, after removing the agreement that would occur by chance alone.

Concept

When two raters (e.g. two doctors, two teachers, two coders) classify the same cases, how often they reach the same decision tells us about the reliability of the measurement. Simple percentage agreement is misleading, though: even two people deciding completely at random will agree some of the time, especially if one category is very common.

Kappa subtracts the agreement expected by chance from the observed agreement and divides by the largest possible excess: κ = (p_o − p_e) / (1 − p_e). Here p_o is the proportion of cases on which the raters chose the same category, and p_e is the chance agreement computed from how often each rater uses each category. κ = 1 means perfect agreement, κ = 0 agreement no better than chance, and negative values agreement worse than chance. For ordered categories, weighted kappa gives partial credit for disagreements between neighbouring categories; with more than two raters, Fleiss' kappa or Krippendorff's alpha is used.

Kappa depends on how common the categories are and on the raters' tendencies to use them: κ can be low even when percentage agreement is very high if one category dominates. So the observed percentage agreement, and where needed measures such as PABAK, should be reported alongside κ. Verbal benchmarks (e.g. 0.41-0.60 'moderate') are only a rough guide. Kappa measures agreement, not whether the raters are right.

When to use it

  • To measure how well two raters agree in the categorical decisions they make about the same cases (inter-rater reliability, agreement between two diagnostic methods).
  • For ordered categories use weighted kappa; with three or more raters use Fleiss' kappa, and with missing ratings or continuous scores use Krippendorff's alpha.
  • If the measurements are continuous (e.g. blood pressure from two devices), the intraclass correlation coefficient or Bland-Altman analysis is appropriate instead of kappa.

An everyday example

At a school, two teachers independently grade the same 100 essays as 'poor', 'fair', 'good' or 'excellent'. They give the same grade to 70 essays, so observed agreement is 70%. But given how often each teacher uses each grade, about 30% agreement would be expected even if they graded at random. Kappa = (0.70 − 0.30) / (1 − 0.30) ≈ 0.57: the teachers agree clearly beyond chance, but their grading criteria are not quite the same.

Parameters in MerQur

Rater 1 and Rater 2

Concept: Two columns holding the two raters' decisions about the same cases; each row is one case. The agreement table has the first rater's decisions in its rows and the second's in its columns, and the diagonal shows the cases on which they made the same decision.

Everyday example: In the school example each row is one essay; the first column holds the first teacher's grade and the second column the second teacher's. Swapping the two teachers does not change kappa.

What it does: The row and column of the agreement table. The category sets of both sides are merged; an unobserved category is added as a zero row/column. Subjects with missing values are excluded pairwise.

Effect on the result: κ, SE, z, p, category κ, Bowker, and Stuart-Maxwell are unaffected by swapping the raters. The weighted κ depends on the category order; numeric codes are sorted numerically, text labels alphabetically, and a warning is written.

On the validation data: In SkinCondition, swapping the raters keeps the linear κ 0.5082 and Bowker 4.5354 the same. In the UCLA film data the table is completed to 3x3: κ = 0.5000, ASE 0.1559, CI [0.1944, 0.8056].

Weight (Unweighted (nominal/binary), Linear weighted (ordinal), Quadratic weighted (ordinal))

Concept: When categories are ordered, accounts for the fact that not every disagreement is equally serious. Unweighted kappa counts only identical decisions as agreement; linear weights give partial credit to neighbouring categories, and quadratic weights penalise disagreements between distant categories much more heavily.

Everyday example: If one teacher says 'good' and the other 'excellent', that is a small disagreement; 'poor' against 'excellent' is a large one. Weighted kappa makes this distinction and, since the grading scale is ordered, suits the school example better.

What it does: The unweighted κ counts only exact agreement on the diagonal. Linear (Cicchetti-Allison) w = 1 - |i-j|/(k-1) and quadratic (Fleiss-Cohen) w = 1 - ((i-j)/(k-1))² give credit for partial agreement. The CI uses the SE under H1, the z test the SE under H0.

Effect on the result: κ, SE, CI, z, p, p_o, p_e change; category κ, Bowker, Stuart-Maxwell, and PABAK do not.

On the validation data: SkinCondition (n = 88): unweighted 0.3449 (SE 0.0724), linear 0.5082 (SE 0.0655), quadratic 0.6607 (SE 0.0616). Eye data (n = 7477): 0.5954 / 0.6524 / 0.7023.

→ Weighted kappa

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets the width of the confidence interval for kappa and the threshold (α) for deciding whether agreement differs from chance. Kappa itself does not change.

Everyday example: With 100 essays, the 95% confidence interval for κ = 0.57 might run from about 0.45 to 0.69, so the true agreement lies somewhere between 'moderate' and 'substantial'. At 99% confidence the interval widens.

What it does: alpha = 1 - confidence level; the κ interval (κ ± z·SE), the decision (p < alpha), the custom weighted κ interval, the bootstrap percentile limits, and the Fleiss κ interval are computed at this level.

Effect on the result: κ, SE, z, p do not change; the interval width and the decision for borderline p change. For Krippendorff α there is no interval or p, so the output does not change.

On the validation data: SkinCondition linear κ 0.5082: 80% [0.4242, 0.5921], 90% [0.4004, 0.6159], 95% [0.3798, 0.6366], 99% [0.3394, 0.6769], 99.9% [0.2926, 0.7237].

→ Confidence level and confidence interval

Multiple raters (None (Cohen's κ, 2 raters), Fleiss κ, Krippendorff's α (nominal), Krippendorff's α (ordinal), Krippendorff's α (interval), Krippendorff's α (ratio)) and Additional raters

Concept: Chooses the measure to use when there are more than two raters. Fleiss' kappa generalises kappa to situations where each case is classified by the same number of raters; Krippendorff's alpha allows missing ratings and uses a difference measure suited to nominal, ordinal, interval or ratio data.

Everyday example: If four teachers grade the essays, Fleiss' kappa sums up the agreement of all four in one number. If some teachers never read some of the essays, Krippendorff's alpha can still be computed with those gaps.

What it does: Fleiss κ (Fleiss, 1971) excludes subjects with missing values. Krippendorff α is computed from the coincidence matrix with the selected difference function; it allows missing values and gives no p or interval. The raters are Rater 1 and 2 plus the additional columns.

Effect on the result: Weight, category κ, PABAK, bootstrap, and the custom weight matrix are not used in the multi-rater path, and a note is written in the output. The additional raters list has no effect in the Cohen path.

On the validation data: Wikipedia Fleiss example: κ = 0.2099. Krippendorff (2011) data C: nominal 0.7434, ordinal 0.8154, interval 0.8491, ratio 0.7974.

→ Fleiss' κ and Krippendorff's α (multiple raters)

Extra outputs: Category κ + symmetry tests

Concept: Category kappas show in which categories agreement is good and in which it is weak. Symmetry tests check whether one rater systematically chooses higher or lower categories than the other, that is, whether disagreements pile up in one direction.

Everyday example: The teachers may agree well on 'poor' essays but often split between 'good' and 'excellent'; category kappas reveal this. If the first teacher always gives the higher grade when they disagree, the symmetry tests pick up this difference in strictness.

What it does: For each category, Cohen κ, SE, and z in the "this category / others" 2x2 table; the Bowker symmetry test (pairs with n_ij + n_ji = 0 are skipped and df is reduced accordingly); the Stuart-Maxwell marginal homogeneity test (df = k - 1).

Effect on the result: Extra rows only.

On the validation data: SkinCondition category κ 0.5421 / 0.1904 / 0.2063 / 0.5231; Bowker χ²(5) = 4.5354, p = 0.4752 (a symmetric pair with both cells empty does not count toward the degrees of freedom); Stuart-Maxwell 3.2476, p = 0.3550.

→ Kappa statistic (Cohen's κ)

Extra outputs: PABAK / κ_max

Concept: When one category is very common, or when raters use categories with different frequencies, kappa can be low despite high percentage agreement. PABAK is an agreement measure adjusted for these prevalence and bias effects; κ_max is the largest kappa attainable given how often each rater uses each category.

Everyday example: If 90% of the essays one year are 'good', kappa can be low even with 92% agreement, because the agreement expected by chance is already very high. PABAK and κ_max then help tell whether a low kappa reflects genuinely weak agreement or simply an unbalanced distribution.

What it does: PABAK = (k·p_o - 1)/(k - 1) (Byrt, Bishop, and Carlin, 1993), κ_max (Cohen, 1960); in 2x2 the prevalence index |a - d|/n and the bias index |b - c|/n. They help interpret a low κ under high prevalence or rater bias; they are reported alongside κ.

Effect on the result: Extra rows only.

On the validation data: Wikipedia examples PABAK 0.4 / 0.2 / 0.2, κ_max 0.8 / 0.7826 / 0.4444; Drugs PABAK -0.2857, prevalence 0.0714, bias 0.5000.

→ PABAK and κ_max

Extra outputs: Bootstrap CI

Concept: Finds the confidence interval for kappa by resampling the cases with replacement thousands of times and computing kappa each time, instead of using a formula. It is useful in small or unbalanced samples where the assumptions behind the formula-based interval are doubtful.

Everyday example: It is like building thousands of 'new piles' of 100 essays drawn at random from the original pile, some more than once, and computing the two teachers' kappa for each. The range covered by the middle 95% of those kappas gives the confidence interval.

What it does: Subjects are resampled with replacement 2000 times (seed 12345), κ is computed with the selected weight, and the alpha/2 and 1 - alpha/2 percentiles are taken. The number of resamples cannot be changed in the form.

Effect on the result: Extra interval only; the main κ does not change.

On the validation data: SkinCondition unweighted [0.1979, 0.4898] (asymptotic [0.2030, 0.4868]); eye data linear [0.6388, 0.6659].

→ Bootstrap (resampling)

Custom weight matrix

Concept: Lets the researcher decide how much 'partial agreement' each disagreement counts as. The diagonal corresponds to full agreement (1); the other cells receive weights between 0 and 1 based on subject-matter knowledge.

Everyday example: The school may consider the line between 'poor' and 'fair' critical (it decides pass or fail) but the one between 'good' and 'excellent' minor. A custom weight matrix can give a 'good'-'excellent' disagreement 0.8 credit and a 'poor'-'fair' disagreement 0.

What it does: Computes an extra κ, SE, CI, and p row with the agreement weights supplied by the user (diagonal 1); gives an explicit error if the size is not k x k.

Effect on the result: Extra rows only; the main κ does not change.

On the validation data: The linear matrix entered manually gives 0.5082 (= linear option); the Fleiss-Cohen matrix 0.6607 (= quadratic); a matrix giving 0.5 credit only to the adjacent category 0.4692 (SE 0.0676).

→ Weighted kappa

Category order

Concept: Weighted kappa and ordinal Krippendorff's alpha depend on the distance between categories, so the categories must be in the right order. If they carry text labels, alphabetical order may not match the meaningful order, so the order should be stated explicitly.

Everyday example: Sorted alphabetically, 'excellent', 'fair', 'good', 'poor' would put 'excellent' next to 'fair'. Writing the order as 'poor, fair, good, excellent' lets weighted kappa measure the size of the disagreements correctly.

What it does: Gives the low-to-high order of the categories, separated by commas, for the weighted κ and the ordinal Krippendorff α. It makes the order explicit for ordinal categories with string labels (e.g., "Never Fun ... Always fun").

Effect on the result: The weighted κ, SE, CI, z, and p change; the unweighted κ does not.

On the validation data: In SexualFun, with the natural order the linear κ = 0.2374; without an order MerQur falls back to alphabetical order (with a warning) and κ = 0.0655. MS Winnipeg natural order 0.3797, alphabetical 0.1767.

Example on real data

Data: SkinCondition, SAS/STAT PROC FREQ agreement example

Variables: rater 1 Derm1_kod, rater 2 Derm2_kod (ordinal 1-4 ratings of 88 patients by two dermatologists)

  • Question: Do the four-level ratings (1-4) given by two dermatologists to the skin condition of 88 patients agree beyond chance? The unweighted Cohen kappa is used.
  • Descriptive: Observed agreement p_o = 0.511 (the same rating for about half of the patients); agreement expected by chance alone p_e = 0.254.
  • Result: κ = 0.345, z = 5.64, p < .001. The null hypothesis (κ = 0) is rejected: agreement is above chance.
  • Effect size: κ = 0.34 is "slight" agreement on the MerQur interpretation scale; even the upper limit of the 95% CI [0.203, 0.487] stays below good agreement.
  • Note: Because the ratings are ordinal, a weighted kappa (partial credit for close ratings) may be more appropriate; the linear weighted κ is 0.508.
  • APA: Cohen's κ = .345, 95% CI [.203, .487], z = 5.64, p < .001.

Related concepts

Bootstrap (resampling)Percentiles (quantiles) and percentile definitionsNull hypothesis (H₀) and test valueStandard errorWeighted kappaKappa statistic (Cohen's κ)Fleiss' κ and Krippendorff's α (multiple raters)PABAK and κ_max

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.