MerQuR+Statistics Glossary

Statistics Glossary → Intraclass Correlation Coefficient (ICC)

Intraclass Correlation Coefficient (ICC)analysis

An analysis that expresses, as a reliability coefficient between 0 and 1, how closely several raters, devices or measurement occasions agree when they measure the same subjects.

Concept

A measurement is only useful if it gives similar results regardless of who or what does the measuring. The intraclass correlation coefficient (ICC) addresses this: how much of the total variability in the scores comes from real differences between subjects, and how much from differences between raters and measurement error? An ordinary Pearson correlation only asks whether two sets of measurements rise and fall together; it stays high even if one rater systematically gives everyone 5 points more. The ICC can, if required, take such systematic differences into account.

The logic rests on analysis of variance: the ICC is roughly the ratio of between-subject variance to total variance, ICC = σ²(subject) / (σ²(subject) + σ²(rater) + σ²(error)). Which components enter the denominator depends on the model: the one-way model, where each subject is rated by different raters; the two-way random model, where raters are treated as a sample from a larger pool; and the two-way mixed model, where only these particular raters are of interest. A further distinction is made between absolute agreement (identical scores) and consistency (identical ranking), and between the reliability of a single rater and that of the average of several raters.

The closer the coefficient is to 1, the better the agreement; a common guideline reads values below .50 as poor, .50-.75 as moderate, .75-.90 as good and above .90 as excellent reliability. The confidence interval matters as much as the point value: with few subjects it is very wide and may even include zero. The ICC type used must always be reported, because different types can give quite different values from the same data.

When to use it

  • In inter-rater agreement and test-retest reliability studies where several raters, devices or occasions score the same subjects on a continuous scale.
  • If the raters make categorical decisions (e.g. present / absent), use Cohen's or Fleiss' kappa instead of the ICC.
  • To judge whether two measurement methods are interchangeable in terms of the size of their differences, add a Bland-Altman analysis to the ICC.

An everyday example

At a gymnastics competition, three judges score the same 20 gymnasts' routines out of 10. If one judge gives everyone about half a point more than the others, their scores will not match exactly even if all three rank the gymnasts in the same order. The ICC tells you how much of the variation in the scores is explained by real differences in the gymnasts' performances: a value like 0.85 shows that the judges are largely measuring the same thing, while a value like 0.30 shows that a gymnast's score depends heavily on which judge they happen to get.

Parameters in MerQur

Raters/Measurements (≥ 2)

Concept: For the ICC the data are laid out in wide format: each row is a subject being measured and each column a rater, device or time point measuring that subject. Agreement can only be assessed if each subject has been measured at least twice.

Everyday example: Each row of the table is a gymnast and the three columns are the scores the three judges gave that gymnast. If a judge did not score a gymnast, that gymnast drops out of the comparison, since all three scores are needed to set the judges side by side.

What it does: Each column is a rater, device or time point; each row is a subject (wide format). At least 2 columns are needed.

Effect on the result: A subject with any missing column is dropped (listwise); an error is given if n < 5.

On the validation data: In M255 1413 of the 1428 rows (the complete rows) are used.

ICC model (All (table of 6 types), One-way random, Two-way random, Two-way mixed)

Concept: The ICC model reflects how the raters were chosen and to whom the result should generalise. In the one-way model each subject is measured by different raters; in the two-way random model the raters are regarded as a sample from a larger pool; in the two-way mixed model interest lies only in these particular raters.

Everyday example: If each gymnast was scored by whichever three judges happened to be free, the one-way model fits. If the same three judges scored everyone and the result should generalise to judges at other competitions, choose the two-way random model; if only the agreement among these three judges matters, choose the two-way mixed model.

What it does: "All" tabulates the six Shrout-Fleiss/McGraw-Wong types; the other options additionally report the selected model. If raters vary across subjects a one-way, if they are a random sample a two-way random, and if only these raters matter a mixed model is suitable (Koo and Li, 2016).

Effect on the result: The selected-ICC section and the starred row of the table. The two-way random and mixed models give the same number; only the interpretation note differs.

On the validation data: In the Shrout-Fleiss data ICC(1,1) = 0.1657 [-0.1329, 0.7226], F(5, 18) = 1.795, p = 0.165; ICC(A,1) = 0.2898 [0.0188, 0.7611]; ICC(C,1) = 0.7148 [0.3425, 0.9459].

→ Intraclass correlation (ICC)

ICC type (Absolute agreement, Consistency)

Concept: Absolute agreement asks whether the raters give exactly the same score to the same subject, so systematic differences between raters count as error. Consistency asks only whether the ranking and relative differences are the same; a rater who gives everyone a fixed amount more is not penalised.

Everyday example: If one judge gives every gymnast exactly half a point more than the other two, the ranking is unchanged and consistency stays high. But if a medal depends directly on the score, that half point matters, and absolute agreement should be chosen.

What it does: Absolute agreement adds the systematic difference between rater means to the denominator; consistency does not. Choose absolute agreement for interchangeability of measurements, consistency for rank agreement only.

Effect on the result: Effective only in two-way models. For absolute agreement the confidence interval uses Satterthwaite degrees of freedom (McGraw and Wong, 1996).

On the validation data: Shrout-Fleiss single measures 0.2898 (absolute) and 0.7148 (consistency); the rater means differ a lot (MSC = 32.49, MSE = 1.02).

→ Absolute agreement and consistency (ICC types)

Unit (Single measures, Average measures)

Concept: The single-measures ICC gives the reliability of one rater's score, for when only one rater will be used in practice; the average-measures ICC gives the reliability of the mean of several raters, for when that mean will be used. Averaging several scores partly cancels individual errors, so the mean is more reliable.

Everyday example: If a gymnast's final score is the average of the three judges, report the average-measures ICC. If a small club competition can only provide one judge, the single-measures ICC tells you how reliable that one judge's score is; this value is usually lower.

What it does: Gives the reliability of a single rater or of the mean of k raters (Spearman-Brown).

Effect on the result: Effective except under "All"; the average-measures ICC is always larger than the single-measures one.

On the validation data: In the Shrout-Fleiss data ICC(A,1) 0.2898 while ICC(A,k) 0.6201 [0.0711, 0.9272]; ICC(C,k) 0.9093 (equal to Cronbach's alpha).

→ Spearman-Brown formula (single and average measures)

Confidence Level (%90, %95, %99, custom)

Concept: Sets how much assurance there is that the interval computed from the sample captures the true ICC in the population. The higher the confidence level, the wider the interval.

Everyday example: If the judges' agreement is ICC = 0.80, you can say with 95% confidence that the true agreement lies between, say, 0.62 and 0.91. Asking for 99% confidence widens the interval further with the same data.

What it does: Sets the confidence interval level of the selected ICC and the exact table. The default six-type table and the "Recommended" row are always 95%; with model "All" and the exact table off, this field is disabled.

Effect on the result: Only the selected ICC and the "exact confidence interval" additional output.

On the validation data: In the Shrout-Fleiss data ICC(A,1) 90% [0.0429, 0.6911], 95% [0.0188, 0.7611], 99% [-0.0129, 0.8703]; mixed ICC(C,k) 97.5% [0.6094, 0.9897].

→ Confidence level and confidence interval

Test value (H0: ICC = rho0)

Concept: The usual test only asks whether the ICC is greater than zero, yet an agreement just above zero is usually inadequate in practice. The test value lets you test whether the ICC is significantly greater than a pre-set threshold (e.g. 0.70).

Everyday example: The federation requires an agreement of at least 0.70 for a judging panel to count as reliable. With the test value set to 0.70, the question becomes not 'do the judges agree at all?' but 'does their agreement clear the 0.70 bar?'.

What it does: Tests with an F test whether the ICC is greater than rho0 (McGraw and Wong, 1996); 0 is the usual test.

Effect on the result: The F, the second degrees of freedom (Satterthwaite for absolute agreement) and p in the selected ICC and the exact table; the ICC and confidence interval do not change.

On the validation data: In the Shrout-Fleiss data, rho0 = 0.3, ICC(A,1): F = 0.956, df2 = 4.75, p = 0.522 (at rho0 = 0 F = 11.03, p = 0.0001). ICC(A,k): F = 3.035, df2 = 7.14, p = 0.088.

→ Null hypothesis (H₀) and test value

Additional output: 6 ICC types: selected CI level and 4 decimals (exact formulas)

Concept: Shows the six common ICC types (one-way, two-way absolute agreement and consistency, each for single and average measures) in one table, with the chosen confidence level and test value and more decimals. It makes it easier to see why the types give different values.

Everyday example: Seeing side by side that the judges' agreement is 0.62 for a single judge, 0.83 for the average of three, and somewhat higher under the consistency definition makes it easier to decide which figure to report.

What it does: Recomputes the six types with the confidence level and rho0 of the form, to 4 decimals.

Effect on the result: Additional table only.

On the validation data: In the Shrout-Fleiss data at 90% and rho0 = 0.3 ICC(1,1) F = 0.661 (p = 0.657); ICC(C,1) F = 4.063 (p = 0.016).

→ Intraclass correlation (ICC)

Additional output: ANOVA / variance components table

Concept: The analysis-of-variance table from which the ICC is computed: it splits the total variability among subjects, raters and error. A significant rater row means the raters' average scores differ systematically.

Everyday example: The table shows whether most of the score differences come from the gymnasts genuinely performing differently, or from some judges being generally generous and others generally strict.

What it does: Gives the sums of squares/mean squares/F/p for subjects, raters, error and within subjects, and the two-way variance components.

Effect on the result: Additional table only.

On the validation data: Shrout-Fleiss sums of squares: subjects 56.208, raters 97.458, error 15.292, total 168.958. Variance components: subjects 2.556, raters 5.244, error 1.019.

→ Variance components

Additional output: SEM and MDC95 (test-retest)

Concept: The standard error of measurement (SEM) gives, in score units, how far a person's observed score typically departs from their true score. The minimal detectable change (MDC95) is the smallest difference between two measurements that is too large to be explained by measurement error.

Everyday example: If the SEM is 0.3 points, a gymnast's score typically wobbles by about ±0.3 points because of measurement error. If the MDC95 is about 0.8, a rise of 0.5 points between two competitions is not evidence of real improvement; a gain of at least 0.8 points is needed.

What it does: SEM = SD x sqrt(1 - ICC) (SD is the standard deviation of all scores) and MDC95 = 1.96 x sqrt(2) x SEM; a sqrt(MSE)-based SEM is also given.

Effect on the result: Additional output only; depends on the selected ICC model (ICC(A,1) is used for "All").

On the validation data: In the Shrout-Fleiss data with 'All' and ICC(A,1): SD = 2.710, SEM = 2.284, MDC95 = 6.33; with mixed consistency selected, ICC(C,1) = 0.7148 and SEM = 1.447.

→ Standard error of measurement (SEM) and minimal detectable change (MDC)

Example on real data

Data: Interpretation: R irr anxiety (n = 20 subjects, 3 raters). Parameter examples: the Shrout-Fleiss data and the M255 data.

Variables: raters rater1, rater2, rater3 (anxiety ratings of 20 subjects)

  • Question: In the R irr anxiety data, how well do the anxiety ratings given by three raters to 20 subjects agree with each other?
  • Model choice: Since the raters are regarded as sampled from a wider population of raters, the two-way random, absolute agreement, single-measures ICC(A,1) is recommended.
  • Result: ICC(A,1) = 0.198, 95% CI [-0.039, 0.494], F(19, 38) = 1.83, p = .056; agreement is not significantly greater than zero and is "poor" by the Cicchetti (1994) criterion.
  • Other types: One-way ICC(1,1) = 0.175 and consistency ICC(C,1) = 0.216; reliability is higher for the mean of the three raters (ICC(A,k) = 0.426, ICC(C,k) = 0.453 "fair").
  • Additional finding: The rater main effect (mean square 4.817, F = 3.33, p = .047) shows that the raters' mean scores differ systematically; this makes absolute agreement lower than consistency.
  • Limitation: n = 20 is a small sample; the confidence intervals are very wide and include zero.
  • APA: ICC(A,1) = .20, 95% CI [-0.04, 0.49], F(19, 38) = 1.83, p = .056.

Related concepts

Null hypothesis (H₀) and test valueAnalysis of variance (ANOVA) tableIntraclass correlation (ICC)Standard error of measurement (SEM) and minimal detectable change (MDC)Absolute agreement and consistency (ICC types)Spearman-Brown formula (single and average measures)Variance componentsFleiss' κ and Krippendorff's α (multiple raters)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.