Statistics Glossary → Bland-Altman Agreement Analysis
Bland-Altman Agreement Analysisanalysis
An analysis that assesses whether two methods measuring the same thing can be used interchangeably, by computing the mean difference between them (bias) and the limits of agreement within which most individual differences fall.
Concept
Can a new measurement method (cheaper, faster or less painful) replace the old one? Correlation cannot answer this: two methods can be very highly correlated and still give systematically different values; if one always reads 10 units higher than the other, the correlation is perfect but there is no agreement. Bland-Altman analysis looks directly at the differences between the two measurements.
For each item the difference and the mean of the two methods are computed. The mean of the differences is the bias: how much one method reads higher or lower on average. If the differences are roughly normal, about 95% of individual differences lie within the limits of agreement: LoA = bias ± 1.96 × the standard deviation of the differences. The plot of differences against means shows whether the difference changes with the size of the measurement (proportional bias).
Whether the limits of agreement are acceptable is decided by subject-matter knowledge, not by statistics: what size of difference is clinically or practically negligible should be set in advance. Because the bias and limits of agreement are estimated from a sample, they should be reported with confidence intervals; in small samples these intervals are wide. If the difference grows with the size of the measurement, percentage differences, log ratios or regression-based limits of agreement are more suitable.
When to use it
- To assess whether two methods, devices or observers measuring the same quantity can be used interchangeably (method comparison studies).
- If there are several paired measurements per person, use the repeated-measures Bland-Altman method; if the measurements are categorical, Cohen's kappa is appropriate.
- If a method's repeatability or several raters' reliability is to be summarised in one coefficient, the intraclass correlation coefficient or Lin's concordance correlation coefficient can be reported alongside Bland-Altman.
An everyday example
A family practice wonders whether a cheap blood pressure monitor that patients use at home agrees with the standard device in the surgery. The systolic pressure of 50 patients is measured with both. The home device reads 2 mmHg higher on average (bias), and the standard deviation of the differences is 5 mmHg; the limits of agreement come out at about −8 to +12 mmHg. So for a given patient the home device may read anywhere from 8 mmHg below to 12 mmHg above the standard device. If the doctor set ±10 mmHg as the acceptable difference for treatment decisions, the upper limit exceeds this threshold and the home device should not replace the standard one.
Parameters in MerQur
Method 1 and Method 2
Concept: The two measurement methods being compared; each row holds the values of the same item measured by both methods, and the difference is the first minus the second. Putting the reference (old) method second makes the difference easy to read as 'new − old'.
Everyday example: In the blood pressure example the first method is the home device and the second the standard device; a bias of +2 mmHg reads as 'the home device reads 2 mmHg higher on average'. Reversing the order makes the bias −2, but the width of the limits of agreement does not change.
What it does: The two measurements being compared; difference = method 1 - method 2. Putting the established (old) method second makes the "new - old" interpretation easier.
Effect on the result: If the order changes, the sign of the bias, the limits of agreement and the slope changes; SD and interval widths stay the same.
On the validation data: Wright 1 - mini Wright 1: bias -2.1176 [-22.0488, 17.8135]; in reverse order +2.1176 [-17.8135, 22.0488].
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets the level of the confidence intervals for the bias and the limits of agreement. The limits of agreement themselves are defined to cover about 95% of individual differences and do not change with this choice; the confidence intervals show the uncertainty of estimating those limits from a sample.
Everyday example: With 50 patients, if the upper limit of agreement is +12 mmHg its 95% confidence interval might run from about +9.6 to +14.4 mmHg. Most of that interval lies above the doctor's +10 mmHg threshold, but its lower end dips below it; a firmer decision would need more patients.
What it does: Sets the level of the bias interval (t, df = n - 1), the limit-of-agreement intervals (SE = SD·√(3/n), t) and the slope interval. The limits of agreement themselves are always bias ± 1.96·SD (about 95% coverage).
Effect on the result: Only the confidence intervals change; the comment line states that the coverage is always 95%, and the table labels show the selected level.
On the validation data: Bias interval: 90% [-18.5323, 14.2970], 95% [-22.0488, 17.8135], 99% [-29.5786, 25.3433]; the lower end of the lower limit-of-agreement interval is -112.6191 at 95% and -143.4800 at 99.9% (t distribution).
Proportional-bias line
Concept: Adds to the plot the regression line of the differences on the mean of the two measurements. If the line slopes clearly, the difference between methods changes with the size of the measured value (proportional bias), and a single bias value is misleading.
Everyday example: If the home device agrees with the standard one at normal pressures but reads clearly higher at high pressures, the line slopes upward. This means the device is least reliable exactly where treatment decisions matter, at high values.
What it does: Adds the difference ~ mean OLS line to the plot; for visually examining whether the difference changes with the magnitude of the measurement (Bland and Altman, 1999).
Effect on the result: Plot only; the slope test is in the text in every case.
On the validation data: Slope 0.0287 (SE 0.0882, p = 0.7495) is reported in every run.
Difference scale (extra) (Raw difference, Percentage difference (% of mean), Ratio (log scale))
Concept: When differences grow in proportion to the size of the measurement, expresses the difference as a percentage or ratio instead of in raw units. Limits of agreement computed on the log scale, once transformed back, give proportional limits such as 'one method reads between 90% and 115% of the other'.
Everyday example: If a laboratory measures cholesterol with two methods and the differences are 5 units at low values but 20 at high values, percentage differences make more sense than raw ones: for example, 'the new method reads between −6% and +8% of the old one'.
What it does: An additional analysis for when the difference grows in proportion to the mean: percentage difference 100·(y₁ - y₂)/mean or log ratio; on the log scale the limits of agreement are back-transformed to ratios.
Effect on the result: An additional section; the main results use the raw difference. Normality, non-parametric LoA, regression LoA, exact interval, Δ decision and repeated measures are computed on the scaled difference; the headings state the scale.
On the validation data: PEFR log: bias -0.0118, ratio 0.9883, limit-of-agreement ratio [0.7783, 1.2550]; percentage: bias -1.158, LoA [-24.87, 22.55]. Shapiro-Wilk on the log difference W = 0.849, p = 0.010.
Additional output: Normality of differences
Concept: The reading of the limits of agreement as 'covering about 95% of the differences' assumes the differences are roughly normal. A normality test, skewness and kurtosis, and a Q-Q plot check this assumption.
Everyday example: If the home device gave very high readings for a few patients because of a cuff error, the distribution of differences becomes skewed. Limits of agreement based on the normal distribution then misstate the real coverage; non-parametric limits or a look at the outliers are needed.
What it does: The Shapiro-Wilk test of the differences, skewness and kurtosis, and a Q-Q plot; checks the normal-distribution assumption of the limits of agreement.
Effect on the result: An additional section.
On the validation data: PEFR W = 0.9579, p = 0.5931; unbiased skewness/kurtosis G1/G2: 0.283 / 0.371.
Additional output: Lin CCC
Concept: Lin's concordance correlation coefficient (CCC) sums up in one number, between 0 and 1 (down to −1 for negative agreement), both how related the two measurements are and how far they give the same values. Unlike Pearson's correlation it also penalises systematic bias; it is used alongside the limits of agreement, not instead of them.
Everyday example: If the home device always read exactly 10 mmHg higher, Pearson's correlation would be 1 but the CCC would fall clearly below 1. The CCC thus captures the difference between 'the two devices rise and fall together' and 'the two devices give the same value'.
What it does: Lin's (1989) concordance correlation coefficient, its Fisher z interval and the C_b bias correction factor; a single-number summary of agreement, reported alongside the LoA, not instead of it.
Effect on the result: An additional section.
On the validation data: PEFR CCC 0.9427 [95% 0.8505, 0.9787], C_b 0.9994; RV/IC 0.6614 [0.5090, 0.7736].
Additional output: Non-parametric LoA
Concept: If the differences are not normal, takes the limits of agreement from the differences' own percentiles instead of a formula: the values below which 2.5% and 97.5% of the differences fall. A reasonably large sample is needed for it to be reliable.
Everyday example: Had the doctor collected measurements from 500 patients, the differences could be sorted and the cut-offs of the lowest and highest 2.5% taken as the lower and upper limits of agreement. With skewed differences this gives a more honest result than the normal formula.
What it does: The 2.5th, 50th and 97.5th percentiles of the differences (linear interpolation, Hyndman-Fan type 7); appropriate if the differences are not normal and n is large.
Effect on the result: An additional section.
On the validation data: PEFR [-65.8, 68.6], median -8. At n = 17 the extreme percentiles are effectively the smallest and largest observations.
Additional output: Regression-based LoA
Concept: For situations where both the bias and the spread of the differences change with the measurement level, computes the limits of agreement as sloping or curved lines that depend on the size of the measurement, rather than flat lines. A separate range of agreement can then be read for each measurement level.
Everyday example: If the home device deviates by ±6 mmHg around 120 but by ±15 around 180, a single range of agreement is misleading. Regression-based limits let the doctor say 'the device is adequate at normal pressures but not at high ones'.
What it does: Bland and Altman (1999): D = b0 + b1·A, |e| = c0 + c1·A, LoA(A) = D(A) ± 1.96·√(π/2)·(c0 + c1·A); for when the mean or spread of the difference changes with the level of measurement.
Effect on the result: An additional section and curved limits on the plot.
On the validation data: PEFR b0 = -15.0675, b1 = 0.0287, c0 = 67.7181, c1 = -0.0855; RV/IC b0 = 0.4224, b1 = 0.0358.
Additional output: Exact LoA CI
Concept: Computes the confidence intervals of the limits of agreement exactly from the noncentral t distribution instead of an approximate formula. The approximate formula makes the intervals too narrow in small samples.
Everyday example: Had the doctor worked with only 12 patients, the approximate formula would understate the uncertainty of the limits of agreement. The exact interval better reflects how fragile a 'device is adequate' decision based on few patients is.
What it does: Exact limit-of-agreement intervals based on the Carkeet (2015) noncentral t distribution; at small n the approximate SE = SD·√(3/n) interval is too narrow.
Effect on the result: An additional section; the exact interval is used in the Δ decision.
On the validation data: PEFR lower limit [-124.16, -53.10], upper limit [48.86, 119.93]; with the approximate method [-112.62, -43.58] and [39.34, 108.38] (exact interval from noncentral t quantiles).
Clinical acceptance limit Δ
Concept: The largest difference between the two methods regarded as practically negligible (±Δ). If the limits of agreement (and preferably their confidence intervals) lie within this limit, the methods are considered interchangeable. Δ should be set from subject-matter knowledge, without looking at the data.
Everyday example: Before the study the doctor sets Δ = 10, reasoning 'if the two devices differ by at most ±10 mmHg my treatment decision does not change'. With limits of agreement of −8 to +12, the upper limit exceeds Δ and the verdict is 'not in agreement'.
What it does: Reports whether the limits of agreement and their intervals are within a pre-specified ±Δ (0 = none). The acceptance limit must be set without seeing the data.
Effect on the result: An additional row (Yes/No). If a subject is selected, the decision follows the repeated-measures LoA; if a scale is selected, it is interpreted in the unit of the scaled difference.
On the validation data: PEFR Δ = 80: Yes within LoA, No within the interval; Δ = 120: Yes/Yes; Δ = 120 + exact interval: Yes/No.
Subject (repeated measures)
Concept: When there are several paired measurements per person, those measurements are not independent. If the subject variable is given, the variability of the differences is split into between-person and within-person components and the limits of agreement are computed accordingly; treating the pairs as independent misstates the limits.
Everyday example: If the doctor measured each patient's pressure with both devices on three days, there are 150 pairs for 50 patients, but they are not 150 independent people. With the subject column chosen, the analysis takes each patient's repeated measurements into account.
What it does: If there is more than one pair per subject, computes σ²_b + σ²_w and the limits of agreement with the Bland and Altman (2007) "true value varies" method.
Effect on the result: The main LoA, the Δ decision and the APA summary use the repeated-measures LoA; the independent-pairs LoA is given in a separate row. With a single pair per subject an explicit note is written; the subject column cannot be a measurement column.
On the validation data: RV/IC σ²_w = 0.2276, σ²_b = 0.7498, SD = 0.9886, LoA [-1.3356, 2.5399] (treated as independent, SD 0.9611, LoA [-1.2815, 2.4858]); the variance components are computed from the one-way ANOVA mean squares (type I).
Example on real data
Data: Bland and Altman (1986) Lancet, PEFR data
Variables: method 1 wright1 (Wright peak flow meter, first reading), method 2 mini1 (mini Wright meter, first reading)
- Question: Do the Wright peak flow meter and the mini Wright meter give agreeing PEFR (l/min) measurements on the same people (n = 17)?
- Result (bias): Mean difference (wright1 - mini1) = -2.12 l/min, 95% CI [-22.05, 17.81]; t(16) = -0.23, p = .825. There is no systematic difference.
- Limits of agreement: SD(difference) = 38.77; LoA = -78.10 to 73.86 l/min (width 151.96). Individual readings of the two meters can differ by up to 76 l/min; the LoA confidence intervals are [-112.62, -43.58] for the lower limit and [39.34, 108.38] for the upper limit.
- Proportional bias: In the regression of the difference on the mean the slope is 0.029, t(15) = 0.33, p = .749; the difference does not depend on the measurement level.
- Additional finding: Pearson r = 0.943 (p < .001); but high correlation does not show agreement, the LoA width determines agreement.
- Limitation: With n = 17 the LoA confidence intervals are wide; whether a difference of about 76 l/min is clinically acceptable must be judged with subject-matter knowledge.
- APA: Bias = -2.12 l/min, t(16) = -0.23, p = .825; 95% limits of agreement [-78.10, 73.86].
Related concepts
Normality assumptionLimits of agreement (Bland-Altman)Proportional biasLin's concordance correlation coefficient (CCC)Noncentral t confidence intervalDistribution-free (order-statistic) confidence intervalVariance componentsFisher z transformation (correlation confidence interval)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.