Statistics Glossary → Cochran-Mantel-Haenszel (CMH) Test
Cochran-Mantel-Haenszel (CMH) Testanalysis
A test of the association between two binary variables that combines the 2x2 tables at each level of a third, confounding variable, keeping that variable's influence out of the comparison.
Concept
The association between two variables can be distorted by a third variable linked to both (a confounder). If a treatment was mostly given at a hospital with milder cases, for instance, the treatment looks good although recovery is really driven by mildness. When all the data are pooled into one table, even the direction of the association can reverse (Simpson's paradox). The CMH test deals with this by treating each level of the confounder (each stratum) as a separate 2x2 table.
The association is assessed within each stratum and the information is combined across strata: the test compares the observed count in one cell of each stratum with the count expected under independence in that stratum and adds up the differences. Comparisons are thus made only between units in the same stratum. The result is summarised in a single value that applies across strata: the Mantel-Haenszel common odds ratio (OR_MH). OR_MH = 1 means there is no association once the confounder is taken into account.
A common odds ratio only makes sense if the strength of the association is similar across strata; this is checked with the Breslow-Day test of homogeneity. If the odds ratios differ clearly between strata (effect modification), stratum-specific results should be reported instead of a single common value. The test controls only the confounder it includes; it offers no protection against unmeasured confounders.
When to use it
- To examine the association between two binary variables stratified by a confounder (centre, age group, sex, village); to combine centres in a multicentre study.
- With no confounder, or when one table suffices, use the chi-square test of independence or Fisher's exact test; when counts within strata are very small, the exact conditional test is preferred.
- To control several confounders or continuous ones at the same time, logistic regression is more suitable; to study the mutual associations of three or more categorical variables, a log-linear model.
An everyday example
A farming cooperative tests a new seed treatment in three villages: in each village some fields are sown with treated seed and some with untreated seed, and whether the seedlings catch a fungal disease is recorded. The villages differ in climate and soil, so disease rates vary from village to village, and treated seed may have been used mostly in the wettest village. Pooling all fields into one table would mix up the treatment's effect with the village differences. The CMH test compares treated and untreated fields only within the same village and then combines the results; if the common odds ratio comes out at, say, 0.45, the treatment roughly halves the odds of disease once village differences are set aside.
Parameters in MerQur
Row (binary) and Column (binary)
Concept: The two binary variables forming the row (usually exposure or treatment) and column (outcome) of the 2x2 table in each stratum. The test statistic is unaffected by swapping them; which group the odds ratio, relative risk and risk difference compare against which depends on this choice and on the order of the levels.
Everyday example: In the cooperative example the rows show whether the seed was treated and the columns whether the seedlings fell ill. The odds ratio then answers 'the odds of disease in treated fields are how many times those in untreated fields?'
What it does: The row and column of the 2x2 table in each stratum. Levels are sorted in string order; the OR is computed as ad/bc relative to the (1st row, 1st column) cell.
Effect on the result: CMH χ², Breslow-Day, Cochran, and the exact p are unaffected by swapping; if row and column are swapped together, OR_MH stays the same. RR and RD change because they are defined as "1st column risk, 1st row / 2nd row". If row/column are not binary and the generalized CMH is not selected, an explicit error is given.
On the validation data: Migraine CMH = 8.3052, OR_MH = 3.3132 [1.4456, 7.5934]. In the default layout (row Treatment [Active, Placebo], column Response [Better, Same]) RR_MH = 2.1636 [1.2336, 3.7948] and RD_MH = 0.2738 (SE 0.0904) [0.0965, 0.4510]. When row and column are swapped, OR_MH stays the same, RR_MH = 1.7193 [1.1972, 2.4689] and RD_MH = 0.2917 (SE 0.0954). In UCBAdmissions OR_MH = 1.1053 (levels are sorted alphabetically, in the reverse direction 1/1.1053 = 0.9047; the note "levels in alphabetical order" is written).
Stratum
Concept: The confounding variable whose influence is to be kept out; each of its levels forms a separate 2x2 table (stratum). Because comparisons are made only within a stratum, this variable's links with the two main variables cannot distort the result.
Everyday example: In the cooperative example the stratum is the village. Since treated and untreated fields are compared only within the same village's climate and soil, the effect of 'there is a lot of disease in the wet village anyway' does not mix with the treatment's effect.
What it does: The confounding variable; each level forms a separate 2x2 table. The main CMH uses all strata, the extra rows the strata with n >= 2.
Effect on the result: Changing roles leads to an entirely different question. Strata with a zero margin do not affect the main CMH; for Breslow-Day and Tarone the uninformative strata are removed.
On the validation data: With stratum = Response and column = Gender, CMH = 0.0236, p = 0.878. Rabbits has two strata with a zero margin; the main CMH is 5.6571, Breslow-Day 8.6273 (df 2, p 0.0134) and Tarone 8.3583 (p 0.0153).
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets the decision threshold (α = 1 − confidence level) and the width of the confidence intervals for the common odds ratio, common risk ratio and risk difference, and for the stratum-specific odds ratios. The CMH statistic and p value do not change.
Everyday example: With a common odds ratio of 0.45, the 95% confidence interval might run from about 0.28 to 0.72; since it excludes 1, the treatment's effect is reliable. At 99% confidence the interval widens.
What it does: alpha = 1 - confidence level. The OR_MH interval (Robins-Breslow-Greenland), stratum OR (Woolf), RR_MH (Greenland-Robins), and RD_MH (Sato) intervals, the exact interval, and the decision are computed at this level.
Effect on the result: The interval widths and the decision for borderline p change; the CMH statistic and p do not. The generalized CMH has no interval.
On the validation data: Migraine OR_MH 3.3132: 80% [1.9263, 5.6985], 90% [1.6518, 6.6455], 95% [1.4456, 7.5934], 99% [1.1140, 9.8541], 99.9% [0.8232, 13.3339]; exact interval 95% [1.3534, 8.4551]; at 99.9% the decision is "not rejected" (p = 0.0040 > 0.001).
Extra outputs: Common RR / RD
Concept: The relative risk (how many times one group's risk is the other's) and risk difference (the gap between the two risks), combined across strata. They are more intuitive than the odds ratio but only meaningful in cohort or experimental designs; in case-control studies risk cannot be estimated directly and the odds ratio is interpreted.
Everyday example: Because the cooperative followed the fields forward in time, risk can be computed directly. A common risk ratio of 0.55 means that, with village differences set aside, the risk of disease in treated fields is about half that in untreated ones; the risk difference expresses this as, say, '15 fewer diseased fields per 100'.
What it does: The Mantel-Haenszel common risk ratio (Greenland-Robins, 1985 variance) and common risk difference (Sato, 1989 variance); risk = proportion of the 1st column level, 1st row compared with 2nd row. Meaningful in cohort studies and clinical trials; in case-control studies the OR is interpreted.
Effect on the result: Extra rows only. Undefined in r x c x k tables, and the note "common RR/RD and the exact test are defined only for 2x2xK" is written.
On the validation data: Migraine RR_MH 2.1636 [1.2336, 3.7948], RD_MH 0.2738, SE 0.0904, CI [0.0965, 0.4510] (Mantel-Haenszel weighted common risk difference). Promotion data 1.2327 / 0.1887 (also computed when OR_MH is infinite).
Extra outputs: Generalized CMH
Concept: Extends the CMH idea when the row or column variable has more than two categories. It answers three different questions: is there a linear trend between two ordered variables, do the row groups differ in the mean of an ordered outcome, and is there any association at all.
Everyday example: Had the cooperative used three groups, untreated, low dose and high dose, and graded disease as 'none, mild, severe', a 2x2 table would not suffice. The generalised CMH test can ask 'does disease get milder as the dose rises?' while still keeping village differences out.
What it does: For r x c x k tables, the nonzero correlation (df 1), row mean scores (df r-1), and general association (df (r-1)(c-1)) statistics. The table scores are the numeric level values; if not numeric, 1..k is used.
Effect on the result: In 2x2xK all three statistics equal the uncorrected MH χ². Ordinal categories should be given with numeric codes.
On the validation data: Agresti Table 7.8 (Satisfaction): correlation 6.6235 (p 0.0101), row mean 9.2259 (p 0.0264), general association 10.2001 (df 9, p 0.3345). With string levels, code point order and scores 1..k are used.
Extra outputs: Exact conditional test
Concept: When counts in the strata are small, computes the p value, and an exact confidence interval for the common odds ratio, from the exact probabilities of all possible tables with the stratum totals fixed, without relying on the chi-square approximation.
Everyday example: If the cooperative tested only a few fields per village, some strata might have 0 or 1 diseased fields. The exact conditional test is then more trustworthy than the approximate CMH test.
What it does: Uses the distribution of S = Σa_k conditional on the stratum margins; the two-sided p is the sum of values with probability less than or equal to the observed one. Gives the conditional MLE common OR and the exact interval (alpha/2 in each tail). Used with small stratum frequencies.
Effect on the result: Extra rows only; the main CMH does not change.
On the validation data: Rabbits p = 0.03994, MLE 10.3610, CI [1.0774, 531.5128]. Migraine p = 0.005059 (sum of the conditional probabilities of the S values whose probability does not exceed that of the observed S). Promotion data: p = 0.05625, MLE infinite, CI [0.9910, infinite].
H₀ common OR value
Concept: Tests whether the common odds ratio differs from a value chosen by the researcher rather than from 1, the 'no association' value. It is used to compare with a previous study's finding or with an acceptable effect threshold.
Everyday example: If the old seed treatment is known to cut the odds of disease to 0.70, the cooperative can ask 'is the new treatment different from the old one?'. The H₀ value is then set to 0.70, and the test asks whether the common odds ratio of 0.45 lies significantly far from it.
What it does: When a value other than 1 is entered, z = (ln OR_MH - ln ψ0)/SE and the two-sided p are written; if the exact test is selected, the exact p is computed under ψ0.
Effect on the result: Extra row only; no row is written when ψ0 = 1. When OR_MH is infinite no raw "inf/nan" appears in the output.
On the validation data: Migraine: for ψ0 = 2, z = 1.1928, p = 0.2329; for ψ0 = 0.5, p < 0.001; for ψ0 = 100, z = -8.052; for ψ0 = 3.313, p = 0.9999.
Example on real data
Data: SAS/STAT 15.1 Example 44.7 (Migraine)
Variables: row Treatment, column Response, stratum Gender
- Question: Controlling for gender, is migraine treatment (active / placebo) associated with response (better / same)?
- Assumption: The common odds ratio must be homogeneous across strata: Breslow-Day χ²(1) = 1.4929, p = 0.2218; the ORs of the female (OR = 5.82) and male (OR = 2.04) strata are taken as homogeneous, so the common OR can be interpreted.
- Result: CMH χ²(1) = 8.31, p = .004. H0 (common OR = 1) is rejected: adjusting for gender, treatment and response are associated.
- Effect size: OR_MH = 3.31, 95% CI [1.45, 7.59]; the odds of a "better" response with active treatment are about 3.3 times those with placebo; the interval does not include 1.
- Additional finding: Cochran χ² = 8.47 (p = .004) and the continuity-corrected MH χ² = 7.20 (p = .007) give the same decision.
- APA: CMH χ²(1) = 8.305, p = .004, OR_MH = 3.313 (95% CI [1.446, 7.593]).
Related concepts
Null hypothesis (H₀) and test valueMantel-Haenszel common odds ratioStratified analysis (stratum variable)Breslow-Day test (homogeneity of odds ratios)Exact test and exact p-valueOdds ratio (OR)Confounding variableRelative risk (RR) and risk difference (RD)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.