MerQuR+Statistics Glossary

Statistics Glossary → Multiple Comparison Correction

Multiple Comparison Correctionanalysis

Methods that adjust p values or the decision threshold when many hypothesis tests are run in the same study, to stop results that are 'significant' by chance from piling up.

Concept

Choosing α = 0.05 for a single test keeps the chance of wrongly calling a result 'significant' when there is no real effect at 5%. But if 15 independent tests are run in the same study and none involves a real effect, the chance that at least one comes out significant by chance rises to about 54%. The more tests, the bigger the problem: genetics, questionnaire items, subgroup analyses and all pairwise group comparisons are typical examples. Multiple comparison corrections control this accumulation.

There are two different targets. The familywise error rate (FWER) is the probability of at least one false positive in the family of tests; Bonferroni (multiplying each p by the number of tests), Šidák, Holm, Hochberg and Hommel control it. Step-wise methods such as Holm sort the p values from smallest to largest and relax the threshold step by step, giving the same guarantee as Bonferroni with more power. The false discovery rate (FDR) is the expected proportion of false results among those declared 'significant'; Benjamini-Hochberg and its variants control it. FDR is a more lenient criterion and detects more real effects when there are many tests.

Which method to choose depends on the purpose of the study: if even one false positive would have serious consequences (e.g. approving a drug), use FWER; if many candidates are being screened (e.g. deciding which genes to study further), FDR is appropriate. Some methods assume the tests are independent or positively related; with complex dependence between tests, more cautious methods such as Benjamini-Yekutieli are used. The family of tests to be corrected should be defined before looking at the data.

When to use it

  • When several hypothesis tests are run in the same study (many outcomes, many subgroups, all pairwise group comparisons) and the risk of some findings being significant by chance should be controlled.
  • For a few confirmatory tests, Holm or Hochberg is preferred; for many exploratory tests, Benjamini-Hochberg (FDR). With complex dependence between tests, Benjamini-Yekutieli is safer.
  • If the comparisons are between pairs of groups after an ANOVA, ANOVA-specific post-hoc tests such as Tukey's HSD, Dunnett or Games-Howell also solve this problem directly.

An everyday example

A school psychologist compares test-anxiety scores across six classes and tests every pair of classes, 15 pairs in all, with separate t-tests. Without correction, 4 pairs come out significant at p < 0.05. But even if the classes did not differ at all, 0.75 false alarms would be expected on average among 15 tests. After a Holm correction only 2 pairs remain significant; the differences in these two pairs are much harder to explain by chance. With Benjamini-Hochberg, 3 pairs might remain significant; the procedure then keeps the expected share of false findings among them at no more than about 5%.

Parameters in MerQur

p list

Concept: The correction is made jointly over all the raw p values belonging to the same 'family of tests'; the adjusted version of a p value depends on the other p values in the family and on the number of tests. That is why which tests enter the family must be decided before looking at the results.

Everyday example: The psychologist enters the p values of all 15 class pairs as one list. Had only the 4 'interesting-looking' p values been entered, the correction would be far too mild and the false-alarm risk hidden.

What it does: One test per row; the "label: p" format and values separated by commas or semicolons are accepted, a row starting with "#" is a note, an invalid row is skipped. Values outside 0 <= p <= 1 and a single p value stop the analysis with a warning.

Effect on the result: Determines the set of p values to be adjusted.

On the validation data: Different notations of the BH 15 p values give the same result. For the four lists (15 + 24 + 121 + 50 p), the Bonferroni, Šidák, Holm, Holm-Šidák, Hochberg, Hommel, BH and BY adjusted p values are computed in every run (8 methods x 2175 rows).

→ p-value

α (0.001-0.999)

Concept: α is the accepted upper limit for the familywise error rate or the false discovery rate. A test counts as significant if its adjusted p value is at most α. For FWER methods, α = 0.05 means 'the chance of at least one false alarm across the whole family is at most 5%'; for FDR methods it means 'at most 5% of the findings declared significant are false'.

Everyday example: If the psychologist applies Holm with α = 0.05, the risk of at least one false alarm across all 15 comparisons together does not exceed 5%. Choosing α = 0.10 makes more pairs significant, but doubles that risk too.

What it does: Sets the decision threshold (adjusted p <= α). In two-stage FDR (TSBH, BKY) it also affects the adjusted p; the raw significance count uses p < α.

Effect on the result: The numbers of significant tests change; in most methods the adjusted p does not.

On the validation data: BH 15 p, number significant for Bonferroni / Holm / Hochberg / Hommel / BH / BY: α = 0.05: 3 / 3 / 3 / 3 / 4 / 3; α = 0.01: 2 / 2 / 2 / 2 / 3 / 2; α = 0.10: 3 / 3 / 3 / 4 / 9 / 3.

→ Significance level (α)

Input mode (List, From data)

Concept: The raw p values can be taken ready-made from another analysis, or generated directly from the raw data by pairwise tests between groups. The second route ensures that the family of tests is built completely and consistently.

Everyday example: The psychologist can copy and paste p values from tests run earlier. Or they can select the class and anxiety-score columns; the p values of the 15 pairs are generated from the data and no pair is forgotten.

What it does: In "From data" mode, pairwise test p values are generated from the group and measure columns, written into the p box and adjusted with the same methods.

Effect on the result: Back in list mode the data selections have no effect. If p generation fails (e.g. a Reference level not in the data), an error message is shown and old p values are not reported.

Group and measure column (from-data mode)

Concept: When p values are generated from the data, the group column defines the categories to compare and the measurement column the numeric outcome being compared. An observation with a missing measurement is left out of the tests.

Everyday example: The group column is the class name (6-A, 6-B, …) and the measurement column the test-anxiety score. Pupils who left the questionnaire blank are not counted.

What it does: Selects the group (categorical or text) and measure column. Missing measurements are dropped by row; the level order is the order of appearance (airquality: May, Jun, Jul, Aug, Sep).

Effect on the result: Sets the groups and the measure from which the raw p values are computed.

Comparison (each group vs reference, all pairs) and Reference level

Concept: Which comparisons are made sets the size of the family of tests: for k groups, comparing each against a single reference group means k − 1 tests, while all pairs means k(k − 1)/2 tests. The smaller the family, the milder the correction and the greater the power to detect real differences.

Everyday example: All pairs of six classes make 15 tests. If the psychologist's real question is 'does class 6-A, which ran an anxiety-prevention programme, differ from the others?', only 5 comparisons against 6-A are made and the correction is much milder.

What it does: Compares each level with the Reference level (k - 1 tests) or all pairs (k(k - 1)/2 tests). If the Reference level is blank, a level named "control / kontrol / placebo" or the first level is used.

Effect on the result: The number of tests and the adjustment family change; in pairs mode the Reference level has no effect.

On the validation data: airquality (Welch): 4 tests for Reference level May, 10 tests for pairs. The Holm-adjusted values are computed from the separate-variance Welch p values (e.g. May-Jul 0.00026, Jun-Jul 0.01527).

→ Post hoc tests (pairwise comparisons)

Pairwise test (Welch t, Student t, Pooled t, Mann-Whitney U)

Concept: Whatever pairwise test the raw p values are computed with, the correction inherits that test's assumptions. Welch's t does not assume equal variances; Student's t pools the two groups' variances; the pooled t uses the common variance of all groups; the Mann-Whitney U is rank-based and does not require normality.

Everyday example: If anxiety scores are very spread out in some classes and tightly clustered in others, Welch's t gives more reliable raw p values. If scores in small classes are clearly skewed, the Mann-Whitney U may be chosen.

What it does: Selects the test that produces the raw p values. The pooled t uses the SD of all groups (df = N - k).

Effect on the result: The raw and adjusted p values change.

On the validation data: airquality pairs: with the pooled t, the Holm-adjusted p values are May-Jul 0.000264, Jun-Aug 0.0499. Mann-Whitney PlantGrowth trt1-trt2: p = 0.01133 (continuity-corrected normal approximation).

→ Pooled variance (pooled standard deviation)

Additional methods: Holm-Šidák, two-stage BH (TSBH), Benjamini-Krieger-Yekutieli (BKY), Storey q-value

Concept: The additional methods are more powerful versions of the basic ones, or versions with different assumptions. Holm-Šidák is a slightly more powerful form of Holm using Šidák thresholds. Two-stage FDR methods (TSBH, BKY) first estimate the share of tests with no real effect and then use it for a more powerful correction. Storey's q-value is, for each test, 'the smallest false discovery rate at which this result would be called significant'.

Everyday example: If most of the 15 pairs involve a real difference, the two-stage methods notice this and may declare slightly more pairs significant than BH. If a pair has a q-value of 0.03, then when it and all pairs with smaller p values are called significant, the expected false discovery rate is about 3%.

What it does: Holm-Šidák (step-down Šidák, FWER); two-stage BH (TSBH); BKY (2006) two-stage FDR; Storey q-value (λ = 0.5). Added to the table as extra columns.

Effect on the result: Extra columns only.

On the validation data: The Holm-Šidák adjusted p values follow the Holm (1979) definition. The BKY decisions follow BKY (2006) Definition 6 (BH 15 p, α = 0.05: 8 rejections). Storey q: π0 = 0.667 for the BH 15 p (λ = 0.5).

→ Holm (step-down Bonferroni) method

Example on real data

Data: Benjamini and Hochberg 1995 p list; airquality, R datasets

Variables: the 15 p values of the Benjamini and Hochberg (1995) example are pasted into the p box

  • Question: Which of the 15 tests in the Benjamini and Hochberg (1995) example remain significant once the multiple testing problem is taken into account?
  • Raw result: Without adjustment 9 of the 15 tests appear significant at p < 0.05; with this many tests the type I error accumulates.
  • Familywise error (FWER): With Bonferroni, Šidák, Holm, Hochberg and Hommel only the first 3 tests remain significant (e.g. Bonferroni p03 = 0.0285, Holm p03 = 0.0247).
  • False discovery rate (FDR): With BH 4 tests are significant (BH p = 0.0356 for p04); the more conservative BY method for dependent tests keeps 3.
  • Interpretation: FDR methods are more powerful than familywise methods; BH is recommended for exploratory studies with many tests, Holm or Hochberg for a few confirmatory tests.
  • APA: Of the 15 comparisons, 9 were significant without adjustment; 4 remained significant with the Benjamini-Hochberg correction (FDR = .05) and 3 with the Holm correction.

Related concepts

Šidák correctionPooled variance (pooled standard deviation)Post hoc tests (pairwise comparisons)Holm (step-down Bonferroni) methodp-valueFalse discovery rate (FDR, Benjamini-Hochberg, Storey q)Reference category (comparison level)Multiple comparisons and the familywise error rate (FWER)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.