MerQuR+Statistics Glossary

Statistics Glossary → Multiple comparisons and the familywise error rate (FWER)

Multiple comparisons and the familywise error rate (FWER)concept

The probability of wrongly rejecting at least one true null hypothesis within a family of tests; multiple-comparison corrections exist to control it.

When a single test is run at α = 0.05, the probability of a false positive is 5%. If m independent tests are run, however, the probability of at least one false positive, even when every null hypothesis is true, is 1 - (1 - α)^m; with 10 tests that is about 40%. The familywise error rate (FWER) is this overall risk. The Bonferroni correction multiplies each p value by m (or divides α by m); the Šidák correction uses 1 - (1 - p)^m; Holm's method orders the p values and adjusts them step by step, giving the same control with more power.

FWER control is conservative: it keeps false positives tightly in check, but as the number of tests grows so does the chance of missing real effects. In exploratory screens with many tests, the false discovery rate (FDR) is controlled instead; the Benjamini-Hochberg procedure bounds the expected proportion of false findings among the rejected hypotheses. Which tests count as a 'family' should be decided in advance from the research question. Unadjusted approaches such as LSD do not control the FWER.

An everyday example

Your chance of winning with a single lottery ticket may be tiny, but buy a hundred tickets and the chance that at least one wins rises noticeably. Statistics works the same way: if a student tests whether 20 different foods are related to exam success at the 5% level, they can expect about one 'significant' result even if no real relationship exists. Judging each test against a Bonferroni threshold of 0.05 / 20 = 0.0025 avoids this trap.

In MerQur

The Multiple Comparison Correction analysis is devoted to it. It is offered as 'EMM / simple-effect adjustment (Bonferroni, Šidák, LSD)' in Two-Way and Mixed-Design ANOVA, as 'Category-wise tests (None, Holm, Bonferroni)' in Chi-Square Goodness of Fit, and as 'p adjustment' in the MR crosstabs and Correlation Analysis. The post hoc tests of One-Way ANOVA control the familywise error rate; Repeated Measures ANOVA offers Bonferroni and Holm, Kruskal-Wallis Dunn (Bonferroni, Holm, Šidák ...), Kaplan-Meier 'Multiple comparison correction' and Getis-Ord Gi* 'α (alpha) and multiple-comparison correction'; Point Pattern Analysis warns that the pointwise envelope is not corrected for multiple comparisons.

Analyses where this concept appears

One-Way ANOVATwo-Way ANOVARepeated Measures ANOVAMixed-Design ANOVA, Split-PlotMultiple Comparison CorrectionKruskal-WallisChi-Square Goodness of FitMR × Categorical CrosstabMR × MR CrosstabCorrelation AnalysisKaplan-Meier Survival AnalysisMoran's I (Spatial Autocorrelation)Getis-Ord Gi* HotspotPoint Pattern Analysis (Ripley K/L, G/F/J, CSR)

Related concepts

Bonferroni correctionHolm (step-down Bonferroni) methodFalse discovery rate (FDR, Benjamini-Hochberg, Storey q)Post hoc tests (pairwise comparisons)