MerQuR+Statistics Glossary

Statistics Glossary → Post hoc tests (pairwise comparisons)

Post hoc tests (pairwise comparisons)concept

Comparisons that, once an omnibus test (such as ANOVA) has found a difference among groups, identify which pairs differ while controlling the multiple-testing error.

The ANOVA F test only says that at least one group differs from the others. To see which pairs differ, all pairs are compared; but with k groups there are k(k − 1)/2 tests, so the probability of at least one false alarm (the familywise error rate) rises quickly. Post hoc methods try to keep that rate at the chosen α level.

The methods differ in their assumptions and aims: Tukey's HSD suits all pairwise comparisons with equal variances, Games-Howell and Tamhane's T2 unequal variances, Dunnett comparing every group with a single control, and Scheffé any comparison, including complex non-pairwise contrasts. Bonferroni is simple but conservative, while LSD does not control the familywise error at all. For a few comparisons fixed in advance on theoretical grounds, planned contrasts are more powerful than post hoc tests. After nonparametric tests, rank-based methods such as Dunn, Conover-Iman and Nemenyi are used.

An everyday example

The effect of four fertilizers on tomato yield is compared and the ANOVA is significant. If each of the six pairwise comparisons were tested separately at 5%, the chance of at least one false 'difference' would be about 26% even if no difference existed. Tukey's HSD keeps that chance at 5% and shows that only fertilizer C gives a higher yield than A and B.

In MerQur

The 'Post-hoc test' list in One-Way ANOVA (Tukey HSD, Games-Howell, Scheffé, Dunnett, Tamhane T2 and others); the corresponding options in Two-Way ANOVA, ANCOVA, MANOVA and Repeated Measures ANOVA; the rank-based post hoc tests of the Kruskal-Wallis and Friedman tests; pairwise McNemar tests in Cochran's Q; and pairwise group comparisons in Kaplan-Meier.

Analyses where this concept appears

One-Way ANOVATwo-Way ANOVARepeated Measures ANOVAMixed-Design ANOVA, Split-PlotMANOVAANCOVAMultiple Comparison CorrectionKruskal-WallisFriedmanCrosstabMR × Categorical CrosstabMR × MR CrosstabCochran's Q TestLinear Mixed Models (LMM)Nested Mixed Model (Nested LMM, Multi-Level Hierarchical Mixed Model)Crossed Mixed Model (Crossed LMM, Crossed Random Effects Mixed Model)Generalized Estimating Equations (GEE)Generalized Linear Mixed Models (GLMM)Bayesian ANOVAKaplan-Meier Survival Analysis

Related concepts

Tukey's HSD testMultiple comparisons and the familywise error rate (FWER)Bonferroni correction