MerQuR+Statistics Glossary

Statistics Glossary → Exact test and exact p-value

Exact test and exact p-valueconcept

A test that computes the p value from the exact null distribution of the test statistic rather than from an approximating distribution (normal, chi-square).

Many tests rely on the fact that, as the sample grows, the test statistic approaches a normal or chi-square distribution (the asymptotic approximation). With small samples, sparse tables or many ties this approximation can break down. An exact test instead enumerates every data arrangement that could occur if the null hypothesis were true (for example, every possible assignment of observations to groups, or every table with the same margins) and reports as the p value the total probability of arrangements at least as extreme as the observed one. Fisher's exact test, the binomial test and the exact Mann-Whitney U distribution are familiar examples.

Because they come from discrete distributions, exact p values are usually conservative: the actual Type I error rate can fall below α, and the mid-p value is used to reduce this conservatism. When the number of possible arrangements becomes very large the computation gets heavy, and a Monte Carlo p value based on randomly sampled arrangements is used instead. 'Exact' does not mean the test is assumption-free; it only means the p value involves no approximation.

An everyday example

Seven of eight people who try a café's new dessert prefer it to the old one. Under equal preference (p₀ = 0.5), the probability that 7 or 8 of the 8 pick the new dessert can be computed exactly: (8 + 1)/256 ≈ 0.035 (one-sided). With such a small sample this exact probability is used instead of trusting a normal approximation.

In MerQur

Appears as 'p computation' (Monte Carlo, Exact, Automatic) in the permutation test, as 'Computation method' in the Mann-Whitney U test and 'p method' in the Wilcoxon test (Auto, Exact, Asymptotic), with 'Permutation p' (Exact, Monte Carlo) in both, and as the 'Exact p' checkbox in the runs test. 'Exact / Monte Carlo p' in the chi-square independence and goodness-of-fit tests, the 'Exact (binomial)' method in McNemar, the 'Exact conditional test' in Cochran-Mantel-Haenszel, the exact (permutation) p in Cochran's Q, Fisher's exact test in the MR crosstabs, the Monte Carlo p in crosstabs and the exact p in the Mann-Kendall test can also be chosen; the binomial, sign and Fisher's exact tests rest directly on the exact distribution.

Analyses where this concept appears

Permutation TestMann-Whitney UWilcoxon Signed-RankBinomial TestSign TestRuns TestChi-Square IndependenceChi-Square Goodness of FitFisher's Exact TestMcNemar TestCochran-Mantel-Haenszel (CMH) TestCrosstabMR Frequency AnalysisMR × Categorical CrosstabMR × MR CrosstabCochran's Q TestCorrelation AnalysisConfusion Matrix MetricsMann-Kendall + Sen's SlopeAge Standardization and SMR

Related concepts

Asymptotic (normal) approximationMonte Carlo p-valueMid-p value