MerQuR+Statistics Glossary

Statistics Glossary → Permutation p-value

Permutation p-valueconcept

A p value based on the distribution of a test statistic when group labels are shuffled at random, requiring no distributional assumption.

If the null hypothesis is 'no difference between groups', the group labels are exchangeable: which observation belongs to which group should not matter. The labels are shuffled many times, the test statistic is recomputed each time, and the position of the observed statistic in this permutation distribution gives the p value. With paired data the signs of the differences are flipped instead of the labels.

Enumerating every possible arrangement gives the exact p value; as this is usually infeasible, a Monte Carlo approximation with B random permutations is used, and p is typically computed as (number of results at least as extreme as observed + 1)/(B + 1), so it is never zero. A Monte Carlo p value varies slightly with the random seed, less so as B grows. The method does not require normality but does assume exchangeability; for example, it can mislead when testing a mean difference between groups with very different variances.

An everyday example

A teacher sees a 6-point difference between the mean exam scores of two classes. She puts the students' names in a bag, reassigns them at random to two classes 10,000 times and computes the difference each time. If only 150 of these shuffles give a difference of 6 points or more, the permutation p value is about 0.015.

In MerQur

In the Permutation Test, the 'Design' (two independent groups, k groups, paired) and 'B and Seed' settings; in the Mann-Whitney U and Wilcoxon tests, the 'Permutation p' option (None, Exact, Monte Carlo). In Moran's I and Getis-Ord Gi* Hotspot, the 'Permutation' option (0 analytic, 99, 999, 9999) gives a pseudo p value. The 'Monte Carlo p' checkbox of the Kruskal-Wallis and Friedman tests and the 'p computation' option of Cochran's Q test (Asymptotic χ², Monte Carlo, Exact (permutation)) produce p from the permutation distribution; GWR offers the 'Monte Carlo test of local coefficient variability and Number of permutations', and the Generalized Additive Model gives a permutation p value for the k-index.

Analyses where this concept appears

Permutation TestMann-Whitney UWilcoxon Signed-RankKruskal-WallisFriedmanCochran's Q TestGeneralized Additive Model (GAM, Nonlinear Regression)GWR (Geographically Weighted Regression)Moran's I (Spatial Autocorrelation)Getis-Ord Gi* Hotspot

Related concepts

Monte Carlo p-valueBootstrap (resampling)Random seed