MerQuR+Statistics Glossary

Statistics Glossary → Permutation Test

Permutation Testanalysis

A test that needs no distributional assumptions: it builds from the data itself the distribution of 'what would we see if there were really no difference?' by randomly reassigning group labels (or the signs of paired differences), and computes the p value from that distribution.

Concept

Classical tests compute the p value by assuming that the test statistic follows a particular theoretical distribution (t, F) under the null hypothesis, which in turn rests on assumptions such as normality. The permutation test generates this distribution from the data instead. Its logic is simple: if there is truly no difference between the groups, the label saying which group an observation belongs to is arbitrary, and shuffling the labels should produce similar differences.

First the test statistic (e.g. the difference between two group means) is computed on the observed data. Then the group labels are randomly reassigned among the observations and the statistic is recomputed each time; in a paired design, the sign of each difference is flipped at random instead. This is repeated for every possible arrangement (an exact test) or for thousands of random arrangements (Monte Carlo). The p value is the proportion of shuffled datasets that give a statistic as extreme as, or more extreme than, the observed one.

The test does not require normality, but it does have an assumption: under the null hypothesis the observations must be exchangeable between groups. With the mean difference as the statistic, the test actually checks whether the two distributions are the same; it can come out significant even if the groups differ only in spread, not in mean. When the question is only about the mean difference and the variances may differ, a standardised statistic such as Welch's t is safer. Observations must be independent; the test gives its firmest interpretation in experiments where the labels were genuinely assigned at random.

When to use it

  • When comparing two or more groups with a small sample and doubtful normality, as a distribution-free alternative to the t-test and ANOVA.
  • When two measurements on the same units are compared, use the sign-flipping (paired permutation) test; if a classical rank-based distribution-free option is wanted, the Mann-Whitney U or Wilcoxon signed-rank test can be used.
  • If the aim is not to test a difference but to find a confidence interval for a statistic, the bootstrap confidence interval is preferred.

An everyday example

A small bookshop tries two window displays: display A for 8 days and display B for 8 days. Mean daily sales are 1200 TL under A and 1450 TL under B, a difference of 250 TL. With so few days, nobody can be sure the distribution is normal. The permutation test takes the 16 days' sales figures, randomly reassigns the 'A' and 'B' labels to the days, and computes the mean difference each time. There are 12,870 ways to split 16 days into two sets of 8; if only 2% of them give a difference of 250 TL or more in favour of B, the two-sided p is about 0.04 and the difference is hard to explain by chance alone.

Parameters in MerQur

Dependent variable and Grouping variable

Concept: The dependent variable is the numeric outcome being compared; the group variable is the column whose labels will be shuffled. The sign of the difference depends on the order in which the groups are taken: first group minus second group.

Everyday example: Daily sales is the dependent variable and window display (A/B) is the group variable. If the difference is defined as 'A − B', B's advantage appears as −250 TL; the sign shows only the direction, not the size.

What it does: Selects the numeric outcome and the group column. Levels are sorted alphabetically; difference = first level - second level. The two-group design needs exactly 2 levels; otherwise it stops with a warning.

Effect on the result: Determines the whole output.

On the validation data: hsb2 write: female = 0 n = 91, M = 50.1209; female = 1 n = 109, M = 54.9908; observed difference -4.8699.

Design (Two independent groups, k independent groups (permutation F), Paired (sign-flip))

Concept: The design determines what is shuffled. With two independent groups, and also with three or more groups, the group labels are shuffled, using the one-way ANOVA F statistic in the latter case. In a paired design each unit's two measurements belong together, so instead of labels, the signs of the differences are flipped at random.

Everyday example: If the bookshop tries three displays, the k-group design is used. If old and new displays are compared in the same 10 branches, the paired design is chosen: the sign of each branch's 'new − old' difference is flipped at random, like tossing a coin.

What it does: Two groups: label shuffling; k groups: the permutation distribution of the one-way ANOVA F (Monte Carlo only, upper tail); paired: the sign-flip test of the differences (T = mean(d)).

Effect on the result: The definition of the statistic and p changes. With two-level data the k-group design is equivalent to the two-sided mean difference test.

On the validation data: alpha k-group F = 2.6130, permutation p = 0.0754; hsb2 write ~ prog F = 21.27, p = 0.0001 (= 1/(B + 1)); depression paired exact two-sided p = 0.02734.

→ Permutation p-value

2nd measurement (paired design only)

Concept: In a paired design each unit has two measurements, and the difference is defined as the first measurement minus the second. The order of the measurements sets the sign of the difference and hence the direction of a one-sided hypothesis.

Everyday example: If the first measurement is sales under the new display and the second under the old one, a positive difference shows that the new display sells more. Choosing the reverse order expresses the same result as a negative difference.

What it does: Defines the difference as d = first measurement - second measurement. If the same column is selected, it stops with a role warning.

Effect on the result: Sets the sign of the difference in the paired design; no effect in the two-group design.

Test statistic (Mean difference, Median difference, Welch t; two groups only)

Concept: A permutation test can be built on any statistic; the chosen statistic determines what the test is sensitive to. The mean difference is the most common; the median difference is more resistant to outliers; Welch's t is a standardised statistic that divides the difference by its uncertainty and largely keeps its validity for the mean difference even when the group variances differ.

Everyday example: If a school made a bulk book purchase on one of the B days, that day's sales inflate the mean; the median difference is barely affected by this single day. If sales on A days are very consistent while those on B days fluctuate a lot, Welch's t is the more reliable choice.

What it does: Selects the statistic for the permutation distribution; Welch t is a studentized statistic (Janssen, 1997).

Effect on the result: The observed value and p change. Ignored in the paired and k-group designs.

On the validation data: diffusion (exact, all 3003 assignments counted): mean difference p = 0.131868; median difference (-0.515) p = 0.082584; Welch t (-2.0414) p = 0.079254.

Hypothesis direction (Two-sided, Right-tailed, Left-tailed)

Concept: The direction of the hypothesis determines which values in the shuffled data count as 'as extreme as observed': in a two-sided test large differences in both directions count, in a one-sided test only differences in the pre-specified direction.

Everyday example: If the bookshop only asks 'does display B sell more?', only differences of 250 TL or more in favour of B are counted. If it asks 'is there any difference between the displays?', differences of 250 TL or more in favour of A are counted too, and p roughly doubles.

What it does: Two-sided |T| >= |t|, right-tailed T >= t, left-tailed T <= t; the direction is first group - second group. Ignored in k groups.

Effect on the result: p and the H₁ text change; the classic t comparison row also shows the p in the selected direction.

On the validation data: diffusion exact right-tailed p = 0.938062, left-tailed p = 0.063603; neuropathy left-tailed MC p = 0.0990.

→ One-sided and two-sided tests (hypothesis direction)

p computation (Monte Carlo, Exact, Automatic)

Concept: An exact permutation test tries every possible arrangement of the labels and gives an exact p value, but the number of arrangements explodes as the sample grows. The Monte Carlo approach picks thousands of them at random; the p value is then an estimate, and its precision can be shown with a confidence interval.

Everyday example: There are 12,870 ways to split 16 days into two sets of 8; a computer easily tries them all and gives the exact p. Had the bookshop worked with 60 days of data, the number of possible splits would be astronomical, and a Monte Carlo estimate with 10,000 random arrangements would be used.

What it does: Monte Carlo: p = (b + 1)/(B + 1) (Phipson and Smyth, 2010) with a 99% Clopper-Pearson CI. Exact: all C(n, n1) assignments (<= 500000; paired 2^n, n <= 20), p = b/N. Automatic: exact when enumerable; beyond the limit, or in k groups, it switches to Monte Carlo with a warning.

Effect on the result: The type and precision of p change. With the exact method only the number of enumerated assignments is reported.

On the validation data: diffusion MC p = 0.134287, exact 0.131868; mouse exact (11440 assignments) 0.278147. hsb2 write ~ female MC p = 0.0005.

→ Exact test and exact p-value

B and Seed

Concept: B is the number of random arrangements tried in the Monte Carlo method; the larger it is, the more precise the p estimate. The seed is the starting value of the random number generator; the same seed produces the same arrangements and hence the same result.

Everyday example: With 1,000 permutations p might come out at 0.038 on one run and 0.046 on another; with 10,000 permutations this fluctuation shrinks a lot. If the seed is stated in the report, someone else obtains exactly the same p.

What it does: Sets the number of Monte Carlo permutations and the seed; the bootstrap CI comes from the same random stream with B_boot = min(B, 5000).

Effect on the result: The MC p changes by the MC margin.

On the validation data: diffusion: B = 200 p = 0.1294, 2000 p = 0.1264, 10000 p = 0.1343; seed 0 p = 0.1340, 12345 p = 0.1285 (exact 0.1319; all within the MC margin).

→ Permutation p-value

Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)

Concept: The confidence level sets the significance threshold (α) used for the decision and how 'sure' the bootstrap confidence interval for the mean difference is. It does not change the p value itself.

Everyday example: p = 0.04 is significant at the 95% level and not at the 99% level. The bootstrap interval may say that display B raises daily sales by 'between 20 and 480 TL' on average, showing the size of the difference alongside the decision.

What it does: Sets the decision α (1 - level) and the level of the percentile bootstrap CI of the mean difference.

Effect on the result: The CI and the decision row change; p does not.

On the validation data: hsb2 write 95% bootstrap CI [-7.4005, -2.2211]; diffusion 90% [-0.5951, -0.0869], 95% [-0.644, -0.040], 99% [-0.7351, 0.0490]; resampling is done within groups (stratified).

→ Confidence level and confidence interval

Effect size (not a form parameter; reported in all runs)

Concept: An effect size expresses the size of the difference in standard-deviation units and shows practical importance independently of the p value. Hedges' g corrects Cohen's d's tendency to slightly overstate the effect in small samples; in the paired design, d_z uses the standard deviation of the differences.

Everyday example: If the standard deviation of sales is 230 TL, the 250 TL difference amounts to about d = 1.1, a large effect. With small samples of 8 days each, Hedges' g pulls this down slightly, to about 1.0.

What it does: Two groups: Cohen d (pooled SD) and Hedges g = d·J(n1 + n2 - 2); paired: d_z = mean(d)/SD(d) and g = d_z·J(n - 1); J = Γ(m/2)/(√(m/2)·Γ((m - 1)/2)) (Hedges, 1981).

Effect on the result: Written as an extra row in every run; it does not change p.

On the validation data: hsb2 d = -0.5302, g = -0.5282. In the paired design depression g = 0.9133, sleep g = -1.1739.

→ Effect size

Example on real data

Data: UCLA OARC hsb2 (High School and Beyond, n = 200)

Variables: dependent write (writing score), group female (0 male, 1 female)

  • Question: Does the mean writing score differ between male (female = 0, n = 91) and female (female = 1, n = 109) students? It is tested without a distributional assumption by shuffling the group labels.
  • Result: Observed difference -4.87 (50.12 - 54.99); with 10000 permutations the two-sided p = 0.0005 (p = (b + 1)/(B + 1)). The null hypothesis is rejected.
  • Effect size: Cohen d = -0.53, Hedges g = -0.53; a medium effect.
  • Additional finding: The bootstrap 95% CI of the difference [-7.40, -2.22] does not contain 0; the classic t-test gives the same decision (t = -3.73, p = .0002).
  • Interpretation: The permutation test does not require normality or equal variances; the agreement of the parametric and permutation results here shows that the decision is robust.
  • APA: Permutation test, B = 10000, difference = -4.87, p < .001, d = -0.53.

Related concepts

Permutation p-valueEffect sizeMonte Carlo p-valueNormality assumptionOne-sided and two-sided tests (hypothesis direction)Distribution-free (order-statistic) confidence intervalExact test and exact p-valueCohen's d for paired data (d_z, d_av, d_rm)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.