MerQuR+Statistics Glossary

Statistics Glossary → Friedman

Friedmananalysis

A test that needs no normality, for designs in which the same people (or blocks) are measured under three or more conditions: it checks whether the conditions differ systematically, based on how each person ranks the conditions within themselves.

Concept

Repeated measures ANOVA compares the means of the same people under several conditions and assumes normality and sphericity. When the data are ordinal (e.g. ratings from 1 to 10), the distributions are skewed or the sample is small, these assumptions weaken. The Friedman test uses each person's own ranking instead of the values themselves. Differences in level between people (one person rating everything high, another everything low) therefore do not affect the result; only the order of the conditions within each person matters.

Each person's (block's) values under the k conditions are ranked among themselves from 1 to k; tied values receive the average rank. Each condition's rank sum or mean rank across all people is then computed. If the conditions do not differ, every condition's mean rank should be around (k + 1)/2. The Friedman statistic summarises departures from this value and, in large samples, approximately follows a chi-square distribution with k − 1 degrees of freedom. Kendall's W turns the same information into an agreement coefficient between 0 and 1: it shows how similarly people rank the conditions.

The people (blocks) are assumed to be independent of one another. Because the test uses only ranks, it says little about the size of the differences between conditions, and when the data really are normal it is less powerful than repeated measures ANOVA. A significant result says that at least one condition differs from the others; post-hoc tests show which conditions differ, and the Page test is used when an order among the conditions was predicted in advance.

When to use it

  • When the same people or blocks are measured under three or more conditions and the data are ordinal, the distributions are not normal, or the sample is small.
  • With only two conditions, use the Wilcoxon signed-rank test; if the conditions were studied with different, independent people, use the Kruskal-Wallis test.
  • If the data are near-normal and the sample adequate, repeated measures ANOVA is more powerful; if the conditions are expected to rise or fall in a particular order, the Page trend test is more suitable.

An everyday example

A coffee shop tries three roast levels (light, medium, dark) of the same bean and asks a 12-person tasting panel to rate each one's bitterness from 1 to 10. Some panellists give every coffee high scores, others low scores; what matters is how each panellist ranks the three coffees within their own ratings. The Friedman test ranks each panellist's scores from 1 to 3 and compares the roasts' mean ranks. If the mean ranks are 1.3 for light, 2.0 for medium and 2.7 for dark with p = 0.003, the roasts differ in bitterness; a Kendall's W of 0.5 says that the panellists show moderate-to-strong agreement in their rankings.

Parameters in MerQur

Post-hoc test (None; pairwise Wilcoxon + Bonferroni, Holm, BH, Šidák; Nemenyi; Conover (Holm); Dunn-Bonferroni)

Concept: When the Friedman test is significant, post-hoc tests show which pairs of conditions differ while controlling the multiple-comparison risk. One route applies a Wilcoxon signed-rank test to each pair and adjusts the p values; the other uses tests based on Friedman's own within-person ranks, such as Nemenyi, Conover or Dunn. Conover is usually the most powerful, Nemenyi and Dunn more cautious.

Everyday example: Three roast levels form three pairs. Nemenyi's test might conclude that 'the dark roast is clearly more bitter than the light one, while the medium roast does not differ significantly from either'.

What it does: If Friedman is significant, tests which condition pairs differ. There are two families: (1) the pairwise Wilcoxon signed-rank test with p adjustment; (2) Nemenyi, Conover (Holm), and Dunn-Bonferroni based on the within-block ranks.

Effect on the result: Adds only the post-hoc section; χ², p, and W do not change. The rank-based tests are consistent with Friedman (Conover is the most powerful, Nemenyi and Dunn are more conservative; Pohlert, 2014).

On the validation data: Round Out - Wide Angle: pairwise Wilcoxon raw p = 0.0179, Bonferroni 0.0537 (not significant); Nemenyi p = 0.0044; Conover Holm p = 0.0021; Dunn Bonferroni p = 0.0046. Narrow Angle - Wide Angle: raw p 0.0016, Nemenyi 0.0614 (not significant), Conover 0.0246.

→ Post hoc tests (pairwise comparisons)

Descriptives (checkbox; default off)

Concept: Gives each condition's summary: count, mean, standard deviation, standard error, median, minimum and maximum, skewness and kurtosis. Only people with a measurement under every condition are used, so the summary matches the data the test uses.

Everyday example: The table shows a median bitterness score of 8 for the dark roast and 4 for the light one. If a panellist left without tasting the medium roast, their scores enter no condition's summary.

What it does: n, mean, SD, SE, median, minimum, maximum, skewness, kurtosis per condition (on listwise-deleted data).

Effect on the result: The test does not change.

On the validation data: hsb2 read: M = 52.2300, SD = 10.2529, median 50, skewness 0.196, kurtosis -0.623.

Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)

Concept: The confidence level sets the significance threshold used to reject the null hypothesis (α = 1 − confidence level) and how 'sure' the interval for Kendall's W is. It does not change the Friedman statistic or the p value.

Everyday example: p = 0.003 is significant at both the 95% and the 99% level. The interval for Kendall's W, however, comes out wider when 99% confidence is requested than at 95%.

What it does: α = 1 - level. Determines the H0 decision, the post-hoc marks, and the level of the Kendall W bootstrap CI.

Effect on the result: The custom value is active only when "Other" is selected. The star in the decision row follows fixed .05 thresholds (at 90% "H₀ rejected ns" can appear).

On the validation data: Hypnosis (p = 0.0917): "not rejected" at 95% and 99%, "rejected" at 90% and custom 80%. RoundingTimes Kendall W CI: 90% [0.0706, 0.5743], 95% [0.0444, 0.6223], 99% [0.0195, 0.7162].

→ Confidence level and confidence interval

Pairwise detail: W, Z, r (checkbox)

Concept: The detail of the pairwise Wilcoxon comparisons gives, for each pair of conditions, the test statistic, the standard normal value Z, the raw and adjusted p values, and the effect size r = |Z|/√n, where n is the number of people (blocks).

Everyday example: For dark versus light roast, Z might be 2.9, giving r = 2.9/√12 ≈ 0.84 for 12 panellists, a very large effect. The r between medium and light might be around 0.4.

What it does: Pairwise Wilcoxon table: W (smaller rank sum), Z (normal approximation without continuity correction), raw p, adjusted p, r = |Z|/√n (Rosenthal, 1991; denominator = number of blocks).

Effect on the result: Extra table only.

On the validation data: RoundingTimes |Z| 0.779 / 2.368 / 3.153, W 68 / 54 / 30.5, r 0.166 / 0.505 / 0.672; raw p 0.4357 / 0.0179 / 0.0016 (asymptotic). hsb2 read - write: W = 7944.0, Z = -0.903, p = 0.3663.

→ r effect size from Z/√N

Kendall's W CI (checkbox)

Concept: Kendall's W is also an estimate computed from the sample. Its confidence interval is found by resampling the people (blocks) with replacement and recomputing W each time. With few people the interval is very wide.

Everyday example: With 12 panellists and W = 0.5, the interval might be as wide as [0.2, 0.8]; agreement may be moderate or strong. With a panel of 50 the interval would be clearly narrower.

What it does: Percentile bootstrap CI with resampling of blocks (B = 2000, seed 12345), level = 1 - α.

Effect on the result: Extra row only. With small n the interval is very wide.

On the validation data: hsb2 95% [0.0002, 0.0242]; Hypnosis 95% [0.1062, 0.6437].

→ Kendall's W (coefficient of concordance)

Monte Carlo p (checkbox)

Concept: The chi-square approximation to the Friedman statistic can weaken with few people or many ties. The Monte Carlo p value rests on the idea that under the null hypothesis each person's ranks are arbitrary across the conditions: each person's ranks are shuffled among themselves and the statistic is recomputed thousands of times.

Everyday example: If a panellist gave two coffees the same score, a tie arises. Thousands of fake tastings are produced by shuffling each panellist's three ranks among themselves; the share of results as extreme as the observed one gives a p that does not rely on the approximation.

What it does: Produces p from the permutation of ranks within blocks (B = 10000, seed 12345), p = (b + 1)/(B + 1), 99% CI.

Effect on the result: Extra p row only.

On the validation data: RoundingTimes 0.0025 (asymptotic 0.0038); Hypnosis 0.0967 (asymptotic 0.0917); hsb2 0.7340.

→ Monte Carlo p-value

Page (checkbox)

Concept: The Page test checks whether the conditions rise steadily in a pre-specified order (e.g. as dose, duration or intensity increases). When such an order is predicted, it is more powerful than the Friedman test, which asks only 'is there any difference?'. The order must be fixed before looking at the data.

Everyday example: The coffee shop's expectation is clear: the darker the roast, the more bitter the coffee. The Page test tests the order light < medium < dark and detects this steady increase more easily than the Friedman test.

What it does: Page (1963) ordered alternatives test: L = Σ j·Rj; the order of the checked columns is the predicted increase; one-sided normal approximation (no continuity correction). For a decreasing trend, the columns must be selected in reverse order.

Effect on the result: Extra row only.

On the validation data: hsb2 L = 2410, p = 0.3085; RoundingTimes L = 243, p = 0.9992 (the data decrease, the predicted direction is an increase); Hypnosis L = 180, p = 0.9928.

→ Page's test (ordered alternatives)

Example on real data

Data: RoundingTimes, Hollander and Wolfe 1973, R ?friedman.test

Variables: measurement columns Round Out, Narrow Angle, Wide Angle (times of 22 players for three base-running methods)

  • Question: Does the time of 22 baseball players to run to first base differ among three running paths (Round Out, Narrow Angle, Wide Angle)? Because the same players are measured under three conditions, the Friedman test, the rank-based counterpart of repeated measures ANOVA, is used.
  • Descriptive: Mean ranks are Round Out 2.41, Narrow Angle 2.14, Wide Angle 1.45; the shortest times are on the Wide Angle path (median 5.45 s).
  • Result: χ²(2) = 11.14, p = 0.0038. The null hypothesis is rejected: at least one running path differs from the others.
  • Effect size: Kendall W = 0.25; there is moderate agreement among players in the ranking of the conditions.
  • Next step: To see between which paths the difference lies, pairwise comparisons (Wilcoxon or Nemenyi, adjusted) should be performed.
  • APA: χ²(2, N = 22) = 11.14, p = .004, Kendall W = .25.

Related concepts

Monte Carlo p-valuePost hoc tests (pairwise comparisons)Asymptotic (normal) approximationKendall's W (coefficient of concordance)Page's test (ordered alternatives)r effect size from Z/√NContinuity correction (Yates)Ties and zero differences (rank tests)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.