Statistics Glossary → Sign Test
Sign Testanalysis
A test that looks only at the direction (plus or minus) of the differences between two measurements on the same people, checking whether increases and decreases are equally frequent, that is, whether the median difference is zero.
Concept
Sometimes the size of the difference between two measurements cannot be trusted, only its direction. A patient may say 'I feel better than before' without being able to put a number on how much; or the scale may make comparing the sizes of differences meaningless. The sign test is the paired test with the fewest assumptions for such situations: for each pair it looks only at whether the difference is positive or negative.
If the median difference really is zero, a difference is equally likely to be positive or negative, that is, 50%. The number of plus signs among the n non-zero differences then follows a binomial distribution with a success probability of 0.5. The test computes exactly from this distribution the probability that a split of signs as unbalanced as the observed one would arise by chance. Pairs with a difference of exactly zero carry no information about direction and are dropped from the test.
The test assumes only that the pairs are independent of one another; the differences need not be normal or symmetric. The price of this flexibility is power: because it ignores the size of the differences, it is less powerful than the Wilcoxon signed-rank test when the sizes can be trusted, and less powerful than the paired t-test when the data are normal. The result is reported with the proportion of plus signs and the confidence interval of the median difference.
When to use it
- When two measurements on the same people, or matched pairs, are compared and only the direction of the differences can be trusted; for example with ordinal scales, 'better / worse' judgements, or data containing extreme differences.
- If the sizes of the differences are meaningful and their distribution roughly symmetric, the Wilcoxon signed-rank test is more powerful; if the differences are near-normal, the paired t-test is.
- The same logic applies when asking whether the median of a single sample differs from a given value; if the two measurements are binary (yes/no), use the McNemar test.
An everyday example
A care home gives 20 residents a new mattress and asks them about their sleep quality one month before and one month after, on a five-step scale from 'very poor' to 'very good'. Since the steps cannot be taken as equally spaced, only the direction of change is used: sleep improved for 13 residents, worsened for 4 and did not change for 3. Dropping the unchanged leaves 17 residents. If the mattress had no effect, improving and worsening would be as equally likely as heads and tails; the chance of 13 or more of the same side in 17 tosses, in either direction, is about 5%. With a two-sided p ≈ 0.049, the result is borderline significant.
Parameters in MerQur
Column 1 and Column 2
Concept: The sign test treats the two measurements of the same unit as a pair and subtracts the second measurement from the first for each pair. The order of the columns sets the sign of the differences; the two-sided result is unaffected by it, but the direction of a one-sided hypothesis and the sign of the median difference are. A pair with a missing measurement cannot be used.
Everyday example: If the first column is the sleep score 'after the mattress' and the second 'before', a plus sign means improvement. With the reverse order, the same 13 improvements appear as minus signs; the two-sided p is still 0.049.
What it does: Selects the two paired measurements; difference d = Column 1 - Column 2 (a pair with a missing value is dropped). If the same column is selected twice, the analysis stops before starting.
Effect on the result: When the order changes, the positive and negative counts swap and the median difference and CI change sign; the two-sided p stays the same.
On the validation data: hsb2 write - read: 97 / 88, median 0 (zeros included); read - write: 88 / 97; p = 0.556535 in both.
Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)
Concept: The confidence level sets how 'sure' the interval for the median difference is and the threshold (α) for the significance decision. The interval is built from two values chosen among the sorted differences; because the binomial distribution is discrete, the achieved confidence level does not equal the requested one exactly and usually ends up slightly above it.
Everyday example: With 20 residents and 95% confidence requested, the interval is built from two of the sorted differences and the achieved level might be, say, 95.9%. That is why both the requested and the achieved level are reported.
What it does: The requested level of the distribution-free CI for the median difference and the decision threshold.
Effect on the result: The CI limits and the achieved coverage change; p does not change. Because the CI is built from order statistics, the achieved level differs from the requested one and is written separately.
On the validation data: shoes: the same order [-0.8, 0.1] at 90% and 95% (achieved 97.9%); 99% [-1.1, 0.2] (99.8%). hsb2 95% [-1, 2] (coverage 96.0%); anorexia median 1.65, 95% [-0.1, 3.7] (95.56%).
Hypothesis direction (Two-sided, Right-tailed, Left-tailed)
Concept: The direction of the hypothesis states whether the question concerns any change (two-sided) or a change in one direction only (one-sided). A one-sided test detects a predominance in that direction more easily; the direction must be chosen before looking at the data.
Everyday example: If the care home asks only 'does the new mattress improve sleep?', the test is one-sided and p is about 0.025 for the 13-to-4 result. If the possibility that the mattress worsens sleep also matters, a two-sided test is chosen and p is 0.049.
What it does: "Right-tailed" is the alternative in which Column 1 > Column 2 differences dominate; the exact binomial p becomes one-sided.
Effect on the result: p and the decision change. The CI becomes one-sided ("[-∞, upper]" or "[lower, ∞]", "[one-sided]"); the H₁ row is written with a direction.
On the validation data: hsb2: two-sided 0.556535, right-tailed 0.278267, left-tailed 0.768848. shoes left-tailed 0.054688.
Effect size (checkbox)
Concept: The effect size used is the proportion of plus signs among the non-zero differences (p+); 0.5 means no effect at all. Cohen's g is how far this proportion departs from 0.5 (p+ − 0.5); roughly, 0.05 counts as a small, 0.15 a medium and 0.25 a large effect.
Everyday example: Since 13 of the 17 residents improved, p+ = 13/17 ≈ 0.76 and Cohen's g ≈ 0.26, a large effect. Although the p value is borderline because the sample is small, the direction of change is fairly consistent.
What it does: p+ = n+/(n+ + n-) and Cohen g = p+ - 0.5 (Cohen, 1988, chapter 5).
Effect on the result: Extra row only.
On the validation data: hsb2 p+ = 0.5243, g = +0.0243; shoes 0.2 and -0.3; anorexia 0.5915 and +0.0915 (equals the manual formula).
Large-sample Z (checkbox)
Concept: As the number of non-zero differences grows, the binomial distribution of the number of plus signs approaches the normal distribution, and the test can also be expressed as a Z value. When approximating the discrete distribution with a continuous one, a half-unit continuity correction is applied. This is extra information only; the decision is always made with the exact binomial p.
Everyday example: For 17 residents, Z = (13 − 8.5 − 0.5)/(√17/2) ≈ 1.94, giving a two-sided p ≈ 0.052, while the exact p is 0.049. With a small sample the two differ slightly, and in a borderline decision this difference matters; that is why the exact p is used.
What it does: The continuity-corrected normal approximation Z = (n+ - n/2 ∓ 0.5)/(√n/2) and p in the selected direction. The decision is always made with the exact p.
Effect on the result: Extra row only.
On the validation data: hsb2 Z = 0.5882, p = 0.5564; the sign of Z depends on the direction of the difference (Column 1 - Column 2). shoes Z = -1.5811, p = 0.1138.
Fixed output: median difference and CI
Concept: The median difference is the middle value when all differences are sorted and gives the typical size of the change. Its confidence interval is built without distributional assumptions, from two of the sorted differences chosen according to the binomial distribution. While the test leaves out zero differences, the median difference and its interval are computed from all pairs, zeros included.
Everyday example: If the middle value of the 20 residents' sorted sleep-score differences is +1 step, the typical change is an improvement of one step. If the confidence interval of the median difference is [0, 1], the change may be nothing or one step, a picture consistent with the borderline p value.
What it does: The median difference and CI are computed from all differences including zeros; the test uses the nonzero differences.
Effect on the result: Written in every run; among the options only the confidence level and the direction affect the CI.
On the validation data: hsb2 median 0; read - write [-2, 1]; anorexia median 1.65.
Example on real data
Data: UCLA OARC hsb2 (High School and Beyond, n = 200)
Variables: column 1 write (writing score), column 2 read (reading score); difference = write - read
- Question: Is the median of the paired difference between writing and reading scores zero? The sign test uses only the direction (plus/minus) of the differences.
- Descriptive: Of the 200 pairs, 97 have write > read and 88 have write < read; 15 equal pairs are dropped from the test (effective n = 185). Median difference 0.
- Result: Exact binomial p = 0.557. The null hypothesis cannot be rejected: positive and negative differences occur about equally often.
- Confidence interval: CI for the median difference [-1, 2] (because order statistics are discrete, the achieved level is 96%); the interval includes zero.
- Additional finding: In the same data the Wilcoxon signed-rank test is also not significant (p = 0.366); the sign test is less powerful because it does not use the magnitude of the differences.
- APA: Sign test, 97 positive and 88 negative differences, p = .557.
Related concepts
Effect sizeSignificance level (α)Asymptotic (normal) approximationDistribution-free (order-statistic) confidence intervalOne-sided and two-sided tests (hypothesis direction)Exact test and exact p-valueContinuity correction (Yates)Ties and zero differences (rank tests)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.