MerQuR+Statistics Glossary

Statistics Glossary → Paired t-Test

Paired t-Testanalysis

A test of whether the mean difference between two measurements taken on the same people, or on matched units, truly differs from zero.

Concept

The most direct way to see the effect of an intervention is to measure the same people before and after. These two measurements are not independent: someone who starts high usually stays relatively high afterwards. The paired t-test exploits this dependence; it removes the large differences between people from the calculation and focuses only on each person's own change. That is why applying an independent-samples t-test to such data is wrong and often misses a real effect.

The test first computes a difference score for each unit (d = measurement 1 − measurement 2) and then asks, as in a one-sample t-test, whether the mean of these differences differs from zero: t = d̄ / (s_d/√n), with n − 1 degrees of freedom. The higher the correlation between the two measurements, the smaller the standard deviation of the differences and the more easily the test detects the same mean difference.

The assumption is that the difference scores, not the measurements themselves, are approximately normal. The units must be independent of one another. In a before-after design without a control group, the observed change cannot be attributed to the intervention alone; spontaneous improvement over time, seasonal effects or regression to the mean may also contribute. The result is reported with the mean difference, its confidence interval and an effect size.

When to use it

  • When the same people are measured twice (before and after an intervention, under two conditions) or units are matched in pairs (twins, left and right eye, matched case-control pairs).
  • If the two measurements come from different, independent people, use the independent-samples t-test; with three or more measurements on the same people, use repeated-measures ANOVA.
  • If the difference scores are clearly non-normal and the sample is small, the Wilcoxon signed-rank test is preferred.

An everyday example

A company puts 20 employees through an eight-week walking programme and measures their resting heart rate before and after. The mean heart rate falls from 78 to 74 beats per minute. Although resting rates vary widely between employees, from 60 to 95, each person's own change is much more consistent: the standard deviation of the differences is 6 beats. The paired t-test finds this 4-beat mean drop significant, t(19) ≈ −2.98, p ≈ 0.008; the 95% confidence interval for the difference points to a drop of roughly 1 to 7 beats.

Parameters in MerQur

Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)

Concept: The confidence level sets how 'sure' the interval for the mean difference is; subtracting it from 1 gives the significance threshold α used for the decision.

Everyday example: With 95% confidence the walking programme can be said to lower heart rate by 1 to 7 beats on average. Asking for 99% confidence widens the interval, perhaps bringing it very close to 0.

What it does: α = 1 - confidence level.

Effect on the result: The CI of the difference, the d CIs, the bootstrap CI, the one-sided CI, the decision and TOST ((1 - 2α) CI) change; t and p do not. The custom value applies only when "Other" is selected.

On the validation data: For pressure, CI of the difference 90% [-4.8549, 1.1882], 95% [-5.5365, 1.8698], 99% [-7.0589, 3.3922], custom 99.9% [-9.2986, 5.6319].

→ Confidence level and confidence interval

Hypothesis direction (Two-sided, Right-tailed, Left-tailed)

Concept: The direction of the hypothesis states whether the question concerns any change (two-sided) or only an increase or only a decrease (one-sided). The sign of the difference depends on which measurement is subtracted from which.

Everyday example: If the programme only counts as useful if it lowers heart rate, the question is one-sided. With the difference defined as 'after − before', a drop is negative and a left-sided test is chosen; defined as 'before − after', the same drop is positive and a right-sided test is chosen.

What it does: H1: μ_diff ≠ 0, > 0 or < 0 (difference = 1st measurement - 2nd measurement). The column order sets the sign of the difference.

Effect on the result: p becomes one-tailed; a one-sided CI row is added.

On the validation data: For pressure, two-sided p = 0.299163; right-tailed p = 0.850418, one-sided 95% CI [-4.8549, ∞]; left-tailed p = 0.149582, [-∞, 1.1882].

→ One-sided and two-sided tests (hypothesis direction)

Hedges g (checkbox)

Concept: A version of the difference-score effect size d_z that corrects its tendency to slightly overstate the effect in small samples.

Everyday example: If d_z = 0.67 with 20 employees, Hedges' g comes out at about 0.64. Had only 6 employees been measured, the correction would be larger.

What it does: The small-sample correction of d_z, g = d_z · J(n - 1) (exact gamma J).

Effect on the result: Adds a row to the effect size section.

On the validation data: For pressure, d_z = -0.3146, g = -0.2925. The small-sample correction J is computed with df = n - 1.

→ Hedges' g

d_av (checkbox)

Concept: Divides the mean difference by the average of the two measurements' own standard deviations rather than by the standard deviation of the differences. The effect is thus expressed in terms of the usual spread between people and can be compared with d values from studies with independent groups.

Everyday example: If employees' heart rates vary between people with a standard deviation of about 10 beats, the 4-beat drop gives d_av ≈ 0.4. This can be set beside the d from a study that examined the same question with two separate groups.

What it does: Divides the mean difference by the average SD of the two measurements; comparable with the d of independent groups (Cumming, 2012; Lakens, 2013, equation 10).

Effect on the result: Extra row only.

On the validation data: For pressure, -0.2854 = difference / ((6.9326 + 5.9161)/2); the denominator is the arithmetic mean of the two SDs. anorexia 0.4182.

→ Cohen's d for paired data (d_z, d_av, d_rm)

d_rm (checkbox)

Concept: An effect size recommended for repeated measures: it takes the correlation between the two measurements into account and converts the difference to the scale of between-person spread. It prevents d_z from inflating the effect when the correlation is high.

Everyday example: If before and after heart rates are strongly related (r ≈ 0.8), the differences are very consistent and d_z looks large at 0.67. Taking this relationship into account, d_rm expresses the effect at about 0.4, on the between-person scale.

What it does: Repeated-measures d that takes the correlation between the measurements into account (Lakens, 2013, equation 9).

Effect on the result: Extra row only.

On the validation data: For pressure, -0.2819; the denominator is the SD that accounts for the repeated-measures correlation, with no small-sample correction.

→ Cohen's d for paired data (d_z, d_av, d_rm)

Normal-approximation CI for d_z (checkbox)

Concept: d_z is also an estimate computed from the sample; this option gives a confidence interval for it based on a normal approximation. The approximation works well in large samples.

Everyday example: For d_z = 0.67 with 20 employees, the interval comes out at roughly [0.2, 1.1]; the effect may be medium or large. With 200 employees the interval would be much narrower.

What it does: Gives the large-sample interval d ± z·√(1/n + d²/(2n)).

Effect on the result: Extra row only.

On the validation data: For pressure, 95% [-0.8942, 0.2651] (equals the manual formula); the noncentral t interval is [-0.8884, 0.2727]. At small n, nct is preferred.

→ Asymptotic (normal) approximation

d_z CI, noncentral t (checkbox)

Concept: Computes the confidence interval of d_z exactly from the noncentral t distribution instead of an approximate formula; it is more accurate in small samples.

Everyday example: With only 10 employees, the normal-approximation interval may look narrower than it should; the noncentral-t method is more reliable here. As the sample grows the two intervals converge.

What it does: Gives the exact confidence interval for d_z.

Effect on the result: Extra row only; a two-sided interval is given even with a one-sided selection.

On the validation data: For pressure, 95% [-0.8884, 0.2727]; the interval is obtained by inverting the noncentral t distribution.

→ Cohen's d for paired data (d_z, d_av, d_rm)

Shapiro-Wilk and Kolmogorov-Smirnov (on the differences, checkboxes)

Concept: In the paired t-test the normality assumption applies to the difference scores, not to the two measurements themselves; these tests check whether the differences are bell-shaped.

Everyday example: Even if before and after heart rates are each skewed, the test is valid if people's changes are symmetrically distributed. But if a few employees' heart rates dropped dramatically, the distribution of differences is distorted and the normality test shows it.

What it does: Tests the normality of the differences (no Lilliefors correction for K-S).

Effect on the result: Extra rows only.

On the validation data: For pressure, Shapiro W = 0.9007, p = 0.1619. K-S D = 0.1932, p = 0.6938 (from the exact distribution with tied data).

→ Shapiro-Wilk test

Descriptives (checkbox)

Concept: Summarises the two measurements and the difference scores. When the standard deviation of the differences is smaller than those of the two measurements, it shows how much between-person spread the pairing removes.

Everyday example: The table shows a standard deviation of 10 beats for heart rate before and after, but only 6 for the difference. This says that employees differ a lot from one another, yet each changed in a similar way.

What it does: Gives n, mean, SD, SE, minimum, maximum, skewness and kurtosis for the two measurements and the difference.

Effect on the result: Extra table only.

On the validation data: For pressure, difference M = -1.8333, SD = 5.8284, minimum -9, maximum 8, skewness 0.573, kurtosis -0.552 (adjusted G1/G2).

Bootstrap CI (checkbox)

Concept: Finds the confidence interval of the mean difference not from a formula but by drawing thousands of new samples from the difference scores with replacement and computing each one's mean; it does not require normally distributed differences.

Everyday example: The 20 employees' heart-rate changes are written on cards in a bag; 20 cards are drawn with replacement and averaged. Over thousands of repetitions, the range covered by the middle 95% of those means gives the confidence interval.

What it does: Gives a percentile bootstrap CI for the mean difference.

Effect on the result: Extra row only.

On the validation data: For pressure, 95% [-4.8333, 1.4167] (t CI [-5.5365, 1.8698]); the limits can vary within the Monte Carlo error margin depending on resampling.

→ Bootstrap (resampling)

Equivalence test, TOST (checkbox; lower and upper bound)

Concept: Used to show that two measurements are practically 'equivalent', rather than merely 'not different'; the range of negligible differences is set in advance, and two one-sided tests check that the mean difference lies within it.

Everyday example: The company wonders whether a cheap wrist monitor gives the same result as an expensive chest strap and regards a difference of ±2 beats as negligible. The same employees are measured with both devices; if TOST shows that the mean difference lies within ±2 beats, the cheap device is judged equivalent.

What it does: Tests with two one-sided tests whether the difference lies within the [lower, upper] bounds.

Effect on the result: TOST rows only; the bounds have no effect when TOST is off, and if lower >= upper the analysis stops with a warning.

On the validation data: For pressure, ±0.5: p_TOST = 0.7776; ±6: p_TOST = 0.0154, 90% CI [-4.8549, 1.1882] within the bounds (equivalent); ±3: p_TOST = 0.2512.

→ Equivalence test (TOST)

Chart (Individual lines, difference histogram and normal curve, Q-Q plot of differences, Bland-Altman)

Concept: Individual lines show each person's change from before to after; the difference histogram and Q-Q plot show the distribution of the differences. The Bland-Altman plot draws the difference against the mean of the two measurements; it shows whether the difference changes with the level measured and displays the limits of agreement (mean difference ± 1.96 standard deviations).

Everyday example: The individual lines show that most employees' lines slope downwards while a few rise. The Bland-Altman plot may reveal that those with higher heart rates dropped more, showing that a single mean difference does not represent every employee equally.

What it does: Selects the plot in the Chart tab.

Effect on the result: The numeric results do not change.

On the validation data: The chart shows the means, the mean difference and the Bland-Altman limits Δ̄ ± 1.96·SD = [-13.2569, 9.5903] (Bland and Altman, 1986).

→ Histogram and bin count

Example on real data

Data: Hand et al. 1993, R MASS::anorexia (n = 72)

Variables: 1st measurement Postwt (weight after treatment), 2nd measurement Prewt (weight before treatment)

  • Question: Does the post-treatment weight (Postwt) of 72 patients with anorexia differ from their pre-treatment weight (Prewt)?
  • Assumption: For the difference scores Shapiro-Wilk W = 0.975, p = 0.154; normality of the differences is met.
  • Result: t(71) = 2.94, p = .004. The null hypothesis is rejected: the mean post-treatment weight (85.17) is higher than before (82.41).
  • Effect size: Cohen d_z = 0.35; a small-to-medium effect.
  • Descriptive: Mean gain 2.76 lb (SE = 0.941), 95% CI [0.89, 4.64]; the correlation between the pre and post measurements is r = 0.33 (p = .004), which limits the gain from pairing.
  • Limitation: The data combine three treatment groups (including the control); attributing the gain to treatment requires a group comparison.
  • APA: t(71) = 2.94, p = .004, d = 0.35.

Related concepts

Equivalence test (TOST)Hedges' gHistogram and bin countBootstrap (resampling)Asymptotic (normal) approximationCohen's d for paired data (d_z, d_av, d_rm)One-sided and two-sided tests (hypothesis direction)Shapiro-Wilk test

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.