Statistics Glossary → Independent-Samples t-Test
Independent-Samples t-Testanalysis
A test of whether the means of two independent groups truly differ, by asking whether the observed difference can be explained by chance.
Concept
The means of two groups are almost never exactly equal in a sample; the real question is whether the observed difference comes from the groups truly differing or from the randomness of sampling. The t-test answers this by dividing the mean difference by its standard error: t = (mean₁ − mean₂) / SE. The larger the difference relative to its standard error, the larger t and the smaller the p value.
The test assumes that observations are independent (each person belongs to only one group), that the outcome is continuous and that it is approximately normal within each group. With adequate group sizes (roughly 30 or more per group) departures from normality have little effect. When the variances are unequal, the Welch correction is used instead of the classical (Student) t-test.
The result does not end with a 'difference / no difference' decision: the size of the difference (effect sizes such as Cohen's d) and its confidence interval show the practical importance of the finding.
When to use it
- When comparing two separate groups on one continuous measure (e.g. two treatment groups, two schools, women and men).
- If the same people were measured twice, use the paired t-test instead.
- With three or more groups use one-way ANOVA; with a strongly skewed outcome and a small sample, the Mann-Whitney U test is preferred.
An everyday example
Suppose two classes in a primary school were taught reading with different methods. At the end of term, class A averages 72 on a reading test and class B averages 68. Does this 4-point difference come from the methods really working differently, or did it arise because the two classes happened to contain different pupils? The independent-samples t-test takes the spread of scores in each class into account and gives the probability that a difference this large would arise by chance (the p value).
Parameters in MerQur
Variance (Auto (Levene), Equal (classic t), Welch (unequal))
Concept: Determines whether the two groups' spreads are treated as equal when computing the uncertainty of the mean difference. If equal, the variances are pooled (Student); if not, each is used separately (Welch).
Everyday example: Suppose class A's scores cluster between 65 and 80 while class B's spread from 40 to 95. When comparing two classes with such different spreads, the Welch option takes each class's own spread into account.
What it does: Computes the standard error of the mean difference with a pooled variance (Student, df = n1 + n2 - 2) or with separate variances (Welch-Satterthwaite df). In automatic selection Student is used if the mean-centred Levene p > α, otherwise Welch.
Effect on the result: t, df, p, the CI of the difference and the SE/df of TOST change; Cohen d, the descriptives and the assumption rows do not.
On the validation data: In hsb2 Levene F = 11.1333, p = 0.001013, so the automatic selection is Welch: t = -3.6564, df = 169.71, p = 0.000341. Equal variance: t = -3.7341, df = 198, p = 0.000246.
Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)
Concept: Sets how 'sure' the interval estimates in the results are and the threshold (α) for deciding that a difference exists.
Everyday example: The head teacher asks 'how far behind A is class B?'. With 95% confidence you answer 'between 2 and 7 points'; if they want 99% confidence the range widens to something like '1 to 8 points'.
What it does: α = 1 - confidence level; sets the width of all CIs, the H0 decision threshold and the threshold of the automatic Student/Welch decision. The custom value is active only when "Other" is selected.
Effect on the result: t and p do not change (unless the automatic decision changes); the CI of the difference, the d CIs, the bootstrap CI, the (1 - 2α) CI of TOST, the chart CI and the "met / VIOLATED" wording of the assumption rows change.
On the validation data: Welch CI of the difference 90% [-7.0727, -2.6672], 95% [-7.4992, -2.2407], 99% [-8.3397, -1.4002]. At 99.9%, α = 0.001 and Levene p = 0.001013 > 0.001, so the automatic decision switches to Student (t = -3.7341, df = 198).
Hypothesis direction (Two-sided, Right-tailed (>), Left-tailed (<))
Concept: States whether the research question asks 'is there a difference?' (two-sided) or is posed in one pre-specified direction ('is A higher than B?').
Everyday example: If the new reading method will only be adopted if it does better than the old one, the question is one-sided: 'is the mean of the class taught with the new method higher?'. If it is unknown whether the method helps or harms, a two-sided question is asked. The direction must be chosen in advance, without looking at the data.
What it does: H1: μ1 ≠ μ2, μ1 > μ2 or μ1 < μ2. The first group is the one seen first in the file; the text uses the group names ("H₁: μ(0) > μ(1)").
Effect on the result: p becomes one-tailed; a one-sided (1 - α) CI row is added; t and the two-sided CI do not change.
On the validation data: In hsb2, Welch right-tailed p = 0.99983, left-tailed p = 0.00017; for left, one-sided 95% CI (-∞, -2.6672].
Hedges g (checkbox)
Concept: A version of Cohen's d that corrects its tendency to slightly overstate the effect in small samples.
Everyday example: If each class has only 8 pupils, the computed effect size comes out somewhat inflated; Hedges' g trims this inflation. With 100 pupils per class, d and g are almost the same.
What it does: Corrects the small-sample bias of Cohen d with the J factor (Hedges and Olkin, 1985); J uses the exact gamma formula.
Effect on the result: A "Hedges g" row is added to the effect size section.
On the validation data: hsb2 g = -0.5282 (d = -0.5302). In small samples golf (7 + 7) -1.915491, rice 1.497732.
Glass Δ (checkbox)
Concept: An effect size that scales the difference by the spread of the control (comparison) group only, rather than by the two groups' pooled spread.
Everyday example: Let the class taught with the old method be the control group. If the new method also changed how spread out the scores are, it is more meaningful to express the difference as 'how many times the old class's usual variation'; Glass's Δ gives exactly that.
What it does: Divides the difference only by the SD of the second group (taken as the control).
Effect on the result: Adds a row to the effect size section.
On the validation data: hsb2 Δ = -0.5987 (denominator is the SD of the females, 8.1337); golf -1.8611.
CLES (checkbox)
Concept: The common-language effect size: the probability that a randomly chosen member of the first group scores higher than a randomly chosen member of the second. 50% means no difference.
Everyday example: You pick one pupil at random from class A and one from class B and compare their reading scores. If CLES = 0.65, the pupil from A comes out ahead in about 65% of such comparisons. It is the easiest way to explain the difference to someone who knows no statistics.
What it does: Common language effect size Φ(d/√2) = P(X1 > X2) (assuming normality and equal variances; McGraw and Wong, 1992).
Effect on the result: Extra row only.
On the validation data: hsb2 0.3539 (the probability that a randomly chosen male's writing score exceeds a randomly chosen female's); golf 0.0739.
CI for Cohen d, normal approximation (checkbox)
Concept: Cohen's d itself is an estimate computed from the sample; this option gives a confidence interval for d using a normal approximation.
Everyday example: Having found d = 0.5, you want to know how precise this 'medium' effect is. If the confidence interval is [0.2, 0.8], the true effect may be small or large; with classes of 30 the interval is wide, with schools of 300 it narrows.
What it does: Gives the interval d ± z·SE, SE = √((n1 + n2)/(n1n2) + d²/(2(n1 + n2))) (Hedges and Olkin, 1985).
Effect on the result: Extra row only.
On the validation data: hsb2 95% [-0.8133, -0.2471]. In small samples the noncentral t interval is preferred.
d CI, noncentral t (checkbox)
Concept: Computes the confidence interval of d exactly from the 'noncentral' form of the t distribution instead of an approximate formula; it is more accurate, especially in small samples.
Everyday example: With 10 pupils per class, the normal-approximation interval can look narrower than it should. The noncentral-t method is like weighing a bag on scales, while the approximate formula is like judging its weight by eye. As the sample grows the two converge.
What it does: Finds the CI of the ncp from the noncentral distribution of the Student t and scales it by √(1/n1 + 1/n2) (Steiger and Fouladi, 1997). It is computed from the Student t even if Welch is selected.
Effect on the result: Extra row only.
On the validation data: hsb2 95% [-0.8127, -0.2464]. Rice 99% upper limit 2.875525 (from the exact inversion of the noncentral t distribution).
Shapiro-Wilk (checkbox)
Concept: Tests whether the scores in each group are normally (bell-shaped) distributed. If p < α the distribution departs clearly from normal.
Everyday example: If the reading test was made too easy, most pupils score near the maximum and the distribution piles up at one end; Shapiro-Wilk detects this. With more than 30 pupils per class the t-test is hardly affected by this departure; with few pupils, the Mann-Whitney U test should be considered.
What it does: Runs a normality test in each group (3 <= n <= 5000).
Effect on the result: Extra rows only.
On the validation data: hsb2 W = 0.9499 (p = 0.0015) and 0.9459 (p = 0.0002), both VIOLATED.
Levene (checkbox)
Concept: Tests whether the two groups' spreads (variances) are equal and guides the choice between Student and Welch.
Everyday example: If everyone in class A scored around 70 while class B has pupils scoring both 40 and 95, the two classes differ in 'consistency'. Levene's test says whether this difference can be explained by chance.
What it does: Shows the mean-centred Levene and median-centred Brown-Forsythe rows; it only turns the display on or off, the automatic decision is always made with Levene.
Effect on the result: The decision does not change.
On the validation data: Levene W = 11.1333, p = 0.001; Brown-Forsythe W = 9.6197, p = 0.0022. When off, the automatic decision is still Welch.
Bartlett (checkbox)
Concept: Another test of equal variances; powerful when the data are normal but very sensitive to departures from normality.
Everyday example: If the scores form a neat bell curve, Bartlett's test detects a difference in spread better than Levene's. But if a few pupils' very low scores distort the distribution, Bartlett may raise a false alarm that 'the variances differ'; so it should be read together with Levene.
What it does: Chi-square test of equal variances, sensitive to the normality assumption.
Effect on the result: Extra row only.
On the validation data: hsb2 T = 5.4972, p = 0.019.
Variance ratio, folded F (checkbox)
Concept: The ratio of the larger variance to the smaller (F). A ratio near 1 means similar spreads; much larger than 1 means different spreads.
Everyday example: If class B's score variance is 1.6 times class A's, B's scores are noticeably more spread out. The folded F test gives the probability of the ratio being this large by chance.
What it does: F' = larger variance / smaller variance, two-sided p = 2·P(F > F').
Effect on the result: Extra row only.
On the validation data: hsb2 F(90, 108) = 1.6052, p = 0.0187; graze 1.23 / 0.6981.
Descriptives (checkbox)
Concept: Gives a summary of each group: count, mean, standard deviation, standard error, median, minimum and maximum, skewness and kurtosis.
Everyday example: It is like taking a snapshot of the two classes before the test: how many pupils, what is the average, what are the lowest and highest scores, are the scores piled up at one end? Looking at this table before interpreting the test reveals mistakes early.
What it does: Gives n, mean, SD, SE, median, minimum, maximum, skewness and kurtosis per group.
Effect on the result: Extra table only.
On the validation data: Male M = 50.1209, SD = 10.3052, SE = 1.0803; female M = 54.9908, SD = 8.1337, SE = 0.7791. Skewness and kurtosis are the adjusted G1/G2.
Bootstrap CI (checkbox)
Concept: Computes the confidence interval of the mean difference by drawing thousands of resamples from the data itself instead of a formula; it needs no distributional assumption.
Everyday example: You put the score cards of classes A and B in two bags, draw new 'classes' from each bag with replacement, and note the mean difference each time. The range the difference wanders over in 5000 draws gives the confidence interval.
What it does: Gives a stratified percentile bootstrap CI for the mean difference (B = 5000, fixed seed 12345).
Effect on the result: Extra row only.
On the validation data: hsb2 95% [-7.44, -2.3153] (slightly narrower than the t CI); the limits can vary within the Monte Carlo error margin depending on resampling.
Student + Welch rows (checkbox)
Concept: Shows the equal-variance (Student) and unequal-variance (Welch) results side by side in one table.
Everyday example: A reviewer may ask 'would the result change if equal variances were assumed?'. With the two rows side by side, it is clear at a glance whether both lead to the same decision.
What it does: Shows the equal-variance and Welch rows together in one table.
Effect on the result: Display only.
On the validation data: Student t = -3.7341, df = 198, SE = 1.3042; Welch t = -3.6564, df = 169.7, SE = 1.3319. The internal Welch df = 169.7074.
Equivalence test, TOST (checkbox; lower and upper bound)
Concept: Used to show that two groups are practically 'equivalent', rather than merely 'not different'; the range of differences regarded as negligible is set in advance.
Everyday example: A school wants to switch from an expensive printed book to a free digital one and finds a reading-score difference of at most 3 points acceptable. If TOST shows that the difference between the classes stays within ±3 points, the digital book is judged 'good enough'.
What it does: Schuirmann two one-sided tests: H01: Δ <= lower, H02: Δ >= upper; p_TOST = max(p1, p2); (1 - 2α) CI. SE and df come from the main test (the Student/Welch decision).
Effect on the result: TOST rows only; the bounds have no effect when TOST is off, and if lower >= upper the analysis stops with a warning.
On the validation data: hsb2 ±0.5: p_TOST = 0.999372 (no equivalence); ±5: 0.461165; [-8, -2]: 0.016294 (equivalent).
Chart (Box plot, Normal Q-Q, Mean ± CI)
Concept: Selects the visual summary of the results: the box plot shows the distribution and outliers, the Q-Q plot normality, and the mean ± CI plot the means and their uncertainty.
Everyday example: When presenting the two classes at a parents' meeting, the box plot shows 'class B's scores are more spread out and there are a few very low scores', while the mean ± CI plot conveys 'the gap between the class means is larger than their uncertainty' in one picture.
What it does: Changes only the Chart tab; "Mean ± CI" draws the t-based (1 - α) CI per group.
Effect on the result: The numeric results do not change.
On the validation data: 95% group CIs male [47.9747, 52.2670], female [53.4466, 56.5351].
→ Box plot
Example on real data
Data: UCLA OARC hsb2 (High School and Beyond, n = 200)
Variables: group female (0 male, 1 female), value write (writing score)
- Question: Does the mean writing score differ between male (female = 0, n = 91) and female (female = 1, n = 109) students?
- Assumptions: Levene F = 11.13, p = .001; the variances are not equal, so MerQur uses the Welch t-test. Shapiro-Wilk indicates a violation in both groups (p = .0015 and p = .0002), but the group sizes are large enough for the t-test to be robust.
- Result: Welch t(169.71) = -3.66, p < .001. The null hypothesis is rejected: the female mean (M = 54.99) is higher than the male mean (M = 50.12).
- Effect size: Cohen d = -0.53; a medium effect (the sign comes from the direction of the 0 - 1 difference).
- Descriptive: Mean difference -4.87 points (SE = 1.332), 95% CI [-7.50, -2.24]; male scores are more spread out (SD 10.31 and 8.13).
- APA: t(169.71) = -3.66, p < .001, d = -0.53.
Related concepts
Welch correctionLevene's testConfidence level and confidence intervalCohen's d (effect size)Bootstrap (resampling)Equivalence test (TOST)Common-language effect size (CLES)Hedges' gGlass's ΔFolded F test (variance ratio)Bartlett's test of equal variances
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.