Statistics Glossary → ANCOVA
ANCOVAanalysis
Analysis of covariance: comparing group means on an outcome while statistically holding constant the effect of one or more continuous variables (covariates) that also influence the outcome.
Concept
When groups are compared, the outcome is often affected by more than group membership itself. For instance, when three classes are taught with different methods, end-of-term scores also depend on what pupils knew at the start of term. Analysis of covariance (ANCOVA) adds this continuous variable (the covariate) to the model and compares the groups 'for people with the same covariate value'. This has two benefits: part of the initial differences between groups is removed, and because unexplained variability in the outcome shrinks, the test gains power.
The model combines ANOVA and regression: outcome = group effect + slope × covariate + error. Each group's mean is moved to the value expected at the overall mean of the covariate; these are called adjusted (estimated marginal) means. The F test for the group effect tests the differences between these adjusted means. The more strongly the covariate is related to the outcome, the more the error variance shrinks.
Besides the ANOVA assumptions there are two important ones: the relationship between covariate and outcome is linear, and its slope is the same in every group (homogeneity of slopes). If the slopes differ, the group difference changes with the value of the covariate and a single adjusted difference misleads; a separate-slopes model and the Johnson-Neyman method are then used. The covariate should be measured before group assignment and not be affected by the group. In studies without random assignment, ANCOVA does not remove all initial differences; it adjusts only for the covariate that was measured.
When to use it
- When groups are compared on an outcome and a continuous variable that strongly affects that outcome (e.g. a pre-test score, age, starting weight) should be taken into account.
- In pre-test/post-test designs, to compare post-test scores adjusted for the pre-test; if the same question is to be seen together with the effect of time, mixed-design ANOVA can also be used.
- If the slope of the covariate-outcome relationship differs between groups, use a separate-slopes model (Johnson-Neyman); with several outcome variables, MANCOVA is preferred.
An everyday example
A school teaches three maths classes with different methods: traditional lecturing, learning through games and a digital app. The end-of-term exam means are 62, 70 and 74. But pupils in the digital class had also scored higher than the others on the pre-test at the start of term. ANCOVA takes the pre-test score as a covariate and compares the classes 'for pupils who started at the same level'. If the adjusted means come out at 64, 70 and 71, part of the digital method's raw advantage came from its initial head start; the advantage of games and of the digital method over traditional lecturing, however, survives the adjustment.
Parameters in MerQur
Sum of squares (SS) type (Type III, Type II, Type I)
Concept: The sum-of-squares type determines whether the group effect is adjusted for the covariate. Type I tests effects in the order they enter the model; if the group enters first, it measures the raw difference not adjusted for the covariate. In Types II and III the group effect is adjusted for the covariate, which is the real question of ANCOVA.
Everyday example: Under Type I with the group entered first, the difference between methods is measured on the raw means of 62, 70 and 74 and the pre-test difference is ignored. Type III tests the difference for pupils with the same pre-test score, that is, on the adjusted means.
What it does: Sets whether the factor is adjusted for the covariate (and the second factor). In Type I the factor enters first (not adjusted for the covariate); in Type II and III it is adjusted for the covariate.
Effect on the result: The SS, F, p, η²ₚ, ω²/ε² of the factor (and interaction) row and the decision change. The covariate row and the adjusted means do not.
On the validation data: hsb2 prog + read: Type III prog SS 650.259965 (F 5.867, p 0.0034); Type I prog SS 3175.697857 (F 28.65). DrugTest Drug: Type III F(2, 26) = 2.136, p = 0.1384; Type I F = 9.149, p = 0.0010.
Post-hoc test (None, Bonferroni, LSD, Šidák, Tukey-Kramer)
Concept: When the group effect is significant, post-hoc tests check which pairs of groups differ using the covariate-adjusted means rather than the raw ones, while controlling the multiple-comparison risk. Tukey-Kramer is the adaptation of Tukey's method for unequal group sizes.
Everyday example: If the adjusted means are 64, 70 and 71, a Bonferroni comparison may show that games and the digital app do not differ from each other, but that both are higher than traditional lecturing.
What it does: Compares the adjusted (EMM) mean differences; the SE of the difference comes from the full covariance matrix (√(L V L')), df is the error df.
Effect on the result: Only p and the simultaneous CI width change; Δ, SE and t stay the same.
On the validation data: DrugTest A - F (Δ = -3.446, SE = 1.8868, t(26) = -1.826): LSD p = 0.0793, Bonferroni 0.2379, Šidák 0.2195, Tukey-Kramer 0.1809.
Effect sizes: ω², ε² (checkboxes)
Concept: An effect size shows how much of the variability in the outcome is explained by the group difference. ω² and ε² estimate the population value with less bias than η², which tends to overstate this share in a sample.
Everyday example: If ω² = 0.05, then after controlling for the pre-test, about 5% of the variation in exam scores is related to the teaching method. This is a small-to-medium effect; the school can weigh the cost of changing methods accordingly.
What it does: ω² = (SS_A - df_A·MSE)/(SST + MSE), ε² = (SS_A - df_A·MSE)/SST; SST is the corrected total SS, 0 if negative.
Effect on the result: Extra rows only.
On the validation data: DrugTest Type III ω² = 0.0279, ε² = 0.0283 (η²ₚ = 0.1411); Type I ω² = 0.2004, ε² = 0.2029.
Assumption checks: Homogeneity of slopes (checkbox)
Concept: Homogeneity of slopes means that the relationship between covariate and outcome has the same slope in every group. If the group × covariate interaction is significant, the assumption is violated: the difference between groups changes with the value of the covariate and a single adjusted difference loses its meaning.
Everyday example: If, under traditional lecturing, pupils with high pre-test scores do much better in the exam while under game-based learning the pre-test matters little, the slopes differ. This may mean that game-based learning helps especially those who started weak; a single 'average' difference hides this.
What it does: Runs a nested model F test by adding factor x covariate terms to the y ~ factor + covariate model.
Effect on the result: Extra test row only.
On the validation data: hsb2 prog x read F = 0.012, p = 0.9878; DrugTest F(2, 24) = 0.593, p = 0.5606 (not significant; the slopes are taken as homogeneous).
Assumption checks: Levene (checkbox)
Concept: ANCOVA assumes that the error variance remaining after the covariate is accounted for is equal across groups; Levene's test checks this using the model residuals.
Everyday example: If, after controlling for the pre-test, pupils' scores in the digital class scatter widely around their predicted values while those in the other classes stay close, Levene's test flags it.
What it does: Applies a mean-centred Levene test to the residuals of the main-effects model by group (cell for two factors).
Effect on the result: Information row only.
On the validation data: hsb2 F(2, 197) = 3.3637, p = 0.0366; DrugTest F(2, 27) = 5.858, p = 0.0077.
Assumption checks: Shapiro (checkbox)
Concept: The normality assumption applies not to the raw scores but to the model residuals, that is, each observation's deviation from the model's prediction. Shapiro-Wilk tests whether these residuals are normally distributed.
Everyday example: For each pupil, the deviation of the exam score from the score expected given their class and pre-test is computed. If these deviations are skewed to the left because a few pupils left the exam blank, the test shows it.
What it does: Tests the normality of the residuals.
Effect on the result: Information row only.
On the validation data: DrugTest W = 0.9502, p = 0.1708.
Extra outputs: Parameter estimates (checkbox)
Concept: The parameter estimates show the regression model behind ANCOVA: the slope of the covariate, and how far each group differs from the reference group with the covariate held fixed.
Everyday example: If the slope of the covariate is 0.6, a pupil who scored 10 points higher on the pre-test is expected, within the same class, to score 6 points higher in the exam. If the reference group is the digital class, a 'traditional' coefficient of −7 says that a pupil with the same pre-test score would score 7 points lower in the traditional class.
What it does: Gives the B, SE, t, p and CI table with indicator coding in which the last level is the comparison level.
Effect on the result: Extra table only.
On the validation data: DrugTest Intercept -0.434671, Drug A -3.446138 (SE 1.886781), Drug D -3.337167; common slope 0.987184 (SE 0.164498).
Extra outputs: Covariate checks (checkbox)
Concept: The covariate checks show two things: whether the covariate differs between groups, and how strongly covariate and outcome are related within each group. If the covariate differs greatly between groups, the adjustment relies on extrapolation into regions with little data; if the covariate is unrelated to the outcome, adding it to ANCOVA brings no benefit.
Everyday example: If the pre-test means are 50, 52 and 58, the classes started at different levels and the adjustment matters. But if even the lowest pre-test score in one class exceeds the highest in another, comparing those two classes 'at the same pre-test level' rests on an assumption with no real support in the data.
What it does: Gives the one-way ANOVA of covariate ~ factor and the within-group y ~ covariate relation (r, slope, p).
Effect on the result: Extra section only.
On the validation data: DrugTest PreTreatment ~ Drug F(2, 27) = 1.659, p = 0.2092; r = 0.764 in group A, 0.7452 in group B, p = 0.0101.
Extra outputs: Separate slopes (J-N) (checkbox)
Concept: When homogeneity of slopes fails, each group is given its own slope. The Johnson-Neyman method calculates over what range of covariate values the difference between two groups is significant, the 'region of significance'.
Everyday example: If game-based learning and traditional lecturing have different slopes, Johnson-Neyman might conclude that 'for pupils with a pre-test score below 45, game-based learning is significantly better; above 45, the two methods do not differ'.
What it does: In the y ~ factor x first covariate model, gives the slope, SE, t, p and CI per group; with two groups computes the Johnson-Neyman significance region (Bauer and Curran, 2005). With a second factor the section is not produced and a warning is written.
Effect on the result: Extra section only.
On the validation data: hsb2 female: slope 0.6360 in the male group, 0.5021 in the female group; J-N region (α = 0.05) -36.20 < read < 67.11, (α = 0.01) 10.18 < read < 63.78.
Extra outputs: Save predicted/residual (checkbox)
Concept: The model produces, for each observation, a prediction that takes the group and the covariate into account; the residual is the difference between the observed value and this prediction. Saving them to the dataset makes further examination of the residuals possible.
Everyday example: If a pupil's expected score given their pre-test and class is 68 and they scored 45, the residual is −23. The teacher can check whether this pupil was ill on the exam day.
What it does: Adds the ANCOVA_PRED_ and ANCOVA_RESID_ columns to the dataset.
Effect on the result: Columns are added to the dataset; the test does not change.
Confidence Level (%90, %95, %99, Other (custom), 80.0-99.9)
Concept: The confidence level sets how 'sure' the intervals for adjusted means, differences and slopes are, and the threshold (α) for significance decisions. It does not change the F and p values.
Everyday example: With 95% confidence the advantage of game-based learning over lecturing might be given as 'between 2 and 10 points'. Asking for 99% confidence widens the interval, and the Johnson-Neyman region of significance also shrinks.
What it does: α = 1 - confidence level; the adjusted mean CI, the post-hoc CI, the parameter and slope CIs and the J-N region use this α.
Effect on the result: CI widths and decisions change; F and p do not.
On the validation data: hsb2 group 2 adjusted mean 95% CI [52.900, 55.898]; DrugTest group A 90% CI [4.517, 8.913], 99% [3.135, 10.295].
2nd factor (optional)
Concept: When a second factor is added the model becomes a two-way ANCOVA: the main effects of both factors and their interaction are tested with the covariate controlled. If there is an interaction, the effect of one factor is examined separately at each level of the other.
Everyday example: If gender is added as a second factor, the analysis tests, with the pre-test controlled, whether the teaching methods' effects differ between girls and boys.
What it does: Builds the y ~ A x B + covariate model; gives the adjusted means of B, the cell EMMs and the simple effects.
Effect on the result: Extra sections; the separate slopes section is disabled.
On the validation data: hsb2 prog @ female = 0: F(2, 193) = 8.703, p = 0.0002.
Model diagnostics: Lack of fit
Concept: The lack-of-fit test checks whether the model captures well enough the means of observations that share the same combination of predictor values. A significant result shows that the model is missing something, such as a curved effect of the covariate or an interaction. The test is meaningful in data where the same combination of values occurs more than once.
Everyday example: If pre-test scores are whole numbers, several pupils in the same class will share a pre-test score. If these pupils' actual exam means depart systematically from the model's straight line, for instance because scores hit the ceiling at very high pre-test values, the test detects it.
What it does: Tests with an F test the adequacy of the model relative to the full means model over the combinations of predictor values (cells).
Effect on the result: Extra test row only. A saturated model gets an "undefined" note because pure error and lack of fit cannot be separated.
On the validation data: mtcars mpg ~ cyl + gear: F(4, 24) = 0.5612469775, p = 0.6929786891.
Model diagnostics: Heteroscedasticity tests (Breusch-Pagan, White)
Concept: Heteroscedasticity means that the spread of the residuals changes with the model terms. The Breusch-Pagan and White tests check for this relationship; a significant result shows that the F tests are less reliable and that robust methods should be considered.
Everyday example: If pupils with low pre-test scores scatter widely around their expected exam scores while those with high pre-test scores stay close, the variance of the residuals changes with the pre-test and the Breusch-Pagan test picks this up.
What it does: Tests the relation of the residual variance to the model terms (same definitions as in two-way ANOVA).
Effect on the result: Extra test rows only.
On the validation data: mtcars: Koenker 10.0526523650, original Breusch-Pagan 8.4862948732, F 4.2749929042, White 11.1991024456 (df = 6).
Custom contrast (L)
Concept: A custom contrast tests a specific question defined by the researcher through weights placed on the model coefficients. It is used for pre-planned comparisons that the standard post-hoc tests do not answer.
Everyday example: If the school's real question is 'is the average of the two new methods higher than traditional lecturing?', a contrast giving the traditional class a weight of −1 and each of the other two classes 0.5 tests this in a single test, with the pre-test controlled.
What it does: Tests a linear contrast (L β) written by the researcher on the model coefficients; rows are entered as weights on coefficient names or as a list of numbers. Gives the estimate, SE, test and CI per row, and a joint Wald test for several rows.
Effect on the result: Only an extra contrast section is added; with an empty field the output does not change. The SS type does not affect the contrast, which is built on the coefficients.
On the validation data: mtcars mpg ~ cyl * am + wt: estimates 1.89985083591 (SE 1.61043235526) and -3.86814977484 (SE 1.40652267139); joint Wald F(2, 25) = 5.41642255799.
Example on real data
Data: UCLA OARC hsb2 (High School and Beyond, n = 200)
Variables: dependent write (writing score), factor prog (programme type), covariate read (reading score)
- Question: With the reading score (read) held constant, do mean writing scores differ by programme type (prog: 1 general, 2 academic, 3 vocational)?
- Assumptions: Homogeneity of regression slopes is met (prog × read F = 0.012, p = .988). On the model residuals Levene F(2, 197) = 3.36, p = .037; homogeneity of error variance is slightly violated.
- Result: With Type III SS, prog F(2, 196) = 5.87, p = .003; covariate read F(1, 196) = 69.33, p < .001. With reading controlled, the programme types still differ significantly.
- Effect size: Partial η² prog .057 (small-to-medium), read .261 (large); model R² = 0.393.
- Adjusted means (read = 52.23): academic 54.40, general 52.50, vocational 49.61; part of the difference between the raw means (56.26, 51.33, 46.76) comes from differences in reading ability.
- APA: F(2, 196) = 5.87, p = .003, η²p = .06 (covariate read: F(1, 196) = 69.33, p < .001, η²p = .26).
Related concepts
Johnson-Neyman technique (separate slopes)Homogeneity of regression slopesOmega squared (ω²)CovariateBetween-subjects factorRegression coefficient (B) and parameter estimatesLack-of-fit testHeteroscedasticity
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.