Statistics Glossary → Mixed-Design ANOVA, Split-Plot
Mixed-Design ANOVA, Split-Plotanalysis
An analysis of variance for designs in which different groups are each measured several times, testing together the group difference, the change over time and, above all, whether the groups change differently over time (the group × time interaction).
Concept
One of the strongest designs for seeing the effect of an intervention is to divide people into groups and measure each group several times, before and after the intervention. There are two kinds of factor: a between-subjects factor that separates people (e.g. type of programme) and a within-subject factor that is repeated on the same person (e.g. time of measurement). Mixed-design ANOVA answers three questions together: do the groups differ overall, is there change over time, and does the change differ between groups? In intervention research the main interest is usually in this last question, the group × time interaction.
The analysis splits variability into two layers. The group effect is tested against the variability between people, while the time effect and the group × time interaction are tested against the variability within people; each effect therefore has its own error term and degrees of freedom. If the interaction is significant, the main effects on their own can mislead; the groups need to be compared at each time point, or the time effect examined within each group (simple effects).
The assumptions combine those of both designs: group variances are equal at each time point (Levene), the covariance structure of the repeated measurements is the same across groups (Box's M), and the within-subject measurements satisfy sphericity (Mauchly). If sphericity fails, the Greenhouse-Geisser or Huynh-Feldt correction is used. In the classical method, people with a missing measurement drop out of the analysis; with a lot of missing data, mixed-effects models are more suitable.
When to use it
- When two or more groups are measured on the same outcome at several times or under several conditions and the groups' courses over time are to be compared (treatment-control follow-up studies, evaluations of training programmes).
- If there are no groups and only the same people's measurements are compared, use repeated measures ANOVA; with only a pre and a post measurement, where the aim is to compare the post measurement adjusted for the pre measurement, ANCOVA may also be considered.
- If measurement times vary between people, there is a lot of missing data, or time needs to be modelled as a continuous variable, mixed-effects (multilevel) models are preferred.
An everyday example
In a school district, 40 pupils are randomly assigned to two reading programmes: 20 to a new read-aloud programme and 20 to the standard one. Reading speed (words per minute) is measured at the start, at 3 months and at 6 months. At the start both groups read about 60 words per minute; by month 6 the standard group reaches 75 and the new-programme group 90. In the mixed-design ANOVA, the time effect shows that both groups improved, while a significant group × time interaction shows that pupils in the new programme improved faster; this is the real evidence for the programme's effect.
Parameters in MerQur
Simple effects and pairwise comparison (checkbox)
Concept: A simple effect is the effect of one factor tested at a single level of the other. If the group × time interaction is significant, it is examined from two directions: the groups are compared at each time point, and each group's change over time is tested separately.
Everyday example: The two programme groups may show no difference at the start but a significant 15-word difference at month 6. It may also turn out that both groups improved over time, but that the gain was larger in the new programme.
What it does: When the interaction is significant, tests the group difference at each time point and the time effect in each group.
Effect on the result: The group effect is tested at each within-subject level with F(G - 1, N - G); the time effect in each group with the pooled covariance using Hotelling T² exact F and Pillai; pairwise comparisons use df = N - G.
On the validation data: exer + diet: diet | time = 1 F(1, 28) = 3.701, p = 0.0646; time | diet = 2 Pillai = 0.3087, F(2, 27) = 6.027, p = 0.0069.
EMM / simple-effect adjustment (Bonferroni, Šidák, LSD)
Concept: Pairwise comparisons within simple effects mean several tests, which raises the chance of at least one false alarm. Bonferroni and Šidák control this risk; LSD makes no adjustment.
Everyday example: Within each group, three time points form three pairs (start-month 3, start-month 6, month 3-month 6). With Bonferroni, a comparison with p = 0.03 on its own rises to 0.09 and is not significant at the 5% level.
What it does: Multiple comparison adjustment for the pairwise comparisons in the simple effects; the family is the pairs in each slice.
Effect on the result: Effective only when simple effects are on. The pairwise asterisk uses a fixed 0.05.
On the validation data: exer + exertype, time = 2: at rest - running Šidák p < 0.0001, at rest - walking 0.4557.
Levene (checkbox)
Concept: The group comparison assumes that the groups' variances are equal at each time point. Levene's test checks this equality separately for each time point.
Everyday example: If at month 6 reading speeds in the new-programme group range from 70 to 110 while those in the standard group cluster between 70 and 80, the variances differ at this time point and Levene's test flags it.
What it does: Tests, mean-centred, the equality of variances between groups at each within-subject level.
Effect on the result: Information rows only.
On the validation data: exer + exertype: time 1 F(2, 27) = 0.7777, p = 0.4695; time 3 F = 13.7097, p = 0.0001.
Box M (checkbox)
Concept: Box's M test checks whether the variances of the repeated measurements and the correlations between them, that is, the covariance matrices, are the same across groups. It is a very sensitive test that finds even small differences significant in large samples, so it is usually judged against a stricter threshold (e.g. 0.001).
Everyday example: If in the standard programme pupils' month-3 and month-6 speeds are very closely related, while in the new programme the relationship is weak because some pupils make unexpected leaps, the two groups' covariance structures differ and Box's M may pick this up.
What it does: Tests with the χ² approximation the equality across groups of the covariance matrices of the within-subject levels.
Effect on the result: Information row only.
On the validation data: exer + diet: χ²(6) = 13.2674, p = 0.0390.
Sphericity correction (Automatic, Greenhouse-Geisser, Huynh-Feldt, Lower bound, All)
Concept: The within-subject effects (time and group × time) assume sphericity: the variances of the difference scores between pairs of time points should be equal. When it fails, the degrees of freedom are shrunk by a factor ε; Greenhouse-Geisser is the more cautious correction, Huynh-Feldt slightly more liberal, and the lower bound the most cautious of all.
Everyday example: If the change in reading speed from the start to month 3 varies greatly between pupils while the change from month 3 to month 6 is consistent, sphericity fails. If the corrected p values still find the interaction significant, the conclusion about the programme's effect is robust despite the violation.
What it does: Produces corrected df and p for the within-subject and interaction rows; ε is computed from the within-group pooled covariance.
Effect on the result: With Automatic, a GG/HF p row is added when Mauchly is violated. In the multi-factor path Automatic behaves like All.
On the validation data: exer + diet, time: GG ε = 0.7538, F(1.51, 42.21), p = 0.000302; HF ε = 0.8158, p = 0.000194; lower-bound p = 0.001860.
Generalized η² (checkbox)
Concept: Generalised η² gives the size of an effect in a way that does not depend on the design; because it also takes between-person differences into account, effects from repeated-measures and independent-groups studies become comparable.
Everyday example: The reading programme's effect may have been studied here with a mixed design, and in another study only with month-6 scores from two independent groups. Generalised η² allows the effects in the two studies to be discussed on the same scale.
What it does: Gives the generalized η² (Olejnik and Algina, 2003).
Effect on the result: Extra row only.
On the validation data: exer + diet: diet 0.0726, time 0.1136, interaction 0.0118 (generalized η²).
Multivariate tests (checkbox)
Concept: The multivariate approach treats the differences between time points together as a vector and tests whether the groups differ on it. It does not require sphericity, so it offers an independent second opinion when sphericity is badly violated.
Everyday example: For each pupil, the gains 'from start to month 3' and 'from month 3 to month 6' are taken together, and the two programmes' gain profiles are compared. If the result points the same way as the univariate interaction test, the finding is more trustworthy.
What it does: Gives the MANOVA of the difference scores ~ group.
Effect on the result: Extra row only.
On the validation data: exer + diet time Pillai 0.3565, F(2, 27) = 7.4790; Wilks 0.64349965, F 7.48. With three groups the Hotelling-Lawley F uses the McKeon approximation: F(4, 30.19) = 13.6954.
→ Multivariate test statistics (Wilks' Λ, Pillai's trace, Hotelling-Lawley, Roy)
Chart CI (checkbox)
Concept: Error bars on the interaction plot show the uncertainty of each mean. Whether bars overlap or not does not replace a significance test on its own, but it gives a visual sense of how precisely the differences are estimated.
Everyday example: If at month 6 the new programme averages 90 words and the standard one 75, the confidence-interval bars show how precise each point is. If the bars are short and far apart, the difference looks reliable.
What it does: Adds error bars to the interaction plot only.
Effect on the result: The text output does not change.
Polynomial contrasts (None, Linear, Quadratic, Cubic)
Concept: If the time points are ordered and equally spaced, polynomial contrasts test the shape of the change: the linear component shows a steady rise, the quadratic component a curve that speeds up or slows down. The group × trend interaction tests whether the groups differ in these trends.
Everyday example: Measurements at months 0, 3 and 6 are equally spaced. If the group × linear trend interaction is significant, the steady rate of gain in reading speed differs between the two programmes; the line is steeper in the new programme.
What it does: Tests within-subject trends and the group x trend interaction (Type III, unweighted group means).
Effect on the result: Contrast rows only.
On the validation data: exer + diet linear F(1, 28) = 14.2016, p = 0.000779; diet x linear F = 1.4148, p = 0.244245.
Additional within-subject factor, additional between-subjects factor, and covariate(s)
Concept: The design can be extended by adding a second within-subject factor (e.g. two types of text), an extra between-subjects factor (e.g. gender) or a numeric person-level covariate (e.g. initial vocabulary). Each additional factor brings new main effects and interactions.
Everyday example: Suppose that at each measurement pupils read both a story and an informational text; text type becomes a second within-subject factor. If a vocabulary test score is added as a covariate, the programmes' effects are compared after adjusting for differences in initial language level.
What it does: Runs the repeated-measures model with a within-subject factor and several between-subjects factors together.
Effect on the result: In this path the Levene, Box M, generalized η², multivariate tests, chart CI and polynomial options are not read.
On the validation data: In the exer exertype x diet x time and dogs Drug x Depleted x Time designs all effects are reported. Numerically coded between-subjects factors (e.g. exertype 1/2/3) can also be selected.
Sums of squares in unbalanced designs
Concept: When the groups contain unequal numbers of people, the effects do not separate cleanly and one must decide how to compute the sums of squares. The Type III approach gives each group equal weight regardless of its size (unweighted means), so the larger group does not dominate the result.
Everyday example: If 3 pupils in the new-programme group changed schools, leaving groups of 17 and 20, the design is unbalanced. Type III gives the two programmes equal weight when computing the time effect, so the 'average pupil' does not drift towards the standard-programme pupils.
What it does: In unbalanced groups (dogs 8 / 7), computes the SS of the within-subject main effect as Type III (unweighted).
Effect on the result: In an unbalanced design the within-subject SS, F and p follow this definition; a subject with a missing observation is also excluded from the cell means.
On the validation data: dogs Depleted SS 12.045434, F 20.487918; Drug SS 13.247972, F 9.402302 (Type III). Dog 6, with a missing observation, is excluded (n = 7, mean -2.519932).
Example on real data
Data: UCLA OARC exer (30 subjects, 90 observations)
Variables: dependent pulse, subject id, between-subjects exertype (exercise type), within-subjects time (measurement time)
- Question: Does pulse change with exercise type (at rest, running, walking leisurely; between subjects) and measurement time (1-3; within subjects), and do the two factors interact? (30 subjects, 90 observations)
- Assumption: Mauchly W = 0.968, p = .656; sphericity holds (GG ε = 0.969, HF ε = 1.000), so the uncorrected F can be used.
- Result: Exercise type F(2, 27) = 27.00, time F(2, 54) = 23.54, interaction F(4, 54) = 15.51; p < .001 in all three.
- Effect size: Partial η² exercise type .667, time .466, interaction .535; all are large effects.
- Descriptive: Pulse stays flat at rest (90.2 → 91.4), rises slightly when walking (93.1 → 95.9) and rises markedly when running (96.1 → 126.0); the significant interaction shows that the time effect comes mainly from the running group.
- Limitation: Because the interaction is significant, the main effects should not be interpreted alone; the simple effects (time within group) should be examined.
- APA: Interaction F(4, 54) = 15.51, p < .001, η²p = .54; exercise type F(2, 27) = 27.00, p < .001, η²p = .67; time F(2, 54) = 23.54, p < .001, η²p = .47.
Related concepts
Levene's testBox's M test (equality of covariance matrices)Simple effectsGeneralized eta squared (η²G)Multivariate test statistics (Wilks' Λ, Pillai's trace, Hotelling-Lawley, Roy)Error bars (mean ± CI plot)Greenhouse-Geisser and Huynh-Feldt correctionsWithin-subject (repeated-measures) factor
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.