Statistics Glossary → Nested Mixed Model (Nested LMM, Multi-Level Hierarchical Mixed Model)
Nested Mixed Model (Nested LMM, Multi-Level Hierarchical Mixed Model)analysis
A mixed model for hierarchical trials in which subgroups sit within only one higher group (farms within a region, blocks within a site), splitting the variability in the outcome by level and testing each effect against the correct error term.
Concept
In many trials the groups are nested: each region has several farms, each site several blocks, each school several classes, and a subgroup belongs to only one higher group. 'Farm 1' in the first region and 'farm 1' in the second are not the same unit. If this structure is ignored, a higher-level effect (e.g. a difference between regions), which is really measured across a few subgroups, is tested as if it were measured across hundreds of plants, and significance is overstated.
The model treats one factor (R; e.g. a variety or family) together with the higher group (P), the subgroup within it, F(P), and their interactions: y = R + P + R×P + F(P) + R×F(P) + e. Expected mean squares (EMS) determine which term each effect is tested against; the higher-group effect, for example, is compared with the variability among subgroups, not with individual error. REML also estimates the variance component of each level, showing how much of the total variability comes from which level.
Whether a factor is treated as fixed or random changes both the tests and the interpretation: with a fixed factor the results apply only to the levels in the trial, while with a random factor the levels are regarded as a sample from a wider population. With few higher groups (e.g. only 2 or 3 sites), the degrees of freedom for the higher-group test are very small and its power is low. With unbalanced data, EMS-based F tests are approximate, and the REML results should be relied on.
When to use it
- When the experimental units are arranged hierarchically and the contribution of each level is to be examined separately (blocks within sites, farms within regions, classes within schools; site, block and family effects in forest tree breeding).
- If the grouping factors are crossed rather than nested (every worker uses every machine), a crossed mixed model is used.
- For a more flexible structure, continuous covariates or repeated measures, a general linear mixed model is preferred; for binary or count outcomes, a generalised linear mixed model.
An everyday example
A seed company tests four wheat varieties in three regions. Four farms were chosen in each region and all four varieties were sown on every farm; farms are nested within regions. The nested mixed model splits the differences in yield into variety, region, farm within region and their interactions. It tests the regional difference against how much the farms within each region differ from one another, and might show, for example, that 10% of the variance comes from differences between farms and 60% from plots within the same farm.
Parameters in MerQur
Variable selection: Dependent variable (DV), Replication (R), Upper-level group (P), Lower-level group (F, within P)
Concept: Defines the structure of the model: the factor of interest (R), the higher group (P) and the subgroup F(P) that exists within only one higher group. A subgroup's identity is set together with its higher group; identically labelled subgroups in different higher groups are distinct units.
Everyday example: In the wheat example R is variety, P is region and F(P) is farm within region. Even if farms are numbered 1 to 4 in every region, 'region 1, farm 1' and 'region 2, farm 1' are different farms, and the model treats them so.
What it does: Builds the model. The identity of F is the (P, F) pair; nesting is coded correctly even if block labels repeat in two locations.
Effect on the result: Determines all tables; it is the model definition, not an option comparison.
On the validation data: Sums of squares match Jayaraman Table 6.3 to the second decimal (family 80533.37, location 48900.46, family×location 35349.37, tree 188799.67).
SS method (Type I (sequential) / Type III (partial - for unbalanced designs))
Concept: Determines how the sum of squares is partitioned. Type I adds terms sequentially; Type III measures each term's contribution with all other terms in the model. With balanced data they agree; with unbalanced data they differ.
Everyday example: If every farm has the same number of plots of each variety, the two methods give the same result. If some farms lost plots to damage, the results diverge, and Type III is the more suitable choice for the unbalanced design.
What it does: Type I is the sequential and Type III the partial (contribution with all other terms in the model) sum of squares.
Effect on the result: Changes SS, MS and F only for unbalanced data; does not change the REML section. For unbalanced data the warning "EMS F tests are approximate, rely on the REML tests" is given.
On the validation data: The design is balanced, so Type I and Type III are identical (family F 7.129, location F 21.128). In the unbalanced derived data family F 6.989 -> 6.902.
Upper group (P) type (Fixed (default) / Random (R×P also random))
Concept: Determines whether the higher group is treated as fixed or random. With a fixed factor only the levels in the trial are of interest; with a random factor the levels are regarded as a sample from a wider population and conclusions generalise to it. This choice changes which error term each effect is tested against.
Everyday example: If the company cares only about these three regions, region is fixed. If the regions were drawn at random from many in the country and the results are to apply nationwide, region is random; the variety differences are then tested more cautiously, taking region-to-region variability into account.
What it does: Sets whether P is only these levels (fixed) or a sample from a population (random); the error terms change according to EMS.
Effect on the result: With P random, the denominator for R is MS_R×P and for P it is MS_F(P) + MS_R×P - MS_R×F(P) (Satterthwaite df). (1|P) and (1|R:P) are added to the REML model.
On the validation data: Family F 7.129 (p = 0.00056) -> 2.278 (p = 0.194); location F 21.128 (df 1 and 4) -> 6.863 (df 1 and 4.38; p = 0.0537); synthetic MS 7125.213.
Covariate / extra fixed factor
Concept: Adds an extra fixed term (a covariate or factor) to the REML model that estimates the variance components, removing the effect of a variable that influences the outcome but is not of interest.
Everyday example: If the slope of the plots affects yield, slope is added as a covariate. Differences due to slope are then removed from the residual variance and the between-farm variance is estimated more accurately.
What it does: Adds a fixed term only to the REML mixed model; the EMS table stays without the covariate.
Effect on the result: Changes the REML variance components and fixed-effect tests; does not change the EMS table.
On the validation data: Adding tree order (categorical): σ²(family × block) 201.72 -> 200.48, σ²(e) 1048.89 -> 1056.32; tree F(4, 176) = 0.683, p = 0.604.
Upper-group comparisons ((none) / Tukey HSD / Bonferroni / Šidák / Unadjusted (LSD))
Concept: When the higher group is fixed, compares the higher-group means in pairs. Because the risk of false alarms grows with the number of comparisons, Tukey, Bonferroni or Šidák corrections are used.
Everyday example: With three regions there are three pairwise comparisons: 1-2, 1-3, 2-3. Because these tests rest on the differences between farms within regions rather than on individual plot error, only large differences are significant when there are few farms.
What it does: With P fixed, tests pairwise differences of upper-group means with the MS_F(P) error term and df_F(P); produces letter groups.
Effect on the result: Changes only the comparison table. With P random the comparison is skipped with a warning.
On the validation data: One comparison: difference -30.093, SE 6.547, t(4) = -4.596, p = 0.01006; all four adjustments give the same p.
Residual normality and homogeneity of variance
Concept: Tests whether the model residuals are normally distributed (Shapiro-Wilk) and whether the higher groups have equal variances (Levene, Brown-Forsythe), checking the assumptions of the F tests.
Everyday example: If yields fall in a narrow range in one region but spread widely in another, Levene's test reveals it. Comparisons between regions should then be interpreted with care.
What it does: Shapiro-Wilk on the residuals of the full OLS model; Levene (mean-centred) and Brown-Forsythe (median-centred) tests across P levels.
Effect on the result: Adds only diagnostic output.
On the validation data: Shapiro-Wilk W = 0.98568, p = 0.0282; Levene F(1, 214) = 11.402, p = 0.00087; Brown-Forsythe F = 11.745, p = 0.00073.
Confidence Level (90% / 95% / 99% / Other (custom))
Concept: Sets the confidence level of the intervals for the higher-group means and the variance components; the higher the level, the wider the interval.
Everyday example: The mean yield in the first region might lie between 410 and 470 kg per decare with 95% confidence; at 99% the interval widens.
What it does: Sets the t confidence interval of the upper-group means and the log-Wald interval of the REML variance components.
Effect on the result: Only the interval limits change; F tests do not.
On the validation data: Vellanikkara mean interval 95% [76.810, 96.042]; 90% [78.378, 94.474]; 99% [73.705, 99.147].
REML section and h² (in every run)
Concept: Gives the REML variance component of each level and its uncertainty. When the subgroup is a genetic family (half-sibs), heritability (h²), the share of the observed variation due to inherited differences, can be estimated from the family variance; when the subgroup is a block or farm, h² has no genetic meaning.
Everyday example: The variance components show how much of the variation in yield comes from differences between farms and how much from plots within the same farm. In a forest breeding trial where the subgroups were tree families, h² would show how much of the height differences comes from inherited differences that breeding can pass on.
What it does: Produces σ² components, standard errors (analytic information matrix), Wald Z, log-Wald interval, Type III F (Satterthwaite) and h²: h²_i = 4σ²_F / (σ²_F + σ²_RF + σ²_e).
Effect on the result: Given in every run next to the EMS table; independent of the options.
On the validation data: h² 0.00491 and 0.02391; with this mapping F is a block, so h² has no genetic meaning. σ²(block[loc]) SE 49.604; location denominator df 4.0000.
EMMEANS factor ((none) / Upper group (P) / Replication (R)) and EMMEANS p adjustment (Bonferroni / Šidák / Tukey / No adjustment)
Concept: Estimated marginal means are model-based level means with the other effects balanced out; with unbalanced data they are fairer than raw means. Pairwise comparisons are given with a multiple-comparison correction.
Everyday example: If some plots were lost in one region, that region's raw mean may be skewed by which plots remained. The estimated marginal mean corrects for this imbalance and compares the regions fairly.
What it does: Computes the estimated marginal means and pairwise comparisons of the selected fixed factor from the REML mixed model with Satterthwaite df; valid for unbalanced data too. If P is random, no EMM is given for P and a warning is written.
Effect on the result: Only EMM and pairwise tables are added; the EMS table, F tests and variance components do not change. The adjustment changes only the pairwise p values.
On the validation data: In independent nested trial data P1: EMM 10.976268, SE 0.33904, df 8.884; multiple comparisons are given with the selected adjustment (e.g. Tukey).
Example on real data
Data: Jayaraman 1999 bamboo progeny trial, agridat jayaraman.bamboo
Variables: dependent height, replication family, upper group loc (location), lower group block (within location)
- Question: How does bamboo seedling height vary by family (6 half-sib families), location (2 locations) and blocks nested within location? Blocks were treated as random within location.
- Assumption: The location effect is tested with an error term of only 4 degrees of freedom; power is low because of the small number of blocks, and the Wald test is unreliable because the block variance is at the boundary (≈ 0).
- Result: With EMS error terms, the family effect F(5, 20) = 7.13, p < .001, the location effect F(1, 4) = 21.13, p = .010 and the family x location interaction F(5, 20) = 3.13, p = .030 are significant. The block effect is not significant (F(4, 20) = 1.02, p = .419); the family x block interaction is significant (F(20, 180) = 2.15, p = .004). Mean height is 86.43 (SD 50.41) in Vellanikkara and 56.33 (SD 28.55) in Nailambur.
- Effect size: σ² block(location) = 1.54 (0.1%), σ² family x block = 201.72 (16.1%), σ² error = 1048.89 (83.8%). Individual narrow-sense h² = 0.005, family-mean h² = 0.024 (half-sib assumption, σ²A = 4σ²F). In this data mapping F is the block; h² is meaningful only under the assumption that F is a half-sib family, so here it should not be interpreted as a genetic heritability.
- APA: Family F(5, 20) = 7.13, p < .001; location F(1, 4) = 21.13, p = .010; family x location F(5, 20) = 3.13, p = .030.
Related concepts
F testExpected mean squares (EMS)Restricted maximum likelihood (REML)Variance componentsTypes of sums of squares (Type I, II, III)Estimated marginal means (EMM)Fixed and random effectsSatterthwaite and Kenward-Roger degrees-of-freedom approximations
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.