MerQuR+Statistics Glossary

Statistics Glossary → Linear Mixed Models (LMM)

Linear Mixed Models (LMM)analysis

A regression method for dependent data, such as repeated measurements of the same person or observations within the same school, hospital or plot, that models a continuous outcome with fixed effects common to the whole sample and random effects that vary from group to group.

Concept

Ordinary regression assumes that every observation is independent. Yet a child's height measured at four ages, the scores of pupils in the same class or the yields of plants on the same plot resemble one another. If this similarity is ignored, the data seem to carry more information than they do, standard errors come out too small and significance is overstated. A linear mixed model builds this grouped structure explicitly into the model.

The model has two kinds of effect: y = Xβ + Zb + ε. Fixed effects (β) are relationships, such as age or treatment, summarised by a single value for the whole sample. Random effects (b) are each group's deviations from the average; for example each person's own starting level (random intercept) and own rate of change (random slope). The model estimates the variance of these deviations and splits the total variability into between-group and within-group components; the intraclass correlation (ICC) shows how much of the variance comes from differences between groups. In repeated measures, structures such as consecutive measurements being more alike can also be modelled through the residual covariance.

Mixed models cope with unbalanced data and missing measurements much better than repeated-measures ANOVA: a person with some missing measurements is not dropped entirely. They assume that random effects and residuals are approximately normal and that the model is correctly specified. Variance components are usually estimated by REML; ML should be used when comparing models with different fixed effects. With few groups (e.g. fewer than 5), the between-group variance cannot be estimated reliably.

When to use it

  • When studying a continuous outcome in data where the same units are measured repeatedly over time (longitudinal) or where observations are clustered in groups such as schools, hospitals or plots.
  • If the outcome is binary or a count use a generalised linear mixed model (GLMM); if only average (population-level) effects matter rather than individual ones, generalised estimating equations (GEE) can also be used.
  • If the data are balanced, complete and have only a few measurement times, repeated-measures ANOVA is a simpler option; with no grouping, ordinary multiple regression is sufficient.

An everyday example

A dietitian weighs 60 people on two different eating programmes every month for six months. Each person's starting weight differs, the rate of weight loss varies from person to person, and some people missed weigh-ins in some months. The linear mixed model takes person as a random intercept and time as a random slope and examines 'programme' and 'month' as fixed effects. It might show that people on programme A lose 1.2 kg a month on average and those on B 0.8 kg, together with the probability that the difference is due to chance, without discarding people with missing measurements.

Parameters in MerQur

Fixed effects, Dependent variable (DV) and Group variable (random intercept)

Concept: Fixed effects are predictors summarised by a single coefficient for the whole sample; the grouping variable divides the observations into clusters of similar units and defines the random intercept that gives each cluster its own level.

Everyday example: In the diet example the dependent variable is weight, the fixed effects are programme and month, and the grouping variable is the person. The random intercept accounts for how much heavier or lighter than average each person starts.

What it does: Selects the fixed part of the model, the continuous dependent variable and the grouping variable of the random intercept. Text columns are indicator-coded with the first level left out. The analysis is refused if n < 30 or groups < 5.

Effect on the result: All fixed-effect and variance-component estimates change. Nakagawa R² and ICC are given in every run.

On the validation data: sleepstudy (1 | Subject): Days 10.4673 (SE 0.8042, df 161), σ²_Subject 1378.18, σ²_e 960.46; R²m 0.2799 (performance 0.27989).

→ Fixed and random effects

Random slope (optional) and Additional random slopes

Concept: A random slope lets the effect of a predictor vary from group to group. The fixed effect then becomes the 'average slope', and its standard error also reflects the differences in slope between groups.

Everyday example: Not everyone loses the same amount each month; some lose 2 kg, others 0.3 kg. Making 'month' a random slope gives each person their own rate of weight loss and shows the uncertainty of the average rate more honestly.

What it does: Adds a separate slope for each group: (1 + x | g). Additional slopes are for more than one numeric slope.

Effect on the result: The SE and df of the slope's fixed effect grow (between-group slope variability is taken into account).

On the validation data: sleepstudy Days SE 0.8042 (df 161) -> 1.5458 (df 17); σ²_Intercept 612.09, σ²_Days 35.07; REML criterion 1743.628.

→ Random slope

Random-effects covariance (Unstructured (UN, correlated), VC / diagonal (uncorrelated))

Concept: Determines whether the random intercept and random slope are allowed to be correlated; it matters only when a random slope is present.

Everyday example: If those who start heavier lose weight faster, the intercept and slope are related. The unstructured (correlated) option estimates this link; the diagonal option assumes it is zero.

What it does: Frees (UN) or fixes to zero (VC) the covariance between intercepts and slopes.

Effect on the result: No effect without a slope.

On the validation data: sleepstudy with slope: UN -2RLL 1743.628 -> VC 1743.669; σ²_Intercept 627.57, σ²_Days 35.86; Days SE 1.5596.

→ Covariance (correlation) structure (UN, VC, AR(1), exchangeable)

Estimation method (REML, ML)

Concept: REML estimates the variance components almost without bias by accounting for the degrees of freedom used up in estimating the fixed effects. ML is needed to compare models with different fixed effects by a likelihood ratio test or AIC.

Everyday example: REML is used to report how large the differences in weight between people are. The question 'does adding programme to the model significantly improve the fit?' is answered by comparing two models fitted with ML.

What it does: REML gives unbiased estimates of the variance components; ML is needed for LRT/AIC comparison of models with different fixed effects.

Effect on the result: Variance components, -2LL and information criteria change.

On the validation data: sleepstudy with slope ML: deviance 1751.939; Potthoff-Roy UN ML: -2LL 419.477.

→ Restricted maximum likelihood (REML)

Fixed-effect df method (Satterthwaite t (default), Kenward-Roger, Wald z (asymptotic normal))

Concept: In mixed models the degrees of freedom of fixed-effect tests are not known exactly and are computed approximately. Satterthwaite and Kenward-Roger give more cautious results in small samples; Wald z is a large-sample approximation.

Everyday example: With only 12 people, the test of the programme effect really rests on 12 people, not on 72 weigh-ins. Satterthwaite or Kenward-Roger reflects this with small degrees of freedom, whereas Wald z may make the p value too small in this case.

What it does: Sets the degrees of freedom of the t tests and intervals; Kenward-Roger also inflates the SE for small samples; Wald z uses infinite df (normal).

Effect on the result: df, p and intervals change; variance components and R² do not depend on the df method.

On the validation data: In the random-slope model (Days | Subject), sleepstudy is balanced, so Kenward-Roger = Satterthwaite (Days df 17). In the random-intercept model (1 | Subject; Days SE 0.8042, df 161) the Days interval is [8.891, 12.044] with Wald z and [8.879, 12.056] with Satterthwaite.

→ Satterthwaite and Kenward-Roger degrees-of-freedom approximations

Interaction: Add interaction between the first 2 fixed effects (var1 × var2) and Additional interaction terms

Concept: An interaction means that the effect of one fixed effect depends on the level of another; it is tested by adding the product of the two variables to the model.

Everyday example: A 'programme × month' interaction tests whether the rates of weight loss differ between the two programmes. If it is significant, the gap between the programmes widens or narrows over time.

What it does: The checkbox adds the a:b term of the first two selected fixed effects; the text field accepts "a:b", "a*b", "a:b:c".

Effect on the result: A product coefficient is added; the main effects become conditional.

On the validation data: opposites time:ccog 0.4329 (SE 0.1619, df 33); the checkbox and both spellings give the same model.

→ Interaction effect

2nd grouping variable and 2nd grouping relation (nested (within group 1), crossed)

Concept: Adds a second grouping level. In a nested structure the subgroups exist within only one higher group (class within school); in a crossed structure the two groupings can occur in any combination (pupil and rater).

Everyday example: If the participants attend 6 different diet centres, person is nested within centre, and differences between centres are estimated as a separate variance component.

What it does: Adds an optional second random intercept: nested (1 | g1:g2), e.g. a split plot; crossed (1 | g2).

Effect on the result: An extra variance component. If the second group is also a fixed effect, the model is not identifiable and a clear error is given.

On the validation data: Oats (1 | Block) + (1 | Block:Variety): σ² 214.48 and 108.94, σ²_e 165.56; Variety F(2, 10) = 1.485. Machines: Machine F(2, 10) = 20.58.

→ Nested and crossed random effects

EMMEANS factor and EMMEANS p adjustment (Bonferroni, Šidák, Tukey, No adjustment)

Concept: Estimated marginal means are the model-based means of each level with the other variables held fixed; pairwise differences between levels are tested with a multiple-comparison correction.

Everyday example: The mean weight of programmes A and B at month six is computed taking differences in starting weight into account. With three programmes there would be three pairwise comparisons, and a Tukey or Bonferroni correction would limit the risk of false alarms.

What it does: Gives the estimated marginal means of a categorical fixed effect (covariates at their means) and their pairwise differences.

Effect on the result: An extra table; if a factor that is not a fixed effect is selected, a warning is given and EMM is skipped.

On the validation data: Oats: Golden Rain 104.50, Marvellous 109.79, Victory 97.63 (SE 7.798). Marvellous - Victory p = 0.1164; Bonferroni 0.3492, Šidák 0.3102, Tukey 0.2458.

→ Estimated marginal means (EMM)

Residual covariance structure (R / REPEATED) (σ²I - independent (default), AR(1) - first-order autoregressive, CS - compound symmetry, UN - unstructured, TOEP - Toeplitz (banded), CSH - heterogeneous compound symmetry, ARH(1) - heterogeneous AR(1)), Time/order variable (REPEATED), Repeated-measures subject (SUBJECT) and No random effects (R structure only; gls / MMRM)

Concept: Describes the relationship among the errors of a person's repeated measurements. For instance AR(1) assumes that measurements close in time are more alike than those far apart, while compound symmetry assumes all pairs of measurements are equally correlated.

Everyday example: A person's weights in months 2 and 3 are closer to each other than those in months 1 and 6. AR(1) models this fading similarity; choosing the right structure makes the standard error of the programme effect more reliable.

What it does: Sets the residual covariance of repeated measurements on the same subject. "No random effects" drops G and fits the model with the residual structure only (gls / MMRM).

Effect on the result: Covariance parameters, -2LL and fixed-effect SE/df change. The interval of a boundary variance is shown as "-"; CS + random intercept is not identifiable and a warning is given.

On the validation data: opposites data (Singer and Willett), no random effects, REML: -2RLL UN 1255.789, CS 1287.048, CSH 1284.954, AR(1) 1265.876, TOEP 1258.081; CS covariance 900.07 / residual 331.28.

→ Covariance (correlation) structure (UN, VC, AR(1), exchangeable)

Compare R structures by AIC/BIC

Concept: Fits different residual covariance structures with the same fixed effects and compares them by information criteria (AIC, BIC); a smaller value means a better balance between fit and complexity.

Everyday example: A single table shows which of compound symmetry, AR(1) and unstructured best fits the weigh-in data. The structure with the smallest AIC is usually preferred.

What it does: Gives a -2LL, AIC, AICC and BIC table for seven structures with the same fixed effects and the same G.

Effect on the result: An extra table.

On the validation data: opposites (no G): smallest AIC TOEP, 1274.08; the -2LL values match the individual structure runs.

→ Covariance (correlation) structure (UN, VC, AR(1), exchangeable)

Likelihood-ratio tests (random terms + fixed terms, ML)

Concept: Tests by likelihood ratio whether removing a term makes the fit significantly worse. For random terms it shows whether a variance component is needed; for fixed terms, whether a predictor contributes.

Everyday example: If the fit worsens markedly when the random slope is removed, people's rates of weight loss really do differ, and giving each person their own slope is necessary.

What it does: Performs an LR test by dropping each random term with the same method and each fixed term with ML.

Effect on the result: An extra table.

On the validation data: sleepstudy with slope, REML: dropping the slope LRT = 42.837, df 2; Days ML LRT = 23.537.

→ Likelihood ratio test (LRT)

Residual diagnostics + random-effect predictions (BLUP) and Save predictions and residuals to the dataset (PRED, RESID, FIXPRED)

Concept: Residuals are used to check the model's assumptions (normality, outliers). BLUPs are predictions of each group's random effect; predictions for groups with few observations are pulled towards the overall mean (shrinkage).

Everyday example: Each person's BLUP shows how much faster or slower than average they lose weight. For someone weighed only twice, the prediction lies closer to the average because there is little data.

What it does: Gives a conditional residual summary (Shapiro-Wilk, |e/σ| > 3), residual plots and BLUP ± conditional SD for each group; the save option adds PRED, RESID and FIXPRED columns.

Effect on the result: Extra tables and data columns. Covariance-parameter SEs come from the analytic information matrix.

On the validation data: sleepstudy with slope: Cov(Intercept, Days) SE 46.678; BLUPs are given with their conditional variances; PRED/RESID/FIXPRED are saved to the dataset.

→ Residual plots and residual diagnostics

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets the confidence level of the intervals for the fixed effects; the higher the level, the wider the interval.

Everyday example: Monthly weight loss might be between 0.9 and 1.5 kg with 95% confidence; at 99% the interval widens to something like 0.8 to 1.6 kg.

What it does: Sets the level of the fixed-effect intervals.

Effect on the result: Only the intervals change.

On the validation data: sleepstudy with slope, Days: 95% [7.206, 13.729]; 90% [7.778, 13.156]; 99% [5.987, 14.947].

→ Confidence level and confidence interval

Reference level, EMM second factor (compare within each level), Custom contrast (L) and report of rows outside the analysis

Concept: The reference level sets which level categorical variables are compared with. A custom contrast tests a researcher-defined combination of fixed effects; the report of rows outside the analysis shows observations left out because of missing values.

Everyday example: If 'control programme' is the reference, the other programmes are read relative to it. The report of excluded rows shows, for example, that rows with no recorded weight did not enter the model, so it is clear how many weigh-ins were used.

What it does: Selects the reference level of categorical fixed effects, compares groups at each time in a group × time interaction, tests a linear combination of fixed effects (L matrix) and reports rows excluded because of missing values.

Effect on the result: Extra comparison tables and a report; the main model does not change.

On the validation data: In the BodyWeight subset the level choice, group × time EMM, custom contrast and the excluded-rows report were tested: of 153 rows 111 enter the analysis and 42 are left out.

→ Reference category (comparison level)

Example on real data

Data: Potthoff and Roy 1964 dental growth (SAS MIXED Example 83.2)

Variables: dependent y (dental distance), fixed effects Gender and Age, group Person (random intercept)

  • Question: Does dental distance (y) increase with age in children, and do boys and girls differ? Because each child was measured 4 times, person was modelled as a random intercept (REML).
  • Assumption: The model assumes the same age effect for everyone; a sex-specific growth rate (Gender x Age) or a random slope should be tested separately.
  • Result: Dental distance increases by 0.660 mm per year (SE = 0.062, t(80) = 10.72, p < .001). Boys are 2.321 mm larger than girls (SE = 0.761, t(25) = 3.05, p = .005, 95% CI [0.753, 3.889]). Between-person variance 3.267, within-person residual variance 2.049; ICC = .615.
  • Effect size: Nakagawa R² marginal = .398 (fixed effects only), conditional = .768 (fixed + random); -2LL (REML) = 437.51, AIC = 447.51.
  • APA: Gender: b = 2.32, SE = 0.76, t(25.00) = 3.05, p = .005; Age: b = 0.66, SE = 0.06, t(80.00) = 10.72, p < .001; R²m = .40, R²c = .77, ICC = .61.

Related concepts

Likelihood ratio test (LRT)Fixed and random effectsCovariance (correlation) structure (UN, VC, AR(1), exchangeable)Estimated marginal means (EMM)Satterthwaite and Kenward-Roger degrees-of-freedom approximationsRandom slopeNested and crossed random effectsBest linear unbiased prediction (BLUP)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.