Statistics Glossary → Hierarchical Bayesian Regression
Hierarchical Bayesian Regressionanalysis
A multilevel regression model for grouped data, such as pupils within schools or patients within hospitals, that estimates the effects of predictors while accounting for differences between groups and gives each group an estimate improved by 'borrowing strength'.
Concept
When observations come in groups, members of the same group resemble one another: pupils in the same school are taught by the same teachers, and repeated measurements on the same person come out close together. Ordinary regression that ignores this dependence understates the standard errors, while fitting a separate regression for each group gives very unstable results in small groups. The hierarchical model takes a path between these two extremes.
The model lets each group have its own intercept (and, if wanted, its own slope), but assumes that these group values come from a common distribution: y_ij = (β₀ + u_j) + β₁x_ij + ε_ij, u_j ~ N(0, σ²_u). As a result, each group's estimate is a weighted average of its own data and the mean of all groups; groups with few observations are pulled more towards the overall mean (shrinkage). These group estimates (BLUPs) are empirical Bayes estimates. The intraclass correlation (ICC = σ²_u / (σ²_u + σ²_ε)) shows how much of the total variation comes from differences between groups. In this analysis the model is estimated by REML and the need for the group structure is judged by a BIC-based Bayes factor; no fully Bayesian (MCMC) posterior is produced.
Assumptions: group effects and residuals are roughly normal, groups are independent of one another, and the relationship is linear. With few groups (e.g. fewer than 5), the variance components cannot be estimated reliably. The fixed effects describe the average relationship across all groups, and the variance components describe how far the groups deviate from that average.
When to use it
- When the data fall into natural groups (school-pupil, hospital-patient, repeated measurements on the same person, region-household) and both the overall relationship and the differences between groups matter.
- If the effect of a predictor is suspected to vary from group to group, use a random-slope model; if the groups are themselves nested in higher groups (classes within schools), use a nested model.
- If the outcome is binary or a count, use a generalised linear mixed model (GLMM); if only the average effect is wanted and the group structure is treated as a nuisance, regression with cluster-robust standard errors or GEE is an alternative.
An everyday example
A provincial education office wants to explain the maths scores of 1,200 pupils in 25 schools by weekly study time. Some schools have only 15 pupils. The hierarchical model gives each school its own starting level but balances the estimate for a 15-pupil school with information from the other schools. The result might read: 'each extra hour of study raises the score by 2 points on average, and 20% of the variation in scores comes from differences between schools'.
Parameters in MerQur
Dependent variable, Fixed effects (X), Grouping variable
Concept: Fixed effects give the average relationships assumed common to all groups; the grouping variable states which cluster each observation belongs to and defines the model's random intercept.
Everyday example: The maths score is the dependent variable, weekly study time the fixed effect and school ID the grouping variable. The model answers 'what is the effect of study time in an average school?' and 'how much do schools differ in their starting level?' together.
What it does: Variable selection. Text fixed effects are dummy-coded (the alphabetically first level is the baseline level). At least 15 observations and 3 groups are required; an intercept-only model (without selecting a fixed effect) works.
Effect on the result: All fixed effects, variance components, the ICC and the BLUPs change.
On the validation data: radon: intercept 1.46160, basement -0.69299; σ²_u = 0.10773, σ²_ε = 0.57091; SE 0.05158 and 0.07043.
Random slopes (optional)
Concept: A random slope lets the effect of a predictor vary from group to group; the spread of the slopes and their correlation with the intercepts are also estimated.
Everyday example: In some schools study time may affect scores strongly and in others weakly (e.g. more strongly where there is a supervised study programme). A random slope models this difference and answers 'how much does the effect of studying vary from school to school?'.
What it does: The (1 + x | group) model; the intercept-slope covariance is free. It is valid only for numeric variables selected among the fixed effects; if a variable that is not in X is selected as a slope, it is ignored.
Effect on the result: The fixed effects and variance components change.
On the validation data: sleepstudy: β = 251.405 (SE 6.825) and 10.467 (SE 1.546); Var(intercept) 612.09, Var(Days) 35.07, correlation 0.066, σ²_ε 654.94; REML -2LL = 1743.63.
Nested subgroup (optional)
Concept: Nested grouping models a structure in which subgroups sit inside higher-level groups; a separate variance component is estimated for each level.
Everyday example: If pupils belong to classes and classes to schools, school and class-within-school are added as two levels. This separates how much of the variation in scores comes from differences between schools and how much from differences between classes in the same school.
What it does: Builds the (1 | g/g2) = (1 | g) + (1 | g:g2) model.
Effect on the result: An additional variance component is added.
On the validation data: Pastes: batch 1.6573, batch:cask 8.4337, σ²_ε 0.6780, intercept 60.0533 (SE 0.6769); REML -2LL = 246.99.
Confidence Level (%90 / %95 / %99 / Other, custom value)
Concept: Sets the confidence level of the interval estimates for the fixed effects, the group estimates (BLUPs) and the ICC; the higher the level, the wider the intervals.
Everyday example: If the study-time effect is between 1.6 and 2.4 points per hour with 95% confidence, the interval widens somewhat at 99%.
What it does: The level of the fixed-effect interval (β ± z·SE, "Wald CI"), the BLUP interval and the ICC interval.
Effect on the result: Only the interval limits change.
On the validation data: radon basement 95% [-0.8310, -0.5550].
Profile CI for variance components
Concept: Variance components cannot be negative and their sampling distributions are often skewed, so a profile likelihood interval, which follows the shape of the likelihood, is more accurate than a symmetric 'estimate ± 2 SE' interval.
Everyday example: If the between-school standard deviation is estimated at 6 points, the profile interval may be asymmetric, such as [4, 9]: the true value could be a little below 6 but could also be much higher. A symmetric interval could even push the lower limit below zero.
What it does: Profile likelihood confidence interval for σ_u and σ_ε.
Effect on the result: Only adds intervals.
On the validation data: radon 95% σ_u [0.24351, 0.42225], σ_ε [0.72054, 0.79260]; sleepstudy random intercept σ_u [26.007, 52.936]; in the slope model σ_u [14.381, 37.716] (profile likelihood intervals).
ICC interval bootstrap samples (B)
Concept: The interval for the ICC is found by generating many artificial data sets from the fitted model and recomputing the ICC in each (parametric bootstrap). More replications make the interval limits more stable.
Everyday example: The model generates artificial data sets of 25 schools and 1,200 pupils 1,000 times and computes the ICC each time. The middle 95% of these 1,000 values gives the confidence interval for the ICC; with only 200 replications, the limits can shift a little from run to run.
What it does: The number of parametric bootstrap iterations for the ICC interval (between 100 and 20 000; default 1000; fixed seed).
Effect on the result: Only the ICC interval changes (Monte Carlo error about 1/√B); the estimates do not change. The run time increases linearly with B.
On the validation data: radon 95% ICC interval: [0.0774, 0.2416] for B = 200, [0.0863, 0.2385] for 1000, [0.0845, 0.2348] for 3000.
Example on real data
Data: Belenky et al. 2003 sleepstudy, R lme4
Variables: dependent Reaction (reaction time, ms), fixed effect Days (days of sleep deprivation), group Subject (18 participants)
- Question: How much does the number of days of sleep deprivation (Days) increase reaction time (Reaction, ms)? Because the 18 subjects were measured every day, the observations are dependent within subject; a hierarchical model with a random intercept for subject is fitted.
- Result: Each additional day of sleep deprivation lengthens reaction time by 10.47 ms on average (SE = 0.80, z = 13.02, p < .001, 95% CI [8.89, 12.04]); the baseline level is 251.41 ms.
- Variance components: Between-subject variance σ²_u = 1378.18 (SD 37.12), residual variance σ²_ε = 960.46; ICC = 0.589 (95% bootstrap CI [0.36, 0.74]): about 59% of the total variability comes from differences between subjects.
- Model comparison: The BIC-approximated Bayes factor of the hierarchical model against single-level OLS is BF₁₀ = 8.64e+21 (BIC 1814.85 and 1915.87): the evidence for accounting for the group structure is decisive.
- Additional finding (empirical Bayes): The slowest subject, 337, is 72.18 ms above the mean (95% interval [53.61, 90.75]); the BLUPs shrink the group means toward the grand mean.
- Limitation: The method is empirical Bayes (REML); the fixed-effect intervals are Wald z intervals, not fully Bayesian credible intervals. The model assumes that the slope does not vary by subject.
- APA: Linear mixed model (REML, N = 180, K = 18): Days β = 10.47, SE = 0.80, z = 13.02, p < .001; ICC = .59, BF₁₀ (LMM/OLS) = 8.64e+21.
Related concepts
Intraclass correlation (ICC)Variance componentsRandom slopeFixed and random effectsBest linear unbiased prediction (BLUP)Nested and crossed random effectsBayes factor (BF₁₀)Restricted maximum likelihood (REML)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.