Statistics Glossary → Meta-Analysis
Meta-Analysisanalysis
A method that combines the results of several studies on the same question into a single overall effect estimate, weighting each study by its precision, and examines how the studies differ.
Concept
A single study often rests on a small sample and its result is uncertain; studies on the same topic may give different or even conflicting results. Meta-analysis puts the studies' effect sizes (e.g. standardised mean difference, log odds ratio, correlation) on a common scale and combines them. This gives a more precise overall estimate and makes it possible to ask why results vary from study to study.
Each study's effect is weighted by its precision (the inverse of its variance): large, precise studies influence the pooled estimate more. The fixed-effect model assumes that all studies measure one common true effect. The random-effects model accepts that the true effect may vary between studies and estimates this variation as τ²; the weights become 1/(vᵢ + τ²). Heterogeneity is summarised with Cochran's Q test and I²: I² is the share of the observed variation that comes from real differences rather than sampling error. The prediction interval shows the range within which the true effect in a new study may lie.
A meta-analysis can be no better than the quality and selection of the studies it includes. The easier publication of studies with significant results (publication bias) can inflate the pooled effect; the funnel plot, the Egger and Begg tests and trim-and-fill investigate this but cannot prove it. When heterogeneity is high, a single common effect can mislead; subgroup analysis and meta-regression look for the source of the differences, but these are observational comparisons. With few studies, τ² and the heterogeneity measures are very uncertain.
When to use it
- To combine quantitatively, in a systematic review, the results of several studies addressing the same research question.
- If the studies differ in methods, samples or settings, prefer the random-effects model; if they are genuinely similar studies run under the same protocol, the fixed-effect model.
- If the studies' raw data are available, individual participant data meta-analysis (multilevel models) is a stronger alternative; if several treatments are to be compared in one network, network meta-analysis is used.
An everyday example
A researcher gathers 12 studies on how free school breakfast programmes affect pupils' exam performance. Some found a large effect, some almost none; a few involved only 40 pupils, others thousands. A random-effects meta-analysis might find a standardised mean difference of 0.30 (95% CI 0.18 to 0.42): on average the programmes show a small-to-moderate positive effect. If I² = 45%, there is moderate real variation between studies; a prediction interval of −0.05 to 0.65 means that in a new school the programme's effect could be nil or clearly positive. A subgroup analysis could check whether the effect is larger in primary than in secondary schools.
Parameters in MerQur
Meta-regression moderators
Concept: Meta-regression tries to explain the studies' effect sizes with study-level characteristics (moderators). It shows how much of the variation between studies these characteristics account for; with few studies, though, the results are unreliable (about 10 studies per moderator is recommended).
Everyday example: If programme length (in months) is added as a moderator to the breakfast studies, meta-regression answers 'does the effect grow as the programme gets longer?'. With only 12 studies, adding several moderators at once makes the results unstable.
What it does: Explains the effect size with study-level variables (mixed-effects meta-regression); numeric or categorical moderators are selected. A ratio of about ≥ 10 studies per moderator is recommended.
Effect on the result: Gives the coefficients, QM (Wald or F), QE, residual τ² and R². Text columns are dummy-coded (the alphabetically first level is the baseline).
On the validation data: BCG (log RR, REML) for ablat b = -0.0291 (SE 0.0072), residual τ² = 0.0763, R² = 75.63%, QM = 16.36, QE = 30.73; for alloc QM = 1.7675. Raudenbush weeks (centered): b = -0.157, τ² = 0, R² = 100%.
Study label, Input format, Effect measure and CI level of input
Concept: Sets the form in which each study's result is entered (effect size and standard error, group means and standard deviations, 2x2 counts, correlation and sample size) and which common effect measure it is converted to. Each study's effect size and variance are computed from these inputs.
Everyday example: Some breakfast studies may report the two groups' means and standard deviations; these are converted to standardised mean differences. If the studies reported 'passed / failed the exam' counts, the log odds ratio would be chosen as the common measure.
What it does: The study-name column; the form of the results (effect + SE / variance / confidence interval, two-group mean-SD-n, 2x2 counts, correlation + n) and the column mapping; the effect measure (log OR, log RR, RD; SMD/MD for means, Fisher z for correlation); for interval input, the interval level (90%, 95%, 99%).
Effect on the result: Each study's yᵢ and vᵢ are computed from these choices; 0.5 is added to a zero cell (escalc to = "only0"). With a ratio measure, studies with yᵢ ≤ 0 are excluded and reported. A wrong interval level silently changes vᵢ. The label appears only in tables and plots.
On the validation data: BCG REML: log OR -0.7452, log RR -0.7145, RD -0.0252 (I² 99.94%). Hackshaw effect + variance and effect + 95% interval inputs give the same log OR 0.2189 (SE 0.0494); with 90% selected 0.2035, with 99% 0.2409. Normand MD -15.106 days (SE 8.947).
Model (Random effects, Fixed effect (common effect))
Concept: The fixed-effect model assumes that all studies measure the same true effect and that the differences between them come only from sampling error. The random-effects model accepts that the true effect varies from study to study; its intervals are therefore wider and it gives relatively more weight to small studies.
Everyday example: Breakfast programmes were run in different countries, age groups and with different menus, so the true effect is not expected to be the same everywhere. The random-effects model is then more realistic.
What it does: In random effects the true effects vary between studies (τ² > 0); in fixed effect a single common effect is assumed. Random effects is recommended if the studies differ in method and sample (Borenstein et al., 2009).
Effect on the result: Weights become 1/(vᵢ + τ²) or 1/vᵢ; random-effects intervals are wider; the prediction interval is shown only in random effects.
On the validation data: BCG log RR: random effects -0.7145 [-1.0669, -0.3622]; fixed effect -0.4303 [-0.5097, -0.3509] (SE 0.0405). Q = 152.23 is the same in both models.
τ² estimator (REML (restricted maximum likelihood), DerSimonian-Laird, Maximum likelihood, Paule-Mandel, Sidik-Jonkman, Hedges (HE))
Concept: Sets how the between-study variance of the true effects (τ²) is estimated in the random-effects model. τ² affects the weights and thus the pooled estimate, the intervals and I². REML and Paule-Mandel are generally recommended; DerSimonian-Laird is the classic choice but can underestimate τ², especially with few studies.
Everyday example: For the 12 breakfast studies, different estimators might give somewhat different values for τ², such as 0.03 or 0.05. If the conclusion is very sensitive to the choice of estimator, the finding should be interpreted with care.
What it does: Selects the estimation method of the between-study variance τ². REML or Paule-Mandel is recommended (Veroniki et al., 2016); DL is the classical default.
Effect on the result: τ² -> weights -> pooled estimate, SE, I², prediction interval. With few studies τ² is uncertain.
On the validation data: BCG log RR: REML 0.3132 (b = -0.7145), DL 0.3088, ML 0.2800, PM 0.3181, SJ 0.3455, HE 0.3286. In Raudenbush the difference is larger: ML 0.013, REML 0.019, DL 0.026, HE 0.080. BCG ablat + SJ τ² = 0.2318.
Knapp-Hartung (HKSJ)
Concept: A correction for the random-effects model that scales the standard error of the pooled estimate to the observed heterogeneity and uses the t distribution instead of the normal. It helps the confidence interval cover the true value as often as it should, especially when there are few studies.
Everyday example: With only 12 studies, the standard method may make the confidence interval too narrow. The Knapp-Hartung correction might widen it, say from 0.18-0.42 to 0.15-0.45, reflecting the uncertainty of the finding more honestly.
What it does: Scales the SE and uses the t(k - p) distribution; improves the coverage probability with few studies (IntHout et al., 2014).
Effect on the result: The pooled SE, interval and p; meta-regression coefficients (t) and QM (F). It has no effect in fixed effect (disabled in the interface).
On the validation data: BCG log RR: SE 0.1798 -> 0.1808, interval [-1.1084, -0.3206], p 0.0001 -> 0.0019; the prediction interval with t(k - 1) [-1.9960, 0.5670]. Raudenbush: p 0.105 -> 0.191.
Prediction interval (z (metafor default), t(k - 2) (Riley 2011, Stata))
Concept: The prediction interval shows not where the average effect lies but where the true effect in a new study may lie; it includes both the uncertainty of the pooled estimate and the variation between studies (τ²). Whether the critical value is taken from the normal or the t distribution noticeably affects its width when there are few studies.
Everyday example: Even with an average effect of 0.30, a prediction interval of −0.05 to 0.65 shows that a new school programme could see an effect close to zero. Using t(k − 2) with few studies makes this interval wider and more realistic.
What it does: Selects the critical value of the random-effects prediction interval: z (default; t(k - p) if Knapp-Hartung is on) or t(k - 2) (Riley et al., 2011). Interval = estimate ± critical x √(SE² + τ²). When k is small, t(k - 2) better reflects the uncertainty of τ².
Effect on the result: Only the prediction interval; the estimate, SE and confidence interval do not change. The method used is written in the text.
On the validation data: BCG log RR 95%: z [-1.8667, 0.4376] -> t(11) [-2.0084, 0.5793]; pupiliq10 90%: z [-0.3506, 0.6177] -> t(8) [-0.4138, 0.6809].
Mantel-Haenszel
Concept: A fixed-effect method that combines studies reported as 2x2 counts by using each study's table directly. When events are rare and some cells are zero, it is more robust than the inverse-variance method.
Everyday example: Had the breakfast programmes been studied for a rare outcome such as dropping out of school, many studies would have 0 or 1 dropouts. The Mantel-Haenszel method gives a more reliable common odds ratio for such sparse tables.
What it does: Combines 2x2 tables with the Mantel-Haenszel method (fixed effect; robust in sparse data).
Effect on the result: The common OR/RR/RD and SE (Robins-Breslow-Greenland, Greenland-Robins or Sato variance) and MH weights; publication bias and sensitivity analyses continue with inverse-variance fixed effect. It has no effect in random effects.
On the validation data: Fixed-effect inverse-variance log RR -0.4303 (SE 0.0405) -> MH -0.4537 (SE 0.0393). Rothman Table 15-1 (90%): MH RD 0.0349 [-0.0176, 0.0874].
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets the level of all confidence intervals for the pooled estimate, individual studies, subgroups and meta-regression coefficients, and of the prediction interval. The estimates themselves do not change.
Everyday example: If the 95% confidence interval for the pooled effect of 0.30 is 0.18-0.42, the 99% interval is wider, perhaps 0.14-0.46. If it excludes zero in both cases, the conclusion that the average effect is positive is robust.
What it does: Sets the level of all confidence intervals (pooled, study, Q-profile, subgroup, meta-regression) and of the prediction interval.
Effect on the result: Only the critical value changes; the estimate, SE and p do not change.
On the validation data: BCG log RR 95% [-1.0669, -0.3622]; 90% [-1.0102, -0.4188]; 99% [-1.1776, -0.2514]; custom 80% [-0.9449, -0.4841]. The prediction interval uses the same level (90%: [-1.6815, 0.2524]).
Subgroup (moderator)
Concept: Splits the studies into groups by a categorical characteristic, computes a separate pooled effect in each group and tests the difference between groups. It is the simplest way to look for the source of heterogeneity; but since the groups were not randomly assigned, any differences found may be entangled with other characteristics.
Everyday example: The breakfast studies are split into primary and secondary schools: the effect might be 0.40 in primary and 0.15 in secondary schools. The between-group test says whether this difference can be explained by chance; but if the primary-school studies were also done in poorer areas, the difference may stem from poverty too.
What it does: Builds a separate model at each level of a categorical moderator and tests the difference between groups.
Effect on the result: Group estimates (separate τ²), Q_between (fixed-effect Q over the group estimates) and QM with a common τ².
On the validation data: alloc: alternate -0.5408, random -0.9710, systematic -0.4242; Q_b = 1.861 (p = 0.394), common-τ² QM = 1.7675 (p = 0.4132). pupiliq10 week1: 0.581 / -0.033, Q_b = 8.18.
Publication bias: Egger
Concept: Tests the tendency of small studies to show larger effects (asymmetry of the funnel plot) by regressing the effect size on its standard error. It may signal publication bias but can also arise for other reasons; with fewer than about 10 studies its power is low.
Everyday example: If the small breakfast studies systematically found large effects and the large ones small effects, Egger's test may be significant. This suggests that small studies finding no effect may have gone unpublished.
What it does: Builds a yᵢ ~ SE regression for funnel asymmetry (small-study effect) with the model's own method; interpreted when k ≥ 10 (Sterne et al., 2011).
Effect on the result: z (t in Knapp-Hartung) and p; limit estimate (SE -> 0).
On the validation data: BCG log RR z = -0.8033, p = 0.4218; pupiliq z = 2.53, p = 0.0115.
Publication bias: Begg
Concept: Another test of funnel asymmetry, using a rank correlation (Kendall's τ): it checks whether there is a relationship between the standardised effects and their variances. It is generally less powerful than Egger's test.
Everyday example: Looking at Begg's test alongside Egger's shows whether the two methods point the same way. If neither is significant there is no strong sign of publication bias, though with 12 studies this is no firm guarantee.
What it does: Computes Kendall τ between the standardized effect and the variance (Begg-Mazumdar).
Effect on the result: τ and p.
On the validation data: BCG log OR (the model in the interpretation): τ = 0.0256, p = 0.9524 (Begg rank correlation).
Publication bias: Trim-and-fill
Concept: Estimates the number of studies thought to be missing to make the funnel plot symmetric, adds these hypothetical studies and computes an adjusted pooled estimate. It is a sensitivity analysis; the added studies are not real.
Everyday example: Trim-and-fill might estimate 3 missing breakfast studies in the lower-left corner of the funnel; adding them lowers the pooled effect from 0.30 to, say, 0.22. If the effect shrinks but stays above zero, the conclusion is relatively robust to publication bias.
What it does: Estimates the number of missing studies and the adjusted pooled estimate with Duval-Tweedie L₀.
Effect on the result: k₀ and the adjusted estimate. The adjusted model is built with the model's Knapp-Hartung and confidence level choices; this is written as a note in the output.
On the validation data: BCG log RR random effects k₀ = 1 (right), adjusted -0.6571; log RR fixed effect k₀ = 4; BCG log OR random effects (the model in the interpretation) k₀ = 0; nsaids k₀ = 10, OR = 2.815.
Publication bias: Fail-safe N
Concept: Calculates how many more unpublished studies with zero effect would have to exist to remove the significance of the combined result. A large number suggests the result is robust; but because the method considers only significance, not effect size, it is a limited indicator.
Everyday example: If the fail-safe N is 150, there would have to be 150 null studies sitting in file drawers to make the breakfast programmes' effect non-significant. It is unlikely that so many unpublished studies exist on this topic.
What it does: Rosenthal's fail-safe N: the number of zero-effect studies needed to make the combined result non-significant.
Effect on the result: An additional row.
On the validation data: BCG log RR N = 598, log OR (the model in the interpretation) N = 607 (threshold 5k + 10 = 75; 'robust').
Publication bias: Contour funnel
Concept: Adds bands to the funnel plot showing which significance levels the studies fall into. If the apparently missing studies cluster in the non-significant region, the asymmetry is more likely to come from publication bias.
Everyday example: If the gap in the breakfast studies' funnel plot lies exactly in the p > 0.10 region, small studies that 'found nothing' may have gone unpublished. If the gap lies within the significance bands, the asymmetry may have another cause.
What it does: Adds p = .10/.05/.01 significance bands to the funnel plot.
Effect on the result: Plot only; numeric outputs are the same.
Sensitivity: Leave-one-out
Concept: Removes each study in turn and recomputes the pooled estimate. If the result changes noticeably when a single study is removed, the overall finding depends too heavily on that study.
Everyday example: If removing one large study of thousands of pupils raises the pooled effect from 0.30 to 0.38, that study is pulling the result down. If the effect stays between 0.26 and 0.34 whichever study is removed, the result is robust.
What it does: Leaves out each study in turn and re-estimates the model.
Effect on the result: An additional table.
On the validation data: BCG log RR: without TPT Madras -0.7948 (τ² 0.2732), without Hart and Sutherland -0.6284; significance does not change in any removal.
Sensitivity: Influence diagnostics
Concept: Shows how poorly each study fits the model (residuals) and how strongly it influences the pooled estimate (measures such as Cook's distance, DFFITS and leverage). It helps identify outlying or overly influential studies; such studies are not dropped automatically but their reasons are examined.
Everyday example: If one breakfast study found an effect far above the others (e.g. 1.2) and has a high Cook's distance, the researcher looks again at its methods: perhaps a different exam was used, or the sample was highly selected.
What it does: Gives studentized residuals, DFFITS, Cook's distance, COVRATIO, τ²₍₋ᵢ₎, QE₍₋ᵢ₎, hat values and DFBETAS with an "influential" flag (Viechtbauer and Cheung, 2010).
Effect on the result: If a moderator is selected, the diagnostics are computed from the meta-regression model and this is stated in the text.
On the validation data: Hart and Sutherland Cook D = 0.2295, hat = 0.0970; no study is "influential". In the BCG ablat + year model, study 4: Cook D = 7.318, hat = 0.808.
→ Influence diagnostics (Cook's distance, leverage, DFBETAS)
Example on real data
Data: BCG vaccine, Colditz et al. 1994; metadat dat.bcg
Variables: 2x2 counts: vaccinated events tpos / n_t, control events cpos / n_c; study label author
- Question: Does the BCG vaccine reduce the risk of tuberculosis? The 2x2 counts of 13 studies were combined as log odds ratios (log OR) with a random-effects model (REML).
- Result: Pooled OR = 0.4746, 95% CI [0.3296, 0.6835]; z = -4.0057, p < .0001. In the vaccinated, the odds of disease are roughly halved; the effect is significant.
- Heterogeneity: Q(12) = 163.16, p < .0001; τ² = 0.3378 (on the log OR scale), I² = 92.07% (95% CI [81.74%, 97.60%]). Between-study variation is high, so the random-effects model is appropriate.
- Prediction interval: For the OR [0.1435, 1.5696]; in a similar new study the true effect may exceed 1, so the effect may not be protective in every setting (a moderator such as latitude should be sought).
- Publication bias: With the log OR measure, Egger z = -0.7743, p = 0.4388; Begg Kendall τ = 0.0256, p = 0.9524; trim-and-fill k0 = 0; fail-safe N = 607 (threshold 75; 598 with log RR). There is no evidence of a small-study effect.
- Sensitivity: Leaving one out, the OR stays between 0.4367 (TPT Madras) and 0.5151 (Hart & Sutherland); no removal changes significance.
- APA: Random-effects model (REML), k = 13, OR = 0.47, 95% CI [0.33, 0.68], z = -4.01, p < .001; Q(12) = 163.16, p < .001, I² = 92%, τ² = 0.338 (log OR); 95% PI [0.14, 1.57].
Related concepts
Egger's and Begg's testsMeta-regressionBetween-study variance (τ²) and heterogeneity (I²)Publication biasKnapp-Hartung (HKSJ) adjustmentTrim-and-fillFail-safe NPrediction interval
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.