MerQuR+Statistics Glossary

Statistics Glossary → Two-Way ANOVA

Two-Way ANOVAanalysis

An analysis of variance that tests, in one model, the separate effects of two factors on a continuous outcome (main effects) and how they work together (interaction).

Concept

An outcome is usually shaped not by one factor but by several acting together. Two-way ANOVA puts two factors into the model at once and answers three questions together: does the first factor have a main effect, does the second, and do the two interact? An interaction means that the effect of one factor changes with the level of the other, for instance a drug that works only in women. Studying the two factors separately with one-way ANOVAs misses this information.

The total variability is split into parts for each factor, for the interaction and for error (within-cell variability); for each effect, F = mean square of the effect / error mean square. If the interaction is significant, the main effects on their own can mislead; the effect of one factor is then examined separately at each level of the other (simple effects). On an interaction plot, parallel lines mean no interaction; lines that diverge or cross indicate an interaction. When the cells contain unequal numbers of observations (an unbalanced design), the effects do not separate cleanly and the choice of sum-of-squares type (I, II or III) affects the result.

The assumptions of one-way ANOVA apply here to every cell: observations are independent, residuals are approximately normal and cell variances are equal. When variances are unequal, robust standard errors can be used. Adding a numeric covariate to the model turns the analysis into analysis of covariance (ANCOVA), and group comparisons are adjusted for the covariate. Effect sizes such as partial η² show the practical importance of each effect.

When to use it

  • When testing, together, the effects of two categorical factors (e.g. diet and exercise, fertiliser and irrigation, teaching method and gender) on a continuous outcome and the interaction between them.
  • With a single factor use one-way ANOVA; if the same people are measured under several conditions use repeated-measures or mixed-design ANOVA.
  • If several continuous outcomes are studied together, use MANOVA; if residuals are clearly non-normal and variances very unequal, consider robust standard errors or a suitable generalised linear model.

An everyday example

A diet clinic studies how three diets (Mediterranean, low-carbohydrate, standard) and regular exercise (yes, no) affect weight lost over 12 weeks, with 10 clients in each diet-exercise combination. Two-way ANOVA answers separately: do the diets differ in weight loss, do those who exercise lose more, and is the benefit of exercise the same under every diet? If, say, exercise adds an average of 1 kg of extra loss on the standard diet but 4 kg on the low-carbohydrate diet, there is an interaction: the effect of exercise depends on the diet.

Parameters in MerQur

Sum of squares (SS) type (Type III, Type II, Type I)

Concept: When the cells contain unequal numbers of observations, the variability explained by the factors overlaps, and one must decide which effect the shared part is credited to. Type I tests effects in the order they enter the model, Type II adjusts each main effect for the other main effect, and Type III adjusts each effect for all the others in the model, interaction included. In a balanced design all three give the same result.

Everyday example: If some clients dropped out, leaving between 7 and 10 people per cell instead of 10, the design is unbalanced. The effect of diet then depends, under Type I, on whether it enters before or after exercise; Types II and III remove this dependence on order.

What it does: In an unbalanced design, sets which other effects an effect is adjusted for: Type I is sequential, Type II adjusts each main effect for the other main effects, Type III adjusts each effect for all other effects (sum-to-zero contrasts).

Effect on the result: The main effect SS, F, p, η²ₚ, η², ω² and observed power change; the error MS and the interaction row (last term) do not.

On the validation data: Kutner drug: Type III F(3, 46) = 9.046, Type II 9.245, Type I 9.456; interaction 1.067 in all. In balanced warpbreaks the three types coincide. With a covariate the Type I order is A, B, A x B, covariate.

→ Types of sums of squares (Type I, II, III)

Post-hoc test (None, Tukey HSD, Bonferroni, Games-Howell, Scheffé, Šidák, LSD, Tamhane T2, Duncan, SNK, REGWQ)

Concept: Pairwise comparisons that, when the main effect of a factor with three or more levels is significant, find which levels differ from one another while controlling the familywise error rate. A two-level factor involves only one comparison, so no post-hoc test is needed.

Everyday example: If the main effect of diet is significant, Tukey's HSD might conclude that 'Mediterranean and low-carbohydrate do not differ from each other, but both produce more weight loss than the standard diet'. Exercise has only two levels (yes/no), so no post-hoc test is needed for it.

What it does: Pairwise comparison of the observed marginal means of each factor with three or more levels, using the full-model MSE and error df. Skipped for a two-level factor and when a covariate is selected.

Effect on the result: Adds only the post-hoc section; the ANOVA table does not change.

On the validation data: Kutner drug 1-3 (Δ = 17.317): Tukey p = 0.0006; Bonferroni 0.0006, CI [6.094, 28.539]; Scheffé 0.0015; LSD 0.0001. warpbreaks SNK p(L-M) = 0.0025952549.

→ Post hoc tests (pairwise comparisons)

Effect sizes: η²ₚ, η², ω² (checkboxes)

Concept: Effect sizes show how much of the variability each effect explains. η² is the effect's share of the total variability; partial η² (η²ₚ) computes the same share within 'this effect + error' only, leaving the other effects out; ω² estimates the population value with less bias.

Everyday example: If exercise has a partial η² of 0.15, then once the part explained by diet and the interaction is set aside, 15% of the variation in weight loss is related to exercise. In a model with many factors the partial η² values can add up to more than 1; this is not an error.

What it does: η²ₚ = SS/(SS + SS_error); η² = SS/SS_corrected total; ω² = (SS - df·MSE)/(SS_total + MSE).

Effect on the result: Adds columns to the ANOVA table; only η²ₚ appears in the item line of the result text.

On the validation data: Kutner Type III drug: η²ₚ = 0.3711, η² = 0.3209, ω² = 0.2821; η² and ω² are semi-partial (denominator is the total SS).

→ Eta squared (η²) and partial eta squared (η²ₚ)

Assumption checks: Levene (homogeneity of variance) (checkbox)

Concept: Two-way ANOVA assumes equal variances in every cell (every combination of factor levels); Levene's test checks this assumption across all cells. Its median-centred version (Brown-Forsythe) is more robust for skewed data.

Everyday example: Is the spread of weight loss similar in each of the six diet-exercise groups? If, in the exercising low-carbohydrate group, some lost 1 kg and others 12 kg, that group's variance may be larger than the rest, and Levene's test will flag it.

What it does: Tests equality of variances across the cells (A x B); gives the mean-centred Levene and, as information, the median-centred (Brown-Forsythe) row.

Effect on the result: Information row only.

On the validation data: Kutner W = 1.2457, p = 0.2858; hsb2 W = 4.3376, p = 0.0009 (violated), Brown-Forsythe W = 3.6249, p = 0.0037.

→ Levene's test

Assumption checks: Shapiro (residual normality) (checkbox)

Concept: The normality assumption applies not to the raw data but to the model residuals, that is, each observation's deviation from its own cell mean. Shapiro-Wilk tests whether these residuals are normally distributed.

Everyday example: For each client, the deviation of their weight loss from their group's mean is computed. If these deviations are skewed to the right because of a few exceptionally successful clients, the test shows it; with 60 people, moderate departures do little harm.

What it does: Tests the normality of the residuals of the fitted model.

Effect on the result: Information row only.

On the validation data: Kutner W = 0.9645, p = 0.0876; hsb2 W = 0.9747, p = 0.0011.

→ Shapiro-Wilk test

Descriptive means: Cell + marginal means (checkbox)

Concept: Cell means give the mean of each combination of factors; marginal means give the mean at each level of one factor with the other factor set aside. Cell means are used to understand the interaction, marginal means to understand the main effects.

Everyday example: The cell table gives six separate values such as 'the exercising Mediterranean group lost 6 kg on average'. The marginal table says 'regardless of diet, those who exercised lost 5.5 kg on average and those who did not lost 3 kg'.

What it does: Gives the observed mean, SD (n - 1) and n at the cell and marginal level.

Effect on the result: Extra table only.

On the validation data: hsb2 female = 1, ses = 3: M = 58.966, SD = 6.790, n = 29; Kutner drug = 1, disease = 1: M = 29.333, SD = 13.018, n = 6.

→ Estimated marginal means (EMM)

Confidence Level (%90, %95, %99, Other (custom), 80.0-99.9)

Concept: The confidence level sets how 'sure' the intervals for means, differences and parameters are, and the threshold (α) for significance decisions. It does not change the F and p values in the ANOVA table.

Everyday example: With 95% confidence the extra benefit of exercise might be given as 'between 1.5 and 3.5 kg'; asking for 99% confidence widens the interval. If a clinical recommendation is to be written, this width is also taken into account.

What it does: α = 1 - confidence level. The EMM CIs, the post-hoc CIs and decisions, the parameter CIs, observed power, the interaction plot with CIs and the assumption statements use this α.

Effect on the result: When the α-dependent sections are off (ANOVA, assumptions and descriptives only), no visible number changes.

On the validation data: Kutner drug = 1 EMM 95% CI [20.4570, 31.5319], 99% [18.6024, 33.3864]; observed power (drug) 0.9931 at 95%, 0.9975 at 90%.

→ Confidence level and confidence interval

Estimated marginal means (checkbox)

Concept: Estimated marginal means (EMMs) are marginal means computed with every cell given equal weight. In an unbalanced design the observed marginal mean gives more weight to the larger cells; the EMM shows what the mean would be 'if all cells were the same size'.

Everyday example: If most of the clients who exercised happen to be on the Mediterranean diet, the observed mean of the exercise group also carries the effect of that diet. Because the EMM gives each diet equal weight, it frees the benefit of exercise from this imbalance in the diet mix.

What it does: Gives the mean of the unweighted cell means; SE = √(MSE·Σ1/n)/k; CI t(1 - α/2, df_error); pairwise differences with the selected adjustment. Skipped in a covariate or main-effects-only model.

Effect on the result: Extra section only.

On the validation data: Kutner drug EMM: 25.9944, 26.5556, 9.7444, 13.5444; drug 1-3 Δ = 16.25, SE = 4.1451, p (Bonferroni) = 0.0018.

→ Estimated marginal means (EMM)

Simple effects (None / Factor 1 at each level of factor 2 / Factor 2 at each level of factor 1 / Both)

Concept: A simple effect is the effect of one factor tested separately at a single level of the other factor. When the interaction is significant, a main effect can mislead as an 'average effect'; simple effects show where the effect is present and where it is not.

Everyday example: If the interaction is significant, the effect of exercise is tested within each diet separately: the benefit of exercise may be non-significant on the standard diet but clear on the low-carbohydrate diet. This is far more informative than a general sentence such as 'exercise increases weight loss'.

What it does: Gives the F test of one factor at each level of the other (full-model MSE) and the within-slice pairwise differences.

Effect on the result: Extra section only.

On the validation data: Kutner drug @ disease = 2: F(3, 46) = 7.440, p = 0.0004, η²ₚ = 0.3267.

→ Simple effects

EMM / simple-effect adjustment (Bonferroni, Šidák, LSD)

Concept: When several comparisons are made, the chance of at least one false alarm grows; adjustment methods control this. Bonferroni multiplies each p value by the number of comparisons, Šidák makes a slightly less conservative exact adjustment, and LSD makes no adjustment at all.

Everyday example: If the effect of exercise is tested within each of the three diets, there are three comparisons. With Bonferroni, a result with p = 0.02 on its own rises to 0.06 and no longer counts as significant at the 5% level.

What it does: For m comparisons in the family uses p·m, 1 - (1 - p)^m or the unadjusted p.

Effect on the result: Effective only when EMM or simple effects are on.

On the validation data: Kutner drug 1-4: Bonferroni 0.0124, Šidák 0.0123, LSD 0.0021.

→ Multiple comparisons and the familywise error rate (FWER)

Model (Full factorial (A × B), Main effects only (A + B))

Concept: The full factorial model contains both main effects and the interaction; the main-effects-only model assumes that the effects of the two factors simply add up independently. If an interaction really exists, dropping it from the model produces misleading results.

Everyday example: If the benefit of exercise is the same under every diet, the main-effects model is enough and gives a simpler account. But if exercise helps only on the low-carbohydrate diet, using this model would bury that important finding in the averages.

What it does: Removes the interaction term from the model; the error df and MSE change.

Effect on the result: Kutner drug: F(3, 46) = 9.046 in the full model, F(3, 52) = 9.174 in the main-effects model.

→ Interaction effect

Covariates

Concept: A covariate is a numeric variable that affects the outcome but is not the main focus of the study. When it is added to the model (ANCOVA), group comparisons are adjusted for it, and by reducing error variance the test can gain power.

Everyday example: Clients who start heavier usually lose more weight. Adding starting weight as a covariate compares the diets 'for clients with the same starting weight', so differences in starting weight between groups do not distort the result.

What it does: Adds the selected numeric column(s) to the model (ANCOVA).

Effect on the result: Post-hoc and EMM are skipped in this model.

On the validation data: hsb2 + read: female F 16.595 -> 27.876, read F(1, 193) = 110.67.

→ Covariate

Robust SE (HC) (None, HC3, HC0)

Concept: Robust (heteroscedasticity-consistent) standard errors give reliable tests even when the error variance changes from cell to cell; they do not rely on the equal-variance assumption. HC3 is the version that usually performs better in small and medium samples.

Everyday example: If weight loss is much more spread out in the exercising groups, the classical F may understate the uncertainty of those groups. Robust standard errors take the actual spread in each group into account and give a more honest p value.

What it does: Gives Wald F tests with an HC covariance under variance heterogeneity.

Effect on the result: F and p change in Type II and III. With Type I the robust test cannot be applied; a warning is written and the classic Type I is given. SS, MS and η²ₚ are not shown in the robust test.

On the validation data: Kutner Type III drug: classic 9.046, HC3 7.3125, HC0 12.2927.

→ Robust (sandwich) standard errors (HC0-HC3)

Extra output: Observed power (checkbox)

Concept: Observed power is the probability that the test would be significant if the effect size seen in the sample were the true effect. Because it is merely a transformation of the observed p value, it carries no new information; in particular, it should not be used to explain away a non-significant result as 'low power'.

Everyday example: If the interaction is non-significant with p = 0.30, its observed power will also be low; that is just another way of saying what is already known, that the interaction is not significant. To decide how many clients a future study needs, a power analysis should be done in advance with a theoretically justified effect size.

What it does: λ = F·df1, power = P(F'(df1, df_error, λ) > F_{1-α}). Observed power is a transformation of the observed p and has limited interpretive value (Hoenig and Heisey, 2001).

Effect on the result: Extra row only.

On the validation data: Kutner drug λ = 27.138, power 0.9931.

→ Observed (post hoc) power

Extra output: Parameter estimates (checkbox)

Concept: Parameter estimates show the regression model behind the ANOVA: each factor level is coded with an indicator (0/1) variable, and the coefficients give how far that level differs from the chosen reference level.

Everyday example: If the reference levels are the standard diet and no exercise, a 'Mediterranean' coefficient of 2.1 kg means that a non-exercising client loses 2.1 kg more on average on the Mediterranean diet than on the standard one. When there is an interaction, these coefficients hold only at the reference level.

What it does: Indicator coding with the last level as the comparison level; gives B, SE, t, p, CI.

Effect on the result: Extra table only.

On the validation data: Kutner Intercept 14.2, 6.2 for drug = 1 (SE 6.647).

→ Regression coefficient (B) and parameter estimates

Extra output: Save predicted/residual (checkbox)

Concept: The model produces a prediction (the cell mean) for every observation; the residual is the difference between the observed value and this prediction. Saving them to the dataset makes further examination of the residuals possible (finding outliers, other tests, plots).

Everyday example: If a client lost 14 kg and their group's prediction is 6 kg, the residual is 8 kg. If the standardised residual exceeds 3, the dietitian checks that client's record; perhaps they were taking an additional medication.

What it does: Adds the PRE_, RES_, ZRE_ (= residual/√MSE) columns to the dataset.

Effect on the result: Columns are added to the dataset; the test does not change.

Chart (Interaction plot, Interaction plot + CI bars, Residual vs fitted plot, Q-Q plot of residuals)

Concept: An interaction plot shows the levels of one factor on the horizontal axis and each level of the other factor as a separate line: parallel lines mean no interaction, while lines that move apart or cross indicate one. The residual-versus-fitted plot and the Q-Q plot of residuals let you check the equal-variance and normality assumptions by eye.

Everyday example: With the diets on the horizontal axis and exercise yes and no as two lines, the lines lie close together for the standard diet but far apart for the low-carbohydrate diet. This widening tells in one picture that the benefit of exercise varies with the diet.

What it does: Selects the Chart tab. The CI type draws cell mean ± t(1 - α/2, df_error)·√(MSE/n).

Effect on the result: The result text does not change.

→ Interaction plot

Model diagnostics: Lack of fit

Concept: The lack-of-fit test checks whether a simpler model (e.g. main effects only) captures the cell means in the data well enough, by comparing it with the full model that gives each cell its own mean. A significant result shows that the simpler model is missing something, often the interaction.

Everyday example: The clinic wants to work with the main-effects model. If the lack-of-fit test is significant, the actual means of the six groups differ clearly from what that model predicts; in other words, the benefit of exercise varies between diets.

What it does: Tests with an F test whether the main-effect terms are adequate relative to the full cell-means model; for a saturated model an "undefined" note is given.

Effect on the result: Extra test row only.

On the validation data: warpbreaks (main effects only): F = 4.1890689669, p = 0.0210441907.

→ Lack-of-fit test

Model diagnostics: Heteroscedasticity tests (Breusch-Pagan, White)

Concept: Heteroscedasticity means that the spread of the residuals changes with the model terms or with the predicted value. The Breusch-Pagan and White tests check for this relationship; a significant result suggests using robust standard errors.

Everyday example: If residuals are larger in groups with a high predicted weight loss, that is, the results of those expected to lose a lot are more variable, the Breusch-Pagan test picks this up.

What it does: Tests the relation of the residual variance to the model terms: Breusch-Pagan in the original, Koenker and F forms; White with all squares and cross products (n·R², df = rank - 1).

Effect on the result: Extra test rows only.

On the validation data: warpbreaks full model: Koenker 21.5743594325, original 17.5948613787, White 21.5743594325 (df = 5).

→ Heteroscedasticity

Custom contrast (L)

Concept: A custom contrast tests a particular combination of cell or level means, using weights defined by the researcher. It is used to test specific, pre-planned questions that the standard tests do not answer.

Everyday example: The clinic's specific question might be 'is the Mediterranean diet with exercise more effective than the standard diet with exercise?'. The contrast compares these two cells directly and gives the estimated difference with its confidence interval.

What it does: Tests user-defined linear contrasts from an indicator-coded OLS model (first level as baseline); gives the estimate, SE, t, p and CI per row, and a joint F test for several rows.

Effect on the result: A contrast table is added to the text interpretation section.

On the validation data: warpbreaks: contrast rows -0.5 (SE 3.646761346) and -20.5555556 (SE 5.157299354); joint F(2, 48) = 9.309248443, p = 0.00038328.

→ Contrast and custom contrast (L)

Example on real data

Data: UCLA OARC hsb2 (High School and Beyond, n = 200)

Variables: factor 1 female (sex), factor 2 ses (socioeconomic status), dependent write (writing score)

  • Question: Does the writing score differ by sex (female) and socioeconomic status (ses: 1-3), and do the two factors interact?
  • Assumptions: Levene W = 4.34, p < .001 and Shapiro-Wilk on the residuals W = 0.975, p = .001; homogeneity of variance and normality are violated, so the results should be interpreted with care (with n = 200 the F test is fairly robust).
  • Result: With Type III SS, the main effect of sex F(1, 194) = 16.59, p < .001; the main effect of ses F(2, 194) = 6.61, p = .002; the interaction F(2, 194) = 0.13, p = .875 (not significant).
  • Effect size: Partial η² sex .08, ses .06 (medium), interaction .00; model R² = 0.127.
  • Descriptive: Girls (M = 54.99) score higher than boys (M = 50.12), and the high ses group (M = 55.91) higher than the low group (M = 50.62); with no interaction, the sex difference is similar at every ses level.
  • APA: Sex F(1, 194) = 16.59, p < .001, η²p = .08; ses F(2, 194) = 6.61, p = .002, η²p = .06; interaction F(2, 194) = 0.13, p = .875, η²p = .00.

Related concepts

Observed (post hoc) powerPost hoc tests (pairwise comparisons)Estimated marginal means (EMM)Simple effectsInteraction plotHeteroscedasticityEta squared (η²) and partial eta squared (η²ₚ)Types of sums of squares (Type I, II, III)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.