Statistics Glossary → Log-Linear Model
Log-Linear Modelanalysis
A method that models the cell counts of a crosstab formed by three or more categorical variables to find out which variables are associated and whether those associations change across the levels of the others.
Concept
The chi-square test examines the association between two categorical variables; with three variables the questions multiply: are A and B still associated when C is held fixed? Is the association between A and B the same at every level of C? A log-linear model answers these questions while treating all variables symmetrically, without singling out one as the outcome.
The model writes the logarithm of the expected count in each cell as a sum of the variables' main effects and interactions: log μ = λ + λ_A + λ_B + λ_C + λ_AB + λ_AC + λ_BC + λ_ABC. Main effects only reflect how common each category is overall; associations live in the interaction terms. If the λ_AB term is absent, A and B are conditionally independent given C; if λ_ABC is absent, the association between A and B is the same at every level of C (homogeneous association). The saturated model, with every term, reproduces the data exactly; the aim is the simplest model that still explains the data well.
Model fit is judged with G² and χ² statistics comparing the observed counts with those the model expects; a large p value means the model is adequate. Two nested models are compared with ΔG². Under the hierarchy principle, if an interaction is in the model its lower-order terms stay in too. As the number of variables and levels grows the table becomes sparse, and many empty or small cells can make the estimates unstable.
When to use it
- To study the mutual associations of three or more categorical variables without treating any of them as the outcome (questions of conditional independence and homogeneous association).
- With only two variables the chi-square test of independence is enough; if a third variable is merely a confounder to control and two binary variables are of interest, the Cochran-Mantel-Haenszel test suffices.
- If one variable is clearly the outcome (e.g. disease yes/no), logistic or multinomial logistic regression is more directly interpretable; to summarise the association structure of categorical variables visually, correspondence analysis is used.
An everyday example
A secondary school records three things for 600 pupils: whether they eat breakfast regularly, whether they do sport at least three days a week, and whether their term average is high or low. Breakfast and achievement appear to be linked; but could this simply be because breakfast eaters also do more sport? A log-linear model tests whether breakfast and achievement are still associated when sport is held fixed, and whether that association is equally strong among pupils who do and do not do sport. If the model without the three-way interaction fits the data well, each pairwise association is equally strong at every level of the third variable.
Parameters in MerQur
Categorical factors
Concept: The categorical variables that form the crosstab; each combination of levels is a cell, and the model explains the counts in these cells. One level of each variable serves as the reference level; as variables and levels are added, the number of cells grows quickly.
Everyday example: In the school example three binary variables (breakfast, sport, achievement) form a table of 2 × 2 × 2 = 8 cells. Adding grade level with three levels as a fourth variable raises this to 24 cells, some of which may hold very few pupils.
What it does: Selects the categorical variables forming the contingency table; the cell counts are modelled with a Poisson generalized linear model. Levels are sorted (numeric ascending, string alphabetical), the first level is the baseline; unobserved level combinations stay in the table with count 0.
Effect on the result: The number of factors determines the table size, the possible interaction orders, the K-way tables, and the partial association list. With many factors the risk of sparse tables and boundary estimates increases (Agresti, 2013, 9.8).
On the validation data: Reversing the factor order changes only the coefficient names; none of the 80 common outputs changes. In hsb2, 4 factors make 8 of 36 cells zero; MerQur computes G² = 12.7906, df = 16 with the full grid and gives a zero-cell warning.
Frequency (count) column
Concept: When the data are a count table with one row per cell rather than raw data with one row per person, this column gives how many observations each cell holds. If it is not chosen, each row counts as one observation.
Everyday example: Instead of entering 600 pupils one by one, the school can prepare an 8-row table such as 'breakfast yes, sport yes, achievement high, 140'. With the count column chosen as the frequency, the model gives the same result as the 600-row raw data.
What it does: If the data are in count table form (each row a cell and a count), sets the column giving the cell count; if empty, each row is one observation.
Effect on the result: The cell count is the column sum. If one of the factors is also selected as the frequency column, the column is ignored (each row one observation) and a warning is written to the output. A non-numeric frequency column is an input error.
On the validation data: When the same drug data are run as a count table (8 rows) and as raw rows (2276 rows), all 110 outputs match. A numeric non-count column (hsb2 read) as weight gives n = 10446, G² = 97.0396.
Interaction degree
Concept: Sets how many variables the interactions in the model may involve. Main effects only is the model in which all variables are mutually independent; going up to two-way interactions gives the homogeneous association model, which says each pairwise association is the same at every level of the third variable; the highest degree is the saturated model.
Everyday example: At the school, degree 1 tests whether breakfast, sport and achievement are completely independent. Degree 2 tests the model 'breakfast and achievement are associated, but equally strongly among pupils who do and do not do sport'.
What it does: Sets the highest interaction order k in the model (1: mutual independence, 2: homogeneous association, m: saturated model).
Effect on the result: Determines the coefficient table, G²/χ² and df, the starting model of backward elimination, and the base of the custom model comparison. The K-way tables and partial association do not depend on k.
On the validation data: In the drug data k = 1: G² = 1286.02 (df 4; UCLA 1286.02), k = 2: 0.3740 (df 1; Thompson glm 0.3739859), k = 3: 0 (df 0). In the accident data k = 3: 1.3253 (df 1). In the five-way table k = 2: 15.3403 (df 16).
Cell delta
Concept: In the saturated model, empty (zero) cells drive some parameters off to infinity. Adding a small constant (e.g. 0.5) to every cell keeps the estimates finite; it is a pragmatic fix, though, and shifts the results slightly.
Everyday example: If no pupil at the school skips breakfast, skips sport and still has a high average, that cell is zero and some coefficients of the saturated model cannot be computed. Adding 0.5 to each cell makes the calculation possible by saying 'almost no pupils' in that cell.
What it does: Adds a constant (delta) to the cell counts to avoid boundary estimates in zero cells. Recommended only for parameter estimation in the saturated model with zero cells (Agresti, 2013, 10.6).
Effect on the result: Delta is applied only in the saturated model; in a non-saturated model no numeric output changes and the note "delta not applied: the selected model is not saturated" is written.
On the validation data: In the drug data with k = 2 and delta 0.5, G² 0.3740 and the partial association 91.6444 do not change. In the saturated k = 5 with delta 0.5, the boundary estimate becomes finite.
Backward elimination (hierarchical)
Concept: A selection method that starts from a complex model and at each step removes the term whose removal harms the fit least, arriving at a simpler model. Being hierarchical, it only removes an interaction after any higher-order terms containing it have been removed.
Everyday example: The school starts with the model containing every pairwise association. If the model still fits well after the sport-achievement association is removed, that term is dropped; if removing the breakfast-achievement association clearly worsens the fit, that term stays.
What it does: At each step removes the generator term with the largest ΔG² p value if it exceeds the elimination criterion.
Effect on the result: A step table and the final model (G², df, p) are added; the coefficient table stays at the selected degree model.
On the validation data: Drug k = 3: ACM is removed (ΔG² 0.3740, p = .541), final model AC, AM, CM. Accident k = 3: two terms are removed, final model G*L*S, L*S*I, G*I, G² = 3.5914.
Elimination criterion (p)
Concept: The p-value threshold for removing a term during backward elimination: if the p value of the loss of fit from dropping a term exceeds the threshold, the term is considered unnecessary. A smaller threshold leads to more terms being dropped, that is, a simpler model.
Everyday example: In a large sample such as the school's 600 pupils, even very small associations can be significant. Choosing 0.01 instead of 0.05 keeps only the really strong associations in the model and makes the result easier to interpret.
What it does: Sets the term removal criterion (removed if p > criterion); valid only when backward elimination is on. The default is .05; with large n a smaller criterion gives a more parsimonious model.
Effect on the result: The number of removed terms and the final model change.
On the validation data: Accident k = 3: with criterion 0.2 one term, with 0.05 two terms, with 0.001 three terms are removed; at 0.001 the final model G*L*S, G*I, L*I, S*I (G² = 7.4645, df 4) is the model chosen by Agresti/Thompson. In the drug data 0.5 and 0.01 do not change the result (ACM p = .541, the other terms p < 1e-20).
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets the width of the confidence intervals for the model coefficients (the λs). The fit tests and the estimates themselves do not change.
Everyday example: The 95% confidence interval for the breakfast-achievement coefficient shows the plausible range of that association's strength; at 99% the same interval widens and the school's reading becomes more cautious.
What it does: Sets the level of the Wald confidence intervals of the coefficients and λs.
Effect on the result: Only the interval limits; tests and estimates do not change.
On the validation data: Drug k = 2 marijuana coefficient: 80% [-5.9180, -4.7001], 90% [-6.0907, -4.5274], 95% [-6.2404, -4.3777], 99% [-6.5331, -4.0850], 99.9% [-6.8727, -3.7454] (Wald interval).
Custom model (generators)
Concept: Lets the researcher specify the model's terms directly to test a particular hypothesis. The model is written by listing its highest-order terms (generators); [AC][BC], for example, is the model in which A and B are conditionally independent given C.
Everyday example: To test the hypothesis 'once sport is known, breakfast and achievement are unrelated', the school fits the model [breakfast×sport][achievement×sport]. If it fits poorly, the breakfast-achievement association cannot be explained away by differences in sport.
What it does: Defines a hierarchical model with generators ("A*B, C" = [AB][C]); for hypotheses such as conditional independence and for tests that remove a single term from the homogeneous association model.
Effect on the result: A separate G², χ², df, p row and (if nested in the selected degree model) ΔG² are added; residuals and λ are computed for this model. An invalid variable name gives an explicit error; for a non-nested model a "not nested" note is written.
On the validation data: Drug model AM, CM: G² = 187.7543, df 2, ΔG² = 187.3803. Accident model GLS, GI, IL, IS: 7.4645, df 4.
Extra outputs: Cell frequencies and residuals
Concept: Shows the observed and model-expected counts in each cell and the residuals that scale the difference between them. Even when the overall fit is good, it reveals whether the model explains particular cells badly; adjusted residuals beyond about 2 in absolute value stand out.
Everyday example: The model may fit well overall, yet the 'no breakfast, does sport, high achievement' cell may hold far more pupils than expected. A large residual there draws the school's attention to a group of pupils worth looking into.
What it does: Gives observed and expected frequencies; raw, standardized (Pearson), adjusted, and deviance residuals. Serves to find poorly fitting cells (|adjusted residual| > 2).
Effect on the result: Extra tables and plots; the other outputs do not change.
On the validation data: Drug k = 2: adjusted residuals ±0.6333, expected 910.383 / 538.617 / .... In a zero cell the deviance residual is computed as -√(2·expected) (-0.3923).
Extra outputs: Deviation-coded λ
Concept: A coding that expresses coefficients relative to the average of all levels rather than to a reference level: the level effects within a term sum to zero. Each level's effect then reads as 'how far it departs from the average', and the result does not depend on which level is the reference.
Everyday example: If grade (9, 10, 11) were one of the school's variables, reference coding would describe the other grades relative to grade 9. Deviation coding instead describes each grade by how much it differs from the average of the three.
What it does: Gives the deviation (effect) coded λ parameters, whose level λs sum to zero in each term; for interpreting effects independently of the baseline level.
Effect on the result: An extra table (λ, SE, z, p, interval) and the constant λ0 are added.
On the validation data: Drug k = 2: λ_alcohol(0) = -1.5040 (SE 0.1138). HairEyeColor λ(Black, Brown) = 1.0004 (SE 0.1567).
Example on real data
Data: Agresti (1996) ICDA Table 6.3; UCLA OARC
Variables: factors alcohol, smoke, marijuana (alcohol, cigarette and marijuana use); frequency column count
- Question: Can the association among students' alcohol, cigarette, and marijuana use be explained by the homogeneous association model, which includes all two-way associations but excludes the three-way interaction?
- Model fit: G²(1) = 0.374, p = .541; Pearson χ²(1) = 0.401, p = .527. The model fits the data well; the three-way interaction is not needed (each two-way association is the same across the levels of the third variable).
- Result: All three two-way interactions are significant (all p < .001): alcohol x smoke λ = 2.05, alcohol x marijuana λ = 2.99, smoke x marijuana λ = 2.85.
- Effect size: The interaction coefficients are conditional log odds ratios: for example, among smokers the odds of marijuana use are exp(2.8479) = 17.25 times higher with alcohol level held fixed (alcohol x smoke exp(2.0545) = 7.80, alcohol x marijuana exp(2.9860) = 19.81; the ratios are computed from the unrounded coefficients).
- Additional finding: In the K-way tests the three-way effect is zero with G² = 0.374 (p = .541); the strongest partial association is smoke x marijuana (ΔG² = 497.0).
- Model comparison: AIC = 63.42, BIC = 63.97 (8 cells).
- APA: The homogeneous association log-linear model fitted the data well, G²(1) = 0.37, p = .541.
Related concepts
Stepwise variable selectionInteraction effectG-test (likelihood-ratio chi-square, G²)Haldane-Anscombe (zero-cell) correctionStandardized and adjusted residuals (contingency tables)Deviation (effect) codingFrequency weight (summary-table data)Goodness-of-fit tests (Pearson X² and deviance)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.