MerQuR+Statistics Glossary

Statistics Glossary → Multinomial Logistic Regression

Multinomial Logistic Regressionanalysis

A regression model that relates which of three or more unordered categories an outcome falls into (e.g. mode of transport, programme choice) to predictors, comparing each category with a reference category.

Concept

Some outcomes have more than two options with no natural order among them: car, bus or bicycle; academic, general or vocational programme. Studying such an outcome with several separate binary logistic models gives inconsistent probabilities. Multinomial logistic regression handles all categories in a single model, and for each observation the probabilities sum to 1.

One category is chosen as the reference and, for an outcome with K categories, K − 1 equations are fitted; each links the log-odds of one category versus the reference to the predictors: ln[P(k) / P(reference)] = β₀ₖ + β₁ₖx₁ + … Each predictor has a separate coefficient in each equation. exp(β) is an odds ratio: how many times the odds of being in that category rather than the reference change with a one-unit increase.

The model rests on the independence of irrelevant alternatives (IIA): the odds of one option versus another should not depend on whether some other option is available. When options are close substitutes, this assumption can fail. Many categories and predictors quickly multiply the number of parameters, and estimates become unstable for sparse categories.

When to use it

  • When the outcome has three or more unordered categories and you want to explain which one is chosen with predictors (mode of transport, preferred brand, species composition).
  • If the categories have a natural order (low-medium-high), ordinal logistic regression gives a more powerful model with fewer parameters; with only two categories, binary logistic regression is used.
  • If the IIA assumption is seriously violated, models such as nested logit or multinomial probit are considered.

An everyday example

In a city, 600 employees are asked how they get to work: car, bus or bicycle. Commuting distance and household income are also recorded. With 'bus' as the reference, the model might find OR = 1.5 for each additional 10,000 of income in the 'car versus bus' comparison, and OR = 0.7 for each additional kilometre in the 'bicycle versus bus' comparison: as income rises the odds of choosing the car go up, and as distance grows the odds of cycling go down.

Parameters in MerQur

Predictors (X) and Target (Y, multinomial)

Concept: The target is the outcome variable with at least three unordered categories; the predictors are used to explain which category occurs. For K categories, K − 1 equations are fitted.

Everyday example: In the commuting example the target is the 'car / bus / bicycle' column and the predictors are distance and income. With three categories, the model fits two equations: car versus bus and bicycle versus bus.

What it does: Selects an unordered dependent variable with at least 3 classes and the predictors. Classes are ordered by the text form of the values (numeric code 1/2/3 -> "1", "2", "3").

Effect on the result: There are K - 1 equations; if the baseline class is left empty, the first text class is the baseline.

On the validation data: prog (3 classes, n = 200), ICE_CREAM (1/2/3), Style (3), food (5 classes, n = 219).

Reference class

Concept: The baseline category the others are compared with. Changing the reference does not alter the model fit or the predicted probabilities; it only changes which comparison the coefficients describe.

Everyday example: If 'car' rather than 'bus' is the reference, the coefficients describe 'bus versus car' and 'bicycle versus car'. Each employee's predicted probabilities for the three modes stay the same.

What it does: Selects the denominator class. It does not change the model (LL, AIC, fit, probabilities and classification stay the same); it only re-expresses the coefficients as another comparison.

Effect on the result: Equation labels and coefficient/OR values change; LR tests, pseudo-R² and GOF do not. A single case-insensitive match is used with a warning; with no match a clear error lists the available classes.

On the validation data: write coefficient: baseline academic: general -0.05793, vocation -0.1136; baseline "general": academic +0.05793, vocation -0.05567. LL -179.9817 in every case.

→ Reference category (comparison level)

Categorical (factor) variables

Concept: Makes a numerically coded categorical predictor enter each equation with separate coefficients per level (dummy variables) rather than a single slope.

Everyday example: If the area an employee lives in is coded 1 = centre, 2 = suburbs, 3 = rural, these numbers are not measurements. As a factor, the question 'how does living in the suburbs change the odds of choosing the car compared with the centre?' is answered separately for each equation.

What it does: Turns numerically coded categorical predictors (e.g. school 1/2/3) into factors; the first (smallest) level is the baseline. Text columns are already categorical.

Effect on the result: Instead of a single continuous slope, (levels - 1) × (K - 1) dummy coefficients are formed; LL, LR and GOF patterns change.

On the validation data: School preference data: with the field empty School is treated as continuous (-2LL 670.404); with "School" -2LL 666.934, LR 28.4704 (df 6), AIC 682.934.

→ Dummy (indicator) variable coding

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets the confidence level of the intervals for the coefficients and odds ratios; the higher the level, the wider the interval.

Everyday example: If income has OR = 1.5 for 'car versus bus', its 95% interval might be [1.2, 1.9]; at a very high level such as 99.9% the interval may widen enough to include 1.

What it does: Sets the Wald interval for coefficients and ORs: b ± z·SE, OR interval = exp(limits).

Effect on the result: Only the interval columns change; estimate, SE and p do not.

On the validation data: general vs academic, write OR interval: 90% [0.9111, 0.9775]; 95% [0.9049, 0.9842]; 99% [0.8931, 0.9972]; 99.9% [0.8795, 1.0126] (covers 1).

→ Confidence level and confidence interval

Interactions

Concept: An interaction means that the effect of one predictor depends on the level of another; it is tested by adding the product of the two variables, which produces separate parameters in each equation.

Everyday example: The way distance discourages cycling may be weaker among low-income employees, who have limited access to a car. A 'distance × income' interaction tests whether this difference can be explained by chance.

What it does: Adds two-way product terms between selected predictors ("a*b"; "a:b" is treated the same). For a categorical variable a product is formed with each dummy.

Effect on the result: Extra parameters are added to each equation; effect tests are Type II. Under separation MerQur warns and shows no estimates.

On the validation data: ses*write: LL -179.9817 -> -178.2506, AIC 375.96 -> 380.50, Type II LR (ses:write) 3.4623, df 4, p = .484.

→ Interaction effect

Extra output: Likelihood-ratio tests per effect

Concept: Uses a likelihood ratio to test whether all of a predictor's coefficients across the equations are jointly zero, giving one answer to whether that variable affects the outcome overall.

Everyday example: Income being significant in the 'car versus bus' equation but not in 'bicycle versus bus' can be confusing. The per-effect LR test answers 'does income affect choice of transport overall?' with a single χ².

What it does: For each predictor, an LR test against the model with that predictor's parameters removed from all equations (df = number of columns × (K - 1)); Type II in a model with interactions.

Effect on the result: An extra table.

On the validation data: ses 11.0582 (df 4, p .0259), write 31.4469 (df 2).

→ Likelihood ratio test (LRT)

Extra output: Pearson / Deviance goodness of fit

Concept: Compares observed and model-expected category frequencies for groups of observations sharing the same combination of predictor values. With continuous predictors the groups become too small and the test is unreliable.

Everyday example: If the predictors are just a few categories such as 'area' and 'owns a car', each group contains dozens of employees and the test is meaningful. When distance is entered in kilometres, nearly every employee forms a group of their own and the test cannot be interpreted.

What it does: Compares observed and expected frequencies over covariate patterns with Pearson X² and G². Meaningful for grouped data; with a continuous predictor the chi-square approximation is not valid.

Effect on the result: Extra rows.

On the validation data: hsbdemo 64 patterns, Pearson 119.766, Deviance 129.875, df 120 (sparseness warning); school 6 patterns, df 6, Pearson 5.171.

→ Goodness-of-fit tests (Pearson X² and deviance)

Extra output: Classification table

Concept: Assigns each observation to the category with the highest predicted probability and cross-tabulates observed against predicted categories. The overall percentage correct can hide poor performance in sparse categories.

Everyday example: Even if the model classifies 70% of employees correctly, this may come mostly from car users, the largest group. The table may show that only 20% of cyclists are predicted correctly.

What it does: Assigns each observation to the class with the highest predicted probability; gives the observed × predicted table and % correct.

Effect on the result: An extra table.

On the validation data: hsbdemo 61.0% (academic 87.6%, general 15.6%, vocation 46.0%).

→ Classification table (confusion matrix)

Extra output: IIA test (Hausman-McFadden)

Concept: Tests the independence of irrelevant alternatives: when one category is removed from the analysis, the coefficients for the remaining comparisons should not change markedly. A marked change indicates the assumption is violated.

Everyday example: Imagine a metro line is added to the city. If most metro riders come from the bus, the odds of car versus bus change; the options are substitutes and IIA fails. The test catches such problems by checking whether the car-bus coefficients change when, say, the bicycle category is removed.

What it does: Tests independence of irrelevant alternatives: when one alternative is dropped, the remaining coefficients should not change. H = d'(V_r - V_f)⁻¹d.

Effect on the result: Extra rows. If (V_r - V_f) is not positive definite, a "test may be invalid" warning is written.

On the validation data: hsbdemo dropping general: -1.0763 (negative; warning), dropping vocation: 0.5438 (p .969). An independent high-precision computation matches MerQur to 1e-8.

→ Independence of irrelevant alternatives (IIA)

Variable selection method (Enter (all), Forward (entry only), Backward (removal only), Forward stepwise, Backward stepwise), Entry test (Likelihood ratio (LR), Score), Removal test (Likelihood ratio (LR), Wald), PIN, POUT, Always in the model; Overdispersion scale (None (scale = 1), Pearson, Deviance); Frequency weight

Concept: Stepwise selection adds or removes predictors automatically by statistical criteria and is exploratory. An overdispersion scale carries extra variability in the data into the standard errors; a frequency weight states that a row represents several identical observations.

Everyday example: With many candidate variables such as age, sex, income, distance and number of children, stepwise selection suggests a small set that best explains commuting choice; this suggestion should be checked in a new survey. If the data are summary rows such as 'centre, car: 120 people', a frequency weight is used.

What it does: Stepwise selection for the multinomial model (entry by LR or score, removal by LR or Wald), an overdispersion scale and a frequency weight.

Effect on the result: The selected set and the step table; the scale changes the SEs; the weight changes n and the estimates.

On the validation data: The step table gives -2LL, chi-square, df and the selected set at each step; the frequency weight was tested on the housing data.

→ Stepwise variable selection

Example on real data

Data: UCLA hsbdemo (programme choice, 200 students)

Variables: target prog (academic, general, vocation), predictors ses (socioeconomic status) and write (writing score)

  • Question: Can a student's high school programme type (academic, general, vocation) be explained by socioeconomic status (ses) and writing score (write)? Baseline class academic, ses baseline level high.
  • Assumption: Multinomial logit relies on the independence of irrelevant alternatives (IIA); this assumption should be checked separately with the Hausman-McFadden test.
  • Result: LR χ²(6) = 48.23, p < .001. general vs academic: the odds of being in general are 3.20 times higher for low-ses than for high-ses students (95% CI [1.17, 8.76], p = .024); each 1-point increase in write multiplies these odds by 0.944 (p = .007). vocation vs academic: OR = 3.58 for middle ses (95% CI [1.31, 9.74], p = .013); OR = 0.893 for write (95% CI [0.855, 0.932], p < .001).
  • Effect size: McFadden R² = .118, Cox-Snell R² = .214, Nagelkerke R² = .246; AIC = 375.96, BIC = 402.35.
  • APA: χ²(6, N = 200) = 48.23, p < .001, McFadden R² = .12; write (vocation/academic) OR = 0.89, 95% CI [0.86, 0.93], p < .001.

Related concepts

Likelihood ratio test (LRT)Independence of irrelevant alternatives (IIA)Classification table (confusion matrix)Goodness-of-fit tests (Pearson X² and deviance)Interaction effectStepwise variable selectionOdds ratio (OR)Pseudo-R² (McFadden, Cox-Snell, Nagelkerke)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.