MerQuR+Statistics Glossary

Statistics Glossary → Logistic Regression

Logistic Regressionanalysis

A regression model that predicts the probability of a two-category (yes/no) outcome from one or more predictors and expresses the effect of each predictor as an odds ratio (OR).

Concept

In many research questions the outcome is not a measurement but a state: did the patient recover, was the application accepted, did the customer buy? For such binary outcomes ordinary linear regression can produce 'probabilities' below 0 or above 1, and its error assumptions do not hold. Logistic regression models the probability directly and always keeps the predictions between 0 and 1.

Instead of the probability itself, the model links the logarithm of its odds (the logit) to a linear combination of the predictors: ln[p / (1 − p)] = β₀ + β₁x₁ + β₂x₂ + … The coefficients are estimated by maximum likelihood. The exponential of each coefficient, exp(β), is an odds ratio: OR = 2 means that, holding the other predictors fixed, a one-unit increase in that variable doubles the odds of the event; OR = 0.5 halves them. An odds ratio is not a ratio of probabilities; when the event is common the two diverge noticeably.

The assumptions are that observations are independent, that continuous predictors are linearly related to the logit and that there is no severe multicollinearity among predictors. With few events and many predictors, or when a predictor perfectly separates the outcome (separation), the estimates become unstable. Explanatory power is judged by pseudo-R², discrimination by the AUC and goodness of fit by the Hosmer-Lemeshow test; none of these replaces the others.

When to use it

  • When the outcome has two categories (ill/healthy, accepted/rejected, pass/fail) and you want to explain its probability with several factors.
  • If the outcome has three or more unordered categories use multinomial logistic regression; if the categories are ordered, use ordinal logistic regression.
  • If events are very rare or a predictor perfectly separates the outcome, use Firth-corrected logistic regression; if the time until the event matters, survival models such as Cox regression are preferred.

An everyday example

A hospital wants to identify in advance which patients will develop a wound infection after surgery. For 500 patients it has recorded age, length of the operation, whether the patient has diabetes and the ward they stayed on; the outcome is infection (1) or no infection (0). Logistic regression might find OR = 2.4 for diabetes: with age, duration and ward held fixed, the odds of infection for diabetic patients are about 2.4 times those of non-diabetic patients. The model also predicts, for every new patient, an infection probability between 0 and 1.

Parameters in MerQur

Target column

Concept: The outcome variable to be modelled; in logistic regression it can take only two values (event occurred / did not occur).

Everyday example: In the hospital example the target column is the 'infection' column, which holds 1 (developed) or 0 (did not develop) for each patient. If it contains a third value such as 'suspected', the model cannot be fitted.

What it does: Selects the binary dependent variable; if there is a value other than the two classes, the analysis stops.

Effect on the result: Together with the event coding, it sets the sign of all coefficients.

On the validation data: In binary.csv, 400 observations, 127 events.

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets how confidently the intervals for the coefficients and odds ratios are constructed; the higher the confidence level, the wider the interval.

Everyday example: If diabetes has OR = 2.4, its 95% confidence interval might be [1.5, 3.8]; at 99% it widens to something like [1.3, 4.4]. As long as the interval excludes 1, diabetes can be said to raise the risk.

What it does: Sets the Wald confidence interval of the coefficients (b ± z·SE) and the OR interval, its exponential transformation.

Effect on the result: Only the OR interval columns change; the coefficient, SE, z and p do not. The MerQur interval is a Wald interval (gpa 95% Wald 0.1537-1.4544; the profile likelihood interval would be 0.1603-1.4641).

On the validation data: gpa (OR = 2.2345): 90% [1.2947, 3.8568]; 95% [1.1661, 4.2819]; 99% [0.9506, 5.2527].

→ Confidence level and confidence interval

Extra output: Pseudo-R² (CS/Nagelkerke)

Concept: Approximate counterparts of R² in logistic models; they summarise, on a 0 to 1 scale, how much the model improves on an intercept-only model. Their values are usually lower than R² in linear regression and should not be read on the same scale.

Everyday example: If the infection model has Nagelkerke R² = 0.15, age, duration, diabetes and ward help explain infection, but many of its causes remain outside the model.

What it does: Writes the McFadden (1 - LL/LL₀), Cox-Snell (1 - exp(-LR/n)) and Nagelkerke (Cox-Snell/upper bound) R² values.

Effect on the result: Only an additional text row; the model does not change. With Firth on, Nagelkerke and Cox-Snell use the penalized LR.

On the validation data: binary.csv McFadden 0.0829, Cox-Snell 0.0985, Nagelkerke 0.1380. With Firth the endometrial Nagelkerke is 0.6181.

→ Pseudo-R² (McFadden, Cox-Snell, Nagelkerke)

Extra output: Classification + AUC and Classification cutoff

Concept: Counts correct and incorrect classifications by treating observations whose predicted probability exceeds a cutoff as 'events'. The AUC is the probability that a randomly chosen event receives a higher predicted probability than a randomly chosen non-event; it does not depend on the cutoff.

Everyday example: With a cutoff of 0.5, many real infections with predicted probabilities below 0.5 may be missed. If the hospital does not want to miss infections it lowers the cutoff to 0.2; more patients then receive precautions, but false alarms also increase.

What it does: Counts observations with p ≥ cutoff as events; gives the 2x2 table, accuracy, sensitivity, specificity, PPV, NPV and the area under the ROC curve (AUC; independent of the cutoff).

Effect on the result: The cutoff changes only the classification table (and the saved group label); the coefficients and AUC do not change. With a low event rate a 0.5 cutoff lowers sensitivity.

On the validation data: binary.csv AUC = 0.6928. Cutoff 0.50: TN 254, FP 19, FN 97, TP 30 (accuracy 0.71); 0.30: 161/112/42/85; 0.3175 (event rate): 174/99/46/81.

→ Classification table (confusion matrix)

Extra output: Hosmer-Lemeshow

Concept: A goodness-of-fit test of whether the probabilities predicted by the model agree with the observed event frequencies (calibration). A non-significant result means there is no evidence of poor fit; it does not prove good fit.

Everyday example: Among 50 patients for whom the model predicts an infection probability of about 30%, about 15 infections are expected. If 14 or 16 occur the fit is good; if 30 occur, the model underestimates the risk in this group.

What it does: Divides the observations into about 10 groups by predicted probability and compares observed and expected event counts with χ² (goodness of fit; a large p does not indicate poor fit).

Effect on the result: An additional row; the model does not change.

On the validation data: binary.csv χ² = 11.085, df 8, p = 0.1969. Remission: χ² = 4.183, df 8, p = 0.8403.

→ Hosmer-Lemeshow goodness-of-fit test

Extra output: VIF

Concept: The variance inflation factor shows how much a predictor overlaps with the other predictors; the larger it is, the more that coefficient's standard error is inflated and the harder it is to separate the effects.

Everyday example: If length of surgery and length of anaesthesia are both entered, they carry almost the same information and their VIFs are high. It then becomes impossible to tell which one affects infection; usually one of them is dropped.

What it does: Computes VIF = 1/(1 - R²) from the OLS regression of each design column on the others.

Effect on the result: An additional table.

On the validation data: binary.csv gre 1.19, gpa 1.19, rank[2] 2.18, rank[3] 2.10, rank[4] 1.77; birthwt the largest of 8 predictors 1.35.

→ Multicollinearity (VIF, tolerance, condition index)

Event category

Concept: Determines which of the two classes is modelled as the 'event' (Y = 1). Switching the event class flips the signs of the coefficients and inverts the odds ratios (1/OR instead of OR); the model fit is unchanged.

Everyday example: If the event is 'infection developed', diabetes has OR = 2.4; if it is 'no infection', the same relationship appears as OR = 1/2.4 ≈ 0.42. Both describe the same finding from opposite sides.

What it does: Sets the class modeled as Y = 1. If empty, the second in the ordering of the classes is taken (numeric order for numeric codes; larger code = event).

Effect on the result: When the event changes, the sign of all coefficients flips and the ORs are inverted; LR, pseudo-R² and AUC do not change. An invalid event gives an explicit error.

On the validation data: binary.csv event "0": gpa -0.80404, OR interval [0.2335, 0.8575]. Neuralgia event "No"; with vaso event "constrict" LogRate 4.5617, LogVolume 5.1793. In a target coded 9/10 the event is 10.

Reference category (factors) (First level, Last level)

Concept: Determines which level of a categorical predictor the other levels are compared with; each dummy coefficient shows the difference of that level from the reference level.

Everyday example: If 'general surgery' is the reference level of the ward variable, the ORs for the orthopaedic and urology wards are read relative to general surgery. Changing the reference leaves the model's predictions unchanged; only the starting point of the comparisons changes.

What it does: Selects the level left out in the dummy coding of categorical predictors.

Effect on the result: The meaning of the dummy coefficients changes; model fit, LR and predictions stay the same.

On the validation data: binary.csv first level: rank[2] -0.6754, rank[3] -1.3402, rank[4] -1.5515; last level: rank[1] 1.5515, rank[2] 0.8760, rank[3] 0.2113, constant -5.5414.

→ Reference category (comparison level)

Variable selection method (Enter (all), Forward stepwise (LR), Forward stepwise (Wald), Backward stepwise (LR), Backward stepwise (Wald)) with PIN and POUT

Concept: Stepwise methods that automatically add predictors to or remove them from the model according to statistical criteria. They are exploratory; the p values and coefficients of the selected model are optimistic and should be validated on another sample.

Everyday example: Suppose the hospital has 20 candidate risk factors. Forward stepwise selection adds them one at a time, starting with the most useful, and might stop at diabetes, duration and age. This is a hint about which factors may matter, not conclusive evidence.

What it does: Performs forward (FSTEP) or backward (BSTEP) stepwise selection: entry by the Rao score test p < PIN, removal by the LR or Wald test p > POUT. It is exploratory; p-values after selection are optimistic (Harrell, 2015).

Effect on the result: Changes the terms in the final model and all outputs; PIN and POUT act only in selection. With PIN > POUT it runs with a warning. The output states the criterion as "entry: score χ², removal: LR/Wald χ²".

On the validation data: binary.csv: all four methods select three terms (rank score 25.242). birthwt: forward -> ptl, ht, lwt (LR χ² 18.71); backward -> lwt, race, smoke, ht, ui (30.46). Remission forward Wald PIN .30 / POUT .35 -> li, temp, cell.

→ Stepwise variable selection

Hierarchical blocks

Concept: Predictors are added in blocks, in an order chosen by the researcher; the contribution of each block beyond the earlier blocks is tested separately.

Everyday example: First patient factors such as age and diabetes might be entered, then surgical factors such as duration and ward. A significant contribution of the second block shows that surgical conditions are related to infection even after patient characteristics are taken into account.

What it does: Adds terms in blocks in sequence; for each block gives -2LL, the block χ² (against the previous one) and the model χ². If given together with a selection method, selection is not applied and a warning is written.

Effect on the result: The final model equals the one with all terms; the additional table gives the block contribution. An unknown name gives an explicit error.

On the validation data: binary.csv "gre, gpa | rank": Block 1 -2LL 480.344, χ² 19.633 (df 2); Block 2 -2LL 458.517, χ² 21.826 (df 3), model χ² 41.459 (df 5).

→ Hierarchical (blockwise) regression

Categorical (factor) variables

Concept: Makes a variable that is coded with numbers but is really categorical enter the model with a separate coefficient for each level (dummy variables) rather than a single slope.

Everyday example: If wards are coded 1, 2, 3, the model may treat these numbers as measurements and impose a meaningless linear trend across ward codes. Declared as a factor, each ward gets its own coefficient.

What it does: Expands numerically coded columns into dummy variables; text columns are expanded automatically.

Effect on the result: Gives a coefficient per level instead of a single slope; the degrees of freedom increase.

On the validation data: binary.csv rank numeric: LR χ² 40.535 (df 3), gpa 0.7770; factor: 41.459 (df 5), gpa 0.8040. Contraception district (60 levels) as a factor: quasi-separation warning.

→ Dummy (indicator) variable coding

Interaction terms

Concept: An interaction means that the effect of one predictor changes with the level of another; it is tested by adding the product of the two variables to the model.

Everyday example: Whether a long operation raises infection risk more in diabetic than in non-diabetic patients is examined with a 'duration × diabetes' interaction. If it is significant, describing the effect of duration with a single number regardless of diabetes would be misleading.

What it does: "a*b" adds the main effects and the product, "a:b" only the product.

Effect on the result: Product columns are added; the main-effect coefficients take a conditional meaning. In a model with an interaction the Type III test of a main effect uses effect coding.

On the validation data: gre*gpa: LR χ² 44.205; gpa*rank: 41.8645; Neuralgia Treatment*Sex Wald 0.1412; Neuralgia Treatment Type III 8.16. A product-only "gre:gpa" model can also be fitted.

→ Interaction effect

Standard error type (Classic (ML information matrix), HC0, HC1)

Concept: Robust (sandwich) standard errors measure the uncertainty of the coefficients more reliably when the model assumptions do not hold exactly; they change only the standard errors and p values, not the coefficients.

Everyday example: If the model form does not fit some patient groups perfectly, classical standard errors may make the diabetes effect look more precise than it is. A robust standard error gives a more cautious interval in that case.

What it does: Estimates a sandwich (robust) standard error: HC0 or HC1 = HC0·n/(n - k).

Effect on the result: SE, z, p and the OR interval change; the coefficients stay the same. With Firth on, this option is ignored and reported with an explicit warning.

On the validation data: binary.csv gpa SE 0.3318 -> HC0 0.3447, HC1 0.3473. Contraception age SE 0.00754 -> HC0 0.00727, HC1 0.00728.

→ Robust (sandwich) standard errors (HC0-HC3)

Cluster variable (cluster-robust SE)

Concept: A standard error that accounts for observations in the same cluster (school, hospital, district) resembling one another. If this similarity is ignored, the data seem to contain more information than they do and standard errors come out too small; the method needs a reasonable number of clusters.

Everyday example: If the data come from 40 hospitals, patients in the same hospital share operating theatres and hygiene practices. Using hospital as the cluster variable carries this similarity into the standard errors; with only 3 or 4 hospitals the method is unreliable.

What it does: Computes an SE robust to within-cluster dependence: sandwich with score sums within clusters, HC0·G/(G - 1). It should not be used with few clusters (Cameron and Miller, 2015).

Effect on the result: SE, z, p and the interval change; the coefficients are the same. With SE type HC1 a small-sample correction by the number of clusters is applied.

On the validation data: Contraception (G = 60 districts): urban SE 0.1052 -> 0.1890, age 0.00754 -> 0.00686. With few clusters (binary.csv rank, G = 4) the SEs shrink meaninglessly (gpa 0.118).

→ Cluster-robust (clustered) standard errors

Extra output (2): Omnibus + Type III (LR/Wald)

Concept: The omnibus test asks whether all predictors together contribute to the model. Type III tests assess each term as a whole with the others in the model, giving one overall test for a categorical variable with many levels.

Everyday example: The omnibus test answers 'do these four factors together explain infection?'. The Type III test answers 'does the ward variable, with all its levels, add anything beyond the other factors?' with a single χ².

What it does: Gives the model χ² (-2LL difference) and the per-term LR (term removed) and Wald χ² tests; adds a Wald χ² column to the coefficient table.

Effect on the result: An additional table; with Firth on it is produced with the penalized likelihood.

On the validation data: binary.csv model χ²(5) = 41.459, -2LL 458.5175 (constant only 499.9765); Type III LR gre 4.358, gpa 6.014, rank 21.827; Wald 4.284, 5.872, 20.895.

→ Likelihood ratio test (LRT)

Extra output (2): Influence diagnostics

Concept: Diagnostics showing how badly individual observations fit the model (residuals) and how strongly they affect the results (leverage, Cook's D, DFBETA).

Everyday example: A patient who developed an infection despite a very low predicted risk gives a large residual. If removing this patient changes the OR for diabetes noticeably, their Cook's D is high and the record should be checked.

What it does: Gives leverage h, Pearson residual (ZRESID), deviance residual, Cook D and DFBETA; a |ZRESID| > 2 list is produced.

Effect on the result: Additional tables.

On the validation data: vaso: the largest Cook D 0.4291 (observation 4), then 18; h, Pearson and deviance residuals are given for 39 rows.

→ Influence diagnostics (Cook's distance, leverage, DFBETAS)

Extra output (2): Average marginal effects

Concept: Gives the effect on the probability scale instead of as an odds ratio: how many points, on average, a one-unit change in a predictor shifts the probability of the event.

Everyday example: OR = 2.4 says 'the odds are 2.4 times higher', but for a hospital manager 'diabetes raises the probability of infection by about 9 percentage points on average' is easier to grasp. The average marginal effect gives this second statement.

What it does: Gives the average effect on the probability scale (continuous: average derivative; dummy: 1-0 discrete difference), with the SE by the delta method.

Effect on the result: An additional table; not computed in a model with an interaction, with a warning.

On the validation data: binary.csv gpa 0.1564 (SE 0.0630), rank[4] -0.3212; Contraception urban 0.1845.

→ Average marginal effect (AME)

Box-Tidwell

Concept: Tests the assumption that a continuous predictor is linearly related to the logit; a significant result suggests the relationship may be curved.

Everyday example: If length of surgery is thought to raise risk little in the first hours and much faster later, the Box-Tidwell test checks whether the relationship between duration and the logit is a straight line.

What it does: Tests the linearity of the logit by adding an x·ln(x) term for each continuous positive predictor (Hosmer and Lemeshow, 2000, 4.2).

Effect on the result: An additional table; variables with zero/negative values are skipped and this is written.

On the validation data: binary.csv gre p = 0.721, gpa p = 0.709; birthwt age p = 0.329, lwt p = 0.415; vaso Rate p = 0.435, Volume p = 0.870.

→ Box-Tidwell test (linearity of the logit)

Firth correction

Concept: An estimation method that adds a small penalty to the likelihood, reducing small-sample bias and giving finite coefficients even when a predictor perfectly separates the outcome.

Everyday example: If, in a small study, every patient who did not receive a certain antibiotic developed an infection and none who received it did, the ordinary model tries to produce a coefficient heading to infinity. The Firth correction gives a reasonable, interpretable estimate in this situation.

What it does: Applies the penalized likelihood with the Jeffreys prior (Firth, 1993); gives finite estimates under separation.

Effect on the result: Coefficients and SE; the robust SE options are ignored (explicit warning); Nagelkerke/Cox-Snell use the penalized LR.

On the validation data: endometrial NV: ML 19.50 (SE 5458, non-converged) -> Firth 2.9293 (SE 1.4650); the coefficients are 3.77456, 2.92927, -0.03475, -2.60416. Penalized LR χ²(3) = 43.656.

→ Firth correction (penalized likelihood)

Save to dataset

Concept: Adds each observation's predicted probability, predicted class and diagnostic values to the dataset as new columns, so they can be used in later analyses.

Everyday example: If each patient's predicted infection probability is added to the table, nurses can list high-risk patients, or a separate ROC analysis can be run on these probabilities.

What it does: Adds the PRE (probability), PGR (predicted group), COO, LEV, ZRE, DEV columns to the dataset.

Effect on the result: Columns are added to the dataset; PGR follows the cutoff (197 observations are events at a 0.3 cutoff).

On the validation data: The PRE, LEV, ZRE, DEV and COO columns are saved; COO uses the Cook D definition.

Overdispersion scale (None (scale = 1), Pearson, Deviance), Frequency weight, Custom contrast (L)

Concept: Overdispersion means the variability in the data exceeds what the model expects, and it is handled by scaling the standard errors. A frequency weight states that a row represents several identical observations; a custom contrast tests a researcher-defined combination of coefficients.

Everyday example: If the data are stored as summary rows such as 'diabetic, long operation: 40 patients, 12 infections' rather than one row per patient, a frequency weight conveys these counts. A custom contrast tests a specific question such as 'are the effects of the orthopaedic and urology wards equal?'.

What it does: The overdispersion scale (Pearson or deviance) scales the standard errors; the frequency weight defines data in count form; the custom contrast tests a linear combination of the coefficients (L matrix, right-hand-side value).

Effect on the result: SE, test statistics and contrast results; the coefficient estimates do not change (except for the weight).

On the validation data: The overdispersion scale is computed at the observation level (birthwt: df 186). The frequency weight and a custom contrast (right-hand side 0.5) are also supported.

→ Overdispersion and the scale parameter

Example on real data

Data: UCLA binary (graduate admission, 400 applicants)

Variables: target admit (admission 0/1), independents gre, gpa and rank (school rank) as a factor

  • Question: Can graduate admission (admit = 1) be explained by the GRE score, grade point average (GPA) and the prestige rank of the undergraduate school (rank, 1 = highest)?
  • Model test: LR χ²(5) = 41.46, p < .001; together the predictors significantly explain the probability of admission.
  • Coefficients: A 1-point increase in GPA multiplies the odds of admission by 2.23 (OR = 2.2345, 95% CI [1.1661, 4.2819], p = .015); the GRE effect is small but significant (OR = 1.0023 per point, p = .039). Compared with rank 1, applicants from rank 2 (OR = 0.509), rank 3 (OR = 0.262) and rank 4 (OR = 0.212) schools have significantly lower odds of admission.
  • Model fit: McFadden R² = .083, Cox-Snell R² = .098, Nagelkerke R² = .138; AIC = 470.52, BIC = 494.47. The explanatory power is limited.
  • Classification: At the 0.5 cutoff accuracy is 71.0%, sensitivity 23.6%, specificity 93.0%; AUC = 0.693. The model identifies the non-admitted well and the admitted poorly.
  • Limitation: With an event rate of 32%, the 0.5 cutoff lowers sensitivity; the cutoff should be chosen from the ROC curve according to the purpose.
  • APA: χ²(5, N = 400) = 41.46, p < .001, McFadden R² = .08; GPA OR = 2.23, 95% CI [1.17, 4.28], p = .015.

Related concepts

Likelihood ratio test (LRT)Pseudo-R² (McFadden, Cox-Snell, Nagelkerke)Firth correction (penalized likelihood)Hosmer-Lemeshow goodness-of-fit testAverage marginal effect (AME)Box-Tidwell test (linearity of the logit)Classification table (confusion matrix)Hierarchical (blockwise) regression

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.