Statistics Glossary → Multiple Linear Regression
Multiple Linear Regressionanalysis
A method that models how a continuous outcome relates to several predictors, estimating the average change in the outcome associated with each predictor while the others are held constant.
Concept
An outcome (e.g. house price) is usually shaped by several factors together, not one. Simple correlation looks at each factor separately and can mislead when the factors are related to one another: large homes may tend to have more rooms and to be further from the centre. Multiple linear regression puts all the predictors into one model and estimates each one's relationship with the outcome while the others are held constant.
The model writes the outcome as a linear combination of the predictors: y = b₀ + b₁x₁ + b₂x₂ + … + e. The coefficients are estimated so as to minimise the sum of squared differences between observed and predicted values (the residuals), the least-squares method. b₁ is the average change in y for a one-unit increase in x₁ with the other predictors held constant. R² gives the share of variation in the outcome explained by the model; the F test shows whether the model as a whole is significant; each coefficient's t test and confidence interval show that predictor's contribution. Categorical predictors are modelled with dummy variables, and an effect that changes with another variable is modelled with interaction terms.
The model rests on assumptions: the relationship is linear, the residuals are independent, their variance is constant and (for inference in small samples) they are roughly normal. If predictors are very strongly related to each other (collinearity), the coefficients become unstable and their standard errors grow. A few influential observations can change the results noticeably. Regression coefficients show association; they show causation only if the design (e.g. randomisation) supports it. An important variable left out of the model can bias the coefficients.
When to use it
- To examine how a continuous outcome relates to several predictors, to see each predictor's contribution independently of the others, or to predict the outcome.
- If the outcome is binary use logistic regression; if it is a count, Poisson or negative binomial regression; with repeated measurements on the same people or clustered data, use mixed-effects models.
- With many predictors and strong collinearity among them, regularised regression such as ridge or lasso is more suitable; if the relationship is clearly non-linear, polynomial terms or non-linear models are.
An everyday example
An estate agency wants to explain the prices of 200 flats for sale using floor area, number of rooms, building age and distance from the centre. Multiple regression might find b = 15,000 TL for floor area: with the other features the same, each extra square metre adds 15,000 TL to the price on average. If the coefficient for building age is −20,000 TL, a flat of the same size and location loses 20,000 TL on average for each additional year of age. R² = 0.72 shows that about 72% of the variation in prices is explained by these four features; the rest comes from factors not in the model, such as the view, the floor and the flat's condition.
Parameters in MerQur
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets how 'sure' the confidence intervals for the coefficients and predictions are. A higher confidence level means wider intervals; the coefficients, standard errors and p values do not change.
Everyday example: With a floor-area coefficient of 15,000 TL, the 95% confidence interval might run from about 12,000 to 18,000 TL; at 99% it widens to something like 11,000 to 19,000 TL.
What it does: Sets the level (1 - alpha) of the coefficient confidence intervals and prediction intervals; the critical value is t(1 - alpha/2, n - p). The custom level is in the range 80.0-99.9.
Effect on the result: Only the interval limits change; the coefficient, SE, t and p stay the same.
On the validation data: hsb2 female -2.0098 (SE 1.0227, p = .0508): 95% [-4.0268, 0.0072]; 90% [-3.7000, -0.3195]; 99% [-4.6701, 0.6506]. At 90% the interval excludes zero; at 95% and 99% it includes it.
Standard error type (Classic (OLS), HC0, HC1, HC2, HC3)
Concept: Classical standard errors assume that the variance of the residuals is the same for all observations. Robust (HC) standard errors remain valid when this assumption fails, that is, under heteroscedasticity; they do not change the coefficients, only the estimates of their uncertainty. HC3 is recommended in small samples.
Everyday example: If price predictions for cheap flats are off by tens of thousands of TL but for luxury flats by hundreds of thousands, the residual variance is not constant. HC3 standard errors then give more honest confidence intervals for the coefficients.
What it does: Estimates a heteroskedasticity-robust (sandwich) covariance: HC0 White (1980), HC1 = HC0·n/(n - p), HC2 e²/(1 - h), HC3 e²/(1 - h)² (MacKinnon and White, 1985; Long and Ervin, 2000). HC3 is recommended at small n.
Effect on the result: Coefficients do not change; SE, t, p, interval and the overall F (Wald F with the sandwich covariance) change.
On the validation data: elemapi2 acs_k3 SE: classic 4.3711, HC0 4.5913, HC1 4.6206, HC3 4.7154. Overall F: classic 61.01, HC1 Wald F 84.6749. hsb2 female p: classic .0508, HC3 .0592.
Standardized coefficients (β)
Concept: The standardised coefficient (β) shows how many standard deviations the outcome changes for a one-standard-deviation increase in the predictor. It allows the relative influence of predictors measured in different units to be compared; when predictors are correlated, though, this comparison needs care.
Everyday example: Floor area is measured in square metres and distance from the centre in kilometres, so the raw coefficients cannot be compared directly. If β is 0.60 for floor area and −0.25 for distance, floor area is clearly more strongly related to price than distance is.
What it does: β_j = b_j·s_xj/s_y; compares the relative effects of the predictors independent of unit.
Effect on the result: Adds a "β (std)" column to the table. In WLS the weighted SD is used.
On the validation data: elemapi2 meals β = -0.66070, ell -0.14958; hsb2 math .368. census9 WLS medage β = 0.6852.
Variable selection method (Enter (all), Stepwise, Forward, Backward) with PIN and POUT
Concept: Methods that decide on a statistical criterion which predictors enter the model: forward selection adds the most useful terms one at a time, backward elimination removes the least useful one at a time, and stepwise selection combines the two. They are exploratory; in the selected model the p values and R² are optimistic, and the model may not perform as well on new data.
Everyday example: The estate agency might use stepwise selection to find which of 15 features (balcony, parking, lift, floor…) best explain price. But the 5 features selected may also reflect chance quirks of these 200 flats, so the result should be tried out on new listings.
What it does: Selection uses the p-value of the partial F test: a term enters if p is below PIN and leaves if above POUT. If POUT < PIN is given, POUT = 1.1·PIN is taken with a warning. It is exploratory; p-values after selection are optimistic (Harrell, 2015).
Effect on the result: Changes the final model, the step table and all model statistics; a dummy group is a single term. No selection with WLS or the no-intercept model (error message). In Enter, PIN and POUT have no effect.
On the validation data: Fitness data: forward (PIN .50) RunTime, Age, RunPulse, MaxPulse, Weight; step F 84.01, 2.48, 6.70, 4.10, 1.84. Backward (POUT .10) removes RestPulse and Weight, R² = .8368. Selection was also tested on mtcars and the Hald cement data.
Hierarchical blocks
Concept: Adds predictors to the model in groups (blocks) in an order set by the researcher and tests each block's additional contribution to the explained variance (ΔR²). The order comes from theory rather than a statistical criterion.
Everyday example: The agency first enters the flats' physical features (floor area, rooms, age), then location information (distance from centre, district). A ΔR² of 0.15 means that location explains an extra 15% of the variation in price after the physical features are accounted for.
What it does: Divides the variables into blocks entered in sequence (blocks separated by "|", variables by ","); for each block gives R², ΔR², F change and p. Tests the contribution of variable groups added in a theoretical order.
Effect on the result: A block table is added. An unknown name gives an error; if given together with a selection method, the selection is ignored with a warning.
On the validation data: elemapi2, 8 variables | ell: block 2 ΔR² = .00673, F change 16.673 (1, 385), R² .845. hsb2 "math, female | socst": ΔR² = .0239, F = 8.26.
Categorical (factor) variables
Concept: Enters a categorical predictor (e.g. district) into the model not as a numeric variable but through a separate indicator (dummy) variable for each level. A variable with k levels is represented by k − 1 dummies; each coefficient is that level's average difference from the reference level.
Everyday example: If districts are coded 1, 2, 3 and entered as a numeric variable, the model makes the meaningless assumption that district 3 has twice the effect of district 1. Entered as a factor, each district's price difference is estimated separately.
What it does: Expands a numerically coded column into k - 1 indicator (dummy) variables; text columns are expanded automatically. Levels are ordered from smallest to largest (alphabetically in text).
Effect on the result: The coefficients are interpreted relative to the dummy levels.
On the validation data: elemapi2 mealcat (omitted level is the first level): constant 805.718, mealcat[2] -166.324, mealcat[3] -301.338. In census9 region the omitted level is 1 = NE.
Interaction terms
Concept: Models whether the effect of one predictor on the outcome changes with the level of another predictor. Once an interaction is in the model, the main effects are no longer 'overall effects' but conditional effects at the zero (or reference) level of the other variable.
Everyday example: The effect of floor area on price may be larger in the city centre than on the outskirts. Adding a 'floor area*district' interaction lets the model estimate a separate floor-area coefficient for each district: say 20,000 TL in the centre and 10,000 TL on the outskirts.
What it does: "a*b" adds the main effects and the product, "a:b" only the product (model formula notation); it also works with factors. No centering is done.
Effect on the result: A product term is added; the main effects are interpreted as conditional effects.
On the validation data: insurance size*type product -0.00041714 (SE 0.01833, p = .9821). elemapi2 some_col*yr_rnd = 5.99319. The hsb2 math*prog and warpbreaks wool*tension interactions were also tested.
Reference category (factors) (First level, Last level)
Concept: Sets which level serves as the comparison baseline (reference) in dummy coding. The other levels' coefficients are read as differences from this level; the choice does not change the model's fit (R², F), only what the coefficients compare.
Everyday example: With the outskirts as the reference district, a coefficient of, say, +400,000 TL for the central district means 'a similar flat in the centre costs on average 400,000 TL more than one on the outskirts'. With the centre as reference, the same information would appear as −400,000 TL for the outskirts.
What it does: Sets the level left out in dummy coding.
Effect on the result: The interpretation of the coefficients changes; model fit (R², F) does not change. No effect without a factor.
On the validation data: elemapi2 yr_rnd + mealcat, last level: mealcat[1] 281.683, mealcat[2] 117.946. warpbreaks last level (tension M): tension[L] 10.000, tension[H] -4.722.
Extra output: Zero-order / partial / part correlations
Concept: Gives three correlations for each predictor: zero-order (the simple correlation with the outcome), partial (with the other predictors removed from both predictor and outcome) and semi-partial or part (with the other predictors removed from the predictor only). The squared semi-partial correlation is the share of R² that the predictor contributes on its own.
Everyday example: The number of rooms may have a simple correlation of 0.65 with price, but its semi-partial correlation drops to 0.08 once floor area is controlled. This shows that the rooms-price link largely reflects the fact that big homes have many rooms.
What it does: For each predictor gives the simple, partial and semi-partial (part) correlation with y. It cannot be combined with WLS.
Effect on the result: An additional table.
On the validation data: hsb2 female r = -0.1277, partial -0.1394, part -0.1006.
Extra output: Influence diagnostics (Cook D, leverage, DFBETAS...)
Concept: Finds observations that noticeably influence the model on their own or do not fit it. Leverage shows how unusual an observation is in its predictor values; Cook's distance and DFFITS how much the predictions would change if it were removed; DFBETAS how much each coefficient would change; and standardised residuals how far the observation departs from the model.
Everyday example: If among the 200 flats there is a single 600-square-metre villa, its leverage is high and it may alter the floor-area coefficient by itself. If Cook's distance flags this villa, the agency examines it separately or fits the model without it and compares the results.
What it does: Cook D, leverage, centered leverage, Mahalanobis, DFFITS, DFBETAS, ZRESID/SRESID/SDRESID and the Bonferroni outlier test. It cannot be combined with WLS.
Effect on the result: Additional tables.
On the validation data: crime, dc: RSTUDENT 3.76585, leverage 0.53638, Cook D 3.20343, DFFITS 4.05061, DFBETAS(single) 3.1391. For LifeCycleSavings a 50 x 5 DFBETAS table is produced.
→ Influence diagnostics (Cook's distance, leverage, DFBETAS)
WLS weight variable
Concept: Weighted least squares gives each observation a different weight so that more precise observations influence the model more. It is used when observations are means of groups of different sizes or when their variances differ in a known way.
Everyday example: If the agency works with district averages instead of individual flats, a district averaging 80 sales is not as reliable as one averaging only 3. With the number of sales given as the weight, the large districts shape the model more.
What it does: Applies weighted least squares; a row with a missing or ≤ 0 weight is excluded. For observations that are means or measurements with known different variances (group size as weight).
Effect on the result: Coefficients, SE, R², F and diagnostics follow the weighted model; Breusch-Pagan is computed on the transformed model and does not depend on the weight scale (census9: LM = 5.1574, p = .2715).
On the validation data: census9 drate ~ medage + region, weight pop: medage 4.283183 (SE .5393329), region[4] -10.90629, F 37.21, R² .7679. In the unweighted model region[4] -12.390.
Extra output (2): ANOVA table + SEE
Concept: Splits the total variation in the outcome into the part explained by the model (regression) and the unexplained part (residual); the F test is based on the ratio of the two. The standard error of the estimate (SEE) shows, in the outcome's own units, how far observed values typically depart from the model's prediction.
Everyday example: If the standard error of the estimate is 150,000 TL, then when the model predicts a flat's price the actual price usually falls within a band of roughly ±300,000 TL around the prediction. Even with a high R², that deviation can be large in practice.
What it does: Gives the regression, residual and total sums of squares, df, MS, F and the standard error of the estimate.
Effect on the result: An additional table.
On the validation data: hsb2 SS 9543.72074 / 9963.77926 / 19507.5, MS residual 51.0963039, SEE 7.148168. For NIST Longley SS and F, LRE ≥ 12.
Extra output (2): Collinearity diagnostics
Concept: Measures strong linear relationships among the predictors (collinearity). Tolerance and VIF show how much a predictor overlaps with the others; condition indices and variance proportions show which groups of predictors cause the problem. Collinearity makes coefficients unstable and inflates their standard errors.
Everyday example: If floor area and number of rooms are both in the model and are very strongly related, the VIF is high and the coefficient for rooms may turn unexpectedly negative. That does not mean 'rooms lower the price'; it means the effects of the two variables cannot be separated.
What it does: Tolerance (1/VIF), eigenvalues, condition indices and variance proportions (Belsley, Kuh and Welsch, 1980; including the constant, columns scaled to unit length).
Effect on the result: An additional table.
On the validation data: elemapi2 parental education model: avg_ed VIF = 43.5703, condition indices 1.000, 2.918, 4.455, 5.940, 42.036, 65.887.
Extra output (2): RESET test
Concept: Tests whether the model's functional form is correctly specified: if adding the squares and cubes of the fitted values to the model significantly improves its explanatory power, the model may be missing an important non-linear relationship or interaction.
Everyday example: If building age lowers price quickly in the first years and slowly after that, a straight line does not fully capture the relationship. A significant RESET test suggests the agency should consider adding age squared or grouping age into bands.
What it does: Ramsey RESET: tests functional form misspecification with the partial F when ŷ² and ŷ³ are added.
Effect on the result: An additional row.
On the validation data: hsb2 F(2, 193) = 0.6215, p = .538; census9 WLS F(2, 43) = 0.0973, p = .908.
Predictions + CI/PI and Save to dataset
Concept: Gives the model's predicted value for each observation and two different intervals: the confidence interval shows the uncertainty in the average price of flats with those predictor values, while the prediction interval shows where the price of a single new flat may lie. Because it includes individual variability, the prediction interval is always wider.
Everyday example: For a 100-square-metre, 3-room, 10-year-old flat the prediction might be 3,000,000 TL. The confidence interval for the average price of such flats might be 2.9-3.1 million TL, while the prediction interval for the price of the single flat the agency is about to list is much wider, say 2.7-3.3 million TL.
What it does: For each row gives the prediction, the mean confidence interval and the individual prediction interval (rows with missing Y are also predicted); the save option adds the PRE, RES, ZRE, LMCI, UMCI, LICI, UICI columns to the dataset.
Effect on the result: An additional table and columns; the interval level follows the Confidence Level.
On the validation data: airquality Ozone: 146 rows, 35 of them new observations with missing Y; predictions, confidence intervals and prediction intervals are given for these too. Saving was tested on hsb2, crime and airquality.
No-intercept model
Concept: Fits the model without an intercept, forcing the regression line through the origin. It is appropriate only if theory requires the outcome to be zero when all predictors are zero; its R² is defined differently and cannot be compared with that of a model with an intercept.
Everyday example: Dropping the intercept from the price model assumes that 'a 0-square-metre, brand-new flat in the centre costs 0 TL', which is unrealistic given factors such as land value. For a relationship like 'raw material used versus number of parts produced', on the other hand, a no-intercept model can make sense.
What it does: Removes the constant from the model (the regression passes through the origin); R² uses the uncentered definition and cannot be compared with the model with a constant. Only for when y = 0 is theoretically required at x = 0.
Effect on the result: R², F, β and the collinearity diagnostics change.
On the validation data: NIST NoInt1/NoInt2 coefficient LRE 14.8-15 (certified). In hsb2 R² 0.489 -> 0.981 (uncentered definition). In a no-intercept model with a factor MerQur keeps k - 1 dummies.
Diagnostic plot (Residual vs fitted, ZRESID × ZPRED, Residual histogram + normal P-P + Q-Q, Partial regression plots)
Concept: Chooses a plot for checking the model's assumptions visually: the residual-versus-fitted plot shows non-linear patterns and heteroscedasticity, the histogram and Q-Q plot the normality of the residuals, and partial regression plots each predictor's relationship once the others are controlled, along with influential observations.
Everyday example: If the points in the residual-versus-fitted plot fan out like a funnel towards expensive flats, the variance is not constant. If in the floor-area partial regression plot a single point pulls the line towards itself, that flat is an influential observation.
What it does: Selects the visual diagnostic; does not change the numeric results.
Effect on the result: Plot tab only.
On the validation data: ZPRED is the standardized fitted value and ZRESID the residual divided by sigma; the partial regression slopes equal the coefficients (math 0.389310; Frisch-Waugh-Lovell).
Model diagnostics: Lack of fit, Heteroscedasticity tests (Breusch-Pagan, White); Chow break test (None, Observation order, Grouping variable); WLS weight source (power), Power from, to and step
Concept: Additional tests of the model's assumptions and stability: heteroscedasticity tests (Breusch-Pagan, White) check whether the residual variance depends on the predictors; the Chow test checks whether the model's coefficients change between two subgroups or periods (a structural break); the lack-of-fit test checks whether a linear model is adequate when there are repeated observations at the same predictor values. The weight scan helps estimate WLS weights that correct heteroscedasticity.
Everyday example: If the Breusch-Pagan test is significant, the errors in price predictions are larger for big flats. The Chow test could check whether the effect of floor area on price differed between sales before and after a change in mortgage rates.
What it does: Heteroskedasticity (original Breusch-Pagan, Koenker and White), structural break (Chow; observation order or grouping variable), the lack-of-fit F test in replicated designs, and estimation of the weight as w = 1/x^p (power grid).
Effect on the result: Additional test rows; the main model does not change. In a saturated model the lack of fit gives an "undefined" note.
On the validation data: On cars and mtcars the lack-of-fit F test is computed with replicated x values. In the Chow test the break row is 17. In the WLS power grid the best p is 1.5.
Example on real data
Data: UCLA hsb2 (High School and Beyond, 200 students)
Variables: target science (science score), independents math, female, socst, read
- Question: To what extent is the students' science score explained by their math, reading and social studies scores and sex (female)?
- Result: The model is significant: F(4, 195) = 46.69, p < .001; R² = .489, adjusted R² = .479. About half of the variability in the science score is explained.
- Coefficients: Math (b = 0.389, p < .001) and reading (b = 0.335, p < .001) scores significantly increase the science score; with the other variables fixed, girls score 2.01 points lower in science, which is borderline non-significant (p = .051); social studies has no effect (p = .424).
- Assumption: The residuals are normal (Shapiro-Wilk W = 0.997, p = .975) and independent (Durbin-Watson = 2.005); VIF values are below 2.17, no collinearity problem. Breusch-Pagan LM = 10.48, p = .033 rejects homogeneity of variance.
- Limitation: Because of heteroskedasticity the result should be rechecked with the robust (HC3) standard error option.
- APA: F(4, 195) = 46.69, p < .001, R² = .49, adjusted R² = .48.
Related concepts
Analysis of variance (ANOVA) tableWeighted least squares (WLS)Hierarchical (blockwise) regressionRegression through the origin (no intercept)Multicollinearity (VIF, tolerance, condition index)Standardized regression coefficient (β)Ramsey RESET testInfluence diagnostics (Cook's distance, leverage, DFBETAS)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.