Statistics Glossary → Robust Regression (M-estimator)
Robust Regression (M-estimator)analysis
A family of linear regression methods that stops outliers, and leverage points with unusual predictor values, from distorting the coefficients by limiting their influence.
Concept
Ordinary least squares (OLS) minimises the sum of squared residuals; because squaring punishes large errors very heavily, a single outlier can pull the regression line towards itself. A data-entry error, a measurement glitch or a few observations from a genuinely different subgroup can thus change the whole result. Robust regression aims to estimate the relation followed by the bulk of the data while limiting the influence of a few unusual observations.
M-estimators replace the squared residual with a function that gives less weight to large residuals: small residuals are treated as in OLS, while the influence of large ones is capped or removed. This protects against outliers in the outcome but not against leverage points, which carry unusual values in the predictors. S and LTS estimators have a high breakdown point, meaning they give sensible results even when a large part of the data (up to about half) is contaminated. The MM-estimator is usually the recommended choice because it combines a high breakdown point with precision close to OLS when the data are clean (high efficiency).
The weights in the output of robust methods show which observations were found to be unusual and down-weighted; these observations should be examined rather than deleted, since they sometimes hold the most interesting information. If the robust and OLS coefficients are close, the outliers have little effect on the result; if they differ markedly, the OLS result rests on a few observations. Robust regression does not remove the linearity assumption and does not fix a misspecified model.
When to use it
- When outliers or leverage points are suspected and the relation representing the majority of the data is to be estimated; also to check how robust an OLS result is.
- If there are no outliers and the OLS assumptions hold, ordinary linear regression is more precise and easier to interpret.
- If the real interest lies in different parts of the distribution rather than the mean, or the error distribution is skewed, consider quantile regression (including median regression).
An everyday example
An estate agency examines the relation between floor area and sale price for 40 flats in a neighbourhood. Most flats follow a regular pattern, but two records are problematic: in one, the price was mistakenly entered with an extra zero; the other is a 400-square-metre penthouse with a sea view. The OLS line is pulled towards these two points and overstates the price per square metre. The MM-estimator gives these two observations very low weight and estimates the relation followed by ordinary flats, and the agency can see from the weights table which records it should check again.
Parameters in MerQur
Explanatory variable(s) and Target column
Concept: The dependent variable is the outcome to be explained; the explanatory variables are those thought to predict it. Categorical predictors enter the model as dummy variables compared with a reference level.
Everyday example: Sale price is the dependent variable; floor area, building age and whether there is a lift (yes/no) are the predictors. The lift coefficient shows how much more flats with a lift cost on average, other features being equal.
What it does: Selects the dependent and explanatory variables; categorical variables are expanded into dummy variables (the first level is the baseline level).
Effect on the result: All estimates.
On the validation data: stackloss, the Air.Flow slope is 0.9388 in the MM estimator.
Estimator (MM, M, S, LTS)
Concept: Estimators offer different kinds and degrees of protection against outliers. M is robust only to outliers in the outcome; S and LTS work even when a large part of the data is contaminated but are less precise when the data are clean; MM combines both properties. The breakdown point is the share of contaminated data an estimator can tolerate before it becomes meaningless.
Everyday example: If the only problem is a flat whose price was entered wrongly, an M-estimator is enough. But a record whose floor area is also unusual, like the 400-square-metre penthouse, may slip past M; MM limits the influence of both problematic records.
What it does: MM (S start + bisquare c = 4.685061, 50% breakdown point, 95% efficiency; the Yohai 1987 MM estimator) is the default; M (the selected norm; resistant only to outliers in the y direction); S (bisquare c = 1.547645); LTS (h = [(n + p + 1)/2], FAST-LTS). MM is recommended for data that may contain leverage points (Maronna et al., 2006).
Effect on the result: Coefficients, scale and weights.
On the validation data: stackloss, Air.Flow slope / scale: MM 0.9388 / 1.9123; M-Huber 0.8294 / 2.4405; S 0.8496 / 1.9123; LTS 0.7409 / 0.9888 (H = 13 in LTS).
→ Robust regression estimators (MM, S, LTS) and the breakdown point
Norm (M-estimator: Huber-T, Tukey biweight, Hampel)
Concept: The norm (psi function) determines how much weight an observation gets according to the size of its residual. Huber's function caps the influence of large residuals but never removes it; Tukey's biweight and Hampel's functions can remove the influence of observations with very large residuals entirely.
Everyday example: Under the Huber norm, the flat whose price has an extra zero still carries some influence. Tukey's biweight ignores this record almost completely, because its residual is extremely large compared with the other flats.
What it does: Selects the psi function of the M estimator; active only in M.
Effect on the result: Coefficients and scale; no effect in MM, S and LTS.
On the validation data: stackloss slope: Huber 0.8294 (1 outlier), Tukey bisquare 0.9276 / scale 2.2819, Hampel (2, 4, 8) 0.7411 / scale 3.0880.
Tuning constant (M)
Concept: The tuning constant sets how large a residual must be before it counts as 'unusual' and is down-weighted. A smaller constant down-weights more observations and resists outliers better, but loses precision when the data are clean; the usual values keep about 95% of OLS efficiency when the data are normal.
Everyday example: If the constant is set very small, not only faulty records but also slightly pricey yet perfectly normal flats lose weight, and the estimate becomes needlessly uncertain. The default constant trims only the clearly deviating records without touching ordinary flats.
What it does: The tuning constant of the M estimator (Huber t, Tukey c); 0 uses the default value (1.345 / 4.685, 95% efficiency under the normal distribution). A smaller value is more resistant and less efficient (Huber, 1981). Disabled when Hampel is selected.
Effect on the result: Coefficients and the number of outliers.
On the validation data: stackloss: Huber t = 1: 0.8296 (4 outliers); t = 2: 0.7411; Tukey c = 4: 0.9398; c = 6: 0.7667.
Scale estimator (MAD, Huber proposal 2)
Concept: To judge how large residuals are, a scale, i.e. the typical residual size, is needed. Because the standard deviation is inflated by outliers, robust methods use scales unaffected by them, such as the median absolute deviation (MAD); Huber's proposal 2 is another approach that estimates the scale together with the coefficients.
Everyday example: The flat whose price has an extra zero inflates the standard deviation of the residuals enormously, making every other flat look 'ordinary'. The MAD is hardly affected by this single record and correctly reflects the typical deviation of ordinary flats.
What it does: Determines the residual scale in the M estimator: the rescaled MAD (median absolute residual / 0.6745) or Huber proposal 2. No effect in MM, S and LTS.
Effect on the result: Scale and coefficients.
On the validation data: stackloss Huber: MAD scale 2.4405; Huber proposal 2 scale 2.8551, slope 0.8167 (Huber d = 1.345).
→ Robust regression estimators (MM, S, LTS) and the breakdown point
Max iterations (M)
Concept: The M-estimator is computed by an iterative method that updates the weights and coefficients in turn (iteratively reweighted least squares). The cap sets how many rounds the algorithm will try before the changes become small enough; hitting the cap means the result is not yet stable.
Everyday example: In each round the algorithm lowers the weight of the faulty price record a little more and redraws the line. After a few rounds the line stops moving; hitting the cap usually means the data or the model need a second look.
What it does: Sets the upper limit on the IRLS iterations; active only in M. A warning is given if it stops before convergence.
Effect on the result: Coefficients at the truncated iteration.
On the validation data: At 10 / 50 / 200 iterations the largest difference in Huber is 6e-5; Tukey stops before convergence at 10 iterations (slope 0.92740; converged 0.92756).
Leverage / outlier diagnostics (MCD robust distance)
Concept: Leverage diagnostics find observations in unusual positions in predictor space using a robust distance (MCD) and classify them together with their residuals. A good leverage point has unusual predictor values but fits the general relation; a bad leverage point is both unusually placed and far from the relation, and is the kind of observation that distorts the line most.
Everyday example: A 200-square-metre villa priced in line with its size is a good leverage point: its size is unusual but it follows the pattern. The sea-view penthouse is a bad leverage point, being both very large and priced far above the pattern.
What it does: Gives the reweighted MCD (FAST-MCD) distance of the explanatory variables and the classification into regular / vertical outlier / good / bad leverage with |r/s| > 2.5 (Rousseeuw and van Zomeren, 1990).
Effect on the result: An additional table only.
On the validation data: stackloss MM: 11 / 1 / 8 / 1; threshold sqrt(chi-square(.975, 3)) = 3.0575. hbk MCD unscaled determinant 0.3506879.
→ Influence diagnostics (Cook's distance, leverage, DFBETAS)
Save weights, residuals and predictions to the dataset
Concept: Adds to the dataset each observation's predicted value, residual and the weight the robust method gave it. Observations with low weight are unusual records worth examining in detail.
Everyday example: The record showing a weight of 0.02 is the flat whose price has an extra zero; the agency can check this row against the land registry and correct it. Flats with weights near 1 are records that follow the general pattern.
What it does: Adds the ROB_PRED, ROB_RESID, ROB_WEIGHT (and, if diagnostics are on, ROB_RD) columns.
Effect on the result: New columns only.
On the validation data: The added columns hold the fitted value, residual and robust weight for each stackloss observation (the LTS weight depends on the scale definition).
Confidence Level (%90, %95, %99, custom)
Concept: Sets how much assurance the coefficients' confidence intervals give of covering the true value. A higher level gives a wider interval.
Everyday example: If the robust estimate of price per square metre is 25,000 lira, you can say with 95% confidence that the true value lies between, say, 22,000 and 28,000 lira; asking for 99% widens the interval.
What it does: Sets the confidence interval level (t(n - p) in MM, normal in M and S).
Effect on the result: Only the coefficient confidence intervals; they are shown in the text on each coefficient's line ("90% CI [lower, upper]") and in the table in the "CI (lower)" and "CI (upper)" columns.
On the validation data: MM 90% Air.Flow [0.7346, 1.1431]; M-Huber 97.5% Air.Flow [0.5806, 1.0782]; M-Huber 95% Air.Flow [0.611818, 1.046951].
Example on real data
Data: Brownlee 1965 stackloss, R datasets
Variables: target stack.loss (ammonia loss), explanatory Air.Flow, Water.Temp (water temperature), Acid.Conc. (acid concentration)
- Question: In an ammonia plant, how is stack loss (stack.loss) related to air flow, water temperature and acid concentration, and how much do outliers affect OLS?
- Model: MM estimator (S start, 95%-efficient bisquare, c = 4.685); N = 21; robust scale = 1.912.
- Result: Air.Flow b = 0.939 (SE = 0.117), p < .001, 95% CI [0.691, 1.187]; Water.Temp b = 0.580 (SE = 0.263), p = .042, CI [0.025, 1.134]; Acid.Conc. b = -0.113 (SE = 0.070), p = .125.
- Model fit: Robust R² = .959 (adjusted .952); Wald χ²(3) for all slopes = 144.67, p < .001.
- Outliers: 2 observations have an MM weight below 0.5; the OLS coefficients differ clearly (Water.Temp OLS 1.295, robust 0.580; a 55% difference).
- APA: Air.Flow: β = +0.939, SE = 0.117, p < .001, 95% CI [0.691, 1.187]; Water.Temp: β = +0.580, SE = 0.263, p = .042 (MM, N = 21, outliers = 2).
Related concepts
Median absolute deviation (MAD) and modified z-scoreRobust regression estimators (MM, S, LTS) and the breakdown pointM-estimator (Huber, Tukey biweight)Influence diagnostics (Cook's distance, leverage, DFBETAS)OutlierMinimum covariance determinant (MCD) and robust distanceConvergence and maximum iterationsTruncation
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.