MerQuR+Statistics Glossary

Statistics Glossary → LOESS (Local Regression)

LOESS (Local Regression)analysis

A method that shows how an outcome changes with one or two explanatory variables through a smooth curve (or surface) built from small regressions fitted locally to the data, without assuming in advance a shape such as a line or a parabola.

Concept

Linear regression assumes from the start that the relationship is a straight line, and polynomial regression that it is a particular curve. Often, though, the shape is unknown: fuel consumption may be high at low speeds, lowest at moderate speeds and high again at high speeds. LOESS (local regression) is a flexible smoother that learns the shape of the relationship from the data and shows such curves without trying to fit them into a single formula.

For each point on the curve, a neighbourhood of the observations closest to that point is taken. The span sets the size of the neighbourhood: span = 0.75 means 75% of the observations are used in each local fit. The observations in the neighbourhood are weighted by their distance from the point (closer ones weigh more; tricube weights), and a weighted line or parabola is fitted to them. The value of this local fit at the point becomes the value of the curve. A small span captures detail but gives a noisy curve; a large span gives a smoother curve that may miss detail.

LOESS produces a curve, not a table of coefficients, so it does not answer 'how much does y rise when x rises by one unit?' with a single number. The model's complexity is summarised by the equivalent number of parameters (enp). The confidence band is pointwise: it holds separately at each x value and is not a simultaneous band for the whole curve. At the ends, where data are sparse, and outside the observed range the curve becomes unreliable; with many explanatory variables the method also becomes inefficient.

When to use it

  • When the shape of the relationship between an outcome and one or two numeric explanatory variables is unknown, and you want to explore the data, see the curvature or check whether a linear model is adequate.
  • If the shape is known from theory (e.g. quadratic), polynomial regression or response surface regression gives more interpretable coefficients.
  • If there are many explanatory variables and each effect is to be modelled with a flexible curve, generalised additive models (GAMs) or spline regression are preferred.

An everyday example

A logistics company has recorded the fuel consumption of 60 lorries at different average speeds. A straight line is forced to say consumption keeps rising or keeps falling with speed. A LOESS curve might instead show consumption is high below 50 km/h, lowest around 70-80 km/h and rising fast above 90 km/h. Looking at this curve, the company can decide which speed range to recommend to its drivers.

Parameters in MerQur

Explanatory variable(s)

Concept: The explanatory variable is the numeric variable forming the horizontal axis of the curve. With one explanatory variable a curve is fitted; with two, a surface (a two-dimensional 'landscape').

Everyday example: In the lorry example the explanatory variable is average speed. If load weight is chosen as well, LOESS draws a surface showing how consumption changes with speed and load together.

What it does: Selects one or two numeric explanatory variables on which the smoothing is based; with two, a surface is fitted.

Effect on the result: Determines the neighbourhood distance and the model dimension.

On the validation data: One explanatory variable (speed) in cars, two (C and E) in ethanol.

Target column

Concept: The target is the numeric outcome whose variation is shown after smoothing.

Everyday example: The target is fuel consumption in litres per 100 km; the curve shows the expected consumption at each speed.

What it does: Selects the numeric dependent variable to be smoothed.

Effect on the result: Determines all outputs.

On the validation data: cars: dist (stopping distance).

Span selection (Fixed (manual), Automatic: AICc, Automatic: GCV)

Concept: The span can be given by hand or chosen from the data. AICc and generalised cross-validation (GCV) are criteria that balance goodness of fit against the complexity of the curve; the span giving the smallest value is chosen.

Everyday example: If it is unclear which span suits, AICc is chosen; the criterion suggests the span that captures the true shape of the consumption curve without adding needless wiggles.

What it does: With fixed selection the span is entered by hand (default 0.75). With AICc or GCV all neighbourhood sizes with span <= 1 are scanned and the one minimising the criterion is chosen; ties take the largest span.

Effect on the result: Determines the span and hence the roughness of the curve. With automatic selection the span is determined from the data by the AICc or GCV criterion.

On the validation data: cars: span 1.0 with AICc (minimum AICc 6.56859516), 0.98 with GCV; ethanol: 0.27272727 with AICc, 0.22727273 with GCV.

→ Smoothing parameter (λ, span)

Smoothing (span) (default 0.75)

Concept: The span is the share of observations used in each local fit and sets how smooth the curve is. A small span gives weight to local detail, a large span to the overall trend.

Everyday example: With span 0.3, each point is computed from only the 18 nearest lorries (30% of 60) and the curve is jittery. With span 0.9 almost all lorries are used and the curve is smooth, but the dip around 70-80 km/h may fade.

What it does: Sets the proportion of observations used in each local fit: q = floor(n x span). If the span exceeds 1, all observations are used and the neighbourhood radius is enlarged.

Effect on the result: A small span gives a more flexible curve, a large span a smoother one; changes enp and the residual standard error.

On the validation data: cars, degree 2: span 0.75 gives enp 4.83393101, residual SE 15.31087491; span 1.5 gives enp 3.19816237, residual SE 15.16661435.

→ Smoothing parameter (λ, span)

Local polynomial degree (1 (locally linear), 2 (locally quadratic))

Concept: The form of the small regression fitted in each neighbourhood. A locally linear fit captures only the slope, a locally quadratic fit also the curvature; for relationships with peaks and dips, degree 2 usually works better.

Everyday example: If the consumption curve has a dip around 75 km/h, the locally quadratic fit draws this minimum more accurately; the locally linear fit may flatten it a little.

What it does: Sets the degree of the local polynomial fitted in each neighbourhood (1 linear, 2 quadratic).

Effect on the result: Locally linear is less flexible; locally quadratic follows curvature and peaks better but increases variance.

On the validation data: cars: degree 1 and span 0.5 give enp 4.01730395, residual SE 15.289023; degree 2 and span 0.75 give enp 4.83393101, residual SE 15.31087491.

→ Smoothing parameter (λ, span)

Family (Gaussian (least squares), Symmetric (robust, bisquare))

Concept: The Gaussian family uses ordinary least squares and is sensitive to outliers. The symmetric (robust) family iteratively lowers the weight of observations with large residuals, so a few extreme values cannot pull the curve towards themselves.

Everyday example: If one lorry's consumption was recorded at twice the usual because its fuel tank was leaking, the Gaussian curve bulges upward at that speed. The robust family greatly reduces the influence of this single record.

What it does: Gaussian is the least-squares fit. The symmetric family applies 4 iterations of bisquare robust weighting and resists outliers.

Effect on the result: Changes the fitted values and the residual standard error.

On the validation data: cars, degree 2, span 0.75: Gaussian residual SE 15.31087491, symmetric 13.24337049.

→ M-estimator (Huber, Tukey biweight)

Statistics (Approximate (R default), Exact (SAS DIRECT))

Concept: In LOESS, statistics such as degrees of freedom and residual standard error are computed from the smoothing matrix. This can be done quickly with an approximate formula or exactly with the full matrix; the two usually give very close results.

Everyday example: If the results must match another program exactly, the computation type that program uses is chosen. With 60 lorries, the difference between the two shows only in the second decimal place.

What it does: In approximate mode delta1/delta2 use the Cleveland-Grosse approximation; in exact mode they come from the full smoothing matrix (n <= 4000).

Effect on the result: Changes delta1, delta2, the lookup df, enp and the residual SE; does not change trace(L) or the fitted values.

On the validation data: cars, degree 2, span 0.75: approximate enp 4.83393101, delta1 44.23236615, residual SE 15.31087491; exact enp 4.90741774, delta1 44.30585288, residual SE 15.29817216.

Normalize ([Parameters] check box; default on)

Concept: When two explanatory variables are measured in different units, 'closeness' is dominated by the variable measured in large numbers. Normalisation divides each explanatory variable by its own spread, so both enter the neighbourhood calculation on an equal footing.

Everyday example: If speed ranges over 40-110 km/h and load over 2,000-20,000 kg, without normalisation the neighbourhood is decided almost entirely by load. With normalisation, speed and load count with similar weight.

What it does: With two explanatory variables, divides each by its 10% trimmed standard deviation (default); has no effect with one explanatory variable.

Effect on the result: Changes the scale of the explanatory variables, hence the neighbourhood and all statistics.

On the validation data: ethanol, degree 2, span 0.5: on gives enp 13.64520361, residual SE 0.24929388; off gives enp 11.6500919, residual SE 0.45843128.

→ z-score (standardization)

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets the confidence level of the interval at each point of the curve. These intervals are pointwise: each holds separately at its speed value, with no guarantee that the whole curve stays inside the band at once.

Everyday example: If the 95% confidence interval for expected consumption at 75 km/h is 28-30 litres, it applies to that speed only. Around 110 km/h, where data are scarce, the interval widens noticeably.

What it does: Sets the confidence level of the prediction confidence interval (pointwise, not a simultaneous band). A custom level is entered in the range 80.0-99.9.

Effect on the result: Changes only the confidence interval limits.

On the validation data: cars: at new points the 95% prediction, SE and interval are given; in the Gas data the 99% interval is computed for 22 observations.

→ Confidence level and confidence interval

Save to dataset ([Parameters] check box)

Concept: Adds to the data set, as new columns, the prediction the curve gives for each observation (fitted value) and its difference from the observed value (residual). Residuals are useful for finding outlying observations or for further analyses.

Everyday example: Each lorry's row gets 'expected consumption' and 'deviation from expected' columns. The lorries with the largest deviations can be checked first for maintenance.

What it does: Adds the "<target>_loess_fit" and "<target>_loess_resid" columns to the dataset.

Effect on the result: Does not change the computation.

On the validation data: cars: the dist_loess_fit and dist_loess_resid columns are added.

Example on real data

Data: Ezekiel 1930, R datasets

Variables: target dist (stopping distance), explanatory speed

  • Question: How does the stopping distance of cars (dist) change with speed without assuming a parametric form? (cars, n = 50)
  • Model: LOESS, span = 0.75 (37/50 observations in each local fit), local polynomial degree 2, Gaussian family, direct computation.
  • Fit: Equivalent number of parameters 4.83, trace(L) = 5.30, residual standard error 15.31, residual SS = 10369.08, pseudo R² = .68.
  • Additional statistics: δ1 = 44.23, δ2 = 43.92, lookup degrees of freedom 44.55; AICc = 6.63, GCV = 5.19.
  • Interpretation: The curve shows a relationship that increases with speed and steepens at high speeds; it is a more flexible fit than a linear model, with about 5 parameters.
  • APA: LOESS (degree = 2, span = 0.75), N = 50, enp = 4.83, residual SE = 15.31, pseudo R² = .68.

Related concepts

Smoothing parameter (λ, span)M-estimator (Huber, Tukey biweight)z-score (standardization)Generalized cross-validation (GCV)Residual plots and residual diagnosticsOutlierConvergence and maximum iterationsTies and zero differences (rank tests)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.