MerQuR+Statistics Glossary

Statistics Glossary → Partial Least Squares Regression (PLS)

Partial Least Squares Regression (PLS)analysis

A regression method that reduces many strongly correlated predictors to a few latent components most related to the response, and models the response with those components.

Concept

A spectrometer produces measurements at hundreds of wavelengths for a single sample, and measurements at neighbouring wavelengths carry almost the same information. In such data the number of predictors often exceeds the number of observations and the predictors are extremely interdependent; ordinary least squares either cannot be fitted at all or gives wildly unstable coefficients. Partial least squares (PLS) regression gathers this information into a few summary variables, the latent components, and predicts the response from them.

Principal component analysis also summarises the predictors, but it does so by looking only at the predictors' own variability; the components it finds may be unrelated to the response. PLS instead chooses components that both capture variability in the predictors and have the largest covariance with the response. The first component is the direction that best meets this criterion; the second is extracted in the same way from what the first left unexplained, and so on. The final model can also be written as an ordinary regression equation giving each predictor a coefficient.

The most critical decision is the number of components: too few miss information, too many bring noise into the model and lead to overfitting. The number is chosen by cross-validation, looking at the prediction error on data not used to build the model (RMSECV) or at Q². The importance of predictors is judged with VIP scores; predictors with VIP > 1 are usually regarded as important. The model depends on the conditions under which the predictors were measured and should be revalidated before use with a different instrument or type of sample.

When to use it

  • When the predictors are numerous and strongly correlated, especially when they outnumber the observations: spectroscopy, chemometrics, rapid quality analysis of food and agricultural products, sensor data.
  • If there are few predictors with moderate relations among them, ordinary multiple regression is more suitable; if variable selection is also wanted, LASSO or ElasticNet.
  • If the aim is not to predict a response but only to summarise the structure of the predictors, use principal component analysis.

An everyday example

An apple packing plant wants to know each apple's sugar content without cutting it. Near-infrared light is shone on the apples passing along the belt and reflectance is measured at 300 wavelengths, but the sugar content of only 80 apples has actually been measured in the laboratory. With 300 predictors and 80 observations, ordinary regression cannot be fitted. PLS summarises the spectrum in, say, five latent components and predicts sugar content from them. If the model performs well in cross-validation, the plant can grade every apple within seconds without sending it to the lab.

Parameters in MerQur

Predictors (X) and Target (Y)

Concept: The predictors are the many measurements used to predict the response; the response is the variable to be predicted. Because predictors may be on different scales they are usually standardised, so that variables measured in large numbers do not dominate the components.

Everyday example: Reflectance at the 300 wavelengths are the predictors and the lab-measured sugar content is the response. Standardisation stops wavelengths with generally high reflectance from looking more important for that reason alone.

What it does: Selects the predictor and response variables (single-response PLS1); X and Y are standardized and the coefficients are converted back to the original scale.

Effect on the result: All outputs.

On the validation data: gasoline training R2: 1 component 0.3159, 2 components 0.7929, 5 components 0.9891, 10 components 0.9957, 20 components 0.9999.

Number of components

Concept: The number of components sets how many latent variables the model uses to summarise the predictors' information. Each new component improves the fit to the training data, but beyond some point it only learns noise and worsens prediction on new data.

Everyday example: A two-component model might capture the main pattern in sugar content; with five components prediction improves clearly; with fifteen the model fits the 80 lab apples perfectly but predicts new apples worse.

What it does: Sets the number of latent components kept in the model; it should be selected by cross-validation (minimum RMSECV or entering the plateau). The requested number is clipped to min(requested, p, n - 1, rank(X)) and the clipping is reported with a warning.

Effect on the result: Coefficients, VIP, training R2, explained variance.

On the validation data: gasoline R2 (5-fold CV): -0.31 at 1 component; 0.956 at 5 components; 0.944 at 10 components. Q2 (5-fold, PRESS): 5 components 0.9739. Penta (X rank 11): when 12, 14 and 20 components are requested, it is clipped to 11 with a warning.

Select number of components by CV

Concept: Cross-validation computes the prediction error on data not used to build the model for different numbers of components. RMSECV shows the typical size of the prediction error, and Q² how much of the response's variability the model can also predict in new data; the suitable number of components is where the error is smallest or stops falling noticeably.

Everyday example: If RMSECV is 1.4 Brix with one component, 0.8 with three and 0.6 with five, and then starts rising again, five components is the right choice: the sugar prediction for a new apple is typically off by about 0.6 Brix.

What it does: Computes PRESS, RMSECV = sqrt(PRESS/n) and Q2 = 1 - PRESS/SST from 0 up to the maximum number of components; marks the smallest RMSECV. The number of components of the main model does not change (it is only a suggestion).

Effect on the result: The CV table only.

On the validation data: gasoline LOO RMSECV: 0 components 1.5451; 1 component 1.3211; 2 components 0.7857; 3 components 0.2869; best 6 components.

→ k-fold cross-validation

CV type (LOO, 10-fold consecutive)

Concept: Leave-one-out (LOO) cross-validation sets aside each observation in turn as the data to be predicted; k-fold cross-validation splits the data into k parts and leaves each out in turn. If similar or replicate measurements sit next to each other in the data, LOO can be optimistic; consecutive folds leave such groups out together and give a more cautious estimate.

Everyday example: If each apple was scanned twice and the two scans sit one after the other in the data, LOO predicts one scan using its twin and exaggerates the model's success. Ten consecutive folds leave both scans of the same apple out together.

What it does: Selects leave-one-out (LOO) or 10 consecutive-fold cross-validation; it also determines the jackknife segments. No effect when CV and jackknife are off.

Effect on the result: The CV table and the jackknife intervals.

On the validation data: gasoline 10-fold: best 4 components (6 in LOO).

→ k-fold cross-validation

Max. components (CV)

Concept: Limits the largest number of components tried in cross-validation. The cap should be large enough to comfortably include the expected best number and small enough to avoid needless computation.

Everyday example: Since the best number in spectroscopy models is often below 10, trying up to 15 components for the sugar model is enough; trying 60 components with 80 apples would only show how overfitting grows.

What it does: Sets the upper limit tried in cross-validation; min(value, p, n - 2).

Effect on the result: The CV curve and the suggested number of components.

On the validation data: gasoline: best 1 for 1; 4 for 5; 6 for 10 and 30.

→ k-fold cross-validation

Additional Y (PLS2)

Concept: PLS2 models several responses together with the same latent components. If the responses are related, shared components can be more efficient; if they are unrelated, a separate PLS1 model for each response usually predicts better.

Everyday example: If the plant wants to predict firmness and acidity as well as sugar content from the same spectrum, PLS2 can be used. Because all three change together with ripeness, shared components may describe them all well.

What it does: Models several responses with common components; the main table is the PLS1 model of the first Y, and PLS2 is given in a separate section.

Effect on the result: An additional section.

On the validation data: oliveoil 6 sensory Y, 2 components: yellow constant 106.5030; R2 yellow 0.4541, brown 0.7349.

Extra output: Weights/loadings/scores

Concept: Weights show which combination of predictors each component is built from; loadings show how strongly the predictors relate to the components; scores show each observation's position on the components. These outputs let you look inside the model and spot unusual samples.

Everyday example: If the first component's weights are high in a particular wavelength range, that component reflects absorption in that region. An apple that sits far from all the others on the score plot, perhaps because it is bruised or was scanned wrongly, deserves a closer look.

What it does: Gives the X weights W (unit norm), the X loadings P and the Y loadings q.

Effect on the result: An additional table only.

On the validation data: W has unit norm; because X is scaled by the 1/n sd, the q loadings differ by a factor of sqrt((n - 1)/n) from the (n - 1) scaling.

→ Factor (component) loadings

Extra output: Jackknife CI

Concept: The jackknife approximates the coefficients' standard errors and confidence intervals by looking at how much the coefficients estimated with each cross-validation part left out vary. Predictors whose intervals exclude zero contribute stably to the model.

Everyday example: If a wavelength's coefficient is always positive and of similar size whichever group of apples is left out, that wavelength is a reliable source of information for predicting sugar.

What it does: From the CV segments, gives the jackknife coefficient variance (Martens and Martens 2000), t = b/SE, degrees of freedom (segments - 1) and the confidence interval.

Effect on the result: An additional table only.

On the validation data: gasoline LOO and 10-fold, yarn 10-fold: SE, t, p and interval are computed from the CV segments.

→ Jackknife

Extra output: Predicted-observed plot

Concept: The predicted-observed plot draws the model's predictions against the actual values. If the points gather in a narrow band around the 45-degree line, prediction is good; a systematic deviation in some range shows that the model is weak there.

Everyday example: If the plot shows that the sugar content of very sweet apples is systematically underpredicted, it is clear that more sweet apples need to be added to the training data.

What it does: Adds a plot of predicted and observed values.

Effect on the result: Graphics only; the numerical output does not change.

On the validation data: No numerical difference.

Example on real data

Data: gasoline NIR spectra, Kalivas 1997, R pls (training set)

Variables: target octane (octane number), predictors nir900-nir1700 (NIR absorbance at 401 wavelengths)

  • Question: How well does the NIR spectrum of 50 gasoline samples at 401 wavelengths predict the octane number? (The number of predictors far exceeds the number of observations and the predictors are very highly correlated; OLS cannot be fitted.)
  • Model: 2-component PLS1, X scaled. The first component explains 64.31% of the X variance and the second 20.93% (total 85.24%).
  • Result: The two components explain 79.29% of the octane variance (training R² = 0.7929); the first component contributes only 31.59%, so the Y-related information lies mostly in the second component.
  • Model fit (validation): In 5 consecutive folds Q² (PRESS) = 0.7076, fold mean R² = 0.4879. Since Q² is above 0.5, the model also predicts well out of fold; the fold mean R² is affected by the narrow Y range within individual folds.
  • Additional finding: The highest VIP values are around 1204-1214 nm (highest 1.96 at 1208 nm) and 1356-1362 nm (about 1.94); these are the most informative wavelengths for octane (VIP > 1 is considered important).
  • Limitation: The number of components defaults to 2; the optimal number should be selected by CV.
  • APA: PLS regression (2 components), R² = .79, Q²(5-fold) = .71, N = 50, 401 predictors.

Related concepts

Degrees of freedomJackknifek-fold cross-validationFactor (component) loadingsPrediction error measures (RMSE, MAE, MAPE, MASE)Q² predictive relevance (Stone-Geisser, PRESS)Leave-one-out (LOO)Proportion of explained variance

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.