MerQuR+Statistics Glossary

Statistics Glossary → Discriminant Analysis (LDA / QDA)

Discriminant Analysis (LDA / QDA)analysis

A multivariate method that finds the combinations of continuous variables that best separate groups known in advance, and uses these combinations to estimate which group a new observation belongs to.

Concept

Discriminant analysis answers two questions together: do the measured variables separate the known groups, and if so, which variables contribute most to the separation; and to which group should a new observation of unknown membership be assigned? In this sense it is both a descriptive method (how do the groups differ) and a predictive one (classify a new observation).

Linear discriminant analysis (LDA) seeks linear combinations of the variables (discriminant or canonical functions) for which the between-group difference is largest relative to the within-group spread. With K groups and p variables at most min(K − 1, p) functions are found; the eigenvalue and canonical correlation of each function show its separating power. The overall significance of the separation is tested with Wilks' Λ: Λ = |W| / |T|, the ratio of within-group variability to total variability; the smaller Λ, the better the groups are separated. For classification, group membership (posterior) probabilities are computed for each observation, and it is assigned to the most probable group.

LDA assumes that within each group the variables are multivariate normal and that the groups have equal covariance matrices; the latter is tested with Box's M, but the test is very sensitive in large samples and to departures from normality. If the covariances clearly differ, quadratic discriminant analysis (QDA), which allows each group its own covariance, can be used; because QDA estimates more parameters, it is unstable in small groups. The accuracy on the data used to build the model (resubstitution) is optimistic; realistic performance is assessed by leave-one-out or k-fold cross-validation.

When to use it

  • When the groups of the observations are known in advance and the aim is to see how far, and along which dimensions, a few continuous measurements separate them, and to classify new observations (species identification, occupational groups, disease subtypes).
  • If the predictors include categorical variables, or normality is badly violated, logistic regression (binary outcome) or multinomial logistic regression (several groups) is more suitable.
  • If the main aim is to test whether the group mean vectors differ, MANOVA is used; if the groups are not known in advance and are to be discovered, cluster analysis.

An everyday example

A grain merchant wants to tell which of three varieties each wheat lot arriving at the depot belongs to, from measurements of kernel length, kernel width and protein content. A discriminant analysis built on 150 lots of known variety might show that a combination such as 'the difference between length and width' separates the first variety from the others, while protein content separates the second from the third. When a new lot arrives, its measurements are plugged into these functions and the lot is classified as, say, the second variety with 88% probability.

Parameters in MerQur

Explanatory variable(s) and Target class

Concept: The predictors are the continuous measurements used to separate the groups; the target class is the known group membership of the observations. The method looks for the combination of these measurements that best separates the groups.

Everyday example: For the merchant the predictors are kernel length, width and protein content; the target class is the wheat variety established for certain in the laboratory. Lots of unknown variety are classified after the model is built.

What it does: Numeric explanatory variables (at least 2) and the class variable; at least 20 complete rows are required.

Effect on the result: The model, the canonical solution, Box's M and all additional outputs.

On the validation data: (UCLA): Canonical r 0.7207 and 0.4927, eigenvalue 1.0805 and 0.3205, Wilks 0.3640 and 0.7573, χ²(6) 242.55, resubstitution accuracy 0.7500; the classification table is reported.

Method (LDA (linear) / QDA (quadratic) / kNN (nonparametric) / Kernel density (nonparametric))

Concept: LDA assumes one covariance matrix shared by all groups and draws straight boundaries between them; QDA allows each group its own covariance and draws curved (quadratic) boundaries. The kNN and kernel density methods make no distributional assumption and base the classification on nearby observations or on a local density estimate.

Everyday example: If the kernel measurements of the three varieties have similar spread, LDA is enough. If one variety's kernels are very uniform and another's very variable, QDA takes this difference into account; if the measurements are far from normal, kNN assigns a new lot to the variety of the lots it most resembles.

What it does: LDA builds a linear boundary with the pooled covariance, QDA a quadratic boundary with a covariance per class.

Effect on the result: Classification, posterior probabilities, LOO and CV; the canonical solution and Box's M are independent of the method.

On the validation data: iris LDA and QDA resubstitution 0.98 and 0.98, LOO 0.98 and 0.9733 (QDA: 3 errors in resubstitution, 4 in cross-validation). UCLA QDA 0.7623. crops QDA (class-proportion prior) resubstitution 0.8889, LOO 0.4444 (error rates 0.1111 and 0.5556).

→ Covariance and correlation matrix

CV folds

Concept: In k-fold cross-validation the data are split into k parts; each time the model is built on k − 1 parts and classifies the remaining one. Accuracy is thus computed on observations not used to build the model, and is more realistic than resubstitution accuracy.

Everyday example: The 150 lots are split into 5 parts; each time, a model built on 120 lots predicts the variety of the remaining 30. The average accuracy over the five rounds is closer to the success the merchant can expect on new lots.

What it does: k-fold CV accuracy (unshuffled stratified folds; k is reduced to the smallest class size).

Effect on the result: Only the CV accuracy and its spread change; the resubstitution accuracy and the canonical solution do not change.

On the validation data: iris k = 2, 5, 10: 0.9733 ± 0.0133, 0.98 ± 0.0267, 0.98 ± 0.0427; crops k = 10, reduced to 6 (smallest class 6), 0.3889 ± 0.2833. In QDA, if n <= p per class in a fold, the CV gives "-" and an explicit warning.

→ k-fold cross-validation

Prior probabilities (Equal (1/K; default) / Class proportions / User-defined) and Prior values

Concept: The prior probability is the probability that an observation belongs to each group before its measurements are seen. If one group is in fact far more common, the priors shift the classification boundary in its favour; the canonical functions themselves do not change.

Everyday example: If 70% of the lots arriving at the depot are usually the first variety, then for a lot whose measurements are ambiguous the first variety is a more likely guess. Setting the priors to 'class proportions' takes this into account; choosing 'equal' treats every variety as equally likely.

What it does: Equal (1/K; default), class proportions, user-defined (in alphabetical class order; normalized to 1).

Effect on the result: Classification boundaries, posterior probabilities, the Fisher classification constant (+ ln prior), LOO; it does not affect the canonical solution.

On the validation data: (UCLA, equal / proportions / 0.2-0.3-0.5): Accuracy 0.7500, 0.7582, 0.7049; LOO 0.7459, 0.7500, 0.6885. The iris entry "2 1 1" is normalized to (0.5, 0.25, 0.25). In crops the Fisher constants are computed with the selected class-proportion priors.

→ Prior class probabilities (discriminant analysis)

Additional output: Fisher classification function coefficients

Concept: Fisher's classification functions are a separate linear scoring formula for each group; a new observation's measurements are plugged into each formula and it is assigned to the group with the highest score. Unlike the canonical functions, they are used directly for classification.

Everyday example: The merchant writes the three formulas for the three varieties in a table. They plug the new lot's length, width and protein values into each formula; the variety whose formula gives the largest result is taken as the lot's variety, with no computer needed.

What it does: c_k = S_w⁻¹x̄_k and the constant -½x̄_k'S_w⁻¹x̄_k + ln(prior); a new observation is assigned to the class with the largest value.

Effect on the result: The classification does not change; a coefficient table for classifying new observations by hand is added.

On the validation data: UCLA equal-prior constants -23.1416, -25.3403, -20.5626 (the constants include the ln(prior) term). For crops all 25 Fisher coefficients are reported.

→ Fisher classification functions and canonical discriminant functions

Additional output: Tests of equality of group means (univariate Wilks Λ / F)

Concept: For each predictor on its own, it tests whether the group means are equal (equivalent to a one-way ANOVA). The univariate Wilks' Λ shows how much of the total variability in that variable lies within the groups; a small value means strong separation.

Everyday example: If Wilks' Λ = 0.40 for protein content, most of the variation in protein comes from differences between varieties, and protein on its own is a good separator. If Λ = 0.95 for kernel width, the varieties hardly differ in width.

What it does: For each explanatory variable, Wilks Λ = SS_within/SS_total and the one-way ANOVA F.

Effect on the result: The model does not change; a table showing how well each predictor alone separates the groups is added.

On the validation data: iris F 119.26, 49.16, 1180.16, 960.01; UCLA social Λ 0.6040, F 79.009.

→ Multivariate test statistics (Wilks' Λ, Pillai's trace, Hotelling-Lawley, Roy)

Additional output: Leave-one-out (LOO) cross-validation

Concept: Each observation is classified by a model built on the rest of the data with that observation left out. Because an observation cannot contribute to its own classification, this accuracy is less optimistic than the accuracy computed on the data used to build the model.

Everyday example: For each of the 150 lots, the model is rebuilt on the other 149 as though that lot had never been seen, and its variety is predicted. If resubstitution accuracy is 92% and LOO accuracy 86%, expecting about 86% success on new lots is more realistic.

What it does: Each observation is classified by the model fitted without it; a cross-validated classification table and accuracy are given.

Effect on the result: The model does not change; an accuracy estimate less optimistic than the resubstitution accuracy is added.

On the validation data: UCLA LDA LOO 0.7459; crops LDA with class proportions LOO 0.3333 (the cross-validation classification table is reported).

→ Leave-one-out (LOO)

Additional output: Stepwise variable selection (Wilks; F-to-enter/F-to-remove below) and Stepwise selection (Wilks): F-to-enter/F-to-remove thresholds

Concept: Stepwise selection enters or removes variables one at a time according to their contribution to separation; to enter, a variable's partial F must exceed a threshold, and to be removed it must fall below another. This suggests a subset that gives similar separation with fewer variables; but the selection depends on the data and may not be repeated on new data.

Everyday example: If the merchant wants to take only a few measurements rather than dozens, stepwise selection might first enter protein content, then kernel length, and find the contribution of width negligible. It then becomes possible to make do with just two measurements at the depot.

What it does: Entry with partial F >= the entry threshold (default 3.84), removal with < the removal threshold (2.71) (Wilks Λ criterion); the step table is reported, and the main model stays with all variables.

Effect on the result: The main model does not change (with the refit option off); a step table is added. The thresholds change the selected variables and their order.

On the validation data: iris entry order Petal.Length (F 1180.16), Sepal.Width (43.04), Petal.Width (34.57), Sepal.Length (4.72); UCLA social (79.01), outdoor (47.51), conservative (22.55). In UCI Wine, with the threshold pairs 3.84/2.71, 10/8 and 1/0.5, the selected variables, the entry order and the partial F and Wilks Λ of each step are reported in the step table.

→ Stepwise variable selection

Additional output: Covariance matrices (pooled, group, total)

Concept: Covariance matrices summarise the variances of the variables and how they vary together: the pooled within-group matrix is the shared structure LDA uses, the group matrices show each group's own structure, and the total matrix the structure of all the data regardless of group.

Everyday example: Looking at the group matrices, the merchant may see that in one variety long kernels are usually also wide, whereas in another length and width are hardly related. Such a difference is a hint to consider QDA instead of LDA.

What it does: Writes the pooled within-group, per-group and total covariance matrices.

Effect on the result: The model does not change; covariance tables are added.

On the validation data: UCLA pooled outdoor variance 16.9397.

→ Covariance and correlation matrix

Additional output: Canonical score plot (LD1 × LD2 + centroids) and Save predicted class, D scores and membership probabilities

Concept: The canonical score plot shows each observation in the plane formed by the first two discriminant functions; how distinct or overlapping the groups are, and where their centres lie, can be seen at a glance. The save option adds each observation's assigned class, scores and group membership probabilities to the dataset.

Everyday example: If in the plot the lots of the first variety gather in one corner while the second and third varieties partly overlap, most confusion will be between these two. With the saved probabilities, the merchant can send lots classified with less than 60% probability to the laboratory.

What it does: Dis1 x Dis2 scatter and group centroids (the numeric output does not change); writing the class, canonical scores and posterior probabilities to the dataset.

Effect on the result: The numerical results do not change; a plot and new columns in the dataset are added.

On the validation data: In iris, UCLA and crops, the assigned class, the posterior probabilities and (with the sign rule) the discriminant scores are saved for every row.

→ Fisher classification functions and canonical discriminant functions

kNN: number of neighbours k, Kernel: radius r, Kernel (Uniform / Normal / Epanechnikov / Biweight / Triweight), Distance metric (Mahalanobis (pooled covariance) / Diagonal (pooled variances) / Euclidean)

Concept: Among the non-parametric methods, kNN classifies a new observation by the groups of its k nearest observations, and the kernel method by the local density of each group within a given radius. The distance metric sets how 'closeness' is measured: Mahalanobis distance takes the variables' scales and their relationships into account, Euclidean distance does not.

Everyday example: With k = 5, a new lot counts as the second variety if 4 of the 5 lots most similar to it are of the second variety. Because length is measured in millimetres and protein in per cent, Mahalanobis distance stops the difference in units and the link between length and width from distorting the result.

What it does: Nonparametric classification: k nearest neighbours (Mahalanobis FULL or Euclidean distance; equal or proportional prior) and kernel density estimation (radius r, pooled within-group V, full sample).

Effect on the result: Effective only with the kNN and kernel methods; classes, posterior probabilities, resubstitution and LOO accuracy change.

On the validation data: iris: under k = 5 (FULL, equal prior), k = 5 (Euclidean, proportional prior), normal r = 0.8, uniform r = 1.5 and Epanechnikov r = 1, the resubstitution and LOO accuracies, misclassified rows and posteriors are reported.

→ Number of nearest neighbours (k) and the neighbourhood graph

Stepwise selection (Wilks): refit the model with the selected variables (checkbox; default off)

Concept: Stepwise selection only produces a suggestion table; if this option is ticked, the model is rebuilt with the variables retained by the selection and all results are reported for this smaller set. Because the selection was made on the same data, the rebuilt model's performance may be somewhat optimistic.

Everyday example: If stepwise selection suggested only protein content and kernel length, the merchant ticks this option to rebuild the model with these two measurements and sees how far its classification accuracy differs from that of the three-measurement model.

What it does: If checked, the main model is refitted with the variables remaining after stepwise selection, on the same observations; the step table is kept and the refitted variables are written. If unchecked, the model keeps all variables. If fewer than 2 variables or all candidates are selected, the model does not change and a note is written.

Effect on the result: Classification accuracy, canonical functions, Wilks Λ and the other additional outputs change according to the selected variable set. The step table (order, partial F, Wilks Λ) and the entry/removal thresholds do not change.

On the validation data: In simulated data (3 classes x 30 observations, x1-x5) with the default thresholds x1 (F = 55.7803644115) and x2 (F = 5.6804512930) are selected; with equal prior probability the refitted LDA accuracy is 0.7111111111 for {x1, x2} and 0.7333333333 for {x1, x2, x3, x5}. With the threshold pair 10/8 only x1 is selected, and with 1/0.5 x1, x2, x5, x3 are selected.

→ Stepwise variable selection

Example on real data

Data: UCLA OARC discrim (job type, n = 244)

Variables: target job (1 customer service, 2 mechanic, 3 dispatcher), explanatory outdoor, social and conservative (interest scores)

  • Question: To what extent do the interest scores outdoor, social and conservative separate three job groups (1 customer service, 2 mechanic, 3 dispatcher)?
  • Assumption: Box's M = 26.12, F(12, 233002) = 2.14, p = .012; the group covariance matrices are not equal at the 5% level, but the test is very sensitive in large samples (LDA is acceptable with the α = .001 criterion; a comparison with QDA is recommended).
  • Result: The two canonical functions together are significant, Wilks Λ = 0.364, χ²(6) = 242.55, p < .001; the second function alone is also significant (Λ = 0.757, χ²(2) = 66.72, p < .001).
  • Effect size: The first function has eigenvalue 1.08 (77.1% of the variance, canonical r = .72), the second 0.32 (22.9%, r = .49).
  • Additional finding: The first function is determined mostly by social (-0.83) and conservative (0.52), the second by outdoor (0.93); the customer service group lies at the negative end of the first function (-1.22), the dispatcher group at the positive end (1.42). With equal priors the resubstitution accuracy is .75 and the 5-fold CV accuracy .75 (SD = .08).
  • APA: LDA (N = 244, k = 3): F1 eigenvalue = 1.08, Rc = .72, Λ = .364, χ²(6, N = 244) = 242.55, p < .001; accuracy = .75; Box's M = 26.12, F(12, 233002) = 2.14, p = .012.

Related concepts

Fisher classification functions and canonical discriminant functionsMultivariate test statistics (Wilks' Λ, Pillai's trace, Hotelling-Lawley, Roy)Stepwise variable selectionLeave-one-out (LOO)Prior class probabilities (discriminant analysis)Covariance and correlation matrixBox's M test (equality of covariance matrices)Number of nearest neighbours (k) and the neighbourhood graph

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.