MerQuR+Statistics Glossary

Statistics Glossary → Latent Class Analysis (LCA)

Latent Class Analysis (LCA)analysis

A model that estimates how many unobservable subgroups (latent classes) lie behind the patterns of answers to categorical questions (yes/no, present/absent and so on), and the probability that each individual belongs to each group.

Concept

In many studies the group of interest cannot be measured directly: groups such as 'students at risk', 'those who are truly ill' or 'loyal customers' can only be inferred from patterns of answers on a set of categorical indicators. Latent class analysis (LCA) assumes that behind these indicators lie a few hidden subgroups with distinct response tendencies, and estimates the number, size and characteristics of these groups from the data.

The model estimates two kinds of parameters: the share of each class in the population, and the probability of a given answer to each indicator within each class (the class-conditional response probabilities). The key assumption is local independence: for individuals in the same class the indicators are independent of one another, so all of the association between indicators comes from differences between classes. Estimation uses the EM algorithm; posterior class probabilities are computed for each individual, who can be placed in the most probable class (modal assignment). The number of classes is chosen by comparing models with different numbers of classes using information criteria such as BIC and likelihood ratio tests such as LMR and BLRT.

The classes are a summary of the data, not groups known to exist; the chosen number of classes should be judged jointly on the criteria, the sample size and interpretability. Because the EM algorithm can get stuck in local optima, many random starts are needed. Entropy shows how clearly individuals are assigned to classes (a value near 1 means good separation). If local independence is seriously violated, the model may suggest more classes than needed to account for the extra association between indicators.

When to use it

  • When individuals are to be divided, by their answer patterns on a set of categorical indicators (binary or with several categories), into subgroups whose number is not known in advance (survey typologies, agreement between diagnostic tests, behaviour patterns).
  • If the indicators are continuous measurements, latent profile analysis is used; if the latent structure is thought of as a continuous dimension (e.g. an ability level) rather than groups, item response theory or factor analysis.
  • If a quick exploratory grouping without a probability model or fit comparison is enough, hierarchical clustering with a suitable distance measure (e.g. Jaccard, Gower) can also be used.

An everyday example

A school counsellor asks 300 pupils six yes/no questions, such as whether they do their homework on time, take part in class, are often absent, or feel lonely at school. Looking at the answers one by one gives a confusing picture; latent class analysis might find, behind these answer patterns, three groups: 'well adjusted and engaged' (55%), 'behind academically but sociable' (30%) and 'both often absent and lonely' (15%). The probability of belonging to each group is also computed for every pupil; those who appear to belong to the third group with 90% probability can be invited to talk first.

Parameters in MerQur

Maximum number of classes

Concept: The number of latent classes cannot be read directly from the data, so models with 1, 2, 3… classes are estimated separately and compared. The maximum number of classes is the upper limit of this comparison; the selected number cannot fall outside this range.

Everyday example: The counsellor tries models with between 1 and 5 classes. Had the limit been set at 3, four distinct groups of pupils could never have been seen, even if they existed in the data.

What it does: Models with 1..K classes are estimated and compared with the criteria.

Effect on the result: Sets the range of models compared; the K selected by the criterion is limited to this range.

On the validation data: (carcinoma, K = 1..4 log-likelihood): -524.4648, -317.2568, -293.7050, -289.2858 (global maximum). With maximum K of 4, 3 is selected by BIC; with 6, K = 5 and 6 give -286.8843 and -286.1866, and again 3 is selected (the K = 6 parameters are weakly identified: different solutions at the same log-likelihood).

→ Choosing the number of clusters

Number of classes to report (0 = automatic)

Concept: Sets which of the compared models is reported in detail. With automatic selection the information criterion picks the best model; the researcher can also ask for a particular number of classes for reasons of interpretability or theory.

Everyday example: Even if BIC suggests four classes, if the fourth class has only 8 pupils and carries no meaningful interpretation, the counsellor may choose to report the three-class model in detail.

What it does: Number of classes of the model reported in detail; with 0 the best K by the selection criterion is reported.

Effect on the result: Class proportions, conditional probabilities, fit and class assignments change with the selected K.

On the validation data: 2 classes: AIC 664.514, BIC 706.074. 3 classes: class proportions 0.4447, 0.3736, 0.1817; G² 15.26171, X² 20.50335, residual df 95, npar 23. 4 classes: log-likelihood -289.2858 (the global solution; a local maximum at about -289.79 also exists).

→ Choosing the number of clusters

Number of random starts and Random seed

Concept: The EM algorithm starts from a random point and may reach a local peak rather than the highest point of the likelihood. Running many starts and seeing the best log-likelihood repeated across many of them increases confidence that the solution found is the true best; the seed makes the result reproducible.

Everyday example: Someone looking for the highest hill in foggy, hilly terrain who sets off from a single point may climb a small hill and stay there. If most of 20 walkers starting from 20 different points meet at the same summit, the highest hill has most likely been found.

What it does: EM can get stuck at a local maximum; the best log-likelihood and in how many starts it was repeated are reported (Linzer & Lewis 2011).

Effect on the result: Many starts raise the chance of finding the global maximum; with a single start a local solution may be reported.

On the validation data: In carcinoma, 1, 20, 100 starts and seed 7 give the same solution. gss82 K = 3: 100 starts -2754.5454 (global maximum); 1 start -2762.0048 (a local maximum).

→ Random starts and local optima

Missing indicators (Listwise deletion / FIML)

Concept: With listwise deletion, an individual missing even one indicator is dropped entirely. FIML (full information maximum likelihood) uses whatever indicators each individual answered; under the assumption that missingness can be explained by the observed answers (MAR), it gives unbiased estimates and preserves the sample.

Everyday example: If 25 of 300 pupils left the loneliness question blank, listwise deletion leaves these 25 out altogether. FIML keeps them in the model, using their answers to the other five questions.

What it does: Listwise deletion drops rows with missing indicators; in FIML missing indicators do not enter the likelihood (maximum likelihood under MAR).

Effect on the result: n and therefore all estimates change; FIML also uses rows with missing indicators.

On the validation data: (election, 12 indicators, K = 3): Listwise n = 1311 (no covariate) and 1300 (PARTY), log-likelihood -16714.6591 and -16222.3233; FIML n = 1785 and 1760, -21311.5357 and -20609.2728.

→ Listwise and pairwise deletion

Covariates

Concept: Covariates are variables that predict which class individuals belong to; class membership is linked to them by multinomial logistic regression. The question 'which kinds of individuals are more likely to be in a given class?' is thus answered within the same model.

Everyday example: If the counsellor adds year group as a covariate, the model might give a result such as 'pupils in the upper years are twice as likely as those in the lower years to be in the absent-and-lonely group'.

What it does: Predicts class membership with multinomial logit (one-step latent class regression; class 1 is the baseline category).

Effect on the result: Class proportions depend on the covariate; the log-likelihood, conditional probabilities and class assignments may change.

On the validation data: election PARTY: constant 2/1 -3.81813 (SE 0.31109), PARTY 2/1 0.79327 (0.06232), constant 3/1 1.16155 (0.17989), PARTY 3/1 -0.57436 (0.06401). In the cheating data the GPA covariate is also estimated with K = 2.

→ Covariate

Selection criterion (BIC / SABIC / AIC / CAIC)

Concept: Information criteria combine the model's fit to the data (the log-likelihood) with a penalty for the number of parameters; the model with the smallest value is chosen. AIC applies a light penalty, BIC and CAIC heavier ones that lean towards fewer classes; SABIC lies in between and is less conservative than BIC in small samples.

Everyday example: A four-class model may explain the data slightly better than a three-class one but uses more parameters. BIC may decide the small gain is not worth the extra complexity and pick three classes, while AIC may pick four.

What it does: Information criterion for selecting the number of classes; the K with the smallest value is selected.

Effect on the result: Can change only the selected K; the estimates for each K do not change.

On the validation data: (carcinoma, K = 1..4): AIC 1062.930, 664.514, 633.410, 640.572; BIC 1082.324, 706.074, 697.136, 726.463; SABIC 1060.196, 658.655, 624.427, 628.464; CAIC 1089.324, 721.074, 720.136, 757.463; all four select K = 3.

→ Information criteria (AIC, AICc, BIC)

Maximum EM iterations

Concept: The EM algorithm improves the estimates a little with each iteration and stops when the change becomes very small. The maximum number of iterations should be high enough that the algorithm is not stopped before converging; otherwise an unfinished solution is reported.

Everyday example: In each round the class shares and response probabilities become a little clearer; after a few hundred rounds the values stop changing. If the round limit is too low, the calculation is cut off before the pupils' class probabilities have settled.

What it does: Maximum number of EM iterations in each start.

Effect on the result: With few iterations the model may not converge and a lower log-likelihood is reported; MerQur gives a warning.

On the validation data: At 50 iterations K = 4 does not converge (log-likelihood -289.3868, lower than -289.2858) and MerQur gives an explicit warning; K = 1..3 the same.

→ Convergence and maximum iterations

BLRT bootstrap samples (0 = off)

Concept: The parametric bootstrap likelihood ratio test (BLRT) tests whether a K-class model is significantly better than a (K − 1)-class model. The distribution of the likelihood ratio is obtained from datasets simulated from the (K − 1)-class model; a significant p indicates that the extra class is needed.

Everyday example: The model generates hundreds of artificial school datasets on the assumption that 'there were really only two groups of pupils' and looks at how much the three-class model improves on them. If the improvement in the real data is larger than in almost any of these artificial schools, the third class is a real group.

What it does: Parametric bootstrap likelihood ratio test of K against K-1 (McLachlan 1987; Nylund et al. 2007).

Effect on the result: The fit does not change; the table gains a BLRT p column. p depends on the bootstrap samples and the seed.

On the validation data: (100 samples): Observed LR 414.416, 47.104, 8.838 (= 2·Δ log-likelihood); p 0.0099, 0.0099, 0.119.

→ Bootstrap likelihood ratio test (BLRT)

Extra outputs: save

Concept: Adds to the dataset each individual's most probable class (the modal class) and their probabilities of belonging to each class (posterior probabilities). The modal class gives a practical label; the probabilities show how certain that label is.

Everyday example: One pupil may be assigned to the 'absent and lonely' class with probability 0.95, another with 0.52. The second pupil's label is uncertain; the counsellor should base decisions not only on the label but also on these probabilities.

What it does: Adds the modal class and the posterior class probabilities to the dataset.

Effect on the result: The estimates do not change; new columns are added to the dataset.

On the validation data: carcinoma K = 3: gives the 118 x 3 matrix of posterior probabilities.

LMR test (always reported)

Concept: The Lo-Mendell-Rubin (LMR) test also compares a K-class model with a (K − 1)-class model, but uses a corrected approximate distribution of the likelihood ratio instead of simulation. Such special tests are needed because the usual chi-square test is not valid when comparing numbers of classes.

Everyday example: If the LMR test gives p < 0.001 for three classes and p = 0.40 for four, adding a third class clearly improves the model while a fourth adds nothing worthwhile; the counsellor goes ahead with three groups.

What it does: Compares the K-class model with the K-1-class model by an adjusted likelihood ratio; the MerQur correction is LR/(1 + 1/(dp·ln n)) (dp = parameter difference, Lo, Mendell & Rubin 2001).

Effect on the result: The fit does not change; an LMR p column that helps to select the number of classes is given.

On the validation data: carcinoma K = 3: 45.901 (p 2.48e-07); K = 4: 8.613 (p 0.376).

→ Lo-Mendell-Rubin (LMR) test

Example on real data

Data: Agresti 2002 carcinoma, R poLCA package (n = 118)

Variables: indicators A-G (binary carcinoma ratings of 7 pathologists)

  • Question: How many latent patient classes lie behind the binary (cancer present / absent) decisions of seven pathologists on 118 tissue slides?
  • Model selection: Models with 1-4 classes were estimated by EM (20 random starts). The BIC values are 1082.32, 706.07, 697.14 and 726.46; the 3-class model with the smallest BIC was selected (LMR p < .001: 3 classes are better than 2; for 4 classes LMR p = 0.376).
  • Result: Class proportions .44, .37 and .18; with modal assignment 51, 44 and 23 slides.
  • Classification quality: Entropy = 0.926 and mean posterior probabilities 0.88-0.99; the classes are well separated.
  • Descriptive: In the first class the pathologists say "present" almost unanimously, in the second class "absent"; the small third class consists of ambiguous slides on which the pathologists disagree (e.g. probabilities 0.49 and 0.37 for A and G).
  • Limitation: The absolute fit G² = 15.26 and X² = 20.50 (df = 95) does not follow the χ² distribution because of sparse cells; at K = 3 and 4 the two best log-likelihoods are close, so many starts are required.
  • APA: Latent class analysis was estimated with 1-4 classes (EM, 20 random starts, N = 118); the 3-class solution was selected (BIC = 697.14), with class proportions .44, .37, .18 (entropy = .93, LMR p < .001).

Related concepts

Choosing the number of clustersBootstrap likelihood ratio test (BLRT)Lo-Mendell-Rubin (LMR) testEM (expectation-maximization) algorithmRandom starts and local optimaCovariateInformation criteria (AIC, AICc, BIC)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.