MerQuR+Statistics Glossary

Statistics Glossary → Principal Component Analysis (PCA)

Principal Component Analysis (PCA)analysis

A dimension reduction method that condenses many correlated numeric variables into a few uncorrelated components, ordered by importance, while keeping most of the information.

Concept

When several variables partly measure the same thing (e.g. income, house prices and education level), examining them one by one is both laborious and misleading. Principal component analysis (PCA) answers the question 'how far can the total variability in these variables be described by fewer summary variables?'. It is used to see the common structure of items in scale development, to summarise highly correlated predictors in regression, and to display multidimensional data graphically.

Each component is a weighted sum of the variables: PC = a₁x₁ + a₂x₂ + … + a_px_p. The first component is the direction that captures the greatest variability in the data; the second captures the greatest remaining variability subject to being uncorrelated with the first, and so on. The variance each component explains is an eigenvalue of the correlation (or covariance) matrix; in the correlation matrix the eigenvalues add up to the number of variables, so 'eigenvalue / p' gives the share of variance explained by the component. The loadings show how strongly each variable is related to a component and are used to give the component a meaning.

PCA is not a model but a re-expression of the data; it needs no distributional assumption, but it captures only linear relationships and is sensitive to outliers. If the variables are in different units, the correlation matrix (standardised data) should be used. How many components to keep is decided by the Kaiser criterion, a target for explained variance, or parallel analysis; parallel analysis is usually the most reliable. PCA tries to account for all of the variance; if the aim is to model the latent structure behind items through their shared variance, exploratory factor analysis is more suitable.

When to use it

  • When many correlated numeric variables are to be reduced to fewer summary variables, the main axes of variation in the data are to be seen, or uncorrelated component scores are needed for later analyses.
  • If the aim is to interpret latent constructs (factors) behind the items theoretically, exploratory factor analysis is preferred.
  • If the main aim is only visualisation and the relationships are non-linear, t-SNE or UMAP can be used; if the starting point is a distance or similarity matrix, multidimensional scaling.

An everyday example

A municipality wants to compare 40 neighbourhoods on ten indicators such as household income, share of university graduates, house prices, green space per person, population density and average age. Instead of ten separate maps, PCA notices that the indicators move together: income, education and house prices are high or low together. The first component gathers these into a 'prosperity' axis and explains 55% of the total variability; the second is an 'urban crowding' axis setting density against green space. The neighbourhoods can then be placed on a single two-dimensional chart.

Parameters in MerQur

Variables

Concept: The variables to be split into components; the analysis rests on their correlation matrix. The components reflect only the shared variability among the selected variables; a column that never varies (a constant) carries no information and is dropped.

Everyday example: If the municipality added a meaningless number such as 'district code' to its ten indicators, the components would be distorted. An indicator with the same value in every neighbourhood (e.g. a column of 1s) carries no difference at all and is dropped.

What it does: Selects the numeric variables to be decomposed into components; constant columns are dropped and reported in the text with the warning "Column(s) removed from the analysis".

Effect on the result: The size of the correlation matrix; all eigenvalues, loadings, KMO and the Bartlett test are computed from the selected variables.

On the validation data: M255 12 items: KMO 0.9344, Bartlett χ² 8676.71 (df 66).

Number of components

Concept: The number of components kept as a summary. Together, all the components account for all of the variance in the data; keeping only a few means giving up some information in exchange for simplicity. The eigenvalues are not affected by the number kept.

Everyday example: If the municipality keeps two components, it summarises the neighbourhoods on 'prosperity' and 'crowding' axes and retains, say, 70% of the total variability. Keeping five retains 90% but makes the summary more complex.

What it does: The number of components kept under the "Manual" criterion (2..min(p, n-1)).

Effect on the result: The number of loading, coefficient, score and h² columns, and the cumulative variance. The eigenvalues do not change.

On the validation data: k = 2: cumulative 62.3%; k = 3: 68.3%; k = 5: 78.1%; k = 12: 100%. The loadings of the first two components are independent of k (ITEM13 0.7273 and -0.4492).

Missing Data (Drop (listwise) / Fill with mean / Fill with median / Fill with KNN)

Concept: Either removes rows with missing values from the analysis or fills the missing value with the mean, the median or an estimate from the most similar rows. Mean filling slightly weakens variances and correlations; with a lot of missing data, the components can be affected.

Everyday example: If green-space data are missing for three neighbourhoods, these can be deleted, or their green space can be filled with the average of the five neighbourhoods most similar on the other indicators.

What it does: Drops the row with missing values, or fills with the column mean, the median or the mean of the 5 nearest neighbours (nan-Euclidean, raw scale).

Effect on the result: n and the correlation matrix.

On the validation data: M255 λ1: drop n = 1365, 6.2491; mean n = 1428, 6.2374; median 6.2347; KNN 6.2959. airquality: drop n = 111, λ1 2.3599; mean n = 153, 2.1820.

→ Single imputation (mean, median, kNN filling)

Number-of-components criterion (Manual / Kaiser / Cumulative variance target / Parallel analysis (Horn))

Concept: Ways of deciding how many components to keep. The Kaiser criterion keeps components with an eigenvalue above 1, i.e. those explaining more variance than a single variable; the cumulative variance criterion adds components until a set percentage is explained; parallel analysis compares the observed eigenvalues with those from completely random data.

Everyday example: Parallel analysis generates random data of the same size as the municipality's indicators but with no relationships among them. If only the first two eigenvalues of the real data exceed those of the random data, only two components reflect a 'real' structure.

What it does: Kaiser (λ > 1; in covariance, λ > mean); cumulative variance >= target; parallel analysis (observed λ greater than the 95th percentile or the mean of 100 normal simulations). If the recommendation is < 2, 2 are kept for the chart.

Effect on the result: The number of retained components and the loading, score and h² columns that depend on it.

On the validation data: M255 Kaiser 2, variance (80%) 6, parallel 2; USArrests Kaiser 1 (kept 2), parallel 1; Crime Kaiser 2, parallel 1. Because the parallel analysis thresholds are simulation-based, they carry Monte Carlo error.

→ Kaiser criterion (eigenvalue > 1)

Parallel analysis criterion (95th percentile (Glorfeld) / Mean (Horn))

Concept: Sets which random threshold an observed eigenvalue must exceed in parallel analysis: the mean of the random eigenvalues (Horn) or their 95th percentile (Glorfeld). The 95th percentile is a stricter threshold and lowers the chance of keeping components that are large by chance.

Everyday example: If the third component's eigenvalue is a little above the random-data average but small enough to occur in 5% of random datasets, Horn's criterion keeps it, while Glorfeld's leaves it out as 'explainable by chance'.

What it does: The simulation threshold against which the observed eigenvalue is compared; the default is the 95th percentile (Glorfeld 1995).

Effect on the result: The number of components recommended by parallel analysis.

On the validation data: M255 2 in both; USArrests and Crime 1 in both (kept 2).

→ Parallel analysis (Horn) and the MAP criterion

Variance target (%)

Concept: In the cumulative variance criterion, the total percentage of variance the kept components should jointly explain. A high target means more components and less information lost; a low target a simpler summary.

Everyday example: If the municipality wants at least 80% of the differences in the indicators preserved, components are added until this level is reached; that might mean keeping three components, for example.

What it does: The percentage of explained variance to be reached under the cumulative variance target criterion.

Effect on the result: The recommended and retained number of components.

On the validation data: M255 50%: recommended 1 (kept 2); 80%: 6; 90%: 9; 99%: 12.

→ Proportion of explained variance

Save component scores to the dataset

Concept: Component scores are each observation's position on each component, given with mean 0 and standard deviation 1. As uncorrelated summary variables, the scores can be used in later analyses (e.g. regression or clustering).

Everyday example: Each neighbourhood gets a 'prosperity score' and a 'crowding score'. A neighbourhood with a prosperity score of +1.5 is clearly above average; the municipality can use these scores directly when planning how to allocate resources.

What it does: Adds the standard component scores (mean 0, SD 1) as PCA_PC1.. columns; if rotation is selected, the rotated scores are saved.

Effect on the result: Adds new columns to the dataset; the tables and eigenvalues do not change.

On the validation data: In correlation PCA the USArrests state 1 PC1 score is 0.6195; in covariance PCA row 1 is 0.7739. In the rotated scores row 1 is 1.0046.

→ Factor and component scores

Rotation (None / Varimax / Promax (m = 4))

Concept: Rotation repositions the axes without changing the total variance explained by the kept components; the aim is an easier-to-interpret loading pattern in which each variable loads strongly on only a few components. Varimax keeps the components uncorrelated (orthogonal); promax allows them to be correlated.

Everyday example: If before rotation all the indicators load moderately on the first component, it is hard to name. After varimax, income and education stand out on one component and density and green space on the other; if prosperity and crowding are expected to be related in reality, promax is more suitable.

What it does: Rotates the retained loadings with varimax (orthogonal; Kaiser-normalized, criterion 1e-14) or promax (oblique, κ = 4).

Effect on the result: Loadings, scores and the variable-component matching; promax also gives the component correlations. The eigenvalues do not change.

On the validation data: Varimax M255 ITEM13 0.8410; promax USArrests component correlation 0.3069, M255 0.6388.

→ Factor rotation (varimax, promax, oblimin)

Covariance-matrix PCA

Concept: PCA carried out on unstandardised data, i.e. on the covariance matrix. Variables with large variances then dominate the components, so it is preferred only when all variables are measured in the same unit and differences in scale are meaningful.

Everyday example: If house prices are measured in hundreds of thousands and the share of graduates in per cent, in a covariance PCA the first component consists almost entirely of house prices. If all the indicators are, say, spending categories in the same currency, a covariance PCA can make sense.

What it does: PCA without standardization; the eigenvalues and loadings are in raw units.

Effect on the result: Variables with large variance dominate the components; the text says "(covariance matrix)" and applies the matching thresholds to the standardized loadings.

On the validation data: USArrests λ1 = 7011.11 (96.6%; Assault dominates); Temperature λ = 154.3106 and 9.1630.

→ Covariance and correlation matrix

Communalities (h²) + variable MSA

Concept: The communality (h²) is the part of a variable's variance explained by the kept components; a low h² shows that the variable is poorly represented in the summary. MSA measures, by looking at partial correlations, how much each variable shares the common structure with the others; variables with low MSA contribute little to the analysis.

Everyday example: If average age has an h² of 0.20, the prosperity and crowding components reflect only 20% of the neighbourhoods' differences in age, and age should be examined separately. If an indicator's MSA is very low, it has little in common with the others and dropping it may be considered.

What it does: Gives the variance of each variable explained by the retained components (h²) and the variable-level measure of sampling adequacy (MSA).

Effect on the result: Adds h² and MSA columns to the table; the component solution does not change.

On the validation data: M255 k = 2 h² ITEM13 0.7307; MSA ITEM13 0.9225 (the diagonal of the anti-image correlation matrix).

→ Communality

Chart (Scree / Biplot / Loading plot / Component scores)

Concept: The scree plot draws the eigenvalues in order, and the 'elbow' where the curve flattens gives a hint about the number of components. The biplot shows observations and variables in the same plane, the loading plot the variables' loadings on the components, and the score plot the observations' component scores.

Everyday example: In the biplot the neighbourhoods appear as points and the indicators as arrows: if the 'income' arrow points to the right, neighbourhoods on the right rank high on income. The municipality can see in one picture which neighbourhoods stand out on which features.

What it does: Selects the chart to be drawn.

Effect on the result: Does not change the numeric output; in covariance PCA the score and biplot charts draw the standard scores.

→ Scree plot

Example on real data

Data: UCLA OARC M255 survey data (J. Sidanius; 1428 records, 1365 complete)

Variables: variables ITEM13-ITEM24 (12 attitude items)

  • Question: How can the 12 items (ITEM13-ITEM24) of the UCLA M255 survey be summarized in fewer components?
  • Assumption: KMO = 0.934 (excellent) and Bartlett's test of sphericity χ²(66) = 8676.71, p < .001; the correlation matrix is suitable for component analysis. With listwise deletion 1365 of the 1428 rows are used (4.4% loss).
  • Result: The first eigenvalue of the correlation matrix is 6.249 (52.1% of the variance), the second 1.229 (10.2%); only two components exceed the Kaiser criterion (λ > 1). The 5 requested components explain 78.1% of the total variance.
  • Loading pattern: All items load on PC1 between 0.59 and 0.82 (a general factor); PC2 is a contrast component: ITEM19, ITEM20, ITEM18 and ITEM22 load positively, ITEM13, ITEM14 and ITEM15 negatively.
  • Descriptive: PC3-PC5 each explain 5-6% of the variance and rest on one or two items (e.g. ITEM24 0.54 on PC3, ITEM20 0.51 on PC4); 2 components are enough for interpretation.
  • Limitation: PCA summarizes the total variance and does not separate measurement error; if a latent structure is sought, exploratory factor analysis should be preferred. Component signs are arbitrary.
  • APA: A principal component analysis of the correlation matrix (N = 1365; KMO = .93; Bartlett χ²(66) = 8676.71, p < .001) showed that two components had eigenvalues above 1 (6.25 and 1.23; 62.3%); five components explained 78.1% of the variance.

Related concepts

Factor rotation (varimax, promax, oblimin)Factor (component) loadingsEigenvalue and eigenvectorParallel analysis (Horn) and the MAP criterionKaiser criterion (eigenvalue > 1)Factor and component scoresCommunalityScree plot

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.