Statistics Glossary → Variable Clustering (VARCLUS)
Variable Clustering (VARCLUS)analysis
A method that sorts highly correlated numeric variables into groups, represents each group by a single summary component and shows the best representative variable of each group.
Concept
When a survey or dataset contains many variables, some of them may really be measuring the same thing. Variable clustering groups variables rather than observations: variables in the same cluster are strongly related, those in different clusters weakly. It shows which dimensions a large set of items falls into and can reduce the number of variables used in an analysis.
Similarity between variables is measured by correlation (the distance is usually 1 − |r|). Clusters are formed either by merging the most similar variables step by step (agglomerative) or by starting with all variables in one cluster and splitting the most heterogeneous cluster in two (divisive). Each cluster is represented by its first principal component, which best summarises the shared variation of its variables. Two R² values are computed for each variable: with its own cluster's component (R²_own) and with the component of the nearest other cluster (R²_next). A small ratio (1 − R²_own) / (1 − R²_next) shows that the variable sits in the right cluster; the variable with the smallest ratio can be chosen as the cluster's representative.
Variable clustering is exploratory; the number and content of the clusters may change with the algorithm and the number of clusters chosen, and interpretation needs subject-matter support. The method considers only linear relationships (correlations) and needs a sample large enough for the correlation matrix to be reliable. Each variable is assigned to a single cluster; for variables that load on several dimensions, factor analysis gives a more flexible picture.
When to use it
- To sort many related numeric variables (scale items, sensor readings, economic indicators) into meaningful groups, or to reduce the number of variables by choosing one representative from each group.
- If variables may load on several latent dimensions at once, or a scale structure is to be tested, exploratory or confirmatory factor analysis is more suitable.
- To group observations (people, fields, customers) rather than variables, use hierarchical or k-means clustering; to concentrate the information in a few components, principal component analysis is appropriate.
An everyday example
A restaurant chain gives customers a 12-question satisfaction survey: taste of the food, portion size, freshness, attentiveness of the waiters, speed of service, accuracy of the order, cleanliness, noise, comfort of the seating and so on. Variable clustering might sort the 12 questions into three clusters: 'food' (taste, portion, freshness), 'service' (attentiveness, speed, order accuracy) and 'setting' (cleanliness, noise, seating). In each cluster the question with the lowest R² ratio, say 'taste of the food', is chosen as its representative, so the chain could ask just 3 questions instead of 12 in a short survey.
Parameters in MerQur
Variables
Concept: The numeric variables to be clustered. The clustering rests only on the correlations among these variables, so which variables are chosen determines the result directly. A variable that never varies (a constant) has an undefined correlation.
Everyday example: In the restaurant example the 12 survey questions are selected. Had customer age been added too, it might form a cluster of its own if it is not strongly related to any satisfaction question.
What it does: Selects the numeric variables to be clustered (listwise missing deletion; works on the correlation matrix).
Effect on the result: The clustering depends only on the correlation structure of the selected variables. The correlation is undefined for a constant (zero variance) column; MerQur gives a clear message in the interface language.
Algorithm (Average linkage (1-|r|, agglomerative), Divisive (SAS VARCLUS type))
Concept: Sets how the clusters are formed. The average-linkage agglomerative method starts with each variable as its own cluster and merges the most similar clusters step by step; the divisive method starts with all variables together and proceeds by splitting the most heterogeneous cluster in two, reassigning variables between clusters where needed.
Everyday example: The agglomerative method first merges the two most similar questions (e.g. 'taste' and 'freshness'), then adds 'portion' to them. The divisive method first splits all questions into 'food' and 'non-food', then splits the second group into 'service' and 'setting'.
What it does: Average linkage performs hierarchical agglomerative clustering with the 1 - |r| distance; the divisive method splits the cluster with the largest second eigenvalue (orthoblique rotation), reassigns the variables and applies a search phase. Average linkage suits quick exploration, the divisive method a component-based cluster structure.
Effect on the result: Changes cluster membership and all cluster statistics; the output definitions stay the same. The divisive method adds a "Split history" section and uses at most 10 reassignment iterations and 1000/p search steps.
On the validation data: JobRat, K = 3: average linkage total proportion explained 0.6695; divisive 0.6715. USJudgeRatings K = 3 divisive 0.9565, mtcars second-eigenvalue threshold 1 (2 clusters) 0.7536.
MAXEIGEN (divisive)
Concept: In the divisive method, the threshold that decides whether a cluster is split further. If a cluster's second eigenvalue is below the threshold, the cluster is considered well summarised by one dimension and is not split; a threshold of 1 applies a rule similar to the Kaiser criterion.
Everyday example: If the 'service' cluster's second eigenvalue is 0.7 and the threshold is 1, the cluster is not split: the three service questions are adequately captured by one 'service quality' dimension. Had the second eigenvalue been 1.3, the cluster might split into, say, 'waiter attentiveness' and 'kitchen speed'.
What it does: In the divisive method, stops splitting when the largest second eigenvalue falls below this threshold (Kaiser-like criterion; default 0 when the number of clusters is given).
Effect on the result: The number of clusters can be fewer than K (including 1 cluster). The field is disabled in average linkage.
On the validation data: JobRat, divisive, K = 20: 0 -> 13 clusters; 0.7 -> 4 clusters (proportion 0.7236); 1 -> 2 clusters (0.6129); 2 (K = 3) -> 1 cluster (0.5036; at step 1 the second eigenvalue 1.7727 < 2).
Number of clusters (K)
Concept: Sets how many clusters are formed (in the divisive method it is an upper limit). Fewer clusters give a simpler but cruder summary, more clusters a more detailed structure that is harder to interpret; the proportion of total variance explained by the clusters guides this choice.
Everyday example: With 2 clusters the restaurant gets a 'food' versus 'non-food' split; with 3, 'service' and 'setting' appear separately. If 3 clusters explain 70% of the total variance and 4 explain only 74%, the fourth cluster adds little.
What it does: Sets the number of clusters cut in average linkage and the upper limit in divisive (2-20).
Effect on the result: Cluster membership and the proportion of variance explained. If K exceeds the number of variables, it is reduced to the number of variables.
On the validation data: JobRat, average linkage: K = 2 -> 0.5602, 3 -> 0.6695, 4 -> 0.7169, 5 -> 0.7668; K = 20 is reduced to the number of variables (13) (proportion 1.0000).
Additional output: R² and 2nd eigenvalue
Concept: Gives in detail each cluster's first and second eigenvalues and proportion of variance explained, and each variable's R² with its own cluster (R²_own) and with the nearest other cluster (R²_next). A high R² with its own cluster and a low one with the other indicates a well-separated variable.
Everyday example: If 'speed of service' has R²_own of 0.75 and R²_next of 0.20, it clearly belongs to the 'service' cluster. If both values for 'cleanliness' are around 0.40, the question relates to perceptions of both setting and service, and its cluster is debatable.
What it does: Prints the cluster summary (1st and 2nd eigenvalue, proportion explained) and the R²_own / R²_next values per variable at full precision (the main table gives the 1 - R² ratios with 4 decimals).
Effect on the result: An additional table.
On the validation data: JobRat divisive: Communication_Skills R²_own 0.6403, R²_next 0.3599; the 2nd eigenvalue of cluster 1 is 0.7093.
Additional output: Inter-cluster correlations
Concept: Shows the correlations among the components that summarise the clusters. Low correlations between clusters mean they are well separated; high ones suggest that the clusters share a common higher-level dimension.
Everyday example: If the correlation between the 'food' and 'service' components is 0.55, customers who like the food also tend to like the service; perhaps a general sense of 'satisfaction with the restaurant' lies behind both.
What it does: Gives the correlation matrix among the cluster first components and the standardized scoring coefficients (v₁/√λ₁).
Effect on the result: An additional table.
On the validation data: JobRat divisive: r(C1, C2) = 0.6085, r(C1, C3) = 0.5922, r(C2, C3) = 0.4871; Communication_Skills coefficient 0.2122.
Additional output: Dendrogram and k scan
Concept: The dendrogram is a tree diagram showing in what order and at what level of similarity the variables merge. The k scan shows how the proportion of variance explained changes as the number of clusters grows; the point where the curve starts to flatten suggests a reasonable number of clusters.
Everyday example: In the restaurant's dendrogram 'taste' and 'freshness' merge near the bottom, while the 'food' and 'service' clusters join higher up. If the variance explained rises quickly up to 3 clusters and slows afterwards, 3 clusters is a good choice.
What it does: Gives the 1 - |r| average-linkage dendrogram and the curve of the proportion of variance explained for k = 1...10.
Effect on the result: The plot tab turns into the dendrogram and the curve; a scan line is added to the text. Even with the divisive method selected, the tree and scan are average linkage (stated in the heading).
On the validation data: JobRat k = 1...10: 0.504, 0.560, 0.669, 0.717, 0.767, 0.809, 0.846, 0.877, 0.912, 0.940 (computed from each cut of the average linkage tree).
Example on real data
Data: SAS/STAT PROC VARCLUS example (JobRat job ratings)
Variables: 13 job rating items (communication, problem solving, learning ... integrity)
- Question: How do 13 job rating items (n = 103) divide into groups of similar (highly correlated) items, and which item best represents each group?
- Method: Distance 1 - |r|, average-linkage agglomerative clustering, 3 clusters; each cluster is summarized by its first principal component.
- Result: Cluster 1 (interpersonal): Interest_in_People, Interpersonal_Sensitivity, Integrity, Judgement_under_Pressure (1st eigenvalue 3.04, proportion explained 0.760). Cluster 2 (cognitive and work skills): 8 items (1st eigenvalue 4.66, proportion 0.583). Cluster 3: Physical_Ability alone.
- Model fit: The three cluster components explain 66.95% of the total variance.
- Representatives: The items with the lowest 1 - R² ratio represent their clusters: Interest_in_People (0.210), Observational_Skills (0.420), Physical_Ability (0).
- Limitation: Physical_Ability is weakly related to the other items, so it forms a cluster on its own; the number of clusters (3) is the researcher's choice.
- APA: Average-linkage variable clustering gave three clusters; the cluster components explained 67.0% of the total variance.
Related concepts
Linkage method (Ward, complete, average, single)Proportion of explained varianceDendrogram and cophenetic correlationEigenvalue and eigenvectorChoosing the number of clustersCovariance (correlation) structure (UN, VC, AR(1), exchangeable)Coefficient of determination (R²) and adjusted R²Covariance and correlation matrix
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.