MerQuR+Statistics Glossary

Statistics Glossary → Multidimensional Scaling (MDS)

Multidimensional Scaling (MDS)analysis

A method that starts from information on the distances or similarities between objects and draws a 'similarity map' in two or a few dimensions that distorts those distances as little as possible.

Concept

Suppose we have a set of objects and know how far apart (or how similar) every pair is: road distances between cities, similarity ratings consumers give to pairs of products, or differences between species computed from many traits. Multidimensional scaling (MDS) finds an arrangement of points (a configuration) that reflects these distances as faithfully as possible: similar objects end up close on the map and different ones far apart. The map makes the structure of relationships among the objects, and the dimensions behind it, visible.

The method tries to minimise the mismatch between the distances on the map and the original distances. The mismatch is usually measured by Kruskal's Stress-1: Stress-1 = √[Σ(d_ij − d̂_ij)² / Σd_ij²], where d_ij is the distance on the map and d̂_ij the corresponding original distance (or a suitable transformation of it). Metric MDS tries to preserve the distances themselves in proportion; non-metric MDS preserves only their rank order, staying faithful to information such as 'A and B are more alike than A and C'. This is more suitable for data meaningful only at the level of order, such as similarity judgements.

The smaller the stress, the better the map reflects the distances; by Kruskal's widely used guide, below 0.05 is good, below 0.10 acceptable and above 0.20 poor fit. Stress always falls as the number of dimensions grows, so the number of dimensions is chosen by looking at the stress scree plot together with interpretability. The axes of the map can be rotated and reflected; the dimensions are given meaning by comparison with known characteristics of the objects. Because the algorithm can get stuck in local solutions, trying several starts is useful.

When to use it

  • When distances or similarity judgements between objects are directly available or can be computed, and the structure of these relationships is to be seen on a map (product perception maps, similarity of species or sites, relationships among test items).
  • If the data consist of observations measured on variables and the aim is to summarise the variability with interpretable linear components, PCA is a more direct route.
  • If the main aim is to preserve local neighbourhoods and visualise clusters among many observations, t-SNE or UMAP are used; if it is to divide the objects into groups, cluster analysis.

An everyday example

A coffee shop wonders how customers perceive the ten kinds of coffee on its menu. A tasting panel rates every pair of coffees from 1 to 10 on 'how similar are they?'. From these ratings MDS draws a two-dimensional map on which coffees the panel found alike sit side by side. Looking at the map, the owner may notice that the horizontal axis orders the coffees from 'mild to strong' and the vertical axis from 'fruity to cocoa-like'; an empty corner of the map may point to a taste the menu does not yet offer.

Parameters in MerQur

Features

Concept: With raw data, it selects the variables from which distances are computed; with a ready-made matrix, the columns corresponding to the objects to be placed on the map. The map is built from this information alone.

Everyday example: If the coffees are described by acidity, bitterness and body scores, these three columns are selected and the distances between coffees are computed from them. If the 10 × 10 similarity table given directly by the tasting panel is used, the ten coffee columns are selected.

What it does: Selects the observation x variable columns for raw input, or the k columns (k objects) for matrix input.

Effect on the result: All distances and the configuration depend on the selected columns.

On the validation data: USArrests Stress-1 0.0959113.

Number of dimensions (2-5)

Concept: The number of dimensions of the map. Adding dimensions reflects the original distances better and lowers the stress, but makes the map harder to read and its dimensions harder to interpret; two or three dimensions are preferred in most applications.

Everyday example: A two-dimensional map shows the coffees on strength and aroma axes. Adding a third dimension might reveal a further distinction such as 'goes well with milk'; the stress falls, but the map becomes hard to show on a sheet of paper.

What it does: Sets the dimension of the embedding space.

Effect on the result: As the dimension increases, stress decreases and the number of coordinates increases.

On the validation data: USArrests Stress-1 0.09591 in 2 dimensions, 0.03820 in 3, 0 in 5 (only 4 variables). eurodist 0.06657 in 3 dimensions.

Metric MDS (preserves distances)

Concept: Metric MDS requires the distances on the map to be proportional to the original distances; non-metric MDS preserves only the order of the distances. If the numeric size of the distances is meaningful, metric MDS is suitable; if only the 'more similar / less similar' ordering can be trusted, non-metric MDS.

Everyday example: One cannot say that the difference between a rating of 8 and a rating of 4 from the tasters means exactly 'twice as similar'; but it is clear that the pair rated 8 was found more alike than the pair rated 4. This is why non-metric MDS is the more defensible choice for similarity ratings.

What it does: When on, SMACOF is applied at the ratio level; when off, only the rank order is preserved (nonmetric, monotone MDS) and ties are kept equal (secondary approach to ties).

Effect on the result: Stress-1, the coordinates and the fit measures change.

On the validation data: USArrests nonmetric Stress-1 0.0788746. In the intelligence data 0.028897 (ties are handled with the secondary approach).

→ Metric and non-metric MDS (SMACOF)

Distance metric (euclidean, manhattan, cosine, chebyshev, Bray-Curtis, Gower, Jaccard)

Concept: Sets how the distance between objects is computed from raw variables. Euclidean and Manhattan look at the size of the differences, cosine at the shape of the profile, Chebyshev at the largest single difference; Bray-Curtis is designed for abundance data, Jaccard for presence/absence data and Gower for variables of mixed type.

Everyday example: If the coffees are described by numeric taste scores, Euclidean distance is enough. If presence/absence information such as 'chocolate note? caramel note?' is kept for each coffee, Jaccard distance measures similarity more meaningfully by the share of notes they have in common.

What it does: Sets the distance definition for raw input. Euclidean, Manhattan, cosine and Chebyshev work on z-standardized data; Bray-Curtis, Gower and Jaccard on raw data.

Effect on the result: The distance matrix and therefore all outputs change.

On the validation data: USArrests Stress-1: Euclidean 0.09591, cosine 0.16198, Chebyshev 0.15531, Bray-Curtis 0.06141, Gower 0.09692, Manhattan 0.09749. dune: Bray-Curtis 0.18594, Jaccard 0.21138.

→ Distance measure (Euclidean, Manhattan, Gower, great-circle)

Input type (raw variables, ready-made distance matrix, ready-made similarity matrix)

Concept: Sets what the analysis starts from: raw variables (from which distances are computed), a ready-made distance matrix or a ready-made similarity matrix. MDS's particular strength is that it can work even when only pairwise distance or similarity information is available.

Everyday example: If the owner never measured the coffees' taste profiles and only collected answers to 'how similar are these two coffees?' from the panel, the similarity matrix input is used. A map can then be drawn without knowing on which traits the coffees differ.

What it does: Sets whether the input is raw data, a k x k distance matrix or a k x k similarity matrix; for matrix input the distance metric and standardization are not used.

Effect on the result: Determines the computational path.

On the validation data: eurodist Stress-1 0.072161. UScitiesD 0.0016896.

→ Distance measure (Euclidean, Manhattan, Gower, great-circle)

Similarity -> distance (max + min - s, sqrt(1 - s), 1 - s)

Concept: Because MDS works with distances, similarity values must be converted into distances: the most similar pair must correspond to the smallest distance. The conversion is chosen by the type of similarity: inversion for general ratings, √(1 − r) for correlations, 1 − s for proportions.

Everyday example: If the panel gave two coffees a similarity of 9 out of 10, they should be very close on the map; if it gave 2, far apart. The inversion transformation achieves this by turning 9 into a small distance and 2 into a large one.

What it does: Selects the conversion to distance for a similarity matrix only: reversal (max(s) + min(s) - s), sqrt(1 - s) for correlation similarities, 1 - s for proportions.

Effect on the result: The distance matrix and all results.

On the validation data: intelligence sqrt(1 - s): 0.22674. Lawler sqrt(1 - s): 0.24013. Ekman 1 - s: 0.13120.

→ Distance measure (Euclidean, Manhattan, Gower, great-circle)

Colour by group

Concept: Colours the points by a known grouping variable. Because the group information does not enter the calculation of the map, colours gathering together on the map show that the groups truly separate in the distance structure.

Everyday example: If, when the coffees are coloured by the continent their beans come from, the African coffees gather in the 'fruity' corner of the map, there is a link between the tasters' perception and the coffees' origin.

What it does: Colours the points by a known class column (e.g. iris Species).

Effect on the result: Chart only; the numeric output does not change.

On the validation data: No change in the numeric output.

Save coordinates to the dataset

Concept: Adds each object's map coordinates to the dataset as new columns. The coordinates can be used to relate the map's dimensions to known characteristics of the objects (e.g. by correlation).

Everyday example: The owner compares each coffee's coordinate on the first dimension with its caffeine content; if the two are strongly related, naming the first dimension 'strength' is supported.

What it does: Adds the embedding coordinates as new columns (raw input only).

Effect on the result: Only adds columns to the dataset.

On the validation data: The embedding coordinates are added to the dataset as new columns.

z-standardization

Concept: Transforms the variables to mean 0 and standard deviation 1, so that a variable with large units does not dominate the distance. With a ready-made matrix input, and with distances that have their own scaling (Bray-Curtis, Gower, Jaccard), this option has no effect.

Everyday example: If caffeine is measured in milligrams (60-200) and acidity on a 1-10 score, distances computed without standardisation reflect almost nothing but the caffeine difference. After standardisation, acidity also has a say on the map.

What it does: Converts the variables to z-scores with an n divisor; when off, distances are computed in raw units.

Effect on the result: For raw data the distances change; it has no effect for matrix input and for the Bray-Curtis, Gower and Jaccard distances.

On the validation data: USArrests Stress-1 0.09591 when on, 0.01231 when off (in raw units Assault dominates).

→ z-score (standardization)

MDS of variables (1 - r)

Concept: Places the variables rather than the observations on the map; the distance between two variables is taken as 1 − r (the Pearson correlation). Variables with a strong positive relationship end up close, unrelated or inversely related ones far apart.

Everyday example: If the owner maps the taste traits instead of the coffees, they may see bitterness and body side by side while acidity and sweetness stand far apart. This shows which taste traits are perceived together.

What it does: Positions the variables instead of the observations; the distance is 1 - Pearson r and the distance metric is ignored.

Effect on the result: The set of objects and the distance definition change.

On the validation data: USArrests 4 variables Stress-1 0.001254 (MerQur 0.001256). Crime 7 variables 0.06388.

Number of starts

Concept: The MDS algorithm starts from an initial layout, improves it step by step and can get stuck in a local solution. The classical (Torgerson) start is usually a good one; trying additional random starts and keeping the solution with the smallest stress lowers the risk of missing a better solution.

Everyday example: When the map is drawn once, two coffees may have got stuck on the wrong side. Rebuilding the layout from ten different starts and keeping the one with the best fit ensures that the coffees' positions on the map are not down to chance.

What it does: 1 = classical (Torgerson) start only; with a larger value additional random starts are tried and the smallest stress is chosen (against the risk of local minima).

Effect on the result: Stress and the coordinates; random starts can cause differences between runs.

On the validation data: USArrests 0.0959113 with 5 and 20 starts. Wish (7 - s) with 100 starts 0.217747.

→ Random starts and local optima

Fit measures (DAF, Tucker)

Concept: Gives further fit measures alongside Stress-1: DAF (dispersion accounted for) shows how much of the dispersion in the original distances the map captures, and Tucker's congruence coefficient how similar a pattern the two sets of distances follow; the closer both are to 1, the better the fit.

Everyday example: If DAF is 0.98, almost all the differences the tasters perceived are preserved on the two-dimensional map. With this value the owner can present the map to the team as 'a reliable summary of perceptions'.

What it does: Gives the raw stress, normalized raw stress (1 - phi^2), DAF (phi^2) and Tucker's congruence coefficient.

Effect on the result: Only the reported fit measures.

On the validation data: USArrests normalized raw stress 0.0091990, DAF 0.9908010, Tucker 0.9953899. The raw stress is 91.99; its value depends on the divisor of the z standardization.

→ Stress, fit measures and the Shepard diagram (MDS)

Stress scree (1..k dimensions)

Concept: Stress-1 is computed for every number of dimensions from 1 up to the chosen number. The point where the decrease in stress clearly slows down shows that further dimensions bring little gain, and helps in choosing the number of dimensions.

Everyday example: If the stress is 0.30 in one dimension, 0.08 in two and 0.06 in three, going from one to two is a large improvement and from two to three a small gain; the owner can make do with the two-dimensional map.

What it does: Computes Stress-1 for each number of dimensions from 1 to k (stress plot).

Effect on the result: Only the scree output.

On the validation data: In 1 dimension MerQur gives Stress-1 = 0.30800.

→ Stress, fit measures and the Shepard diagram (MDS)

Chart (Configuration, Shepard, Stress scree)

Concept: The configuration plot shows where the objects sit on the map. The Shepard diagram plots the original distances against the map distances; if the points gather around a straight line or a rising curve, the map has preserved the distances well. The stress scree plot shows how stress changes with the number of dimensions.

Everyday example: If, in the Shepard diagram, pairs the tasters found 'very alike' also sit at short distances on the map and pairs found 'very different' at long distances, the map faithfully reflects their perceptions. A pair far off the line points to two coffees that are misrepresented on the map.

What it does: Shows the configuration plot, the Shepard diagram (dissimilarity - embedded distance) or the stress scree.

Effect on the result: The numeric output does not change; if scree is selected and the dimension is 0, 5 dimensions are computed.

On the validation data: No numeric difference.

→ Stress, fit measures and the Shepard diagram (MDS)

Example on real data

Data: McNeil 1977, R datasets USArrests

Variables: features Murder, Assault, UrbanPop, Rape (50 states, raw-variable input)

  • Question: How well can the 50 states be placed on a two-dimensional map according to their similarity on four crime/urbanization indicators?
  • Setting: Euclidean distances from z-scored variables; metric (ratio level) MDS, SMACOF algorithm, 2 dimensions.
  • Result: Kruskal Stress-1 = 0.0959; on the Kruskal (1964) scale (0.025 excellent, 0.05 good, 0.10 fair (the acceptable limit), 0.20 poor) the fit is fair and below the acceptable limit (0.10).
  • Model fit: The squared correlation between the original distances and the map distances is R² = 0.9621; about 96% of the distances are preserved in two dimensions.
  • Descriptive: The first dimension reflects the general crime level (Murder, Assault, Rape), the second largely urbanization (UrbanPop); e.g. Alaska and California are at the high end of the first dimension (2.80 and 2.58).
  • Limitation: The direction and rotation of the dimensions are arbitrary; when interpreting the axes, look at their relationship with the variables. In 3 dimensions the stress drops to 0.038.
  • APA: Metric multidimensional scaling with Euclidean distances from z-scores gave a two-dimensional solution; Stress-1 = .096, R² = .96.

Related concepts

Stress, fit measures and the Shepard diagram (MDS)Distance measure (Euclidean, Manhattan, Gower, great-circle)Metric and non-metric MDS (SMACOF)z-score (standardization)Random starts and local optimaScree plotPearson correlation coefficientTies and zero differences (rank tests)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.