MerQuR+Statistics Glossary

Statistics Glossary → UMAP (Uniform Manifold Approximation and Projection)

UMAP (Uniform Manifold Approximation and Projection)analysis

A fast, non-linear dimension reduction method based on a neighbourhood graph, which places multivariate observations onto a map of two or a few dimensions while preserving each one's near neighbours.

Concept

UMAP (Uniform Manifold Approximation and Projection), like t-SNE, aims to show the local similarity structure of high-dimensional data on a low-dimensional map. Its central idea is that the data lie on a hidden, lower-dimensional curved surface (a manifold) within the high-dimensional space, and that the structure of this surface can be reconstructed from nearest-neighbour relationships. Besides visualisation, it is also used as a non-linear dimension reduction tool for later analyses.

The method works in two stages. First, each observation's k nearest neighbours are found, and from them a neighbourhood graph is built whose edge weights show how strong each neighbourhood is; because distances are scaled by each point's own local density, sparse and dense regions are represented in a balanced way. Then a similar graph is defined on the low-dimensional map, and the points are moved until the mismatch between the two graphs (cross-entropy) is as small as possible. The number of neighbours sets the balance between local detail and overall structure, and the minimum distance (min_dist) how tightly points are packed on the map.

UMAP is usually faster than t-SNE and preserves large-scale structure somewhat better; even so, the axes have no meaning, and cluster sizes and distances between clusters should be read with care, only as a rough hint. The result is sensitive to the number of neighbours, min_dist and the random seed; a structure that also appears under different settings is more trustworthy. Groups seen on the map are no substitute for a cluster analysis; the scaling of the variables and the distance measure clearly affect the result.

When to use it

  • When the local similarity structure of data described by many variables, possibly with many observations (single-cell expression, spectral measurements, image features, survey profiles), is to be visualised quickly, or a non-linear dimension reduction is needed.
  • If the components must be interpreted and distances and variance shares preserved, PCA is preferred; if only a detailed view of local clusters is wanted and the data are small, t-SNE can also be used.
  • If a distance or similarity matrix is available and all distances are to be preserved as faithfully as possible on the map, multidimensional scaling is preferred.

An everyday example

A chamber of agriculture is examining 15 measurements, such as pH, organic matter, nitrogen, phosphorus, potassium and clay content, from soil samples taken from 2,000 fields in the region. Reading this 15-dimensional data as a table is impossible. UMAP places the samples on a two-dimensional map on which similar soils sit side by side, and regions such as 'calcareous and poor in phosphorus' or 'clayey and rich in organic matter' appear. The chamber can prepare fertiliser advice together for fields that fall in the same region of the map.

Parameters in MerQur

Features

Concept: The variables from which the neighbourhoods and the map are computed; because they are standardised, each enters the distance with similar weight. Which variables are chosen directly determines who counts as whose 'neighbour'.

Everyday example: If the soil samples are described by chemical measurements, the map reflects soil types. If irrelevant information, such as a numeric code for the technician who took the sample, is added, soils of the same type start to scatter across the map.

What it does: Selects the columns entering the analysis; the variables are converted to z-scores (n divisor), a constant column stays 0.

Effect on the result: All distance and graph outputs.

On the validation data: The k-NN graph is built for 150 points in iris and 1797 in digits; in iris a duplicated observation creates a tie in the neighbour ranking.

Number of dimensions

Concept: The number of dimensions of the map. Two dimensions are usually used for visualisation; if UMAP is preparing input for later analyses (e.g. clustering), a few more dimensions retain more information. The neighbourhood graph is not affected by the number of dimensions; only the layout changes.

Everyday example: The chamber wants a two-dimensional map to show the fields in a presentation. If it later wants to define soil groups with a cluster analysis, a five-dimensional embedding can help separate groups that overlap in two dimensions.

What it does: Sets the embedding dimension.

Effect on the result: The graph and the a/b parameters do not change; only the embedding and trustworthiness change.

On the validation data: iris trustworthiness 0.9839 in 2 dimensions, 0.9884 in 3, 0.9892 in 5 (the program warns because 5 is greater than the 4 features).

Number of neighbors (n_neighbors)

Concept: The number of neighbours each point is connected to in the neighbourhood graph; it sets the balance between local and overall structure. A small value focuses on very local detail and can break the data into small pieces; a large value takes wider surroundings into account and brings out the overall structure, but can wipe out fine distinctions.

Everyday example: With 5 neighbours each soil sample is linked only to the few samples most like it, and the map splits into small communities, such as the fields of one village. With 50, each sample sees a wider circle and large groups such as 'calcareous soils' appear as a single region.

What it does: The number of local neighbours k (including the point itself). rho_i is the nearest-neighbour distance; sigma_i is found from the condition sum exp(-(d - rho)/sigma) = log2(k). A low value emphasizes local detail, a high value global structure.

Effect on the result: The graph, the number of edges and trustworthiness.

On the validation data: iris: k = 5 794 edges, trustworthiness 0.9843; 15 2678 edges, 0.9839; 50 8666 edges, 0.9800.

→ Number of nearest neighbours (k) and the neighbourhood graph

Min distance (min_dist)

Concept: Sets how close together neighbouring points may be placed on the map. A small value gathers similar points into tight clumps and makes clusters stand out; a large value spreads the points more evenly and shows the overall shape better. It affects only the layout, not the neighbourhood graph.

Everyday example: With a small min_dist, very similar soil samples pile up almost on the same spot and the groups look like sharp islands. With a large one, the samples spread over a wider area and the transitions from one group to another are easier to follow.

What it does: Sets the smallest distance between close points in the embedding; produces the a and b curve parameters. A low value gives tight, a high value loose clusters.

Effect on the result: a, b and the embedding; the graph does not change.

On the validation data: 0.0: a 1.932808, b 0.790495; 0.1: 1.576943 and 0.895061; 1.0: 0.114976 and 1.929237. Trustworthiness 0.981-0.984.

Distance metric (Euclidean, Manhattan, Cosine, Pearson correlation, Chebyshev)

Concept: Defines the distance used to find neighbours in the original space. Euclidean and Manhattan look at absolute differences in values, while cosine and correlation distances look at the shape of the profile rather than the size of the values; the latter are often preferred for spectral and proportional (compositional) data.

Everyday example: If reflectance spectra of the soil samples were measured, two samples may differ in overall brightness yet have spectral curves of similar shape. Correlation distance counts these two samples as neighbours, while Euclidean distance may count them as far apart because of the brightness difference.

What it does: Sets the distance used in the input space (with exact k-NN); cosine or correlation is common for spectral and compositional data.

Effect on the result: The k-NN, the graph and trustworthiness.

On the validation data: The graph is built with the Manhattan, cosine and correlation metrics. Chebyshev has max|.| equalities (ties) at 28 points. The cosine trustworthiness computed with the selected metric is 0.99074.

→ Distance measure (Euclidean, Manhattan, Gower, great-circle)

Random seed

Concept: Fixes the random steps in laying out the map so that the same map is obtained with the same settings. The neighbourhood graph is not affected by the seed; different seeds can show the same structure in slightly different layouts.

Everyday example: When the chamber reproduces the map at the end of the month, it wants the fields to appear in the same places; the same seed ensures this. If the same soil groups appear with different seeds too, the structure is not a matter of chance.

What it does: Sets the seed of the random algorithm (umap-learn random_state); the same seed gives the same result.

Effect on the result: The embedding coordinates; the graph does not change.

On the validation data: The largest difference between two runs with the same seed is 0 (in all 23 runs); seeds 0 and 123 produce different embeddings.

→ Random seed

Supervised target (y)

Concept: Supervised UMAP also feeds a known categorical target into the neighbourhood graph, encouraging observations of the same class to gather together on the map. This highlights the structure that separates the classes; but because the map now reflects the target information as well as the measurements, it cannot be used as evidence that the classes separate 'on their own'.

Everyday example: If the chamber gives the soil classes previously defined by experts as the target, the map shows these classes more clearly separated, and it becomes easier to see which class new samples fall near. But whether the classes truly separate on the chemical measurements must be answered with the unsupervised map.

What it does: Performs supervised UMAP with a categorical target (fuzzy graph intersection with the target, target_weight 0.5). Only categorical columns and integer-coded factors with few levels are listed; rows with an empty target are excluded.

Effect on the result: The graph and the embedding.

On the validation data: iris Species: 5-NN accuracy 1.00. digits digit target: accuracy 0.9955.

Missing data (drop row, Fill with mean, Fill with median)

Concept: Either removes rows with missing values from the analysis or fills the missing value with the column mean or median. Filled-in values can pull an observation artificially towards the average and drop it into an in-between region of the map where it does not really belong.

Everyday example: If potassium was not measured for some fields, these fields can be left off the map. If filled with the mean, they may land between two soil groups on the map and receive the wrong fertiliser advice.

What it does: Deletes missing values listwise or fills them with the column mean/median; the number of dropped rows is reported.

Effect on the result: The number of observations entering the analysis, the k-NN and the graph.

On the validation data: In airquality dropping rows gives n = 111, filling with the mean and median n = 153.

→ Single imputation (mean, median, kNN filling)

Trustworthiness / continuity

Concept: Trustworthiness measures whether points that appear as neighbours on the map really are neighbours in the original space; continuity whether neighbours in the original space remain neighbours on the map. The closer both are to 1, the more faithfully the map preserves local structure.

Everyday example: If trustworthiness is 0.98, two fields side by side on the map really do have similar soil, and the chamber can give joint advice on the basis of this neighbourhood. If continuity is low, some soils that are very alike have landed in separate regions of the map.

What it does: Gives the local neighbourhood preservation measures (k = 5; Venna and Kaski, 2001): trustworthiness is the proportion of neighbours in the embedding that are also neighbours in the original space; continuity is the reverse.

Effect on the result: Only the reported quality measures.

On the validation data: iris trustworthiness 0.98390, continuity 0.98705; digits trustworthiness 0.97980.

→ Trustworthiness and continuity measures

Example on real data

Data: Fisher 1936 iris, R datasets

Variables: features Sepal.Length, Sepal.Width, Petal.Length, Petal.Width (sepal and petal measurements)

  • Question: How do the four measurements of the iris flowers look on a two-dimensional map that preserves local and partly global structure?
  • Setting: The variables are converted to z-scores; 15 neighbours (a balance between local and global structure), min_dist = 0.1 (tight clusters), 2 dimensions.
  • Result: The embedding places the 150 observations in two dimensions; MerQur's automatic cluster heuristic (DBSCAN ε ≈ 0.39) finds 10 small clusters and 20 isolated points (13%). This heuristic shows the local communities in the embedding, not the number of species.
  • Neighbourhood preservation: On the same embedding trustworthiness (k = 5) is 0.984; most close neighbours in the embedding are also neighbours in the original space.
  • Descriptive: In the chart setosa appears as a separate island; versicolor and virginica are partly intermixed in an adjacent region.
  • Limitation: Axis values are not interpreted; distances between clusters are more meaningful than in t-SNE but still approximate. The result is sensitive to n_neighbors, min_dist and the seed.
  • APA: UMAP (n_neighbors = 15, min_dist = 0.1, Euclidean, z-scores) reduced the 150 observations to two dimensions; trustworthiness (k = 5) = .98.

Related concepts

Random seedTrustworthiness and continuity measuresNumber of nearest neighbours (k) and the neighbourhood graphSingle imputation (mean, median, kNN filling)Distance measure (Euclidean, Manhattan, Gower, great-circle)Pearson correlation coefficientz-score (standardization)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.