Statistics Glossary → t-SNE (t-Distributed Stochastic Neighbor Embedding)
t-SNE (t-Distributed Stochastic Neighbor Embedding)analysis
A non-linear visualisation method that places observations described by many variables onto a two- or three-dimensional map so that those close to one another stay close.
Concept
With observations measured on dozens of variables, it is impossible to see directly which ones resemble each other. Linear methods such as PCA can pile groups on top of one another when the relationships are curved or complex. t-SNE (t-distributed stochastic neighbour embedding) focuses on keeping each observation's near neighbours close on the map as well, so local groups in high-dimensional data appear as separate islands on a two-dimensional chart.
The method first computes, in the original space, the probability that one observation would 'pick' another as its neighbour, using a Gaussian curve placed around each observation; the width of the curve is set separately for each point according to the perplexity setting. It then defines similar probabilities on the two-dimensional map using a Student-t distribution, which has heavier tails. The map is adjusted step by step until the Kullback-Leibler divergence KL(P‖Q), which measures the difference between the two probability distributions, is as small as possible. The heavy-tailed t distribution lets dissimilar points move well apart on the map and keeps clusters from crowding together.
t-SNE is a visualisation tool, not a measuring tool: the axes have no meaning, and cluster sizes and distances between clusters usually cannot be interpreted; only the information 'which points are close to each other' is reliable. The result is sensitive to settings such as perplexity, the number of iterations and the initialisation, and should be repeated with different settings before a structure is judged to be real. The method produces no transformation for placing new observations on an existing map, and it is computationally heavy on large data. Clusters seen on the map are no substitute for a cluster analysis.
When to use it
- When the local similarity structure of observations described by many variables (images, gene expression, survey profiles, audio features) is to be explored in a two- or three-dimensional chart.
- If interpretable components, preserved distances or applicability to new observations are needed, PCA is preferred; for a similar map that is faster and better preserves global structure, UMAP.
- If a distance or similarity matrix is available and the distances are to be preserved as faithfully as possible on the map, multidimensional scaling is used.
An everyday example
A music app describes 5,000 songs by 20 audio features such as tempo, energy, danceability, acousticness and loudness. Inspecting this 20-dimensional information by eye is impossible. t-SNE places the songs on a two-dimensional map: songs that are very alike land side by side, and islands such as 'calm acoustic songs' and 'high-tempo dance tracks' appear on the map. The app team can look at which island a song sits on and offer similar recommendations; but it should not be assumed that the distance between two islands shows how different those two styles are.
Parameters in MerQur
Features
Concept: The variables from which similarity between observations is computed. Because the variables are standardised, each enters the distance with similar weight; irrelevant variables blur the true neighbourhoods.
Everyday example: If songs are described by tempo, energy and acousticness, the map reflects musical styles. If irrelevant information such as the upload date is added, songs of the same style start to scatter across the map.
What it does: Input columns; a constant column is dropped; the data are z-standardized (n denominator).
Effect on the result: The distances and neighbourhood probabilities (P) are computed from the selected features.
On the validation data: For iris and digits K = 91 and P are computed (2 constant columns are dropped in digits).
Perplexity
Concept: Perplexity is the setting that sets roughly how many 'effective neighbours' each point has. A small value focuses on very local detail and can produce small, fragmented clusters; a large value takes wider neighbourhoods into account and shows a more general structure. Values between 5 and 50 are usually tried.
Everyday example: With a perplexity of 5, each song looks only at the few songs most like it, and the map breaks up into many small islands. With a perplexity of 50, each song takes a wider circle into account and large styles such as 'dance music' appear as a single region.
What it does: The effective number of neighbours of each point; the Gaussian width σ_i is adjusted by binary search to the entropy log(perplexity). In the Barnes-Hut approximation only K = min(n - 1, ⌊3·perplexity + 1⌋) nearest neighbours are used.
Effect on the result: The balance between local and global structure; KL and trustworthiness change.
On the validation data: iris 5: K = 16, KL (full) 0.3541, trustworthiness 0.986; 30: K = 91, 0.1503, 0.992; 50: K = 149 (n - 1 = 149 instead of ⌊3·50 + 1⌋ = 151), 0.0973, 0.987. Rtsne rejects perplexity 50 and 100 (3·perplexity > n-1); MerQur accepts them.
Iterations
Concept: The number of steps over which the map is adjusted. In the first steps a special phase (early exaggeration) is applied so clusters separate quickly, after which the map settles with fine-tuning; with too few steps the map stops before it has been arranged.
Everyday example: The songs are first scattered roughly over the map, then at each step similar songs are pulled a little closer and dissimilar ones pushed apart. If only 300 steps are taken, the islands may still look mixed up; after 1000 steps the map usually settles.
What it does: The maximum number of iterations; the first 250 iterations are the early exaggeration phase, the lower bound is 500.
Effect on the result: How long the optimization runs; if the KL is undefined, "-" is written.
On the validation data: iris, full KL: 500 (lower bound): 0.1529; 1000: 0.1503; 5000: 0.1501 (early stopping).
Missing Data
Concept: Either removes rows with missing values from the analysis or fills the missing value with the mean, the median or an estimate from the most similar rows. Filled-in values can pull an observation artificially towards the average and affect its position on the map.
Everyday example: If the danceability score was not computed for some songs, these can be left off the map. If filled with the mean, these songs may land between styles, in an in-between region where they do not really belong.
What it does: Drops the row with missing values or fills with the mean, the median or KNN.
Effect on the result: n and the embedding.
On the validation data: airquality: drop n = 111, KL 0.2220; mean n = 153, 0.4055; median 0.3231; KNN 0.3421.
Colour by group
Concept: Colours the points by a known grouping variable. This is purely a visual addition; the group information does not enter the calculation of the map, so if the colours line up with the clusters on the map, the groups genuinely differ in the measurements.
Everyday example: The songs are coloured by the genre labels given by music editors (pop, jazz, classical). If, on a map built from the audio features, the jazz songs gather together on their own, the editors' labels and the audio features confirm each other.
What it does: Colour; if the row limit is exceeded, stratified sampling is done.
Effect on the result: Does not change the embedding.
On the validation data: In iris the embedding is the same; at the digits 500-row limit the strata are proportional (each digit 48-51).
Save coordinates to the dataset
Concept: Adds each observation's two map coordinates to the dataset as new columns. These coordinates can be used for visualisation or exploration, but since the axes have no meaning of their own they should not be read as measurements.
Everyday example: Each song's position on the map is written next to it, and the team can plot the songs in another charting tool with these coordinates. A song having a large first coordinate does not mean it is 'more energetic'.
What it does: Adds the TSNE_1 and TSNE_2 columns to the dataset.
Effect on the result: The embedding does not change.
Random seed
Concept: Fixes the random starting positions and random steps so that the same map is obtained with the same settings. If the start is taken from PCA scores, it is not random and the seed has no effect.
Everyday example: Run twice with a random start, the analysis may show the islands in different corners of the map. With the same seed, the team sees the same map every time and there is no confusion in presentations.
What it does: Random seed for the initialization.
Effect on the result: With the PCA initialization (default) it does not change the embedding; with random initialization and with a non-Euclidean metric it is effective.
On the validation data: With the PCA initialization seeds 0, 42, 123 and 1..10 give exactly the same embedding (by design; PARAMETER HAS NO EFFECT, documented).
Maximum rows
Concept: Because t-SNE's computing load grows quickly with the number of observations, if the number of rows exceeds this limit a random subsample of the data is used. The subsample shows the general structure of the map but may miss rare small groups.
Everyday example: If the app's catalogue holds 200,000 songs and the limit is 5,000, a map of 5,000 randomly chosen songs is produced. A very rare style made up of only 30 songs may barely appear in this sample.
What it does: If exceeded, a seeded (42, fixed) random sample is taken; 0 = unlimited.
Effect on the result: The n used in the embedding; a warning is given in the text when a sample is taken.
On the validation data: digits 500: n = 500 (warning in the text); 0 and 2000: 1797 (the same embedding).
Learning rate
Concept: Sets how far the points are moved at each adjustment step. Too small a rate makes the map progress slowly and remain compressed; too large a rate can make the points fly about erratically.
Everyday example: If the learning rate is very low, the songs move very little at each step and the islands may still be stuck together after 1000 steps. If it is very high, the songs jump about the map and no tidy clusters form.
What it does: The size of the gradient step; "Automatic" max(n/12/4, 50) = 50 (iris).
Effect on the result: Convergence and embedding quality.
On the validation data: 200: KL 0.1540, trustworthiness 0.991 (10-seed median within the limit with Rtsne eta = 200); 1000: KL 0.5803, trustworthiness 0.974.
Early exaggeration
Concept: In the first phase the neighbourhood probabilities of the original space are artificially inflated; this lets similar points form tight groups early and opens gaps between the groups. Larger values usually make the gaps between clusters more pronounced.
Everyday example: Early exaggeration is like the songs finding their 'friends' in the first rounds and gathering in small groups; once the groups have formed, their places on the map are fine-tuned. Without this phase, different styles might stay mixed together in the middle of the map.
What it does: The factor by which the P probabilities are multiplied in the first phase; it lets the clusters separate early.
Effect on the result: The gap between clusters and the final KL.
On the validation data: iris 4: KL 0.1696; 12: 0.1503; 24: 0.1746.
Initialization (PCA / Random)
Concept: Where the points' starting positions on the map come from. PCA initialisation uses the main directions of the data as the starting layout; it gives the same result on every run and better preserves the overall arrangement. A random start may produce a different map for each seed.
Everyday example: With PCA initialisation the songs are first lined up along a main axis such as 'calm to energetic', and t-SNE then separates the local groups. With a random start the songs are sprinkled over the map haphazardly, and the same styles may gather in a different corner on each run.
What it does: PCA scores (PC1 sd 1e-4) or random initialization. With a non-Euclidean metric the engine uses random initialization and the text writes the actual initialization.
Effect on the result: The layout of the embedding; the PCA initialization is deterministic.
On the validation data: Random initialization 10-seed median trustworthiness 0.9910 (PCA 0.9922).
Distance metric (Euclidean / Manhattan / Cosine (1-cos) / Pearson correlation (1-r))
Concept: Defines how similar two observations count as being in the original space. Euclidean and Manhattan distances look at absolute differences in values, while cosine and correlation distances look at the shape of the profile rather than the size of the values.
Everyday example: With cosine distance, two songs whose features are in similar proportions (e.g. both lean towards acousticness rather than energy) count as close even if their loudness differs. With Euclidean distance, a large difference in loudness pushes them apart.
What it does: Input-space distance; in the Barnes-Hut approximation the kNN distance is squared for all metrics.
Effect on the result: The neighbourhood probabilities and the embedding.
On the validation data: iris Manhattan KL 0.1684 (10-seed medians within the limit with Rtsne is_distance); cosine 0.1877; correlation 0.1432.
→ Distance measure (Euclidean, Manhattan, Gower, great-circle)
Embedding dimensions (2 / 3)
Concept: Whether the map has two or three dimensions. Three dimensions can separate groups that overlap in two, but three-dimensional charts are harder to read and to present in print.
Everyday example: If 'acoustic pop' and 'light jazz' partly overlap on the two-dimensional map, a three-dimensional map may separate the two styles at different heights. The team has to rotate the map to examine it, but the separation is clearer.
What it does: The number of dimensions of the embedding.
Effect on the result: The coordinate columns and the KL.
On the validation data: In 3 dimensions the reported KL is 0.7454 (the full KL with Student-t degrees of freedom 1); the internal value during optimization is 0.434 (dof 2 in 3 dimensions).
Trustworthiness / continuity
Concept: Trustworthiness measures whether points that appear as neighbours on the map really are neighbours in the original space; continuity whether neighbours in the original space remain neighbours on the map. The closer both are to 1, the more faithfully the map preserves local structure.
Everyday example: If trustworthiness is high, two songs side by side on the map really are similar songs, and the recommendation system can rely on this neighbourhood. If continuity is low, some songs that are actually very alike have ended up far apart on the map.
What it does: Venna & Kaski trustworthiness and continuity measures at a neighbourhood of k = 5; the original space is measured with the selected metric.
Effect on the result: Reports how well the embedding preserves local neighbourhoods; it does not change the embedding.
On the validation data: On the same embedding iris 0.9922, digits 0.9928; iris cosine 0.99476.
KL divergence (output)
Concept: The Kullback-Leibler divergence measures the mismatch between the neighbourhood probabilities in the original space and those on the map. t-SNE tries to make it as small as possible; a low value shows local neighbourhoods are well preserved, but it is not suitable for comparisons across different datasets or very different perplexity values.
Everyday example: If the KL value is low, the question 'who are this song's closest friends?' gets almost the same answer in the original 20-dimensional data and on the map. If it is high, the map has distorted some neighbourhoods and the settings should be reconsidered.
What it does: The reported KL is the full KL(P||Q) computed from the final embedding by definition (van der Maaten & Hinton 2008).
Effect on the result: A fit measure; a low value shows that local neighbourhoods are well preserved.
On the validation data: The reported value is the full KL (learning rate 1000: 0.5803; 3 dimensions: 0.7454; perplexity 100: 0.01442).
Example on real data
Data: Fisher 1936 iris, R datasets
Variables: features Sepal.Length, Sepal.Width, Petal.Length, Petal.Width (sepal and petal measurements)
- Question: How do the four-dimensional measurements of the iris flowers look on a two-dimensional map that preserves local neighbourhoods?
- Setting: The variables are converted to z-scores; perplexity 30 (about 30 effective neighbours per point), 1000 iterations, PCA initialization (deterministic, independent of the seed).
- Result: Full KL divergence 0.1503 ("very good"); the neighbourhood probabilities of the high-dimensional space are largely preserved in the embedding.
- Descriptive: The embedding shows 2 distinct clusters and 2 isolated points (1%): setosa forms a separate cluster, versicolor and virginica lie adjacent in the second cluster.
- Limitation: Axis values and distances between clusters are not interpreted; t-SNE preserves only local structure. The result is sensitive to the choice of perplexity (e.g. KL 0.354 at 5).
- APA: t-SNE (perplexity = 30, 1000 iterations, PCA initialization, z-scores) reduced the 150 observations to two dimensions; KL divergence = 0.15.
Related concepts
Kullback-Leibler (KL) divergenceTrustworthiness and continuity measuresEarly exaggeration (t-SNE)Perplexity (t-SNE effective number of neighbours)Learning rate (shrinkage)Random starts and local optimaDistance measure (Euclidean, Manhattan, Gower, great-circle)Convergence and maximum iterations
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.