t-distributed stochastic neighbor embedding (t-SNE)

t-distributed stochastic neighbor embedding (t-SNE) is a dimensionality-reduction algorithm that lays high-dimensional data on a two- or three-dimensional map, positioning points so that close neighbors in the original space stay close on the map.

Laurens van der Maaten and Geoffrey Hinton introduced t-SNE in a 2008 paper as a variation on Stochastic Neighbor Embedding. The algorithm turns distances between points into probabilities that one point would pick another as its neighbor, using a Gaussian in the original space and a heavier-tailed Student t-distribution on the map.

Gradient descent then nudges the map points around until the two distributions match as closely as they can, measured by Kullback-Leibler divergence, a score of how much one probability distribution differs from another. The heavy tail is what gives crowded points room to spread out.

The main parameter is perplexity, a smooth stand-in for how many nearest neighbors each point pays attention to. Typical values run between 5 and 50, and scikit-learn defaults to 30.

In LLM work, t-SNE is mostly a way to eyeball an embedding space, projecting a corpus of text vectors down to two dimensions to see whether semantically similar chunks land near each other.

The picture is diagnostic rather than quantitative. t-SNE preserves local neighborhoods but not global geometry, so cluster sizes and the gaps between clusters carry no reliable meaning, and two runs over the same data can settle into different layouts.

The map below runs the algorithm on a fixed set of 150 points that live in eight dimensions as five groups, coloring each point by the group it belongs to in the original space. The neighborhoods pull together as gradient descent runs, and moving the perplexity slider or starting a new run settles the same groups into a different arrangement.

Uniform Manifold Approximation and Projection (UMAP) is the usual alternative when global structure matters.

Related Resources

Tutorial

ChromaDB: Embeddings and Vector Databases in Python

Learn what embeddings and vector databases are, then store, query, and filter text with ChromaDB in Python and use it to give an LLM context.

advanced ai databases data-science machine-learning

For additional information on related topics, take a look at the following resources:

• K-Means Clustering in Python: A Practical Guide (Tutorial)
• Visualizing Data in Python With Seaborn (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Plot With pandas: Python Data Visualization for Beginners (Tutorial)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)
• K-Means Clustering in Python: A Practical Guide (Quiz)
• Visualizing Data in Python With Seaborn (Quiz)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)
• Plot With pandas: Python Data Visualization Basics (Course)