dimensionality reduction

Dimensionality reduction is the process of transforming data that has many features into a version with far fewer, while keeping the structure that matters for the task at hand. A feature is one measured attribute of each data point, such as a column in a table. A vector with hundreds of coordinates might shrink to fifty, or to two for a plot.

Methods fall into two broad families. Feature selection keeps a subset of the original columns and drops the rest. Feature extraction builds new features by combining the originals, so each output dimension mixes information from many inputs.

Extraction methods differ in the kind of structure they preserve:

• Linear projections: Principal component analysis (PCA) and random projection map the data onto a smaller set of straight axes.
• Neighborhood-preserving maps: t-SNE and UMAP keep nearby points together and mostly serve to draw data in two or three dimensions.
• Learned encoders: An autoencoder trains a neural network to squeeze its input through a narrow middle layer and then rebuild it.

Most of these techniques are unsupervised, so they need no labels. They cut the storage and compute cost of later models, filter out noise, and ease the curse of dimensionality, where data spreads too thinly for distances to stay meaningful.

In LLM work, the technique applies mostly to embeddings, which often carry more than a thousand dimensions. Projecting them down to two dimensions shows whether related documents cluster together. Shrinking them lowers the memory and search cost of a vector database.

Some embedding models build the reduction into training. Matryoshka representation learning, introduced in 2022, packs the most important information into the leading coordinates so that a vector can be cut to a shorter prefix. OpenAI’s text-embedding-3 models expose this kind of shortening through a dimensions parameter.

Every reduction discards information. Distances in the reduced space only approximate the originals, and the new dimensions that extraction methods produce rarely correspond to anything a person can name.

Drag the slider to keep fewer or more of 32 dimensions and watch how each pair’s reduced distance falls below its original distance:

Related Resources

Tutorial

K-Means Clustering in Python: A Practical Guide

In this step-by-step tutorial, you'll learn how to perform k-means clustering in Python. You'll review evaluation metrics for choosing an appropriate number of clusters and build an end-to-end k-means clustering pipeline in scikit-learn.

advanced algorithms data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Build a Recommendation Engine With Collaborative Filtering (Tutorial)
• ChromaDB: Embeddings and Vector Databases in Python (Tutorial)
• Vector Databases and Embeddings With ChromaDB (Course)
• Practical Text Classification With Python and Keras (Tutorial)
• K-Means Clustering in Python: A Practical Guide (Quiz)
• Embeddings and Vector Databases With ChromaDB (Quiz)
• Learn Text Classification With Python and Keras (Course)