principal component analysis (PCA)

Principal component analysis (PCA) is a dimensionality-reduction technique that rotates data onto new axes ranked by how much of its spread each one captures. Keeping only the first few axes preserves most of the information in far fewer numbers.

Those new axes are the principal components. The first points in the direction along which the data varies most, and each later component captures the most remaining variance while staying perpendicular to the ones before it. Projecting the data onto the top components yields a compact vector for every point.

Rotate the axis below through a cloud of correlated points to see the share of variance it captures peak at the first principal component, while the perpendicular direction holds the rest:

Karl Pearson described the method in 1901, and Harold Hotelling developed and named it in the 1930s. Modern implementations center each feature on its mean and then compute the components with a singular value decomposition, which is how scikit-learn’s PCA class works. A fitted model reports how much of the total variance each component explains, which guides how many components to keep.

PCA is an unsupervised method, so it needs no labels. It shows up as a preprocessing step that strips redundant or noisy features before a model trains, and as a common answer to the curse of dimensionality.

In LLM work, PCA projects high-dimensional embeddings down to two or three dimensions for plotting, or shrinks them to cut storage and search cost in a vector database. It also often runs first in a t-SNE pipeline to reduce the input to around 50 dimensions.

The method has clear limits. It only finds linear structure, so curved relationships between features get flattened. It’s sensitive to scale, which means features measured in larger units dominate unless the data is standardized first. Outliers can also pull the components toward themselves, and the direction of highest variance isn’t always the one that matters for a given task.

Related Resources

Tutorial

K-Means Clustering in Python: A Practical Guide

In this step-by-step tutorial, you'll learn how to perform k-means clustering in Python. You'll review evaluation metrics for choosing an appropriate number of clusters and build an end-to-end k-means clustering pipeline in scikit-learn.

advanced algorithms data-science machine-learning

For additional information on related topics, take a look at the following resources:

• ChromaDB: Embeddings and Vector Databases in Python (Tutorial)
• NumPy, SciPy, and pandas: Correlation With Python (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Vector Databases and Embeddings With ChromaDB (Course)
• K-Means Clustering in Python: A Practical Guide (Quiz)
• Embeddings and Vector Databases With ChromaDB (Quiz)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)