vision transformer (ViT)

A vision transformer (ViT) is a neural network architecture that applies the transformer to images by cutting a picture into fixed-size patches and treating that sequence of patches the way a language model treats a sequence of tokens.

Each patch, typically 16 by 16 pixels, is flattened and mapped through a trainable linear projection into an embedding. A learnable class token is prepended to the sequence to gather a summary of the whole image, and learnable 1D position embeddings record where each patch sat in the original grid.

A standard transformer encoder then runs self-attention over the whole sequence, so every patch attends to every other patch from the first layer onward rather than building up a receptive field gradually as a convolutional neural network (CNN) does.

The widget below cuts the same picture into patches at three sizes, and the token sequence grows as the patches shrink. Picking a patch then shows what layer one reaches from it: every patch in the image under self-attention, or a single 3 by 3 neighborhood under a convolution.

That global view costs the model its image-specific inductive bias, the built-in assumptions about image structure that help a model learn from fewer examples. Locality and translation equivariance are baked into every convolutional layer but not into attention, so a ViT trained on a mid-sized dataset such as ImageNet trails a comparable CNN.

The 2020 paper by Dosovitskiy and colleagues that introduced the architecture found that ordering reverses once pretraining data reaches 14 to 300 million images.

Hugging Face Transformers exposes the original Google checkpoints, whose names carry the patch size and input resolution, as in google/vit-base-patch16-224, and fine-tuning one on a smaller labeled set is the usual route to a specific task. Patch encoders of this shape now sit behind much of modern computer vision, including the image half of multimodal large language models.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• ChromaDB: Embeddings and Vector Databases in Python (Tutorial)
• Image Segmentation Using Color Spaces in OpenCV + Python (Tutorial)
• Image Processing With the Python Pillow Library (Tutorial)
• Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)
• Process Images Using the Pillow Library and Python (Course)