low-rank adaptation (LoRA)

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning method that adapts a large model to a new task without retraining all of its weights. LoRA freezes those pretrained weights and learns each update as a pair of small matrices injected into the layers of the transformer architecture.

Rather than rewriting a full weight matrix, LoRA represents the change to it as the product of two thin matrices whose shared inner dimension is the rank, typically a small value such as eight or sixteen. The number of trainable parameters drops by orders of magnitude, and the adapters usually attach only to the self-attention projections. Trying different ranks on a single 1024 by 1024 projection shows how steeply the trainable count falls.

Interactive diagram — enable JavaScript to view.

Introduced in 2021 by Hu and colleagues at Microsoft, LoRA reduced the trainable parameters of GPT-3 175B by ten thousand times against full fine-tuning with the Adam optimizer, and its GPU memory requirement by three times. It doesn’t add inference latency, because the trained matrices can be merged back into the frozen weights.

Each adapter is a small file, so one base model can host many of them, one per task. Hugging Face PEFT is a common implementation, and QLoRA extends the method by quantizing the frozen base to 4-bit precision.

Hugging Face Transformers: Leverage Open-Source AI in Python

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

As the AI boom continues, the Hugging Face platform stands out as the leading open-source model hub. In this tutorial, you'll get hands-on experience with Hugging Face and the Transformers library in Python.

intermediate ai

For additional information on related topics, take a look at the following resources:

By Martin Breuss • Updated Aug. 21, 2026
AI Coding Glossary Share Feedback