quantized low-rank adaptation (QLoRA)

Quantized low-rank adaptation (QLoRA) is a memory-efficient fine-tuning method that adapts a large language model by freezing its weights at 4-bit precision and training a small set of low-rank adapter weights on top. The frozen base and the trainable adapters combine on each layer:

QLoRA builds on low-rank adaptation (LoRA), which freezes the base model and learns only a pair of small matrices per layer. The rank is what keeps them small: instead of updating a full weight matrix, LoRA learns two skinny matrices whose product has the same shape as that matrix, so the count of trainable parameters stays tiny.

QLoRA cuts memory further by quantizing the frozen weights to 4 bits, then backpropagating gradients through them into the higher-precision adapters.

Introduced in 2023 by Dettmers and colleagues, QLoRA can fine-tune a 65-billion-parameter model on a single 48 GB GPU while matching the accuracy of full 16-bit fine-tuning. Three techniques make this possible:

• 4-bit NormalFloat (NF4): A data type matched to the roughly normal distribution of pretrained weights, more accurate than plain 4-bit integers.
• Double quantization: Compressing the quantization constants themselves, saving about a third of a bit per parameter.
• Paged optimizers: Using unified CPU-GPU memory to absorb the spikes that would otherwise cause out-of-memory errors.

QLoRA is implemented in libraries such as Hugging Face PEFT and bitsandbytes, and has become a common way to adapt open-weight models on consumer hardware.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

As the AI boom continues, the Hugging Face platform stands out as the leading open-source model hub. In this tutorial, you'll get hands-on experience with Hugging Face and the Transformers library in Python.

intermediate ai

For additional information on related topics, take a look at the following resources:

• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• Embeddings and Vector Databases With ChromaDB (Tutorial)
• Hugging Face Transformers (Quiz)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Building a Neural Network & Making Predictions With Python AI (Course)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)