GPTQ

GPTQ is a post-training quantization method that compresses a trained large language model to 3- or 4-bit weights in a single pass, without retraining.

Introduced in a 2022 paper by Frantar and colleagues, GPTQ quantizes one layer at a time. It feeds a small calibration set of sample text through the model, then rounds each weight matrix column by column. Approximate second-order (Hessian) information updates the columns that aren’t quantized yet, so they absorb the error each rounding step introduces.

Step through that pass below on a single row of six weights, one in each column, headed for a 3-bit grid. Each column snaps to an integer level, and the columns that are still floats absorb its rounding error, which keeps the layer’s output far closer to the original than plain rounding does.

That layer-local correction is what keeps the accuracy loss small. The paper reports quantizing a 175-billion-parameter GPT model in roughly four GPU hours with negligible degradation, and then running it for generative inference on a single GPU instead of a multi-GPU server.

GPTQ is weight-only. The packed integers are restored to 16-bit floats inside a fused inference kernel, so the gain is memory and bandwidth rather than faster integer arithmetic. Hugging Face Transformers exposes GPTQ through the GPTQModel backend, which widens the choice to 2, 3, 4, or 8 bits. Transformers supports AWQ and bitsandbytes as alternatives.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)