GPTQ

GPTQ is a post-training quantization method that compresses a trained large language model to 3- or 4-bit weights in a single pass, without retraining.

Introduced in a 2022 paper by Frantar and colleagues, GPTQ quantizes one layer at a time. It feeds a small calibration set of sample text through the model, then rounds each weight matrix column by column. Approximate second-order (Hessian) information updates the columns that aren’t quantized yet, so they absorb the error each rounding step introduces.

Step through that pass below on a single row of six weights, one in each column, headed for a 3-bit grid. Each column snaps to an integer level, and the columns that are still floats absorb its rounding error, which keeps the layer’s output far closer to the original than plain rounding does.

That layer-local correction is what keeps the accuracy loss small. The paper reports quantizing a 175-billion-parameter GPT model in roughly four GPU hours with negligible degradation, and then running it for generative inference on a single GPU instead of a multi-GPU server.

GPTQ is weight-only. The packed integers are restored to 16-bit floats inside a fused inference kernel, so the gain is memory and bandwidth rather than faster integer arithmetic. Hugging Face Transformers exposes GPTQ through the GPTQModel backend, which widens the choice to 2, 3, 4, or 8 bits. Transformers supports AWQ and bitsandbytes as alternatives.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

As the AI boom continues, the Hugging Face platform stands out as the leading open-source model hub. In this tutorial, you'll get hands-on experience with Hugging Face and the Transformers library in Python.

intermediate ai

For additional information on related topics, take a look at the following resources:

• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Hugging Face Transformers (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)