activation-aware weight quantization (AWQ)

AWQ, or activation-aware weight quantization, is a post-training quantization method that compresses a large language model to 4-bit weights by letting activations, not weight values, decide which weights to protect from rounding.

The method rests on the observation that weights aren’t equally important. Protecting roughly 1% of them, the salient ones, is enough to keep rounding error from degrading the model. AWQ locates those weights by watching which input channels carry the largest activations during a short calibration pass.

Storing the salient channels at higher precision would produce a mixed-precision layout that GPU kernels handle poorly. AWQ instead scales each salient channel up before rounding it and divides the matching activation channel down by the same factor. The layer computes the same result, and the scaled-up weights land on a finer part of the integer grid.

Sliding the scale factor below moves one salient weight along that grid: a larger factor buys it more levels and less rounding error, until it grows into the group’s largest value and widens the shared step for every other weight in the group.

Because AWQ’s scale search only reads activation statistics and runs no backpropagation or reconstruction, it doesn’t overfit its calibration set, so a quantized model keeps its behavior on domains and modalities the calibration data never covered. Weights are packed at 3 or 4 bits in groups, commonly 128 per group, while activations stay in 16-bit floating point.

Ji Lin, Jiaming Tang, and colleagues introduced AWQ in 2023, and the paper won best paper at MLSys 2024. It ships with TinyChat, an inference engine whose fused 4-bit kernels report more than a 3x speedup over a 16-bit Hugging Face baseline on desktop and mobile GPUs.

Hugging Face Transformers and serving stacks such as vLLM load AWQ checkpoints directly, which has made it one of the two common 4-bit formats on the Hugging Face Hub alongside GPTQ.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
• Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• Building Type-Safe LLM Agents With Pydantic AI (Course)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)