GGUF

GGUF is a binary file format that stores a model’s weights along with everything needed to run them, packed into one self-contained file designed for fast loading at inference time.

The llama.cpp project introduced GGUF in August 2023 to replace the earlier formats from GGML, the C tensor library that llama.cpp is built on. Those older formats recorded a model’s hyperparameters as an untyped list, but GGUF stores typed key-value metadata instead, so new fields can be added without invalidating files that already exist.

A single file holds the tensor table and the tensor data along with the architecture description, the tokenizer vocabulary, and the context length. Nothing lives in a companion config file. The tensor data is padded to a fixed alignment, so a runtime can memory-map the file and read weights straight from disk.

That layout is easiest to grasp by walking a real file. The map below decodes a GGUF build of Llama 3.1 8B Instruct region by region, and switching the tensor type shows how the header, the metadata, and the tensor table stay byte for byte the same while only the data region changes size.

Tensors can be kept at full precision or stored in one of many quantized block types, ranging from eight bits per weight down to under two. That mix of portability and compression made GGUF the usual distribution format for open-weight large language models on local hardware, and llama.cpp, Ollama, and LM Studio all read it directly.

Related Resources

Tutorial

How to Use Ollama to Run Large Language Models Locally

Learn how to use Ollama to run large language models locally. Install it, pull models, and start chatting from your terminal without needing API keys.

intermediate ai tools

For additional information on related topics, take a look at the following resources:

• Hugging Face Transformers: Leverage Open-Source AI in Python (Tutorial)
• Python mmap: Improved File I/O With Memory Mapping (Tutorial)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• Hugging Face Transformers (Quiz)
• Python mmap: Doing File I/O With Memory Mapping (Course)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• Building Type-Safe LLM Agents With Pydantic AI (Course)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)