AI gateway
An AI gateway is a server that sits between an application and one or more AI model providers, giving the application a single endpoint for every model it calls while centralizing credentials, routing, and usage controls.
Also called an LLM gateway, it works like a reverse proxy, forwarding each request to the right provider and passing the response back, but specialized for large language model (LLM) traffic. Most gateways expose an OpenAI-compatible API, so switching providers means changing a model name rather than rewriting client code. Typical responsibilities include:
- Routing: Sending each request to a provider or model based on its name, cost, or region
- Fallbacks: Retrying a failed or rate-limited request against another model or provider
- Access control: Issuing per-team keys while the gateway holds the real provider credentials
- Cost control: Counting tokens and spend, and enforcing budgets and rate limits
- Caching and logging: Answering repeated requests from a cache and recording traces for LLM observability
The simulation below sends requests through a gateway. Marking the primary provider as rate-limited shows the gateway falling back to a second provider while the app’s call stays the same:
Some gateways also apply guardrails to prompts and responses. The extra network hop adds latency, and the gateway becomes a single point of failure for every AI feature behind it. Hosted services such as OpenRouter and self-hosted proxies such as LiteLLM implement the pattern.
Related Resources
Tutorial
How to Use the OpenRouter API to Access Multiple AI Models via Python
Access models from popular AI providers in Python through OpenRouter's unified API with smart routing, fallbacks, and cost controls.
For additional information on related topics, take a look at the following resources:
- Accessing Multiple AI Models With the OpenRouter API (Course)
- How to Use Ollama to Run Large Language Models Locally (Tutorial)
- How to Integrate Local LLMs With Ollama and Python (Tutorial)
- Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
- How to Use the OpenRouter API to Access Multiple AI Models via Python (Quiz)
- How to Use Ollama to Run Large Language Models Locally (Quiz)
- How to Get Started With Ollama (Course)
- How to Integrate Local LLMs With Ollama and Python (Quiz)
- Building Type-Safe LLM Agents With Pydantic AI (Course)
- Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)
Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.
By Martin Breuss • Updated Nov. 6, 2026