Skip to content

AI gateway

An AI gateway is a server that sits between an application and one or more AI model providers, giving the application a single endpoint for every model it calls while centralizing credentials, routing, and usage controls.

Also called an LLM gateway, it works like a reverse proxy, forwarding each request to the right provider and passing the response back, but specialized for large language model (LLM) traffic. Most gateways expose an OpenAI-compatible API, so switching providers means changing a model name rather than rewriting client code. Typical responsibilities include:

  • Routing: Sending each request to a provider or model based on its name, cost, or region
  • Fallbacks: Retrying a failed or rate-limited request against another model or provider
  • Access control: Issuing per-team keys while the gateway holds the real provider credentials
  • Cost control: Counting tokens and spend, and enforcing budgets and rate limits
  • Caching and logging: Answering repeated requests from a cache and recording traces for LLM observability

The simulation below sends requests through a gateway. Marking the primary provider as rate-limited shows the gateway falling back to a second provider while the app’s call stays the same:

Interactive diagram — enable JavaScript to view.

Some gateways also apply guardrails to prompts and responses. The extra network hop adds latency, and the gateway becomes a single point of failure for every AI feature behind it. Hosted services such as OpenRouter and self-hosted proxies such as LiteLLM implement the pattern.

How to Use the OpenRouter API to Access Multiple AI Models via Python

Tutorial

How to Use the OpenRouter API to Access Multiple AI Models via Python

Access models from popular AI providers in Python through OpenRouter's unified API with smart routing, fallbacks, and cost controls.

intermediate ai api

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.


By Martin Breuss • Updated Nov. 6, 2026