What is an LLM gateway?

An LLM gateway, also sold as an AI gateway, is a proxy that sits between your applications and the model providers. LLM stands for large language model, the engine behind modern AI, which you pay for by the token: the unit models read and write, roughly three-quarters of a word. The gateway is the central place for API keys, rate limits, spend tracking, and failover across models, so a hundred internal apps do not each hold their own provider key and their own untracked bill.

If you are approving an AI budget, this is the layer that determines whether you can answer three questions later: what did we spend, who spent it, and could the same work have run cheaper. Without a gateway, those answers live scattered across provider dashboards and engineers' memories.

Why it exists, and what breaks without one

AI costs scale with how much your agents read and write, not how often you call them, and agents read and write a lot. The biggest single lever on the bill is model routing: automatically sending simple tasks to a cheap, fast model and reserving frontier models for the requests where judgment matters. Routing needs a place to live, and that place is the gateway.

The second lever is catching pricing changes and cache behavior, and 2026 kept supplying them. Anthropic's Claude Fable 5.1 bills cache hits at $0.25 per million tokens against $1 on Fable 5, a 75% cut on that line. Claude Opus 5.5, announced September 22, cut list prices 20% from Opus 5, to $4 and $20 per million input and output tokens from $5 and $25, and cache reads 60%, to $0.20. The increase Anthropic had scheduled for Sonnet 5 on September 1, to $3 and $15, was cancelled. Then on October 7, Anthropic halved cache reads on Sonnet 5.5 to $0.10 and launched Claude Haiku 5.5 at $0.10 and $0.50, with a threshold attached: once a request's prompt passes 100,000 tokens, every token in that request bills at five times those rates, cached tokens included (Anthropic pricing; Simon Willison; Simon Willison on Haiku 5.5). Each saving only materialized for teams whose calls actually landed as cache hits, or whose routing noticed the change. A gateway is where you would see that a reordered prompt was silently billing at full input rates, or that a long agent session had drifted over a price threshold.

The third reason is damage control. AI model evaluator METR disclosed that an API key stolen in March 2026 ran for about three weeks and consumed credits worth about $600,000, and that there were no caps on token spend on the key (The Hacker News). A hard spend cap per key, enforced at a gateway, is the control that was missing.

One caution belongs in every evaluation: the gateway concentrates your provider keys and your traffic, which makes it a target. In September 2026, the Cybersecurity and Infrastructure Security Agency (CISA) added an improper-authentication flaw in the widely used LiteLLM gateway (CVE-2026-59822) to its Known Exploited Vulnerabilities catalog, meaning attackers were exploiting it in the wild (CISA). Three weeks later, CVE-2026-90898, rated 9.8, let an unauthenticated attacker run commands on the Bifrost AI gateway with one HTTP request (incident record). A control point you do not patch becomes the breach.

What to ask vendors

• Can spend be attributed per team, per application, and per agent, rather than only per provider account?
• Are budget limits hard caps that stop traffic, or alerts that arrive after the money is gone?
• How are routing policies defined, and can we test a policy change against last month's real traffic before enabling it?
• Where do provider keys live, how are they rotated, and who can read them?
• What is the security track record and patch turnaround? Ask specifically how fast known exploited vulnerabilities get fixed in your deployment model.
• How much latency does the proxy add, and what happens to traffic when the gateway itself is down?
• If we leave, does our code keep working? Gateways that mimic a standard API shape are cheaper to exit.