The 7 Best LLM Gateways and Model Routers in 2026, Graded on What Happens Mid-Conversation
Compiled by Lifan Xu. Published September 24, 2026 · Last updated September 24, 2026.
An LLM gateway puts one API between your agents and every model provider, and in 2026 the seven worth shortlisting all handle failover. The difference is what they do to a customer conversation that is halfway through when the upstream model stops answering.
TL;DR: Every gateway here fails over a request; none fails over a conversation — OpenAI's own docs confirm stored response state and prompt caches stay with one provider, so your application must own the transcript whichever gateway you choose.
Methodology & sources
- Seven gateways graded on four operational criteria: failover behaviour, mid-conversation state, per-request cost attribution, and added latency. Self-hosting and licence are listed but not scored.
- Figures come from each vendor's own docs and GitHub READMEs. Every latency number is vendor-published; we did not benchmark these gateways ourselves.
- State and caching behaviour come from OpenAI's conversation state and prompt caching guides.
- All figures verified September 2026. Disclosure: this is Aissist's blog. Aissist sells AI agents for customer service, not an LLM gateway, so no product on this list is ours.
What is the difference between an LLM gateway, a model router, a proxy and a control plane?
An LLM gateway is the single endpoint your application calls; a router, a proxy and a control plane are three jobs that sit inside or around it. Page one of Google uses all four words for the same thing. They are not the same thing.
| Term | What it decides | Example |
|---|---|---|
| LLM proxy | Nothing. Translates one API format into many providers' formats | LiteLLM's Python SDK |
| Model router | Which model gets this request: by cost, task, or health | OpenRouter's Auto Router |
| LLM gateway | Whether the request goes out, and where it goes if it fails: keys, retries, fallbacks, rate limits, logs | Portkey, Cloudflare AI Gateway |
| Control plane | Policy for all of the above across teams: budgets, access, audit | Kong Konnect, Prisma AIRS |
The practical test: a proxy saves you writing adapters, a router saves you money, a gateway saves your uptime, and a control plane saves your security review. One more collision to clear up. On our own site, Aissist's gateways are the connections into your helpdesk and CRM, not to model providers. Same word, opposite end of the agent. The industry named everything "gateway" at once and is now paying for it in search results.
Which LLM gateways are best in 2026?
LiteLLM is the best self-hosted default, and Vercel AI Gateway or OpenRouter are the fastest managed starts. Portkey leads on governance, Bifrost on raw overhead, Kong for teams that already run it, and Cloudflare for edge caching. The table grades each one on the four things that break in production.
| Gateway | Failover behaviour | Mid-conversation state | Per-request cost attribution | Added latency, vendor-published (Sep 2026) | Self-host | Licence |
|---|---|---|---|---|---|---|
| LiteLLM | Ordered fallbacks after retries; cooldowns; separate context-window and content-policy fallbacks | Stateless per request | Spend per key, team and user | 8 ms p95 (4 instances, 1,170 RPS) | Yes | MIT |
| Portkey (Palo Alto Networks) | Ordered list; any non-2xx by default, or chosen status codes | Stateless per request | Per-request cost | "<1ms" | Yes | MIT (gateway) |
| OpenRouter | models array; on context length, moderation, rate limit, downtime | Stateless per request | Billed on the model that actually served | Not published | No | Proprietary |
| Vercel AI Gateway | Provider order plus model fallbacks; BYOK can fall back to system credentials | Stateless per request | Cost and every routing attempt logged per request | "sub-20ms" | No | Proprietary |
| Cloudflare AI Gateway | On errors or preset timeouts; cf-aig-step header shows which step answered | Stateless per request | Aggregate cost analytics | Not published | No | Proprietary |
| Kong AI Gateway | Fails over when a provider is slow or unavailable | Stateless per request | "True cost of each request" incl. cache and service tier | Not published | Yes | Apache 2.0 core; advanced AI features commercial |
| Bifrost (Maxim) | Automatic fallbacks across providers and keys | Stateless per request | Budgets by virtual key, team, customer | 11 µs (5,000 RPS, t3.xlarge) | Yes | Apache 2.0 |
- LiteLLM — best for self-hosted, multi-team spend control. LiteLLM's benchmarks show 8 ms p95 overhead on four instances, but 29 ms p95 on two. Con: that overhead depends on how many instances you run.
- Portkey — best for governance-heavy enterprises. Palo Alto Networks completed its acquisition on 29 May 2026. Con: the roadmap now answers to a security company, for better or worse.
- OpenRouter — best for trying many models fast. Con: bring-your-own-key usage above $25,000 a month carries a 5% fee, per its FAQ.
- Vercel AI Gateway — best for teams with no ops budget. It adds "zero markup to provider token prices, including with BYOK," Vercel's docs state. Con: budgets cover only system-credential spend, so BYOK spend is tracked separately.
- Cloudflare AI Gateway — best for edge caching of repeat prompts. Con: no published overhead figure and no self-host option.
- Kong AI Gateway — best if Kong already runs your APIs. Con: the advanced AI features sit in the commercial tier.
- Bifrost — best for latency-critical, self-hosted traffic. Con: 11 µs is a vendor benchmark on a vendor-chosen instance.
Notice the state column. It says the same thing seven times, and that is the finding.
"Routing non-deterministic AI workloads through a centralized gateway can introduce latency, integration complexity, and potential architectural concentration risk." — Stephanie Walter, Practice Leader, AI Stack, HyperFRAME Research
What happens to a live conversation when the upstream model fails?
The gateway resends the failed turn to the fallback model, and the conversation survives only if that turn carries the full history. Gateways are stateless by design. They retry a request; they do not remember the conversation it belongs to.
That was fine when every call was a self-contained Chat Completions payload. It is not fine now that providers hold state for you. With OpenAI's Responses API, a turn can point at previous_response_id instead of resending the history, and response objects are "saved for 30 days by default," OpenAI's conversation-state guide says. A fallback to Anthropic or Gemini cannot resolve that ID. The customer's refund request becomes a stranger's first message.
Caches break the same way. OpenAI discounts cached input "up to 90%," but "caches are not shared across organizations," its prompt caching guide notes. A different provider means a cold cache. So the first turn after failover is the slowest and most expensive turn in the whole conversation.
In Aissist's own AgentMesh™ deployment, which runs on OpenAI's Responses API with prompt caching keyed per conversation, this is the cost that never appears on a gateway's pricing page. A fallback turn starts cold: full input price, slower first token, on exactly the ticket that was already in trouble. Our rule is that the application stores the transcript and the gateway only ever sees complete turns.
"If Prisma AIRS is the runtime security layer for AI, a gateway that intercepts and governs LLM API traffic is a natural adjacency." — Fernando Montenegro, VP and Practice Lead, Cybersecurity & Resilience, The Futurum Group
Dated note: the AutoGen transition (as of 24 September 2026) The AutoGen README now says AutoGen "is now in maintenance mode" and "will not receive new features or enhancements," and points new users to Microsoft Agent Framework. If your agents run on AutoGen, a migration is coming; our agent orchestration frameworks comparison covers the options. Keep model access behind a gateway and it becomes a framework swap, not a framework swap plus a provider rewire.
How should you choose an LLM gateway?
Choose on who owns failure, not on how many models a gateway lists. Every gateway here reaches the major providers. What differs is where it runs, who pays for its mistakes, and what it tells you afterwards.
- If you must keep data in your own cloud, choose LiteLLM or Bifrost, because both are permissively licensed and self-hosted.
- If you need budgets and audit across many teams, choose Portkey or Kong, because both treat the gateway as a control plane.
- If you have no one to run infrastructure, choose Vercel AI Gateway or OpenRouter, because both are managed and bill per token.
- If first-response time is your SLA, measure p95 with your own traffic, because every figure in the table is vendor-published.
- If you use
previous_response_idor OpenAI's new Agents API sessions, keep a transcript in your own store, because no gateway can move provider-held state to another provider.
That last rule got more urgent this month. OpenAI released its Agents API in public beta on 10 September, with "durable sessions to continue work across turns," its changelog says. Durable for OpenAI. Not portable to anyone else. See our build vs buy guide for where the gateway line sits in a larger AI stack, and token efficiency for why cached tokens matter to the bill.
"AI is evolving so rapidly that organizations often feel forced to choose between two failing strategies" — Lee Klarich, Chief Product & Technology Officer, Palo Alto Networks
Fail over the request, own the conversation
The best LLM gateway for your stack is the one that matches where you run and who governs spend, and the table settles that in a minute. None of them keeps a customer conversation intact across providers, because none of them is meant to. That job sits one layer up, in the application or the multi-agent platform that owns the transcript, the cache strategy and the escalation path. Pick the gateway for uptime and cost. Design the state yourself. For how this layer fits the wider stack, read the rise of AI middleware.
Frequently asked questions
What is an LLM gateway?
An LLM gateway is a single API endpoint that sits between your application and model providers. It handles keys, retries, fallbacks, rate limits and logging, so one provider's outage does not take your product down with it.
What is the difference between an LLM gateway and a model router?
A model router decides which model should handle a request, usually by cost, task or health. A gateway decides whether the request goes out and what happens when it fails. Many products do both, which is why the terms blur.
Does an LLM gateway add latency?
Yes, though vendor figures are small: Bifrost publishes 11 µs at 5,000 RPS and LiteLLM 8 ms p95 on four instances. The bigger latency hit after failover usually comes from a cold prompt cache on the fallback provider.
Is LiteLLM free?
LiteLLM's core gateway is open source under the MIT licence and free to self-host. You still pay the model providers, plus the infrastructure you run it on.
Can I use an LLM gateway with OpenAI's Responses API?
Yes. Vercel AI Gateway, for example, lists the Responses API among its supported formats. But a fallback to another provider cannot reuse previous_response_id, so send full history on any turn that might fail over.
Do I need an AI gateway if I only use one model provider?
Often yes, for retries across regions, spend limits and logs. The case gets stronger the moment a second provider or a second team appears.
