LLM Gateway vs Router vs Inference Provider: The Architecture Boundary

An LLM gateway, a router, and an inference provider are three different layers in the model-serving stack, and confusing them is the most common mistake teams make when architecting AI products. The gateway sits closest to your application and handles auth, normalization, and observability; the router decides which model or provider handles a given request; the inference provider is the entity actually running the weights and returning tokens.

Getting this boundary wrong leads to brittle integrations: teams hardcode a single provider's SDK, then discover months later that adding a fallback model or comparing cost across Claude Sonnet 5, GPT-5.5, and DeepSeek V4 Flash means rewriting large parts of their request handling. This article separates the three layers, shows where responsibilities actually sit, and gives a decision framework for evaluating vendors in each category.

Key Takeaways

What an Inference Provider Actually Does

The inference provider is the layer that owns the model weights, the GPUs (or equivalent accelerators), and the low-level serving engine. This includes commercial API providers such as Anthropic (Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5), OpenAI (GPT-5.5), Google (Gemini 3.5 Flash), Zhipu (GLM-5.2), and DeepSeek (DeepSeek V4 Pro, DeepSeek V4 Flash), as well as self-hosted deployments of open-weight models like Qwen3.7 Plus, MiniMax M3, or Kimi K2.7 Code.

If you self-host, the inference provider layer is software you run yourself. vLLM's serving documentation describes an inference engine built around continuous batching and memory-efficient attention handling for serving LLM workloads at scale, which is the kind of component that sits directly under a self-hosted model deployment. Running vLLM (or a comparable engine) makes you the inference provider for that model, responsible for GPU capacity planning, scaling, and uptime, in exchange for control over cost and data locality.

If you use a commercial API, the provider's infrastructure and rate limits become your constraint. Each provider has its own authentication scheme, request and response schema, error codes, and rate-limit behavior. This is the layer where model quality, context window, and raw latency are actually determined. No router or gateway can change what an inference provider is capable of; they only change how you reach it.

What a Router Does

A router is decision logic that selects a destination, model, or provider for an incoming request. Routing can happen on several axes: cost (send simple queries to a cheap model like DeepSeek V4 Flash or Gemini 3.5 Flash, reserve Claude Opus 4.8 for complex ones), latency (prefer whichever provider currently responds fastest for a given model), or availability (fail over to a second provider if the first is degraded).

OpenRouter's public explainer on model routing describes this pattern at the model level: requests for an open-weight model can be routed across multiple providers that host the same weights, so a single logical model call can be fulfilled by any of several backends. OpenRouter's provider-selection documentation further describes parameters for expressing ordering preferences and for excluding specific providers from consideration, which is the mechanism by which a caller expresses routing intent explicitly rather than leaving it entirely to the platform.

The important architectural point is that routing is a policy, not a product category by itself. You can implement basic routing yourself in application code (a simple if/else on task type or a retry-on-error loop), buy it as a dedicated routing layer, or get it bundled inside a broader gateway. Evaluate routing capability by asking what signals it uses (price, latency, error rate, model capability) and whether those signals are configurable or fixed.

What a Gateway Does

A gateway is the layer your application code actually talks to. Its job is to present one consistent interface regardless of which inference provider ultimately serves the request. A gateway typically includes:

• A unified request and response schema across providers, so switching from GPT-5.5 to Claude Sonnet 5 does not require rewriting your parsing logic.
• Authentication and key management, so provider credentials are not scattered across application code.
• Logging, usage tracking, and cost attribution across every model and provider in use.
• Failover and retry behavior when a provider is slow, rate-limited, or returns an error.
• Optionally, routing logic as one feature among several, rather than the entire product.

TokenLab's model research page catalogs current models across providers, including frontier models (Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5, GPT-5.5, GLM-5.2, Gemini 3.5 Flash), coding-oriented models (Claude Sonnet 5, Kimi K2.7 Code, DeepSeek V4 Pro, DeepSeek V4 Flash), and low-cost routing candidates (DeepSeek V4 Flash, GLM-5.2, Laguna XS 2.1, Hy3, Qwen3.7 Plus, MiniMax M3), which is the kind of reference a gateway user needs when deciding what to route to. See TokenLab's writeup on why a unified AI API gateway matters in 2026 for a fuller treatment of the unification problem itself.

The Architecture Boundary, Concretely

The boundary is easiest to see by tracing a single request through all three layers.

1. Your application sends a chat completion request to the gateway endpoint, specifying a task type or a model preference.
2. The gateway authenticates the request, normalizes it into each candidate provider's expected schema, and hands off to the routing logic.
3. The router evaluates its configured policy (cost ceiling, latency target, or explicit provider order) and picks a destination, for example Claude Sonnet 5 via Anthropic's API, or a self-hosted DeepSeek V4 Pro instance served through vLLM.
4. The inference provider executes the model and returns tokens.
5. The gateway normalizes the response back into a consistent shape and logs the outcome (latency, cost, provider used, success or failure) for your observability layer.

Each layer can fail independently. A provider outage is an inference-layer problem; a bad routing decision (sending long-context requests to a model with a small context window) is a router-layer problem; an inconsistent response schema breaking your parser is a gateway-layer problem. Understanding which layer produced a given failure is what makes debugging tractable. TokenLab's piece on reliability infrastructure for AI APIs covers failure isolation across these layers in more depth.

Decision Checklist

Use this checklist when evaluating whether you need a gateway, a router, a direct provider integration, or some combination.

Question	If yes	If no
Do you call more than one model or provider today, or expect to within 12 months?	You need a gateway or router layer, not direct SDK calls per provider.	A direct provider SDK integration may be sufficient short term.
Do you need automatic failover when a provider is degraded or rate-limited?	You need router-level logic with health checks and fallback ordering.	Manual retries in application code may be acceptable initially.
Do you need per-model cost and usage visibility across providers in one place?	You need gateway-level logging and attribution.	Provider-native dashboards may cover your needs for now.
Are you self-hosting any open-weight models (GLM-5.2, DeepSeek V4 Pro, Qwen3.7 Plus, Kimi K2.7 Code)?	You are also operating an inference-provider layer and need serving infrastructure such as vLLM.	You can rely entirely on commercial API providers.
Do you need to route by task type (cheap model for classification, frontier model for reasoning)?	You need explicit router policy, not just failover.	A single default model may suffice.

Request-Shape Example

The example below illustrates the general shape of a gateway-style request that expresses both a primary model preference and a fallback list, a pattern consistent with how OpenRouter's provider-selection documentation describes expressing routing preferences. Treat field names as illustrative; verify exact parameters against your chosen gateway's current API reference before shipping.

POST /v1/chat/completions
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "model": "claude-sonnet-5",
  "fallback_models": ["gpt-5.5", "deepseek-v4-flash"],
  "routing_policy": {
    "strategy": "cost_then_latency",
    "max_cost_per_1k_tokens": 0.01
  },
  "messages": [
    {"role": "user", "content": "Summarize the attached incident report."}
  ]
}

In this shape, the gateway owns request normalization and the response contract, the router owns interpreting routing_policy and fallback_models, and whichever provider is ultimately selected owns actually generating the completion. Confirm the exact request and response schema against current documentation before relying on any specific field name in production.

Limitations

Public documentation from OpenRouter and vLLM describes general routing and serving mechanisms, not universal guarantees. Exact latency, pricing, and failover behavior vary by provider and change over time, so verify current numbers directly against provider documentation rather than this article. Self-hosting with vLLM shifts operational responsibility (capacity planning, scaling, patching) onto your team; it does not eliminate infrastructure work, it relocates it. No routing policy can compensate for a fundamentally mismatched model choice, such as sending a task requiring long-context reasoning to a model not suited for it; router configuration is not a substitute for evaluating model capability against your workload, which is why maintaining an up-to-date model reference such as TokenLab's model research matters.

FAQ

Is a gateway the same as a router? No. A router is decision logic for selecting a model or provider; a gateway is the broader application-facing layer that includes routing as one possible feature alongside auth, normalization, and logging.

Can I be my own inference provider and still use a gateway? Yes. Self-hosted models served through an engine like vLLM can sit behind the same gateway as commercial API providers, as long as the gateway supports custom or OpenAI-compatible endpoints.

Do I need all three layers on day one? Not necessarily. A single-provider direct integration is reasonable for an early prototype. As soon as you need failover, multi-model cost control, or provider comparison, introducing a gateway with routing becomes worth the engineering cost.

If you are evaluating which layer to adopt first, review current model options at TokenLab's model research page and Get Started with a gateway setup that lets you add routing and providers incrementally rather than committing to one integration upfront.