AI API Rate Limiting: How It Works and How to Handle It

A 429 response from an AI API can mean four different things: too many requests per minute, too many tokens per minute, too many concurrent connections, or a depleted account balance. All four return the same status code, so the fix depends on diagnosing which limit you actually hit, not on retrying harder. This guide breaks down the failure modes, shows what evidence is and is not available on current limits, and gives you a retry and fallback pattern that holds up in production.

Key Takeaways

Understanding AI API Rate Limiting: The Four Failure Modes

Request limits

Most providers count requests per minute (RPM). Exceed it and you get an instant 429, often with an empty body. A user rapidly paging through results, or a cron job firing without a throttle, is a common trigger.

Token limits

The trap teams underestimate most. Providers frequently enforce tokens per minute (TPM) separately from RPM, so you can hit a 429 while comfortably under your request ceiling. Large-context models make this worse: per the TokenLab live pricing evidence (observed 2026-07-07), Claude Opus 4.8 and GPT-5.5 both support roughly 1,000,000+ tokens of context. A call that loads a large document, a full codebase, or a long chat history into that window can consume a large share of your per-minute token budget in a single request, even though it is only "one" request. This is a capacity-planning risk based on context window size, not a measured per-call token count, so validate actual usage against your own logs rather than assuming a fixed figure.

Concurrency limits

Providers may tolerate your average per-minute volume right up until you open fifty streams at once. Concurrency limits cap simultaneous in-flight requests or connections. Streaming responses hold connections open longer, which burns through concurrency slots faster than short, single-shot calls. Coding agents built on Claude Sonnet 5 or Kimi K2.7 Code, and voice interfaces streaming from Gemini 3.5 Flash, are common triggers because they keep many long-lived connections open at once.

Quota or balance exhaustion

This looks identical to a rate limit in your dashboard: calls stop working. But the fix is different. If your account runs out of prepaid credits or hits a hard daily spend cap, the API returns an error that resembles a rate limit. Backoff does nothing here. You need to top up the balance or raise the spend threshold.

Source Snapshot

Data point	Source	Observed at
Model context windows and per-token pricing	TokenLab live model/pricing evidence	2026-07-07
Model SSOT naming (Claude Sonnet 5, GPT-5.5, Gemini 3.5 Flash, etc.)	TokenLab model SSOT	2026-07-07, expires 2026-07-14
Official provider RPM/TPM tier limits	Not available in this evidence set	Not verified, check provider account dashboard
TokenLab gateway header normalization behavior	Not available in this evidence set	Verify in TokenLab API docs before relying on a single header schema

Current Model Context Windows and Pricing (TokenLab Live Evidence)

These figures come directly from the TokenLab live pricing snapshot. They are not RPM or TPM limits, they show why a single call on a large-context model can eat disproportionately into a token budget.

Model	Provider	Context window	Input $/MTok	Output $/MTok	Source	Observed
Claude Sonnet 5	Anthropic	1,000,000	$2.00	$10.00	TokenLab live pricing evidence	2026-07-07
Claude Opus 4.8	Anthropic	1,000,000	$5.00	$25.00	TokenLab live pricing evidence	2026-07-07
Claude Fable 5	Anthropic	1,000,000	$10.00	$50.00	TokenLab live pricing evidence	2026-07-07
GPT-5.5	OpenAI	1,050,000	$5.00	$30.00	TokenLab live pricing evidence	2026-07-07
GPT-5.5 Batch/Flex	OpenAI	1,050,000	$2.50	$15.00	TokenLab live pricing evidence	2026-07-07
Gemini 3.5 Flash	Google	1,048,576	$1.50	$9.00	TokenLab live pricing evidence	2026-07-07
GLM-5.2	Z.ai	1,048,576	$0.93	$3.00	TokenLab live pricing evidence	2026-07-07
Kimi K2.7 Code	Moonshot AI	262,144	$0.74	$3.50	TokenLab live pricing evidence	2026-07-07
DeepSeek V4 Pro	DeepSeek	1,048,576	$0.44	$0.87	TokenLab live pricing evidence	2026-07-07
DeepSeek V4 Flash	DeepSeek	1,048,576	$0.09	$0.18	TokenLab live pricing evidence	2026-07-07
Qwen3.7 Plus	Alibaba	1,000,000	$0.32	$1.28	TokenLab live pricing evidence	2026-07-07
MiniMax M3	MiniMax	1,048,576	$0.30	$1.20	TokenLab live pricing evidence	2026-07-07

Model SSOT note: these names reflect the TokenLab model SSOT observed on 2026-07-07, expiring 2026-07-14. Naming and availability change frequently. Before you hardcode a model string into production code, confirm it still resolves in the TokenLab model directory or model leaderboard.

Fix Rate Limit Failures Without Building Retry Logic From Scratch

Everything above is diagnosis. The remediation work, deciding which model to fall back to, tracking per-user concurrency, and keeping a live picture of which model families are healthy, is exactly what TokenLab's routing layer is built to absorb. Instead of hand-rolling a fallback matrix across five providers, you point requests at TokenLab and let the gateway select among the current model catalog based on availability and your fallback rules.

One honest caveat: TokenLab sits in front of multiple upstream providers, and each upstream returns its own header set, error format, and retry semantics. Whether the gateway fully normalizes every rate-limit header into one consistent schema, or passes some upstream headers through unchanged, is not confirmed in the evidence available for this article. Verify current header behavior in TokenLab's API documentation before you write parsing logic that assumes a single unified schema across every model. Build your header parser defensively, checking for the presence of each field rather than assuming it exists.

Reading Rate-Limit Headers

Providers return rate-limit information in response headers, though exact names vary by provider. A common pattern looks like this:

x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 499
x-ratelimit-reset-requests: 12s
retry-after: 0

Read these on successful responses, not just after a 429. Keep a rolling count of remaining budget and slow down when you dip below a safety threshold, typically 10-20% headroom, though the right number depends on your traffic burstiness and is not something this evidence set can specify for you.

Retry Logic With Explicit Error Handling

A retry helper needs to handle more than the 429 case. It should distinguish transient errors (429, 503, timeouts) from client errors (4xx other than 429) that will never succeed on retry, and it should respect retry-after when present.

async function callWithRetry(fn, maxRetries = 4) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const status = err.status;

      // Client errors other than 429 will not succeed on retry
      if (status && status >= 400 && status < 500 && status !== 429) {
        throw err;
      }

      // Give up after max retries regardless of error type
      if (attempt === maxRetries) throw err;

      // 429: honor retry-after if present, otherwise backoff with jitter
      if (status === 429) {
        const retryAfter = parseRetryAfterHeader(err);
        const delay = retryAfter
          ? retryAfter * 1000
          : Math.pow(2, attempt) * 1000 + Math.random() * 1000;
        await sleep(delay);
        continue;
      }

      // 503 or network timeout: backoff and retry, log for observability
      if (status === 503 || err.code === 'ETIMEDOUT' || err.code === 'ECONNRESET') {
        await sleep(Math.pow(2, attempt) * 1000 + Math.random() * 1000);
        continue;
      }

      // Unknown 5xx: retry with backoff, cap attempts tightly
      if (status && status >= 500) {
        await sleep(Math.pow(2, attempt) * 1000 + Math.random() * 1000);
        continue;
      }

      throw err;
    }
  }
}

Never wrap non-idempotent operations, like a payment charge or a write with side effects, in a naive retry loop. Confirm idempotency or use an idempotency key before retrying those calls.

Token Estimation Accuracy: Why Local Counts Drift From Provider Counts

Client-side token counting libraries approximate the tokenizer a given model actually uses. tiktoken matches OpenAI's family closely but is not guaranteed to match Anthropic, Google, or open-weight tokenizers exactly. Differences show up around special tokens, multi-byte characters, and system prompt formatting, and they compound in long conversations.

Practical steps to reduce drift:

• Read the actual token usage returned in the response body (most providers include prompt and completion token counts) and use that to calibrate your local estimator over time.
• Keep a safety margin, not a hard boundary. If your provider's TPM cap is close to your estimated usage, delay or split the request rather than sending it right at the edge.
• If your prompts are large or repetitive across calls, see TokenLab's coverage of tokenization behavior and prompt splitting strategies for concrete ways to cut tokens per request before you hit a ceiling at all. Reducing tokens per call is often a cheaper fix than negotiating a higher tier.

Traffic Shaping and Model Fallback

Exponential backoff with jitter. Double the wait after each retry and add random jitter so concurrent clients do not retry in lockstep.

Traffic shaping per user or task. Cap concurrent calls per user (for example, 3 concurrent, 5 requests per second burst) so one heavy user cannot exhaust your account-wide limit and degrade everyone else.

Token budget estimation before send. Count tokens client-side, and if a request would push you over your tracked TPM budget, delay or split it rather than sending and hoping.

Model fallback as a safety net. When a primary model returns a 429, route to an alternate with separate limits and comparable capability. A coding task can fail over from Claude Sonnet 5 to DeepSeek V4 Pro or Kimi K2.7 Code. A high-volume, low-cost workload can fail over between DeepSeek V4 Flash, Gemini 3.5 Flash, and Qwen3.7 Plus, all of which sit in the low-cost routing tier of the current model catalog.

Rate Limit Handling Checklist

Category	What to check	Immediate action	Long-term fix
Request limits	RPM header values, burst frequency	Throttle client requests, add a local rate limiter	Per-user tiers, server-side queueing
Token limits	TPM budget per model, average tokens per call, response-reported usage	Pre-count tokens, split large prompts, delay calls near the ceiling	Batching with token budgeting, route high-volume work to lower-cost models
Concurrency limits	Max simultaneous streams or connections	Cap concurrent requests per client, close idle streams	Connection pooling, staggered stream launches
Quota / balance	Account balance, daily spend caps	Top up credits, adjust spend thresholds	Low-balance alerts, prepaid auto-refill

Typical Limit Ranges: What We Can and Cannot Confirm

Secondary searches for this topic often want a table of "typical" RPM/TPM numbers. We are not going to invent one. Published per-tier limits change frequently, vary by account history and usage tier, and are not part of the evidence available for this article.

Question	Status	Verification step
What is a "typical" RPM limit for a frontier model?	Not confirmed in this evidence set	Check your provider's account dashboard or rate-limit response headers directly
What is a "typical" TPM limit for a 1M-context model?	Not confirmed in this evidence set	Log actual usage from response headers over a week of traffic to build your own baseline
Does usage tier change these numbers?	Plausible based on general provider behavior, not benchmarked here	Confirm current tier limits in your provider console
Do TokenLab's aggregated limits match upstream provider limits exactly?	Not confirmed in this evidence set	Verify in TokenLab's API documentation before capacity planning

Limitations

• No official RPM/TPM figures were available for GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, or any other model referenced here. Every number in the pricing tables above is context window size and per-token cost, not a rate limit.
• Whether TokenLab's gateway fully unifies rate-limit headers across every upstream provider, or passes some through unchanged, is not confirmed in this evidence set. Treat header parsing as provider-specific until you verify current gateway behavior in the docs.
• Model names in this article reflect the TokenLab model SSOT observed 2026-07-07, expiring 2026-07-14. Confirm current availability in the model directory before shipping code that hardcodes a model string.
• Token estimation accuracy versus provider-side counting was not benchmarked in this evidence set. Calibrate your own estimator against response-reported usage rather than trusting a fixed offset.

FAQ

Why does my app get 429 errors even though the request count is under the limit? Check the token-per-minute (TPM) budget first. A single large prompt can exhaust the token allowance while the request count stays low. Also check concurrency: open streaming connections can block new requests even when RPM and TPM both look fine.

Should I retry immediately when I receive a 429? No. Wait for the period specified in retry-after, or use exponential backoff with jitter if that header is absent. Immediate retries risk a thundering-herd effect and can extend your lockout window.

How do I know if my local token count will match what the provider actually bills me for? You do not, exactly. Client-side tokenizers are approximations. Read the token usage returned in each response body and use it to calibrate your estimator over time, and keep a safety margin rather than sending requests right at your estimated ceiling.

Does TokenLab expose one unified set of rate-limit headers across every model? Not confirmed in the evidence available here. Different upstream providers return different header formats, and how much normalization TokenLab applies is something to verify in current API docs rather than assume.

How can I avoid rate limits without upgrading my plan? Combine token budgeting, local concurrency caps, and model fallback. Estimate tokens before sending, split long prompts, and route to an alternate model, for example from Claude Sonnet 5 to DeepSeek V4 Pro, when your primary limit is hit.

Get Started

Rate limits are a fact of building on AI APIs, but they do not have to cause outages. TokenLab's gateway gives you access to the current model catalog and live pricing and throughput data so you can route around limits instead of guessing at them. If you are also evaluating aggregator tradeoffs, the OpenRouter comparison covers fallback behavior and operational overhead in more depth. Get your API key at tokenlab.sh and build retry and fallback logic once, not per provider.