LLM Latency vs Throughput: How API Buyers Should Measure Speed

The model that showed its first visible token soonest in our latest test was also the slowest to finish. On 2026-10-03 we streamed one prompt through four models. claude-sonnet-5-5 led on first visible token (2,981 ms) and trailed on total time (12,384 ms). That gap is the core of LLM latency vs throughput: one number says how long a user waits to see text, another says how fast the work completes. This piece defines each metric, shows our small first-party measurements with their limits, and gives a harness you can run against the TokenLab API.

Key Takeaways

Three speed numbers: LLM latency vs throughput in practice

Break "speed" into parts, because each part has a different owner: the network, the model and your own queue.

send request
  -> first stream event          (any SSE chunk; may be a reasoning delta)
  -> first visible content token (first non-empty delta.content)
  -> last token + final usage chunk

Time to first token (TTFT) is the time from sending the request to the first token arriving. We record two versions. The first stream event is the first SSE chunk of any kind. The first visible content token is the first chunk whose choices[0].delta.content is non-empty. For models that reason before answering, the first event can be a reasoning delta (delta.reasoning_content) that lands well before visible text. A harness that watches only delta.content measures time to first visible token, not time to first byte. Users feel the visible one.

Total time runs from sending the request to the end of the stream. It includes reasoning, network time and any retry wait you add on top.

Output tokens per second is output tokens divided by total time. Take the token count from usage.completion_tokens, which includes reasoning tokens. We divide by total time, not by the visible window. Some models in our sample delivered visible text in a few large chunks, so a window-based rate would mislead.

Workload	Watch first	Then	Why
Chat	First visible token	Total time	The user is looking at a blank box until visible text appears.
Agents	Total time	First visible token	Each step usually waits for the previous one, so per-step totals add up.
Batch extraction or summaries	Output tokens per second, and cost per task	Request limit headroom	Nobody watches the first token, and the account's request limit caps how fast you can submit work.

Non-streaming calls, such as structured JSON extraction, only expose total time. Stream them in testing if you want the split.

What we measured on 2026-10-03

Method. This is a first-party TokenLab measurement, observed 2026-10-03, against POST https://api.tokenlab.sh/v1/chat/completions (source). Each request used stream=true and stream_options.include_usage=true. We ran 5 sequential runs per model from a single client in East Asia, using Node 24 fetch. The prompt was "Explain how HTTP keep-alive reduces latency for API clients. Write about 200 words." Every model used default reasoning settings. Timings are client-side wall clock, and all runs succeeded (5/5) for every row below.

Model (max_tokens 2000)	Median first stream event	Median first visible token	Median total time	Median completion tokens (reasoning)	Estimated tokens/s
deepseek-v4.1-flash	2,334 ms	3,925 ms	5,204 ms	453 (225)	87
gemini-3.8-flash	6,892 ms	6,892 ms	8,626 ms	826 (528)	96
claude-sonnet-5-5	2,981 ms	2,981 ms	12,384 ms	418 (not reported)	34
gpt-5.6-luna	3,262 ms	3,262 ms	8,183 ms	313 (42)	38

The tokens/s column is our own estimate: median completion tokens divided by median total seconds, for example 453 ÷ 5.204 ≈ 87. It is a ratio of medians, not a median of per-run rates.

What the numbers show, and no more:

• Ordering by first visible token (claude-sonnet-5-5, gpt-5.6-luna, deepseek-v4.1-flash, gemini-3.8-flash) differs from ordering by total time (deepseek-v4.1-flash, gpt-5.6-luna, gemini-3.8-flash, claude-sonnet-5-5). One metric cannot stand in for the other.
• deepseek-v4.1-flash had a 1,591 ms gap between first event and first visible token. Its median visible output was about 228 tokens, delivered in 87 events over 1,304 ms.
• For gemini-3.8-flash the first event and first visible token share a median of 6,892 ms. Its visible text came in a median of 11 events over 1,734 ms, so the stream was chunky rather than token by token.
• claude-sonnet-5-5 sent a median of 371 events over 9,186 ms, and gpt-5.6-luna sent 265 events over 4,901 ms.

Cost sits beside speed. The prices below come from the TokenLab model API, observed 2026-10-03, with pricing updated 2026-10-02T16:53:30.068Z. The last column is an estimate: median completion tokens times the output rate, output side only. Check the usage records for actual charges.

Model	Input per 1M	Output per 1M	Max output tokens	Output-side estimate, median run	Source
deepseek-v4.1-flash	$0.15 off-peak, $0.30 peak	$0.60 off-peak, $1.20 peak	384,000	453 × $0.60 / 1M ≈ $0.00027 off-peak, ≈ $0.00054 peak	model API
gemini-3.8-flash	$0.375	$1.875	65,536	826 × $1.875 / 1M ≈ $0.00155	model API
claude-sonnet-5-5	$0.60	$3	128,000	418 × $3 / 1M ≈ $0.00125	model API
gpt-5.6-luna	$0.06	$0.36	128,000	313 × $0.36 / 1M ≈ $0.00011	model API

For gpt-5.6-luna, those rates apply when input is at most 272K tokens. Above that, the full request uses 2x input and 1.5x output rates.

Limits of this sample. Five sequential runs from one client in East Asia is a thin slice. The prompt was one short essay request, so results will shift with region, time of day, prompt and output length. We did not test concurrency, so none of this says what happens under load. Treat the table as a template for your own test, run from the region where production runs. If you want a wider view of models, the TokenLab LLM API leaderboard for developers is a starting point. Check its method and date before you trust any speed figure there. Do not swap models silently either: the reliability guide, observed 2026-10-03, notes that a replacement can change quality, price, context length and tools.

Reasoning tokens and max_tokens: why 400 returned almost nothing

We also ran the same prompt with max_tokens set to 400. Same method, same five runs, same date (source).

Model (max_tokens 400)	Median completion tokens	Median reasoning tokens	Visible output (approx.)	Median first visible token	Median total time
deepseek-v4.1-flash	400	400	0 tokens	3,795 ms	4,693 ms
gemini-3.8-flash	396	382	14 tokens	4,199 ms	4,240 ms
claude-sonnet-5-5	400	not reported	400 tokens	3,088 ms	11,078 ms
gpt-5.6-luna	317	42	275 tokens	3,040 ms	8,083 ms

Medians are computed per column, so one row's columns need not describe a single run. That is why deepseek-v4.1-flash shows a first-visible time next to a median of zero visible events.

The finding is plain. deepseek-v4.1-flash and gemini-3.8-flash spent nearly the whole 400-token budget on reasoning and returned almost no visible text. Raising max_tokens to 2000 produced complete answers. Reasoning tokens count inside usage.completion_tokens. They are reported separately in usage.completion_tokens_details.reasoning_tokens.

When we saw the first 400-token result, the failure looked like an empty answer, not a budget problem. Three checks make it visible:

1. Read the final usage chunk, enabled by stream_options={"include_usage": True}. If reasoning_tokens is close to completion_tokens and the text is empty, the budget went to reasoning.
2. Check finish_reason. The streaming guide (observed 2026-10-03) lists length as the value when a token limit is hit.
3. Compare against the model's maximum output. For example, the model API lists 384,000 for deepseek-v4.1-flash and 65,536 for gemini-3.8-flash, so 400 was a tiny slice of what the models allow.

The Create Chat Completion reference (observed 2026-10-03) also lists max_completion_tokens and reasoning_effort. It says supported values depend on the model, so read the model page before setting either. We did not test them, and we ran every row with default reasoning. Size the cap from your own prompt: in our run, a 200-word answer produced roughly 228 to 418 visible tokens, and up to 528 reasoning tokens on top.

A measurement harness for the TokenLab base URL

This script uses the TokenLab base URL, reads token counts from the usage chunk, and records the first stream event separately from the first visible content token. It also keeps response headers for later debugging. It follows the client settings in the docs (timeout=30.0, max_retries=0) so the SDK does not add hidden retries to your timings. The model ID is one we measured; swap in any ID whose accepted request formats include openai_chat_completions.

import json
import os
import statistics
import time

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOKENLAB_API_KEY"],
    base_url="https://api.tokenlab.sh/v1",
    timeout=30.0,
    max_retries=0,
)

PROMPT = "Explain how HTTP keep-alive reduces latency for API clients. Write about 200 words."

def measure(model: str, max_tokens: int = 2000) -> dict:
    start = time.perf_counter()
    raw = client.chat.completions.with_raw_response.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT}],
        max_tokens=max_tokens,
        stream=True,
        stream_options={"include_usage": True},
    )
    headers = raw.headers
    first_event = first_visible = None
    visible_events = 0
    usage = None
    finish_reason = None

    for chunk in raw.parse():
        now = time.perf_counter() - start
        if first_event is None:
            first_event = now
        if chunk.usage is not None:
            usage = chunk.usage
        if not chunk.choices:
            continue
        choice = chunk.choices[0]
        if choice.delta.content:
            visible_events += 1
            if first_visible is None:
                first_visible = now
        if choice.finish_reason:
            finish_reason = choice.finish_reason

    total = time.perf_counter() - start
    details = getattr(usage, "completion_tokens_details", None)
    completion = usage.completion_tokens if usage else None
    return {
        "model": model,
        "request_id": headers.get("x-request-id"),
        "routing_time_ms": headers.get("x-routing-time-ms"),
        "first_event_s": first_event,
        "first_visible_s": first_visible,
        "total_s": total,
        "completion_tokens": completion,
        "reasoning_tokens": getattr(details, "reasoning_tokens", None),
        "visible_events": visible_events,
        "finish_reason": finish_reason,
        "tokens_per_s": completion / total if completion else None,
    }

if __name__ == "__main__":
    runs = [measure("gemini-3.8-flash") for _ in range(5)]
    print(json.dumps(runs[0], indent=2))
    print("median total_s:", statistics.median(r["total_s"] for r in runs))

A few notes on reading the output:

• first_event_s fires on any chunk, including a role-only or reasoning chunk. first_visible_s fires only on non-empty delta.content. If the two differ by seconds, the model is reasoning before it answers.
• Token counts come from the final usage chunk. That chunk has an empty choices list, which is why the loop checks chunk.usage before skipping it.
• Run it five times or more, record the region, and report medians along with the sample size.
• For models that list openai_responses, the streaming guide shows Responses streaming with response.output_text.delta and response.completed events. The same split applies: time the first event, then the first text delta.
• Treat a stream that closes before response.completed or a stop finish reason as incomplete, not as a fast success.

Headers and identifiers to log for latency debugging

A slow request is hard to debug if you only kept a duration. The API reference and the troubleshooting guide, both observed 2026-10-03, name the identifiers worth keeping.

Identifier	Where it appears	Use it for
X-Request-ID	Response header, when available	Support and debugging for one request
request_id	Error bodies, dashboard logs, usage records	Request-level support and reconciliation
X-Routing-Time-MS	Response header, when available	Time spent selecting delivery, before generation
X-Task-ID, task_id, poll_url	Async create and status responses	Image, video, music and 3D jobs
X-Billing-Transaction-ID	Header after final billing, when available	Cost reconciliation
Your own job or user ID	Your application	Finding the request behind a user complaint

For each call, log the endpoint, method, model, status code, timestamp and latency. Add the field names you sent (not the prompt), your retry count, and whether a retry created a new task or resumed one. Our harness adds the split timings and token counts from the usage chunk.

Two cautions. Streaming responses may settle after the stream ends, so a billing header can be absent even when usage is recorded later. For billed amounts, use the Management API usage records instead of response token counts alone. Also log the X-TokenLab-Delivery-Policy value if you set one. The reference says it accepts auto, verified or official, that an invalid value returns 400, and that an unavailable option returns 503 with code: delivery_tier_unavailable and retryable: true. Never log Authorization, API keys, signed URLs, full prompts or personal data.

Console request details are kept for 30 days from capture, so copy the IDs you need into your own logs. In our pipeline we line up these IDs against the requests view; the TokenLab request console post covers that view.

Rate limits, retries and LLM latency vs throughput

Per-stream speed and account throughput are different ceilings. The rate limits guide, observed 2026-10-03, lists the standard tier values below. It notes they may change with the active configuration and are enforced per API key.

Tier	Requests per minute
User	1,000
Partner	10,000
VIP	10,000

When you cross the limit, the API returns 429 rate_limit_exceeded with a Retry-After header. The guide's example body carries "retryable": true and "retry_after": 8. The guide says to trust the returned X-RateLimit-Limit and Retry-After values over any copied table. It also says a faster model does not increase your account's request limit.

Here is the arithmetic for batch work. This is our estimate from the medians above, not a load test. Concurrent streams are roughly requests per second times seconds per request. At 1,000 requests per minute (about 16.7 per second):

• With deepseek-v4.1-flash at a 5.204 s median total, about 87 streams run at once.
• With claude-sonnet-5-5 at a 12.384 s median, about 206 streams run at once.

A slower stream means more concurrency to reach the same request rate. Hitting the limit then adds wait time on top of generation time. Token limits are not covered in the pages we used, so check your account and the model page for those.

How to retry without wrecking your numbers:

• Honor Retry-After when present. Otherwise use exponential backoff with random jitter and a hard attempt cap.
• Set the SDK to max_retries=0 and retry in one place. Otherwise two layers retry the same call and your timings double count.
• Never retry validation, authentication, balance or permission errors. Retry transient 5xx responses with a short backoff.
• After the first event arrives, an interrupted stream is incomplete and is not restarted automatically. Repeating the request may produce different output and another charge.
• Log retry count and wait time separately from model time, so a throttled run does not look like a slow model.

This is the guide's bounded retry pattern:

import random
import time
from openai import RateLimitError

def chat_with_rate_limit_retry(client, model, messages, attempts=4):
    for attempt in range(attempts):
        try:
            return client.with_options(max_retries=0).chat.completions.create(
                model=model,
                messages=messages,
            )
        except RateLimitError as exc:
            if attempt == attempts - 1:
                raise
            header = exc.response.headers.get("Retry-After")
            wait = float(header) if header else min(30, 2 ** attempt + random.random())
            time.sleep(wait)

Also cap concurrency, not just average requests per minute, and queue bursts. Track repeated 429 responses per API key so one client cannot crowd out the rest of your application.

FAQ

Is higher tokens per second always a faster user experience?

No. In our 2026-10-03 sample, claude-sonnet-5-5 showed the earliest first visible token (2,981 ms) but the longest total time (12,384 ms). deepseek-v4.1-flash started visible text later (3,925 ms) and finished first (5,204 ms). Which one feels faster depends on whether the user watches the stream or waits for the full answer. Five runs from one client cannot settle that for your workload.

Why did my streamed response come back with almost no text?

You probably spent the budget on reasoning. At max_tokens 400, deepseek-v4.1-flash used 400 reasoning tokens and returned about 0 visible tokens. Check usage.completion_tokens_details.reasoning_tokens in the final usage chunk and the finish_reason. Then raise the cap; at 2000 both reasoning models in our test answered in full.

Does a faster model raise my rate limit?

No. The rate limits guide says model speed, token limits and account rate limits are separate constraints. A faster stream finishes sooner and frees a concurrency slot, but your per-key request limit stays the same. Ask support for a different limit if you need one.

Which IDs should I send support when a stream is slow?

Send the request_id (or X-Request-ID), the endpoint, method, model, timestamp and status code. Add task_id and poll_url for async work and billing_transaction_id when present. Include your split timings and the safe error body, and redact keys, prompts and private media first.

Run the harness above on your own prompt, then compare what you measure with the TokenLab model rankings.