LLM Latency vs Throughput: How API Buyers Should Measure Speed

When evaluating model APIs, understanding the trade-offs of LLM latency vs throughput is essential for optimizing both user experience and infrastructure costs. Latency measures the time elapsed for a model to respond to a query, while throughput measures the volume of tokens processed or generated by the system over a specific time window. For developers and AI product builders, optimizing for one metric often requires making trade-offs with the other.

Selecting the wrong speed metric can lead to sluggish user interfaces or unnecessarily high API bills. This analysis provides a framework for measuring these metrics, selecting the right APIs for your specific workloads, and implementing optimization strategies.

Key Takeaways

---

Defining the Core Metrics: Latency vs. Throughput

To make informed decisions when purchasing or routing APIs, developers must break down "speed" into distinct, measurable components.

Timeline of an LLM API Request:
[User Sends Request] 
       │
       ▼ (Network Transit + Prompt Processing)
[Time to First Token (TTFT)] <--- Critical for interactive UX
       │
       ▼ (Autoregressive Generation: Tokens Per Second)
[Inter-Token Latency (ITL)]  <--- Dictates reading comfort
       │
       ▼ (Generation Complete)
[Total Latency]              <--- Critical for non-streaming blocking calls

1. Time to First Token (TTFT)

TTFT is the duration between sending an API request and receiving the very first token of the response. This metric includes network round-trip time, prompt serialization, and the time required for the model to process the input tokens (prefill phase). For interactive applications, TTFT is the most important metric because it determines how quickly a user sees a response begin to stream.

2. Inter-Token Latency (ITL)

ITL is the average time elapsed between generating consecutive tokens during the streaming phase. If the ITL is too high, the text streams slower than a human can read, resulting in a frustrating user experience. A stable, low ITL ensures smooth text rendering.

3. Tokens Per Second (TPS)

TPS represents the generation throughput of the model. It is calculated as the total number of output tokens divided by the total generation time (excluding the prefill phase). When evaluating throughput, developers must distinguish between:

• Single-user TPS: The generation speed of a single active stream.
• System throughput: The total number of tokens the API provider can process concurrently across all active users.

4. Total Latency

Total latency is the complete duration of the API request from start to finish. For non-streaming requests, such as structured JSON extraction or background classification, total latency is the primary metric to monitor.

---

The Architectural Trade-offs: Why Speed Varies

The trade-off between LLM latency vs throughput is rooted in the physics of transformer architectures and hardware memory bandwidth. During the prefill phase (which determines TTFT), the computation is highly parallelizable because the entire input prompt is processed at once. This phase is typically compute-bound.

During the generation phase (which determines TPS), the model generates tokens one by one. Each new token requires loading all model weights from High Bandwidth Memory (HBM) to the GPU SRAM. This autoregressive process is memory-bandwidth bound.

Because of these constraints, developers must align their model choices with their primary performance requirements:

• Flagship Models: Models such as Claude Fable 5, Claude Opus 4.8, and GPT-5.5 prioritize reasoning depth over raw speed. They feature massive parameter counts, resulting in higher TTFT and lower TPS.
• Fast, Low-Cost Models: Models like DeepSeek V4 Flash, Gemini 3.5 Flash, and Laguna XS 2.1 are optimized for speed. They use smaller parameter counts, speculative decoding, or distilled architectures to deliver exceptionally low TTFT and high TPS.

Developers can consult the TokenLab LLM API Leaderboard for Developers to compare real-time speed metrics across these model tiers.

---

Decision Framework: When to Prioritize Latency vs. Throughput

The priority between latency and throughput depends entirely on the application use case.

Use Case	Primary Metric	Secondary Metric	Recommended Model Class
Interactive Chatbots	Time to First Token (TTFT)	Inter-Token Latency (ITL)	Fast Frontier (e.g., Gemini 3.5 Flash)
Coding Assistants	TTFT & Single-stream TPS	Total Latency	Specialized Coding (e.g., Claude Sonnet 5, Kimi K2.7 Code)
Bulk Data Extraction	System Throughput	Cost per Task	Low-Cost Open-Weight (e.g., DeepSeek V4 Flash, GLM-5.2)
Autonomous Agents	Total Latency (Non-streaming)	TTFT	High-Reasoning Open-Weight (e.g., DeepSeek V4 Pro)
Image/Video Generation	Total Latency	Cost per Image	Specialized Media APIs (e.g., Nano Banana 2, Seedance)

Interactive Applications (Latency-First)

For conversational UIs, customer support bots, and live search assistants, user retention drops if the system feels unresponsive. Developers should prioritize minimizing TTFT. Even if the total generation takes several seconds, a TTFT under 300 milliseconds keeps users engaged.

Batch Processing and Pipelines (Throughput-First)

For offline tasks like processing thousands of PDF invoices, generating daily reports, or running batch evaluations, TTFT is irrelevant. The goal is to maximize the total volume of tokens processed per minute at the lowest possible cost. Developers should focus on system throughput and cost-efficiency. For an in-depth analysis of cost optimization during batch processing, see the TokenLab AI Model Routing Benchmark Cost Per Task.

---

Measuring API Performance: A Practical Code Example

To accurately measure TTFT, ITL, and TPS, developers must use streaming APIs and record timestamps at specific points in the request lifecycle. Below is a runnable Python script using the OpenAI-compatible client to measure these metrics for a given model.

import time
import os
from openai import OpenAI

# Initialize client (configured for OpenRouter or any compatible provider)
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ.get("OPENROUTER_API_KEY", "your_api_key_here")
)

def measure_api_performance(model_name: str, prompt: str):
    print(f"Evaluating speed metrics for: {model_name}")
    
    start_time = time.time()
    response = client.chat.completions.create(
        model=model_name,
        messages=[{"role": "user", "content": prompt}],
        stream=True
    )
    
    ttft = None
    token_timestamps = []
    total_tokens = 0
    
    for chunk in response:
        chunk_time = time.time()
        # Check if text content is present in the chunk
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            # Estimate token count (1 token ≈ 4 characters for rough measurement)
            estimated_tokens = max(1, len(content) // 4)
            total_tokens += estimated_tokens
            
            if ttft is None:
                ttft = chunk_time - start_time
                print(f"-> Time to First Token (TTFT): {ttft:.3f} seconds")
            
            token_timestamps.append(chunk_time)
            
    end_time = time.time()
    total_duration = end_time - start_time
    generation_time = total_duration - ttft if ttft else total_duration
    
    # Calculate Inter-Token Latency (ITL) and Tokens Per Second (TPS)
    if len(token_timestamps) > 1:
        intervals = [token_timestamps[i] - token_timestamps[i-1] for i in range(1, len(token_timestamps))]
        avg_itl = sum(intervals) / len(intervals)
        tps = total_tokens / generation_time if generation_time > 0 else 0
    else:
        avg_itl = 0
        tps = 0
        
    print(f"-> Total Latency: {total_duration:.3f} seconds")
    print(f"-> Average Inter-Token Latency (ITL): {avg_itl:.3f} seconds")
    print(f"-> Estimated Throughput (TPS): {tps:.2f} tokens/sec")
    print("-" * 50)

# Example usage with a fast, low-cost model
if __name__ == "__main__":
    test_prompt = "Write a 200-word essay on the history of computing."
    # Using a current low-cost routing model example
    measure_api_performance("google/gemini-3.5-flash", test_prompt)

---

Optimization Strategies for API Buyers

If your measurements reveal that your chosen API is too slow or too expensive, several optimization strategies can improve performance.

1. Prompt Optimization and Prefill Reduction

Because the prefill phase scales with the size of the input prompt, reducing prompt length directly decreases TTFT.

• Remove redundant instructions.
• Use system prompt caching if supported by the provider. This allows the API host to cache the compiled state of a long system prompt, bypassing the prefill computation on subsequent requests.

2. Dynamic Provider Routing

According to the OpenRouter provider selection documentation, the performance of a model can vary significantly depending on which underlying host (provider) serves the request. Some providers optimize for low latency, while others offer lower costs at the expense of speed.

By utilizing routing layers, developers can:

• Query multiple providers to find the lowest current latency.
• Set fallback paths so that if a primary provider experiences a latency spike, requests automatically route to a faster alternative.
• Filter providers based on specific performance thresholds.

3. Model Tiering

Do not use flagship models like Claude Fable 5 or GPT-5.5 for tasks that can be handled by smaller models. Implement a routing router that sends simple queries (e.g., classification, formatting) to DeepSeek V4 Flash or GLM-5.2, reserving expensive models only for complex reasoning steps.

---

Limitations of Speed Benchmarks

When evaluating speed metrics, developers should keep the following limitations in mind:

• Network Variance: API latency is highly dependent on the physical distance between your application servers and the API provider's hosting region. Always run benchmarks from servers located in the same region as your production deployment.
• Provider Congestion: Throughput and latency fluctuate throughout the day based on global traffic patterns. A single benchmark run does not represent consistent production performance.
• Token Estimation Discrepancies: Different models use different tokenizers. A model with a higher TPS might not actually be faster if its tokenizer splits words into smaller, more numerous tokens than a competing model.

---

Frequently Asked Questions

Does a higher throughput (TPS) always mean a faster user experience?

No. If an API has high throughput but a poor TTFT, the user will experience a long, unresponsive pause before the text suddenly appears on the screen. For interactive applications, a low TTFT is more critical than a high TPS.

How does prompt caching affect latency?

Prompt caching significantly reduces TTFT for long prompts. By caching the processed tokens of system instructions or context documents, the provider skips the compute-heavy prefill phase on subsequent requests, leading to faster response times.

Should I choose open-weight or closed-source models for the best speed?

It depends on the hosting infrastructure. Open-weight models like Qwen3.7 Plus, GLM-5.2, or DeepSeek V4 Pro can be deployed on dedicated private hardware, allowing you to guarantee throughput. However, managed closed-source APIs often use massive, optimized infrastructure that can be difficult to replicate cost-effectively on private instances. You can compare current performance rankings on the TokenLab Model Rankings.

---

Next Steps

To optimize your application's speed and cost-efficiency, start by measuring your current production workloads using the streaming metrics outlined above.

Ready to evaluate and compare the latest models for your production pipeline? Get Started with TokenLab's comprehensive model rankings to find the optimal balance of latency, throughput, and cost for your application.