AI Model Routing Benchmark: Cost Per Task Beats Cost Per Token

Cost per token tells you what a model charges, not what it costs to finish a job. An AI model routing benchmark built around cost per task measures total spend to reach a correct, usable output, including retries, context length, and output verbosity, which is the number that actually shows up on your invoice.

Key Takeaways

Why Cost Per Token Is the Wrong Metric

Token pricing is the unit economists love and product teams misuse. A model priced at $0.15 per million input tokens looks cheaper than one at $0.50, but that comparison only holds if both models produce the same output length and succeed on the first try.

In practice they rarely do. A verbose model that pads answers with explanation text can burn 3x the output tokens of a terser one, even when both are asked the same question. A model with a lower reasoning ceiling on multi-step coding tasks will often require two or three retries to get a compiling result, and each retry re-sends the full context window. Fireworks AI's blog has documented this pattern repeatedly when comparing throughput-optimized versus accuracy-optimized model variants, noting that raw token price and effective task price diverge once retry rates are factored in (fireworks.ai/blog, observed 2026-07-07).

The result: teams that pick models purely on listed token rates frequently end up paying more per completed task than teams that pick based on task success rate, because failed attempts still consume tokens and add latency-driven engineering overhead.

What Cost Per Task Actually Measures

Cost per task is a derived metric:

Cost per task = (input tokens x input rate + output tokens x output rate) x average attempts to success

The "average attempts to success" term is the part token-rate comparisons skip entirely. It requires you to define what success means for your workload; passing a test suite, matching a JSON schema, rendering a usable image, or hitting a quality threshold on a video generation.

This is why a straight rate comparison, like the kind you'd get from a generic pricing comparison, is a starting point but not a final answer. It tells you what each provider charges per token. It does not tell you how many tokens your specific task type needs on average, or how often a given model needs a second pass.

Benchmark Setup: Comparing Models Across Task Types

A useful routing benchmark separates workloads by task category, because failure modes and verbosity differ sharply between them.

Task type	Primary cost driver	Typical failure mode	Where to check current models
Code generation	Retry rate on compile/test failure	Logic errors, incomplete functions	best AI models for coding
Long-form text	Output token verbosity	Padding, off-topic drift	OpenRouter comparison
Image generation	Per-image flat rate vs. resolution tiers	Prompt misinterpretation, needing regeneration	best AI image models
Video generation	Per-second render cost	Artifacting requiring re-render	best AI video models

For text and code tasks, run each candidate model against a fixed set of representative prompts pulled from your own logs, not generic benchmark suites. Track three numbers per model: average input tokens, average output tokens, and success rate on first attempt. Multiply through the formula above using current published rates, which you should verify directly since pricing changes frequently across providers. The TokenLab model directory lists current rates and context windows across providers in one place, which is useful for pulling baseline numbers before running your own tests (tokenlab.sh/en/models, observed 2026-07-07).

For image and video tasks, the math is simpler because pricing is usually per-output-unit rather than per-token, but the same principle applies: a model with a lower per-image price but a higher regeneration rate due to prompt misreadings can cost more per accepted output.

Routing Strategies That Reduce Cost Per Task

Once you have per-task cost numbers for a handful of models, routing decisions become mechanical rather than a matter of picking a favorite provider.

Route by task category, not by default model. Coding tasks and long-form writing tasks have different optimal models even within the same provider's lineup. A router that sends everything to one default model is leaving savings on the table if that model is over-provisioned for simple tasks and under-provisioned for complex ones.

Set a retry budget per task type. If a task type shows a first-attempt success rate below your threshold, cap retries and fall back to a stronger, more expensive model rather than retrying indefinitely on the cheaper one. Three failed attempts on a cheap model can cost more than one successful attempt on a pricier one.

Cap output length where verbosity doesn't add value. For structured outputs like JSON, code diffs, or API responses, constrain max tokens or use system prompts that instruct concise output. This directly reduces the output-token term in the cost-per-task formula without changing the model.

Re-benchmark quarterly. Provider pricing and model versions change often enough that a routing config tuned six months ago may no longer reflect the cheapest path per task. A live model leaderboard view makes it easier to spot when a new model shifts the cost-per-task ranking for a given category.

Checklist: Evaluating a Model Router for Cost Per Task

Use this before committing to a routing configuration:

• Pulled current token/output rates from the provider directly, not from memory or an old comparison post
• Ran at least 20 representative prompts per task category through each candidate model
• Logged average input tokens, average output tokens, and first-attempt success rate per model per task type
• Calculated cost per task using the retry-adjusted formula, not the raw token rate
• Set a retry cap per task category with a defined fallback model
• Constrained max output tokens for structured or short-answer task types
• Scheduled a recurring review (monthly or quarterly) to re-check rankings as models and prices update

Evidence Boundary

Model-routing research is strongest when it combines three evidence types. Public model directories and official pricing pages tell you the unit price and availability at a point in time. Independent performance sources, such as Artificial Analysis, help expose speed, latency, and broad quality signals. Router research, including RouteLLM, explains why a routing policy can reduce cost while preserving acceptable quality, but it still depends on workload-specific labels and evaluation data.

Do not copy a published routing result into your own stack as a universal savings estimate. The real savings come from your prompt mix, success definition, retry budget, and fallback policy. A customer-support classifier, a coding agent, and a video job orchestrator all have different failure costs. Treat the sources cited here as scaffolding for your own benchmark design: they explain what to measure, while your logs and eval set decide which model actually wins.

FAQ

Does a lower per-token price always mean lower cost per task? No. Retry rate and output verbosity can offset a lower token price entirely. A model priced higher per token but with a higher first-attempt success rate and terser output style frequently costs less per completed task. Test both, using your own prompts, before assuming the cheaper rate wins.

How many test prompts do I need before trusting a cost-per-task number? There's no fixed rule, but fewer than 15-20 prompts per task category tends to produce noisy success-rate estimates, especially for coding tasks where pass/fail is binary. Larger sample sizes matter more for task types with variable difficulty.

Should I route everything through one provider for simplicity? Simplicity has a cost too, but so does fragmentation. Compare providers directly using resources like the OpenRouter comparison and pricing comparison pages, then decide whether multi-provider routing is worth the added integration work for your specific task mix. For teams running high volumes across code, text, image, and video generation, task-based routing across providers is generally more common than single-provider defaults.

---

To see current rates, context windows, and task-specific rankings across providers in one place, Get Started with the TokenLab model directory before running your own cost-per-task tests.

Once you've run your benchmark, compare your results against the field. Browse the model leaderboard to check where your chosen models rank on current pricing and performance data.