TokenLab Fusion: Evidence from the DRACO Weighted-100 Research Dataset

Research status note: TokenLab Fusion: this is a proof-of-recipe, historical strong-baseline research result, not a production claim. Results were produced using a premium Gemini anchor; a no-premium canary run and a final Fusion-only rerun are pending to confirm reproducibility without premium dependencies. DRACO weighted-100 is a TokenLab proprietary internal evaluation and is not an external or standardized benchmark. Interpret accordingly.

Abstract

This report documents the current evidentiary state of TokenLab Fusion, a multi-model inference-time orchestration system evaluated against a fixed, weighted-100 multi-domain research suite (the DRACO weighted-100 research manifest, covering Finance, Shopping/Product Comparison, Academic, Technology, General Knowledge, UX Design, Law, Medicine, Needle-in-a-Haystack, and Personalized Assistant tasks). The central research question is not whether an ensemble of models can post a higher aggregate score than a single model on a benchmark slice, but whether a disciplined recipe (panel composition, synthesis, rubric-aware validation, evidence sourcing, and cost accounting) can be organized into a reproducible, auditable, and upgradeable system that beats strong single-model baselines across domains, not just on a favorable subset.

The strongest evidence to date comes from the weighted-100 research run, a paired, baseline-enabled evaluation against gpt-5.5 and claude-opus-4-8 on the fixed 100-task DRACO dataset. TokenLab Fusion scored a mean of 86.04, with a paired mean delta of +32.60 over gpt-5.5 (win/loss/tie 95/4/1, cost 0.71x, score-per-dollar 2.26x) and +45.63 over claude-opus-4-8 (win/loss/tie 97/2/1, cost 0.69x, score-per-dollar 3.06x). These numbers are strong, paired, and reproducible against a fixed manifest, but they carry two important qualifications that this report treats as first-class findings, not footnotes: the winning recipe used a historical premium Gemini Pro model (gemini-3.1-pro-preview) as its synthesis/judge/validator anchor, and the system's release-readiness state still requires a Fusion-only final rerun before the result can be called complete. The remainder of this report separates what has been proved in the strong-baseline research run from what remains gated before any production claim, and lays out the specific evidence still needed, most urgently a no-premium-Gemini canary, to close that gap.

1. Research Scope and Objective

TokenLab Fusion is defined here as inference-time orchestration across multiple upstream models, not a trained merged-weight model. A single request is routed through several model roles (panelist, synthesizer, validator, judge, reviewer, tool owner), and the recipe controls how outputs are generated, compared, rewritten, and returned. This is closer to a verifiable multi-model production configuration than to a stateless ensemble vote; it shares structural ancestry with Mixture-of-Agents and LLM-Blender, though the recipe-selection and cost-accounting apparatus described below goes beyond either.

The evaluation target is the weighted-100 research suite: a fixed set of multi-domain, evidence-dense, strictly-rubric-scored research tasks. The research program explicitly rejects weak baseline comparisons. The only accepted baselines are gpt-5.5 and claude-opus-4-8, and both are kept strictly out of the Fusion panel, synthesis, judge, validator, and tool-owner roles; contaminating the panel with the baseline model would invalidate any "cheaper multi-model recipe beats a strong single model" claim.

Model identifiers referenced in this paper--gpt-5.5, claude-opus-4-8, gemini-3.1-pro-preview, deepseek-v4-pro, and glm-5.2--are TokenLab Fusion platform logical model IDs and research baseline IDs captured from the TokenLab model single-source-of-truth (SSOT) snapshot observed on 2026-07-07. These identifiers denote internal routing and evaluation labels rather than an independent, publicly audited leaderboard, and no claim is made that these names correspond one-to-one with any external vendor's public release naming. Identifiers such as gpt-5.5 and claude-opus-4-8 are internal baseline and routing labels drawn from the TokenLab SSOT and should not be interpreted as public vendor model version designations. Consequently, the strong-baseline comparisons in this paper should not be read as a complete production readiness claim: the no-premium-Gemini canary evaluation and the remaining Fusion-only rerun have not yet completed and remain gated pending further validation. Results presented here represent a snapshot-in-time research comparison, not a finalized or fully validated production benchmark outcome.

Two product lines share this research infrastructure but diverge on recipe: a Deep Research line (evidence-grounded, source-sufficiency-checked, validator-heavy) and a Coding Agent line (tool-execution-heavy, latency-sensitive, still pre-Phase-0). This report is primarily about the Deep Research line, since that is where paired evidence against strong baselines currently exists.

2. Why a Fixed 100-Task Manifest

A single-slice benchmark run cannot support recipe decisions. The 100-task manifest exists to serve six distinct engineering functions simultaneously: cross-domain recipe selection (avoiding overfitting to one lucky slice), router-policy training data, model-role assignment evidence, cost/quality/latency curve construction, a failure library for regression testing and future model-upgrade canaries, and a reusable layered regression set that does not need a full rerun every time an upstream model changes.

The manifest used across the reported experiments is weighted-100-v1, a deterministic weighted slice with a fixed task-set hash (b08a09aacbc76c5e5aafd5ca0a6fa614a5061bc478f905007ae5aee4a93fc43a). Its domain weighting is not uniform: Finance (20), Shopping/Product Comparison (16), Academic (12), Technology (10), General Knowledge (9), UX Design (9), Law (6), Medicine (6), Needle-in-a-Haystack (6), Personalized Assistant (6). This distribution matters because Fusion's gains are domain-dependent, and Finance in particular carries the most persistent open risk (Section 6).

DRACO weighted-100 is a TokenLab-internal research dataset used exclusively within this paper's TokenLab Fusion evaluation pipeline. It is not an external, publicly maintained benchmark, and it is not affiliated with, endorsed by, or drawn from any third-party benchmark suite. The weighted-100 manifest, including its item sampling, category weighting scheme, and provenance metadata, is governed by a private TokenLab source map that is not published alongside this paper. Reported scores should therefore be interpreted as internal comparative signals specific to TokenLab's evaluation methodology rather than as scores comparable to public leaderboards or community benchmarks. Readers should not assume DRACO weighted-100 results generalize to, or can be cross-referenced against, any other dataset bearing a similar name or structure.

3. Methodology

The research loop is evidence-first: every strong claim must be traceable to a fixed manifest, task IDs, domain weights, runtime settings, raw result files, cost rows, and baseline rows. Aggregate summary numbers without those artifacts are not treated as evidence.

Four methodological commitments matter most.

Shared evidence over opaque native search

Search, fetch, and reader tooling produce an evidence pack external to the models under test, and that same pack is provided to both the panel and the baseline models wherever feasible. This isolates "can the model reason and synthesize well" from "did the model happen to search well," and it is a strict requirement for weighted-100 research suite proof runs, strong-baseline comparisons, model-upgrade canaries, and final reruns.

Role-separated cost accounting

In a baseline-enabled run, the aggregate cost column mixes Fusion calls and baseline calls; using it directly would overstate or understate the Fusion-only cost claim. The weighted-100 research run analysis explicitly excludes baseline and baseline-score rows to arrive at a Fusion-only official total of $12.837461 across 799 cost rows. This distinction, role-separated cost rows rather than blended totals, is the basis for every cost-adjusted quality claim in this report.

Paired comparison over unpaired means

Fusion and baseline outputs are compared on identical task IDs, and results are reported as mean delta, bootstrap confidence interval, sign-test p-value, win/loss/tie count, cost multiplier, latency multiplier, and failure-rate delta. Unpaired means with different task coverage are not accepted as proof.

Resumable, sharded execution

The weighted-100 research run was executed as ten shards of ten tasks each, then aggregated into a single proof artifact. The final Fusion-only rerun (Section 6) uses the same staged, resumable pattern.

Methods dimension	Approach used	Rationale
Task selection	Fixed weighted 100-task manifest, hashed	Prevents cherry-picked slices, enables reruns
Evidence sourcing	Shared, reproducible evidence pack	Isolates reasoning quality from search luck
Baseline isolation	gpt-5.5, claude-opus-4-8 excluded from panel/synth/judge/validator	Prevents baseline contamination of the Fusion claim
Cost accounting	Role-separated rows; Fusion-only totals exclude baseline calls	Avoids blended-cost cost/dollar distortion
Comparison design	Paired, same task IDs, bootstrap CI, sign test	Controls for domain and sample variance
Execution	Sharded, resumable runners	Enables partial progress tracking and staged reruns
Scoring	Rubric-based validator/judge scoring against domain rubrics	Enables validator ablation and score decomposition

TokenLab Fusion evaluates model outputs through an automated scoring pipeline that aggregates multiple metric signals into a composite score. Scoring rules are applied programmatically without manual review at this stage. Public human-audit or inter-rater agreement data for this method has not yet been published; reported results reflect automated evaluation only, pending independent verification.

Data availability

The manifest hash, cost rows, raw result files, model single-source-of-truth snapshot, and source map underlying this study are retained in TokenLab's internal research archive. These materials are access-restricted and are not published alongside this paper. Consequently, public reproducibility is limited until a redacted sample or public subset of the underlying data is released.

4. Recipe Evolution: More Models Is Not the Lever

An early and important finding is negative: adding more models to the panel does not reliably improve quality, and cannot be assumed to. The recipe evolved through a sequence of controlled ablations rather than by scaling panel size.

Fast core as control group

A low-cost, diverse, stable three-model panel (gemini-3.1-flash-lite, deepseek-v4-flash, grok-4-1-fast-non-reasoning) serves as the cost/latency control condition against which every add-back and every validator variant is measured.

Concrete low-cost model mix

TokenLab Fusion is an inference-time orchestration layer, not a merged-weight model: it coordinates calls to several existing models and combines their outputs at runtime. To make this concrete, the baseline models gpt-5.5 and claude-opus-4-8 are excluded from every Fusion role -- panel, synthesis, judge, validator, reviewer, and tool owner -- so that reported gains cannot be attributed to those baselines doing hidden work inside the pipeline.

The default configuration is a fast, low-cost control panel of three models run in parallel, with an optional fourth panelist evaluated in a quality mode, a historical proof-of-ceiling anchor, and a candidate no-premium canary still awaiting paired evidence.

Role	Model(s)	Status
Fast low-cost control panel	gemini-3.1-flash-lite, deepseek-v4-flash, grok-4-1-fast-non-reasoning	Default low-cost panel under test
Quality add-back candidate	kimi-k2.7-code	Evaluated as fourth panelist; not promoted -- adds latency, regresses in some domains with validator
Historical proof anchor	gemini-3.1-pro-preview	Used for synthesis/judge/validator/revision in the strong-baseline recipe; proves a ceiling, not the intended product-cost path
No-premium canary candidate	deepseek-v4-pro (synthesis/validation/revision), glm-5.2 (independent judge)	Intended cost path; pending paired canary evidence

Orchestration proceeds as a fixed sequence rather than a free-for-all of model calls:

1. Build a shared evidence pack once, so every panelist reasons over identical context.
2. Run the low-cost panelists in parallel against that evidence pack.
3. Normalize panel outputs into a common format for comparison.
4. Run panel analysis to surface agreement, conflict, and missing evidence.
5. Synthesize a single answer, then run a rubric-aware validator and revise once if the validator flags issues.
6. Score/judge the final answer, with a single designated tool owner -- panelists never execute tools directly.

The kimi-k2.7-code result and the pending deepseek-v4-pro/glm-5.2 canary are reported separately from the default panel numbers to avoid conflating an evaluated-but-not-adopted configuration with the intended product-cost recipe.

Kimi add-back: a real but marginal signal

A 20-task pilot adding a coding/diversity model to the fast core produced mean score 34.36 versus fast-core 31.37 (paired delta +2.99, 95% CI -0.04 to 6.10, win/loss/tie 11/6/3, cost ~1.01x, latency ~1.57x). The CI straddling zero and a sign-test p of 0.3323 mean this is not strong enough to justify making the addition a latency-sensitive default; it also regressed on some Academic, Medicine, and Technology tasks. The conclusion drawn is role-specific: this class of model is better treated as a coding/front-end/long-context specialist than a general-purpose always-on panelist.

GLM and DeepSeek Pro add-backs need roles, not blanket inclusion

A GLM add-back produced only about +1.22 at 1.49x cost and 1.85x latency on a small slice, useful evidence for a judge/synth/coding-reviewer role, not sufficient evidence for default panel promotion. DeepSeek V4 Pro's evaluation changed materially after a pricing correction (input $0.435, output $0.87 per 1M tokens), which invalidated an earlier "too expensive" judgment and elevated it to a leading candidate for a synth/validator role in a lower-cost recipe (Section 6). The DeepSeek V4 Pro pricing correction derives from the internal model SSOT observed on 2026-07-07 and constitutes a research cost assumption rather than a citation of an external, vendor-published price.

Validator/rewrite is the largest single lever found

In a 20-task expansion, adding a rubric-aware validator/rewrite stage to the fast core produced mean score 81.70 versus 29.88 for fast core alone, a paired delta of +51.81 (95% CI 42.64 to 61.23), win/loss/tie 20/0/0, at 2.14x cost and 1.89x latency. This is not an incremental improvement; it is a structural shift in what the recipe is capable of. It is also not free: the cost multiplier exceeded the automatic keep threshold of 2.0x in one analysis pass, meaning the same result that is unambiguously worth it for a research-deep tier is a real constraint for a fast/cheap product tier. Cheaper alternatives were tested and rejected as defaults: a lite-review/lite-stack variant lost roughly 28 points relative to full validation and lost every paired comparison; selective validation based on weak-domain heuristics saved only about 11% of cost while losing -8.53 mean score, and a high-risk-only selective variant lost more. Selective validation therefore remains an open routing-research problem, not a solved cost-reduction technique.

5. Strong-Baseline Results: What Was Proved

The weighted-100 strong-baseline research run is the strongest evidence artifact in the program. It holds gpt-5.5 and claude-opus-4-8 as baseline-only comparators, excluded from every Fusion role.

Fusion-only summary across the 100 tasks:

Metric	Value
Mean score	86.04
Official total cost	$12.837461
Official mean cost	$0.128375
Platform total cost	$6.568959
Score per official dollar	670.21
Failed calls	0
Failure rate	0.0%
Mean call latency	216.4s
Cost rows	799
Aggregate fallback tasks	0

The reported value of 670.21 denotes the Fusion-only raw mean score divided by the Fusion-only official mean cost, whereas the 2.26x and 3.06x figures represent paired baseline-relative score-per-dollar multipliers computed against matched comparison configurations.

Paired comparison against the two strong baselines:

Baseline	Baseline mean score	Fusion delta	95% CI	Win/Loss/Tie	Fusion cost multiple	Score/$ ratio	Latency multiple
gpt-5.5	53.43	+32.60	28.13 – 37.28	95/4/1	0.71x	2.26x	1.23x
claude-opus-4-8	40.41	+45.63	40.85 – 50.21	97/2/1	0.69x	3.06x	2.15x

Quality and cost-adjusted quality are unambiguous Fusion wins on this manifest. Failure-rate delta of 0.0 percentage points makes stability a tie rather than a differentiator. Latency is a clear Fusion loss: 1.23x slower than gpt-5.5 and 2.15x slower than claude-opus-4-8. The honest product framing from this data is high-quality, cost-efficient, verifiable deep research, not low-latency interactive chat.

Domain breakdown shows positive deltas across every domain against both baselines, which argues against the improvement being a single-domain artifact. Selected values: against gpt-5.5, Finance +37.13, Shopping +38.20, Needle-in-a-Haystack +62.56, Technology +33.49; against claude-opus-4-8, Shopping +55.96, Needle-in-a-Haystack +69.38, Technology +52.43, UX Design +52.77. Finance is also the domain carrying the clearest unresolved risk: eight high-severity evidence warnings, all Finance, flagged finance:needs_more_sources, citing gaps in metric terms, period terms, and current-primary-period sourcing. These are not call failures; they are source-sufficiency warnings, indicating that Finance-specific evidence routing is not yet fully solved even where the score delta is large.

6. Readiness State: Proved vs. Still Gated

It is important to separate two layers of status. This is a proof-of-recipe result, not a shipped-system result, and the readiness tracking used across the program reflects that distinction explicitly.

Passed: fixed weighted 100-task manifest, panel policy, evidence-router shakedown, panel ablation proof, validator/synth ablation proof, the strong-baseline configuration itself, the weak-source-gate proof, recipe artifacts, and the strong-baseline paired proof described in Section 5.

Not yet passed: the Fusion-only final rerun.

The final rerun is not a second comparison against the strong baselines; its purpose is different and narrower: confirm that the frozen winning recipe reproduces on the same 100-task manifest without baseline cost in the run, that runtime configuration matches the recipe definition, that cost rows are complete, and that there is no baseline contamination in the Fusion-only path. As of the latest staged run, coverage stands at 20 of 100 tasks scored, 20 of 20 runtime-compatible, Fusion-only official cost of $2.619654, baseline cost $0, with the remaining 80 tasks to be executed in further offset stages.

This distinction matters for how the weighted-100 research run numbers should be read: the strong-baseline delta is real and paired, but it was measured with a recipe that has not yet been independently reproduced end to end without baseline cost in the loop, and it used a synthesis/validator/judge anchor that the product-cost path is deliberately trying to retire.

On the Gemini Pro anchor specifically, the frozen recipe that produced these numbers used gemini-3.1-pro-preview, a historical premium Gemini Pro model, as the synthesis/judge/validator anchor. That configuration establishes the ceiling for what recipe structure and rubric-aware validation can achieve, but it is not the configuration the product-cost path intends to ship. Because premium Gemini Pro pricing does not fit the target cost profile, the next required evidence artifact is a no-premium-Gemini canary: a paired, smaller-scale rerun substituting deepseek-v4-pro for synthesis/validation and glm-5.2 for judging, run against the same task IDs before any claim is made that the cost-efficient recipe matches the proved quality ceiling.

7. Protocol and Compatibility: Why One Chat Endpoint Is Not Enough

A recipe that only works against a single OpenAI-compatible chat endpoint will not survive contact with real coding-agent clients. Chat Completions is sufficient for plain conversational use but drops semantics that agent clients depend on: typed output items, function-call/tool-result pairing, streaming item events, stateful versus stateless history, and provider-specific ordering guarantees.

The surfaces that matter, and the semantics each one carries, differ enough that they cannot be collapsed into one shape without loss:

Surface	Key semantics that must be preserved
OpenAI Responses	typed output items, developer/system instruction hierarchy, function calls and function-call outputs, previous_response_id, store, streamed item events, usage accounting
OpenAI Chat	legacy messages/tools; usable as a degraded facade, but compatibility losses must be logged
Anthropic Messages	top-level system field, content blocks, tool_use/tool_result, stateless full-history requests, stop_reason, post-200 stream errors
Codex-style client profile	Responses-first, store, previous_response_id, streamed tool arguments, tool-result replay, strict event ordering
Claude Code-style client profile	Anthropic Messages, immediate tool_result ordering, parallel tool batches, disable_parallel_tool_use, model-discovery allowlists

The architectural answer is a canonical intermediate representation (IR). External protocols map into one internal request/response/stream/tool/usage/error/trace representation, and recipe execution and provider adapters operate against that IR rather than against any single wire protocol. The IR is required to carry, at minimum: conversation and turn identifiers plus provider state pointers (previous_response_id); client protocol and public recipe identifier; the provenance and precedence of system/developer/gateway/recipe instructions; messages, content blocks, images, and derived text; tools, tool results, tool choice, and parallel-tool policy; modalities and generation settings; a record of compatibility losses; usage rows and role-separated cost rows; and a trace covering evidence, search, tool, cost, and model-role activity.

The practical justification is concrete: Anthropic's tool_result must immediately follow its matching tool_use; OpenAI Responses function-call outputs must replay by call_id; a Chat facade cannot fully preserve developer-message priority; Claude Code sends stateless full history on every turn, while Responses-based clients rely on a provider-side state pointer. Without an IR and a compatibility suite that exercises these cases directly, a system can pass a benchmark and still fail inside a real agent client because the tool loop never completes correctly.

Current implementation status should be stated precisely rather than optimistically: the compatibility layer exists as protocol documentation, adapter IR definitions, gateway contract fixtures, a turn planner, a stream writer, tool/evidence fixtures, and an aggregate compatibility test suite. It is an offline compatibility skeleton, sufficient to constrain protocol shapes, surface gaps, and prevent early design drift, but it is not yet a live production gateway. It does not yet include real provider execution at scale, a client-facing streaming product surface, live tool execution, production authentication/tenancy, or a persistent trace backend. Public model discovery is designed to expose only namespaced public recipe identifiers; any non-public or non-namespaced model ID request should fail closed (e.g., model_not_public) rather than allow a client to bypass recipe, cost, or trace boundaries.

8. Tool-Use Control Model

Multi-model tool calling is the most operationally dangerous part of any coding-agent extension of this system. If several models can independently issue tool calls in the same turn, the result can be duplicate file edits, duplicate external API calls, duplicate charges, concurrent writes, conflicting shell commands, or credential exposure. The control model adopted is deliberately restrictive:

• A turn has, by default, exactly one active tool owner.
• Other models may act as reviewer or critic and may propose changes, but may not execute tools.
• Tool-shaped text in a model's output is not, by itself, execution authority.
• Every tool call is normalized into a canonical ToolCall IR before any execution decision is made.
• A scheduler classifies each call as executable, advice_only, blocked, or requires_approval.
• Read-only, network-read, and compute-only calls may run concurrently in a parallel group under safe conditions.
• Write, shell-write, payment, credential, and destructive-class calls are serialized and approval-gated by default.
• Tool results are replayed deterministically back into the target protocol's expected shape.
• Partial JSON is never executed; even well-formed JSON must still pass schema, side-effect, approval, and owner checks.

This is a meaningful departure from typical multi-agent demonstrations, where several agents may all attempt to "solve" a problem concurrently. A production-facing coding recipe requires role discipline over execution safety: a reviewer can flag that a proposed patch will break a test suite, but cannot itself run a destructive shell command. A tool owner can initiate a file edit or shell command, but its output is still verified by the gateway before execution.

This also constrains how the model library's tool-capability tags should be read: a "tool-use" tag means a model can plausibly express a tool call, not that it is cleared to be a production tool owner. Promotion to owner status requires passing a compatibility suite covering streamed tool arguments, tool-ID stability, malformed-JSON repair, tool-result replay, parallel-tool policy, usage accounting, and error shape. A related edge case is inline tool "rescue": some providers emit tool calls as assistant text (XML/JSON blocks, private function tags, object or double-encoded arguments). The adapter layer may detect and repair these dialects, but repair only produces a candidate ToolCall IR; it does not grant execution authority, it is scoped to the current owner's response text only (never to user text, tool results, reviewer notes, or the final fused answer), and a repaired call must still clear owner, schema, side-effect, approval, parallelism, and idempotency gates. Repeated malformed JSON should fail that owner's compatibility check for the turn rather than being silently executed.

9. Vision and Web Search: Fairness vs. Product Experience

The evaluation and product paths intentionally diverge here. Benchmark proof uses shared, reproducible evidence; production requests may use native search or vision, but only when traceable.

Search is handled through three distinct paths: shared_evidence (a Fusion-owned, reproducible external search/fetch/read pack, mandatory for weighted-100 research suite proof runs, strong-baseline comparisons, model-upgrade canaries, and final reruns), native_search (a model or provider's own web/browse/grounding capability), and external_search (traceable external tools such as fetch, browser read, or document extraction). Native search is not disallowed; it may improve product experience, but it cannot support a fair benchmark claim, because if a model's own search behavior, retrieved sources, and citation trail are opaque, the comparison stops being a same-task comparison.

Vision follows the same logic. Native vision and derived OCR/caption text are not equivalent inputs, and a comparison across models with different visibility into an image is not a fair vision comparison. The trace records each model's visibility explicitly as native_image, derived_text, or none. The recommended production pattern is a hybrid: primary models may use native vision while lower-cost reviewers work from derived text; current-facts tasks default to external search/fetch, and native search is only permitted once a route has passed a capability probe and produces a complete trace. Evaluation, however, always defaults to shared evidence and a hashed evidence bundle.

10. Model Library as a Role-Aware Selection Schema

The model catalog underlying TokenLab Fusion has moved from a flat capability table to a layered selection schema with fields for cost tier, Fusion roles, capability tags, selection metrics, routing profile, and recommended evaluation track. This reflects a product reality that a single "how strong is this model" ranking does not capture: a model can be an excellent coding reviewer and a poor low-latency research panelist; a model can be inexpensive but have unstable tool-call IDs; a model can carry a native-vision capability tag without yet having trace evidence sufficient for a fairness-gated evaluation.

Layer	Representative role	Product meaning
Cheap always-on	low-cost panel/reviewer/judge candidates	Default panel composition, promoted only via paired canary evidence
Strong reasoning upgrade	hard-task escalation	Synth/validator/judge role for difficult tasks
Coding/agent specialist	tool-loop, repo-scale, front-end review	Coding agent tool owner and reviewer candidates
Vision/search specialist	image/search-heavy product workflows	Requires trace probe before fairness-gated use
Ultra-long-context holdout	huge repo/evidence-bundle diagnostics	Not a default panel member
Black-box comparator	provider-side multi-agent systems	Reference point only; internal Fusion attribution cannot be derived from it

The current judgment on the retired premium anchor deserves explicit statement: the historical premium Gemini Pro model that anchored the weighted-100 proof has been retired from active product-cost recipe planning, and a lower-priced Gemini flash-tier model was also removed from the default plan due to output pricing that does not fit the target recipe cost profile. This does not revise the weighted-100 research run result, that proof stands as recorded, but it does mean the proof and the intended shipping recipe are not currently the same system, which is precisely why the no-premium-Gemini canary is the next required evidence step, not an optional one.

11. Model-Library Upgrade Gates

A recurring operational concern is whether every model change requires a full 100-task rerun. The answer adopted by the program is layered, risk-proportionate gating rather than either extreme (blind full reruns on every change, or blind same-family substitution without evidence):

1. Contract smoke: confirm route, pricing, streaming, context window, output limits, modality, and tool/search/vision flags.
2. Compatibility suite: exercise OpenAI Chat, OpenAI Responses, Anthropic Messages, tool call, tool-result, parallel-tool, usage, error, and stream handling.
3. Sentinel evaluation: run a small domain-specific set matched to the model's claimed role (e.g., finance source sufficiency, coding tool loop, front-end visual review).
4. Paired canary: compare against the current recipe on identical task IDs for quality, cost, latency, and failure rate.
5. Full 100-task run: reserved for candidates that could change the default recipe or a strong-baseline claim.

Even an apparently minor same-family upgrade requires stepping through this funnel: pricing, tool-call behavior, streaming, and paired quality all need independent confirmation rather than being inferred from a shared model family name. This is also why the 100-task manifest is not run continuously; it is reserved for final confirmation, while day-to-day model changes are governed by the cheaper, faster gates above.

12. Related Work and What Is (and Is Not) Reused

The program draws on several public research and tooling directions without adopting their objectives wholesale. Mixture-of-Agents motivates the idea that weaker model outputs can still usefully condition a stronger synthesizer, but the panel-bloat findings in Section 4 caution against layering without ablation. LLM-Blender contributes the candidate-generation, pairwise-ranking, generative-fusion structure that the panel/judge/synthesis stages resemble, though no dedicated trained ranker currently replaces the rubric-based judge. FrugalGPT and RouteLLM motivate cost-aware cascading and routing, but the routing problem here is multi-role (panel, synthesis, judge, validator, search strategy, tool owner can each route independently), not a single binary model choice. Evaluation-harness tooling such as Inspect AI, OpenAI Evals, Promptfoo, and Ragas inform the engineering pattern of a systematic, CI-friendly evaluation loop, but none of them substitute for the fixed weighted manifest, source-sufficiency checks, role-separated cost rows, or the domain-weighted paired proof structure used here.

13. Limitations

Limitation	Current state	Why it matters
Premium anchor dependency	The weighted-100 proof used a historical premium Gemini Pro model as synth/judge/validator	Proves recipe ceiling, not the intended product-cost recipe
Final rerun incomplete	Fusion-only rerun at 20/100 tasks, 20/20 runtime-compatible	Reproducibility without baseline cost is not yet fully confirmed
No public human-audit or inter-rater agreement data	Human-audit and inter-rater agreement results have not yet been published or made publicly available for this evaluation pipeline	Reliability and consistency of automated scores relative to human judgment remain unverified, limiting confidence in absolute score interpretation
Latency disadvantage	1.23x vs. gpt-5.5, 2.15x vs. claude-opus-4-8	May be acceptable for deep research, likely unacceptable for interactive coding agents
Rubric-aware validator	Validator sees the scoring rubric/checklist	Comparisons must disclose this; not directly comparable to rubric-blind systems
Finance source warnings	8 high-severity finance:needs_more_sources warnings	Source sufficiency in Finance is unresolved despite large score deltas
Native capability inference	Search/vision capability tags are not proof of production-grade trace support	Requires per-route probing before any fairness-gated evaluation use
Compatibility layer maturity	Offline skeleton (IR, fixtures, router contract, planner, stream writer)	Not yet a live gateway with real provider execution, auth/tenancy, or persistent trace storage
Pricing/route volatility	DeepSeek V4 Pro pricing correction already changed a strategic decision	Model-library tiering must be re-verified as pricing and route behavior change

14. FAQ

Does the 100-task result mean TokenLab Fusion beats GPT-5.5 and Claude Opus 4.8 in general?

The paired result holds specifically on the fixed weighted-100 research manifest, using shared evidence and role-separated cost accounting, with gpt-5.5 and claude-opus-4-8 held out of the Fusion panel. It is strong, domain-broad evidence within that scope, not a general claim independent of task type, evidence conditions, or manifest composition.

Why does the Gemini Pro anchor matter if the score delta is already large?

The delta was produced by a recipe using a historical premium Gemini Pro model for synthesis, judging, and validation. That configuration establishes what the recipe structure can achieve, but it is not the cost profile the product path intends to ship. Until a no-premium-Gemini configuration is run as a paired canary against the same task IDs, the achieved quality and the intended shipping cost are proved separately, not together.

Is a lower per-call cost the same as a lower cost per unit of quality?

No, and the two are reported separately for this reason. The 0.71x and 0.69x cost multipliers describe raw cost against the two baselines; the 2.26x and 3.06x score-per-dollar ratios describe cost-adjusted quality. Both are needed, because a system can be cheaper per call and still score worse, or more expensive per call and still be more cost-efficient per unit of quality; this evaluation reports both explicitly rather than collapsing them.

Why does validator ablation matter for this result?

The rubric-aware validator/rewrite stage produced the single largest quality change measured in this program (paired delta +51.81 on a 20-task expansion), at roughly 2.14x cost and 1.89x latency versus the fast-core control. Any comparison between recipes, or between Fusion and a baseline, needs to disclose whether the validator stage is active, because it changes both the cost profile and the fairness framing (the validator has visibility into scoring rubrics).

Is TokenLab Fusion ready to run inside a real coding-agent client today?

No. The protocol and tool-use control architecture (canonical IR, compatibility suite, tool-owner model) exists as an offline compatibility skeleton with documented protocol mappings and fixtures, but it does not yet include a live production gateway, real provider execution at scale, authentication/tenancy, or persistent trace storage. The Deep Research proof and the Coding Agent line are evaluated on different timelines, and the coding line has not yet completed its first small-sample validation phase.

15. Research Closeout

What is proved

On a fixed, hashed, weighted 100-task multi-domain manifest, using shared evidence and role-separated cost accounting, a validator-equipped multi-model recipe achieved a paired mean-score delta of +32.60 over gpt-5.5 and +45.63 over claude-opus-4-8, at 0.71x and 0.69x cost respectively, with zero call failures across 799 cost rows. Rubric-aware validation is confirmed as the largest single quality lever tested, with a paired delta of +51.81 over a fast-core control. Positive deltas hold across every domain in the manifest against both baselines, indicating the gain is not concentrated in one task type.

What remains gated

The Fusion-only final rerun of the frozen winning recipe is incomplete (20 of 100 tasks at last check) and must reach full coverage with a clean readiness audit before the proof can be called reproducible end to end. The recipe that produced the proof depends on a historical premium Gemini Pro anchor that the product-cost path intends to retire; no paired evidence yet exists for the intended no-premium-Gemini substitution. Latency remains an unresolved product tradeoff, particularly for any coding-agent application. Finance source sufficiency carries eight unresolved high-severity warnings despite a large score advantage in that domain. The protocol-compatibility layer is architecturally defined but not yet a running production gateway.

What evidence should be collected next

Complete the Fusion-only final rerun on the remaining 80 tasks and rerun the readiness audit to confirm reproducibility without baseline cost in the loop. Run a paired, same-task-ID canary substituting deepseek-v4-pro for synthesis/validation and glm-5.2 for judging in place of the premium Gemini Pro anchor, starting at small scale before any full-manifest claim. Close the eight outstanding Finance evidence-sufficiency warnings with domain-specific source-gate work before treating the Finance domain result as fully resolved. Extend the compatibility suite from an offline skeleton toward live provider execution, and run a small-sample coding-agent validation phase (on the order of five tasks) to test whether the tool-owner/reviewer architecture produces a measurable panel-over-single-model advantage before investing further in coding-agent infrastructure. Each of these is a specific, falsifiable next experiment, not a general roadmap item.