You have a shortlist of AI models, a stack of vendor price cards, and a deadline to make a platform decision. The apparent shortcut is to compare token prices and public benchmark scores. That shortcut can select the model with the lowest unit price and the highest total cost.

You need a benchmark that answers a product question: Which model-and-system combination produces an acceptable result for your workflow, at a predictable total cost, with failures you can contain? The right unit of comparison is not a token or a leaderboard point. It is an accepted outcome.

Key takeaways for model selection

• Optimize for cost per accepted result. Token price does not include retries, failed runs, tool use, validation, human review, or remediation.
• Benchmark the deployed system, not the model in isolation. Prompts, tools, memory, retry rules, context management, and stopping conditions can change the result.
• Make severe failures eligibility gates. A high average score should not compensate for a data-corruption, security, permission, or compliance failure.
• Report first-pass acceptance, retry-adjusted acceptance, failed-attempt spend, and tail cost separately. An average hides the behavior that creates operational surprises.
• Treat model selection as a routing policy. Your best default model, escalation model, and high-risk model may be different.

Measure cost per accepted result, not cost per token

Model price cards matter, but they describe the rate charged for computation rather than the cost of completing your job. A cheaper model can consume more context, generate more output, call more tools, retry more often, or fail after it has already spent most of the task budget.

One agent puzzle evaluation makes the distinction unusually clear. The listed prices favored Grok 4.5, yet its observed task cost was the highest of the four models in that evaluation:

Model	Listed price per million tokens	First-attempt success in the puzzle test	Observed total test cost
Claude Opus 5	$5 input / $25 output	100%	$17.15
GPT-5.6 Sol	$5 input / $30 output	100%	$11.91
Kimi K3	$3 normal input, $0.30 cached input / $15 output	96%	$12.56
Grok 4.5	$2 input / $6 output	75%	$50.66

This is not a universal ranking. It is evidence about one task, one harness, and one set of run conditions. It does establish why list price cannot stand in for product economics. GPT-5.6 Sol and Claude Opus 5 both completed every puzzle on the first attempt. Kimi K3 reached 100% within three attempts. Grok 4.5 reached 88% within three attempts and accumulated the largest bill despite having the lowest listed input and output prices.

Use a denominator that procurement, product, and engineering can interpret in the same way:

Cost per accepted result = total cost of all attempts divided by the number of results that pass the acceptance gate.

If a run produces no accepted result, record it as a failed run. Do not make it disappear from the denominator or hide its spend inside a blended average.

Your total-cost ledger should include:

• Uncached and cached input tokens across every attempt.
• Output tokens, including reasoning, plans, corrections, and verbose dead ends where the provider bills for them.
• Tool, search, retrieval, database, browser, code-execution, and evaluator calls.
• Retries triggered by the model, orchestration layer, validator, or user.
• Spend on tasks that time out, loop, crash, or finish without an acceptable answer.
• Automated validation and guardrail costs.
• Human review, escalation, and correction time.
• Expected remediation when a defect escapes into production.

Loops deserve their own line item and alert. In a wider puzzle experiment, one Gemini 3.6 Flash level generated 1,870 tool calls. Another run consumed more than 500,000 context tokens, and unsuccessful spending exceeded $260 before the experiment stopped. A low-cost model without a stopping policy can spend quickly while making no progress.

Put maximum attempts, tool-call budgets, context budgets, and no-progress detection into the harness before you begin. Log why each run stopped. A timeout, an explicit budget stop, a validator rejection, and a successful completion are different product outcomes even when they produce similar token bills.

Do not report only the mean task cost. Show the median, a high-percentile cost, the most expensive failed run, and total spend on rejected outputs. The median tells you what routine traffic may cost. The tail shows whether the system can create a budget incident.

Benchmark the workflow your product actually runs

A model does not produce an outcome by itself. It operates inside a harness that supplies instructions, context, memory, tools, permissions, retries, validators, and stopping rules. If those elements change between candidates, you are comparing systems. If you freeze all of them, you may be testing some models in an environment that does not suit them.

The practical answer is to write a benchmark contract before running the candidates. It should define:

• Decision: Name the product or platform choice the benchmark will inform. A test without a decision becomes a leaderboard exercise.
• Workload: Use anonymized cases that preserve the length, ambiguity, tool requirements, and failure modes of real traffic. Separate materially different jobs rather than averaging them into one synthetic task.
• Environment: Pin the model version, system prompt, tool definitions, retrieval configuration, memory behavior, context policy, permissions, and orchestration code.
• Budget: Set limits for attempts, tokens, elapsed time, tool calls, and total run cost. Define what happens when a limit is reached.
• Acceptance: Specify observable evidence that the job is complete. Include machine checks where correctness can be verified and blinded human review where judgment is necessary.
• Risk: Identify failures that make a candidate ineligible, even if its aggregate quality is high.
• Economics: Decide whether your denominator is an accepted answer, resolved case, completed workflow, approved code change, converted lead, retained customer, or another product outcome.

Use a controlled track and a deployable track

In the controlled track, give candidates the same core prompt, tools, task set, retry policy, and limits. This tells you how each model behaves inside a common architecture. It is useful when you need portability or expect to route traffic among providers.

In the deployable track, allow model-specific prompting, tool descriptions, context strategies, and orchestration changes that you would realistically maintain in production. This tests the best product configuration available from each candidate, not merely its compatibility with the first harness you built.

Keep the results separate. If the tuned configuration wins, include the engineering and operational overhead of maintaining that configuration in the decision. A model that needs extensive custom handling may still be the right choice, but the customization is part of its cost.

Make acceptance severity-aware

Aggregate scores can hide the one failure that matters. In a Go database-building task, Claude Opus 5 cost $31.71 and scored 98 out of 100, while a Kimi K3 and Grok 4.5 team cost $1.27 and scored 93. Both implementations passed 64 of 65 checks, survived the crash tests, and shipped the same critical bug.

A five-point score difference looks meaningful until both systems violate the same critical requirement. The benchmark did not merely compare quality and price; it exposed a gap in the acceptance system. If the check that failed represents a production invariant, neither candidate should be considered ready.

Split acceptance into distinct layers:

• Hard gates: Correctness, authorization, privacy, data integrity, regulatory, security, and irreversible-action requirements that cannot be traded for a higher average score.
• Graded quality: Completeness, clarity, maintainability, style, helpfulness, and other attributes where better and worse answers can be ranked.
• Recovery: Whether the system recognizes failure, changes strategy, requests clarification, or escalates instead of repeating the same action.
• Containment: Whether a failure stays inside its allowed scope and leaves enough evidence for diagnosis.

Agent evaluations can modify files, send messages, change records, or invoke paid services. Run destructive scenarios in a sandbox against disposable data, use allow-listed tools, and require explicit approval for irreversible actions. A realistic benchmark does not require risking production data to prove that an agent can act.

Turn benchmark results into an executive decision

A product leader needs more than a winner. You need to know where the result holds, what could reverse it, and how the choice will behave at production volume. A useful scorecard keeps quality, reliability, cost, speed, risk, and operability visible instead of compressing them into one composite number.

Decision dimension	What to measure	What it tells you
Outcome quality	Accepted results, hard-gate failures, and graded quality by task segment	Whether the system completes the intended job
Reliability	First-pass acceptance, retry-adjusted acceptance, repeated-run variance, and failure clusters	Whether performance is dependable rather than occasionally impressive
Economics	Cost per accepted result, failed-attempt spend, tool cost, human-review cost, and tail cost	Whether the workflow remains economical under real operating conditions
Speed	End-to-end latency for accepted results, including retries and validation	Whether users receive a usable outcome within the product’s experience constraint
Risk	Failure severity, escaped defects, unsafe actions, and containment behavior	Whether the downside is tolerable for the workflow
Operability	Telemetry, version control, fallback behavior, portability, and ease of debugging	Whether the system can be run and changed without creating hidden dependence

Keep the raw dimensions beside any weighted score. Weighting is a business decision, not a property of the model. A customer-facing support answer, an internal document summary, and an autonomous database migration should not carry the same failure penalty.

Choose a routing policy instead of a universal winner

The puzzle and database results point in different directions because the tasks reward different behavior. GPT-5.6 Sol delivered the lowest cost among the perfect first-attempt puzzle performers. The much cheaper Kimi K3 and Grok 4.5 team came close to Claude Opus 5 on the database score, yet both database implementations missed the same critical requirement. No single rank captures those trade-offs.

Segment your traffic by variables that affect the decision: task type, complexity, context size, tool dependence, latency sensitivity, and failure severity. Then evaluate a routing policy with a default model, a fallback path, and an escalation condition.

A lower-priced model followed by a premium fallback is a hypothesis, not an automatic saving. It works only when the first model accepts enough tasks cheaply and detects its own failures reliably. If the default model produces plausible but incorrect outputs, the fallback may never run. If it retries aggressively before escalating, the two-stage route may cost more than sending the task to the stronger model first.

Benchmark the route end to end. Charge every unsuccessful default attempt to the accepted result eventually produced by the fallback. Measure cases that pass incorrectly as failures, not savings. The routing policy should beat the best single-model alternative on the dimensions your product actually values.

Model diversity also does not guarantee independent failure. The shared database bug shows why you still need external validators, invariant checks, and adversarial cases. Two models can agree and still be wrong.

Run the benchmark like a product decision

The strongest benchmarking process is reproducible enough for engineering and legible enough for finance, security, and procurement. Use this sequence:

1. State the decision. Name the workflow, expected operating environment, candidates, constraints, and decision owner. Record whether you are selecting a default, a specialist, a fallback, or a routing portfolio.
2. Build the evaluation set. Sample representative production-shaped cases, preserve difficult edge cases, and keep part of the acceptance logic hidden from the model. Tag cases by the segments you expect to route differently.
3. Write the rubric before running models. Define accepted, rejected, and escalated outcomes. Separate hard gates from graded preferences and specify how disagreements between automated and human evaluators will be handled.
4. Freeze the controlled harness. Pin model identifiers, prompts, tool schemas, permissions, retry rules, timeouts, context policy, and evaluator versions. Store configuration with every result.
5. Prepare deployable variants. Tune only the elements you would be willing to own in production, and record the implementation effort that each variant requires.
6. Run identical cases repeatedly. Randomize candidate order where order could affect review, blind human evaluators to model identity, and retain complete traces. Record partial progress as diagnostic evidence, not as an accepted outcome.
7. Inspect failure modes before averages. Group failures into misunderstanding, tool misuse, bad planning, looping, premature completion, validator evasion, unsafe action, and infrastructure error. Product fixes differ by failure type.
8. Calculate outcome economics. Include all attempts, tools, evaluators, human work, and rejected-run spend. Report first-pass and retry-adjusted results separately, along with tail cost and latency.
9. Test the proposed operating policy. Evaluate default, fallback, escalation, budget-stop, and human-review behavior as one system. Confirm that failures are contained and observable.
10. Write the decision record. Document the chosen configuration, rejected alternatives, decisive trade-offs, known blind spots, budget limits, fallback behavior, and conditions that trigger a new benchmark.

Rebenchmark when a model version, price, prompt, tool contract, evaluator, context policy, workload mix, or acceptance requirement changes materially. Otherwise, teams end up defending a current platform choice with evidence from a system that no longer exists.

If your current procurement sheet ends at price per million tokens, add cost per accepted result, failed-attempt spend, hard-gate failures, and tail cost. Start with the workflow whose volume or failure consequence makes the decision matter. Benchmark the complete system, choose a default and fallback, and keep the traces needed to change your mind when the evidence changes.

References

• Think Tank AI — GPT-5.6 Sol vs Claude Opus 5 vs Kimi K3 vs Grok 4.5: Who Is The Real Winner

---