An AI agent can look convincing in a demo and still be impossible to operate as a product. Once it enters a real workflow, you need to know what the user asked, which route the system chose, what the agent attempted, where it failed, whether a person repaired the result, what the run cost, and whether anything valuable happened afterward.

A dashboard limited to latency, tokens, and model errors cannot answer those questions. Your infrastructure has to preserve the chain from user intent to business outcome. That chain is what lets product, engineering, operations, finance, and risk teams make decisions from the same evidence.

Define the agent’s job before you instrument it

Start with a job contract, not a model endpoint. A job contract turns an open-ended conversation into an observable unit of work. Without one, the agent can produce a plausible response while your analytics incorrectly label the run a success.

Early agent deployments remain anecdotal, but the useful pattern is consistent: narrow, recurring jobs with reviewable outputs are easier to operate than broad mandates. A task such as checking plan codes, flagging mismatches, and stopping before changing a record has a trigger, a finish line, and a safe review point. A command such as build my business has none of them.

Write the contract before the team chooses models or tools. It should specify:

• Trigger: the event or user request that starts the job.
• Eligible input: the records, documents, accounts, or cases the agent may process.
• Completion condition: the evidence required to call the job finished.
• Prohibited actions: changes the agent must never make without approval.
• Review obligation: who must inspect the result and what they must verify.
• Business outcome: the downstream event that makes the job worth running.
• Accountable owner: the person responsible for the workflow, not merely the model.

Do not define completion as the assistant returned a response. For a reconciliation agent, completion might require every eligible record to be checked, every mismatch to include an auditable identifier, no source record to be changed, and a review artifact to be created. The business outcome might be an approved correction, not the creation of that artifact.

Turn the contract into a trace

Represent each run as a stateful trace. At minimum, distinguish initiation, planning, tool calls, tool results, approval requests, approval decisions, checkpoints, completion, failure, abandonment, and cancellation. Every terminal state needs a reason code. Otherwise, completed, stopped safely, timed out, and silently abandoned will collapse into the same misleading number.

A useful run record includes:

• A stable run_id and, when agents delegate work, a parent_run_id.
• The job type, intent category, tenant, and privacy-safe user identifier.
• The workflow, prompt, policy, tool, and routing versions used for the run.
• Timestamped steps with the selected model, provider, tool, and action type.
• Approval requests, the decision made, and the identity or role that made it.
• Token, tool, orchestration, monitoring, and review costs where they can be attributed.
• The terminal state, primary failure code, contributing failure codes, and retry history.
• Human acceptance, correction, escalation, or override.
• An outcome_id that can join the run to the relevant product or business event.

Do not respond to observability gaps by copying every prompt, credential, document, and tool response into a permanent log. Record the minimum evidence needed to reconstruct a decision. Redact secrets and sensitive fields before ingestion, restrict access to detailed traces, and set retention by data class. A trace that creates a new privacy or security problem is not production-ready infrastructure.

Build a control plane around the execution path

The execution path is where models reason and tools act. The control plane decides what they are allowed to do, how they are routed, how failures are contained, what is metered, and what evidence survives. Keep these concerns separate enough that you can change a model without rebuilding permissions, analytics, billing, and incident controls.

Your control plane should own the following responsibilities:

• Identity and permissions: determine which user, tenant, agent, and delegated sub-agent may invoke each tool or access each data class.
• Routing policy: select a model and provider based on capability, risk, latency, availability, and cost.
• Execution state: preserve checkpoints, tool results, pending approvals, and safe resume points.
• Failure control: apply timeouts, retry budgets, circuit breakers, fallbacks, cancellation, and a kill switch.
• Economic metering: attribute model, tool, infrastructure, monitoring, and review costs to the run and customer.
• Evidence: emit versioned traces, evaluations, audit events, and business-outcome links.

Treat model routing as product policy

A gateway can expose more than 400 models from more than 80 providers, but access to many models is not the same as resilience. A fallback model reached through the same unavailable gateway does not protect the workflow. Nor does a second model help when the real failure is an expired login, a malformed tool schema, or a policy block.

Log each routing decision with the requested capability, candidate routes, selected model and provider, policy version, selection reason, and any fallback attempted. This allows you to answer whether a cheaper route preserved the outcome, whether a faster route increased correction work, and whether a provider incident affected a particular customer segment.

Use automatic fallback for failures you can classify safely, such as transport or capacity errors. Be more cautious with semantic failures. Sending the same ambiguous or unsafe instruction to a different model can repeat the error and add cost. Route low-confidence or high-consequence cases to an evaluator, a narrower workflow, or a human approval step.

Keep route policy in your own versioned configuration, preserve an internal trace identifier across providers, and retain exportable usage logs. Routing and metering naturally converge because the same layer sees model choice, token use, and price. That makes the gateway strategically useful, but it also makes dependency management part of the product architecture.

Instrument the environment, not just the model

Computer-use agents add another failure domain: the environment in which the agent clicks, types, authenticates, and waits. Early user reports have included browser crashes, profile resets, heavy allowance consumption, and constrained apparent concurrency. Those reports are anecdotal, but the engineering implication is sound: a model-success event cannot prove that the browser session remained healthy or that a side effect occurred exactly once.

Capture environment health separately from model health. Record the environment and session identifiers, authentication-state checks, navigation failures, resource or concurrency limits, the last confirmed page state, and the last confirmed side effect. Before retrying an action that sends a message, changes a record, places an order, or triggers a charge, re-read the target system or check an operation key. A blind retry can perform the action twice.

Connect agent performance to a business outcome

The central analytics job is to observe what users ask, identify where the agent fails, and attach those failures to product and business consequences. This requires an intent taxonomy, a shared run identifier, and an outcome event. Without those elements, you can monitor infrastructure but cannot manage the product.

Group requests by the job the user is trying to complete, not merely by the words in the prompt. Preserve an unclassified category and review it regularly. New intents often appear there before they become visible in aggregate metrics. If an intent classifier supplies the category, store its version and confidence so a classifier change does not masquerade as a change in demand.

Measurement layer	Useful signals	Decision it supports
User demand	Eligible requests, attempted runs, intent mix, repeat use, and abandonment before execution	Which jobs deserve capacity, redesign, or removal
Execution health	Terminal-state distribution, tool errors, retries, approval waits, and end-to-end latency	Where infrastructure or workflow reliability is breaking
Output quality	Task-specific evaluation results, human acceptance, material correction, and correction type	Whether the result is usable, not merely complete
Human effort	Review touches, review duration, escalations, overrides, and undo actions	Whether automation is removing work or relocating it
Economics	Inference, tool, orchestration, monitoring, and human-review cost per eligible run, accepted result, and outcome	Whether the workflow can scale economically
Business effect	A workflow-specific outcome and time to that outcome	Whether agent use creates value for the customer and the business

Use denominators that expose failure

A completion rate based only on started runs can hide demand that never reached execution. An acceptance rate based only on completed runs can hide timeouts and abandoned runs. A cost calculation based only on successful runs can hide the expense of retries and human repair.

Define an eligible-run denominator from the job contract, then calculate:

• Completion yield: completed runs divided by eligible runs.
• Accepted-output yield: outputs accepted without material correction divided by eligible runs.
• Outcome yield: qualifying business outcomes divided by eligible runs.
• Cost per accepted output: all attributable run and review costs divided by accepted outputs.
• Cost per outcome: all attributable costs divided by qualifying business outcomes.

The distinction between completion and acceptance is essential. A tool can return a success status while writing the wrong value. A model can follow the requested format while giving an unusable answer. A reviewer can accept the output while the customer still fails to complete the workflow. Each event answers a different question.

Segment every metric by job type, workflow version, route, provider, tool version, permission mode, customer segment, and review policy. Overall averages can improve simply because the mix shifted toward easier jobs. The segmented view tells you whether the system itself improved.

Separate association from product impact

An agent run may precede an outcome without causing it. Define the qualifying outcome and its attribution window before rollout. When the workflow and traffic support it, use a randomized holdout, staged release, or other credible comparison. If you only have observational data, label the result as an association and compare it with an explicit baseline rather than claiming causal lift.

Operational and product data should join through run_id and outcome_id, but they do not need to live in the same storage system. The important requirement is reproducibility: a product analyst should be able to move from an outcome change to the affected runs, and an engineer should be able to move from a failed trace to its customer and business consequence without exposing unnecessary sensitive content.

Turn failures into a product operating queue

Logs become useful only when a team can classify, prioritize, reproduce, and own the failures inside them. Give each failed or repaired run one primary cause and, where needed, contributing causes. The primary code prevents double counting; the contributing codes preserve the chain of events.

A practical taxonomy separates:

• Demand mismatch: the user asked for a job the agent was not designed to perform.
• Task-definition failure: the finish line, constraint, or required input was ambiguous.
• Planning or output failure: the agent selected a bad approach or produced an unusable result.
• Tool-contract failure: a schema, parameter, response, permission, or integration behaved unexpectedly.
• Environment failure: a browser, session, login, resource limit, or external interface broke the run.
• Policy or approval failure: a safeguard blocked the action, an approval arrived too late, or the workflow attempted an unauthorized step.
• Orchestration failure: state, delegation, retry, cancellation, or checkpoint handling failed.
• Outcome gap: the run completed and may even have passed an evaluation, but the intended product or business result did not occur.

Assign an owner and a response path to every category. A tool-contract problem usually belongs with the integration owner. An unsupported job belongs with product. A recurring environment failure belongs with the platform or reliability owner. An outcome gap requires the product owner to revisit the job design, user experience, or value proposition.

Prioritize with more than failure count. Include affected eligible runs, lost outcomes, wasted cost, human repair effort, customer segment, and consequence severity. A rare action that can duplicate a charge or alter a sensitive record may deserve attention before a frequent formatting defect. Frequency is not a substitute for risk.

Use one release loop for evals and outcomes

My default for a production agent is a weekly failure review until the failure mix and operating load stabilize. The meeting should produce changes to the product or evaluation system, not a tour of dashboards.

1. Rank failure groups by lost outcomes, attributable cost, repair effort, and severity.
2. Inspect representative traces alongside successful traces from the same intent and segment.
3. Identify the failing layer before proposing a model change.
4. Convert reproducible failures into versioned evaluation cases with an explicit pass condition.
5. Change one controllable layer: task boundary, prompt, model route, tool contract, permission, interface, review step, or orchestration policy.
6. Release behind a feature flag or to a bounded cohort.
7. Compare the target evaluation, production failure code, outcome yield, cost, latency, and safety guardrails.
8. Promote, revise, or roll back using criteria agreed before the release.

Offline evaluations and production analytics answer different questions. An evaluation can show that a known failure is reproducible and that a proposed change fixes it under controlled conditions. Production analytics show whether real users receive acceptable results and reach the intended outcome. You need both. A model can improve an evaluation while raising latency, review work, or cost enough to make the product worse.

Do not assume every quality problem requires a larger model. If the trace shows an expired session, missing permission, ambiguous tool response, or undefined finish line, a model upgrade adds expense without repairing the mechanism. Fix the layer that failed.

Key takeaways

• Define the trigger, completion condition, prohibited actions, review point, and business outcome before selecting infrastructure.
• Trace each run across models, tools, approvals, retries, human corrections, and downstream outcomes.
• Keep routing, permissions, state, metering, evidence, and failure controls in a control plane you can version independently.
• Measure completion, acceptance, economics, and outcomes against eligible demand so failed and abandoned runs remain visible.
• Pair eval-driven development with product analytics: evals verify known behavior, while outcome data determines whether the product creates value.
• Prioritize failures by business consequence and repair cost as well as frequency, then assign each category to an accountable owner.

Pick one production workflow with a clear finish line and require the team to explain a single bad run from request to outcome. If the trace cannot show the route, action, failure mechanism, human intervention, full cost, and downstream result, close those gaps before expanding autonomy. More agents will otherwise multiply uncertainty faster than value.

References

• The Creator’s AI — Stripe Buys AI Toll Booth, Grok Ships 24/7 Coworkers
• Amplitude — Connecting Agent Performance to Product Outcomes

---