Your agent passed the pilot. It completes the happy path, the demo looks convincing, and the pressure is now to connect more tools, serve more users, and remove more approvals.

This is where reliability usually changes shape. You are no longer asking whether the model can complete a task. You are deciding how much authority the system can exercise when its inputs are incomplete, its environment has changed, or one plausible action creates an expensive downstream mistake. Scaling safely means expanding that authority deliberately, with controls that can detect, contain, and explain failure.

Scale decision authority, not just request volume

A chat assistant can produce a wrong answer. An agent can turn a wrong answer into a refund, a deleted record, a customer message, a configuration change, or a chain of work assigned to other systems. The important unit of scale is therefore not the number of prompts or even the number of agents. It is the number and consequence of decisions they can commit.

Before you increase traffic, place every use case on an autonomy ladder. This is a design scaffold, not an industry standard, but it forces the right product conversation:

1. Draft: The agent prepares content, code, or a recommendation. A person decides whether anything happens.
2. Read: The agent can retrieve live information and analyze it but cannot mutate state.
3. Propose: The agent selects an action and assembles the evidence needed for approval.
4. Act reversibly: The agent can make bounded changes that can be reliably undone, such as updating a staged record or creating an unpublished artifact.
5. Commit externally: The agent can trigger customer-visible, financial, privileged, or otherwise consequential actions.

Do not promote an entire product from one rung to the next. Promote individual actions. An agent may be safe to classify a support request, retrieve an account record, and draft a response while still requiring approval to send the response or issue a credit. Tool permissions should enforce that distinction; a prompt asking the model to be careful is not an authorization boundary.

Give each workflow a written contract before it receives production authority. At minimum, define:

• The eligible requests and the requests that must be rejected or escalated.
• The authoritative system for each piece of mutable state.
• The tools and operations the agent may use.
• The actions it must never take, even if a user asks.
• The evidence required before each consequential action.
• The conditions under which it must stop, ask a question, or hand control to a person.
• The observable state that proves success.
• The rollback or compensating action available after failure.
• The human owner who can suspend the workflow.

This contract becomes the basis for permissions, evaluations, monitoring, and incident review. Without it, teams debate whether a run felt good. With it, they can determine whether the system operated inside its approved envelope.

The distinction will matter more as capacity becomes abundant. The cost of reaching a fixed level of AI performance was estimated to have fallen about 47% per quarter over three years. Under stated assumptions, AI chips shipped through 2027 could support 30 million to 170 million concurrent frontier-model agents, or almost two billion agents using cheaper models. These are scenario estimates, not a forecast that demand will materialize. They still expose the strategic problem: execution can become cheap much faster than oversight, recovery, and accountability do.

Put checkpoints before commitment, not after completion

Many human-in-the-loop designs wait until the agent produces a final result. That is convenient for implementation and often poor for risk control. By then, the agent may have started with the wrong interpretation, searched the wrong evidence, called unnecessary tools, or changed intermediate state. The reviewer receives a polished output while the consequential mistake is buried in its path.

Place approval immediately before the point where an error becomes expensive to reverse. That principle is especially useful in long workflows because an early planning error can produce relevant-looking evidence and a coherent but misdirected result.

Start by drawing the workflow as a sequence of state transitions: interpret, plan, retrieve, decide, act, and verify. Mark the commitment boundaries. Typical boundaries include sending an external message, modifying a system of record, spending money, changing access, launching a batch operation, or passing instructions to another autonomous system.

Then choose the control for each boundary:

• Automated policy check: Use this when the rule is explicit and machine-testable, such as an allowed operation, account scope, amount ceiling, required field, or data-handling restriction.
• Human approval: Use this when judgment is required and the consequence is material. The person should see the intended action before it is executed.
• Independent verification: Use a separate rule, model, or deterministic check when the proposed action can be evaluated against evidence without repeating the entire task.
• Hard stop: Use this when the workflow has entered an unsupported state, the evidence conflicts, or no safe rollback exists.

An approval screen should not force a person to reconstruct the run. Present a compact decision packet containing the proposed action, the affected records or recipients, the evidence used, the policy checks passed, the expected state change, known uncertainty, and the rollback option. Make the reviewer approve the action, not merely acknowledge a generic risk message.

Research agents illustrate the pattern clearly. Approving a report at the end requires the reviewer to check the question, queries, evidence, synthesis, and conclusions. Approving the search plan before retrieval is cheaper and catches directional errors earlier. The same logic applies to operational agents: inspect the plan before fan-out and inspect the transaction before commitment.

Keep the checkpoint proportional to the consequence. Requiring a person to approve every low-risk read will create queues and habitual rubber-stamping. Removing approval from an irreversible action because the queue is inconvenient converts an operating problem into an uncontrolled risk. The durable fix is to narrow the action, improve the evidence packet, automate explicit policies, or create a reliable compensating action.

Make every consequential action a closed loop

Agents often look reliable while the environment resembles their successful examples. Small changes can reveal that the system learned a shortcut rather than the mechanism that matters. In one documented vision-language-action example, raising a table by just one centimetre broke the model’s behavior. A computer-use agent faces the same class of problem in software: every click changes the state from which the next decision must be made.

Causal world models may eventually help an agent reason more directly about how its actions change its environment, but their value in closed-loop production systems still needs testing. You do not need to wait for that research direction to mature. You can impose a causal discipline on the workflow now:

1. Observe: Read fresh state from the authoritative system and record the version, timestamp, or other concurrency signal.
2. Predict: State the intended change and the invariants that must remain true. Store a concise decision rationale, not hidden model reasoning.
3. Validate: Check identity, scope, permissions, policy constraints, required evidence, and whether the observed state is still current.
4. Act: Execute the smallest permitted operation with least-privilege credentials and an idempotency key where repetition could duplicate an effect.
5. Re-observe: Read the resulting state rather than treating a successful tool response as proof of business success.
6. Reconcile: If the result differs from the prediction, retry only when retrying is safe; otherwise compensate, stop, or escalate.

The re-observation step is frequently missing. A tool can return a successful transport response while the requested change is rejected later, applied to the wrong object, overwritten by another process, or only partially completed. Conversely, an agent may time out after an action succeeded and then create a duplicate when it retries. Verification and idempotency address different failure modes, and a production workflow usually needs both.

Retries should be defined per operation, not enabled as a universal resilience setting. Repeating a search is different from repeating a refund, sending a message, or changing a permission. For every mutating operation, document whether it is idempotent, how duplicate attempts are recognized, which error states permit a retry, and what compensating action restores safety.

Your event log must be able to reconstruct the state transition. Capture the run and workflow-contract versions, model and prompt versions, tool version, pre-action state fingerprint, proposed operation, policy result, approval event, tool request and response, post-action observation, verifier result, recovery attempt, human intervention, latency, and cost. Apply access controls and retention rules to these logs; reliable operations do not justify copying sensitive data indiscriminately.

This instrumentation also improves product decisions. You can distinguish a perception failure from a planning failure, a correct plan from a tool failure, and a successful tool call from a failed business outcome. Without that separation, the default response is usually to change the prompt, even when the prompt was not the broken component.

Evaluate the trajectory and the economics together

A final-answer score is not an agent reliability score. The same acceptable result may come from a compliant path, an unsafe path that happened to work, or an unapproved action later repaired by a person. Your evaluation unit should include the plan, evidence, tool sequence, state changes, verification, and recovery behavior.

Build the evaluation set from the operating envelope you actually intend to support. Include normal requests, ambiguous requests, conflicting evidence, stale state, unavailable tools, partial tool failures, permission failures, duplicate events, prompt injection attempts, and requests that should be refused or escalated. For consequential workflows, deliberately inject failures so you can test the stop and recovery paths before production discovers them.

Repeat each scenario. One clean run cannot establish reliability for a probabilistic system. A useful evaluation precedent comes from ToneBench, which runs each model configuration five times and uses four blinded judges across nine dimensions rather than ranking it from one fortunate generation. Your repeat count and evaluators should reflect your workflow’s risk, but the principle carries over: measure the distribution, not the demo.

Track metrics that reveal operational truth instead of a single blended quality score:

Metric	Definition	What it tells you
Verified completion rate	Eligible tasks that reached an independently observed success state divided by all eligible agent-started tasks	Whether the workflow completes useful work, not merely whether tools return success
Escaped consequential-error rate	Committed tasks with a material error discovered after the final checkpoint divided by committed tasks	Whether your gates prevent harmful mistakes from reaching users or systems
Unplanned intervention rate	Runs requiring human help outside the designed approval flow divided by total runs	How much hidden operational labor the automation creates
Safe recovery rate	Recoverable failures returned to a correct completion or known safe state divided by recoverable failures	Whether the system remains dependable when a component fails
Cost per verified completion	Model, tool, planned review, incident, and rework costs divided by verified completions	Whether cheaper inference translates into cheaper business outcomes

Slice these metrics by workflow version, model, tool, action type, customer segment, permission scope, and failure class. An aggregate can improve while a high-consequence slice gets worse. Set release and rollback thresholds for each risk tier before deployment; choosing the threshold after seeing the result invites teams to rationalize whatever the latest build produced.

Route by uncertainty and consequence

Falling model prices should improve unit economics, but they also make it easier to run unnecessary reasoning, retries, and multi-agent fan-out. Optimize for cost per verified completion rather than cost per token or cost per run.

Use deterministic software for fixed validation, arithmetic, authorization, deduplication, and state transitions. Use a cheaper model for bounded classification, extraction, or drafting when your evaluations show it stays inside the contract. Route ambiguous planning or complex interpretation to a more capable model. Route non-recoverable decisions to a person unless an explicit policy and strong evidence make automation acceptable.

The router should consider the requested action, current permissions, reversibility, evidence quality, detected uncertainty, and downstream blast radius. Routing only by a task label such as “support” or “sales” is too coarse: a support workflow can range from summarizing a ticket to disclosing account data or issuing money.

Be equally strict with caches. Reusing a stable definition is different from reusing an authorization decision or a conclusion derived from mutable account state. Do not cache permission checks, action approvals, or state-dependent conclusions beyond the context in which they were validated.

A simple planning identity keeps the economics honest: expected consequential failures rise with eligible run volume, the escaped-error rate, and the consequence of each failure. Lower inference cost increases the first factor. Your reliability program has to reduce the second factor or constrain the third before volume expands.

Expand autonomy one controlled dimension at a time

A reliable rollout is a sequence of earned permissions. Increasing users, traffic, tools, action scope, and model autonomy simultaneously makes failures hard to attribute and rollback decisions hard to make.

1. Write the contract and action map. Define eligible tasks, state authorities, commitment boundaries, forbidden operations, success evidence, and recovery ownership.
2. Replay and shadow. Run representative historical cases and observe live cases without granting mutation rights. Compare proposed actions with known outcomes and human decisions.
3. Enable bounded reversible actions. Restrict scope, credentials, affected objects, and available operations. Verify every state change independently.
4. Add approvals at commitment boundaries. Give reviewers evidence-rich decision packets and measure approval latency, rejection reasons, and rubber-stamping signals.
5. Exercise degraded paths. Simulate stale reads, tool outages, partial completion, duplicates, policy conflicts, and failed recovery. Confirm that the workflow stops safely.
6. Expand one dimension. Increase either volume, user coverage, tool access, action scope, or reduced oversight. Preserve the ability to identify which change affected the metrics.

Use a launch gate that can say no

Before each expansion, require evidence that the workflow contract is versioned, tool permissions are least-privilege, fresh-state checks run before commitment, repeatable operations are idempotent, postconditions are independently verified, rollback or compensation has been exercised, logs can reconstruct a run, and a named owner can suspend execution.

Also require acceptable results for verified completion, escaped consequential errors, unplanned intervention, recovery, and cost per verified completion. “The model got better” is not a launch criterion. A model, prompt, tool, policy, retrieval source, or permission change can alter system behavior, so treat each as a versioned release and rerun the relevant evaluations.

Key takeaways

• Measure scale by committed decisions and blast radius, not by requests served.
• Place human or automated gates before actions become costly to reverse.
• Close the loop by observing fresh state, predicting the change, acting with bounded authority, and verifying the result.
• Evaluate complete trajectories across repeated runs, including failure and recovery paths.
• Route work according to uncertainty, reversibility, and consequence; optimize for cost per verified completion.
• Expand one dimension at a time so you can attribute regressions and reverse them safely.

Your next step does not require a fleet-wide redesign. Choose one production candidate, mark its commitment boundaries, add fresh-state validation and post-action verification, and measure whether it reaches a proven success state without unplanned help. Increase its authority only after those controls can catch, stop, and explain the failures you know how to induce.

References

• Learn AI Together – LAI #147: Can AI Learn Cause and Effect?
• Epoch AI – The Epoch Brief

---