An AI agent has finished the feature, every test it chose to run is green, and the diff looks plausible. You still do not know whether the change is safe to ship. The missing asset is not more generated code. It is a workflow that makes scope, evidence, risk, and ownership visible.

You can close that gap without slowing every change to a crawl. Give the agent a bounded contract, supply only the context needed for the task, require reproducible proof, and increase human control as consequences rise. That turns AI coding from an impressive individual trick into an engineering capability your organization can govern.

Define done as evidence, not generated output

Code generation is an activity. A reviewable, tested, reversible change is an outcome. If your workflow treats the agent’s final response as completion, the agent gets to choose both what counts as success and whether it succeeded.

Start each task with a compact contract. It does not need to become a lengthy specification, but it must remove the decisions you do not want the agent making implicitly:

• Outcome: State the behavior that must change from the user’s or system’s point of view.
• Non-goals: Name adjacent behavior that must remain untouched. This is one of the simplest ways to prevent an agent from turning a local repair into an unsolicited redesign.
• Change boundary: Identify the service, package, interface, or files that are in scope. Require justification before work crosses that boundary.
• Constraints: Preserve relevant API contracts, data formats, authorization rules, dependency policies, and architectural conventions.
• Acceptance evidence: Specify the tests, checks, or observable behaviors that will demonstrate completion.
• Permissions: Say whether the agent may edit files, install dependencies, run migrations, or call external systems. Permission to write code is not permission to act on production.
• Recovery: For a change that can affect data or customer traffic, require a safe deployment and rollback path before implementation begins.

Compare the instruction “improve model-request reliability” with a contract that says: handle the existing timeout path, preserve the public interface and idempotency behavior, add a regression test for the failure, keep unrelated retry behavior unchanged, and report the exact checks run. The second version gives the agent a target and gives the reviewer a basis for rejecting scope drift.

The named human responsible for the change should accept this contract before substantial editing starts. The agent can expose assumptions and propose acceptance checks, but it should not quietly redefine the requested outcome to fit the patch it found easiest to produce.

Control context, then use an inspect-build-prove loop

Context is a control surface. Too little context makes the agent guess. A large, undifferentiated context dump can bury the instruction that actually matters. Organize what the agent receives into three layers:

• Persistent repository rules: Architecture boundaries, coding conventions, security requirements, standard commands, forbidden operations, and the definition of done belong close to the code.
• Task-local facts: Give the agent the accepted contract, relevant files, current behavior, the exact error or request, and any known constraints.
• Retrieved evidence: Bring in additional documentation, logs, interfaces, or prior decisions when the task needs them. Do not keep every potentially useful artifact in every conversation.

Make instruction precedence explicit. Repository security rules should not disappear because a task prompt is vague, and a task-specific acceptance condition should not be displaced by the agent’s preferred refactor. When two instructions conflict, the agent should stop and surface the conflict instead of choosing silently.

Once the context is clean, run each change through the same operating loop:

1. Inspect without editing. Ask the agent to trace the current execution path, identify the relevant interfaces and tests, and list what it still does not know. This catches mistaken assumptions before they become a broad diff.
2. Propose a bounded plan. The plan should name the intended files, behavioral change, validation steps, and meaningful risks. Any expansion beyond the task contract needs explicit approval.
3. Implement the smallest coherent change. Keep the patch narrow enough that a reviewer can connect each changed block to the accepted outcome. Unrelated cleanup belongs in a separate task.
4. Debug from observations. Feed the agent the exact failing command, relevant error, expected behavior, actual behavior, smallest available reproduction, and the diff since the last working state. Do not ask it to “try another fix” without new evidence.
5. Produce a proof packet. Require a concise record of changed files and reasons, exact commands run, results, checks not run, remaining assumptions, and deployment or rollback considerations.

The proof packet matters because “tests pass” is not reproducible. A reviewer needs to know which tests passed, against which change, and what the agent omitted. The agent should run focused checks while iterating and the broader relevant suite before declaring the patch ready. If a required check cannot run, that is an unresolved release fact, not a detail to hide in conversational output.

This loop also improves debugging discipline. When a proposed fix contradicts the latest evidence, discard the hypothesis. Repeated speculative edits create a larger search space, make regressions harder to localize, and leave reviewers reconstructing which change was supposed to solve which failure.

Test deterministic code and stochastic AI behavior differently

A conventional code check and an LLM behavior evaluation answer different questions. A formatter, type checker, or deterministic unit test should produce the same result from the same code and environment. Repeating it without changing anything adds little evidence. An LLM evaluation can vary even when the surrounding application code is unchanged.

For an important stochastic case, a practical screening baseline is to run the same case five times while holding the prompt, model, temperature, other randomness settings, and retrieved context fixed. If you change those inputs between runs, you are changing the test while trying to measure consistency.

The reason is easy to miss after one successful run. Suppose a difficult case has a 90% chance of passing on any run. The probability of seeing at least one failure across five runs is 1 – 0.9⁵, which is about 41%. One green result can therefore conceal an intermittent failure that a small repeated sample has a meaningful chance of exposing.

Record both of these results:

• The pass rate across all five runs.
• Whether the case passed all five runs without an intermittent failure.

Keep the failed outputs as evidence and classify the failure mode. A single average score can hide the distinction between a harmless wording variation and a broken safety, accuracy, or workflow requirement. Define the release threshold before running the evaluation, based on the consequence of failure. Do not encounter a four-out-of-five result and then lower the standard because the patch is otherwise convenient.

Five runs are not a statistical guarantee, and they should not be presented as certification. They are a practical way to reveal instability that one successful run can miss. Higher-consequence behavior may need a stronger evaluation design and production monitoring, while deterministic application checks should remain ordinary merge gates.

Scale autonomy by consequence and make corrections durable

The agent’s apparent fluency should not determine how much autonomy it receives. Use blast radius, reversibility, data sensitivity, and detectability instead. A narrow internal refactor with strong regression coverage can move through a lighter review path than a change to authorization, billing, customer data, or a production migration.

Review every meaningful AI-generated change through five lenses:

• Intent: Does each material part of the diff map to the task contract? Flag unrelated edits and behavior that the request never authorized.
• Correctness: Examine failure paths, boundary conditions, interface compatibility, and concurrency where relevant. Plausible happy-path code is not enough.
• Test validity: Confirm that the new test would fail without the fix, asserts the intended behavior rather than the implementation, and covers the regression the task claims to address.
• Operational readiness: Check observability, deployment order, configuration changes, compatibility, and rollback. A code-correct patch can still be unsafe to release.
• Security and data handling: Recheck authorization, input validation, secret exposure, dependency changes, and data access rather than assuming the agent preserved existing controls.

For a destructive database operation, never let the agent execute against production merely to validate its work. Prove the change with disposable data, require a human-approved backup and recovery path, and review whether rollback is genuinely possible. The downside is not a failed test; it can be unrecoverable data loss.

A second agent can be useful as a critic. It can search for untested branches, mismatches between the contract and diff, or claims unsupported by the proof packet. That is additional evidence, not independent accountability. A human owner still decides whether the residual risk is acceptable.

Then convert repeated review feedback into repository memory. If humans keep making the same correction, the workflow is missing a durable control:

• Turn a recurring instruction into a concise repository rule.
• Turn a mechanical convention into a formatter, linter, or automated check.
• Turn a recurring defect into a regression test.
• Turn missing validation evidence into a pull-request requirement.
• Turn an architectural boundary into an executable check where practical.

Review those controls as the system changes. Stale instructions create contradictions, and contradictions push the agent back toward guesswork.

Measure the workflow as an engineering system, not as an AI adoption campaign. Useful signals include first-pass review acceptance, substantive revision cycles, CI failures after the agent declared completion, escaped defects, rollbacks, and time from an accepted task to a verified change. Lines generated, prompts sent, and raw agent usage can rise while delivery quality falls.

For product and engineering leaders, this is the real governance work: decide where agents may operate, what evidence each class of change requires, which actions need human approval, and what conditions stop the workflow. Tool selection comes after those decisions.

Key takeaways

• Start with a human-accepted task contract covering outcome, non-goals, constraints, permissions, acceptance evidence, and recovery.
• Separate persistent repository rules from task-local facts and retrieved evidence so the agent can identify the authoritative instruction.
• Use an inspect-build-prove loop, and require exact commands, results, omissions, risks, and rollback notes in the final proof packet.
• Repeat important stochastic LLM evaluations five times with all test inputs fixed; record both the pass rate and whether all five runs passed.
• Increase human control with blast radius and irreversibility, not with the agent’s fluency or the apparent simplicity of its response.
• Convert repeated corrections into instructions, automated checks, regression tests, or review requirements.

Choose one active repository and apply this workflow to the next bounded change. Add a task-contract template and proof-packet template first. The first repeated review correction will show you which durable guardrail to build next.

References

• Learn AI Together – LAI #140: Our AI Coding Playbook

---