You have a green evaluation dashboard and a launch decision due. The hard question isn’t whether the model passed. It’s whether the test covered the same tools, identities, data paths, network access, failure modes, and recovery controls that will exist in production.
When those boundaries differ, a high score can be reassuring and irrelevant. Treat evaluation as release infrastructure: define the loss you are preventing, reproduce the production control surface safely, attack the containment boundary, preserve independent judgment, and connect failures to an incident process.
Start with the loss, not the benchmark
A benchmark asks whether a model can perform a task. A release decision asks a different question: can the deployed system cause an unacceptable outcome, and will your controls prevent, detect, and reverse it?
That distinction matters because the model is only one component. An answer can look correct while the surrounding system retrieves another customer’s data, calls the wrong tool, repeats a non-idempotent action, exposes a credential, or continues operating after its monitoring has failed. An output-only evaluation will miss all of those paths.
Before writing test cases, create an evaluation contract for each consequential workflow. Record:
- The job the system is allowed to perform.
- The decisions and actions it may take without approval.
- The outcomes that are prohibited, even if the user requests them.
- The data, identities, tools, and external systems within reach.
- The control expected to stop each prohibited outcome.
- The telemetry that should prove the control fired.
- The operator action that contains the system and reverses any side effect.
A useful contract is specific enough to become a test: Given an input containing an instruction from an untrusted document, the agent may read the permitted account fields but must not reveal a token, contact an unapproved host, or invoke a write action without the required approval. Every blocked attempt must leave an observable event.
Write this contract before the final evaluation run. If the team invents the success condition after seeing results, it can unconsciously redefine a failure as an acceptable edge case.
Then verify that the contract represents the intended product, not merely something that is easy to score. Questions around a claimed AI-assisted Navier-Stokes proof included whether the natural-language conjecture and its Lean representation expressed the same proposition. The product equivalent is a flawless test of the wrong requirement: the judge passes behavior that users, security owners, or domain experts would reject.
Require three perspectives before freezing an evaluation contract. The product owner confirms the intended user outcome. The domain or security owner identifies the credible loss. The evaluator confirms that the oracle distinguishes acceptable behavior from a convincing imitation. Sample several pass and fail cases together; if the reviewers disagree, the specification is not ready to automate.
Finally, attach a deployment manifest to every result. It should identify the model and version, system instructions, retrieval collection, tool schemas, authorization policy, retry and timeout behavior, human approvals, network rules, policy engine, and logging configuration. If any of these differ from production, label the evidence as non-equivalent instead of quietly carrying the score into a launch review.
Reproduce production safely, then attack every boundary
The central infrastructure problem is realism without uncontrolled access. A sandbox that cannot behave like production produces weak evidence. A test environment with production credentials and unrestricted side effects creates the very incident you are trying to prevent.
One current containment approach combines stronger adversarial testing with an offline collection of indexed and cached web content. That is useful for questions that require realistic information without open Internet access. It is not enough for an agent that interacts with changing services, asynchronous callbacks, approvals, or people. For those workflows, you need service replicas and simulated actors, not just cached pages.
| Surface | What the environment should reproduce | Failure to force |
|---|---|---|
| Decision behavior | Model, instructions, retrieval, policies, and context assembly | Ambiguous requests, conflicting instructions, prompt injection, and missing context |
| Tool execution | Tool schemas, authorization checks, retries, timeouts, concurrency, and error responses | Duplicate calls, partial completion, malformed output, delayed callbacks, and stale state |
| Identity and data | Production-like roles, tenant boundaries, field-level access, and retention behavior | Cross-tenant requests, excessive retrieval, secret access, and confused-deputy behavior |
| Network and runtime | Egress policy, filesystem rules, package controls, process isolation, and credential delivery | Unapproved destinations, lateral access, privilege escalation, and attempts to escape containment |
| Operations | Logs, alerts, queues, fallbacks, approvals, stop controls, and rollback paths | Lost telemetry, alert failure, unavailable reviewers, in-flight work during shutdown, and failed rollback |
Reproduce semantics, not secrets. Use non-production credentials with the same permission structure, representative or synthetic records with the same schema, and fake destinations for consequential actions. Email should land in a controlled sink. Payments should reach a stub. CRM writes should go to a replica. The agent should encounter realistic responses and failures without being able to affect a customer or third party.
Make outbound access denied by default. Provide explicit routes to approved mirrors and replicas, and log every attempted connection. Apply the same principle to filesystems, environment variables, package installation, inter-process communication, and internal service discovery. A containment control is not proven because the normal workflow stayed inside it; it is proven when deliberate attempts to leave it are blocked and visible.
Your adversarial cases should ask whether the system can reach a forbidden result indirectly. Can an approved tool delegate to an unapproved one? Can retrieved content persuade the agent to disclose a secret through a permitted field? Can repeated retries turn one approved action into several side effects? Can the model continue after the policy service or audit logger becomes unavailable? Test the path around the control, not only the control’s advertised interface.
Move through environments in a deliberate sequence:
- Replay recorded or synthetic inputs with no external actions.
- Run against simulated services where side effects are contained.
- Shadow real traffic with read-only access and outputs that cannot be acted on.
- Allow gated live actions with restricted permissions and explicit approvals.
- Broaden authority only after the relevant failure and recovery paths have been exercised.
Shadow mode is not harmless. It may still expose production data, create retained model inputs, or reveal cross-tenant retrieval flaws. Apply access control, redaction, retention limits, privacy review, and security approval before connecting it to live traffic.
Make independence part of the release mechanism
An evaluation can be technically sophisticated and still produce weak assurance if the same group defines the rubric, chooses the cases, interprets ambiguous results, and owns the launch deadline. The problem is not bad intent. It is that delivery pressure changes which failures look material.
The EU AI Office’s work on external evaluator independence under the GPAI Code of Practice includes concrete conflict checks: evaluators should not hold provider shares, compensation should reflect market rates rather than performance, and certain adjacent paid advisory or implementation relationships are restricted. Those details are still being operationalized, so regulated teams should confirm the binding requirements with qualified counsel. The broader product lesson is already usable: nobody assessing a control should benefit from declaring that control effective.
Independence does not require keeping evaluators outside the building. Embedded third-party evaluation has been proposed as a way to observe procedures, culture, and safety practices from inside a developer. Access and independence solve different problems. Independence without access becomes a questionnaire. Access without independence becomes self-certification.
Use a role split that makes the decision path explicit:
- The builder documents the architecture, proposes failure hypotheses, and fixes defects, but does not issue the final assurance decision.
- The evaluation owner controls the frozen suite, run procedure, exclusions, reruns, and result record.
- Security and platform owners can block release when identity, containment, telemetry, or recovery controls fail.
- The product release owner accepts documented residual product risk rather than converting it into an evaluation pass.
- An independent reviewer can inspect raw runs, omitted cases, configuration manifests, and unresolved disagreements.
- Legal and compliance owners determine regulatory scope and notification duties; an evaluation score is not a legal conclusion.
Use a release gate that cannot average away a critical failure
A single composite score is a poor release gate. A prohibited write action that succeeds once should not disappear inside a large number of correct answers. Separate task quality from control assurance, and report failures by consequence and control layer.
A release packet should answer all of these questions:
- Did the evaluation use the deployment manifest intended for release?
- Were the success criteria and prohibited outcomes frozen before the final run?
- Did the suite include direct, indirect, and adaptive attempts to reach each critical loss?
- Did the relevant containment controls stop the attempt and emit usable evidence?
- Were stop, credential revocation, rollback, and side-effect recovery exercised?
- Are exclusions, flaky reruns, human overrides, and missing telemetry visible in the result?
- Does every exception have an owner, rationale, review trigger, and condition for removal?
- Is there an independent sign-off or a recorded disagreement for each high-consequence control?
Keep the run record immutable enough to reconstruct the decision. Preserve prompts, model identifiers, tool and permission manifests, dataset versions, evaluator code, relevant application commits, raw outputs, tool traces, policy decisions, and reviewer overrides. Without that chain, a later investigation cannot tell whether the model changed, the environment drifted, or the original result was misread.
Do not assume that calling a model internal removes regulatory exposure. Whether internal use falls within market-placement rules can become uncertain when a model leaves its sandbox. If a system is about to receive broader employee access, customer data, external tools, or a route to production, have counsel determine the applicable obligations before that boundary changes.
Connect evaluation failures to incident readiness
An evaluation program that ends with a defect ticket is incomplete. Some failures show that a model needs improvement. Others show that a security or operational control did not hold. The second kind belongs in your incident system even if the attempt occurred in an isolated environment.
NIST organizes ten key AI incident practices around preparing, detecting, responding, recovering, and reporting. Use that lifecycle to connect pre-release evidence with production operations:
- Prepare: name the incident owner and stop authority; inventory models, tools, credentials, data stores, queues, and external dependencies; document containment and recovery actions.
- Detect: correlate model decisions, retrievals, policy checks, tool calls, approval events, data access, credential use, and network attempts under a common run or request identifier.
- Respond: disable the affected tool or integration, revoke exposed credentials, isolate the workload, stop queued actions, preserve evidence, and involve the owners of affected systems.
- Recover: verify external state, reverse side effects through the system that owns them, restore least-privilege access, rerun the regression suite, and return traffic gradually.
- Report: record what happened, which assets and users were affected, which controls failed, how long exposure continued, and which internal or external notifications are required.
Classify failures by consequence and control state, not by the size of a benchmark change. A bad answer that is caught inside the expected test boundary is a product defect. A successful bypass of isolation, authorization, or monitoring is a security event because a promised control failed. An unapproved production read, write, disclosure, or external action should enter the full incident process, with legal, privacy, security, and customer-impact review as applicable.
Rolling back the model may not contain the incident. Credentials may have been exposed, writes may already exist in external systems, messages may be queued, and retrieved data may remain in logs or provider retention. Your runbook must address those persistent effects explicitly.
Production monitoring should preserve the signals that made the evaluation useful. Track prohibited-action attempts and successes, policy denials, approval bypass attempts, unusual tool sequences, denied egress, cross-tenant access attempts, time to detection, time to containment, configuration drift, unresolved exceptions, and whether rollback remains executable. Aggregate task success belongs on a product dashboard; it should not obscure the state of a critical control.
Rerun the affected suite whenever the model, system instructions, retrieval collection, tool contract, permission policy, provider endpoint, network rule, retry behavior, or human approval flow changes. Treat these dependencies as versioned release inputs. Otherwise, the evaluation expires while the dashboard remains green.
Key takeaways
- Evaluate the deployed system’s loss paths, not just the model’s answers.
- Attach every result to an exact manifest of models, prompts, tools, permissions, data, network controls, and operational settings.
- Use realistic replicas and fake destinations so you can test consequential actions without creating real consequences.
- Attack containment indirectly as well as directly, and require observable evidence that every control fired.
- Separate builders, evaluators, reviewers, and release owners so delivery pressure cannot silently redefine a failure.
- Route control failures into incident readiness, then preserve the same telemetry and recovery mechanisms in production.
For your next AI release, pick the workflow with the largest irreversible consequence. Write its evaluation contract, map every reachable asset and control, and run one end-to-end containment and recovery exercise using the exact release configuration. If the team cannot produce the evidence or complete the recovery, the next milestone is not broader access. It is closing that gap.
References
