Your AI software factory generated the change, ran the tests, passed the evals, repaired its own failures, and shipped behind a feature flag. The delivery system looks healthy. Then someone asks the product question: did the change actually help customers?
A green pipeline cannot answer that. It proves only what you designed it to verify. If you lead an AI product, your job is to connect the factory’s internal checks to a second loop that observes real use, measures the intended outcome, and turns the result into a product decision.
One release has four different verification problems
An AI software factory is a repeatable system for turning an intent into working software. Agents may plan a change, write code, run tools, inspect failures, revise the implementation, and prepare or deploy the release. Verification loops make this process useful because they keep sending failed work back through the factory.
The mistake is treating verification as one gate. A release can pass several valid checks while still failing the product test. Each layer answers a different question:
| Verification layer | Question it answers | Typical evidence | What a pass does not prove |
|---|---|---|---|
| Change verification | Does the implementation satisfy the specified task? | Static checks, unit tests, acceptance tests, and review findings | That the change works across the complete production workflow |
| System verification | Does the change behave correctly with the surrounding product? | Integration tests, permission checks, migration checks, and AI eval suites | That the release will remain healthy under production conditions |
| Release verification | Is the deployed treatment technically healthy? | Exposure checks, errors, latency, incidents, and rollback signals | That users receive more value or change their behavior |
| Product verification | Did the intended users achieve the intended outcome? | Experiment results, behavioral data, guardrails, and qualitative evidence | That the result will generalize to another audience or treatment |
The first three layers compare the release with expectations you already encoded. The fourth tests the product belief behind the release. That is why internal verification cannot establish that a change moved a product metric. Evidence about customer behavior does not exist until eligible users encounter a clearly identified treatment.
This does not make tests or evals less important. It makes their boundaries explicit. A unit test can catch a broken transformation. An AI eval can detect an unacceptable response on a chosen set of cases. A deployment check can catch rising errors. None can tell you whether the feature solved the bottleneck in the customer’s workflow.
Keep all four layers visible in your release system. Do not let a status such as “verified” collapse them into one ambiguous signal.
Give the factory a product verification contract
A factory can verify only what you make explicit. If the brief says “improve the AI onboarding experience,” an agent can produce plausible software, but it cannot infer which user, behavior, risk, or business result determines success.
Before implementation begins, write a product verification contract. It should answer these questions:
<!– wp:list {
