eval-driven development – Page 11 – Shivam.Consulting Blog

Your team has a polished GenAI prototype. The review goes well. Then the hard questions arrive: Which customer behavior should change? How often does the system fail? What context does it need? Who owns prompt and model changes? Can you release it without creating a permanent escalation queue?

The gap between an impressive demonstration and a dependable product is usually an operating-model problem. A useful GenAI discovery model connects customer evidence, system evaluations, field behavior, and production ownership. It lets you preserve the speed of AI-assisted prototyping while making each experiment answer a real product decision.

Key takeaways for GenAI product leaders

Prove value and capability separately. Customer demand does not prove that a model can perform reliably, and a strong evaluation score does not prove that anyone will change their behavior.
Begin with a workflow, an outcome, and a bounded role for AI. A generic AI use case is too loose to guide discovery or define acceptable performance.
Maintain an evidence chain. Connect each customer story to an opportunity, each opportunity to an assumption, each assumption to an evaluation scenario, and each released behavior to a product outcome.
Run customer learning and system learning in the same weekly loop. They require different methods, but they must meet at one decision: advance, narrow, change, or stop the bet.
Centralize reusable infrastructure and guardrails, not product judgment. Product teams should own the workflow and outcome while shared capabilities provide model access, evaluation tooling, observability, and policy controls.

Start with the workflow and outcome, not the model

GenAI discovery often starts in the wrong direction. A team gets access to a capable model, generates a list of possible features, and builds whichever idea looks most compelling in a demo. The team may learn that the model can produce something plausible, but it still does not know whether the capability removes meaningful friction from a real workflow.

Reverse the sequence. Start with a person trying to achieve an outcome in a specific context. Understand what that person does now, where the workflow breaks, what consequence follows, and which constraints shape a usable solution. Only then should you decide whether generation, retrieval, prediction, conversation, or agentic action is an appropriate intervention.

Write a bet brief that can be disproved

A GenAI bet should fit into a short brief before anyone commits significant engineering capacity. Use this framing:

For a specific actor in a specific workflow moment, the current behavior makes it harder to achieve a desired outcome. A bounded AI capability may improve an observable behavior, provided it stays within defined quality, control, privacy, and safety conditions.

Make every part concrete enough to test:

Actor and moment: Name who encounters the problem and where it occurs in the workflow. A role such as support agent is not enough; identify the decision or task the person is performing.
Current behavior: Describe what people actually do, including workarounds, handoffs, rework, and information they assemble manually. Ground this in past behavior rather than hypothetical interest.
Desired outcome: Define the customer or business result that should improve. Completion, adoption, rework, escalation, abandonment, and repeat use are product signals. A model-quality score is not the outcome.
AI contribution: State whether the system retrieves information, creates a draft, recommends an action, or performs an action. These are materially different product and risk propositions.
Operating boundary: Specify the data context, permissions, supported cases, human checkpoints, and fallback path. A capability without a boundary cannot be evaluated honestly.
Decision: Write what evidence would cause the team to proceed, narrow the scope, change the approach, or stop.

Suppose the opportunity is helping a support agent respond to a complex case. Producing a fluent answer is not the unit of value. The product may need to retrieve the right account context, create a grounded draft, expose its basis, let the agent correct it, and contribute to resolving the issue without avoidable rework. That framing changes what you prototype, what you evaluate, what you instrument, and what you ask users during discovery.

Choose the smallest coherent release

The right initial scope is not the smallest visible feature. It is the smallest end-to-end experience that can produce credible evidence. A narrow workflow with real context, a clear human checkpoint, outcome instrumentation, and a recovery path is more useful than a broad assistant that performs many disconnected tasks. This is how rapid prototyping becomes a method for reducing uncertainty instead of a factory for disposable demonstrations.

Scope the first release along four boundaries:

A defined user and workflow moment.
A bounded set of data and tools the system may use.
A clear level of authority over the resulting action.
A measurable product outcome and a separate quality floor.

Authority matters because the same model behavior can create different consequences. Showing relevant information is different from drafting a message. Drafting is different from recommending that it be sent. Recommending is different from sending it automatically. As authority rises and reversibility falls, require stronger evaluations, clearer permission boundaries, better auditability, and more deliberate human approval. If an action is costly, sensitive, or difficult to reverse, keep a human checkpoint and a safe fallback in the initial operating envelope.

Keep product success, system quality, and operating viability distinct. Product success asks whether behavior and outcomes improve. System quality asks whether outputs meet defined conditions. Operating viability asks whether the product can deliver that behavior with acceptable latency, cost, support burden, and failure recovery. A bet needs evidence in all three categories before a polished prototype should influence a roadmap commitment.

Build a traceable evidence chain

There are two uses of AI in this work, and leaders should not conflate them. AI can assist the discovery process by transcribing interviews, organizing material, and generating alternative interpretations. AI can also be the product capability under investigation. In the first role, its output is an analytical input. In the second, its behavior is an object of evaluation. Neither role turns model output into customer evidence.

Preserve the customer story before looking for patterns

Good discovery begins with a specific story about past behavior. Capture the person’s goal, the context, key moments in the experience, decisions, workarounds, and the needs or pain points that emerged. Synthesize that interview on its own before combining it with other interviews. This prevents a recurring operational mistake: compressing many transcripts into generic themes that are easy to present but impossible to design against.

A practical single-interview record includes participant context, a transcript-linked quote, the sequence of important moments, and opportunities expressed within that person’s situation. Cross-interview synthesis can then organize related opportunities without severing them from their origins. This separation between individual and cross-interview synthesis keeps insights contextual, actionable, and traceable.

Use AI as a notetaker or an additional analytical perspective, but establish simple provenance rules:

Every direct quote must link to the transcript location where it appears. If the wording cannot be verified, treat it as a paraphrase or discard it.
Every opportunity must identify the participant, workflow moment, and evidence from which it was derived.
Generated summaries must be checked against the original material before entering an opportunity map or decision record.
Tone, hesitation, visible confusion, and body language must come from a human observation note when they affect interpretation; a text transcript cannot preserve them reliably.
AI-generated themes may prompt a second look at the evidence, but they do not become evidence through repetition.

If the interview itself is shallow, automation will only process shallow material faster. A model cannot recover a missing motivation, workflow constraint, or decision context that nobody elicited. When synthesis produces vague insights, inspect the interview quality before rewriting the prompt.

Connect every artifact to a decision

The evidence chain should survive the entire path from discovery to production:

A customer story supports an opportunity.
The opportunity supports a product bet.
The bet contains assumptions about value, usability, feasibility, data, trust, and operations.
High-risk assumptions become experiments and evaluation scenarios.
Evaluation scenarios become regression cases when the product changes.
Field behavior connects the released capability to product and business outcomes.
Production failures feed new scenarios, product constraints, and discovery questions.

This chain prevents evidence laundering. A customer describing a difficult workflow does not prove that a proposed solution is desirable. A user liking a prototype does not prove that behavior will change. Passing an offline evaluation does not prove that the interaction fits the workflow. A successful pilot does not prove that the system is repeatable across customers. Each step answers a different question.

Decision	Minimum artifact	Evidence that advances the bet	What to do when it is missing
Is the problem worth solving?	Interview snapshots and an opportunity map	Specific past behavior, context, consequences, and constraints	Return to interviewing or workflow observation
Can GenAI contribute meaningfully?	Assumption ledger and a thin prototype	The capability performs the bounded task on representative examples	Narrow the task, change the system approach, or stop
Can users understand and control it?	Interaction prototype and failure scenarios	Users can interpret the result, correct it, and recover from failure	Change the interaction, authority level, or human checkpoint
Does it change real behavior?	Instrumented field pilot	Observed usage changes a leading product signal without unacceptable quality loss	Investigate workflow fit, adoption friction, or the original value hypothesis
Can the product be operated?	Evaluation harness, monitoring, ownership, and rollback path	Quality remains observable after release and a named owner can respond to degradation	Keep the release bounded until the operating controls exist

Use a living bet packet instead of a presentation deck

Keep the evidence chain in one lightweight packet. It should contain the current problem frame, links to customer evidence, the assumption ledger, prototype configuration, evaluation results, field observations, outcome measures, and the latest decision with its rationale. It is not a status report. It is the memory of the bet.

A new team member should be able to see why the work exists, which uncertainty is active, what failed previously, and what would change the decision. If the packet grows without changing a decision, reduce it. The purpose is traceability, not documentation volume.

Run a weekly dual-track discovery loop

GenAI discovery has two learning tracks. The customer track investigates the workflow, value, behavior, interaction, and trust. The system track investigates model behavior, context quality, tool use, failure modes, and operational constraints. Running only the first creates attractive concepts with unknown feasibility. Running only the second produces technically impressive capabilities looking for a problem.

A weekly loop is a useful default because it keeps the customer and system evidence synchronized. It also forces the team to choose a decision small enough to advance with the evidence available. The cadence is not a sequence of meetings. It is a sequence of testable changes.

Customer and workflow learning

This track uses customer interviews, workflow observation, usability tests, pilot behavior, support signals, and outcome instrumentation. Each activity should begin with the decision it is meant to inform. Do not schedule an interview merely to gather feedback. Decide whether you need to understand the current workflow, test the meaning of an opportunity, observe a recovery interaction, or determine why pilot users are abandoning the capability.

Invite engineering into customer learning when technical context changes the solution space. An engineer hearing how permissions, data fragmentation, or exceptional cases shape the workflow can identify constraints earlier than a written handoff would. The product manager should still own the connection between those constraints and the desired outcome.

System and evaluation learning

This track needs a scenario set, not a folder of memorable outputs. Build it from ordinary customer cases, meaningful edge cases, prohibited behavior, and failures already observed. Synthetic examples can widen coverage, but they should extend a foundation of real workflow evidence rather than replace it.

Each scenario should record:

The user task and its provenance in customer or field evidence.
The input, relevant context, permissions, and available tools.
Acceptable behavior, unacceptable behavior, and the scoring method.
The model, prompt, retrieval, tool, and data configuration used.
The actual result, the failure category when applicable, and any human correction.
The product consequence of the failure, not merely its technical label.

Define acceptance criteria before reviewing a preferred prototype. Otherwise, a persuasive output can cause the team to move the standard after the fact. Automate checks when correctness is directly observable, and retain structured human review where meaning, usefulness, or trust depends on context.

Treat the scenario set as part of the product, not disposable test material. Prompt versions, retrieval configuration, model changes, data quality, and tool behavior all belong to the release surface. Evaluation harnesses, prompt versioning, red-teaming, and production monitoring therefore need to begin in discovery and continue through CI/CD. A scenario that exposes a real failure during a pilot should become a regression case before the next release.

A decision-oriented weekly sequence

Name the decision. Choose the highest-risk assumption blocking progress rather than the most convenient activity.
Inspect the evidence. Review the relevant customer stories, field behavior, scenario results, and known failures.
Design the cheapest credible test. This may be another interview, a workflow prototype, a prompt or retrieval change, a human-assisted simulation, or a bounded field experiment.
Run offline scenarios first. Find obvious quality, safety, grounding, and permission failures before asking a customer to spend time on the prototype.
Observe the capability in the workflow. Watch what users accept, edit, ignore, misunderstand, override, or abandon.
Update both records. Add customer learning to the opportunity and assumption records; add system failures to the scenario set and failure taxonomy.
Make the decision explicit. Advance, narrow, change, pause, or stop. Record the evidence that caused the decision and identify the next uncertainty.

Parallel experimentation is useful only when each experiment resolves a distinct uncertainty. Generating many prototypes against the same vague question increases activity without increasing decision quality. The advantage of GenAI is a lower cost of learning; spend that advantage on better coverage of assumptions, not a larger pile of concepts.

Promote a prototype only when the evidence changes

A prototype should not graduate because it performs well in a stakeholder demonstration. Move it toward a broader pilot only when the team can show:

A valuable workflow and outcome grounded in customer behavior.
A bounded capability with explicit supported and unsupported cases.
Acceptance criteria covering ordinary use, meaningful edge cases, and prohibited behavior.
An interaction that helps users understand uncertainty, correct results, and recover.
Instrumentation for product outcomes, system quality, overrides, and failures.
Ownership for monitoring, escalation, and rollback after release.

Keep the value gate and the quality gate separate. A system can meet its quality threshold and still fail because it adds a step, appears at the wrong moment, or solves a low-value problem. It can also attract strong usage while producing unacceptable errors. The first result sends you back to product discovery; the second requires narrowing authority, strengthening the system, or stopping the release. Blending the gates makes both diagnoses harder.

Design the organization around decision rights and learning

A fast learning loop will stall if every product choice waits for a central AI committee. It will also become fragile if each squad invents its own model access, evaluation approach, privacy controls, and incident response. The operating model must distinguish decisions that require local customer context from capabilities that should be reusable across the company.

Keep the product trio accountable for the outcome

The titles can vary, but the responsibilities cannot remain implicit:

Product leadership owns the desired outcome, opportunity framing, assumption sequence, value evidence, and decision record.
Design and research own workflow understanding, interaction behavior, user control, comprehension, correction, and recovery.
Engineering and data own system architecture, context and data quality, repeatable evaluations, observability, reliability, and rollback.
Domain, privacy, security, and risk partners define non-negotiable acceptance conditions and escalation paths where the use case requires them.
Go-to-market and customer-facing partners help identify adoption constraints and connect pilots to real workflows without substituting enthusiasm for evidence.

No single function can answer every release question. Product cannot declare model quality by itself. Engineering cannot infer customer value from technical performance. Governance cannot decide workflow desirability. Clarify who owns each decision, which evidence is required, and who can block a release when a non-negotiable condition fails.

Use forward deployed engineers where context is the constraint

Complex enterprise workflows often hide critical information in customer configurations, permissions, data conventions, and exception handling. A forward deployed engineer can work with product and design in the customer’s environment, shorten the path from observation to prototype, and turn real failures into reusable product knowledge. That field-facing role is especially useful when edge cases cannot be reproduced from a conference-room description.

The role needs a productization boundary. Otherwise, the team may prove that a skilled engineer can manually make each account successful rather than proving that the product is repeatable. Require every customer-specific exception to become one of four things: a reusable product requirement, an explicit configuration option, an evaluation scenario, or a documented unsupported condition. Field learning, end-to-end instrumentation, and responsible guardrails should strengthen the product system rather than disappear into account-specific work.

Centralize the paved road, not product judgment

A shared AI or platform capability should make the safe path easier. Depending on the organization, that can include approved model access, data controls, common logging, an evaluation framework, prompt and model versioning, reusable interaction patterns, cost and latency visibility, incident procedures, and privacy or safety templates.

The embedded product team should still own the opportunity, supported workflow, acceptance criteria, scenario relevance, product experience, pilot design, and outcome decision. A central group cannot judge those elements without the local customer context. It should provide leverage and enforce genuine non-negotiables, not become an approval queue for every prompt experiment.

Governance works best as a quality system with explicit rules:

Which use cases require review before customer exposure.
Which data classes, actions, and model behaviors are prohibited.
Which evidence must accompany a request to expand authority or reach.
Who approves exceptions and who owns the resulting risk.
What monitoring, audit, escalation, and rollback controls must exist after release.

This structure gives teams room to experiment inside known boundaries. It also prevents a common late-stage failure in which privacy, safety, or operational requirements appear only after the team has committed to an architecture and promised a launch.

Replace demo reviews with evidence reviews

A portfolio review should inspect what the team has learned, not how polished the prototype looks. Ask:

Which decision changed since the previous review, and what evidence changed it?
Which customer story and workflow moment support this opportunity?
What is the highest-consequence failure the current prototype still exhibits?
What product outcome must improve, and which quality condition must not regress?
What is the narrowest safe and coherent release that can produce field evidence?
Who owns the capability when its model, prompt, data, or tool behavior changes?

The answers reveal the right next move. No rich customer evidence means the team needs better discovery, not a better prompt. No scenario set means model comparisons are premature. Strong offline performance with weak field adoption points to workflow or value friction. A pilot that depends on hidden manual intervention is not yet a repeatable product. A capability without monitoring and rollback is not ready for production authority.

Take one active GenAI bet and replace its feature pitch with a falsifiable bet brief, a traceable evidence packet, and an explicit quality-and-value gate. Run the next portfolio conversation against those artifacts. Anything important that is missing is not paperwork to add later; it is the discovery work that should happen before the organization makes a larger commitment.

Tag: eval-driven development

A Practical Operating Model for GenAI Product Discovery