Your AI feature is live, people are using it, and the model passes its quality checks. Then finance asks a reasonable question: What did the investment return? If your answer is adoption, accuracy, or a handful of customer comments, you do not yet have evidence of ROI.

A defensible answer connects model behavior to a completed customer task, that task to a business consequence, and the consequence to the full incremental cost of delivering it. Build that chain and you can decide whether to improve the product, expand it, narrow its target market, or stop investing.

Start with the decision your measurement must support

AI measurement often starts with whatever is easiest to retrieve: response volume, latency, token consumption, thumbs-up ratings, or a model-quality score. Those metrics can help diagnose the system, but they cannot tell you whether the product created enough value to justify its cost.

Start by writing a short measurement contract. It should answer:

• User and context: Who is the product for, and in what situation will they use it?
• Intended task: What job should the user complete with the AI?
• Counterfactual: What would the user do without this capability?
• Successful outcome: What observable event proves that the task was completed acceptably?
• Business mechanism: How should that successful task affect revenue, retention, capacity, service cost, risk, or another business result?
• Decision rule: What evidence would cause you to scale, revise, narrow, or stop the product?

This prevents a common form of metric theater: collecting detailed telemetry without defining what decision the telemetry is supposed to change.

Good output is not universally good output. It is output that fits a particular user, task, and moment. An expert asking for a quick check and a beginner trying to learn the same subject may need very different responses; even a quality such as concision changes with context. A useful evaluation therefore identifies the intended user and task instead of grading isolated responses against a generic preference.

Your measurement system needs distinct layers for technical performance, customer value, and business value:

Layer	Question	Useful evidence	What it cannot prove alone
Technical quality	Did the system behave as designed?	Correctness, groundedness, tool completion, latency, stability, policy compliance	That the user completed a valuable task
Customer outcome	Did the user accomplish the intended job?	Task completion, acceptance without material correction, reduced effort, appropriate escalation	That the outcome generated an economic return
Business outcome	Did successful use improve a result the business values?	Incremental retention, revenue, capacity, service cost, or another defined outcome	That the AI caused the change unless a credible baseline exists

Treat these layers as a causal chain, not a menu of interchangeable KPIs. Technical quality can improve while customers remain unable to finish the task. Customers can finish the task while the economics remain unattractive. A green model dashboard does not compensate for a broken link farther down the chain.

Turn the customer outcome into a context-specific eval suite

An eval is useful when it predicts whether your product will succeed in its intended setting. That requires more than sending generic prompts to a model and assigning a quality score.

1. Build scenarios around real jobs. Cover the user intents, input conditions, task complexity, and failure modes the product is expected to handle. Do not let polished demonstration prompts dominate the set.
2. State the expected outcome. For each scenario, describe what the user must be able to do after receiving the output. This is more useful than asking whether the response merely sounds good.
3. Write observable grading criteria. Replace vague labels such as helpful, clear, or concise with evidence an evaluator can identify. Specify required information, acceptable omissions, allowed uncertainty, and the action the response should support.
4. Separate critical failures from ordinary quality variation. A response can be fluent and still fail because it invents evidence, takes the wrong action, omits a required constraint, or should have escalated to a person.
5. Calibrate the evaluator. If people label examples, align them on the rubric. If an LLM judges outputs, audit its ratings against human judgments and review disagreements. The judge is another measurement instrument, not ground truth.
6. Connect eval results to production outcomes. Test whether scenarios that score well actually lead to successful tasks after release. If they do not, revise the eval rather than defending the score.

A practical rubric usually needs to distinguish several questions: Was the response factually or procedurally acceptable? Did it include the information required for the task? Was it appropriate for this user’s expertise and intent? Did it take or recommend the correct next action? Did it avoid a failure serious enough to invalidate the result?

Do not average away a critical failure. A high overall score can conceal a small set of scenarios in which the product behaves unacceptably. Track those scenarios as explicit guardrails and make them capable of blocking a release, even when the aggregate score improves.

Keep a stable regression set so prompt, model, retrieval, and tool changes can be compared against the same tasks. Add new cases when production reveals a missing behavior, but record why each case entered the suite. Otherwise the benchmark will drift without anyone knowing whether the product improved or the test became easier.

Also inspect the workflow before automating it. AI can produce more of an ineffective process just as easily as it can improve a valuable one. If a step does not contribute to the customer’s outcome, remove or redesign it before teaching an agent to perform it faster.

Instrument the path from AI use to unit economics

Usage is not the value event. A person may invoke an AI capability repeatedly because it works well, because the first response failed, or because the interface makes the correct action unclear. Retries can raise engagement while lowering customer value.

Instrument an event chain that preserves the difference:

1. Eligibility: The user encountered a situation the capability was designed to address.
2. Exposure: The user could see or access the AI capability.
3. Attempt: The user invoked it for a classified task or intent.
4. Outcome: The task succeeded, required correction, escalated, failed, or was abandoned.
5. Customer value: The user completed the larger workflow or reached the intended value event.
6. Business consequence: The successful workflow changed the chosen business outcome.
7. Cost: The company incurred build, operating, review, rework, support, and failure-related costs.

For an agent product, the core discipline is to connect agent usage to successful tasks, retained usage, and cost. Instrumenting eligibility and exposure matters as well: measuring only people who choose to invoke the feature introduces selection effects and hides everyone who could not discover, trust, or use it.

Use successful tasks as the economic denominator

The following definitions turn the event chain into decision metrics:

• Task success rate = successful eligible task attempts divided by all eligible task attempts.
• First-pass success rate = attempts accepted without material correction divided by eligible attempts.
• Escalation rate = tasks transferred to a person divided by eligible attempts. Interpret this against product intent; escalation can be the correct outcome when the AI reaches its boundary.
• Post-success retention = users who return within the product’s relevant usage cycle divided by users who previously completed the value event. Choose that cycle from the natural frequency of the job, not from a convenient reporting window.
• Cost per successful task = all attributable operating costs, including the cost of failed attempts, divided by successful tasks.
• Incremental benefit = the business value for the exposed population minus the value expected from a comparable population using the prior workflow.
• ROI = incremental benefit minus total incremental cost, divided by total incremental cost.

A response is usually the wrong unit for AI economics. The customer does not receive value because the system generated text or called a tool; value arrives when the intended job is completed. Cost per successful task exposes products that look inexpensive per inference but require repeated attempts, corrections, or human rescue.

Keep the cost ledger honest

Separate implementation costs from recurring costs, then separate fixed recurring costs from costs that rise with usage. Depending on the product, the ledger may include:

• Data preparation, workflow integration, product development, evaluation, and rollout work.
• Model inference, external APIs, tool calls, retrieval, storage, and observability.
• Human review, quality assurance, escalation, customer support, and incident handling.
• Rework caused by incorrect output, abandoned tasks, and failed automation.
• Ongoing maintenance when prompts, models, knowledge, tools, or policies change.

Do not quietly convert time saved into financial return. Saved time becomes economic benefit only through an explicit mechanism, such as additional productive capacity or an avoided expense. Report the operational outcome and the monetary conversion assumption separately so a reviewer can challenge one without discarding the other.

If the benefit cannot be translated into money credibly, do not manufacture an ROI percentage. Report cost per successful task alongside the measured customer and business outcomes. A bounded claim supported by evidence is more useful than financial precision built on an untested assumption.

Prove incrementality before you claim causality

An AI product can have high usage and satisfied users without causing the business result you observe. Strong users may adopt the feature first. A process improvement may arrive at the same time. Demand may change independently. You need a credible estimate of what would have happened without the product.

• Measure the existing workflow first. Establish task success, effort, downstream outcomes, and cost for the same eligible population before replacing the process.
• Randomize access when feasible. An A/B test gives you a cleaner counterfactual when users can be assigned safely and the groups experience the same surrounding conditions.
• Use a staged comparison when randomization is not feasible. A phased rollout, matched cohort, or within-user comparison may still be useful, but document the ways the groups could differ and treat the causal claim as weaker.
• Define outcomes before inspecting results. Name the primary customer outcome, business outcome, cost measure, and guardrails in advance. This reduces the temptation to promote whichever metric moved favorably.
• Observe the complete value cycle. Do not declare success before the downstream event in your measurement contract had a reasonable chance to occur.
• Segment by context. Inspect the user types, task types, and complexity bands named in the eval suite. An attractive average can hide a segment that receives no value or creates disproportionate cost.

An experiment cannot rescue the wrong outcome metric. If you test whether the AI raises message volume when the customer actually needs a resolved case, you may obtain a statistically credible answer to an irrelevant question.

Compare offline quality and production outcomes together. The pattern tells you where to investigate:

Eval result	Production outcome	Likely interpretation	Next action
Strong	Strong	The eval appears aligned with customer value	Validate economics and guardrails before expanding
Strong	Weak	The rubric, target task, workflow, or adoption path may be wrong	Inspect failed journeys and redefine the value event
Weak	Strong	The eval may penalize behavior customers find useful	Review the rubric and production examples
Weak	Weak	The product is not delivering the intended outcome	Fix the capability or stop investing in that use case

Set the investment decision before the dashboard becomes politically important. Scale when incremental customer value produces an acceptable business consequence after full costs and guardrails remain sound. Iterate when users succeed but the economics or workflow are weak. Narrow the product when value is concentrated in a specific segment. Stop when the chosen use case does not improve on its counterfactual and there is no credible path to changing that result.

Do not borrow a universal ROI threshold. The acceptable return depends on your economics, risk, strategic intent, and alternatives. A severe guardrail failure can also override a positive average return; expected value does not make an unacceptable failure mode safe.

Key takeaways

• Begin with a specific user, task, counterfactual, value event, business mechanism, and decision rule.
• Evaluate AI outputs in context. Generic quality labels cannot define what is appropriate for every user and job.
• Keep technical quality, customer outcomes, and business outcomes separate, then test the causal links between them.
• Measure task success rather than raw usage, and calculate cost per successful task rather than cost per response.
• Include failed attempts, human review, rework, support, and maintenance in the cost ledger.
• Use a credible comparison to estimate incrementality, and state clearly when the comparison supports association rather than causation.
• Claim monetary ROI only when the conversion from operational improvement to financial benefit is explicit and defensible.

At your next product review, choose one AI workflow and replace its adoption slide with the measurement contract. Name the successful task, instrument its result and full cost, and compare it with the existing workflow. Do not expand the claim until every link from model behavior to business value can be inspected and defended.

References

• Product Talk — Defining What Good Looks Likes: All Things Product Podcast with Teresa Torres and Petra Wille
• Pendo — How Product Builders Can Prove AI ROI in Practice

---