Tag: A/B testing

You have an AI capability that demos well, yet its growth story is still unclear. Some users try it once. The team debates model quality. The roadmap fills with features, while the link to activation, retention, or revenue remains an assumption.

You can fix that by managing AI growth as a measurable path from user intent to trusted value, repeated behavior, and a business outcome. That path tells you where growth is breaking, which experiment to run next, and whether a more capable model would solve the problem at all.

Build the growth thesis as a measurable chain

AI products invite feature-shaped goals: launch a copilot, add an agent, improve the prompt, or introduce recommendations. Those goals describe output. They do not tell you whose behavior should change or why the change matters to the business.

In my product strategy work at HighLevel, I use a simple test: if a roadmap item cannot name the user behavior it should change and the business lever that behavior affects, it is not yet a growth strategy. A North Star and its driver tree force that connection into the open.

Build your driver tree from right to left. Start with the business outcome, identify the customer behavior that can produce it, and then identify the AI-assisted moments that can change that behavior. This order prevents model capabilities from dictating the roadmap.

1. Name the segment. Choose a group with a shared job and context. New administrators setting up an account are more useful than all users because their intent, constraints, and success event can be observed.
2. Define the value moment. State what the user can do after the AI interaction that was difficult before it. An answer displayed is not a value moment. A configured workflow, resolved issue, completed analysis, or approved action can be.
3. Select the behavior change. Decide whether you need more users to reach first value, reach it sooner, repeat a valuable workflow, or adopt an additional capability.
4. Connect the behavior to one growth mechanism. Activation, retention, and expansion require different product decisions. Choose a primary mechanism for the bet instead of claiming that one feature will improve all three.
5. Add quality and trust guardrails. Relevance, correctness, abandonment, corrections, unauthorized actions, privacy exposure, and recoverability can invalidate an apparent growth win.

A practical AI growth equation is: eligible users multiplied by discovery, first successful use, repeat successful use, and downstream conversion. You do not need to treat the equation as a financial model. Use it to locate the weakest link. More traffic will not repair poor first-use success, and better answers will not create growth if eligible users never discover the capability.

Turn the thesis into a one-page decision document before adding projects to the roadmap. It should contain:

• The target segment and the high-value job it is trying to complete.
• The current friction, supported by behavioral evidence or customer discovery.
• The proposed AI intervention and why AI is necessary for this step.
• The primary behavioral outcome and its baseline.
• The activation, retention, or expansion lever that outcome should affect.
• The leading indicators that can move before the business outcome does.
• The quality, reliability, and trust guardrails that must not deteriorate.
• The assumptions that would cause you to stop, narrow, or redesign the bet.

Your outcome statement can follow this form: increase a named behavior for a named segment by improving a specific driver, without degrading named guardrails. Supply the target only after you have a baseline and know what change your measurement system can detect. A target chosen for presentation value is not a strategy.

A worked example: turn AI search into an activation path

Consider a SaaS administrator who searches for help while configuring a workflow. A search team could optimize result clicks and declare success. A growth team traces the job further: query submitted, useful guidance received, setup started, workflow activated, and the workflow used successfully.

If result clicks rise but completed setups do not, the team improved engagement with search rather than activation. If setup completion rises but repeat use does not, the next constraint may be workflow value, onboarding, or the quality of the initial configuration. The query-to-outcome path makes those distinctions visible.

This is why every AI growth bet needs an explicit endpoint. The endpoint is not the response. It is the valuable behavior the response enables.

Choose an intent wedge before choosing the AI experience

A broad assistant usually creates a broad measurement problem. It serves unrelated intents, carries different failure costs, and leaves the team unable to explain why adoption changed. Start with an intent wedge: a narrow set of related requests from one segment, encountered at a meaningful point in its journey.

A strong first wedge has several useful properties:

• The job recurs. Repetition gives the product a chance to create a habit or reduce recurring friction.
• The current path is observable. You can see where users search, abandon, ask for help, switch tools, or fail to complete the workflow.
• Success is verifiable. The product can observe a completed action or downstream outcome instead of relying only on a positive reaction to the answer.
• The job is close to value. Improving it can plausibly affect activation, retention, or expansion.
• The failure is recoverable. Early versions should avoid irreversible or high-cost autonomy when a suggestion, preview, or confirmation can solve the same problem safely.
• The scope is evaluable. The team can assemble representative intents and define what an acceptable response or action looks like.

Use continuous discovery and journey mapping to find that wedge. Review behavioral funnels and query logs, then speak with users who completed the job, abandoned it, and avoided the AI experience entirely. The last group matters because usage data cannot explain a discovery problem among people who never entered the funnel.

Capture each candidate in an opportunity card. Record the segment, trigger, intent in the user’s own language, current workaround, failure consequence, next valuable action, available evidence, trust constraints, and outcome metric. This keeps prioritization centered on customer work rather than the novelty of a model capability.

When comparing opportunities, do not collapse everything into one unexplained score. Look separately at the strength of the evidence, proximity to a growth outcome, frequency of the job, severity of the friction, ability to measure success, and cost of a wrong answer or action. A high-frequency request with no meaningful downstream behavior may be less valuable than a narrower request sitting directly before activation.

Match autonomy to the evidence you have

AI product teams often jump from static software to autonomous agents in one roadmap step. A safer growth path increases autonomy only when the preceding level has demonstrated reliable value.

1. Visibility. Capture and classify what users are trying to accomplish. This exposes unmet demand before you automate anything.
2. Retrieval and explanation. Return relevant, grounded information that helps the user make the decision. A retrieval-first approach is often the cleanest starting point because the evidence and failure points are easier to inspect.
3. Recommendation. Suggest a next action using the user’s context, while keeping the decision with the user.
4. Guided or agentic execution. Prepare or perform a multi-step workflow with appropriate permissions, confirmation, observability, and recovery.

Move up a level when the current experience has repeat use, its major failure classes are understood, and the next level removes a documented point of friction. Do not add agency merely because the model can call tools. An agent that takes the wrong action creates a more serious problem than a search result that fails to earn a click.

The decision rule is straightforward: use the least autonomous experience that can produce the target behavior. This makes learning cheaper, limits risk, and shows whether users want the job completed before you invest in completing it on their behalf.

Instrument the full path from interaction to revenue

You cannot manage AI growth from a usage count. Monthly users of an AI feature can rise because of novelty, forced exposure, or repeated failure. Instrument a closed loop that connects intent, system behavior, user response, task completion, and the relevant business outcome.

A useful event spine contains the following stages:

1. Eligibility and exposure: Was the right user able to discover the capability at the right moment?
2. Intent: What job was the user trying to complete, and how was that intent classified?
3. Response: Which result, recommendation, or planned action did the system produce?
4. User judgment: Did the user select, accept, edit, reject, retry, or abandon it?
5. Execution: Did the user or agent start and complete the intended workflow?
6. Value: Did the product observe the success event defined in the growth thesis?
7. Business outcome: Did the relevant account activate, retain, expand, or contribute to Net Recurring Revenue through the defined path?

Concrete event names make implementation reviews easier. Depending on the experience, you might use events such as ai_query_submitted, ai_answer_shown, ai_recommendation_accepted, ai_action_started, ai_action_completed, ai_answer_corrected, and ai_flow_abandoned. The exact naming convention matters less than preserving the sequence and using it consistently.

Attach the properties needed to diagnose changes: segment, intent class, entry point, answer type, retrieval or ranking version, model and prompt version, experiment assignment, content identifiers, action type, completion status, and failure class. Carry account and customer identifiers into downstream systems only under your approved privacy and data-governance rules.

Raw prompts and conversations can contain personal, confidential, or commercially sensitive information. Logging them by default can create exposure that outlives the experiment. Define redaction, retention, access control, and deletion rules before broad collection. If a diagnostic goal can be met with a classified intent or structured error code, do not retain the raw text merely because it may be useful later.

Organize the resulting metrics into five views:

• Reach: eligible users, exposure, discovery, and first use.
• Experience: acceptance, correction, retry, abandonment, and progression to the next step.
• Task value: successful workflow completion and time to the defined value event.
• Repeat value: return use for the same job and successful use across relevant workflows.
• Business impact: activation, retention, expansion, and revenue outcomes for the target segment.

Sentiment can help you locate frustration, but it should not become the success metric. A polite response can still be wrong, and a frustrated user can still complete the task. Pair inferred sentiment with observable behavior such as correction, abandonment, repeated queries, completion, and downstream product use.

Revenue attribution needs similar discipline. Connect experiment exposure and product behavior to the CRM or revenue system, choose an attribution window that matches the natural decision cycle, and distinguish influenced revenue from causally demonstrated lift. Users who voluntarily adopt an AI capability may already be more engaged, so a dashboard correlation does not prove that AI caused their retention or expansion.

This distinction changes roadmap decisions. Behavioral analytics can reveal where the path is breaking. Controlled experiments are needed when you want to know whether fixing that point changes behavior or dollars.

Turn evaluation and experiments into a delivery system

AI growth execution needs two evidence gates. Offline evaluation asks whether the system performs the intended task well enough to expose safely. Online experimentation asks whether the experience changes customer behavior. Passing one gate does not imply that you will pass the other.

Gate releases with eval-driven development

Build the first evaluation set from real intents in the chosen wedge. Cover common requests, ambiguous requests, known failures, and cases where a wrong answer or action carries a higher cost. Preserve segment and intent labels so an average score cannot hide a severe failure in an important slice.

1. Write the rubric before tuning. Define what must be true for the response or action to pass: correct intent, relevant evidence, accurate guidance, appropriate next step, permitted action, and recoverability where needed.
2. Separate failure classes. Coverage, retrieval, generation, interaction design, tool execution, permissions, and policy failures need different fixes.
3. Version the system. Record the model, prompt, retrieval configuration, content version, tools, and policy configuration associated with each result.
4. Review performance by slice. Inspect high-value intents and high-consequence failures instead of relying only on the aggregate.
5. Keep human review where judgment is material. Automated scoring can accelerate evaluation, but uncertain or consequential cases still need an accountable review path.
6. Promote real failures into the eval set. Production corrections and abandoned workflows should make future regressions easier to catch.

Do not treat every low score as a prompt problem. If the required information is absent, fix content or data coverage. If the right material exists but is not retrieved, fix retrieval or ranking. If the answer is accurate but users cannot act on it, fix interaction design or the handoff into the workflow. Prompt iteration cannot compensate for every layer of the system.

Use online experiments to answer growth questions

Once the experience passes its release gate, write an experiment card that another product leader could audit without attending the planning meeting.

• The target segment and eligibility rule.
• The behavior you expect to change and the mechanism behind that change.
• The control and variant, including model, prompt, ranking, content, or interaction differences.
• One primary behavioral outcome tied to the growth thesis.
• Quality, reliability, cost, privacy, and business guardrails relevant to the change.
• The randomization unit, especially when users within the same account can affect one another.
• The minimum detectable effect and the sample required to evaluate it.
• The natural usage or buying cycle the test must observe.
• The decision rule for shipping, iterating, narrowing, or stopping.

Good early tests isolate a decision. Compare retrieval or ranking approaches when users cannot find the right information. Compare concise and detailed answer formats when comprehension or action is the constraint. Test prompt variants when the failure is genuinely in instruction following or response construction. Test a guided workflow against a static answer when users understand the answer but fail at the next step.

A click is an acceptable primary metric only when the click itself represents value. Otherwise, measure the completed behavior downstream. A compelling answer that produces no useful action is engagement without growth.

Run the first 90 days around evidence, not launch theater

A 30-60-90 operating sequence gives the team enough structure to create momentum while preserving room to learn.

1. Days 1-30: establish the truth. Select the segment and intent wedge. Baseline the driver tree. Map the current journey. Audit events and data access. Build the initial evaluation set. Define privacy and permission constraints. Write the first growth thesis and identify the assumptions most likely to break it.
2. Days 31-60: ship the smallest complete path. Release a thin experience behind a feature flag. Instrument the full event spine. Run offline evaluations and the first controlled experiment. Review failed intents and abandoned workflows every week. Fix the largest diagnosable constraint rather than adding adjacent features.
3. Days 61-90: prove, prune, and scale. Aim to land two or three measurable wins, remove low-signal bets, and decide whether the wedge deserves more distribution, deeper personalization, or greater autonomy. Standardize the operating cadence only after the team has learned which reviews lead to decisions.

The product trio should co-own problem framing and solution shaping. Product owns the growth thesis and trade-offs. Design owns comprehension, control, feedback, and recovery in the interaction. Engineering owns system behavior, instrumentation, reliability, and safe delivery. Data science should help design evaluations, experiments, and attribution. Customer-facing teams should validate whether the job and value proposition match the language customers actually use.

Use a cadence that matches the decision:

• Weekly: inspect intent coverage, failure classes, corrections, abandonment, and unexpected trust issues.
• Every two-week sprint: ship a testable improvement, review the evidence, and update the decision record.
• Monthly: review the driver tree, experiment portfolio, business impact, costs, and cross-functional blockers.
• Quarterly: reset outcomes, stop bets that have lost their evidence, and fund the next constraint in the growth path.

Feature flags, CI/CD, and observability are growth infrastructure because they reduce the cost and risk of learning. They let you separate code deployment from customer exposure, compare variants, detect regressions, and reverse a problematic release. Privacy-by-design, data governance, and observability should be release requirements rather than work deferred until scale.

Watch for five failure modes

• The roadmap is organized by AI features. Reorganize it around segments, intents, and outcomes so multiple solution types can compete for the same problem.
• One quality score hides the system. Break performance into coverage, retrieval, generation, interaction, execution, and policy slices so the owner of the next fix is clear.
• The team optimizes prompts while the funnel is broken elsewhere. Locate the failing step before choosing the technical intervention.
• Autonomy arrives before trust. Start with visibility, retrieval, or recommendations, then increase agency when reliable task completion and recovery have been demonstrated.
• AI usage becomes a vanity metric. Keep the primary outcome downstream of the interaction and tie it to activation, retention, or expansion.

Key takeaways

• An AI growth strategy must connect a defined segment and intent to a valuable behavior and one primary business lever.
• Start with a narrow intent wedge whose success is observable, repeatable, and close to activation, retention, or expansion.
• Use the least autonomous experience that solves the documented friction, then earn the right to add agency.
• Instrument eligibility, intent, response, user judgment, execution, value, and business outcome as one path.
• Use offline evaluations to manage quality and controlled experiments to establish behavioral or revenue impact.
• Treat feature flags, observability, privacy, and data governance as part of the growth system.
• Review failures weekly, ship testable improvements in two-week sprints, and prune bets that do not change customer behavior.

Start with one segment, one recurring intent, and one outcome. Trace the current path event by event, identify its weakest link, and write the smallest experiment that can test your explanation. That is enough to turn AI growth from a feature campaign into a learning system.

References

• Shivam.Consulting Blog – Make AI Search Count: Convert Every Query into Revenue with Visibility, Sentiment, and Action
• Shivam.Consulting Blog – Principal Product Manager Playbook: Strategy, Leadership, and Execution That Scales