Category: Product Management

  • Your Ultimate ProductCon San Francisco 2025 Guide: Best Hotels, Eats & Drinks

    Your Ultimate ProductCon San Francisco 2025 Guide: Best Hotels, Eats & Drinks

    Heading to ProductCon San Francisco 2025? I approach conference travel the same way I approach product strategy: optimize for outcomes, reduce friction, and invest in high-signal experiences. Here’s the playbook I use to choose the right hotel, find memorable meals, and make the most of every hour in the city.

    For lodging, I prioritize walkability, safety, and quiet rooms so I can focus during sessions and recover at night. If you want to be steps from most venues and meetups, SoMa and the Yerba Buena corridor are ideal. InterContinental San Francisco, W San Francisco, and The Clancy (Autograph Collection) are reliable, business-friendly picks with strong Wi‑Fi and ample lobby space for impromptu one‑on‑ones. If you prefer classic energy and transit access, Union Square hotels like Hotel Nikko and The Westin St. Francis work well. For waterfront views and a calmer vibe, Hyatt Regency Embarcadero puts you by the Ferry Building with easy BART and Muni access.

    My booking checklist is simple: reserve early, target a high floor away from elevators, and request early check‑in or late checkout around your session schedule. Loyalty programs often unlock better rates and quiet‑room preferences. If you need heads‑down time between talks, ask about day‑use meeting rooms or find a corner of the lobby with stable bandwidth. I also pack a compact power strip and a long USB‑C cable—two small upgrades that routinely save a day.

    Coffee is the fuel of great product conversations. Near SoMa, I rotate between Blue Bottle (Mint Plaza), Sightglass (7th Street), and Philz (Front Street) for pre‑session caffeine and quick stand‑ups. If I’m on the Embarcadero side, the Ferry Building’s roasters are perfect for early starts, and morning lines move faster than you’d expect if you arrive just after opening.

    For efficient lunches, I favor fast‑casual spots that can handle volume without sacrificing quality. Mixt, Souvla, Sweetgreen, Super Duper Burgers, and The Grove are dependable within a short walk of most downtown venues. When I need a higher‑signal lunch with a partner or prospect, I book a table slightly off the main corridor to avoid the rush—think Mourad for elevated Moroccan in SoMa or Boulevard along the Embarcadero for a polished, quiet conversation.

    Dinner is where the best networking often happens, so I plan for atmosphere, acoustics, and a menu that works for mixed dietary needs. Kokkari Estiatorio (FiDi) excels for executive dinners. Liholiho Yacht Club is a creative, memorable choice for cross‑functional teams. Waterbar or Angler near the waterfront pair great food with views that impress visiting colleagues. For something more casual but still conversation‑friendly, Nopa or Sorella deliver consistently.

    When it’s time for drinks, I think in terms of groups and goals. For panoramic views and small group catch‑ups, The View Lounge (Marriott Marquis) is a classic. For wine‑forward conversations with a quiet ambiance, Press Club near Yerba Buena works well. If you’re hosting a more energetic crew, Charmaine’s (SF Proper Hotel), Dirty Habit (Hotel Zelos), or 25 Lusk offer space, good music, and reliable service. For craft cocktails, Pacific Cocktail Haven and ABV are standouts if you don’t mind a short ride.

    Transit and timing matter. From SFO or OAK, BART is often the fastest, most predictable route downtown; rideshare is convenient late at night. I walk whenever possible, but I time routes along well‑lit, busier streets and avoid sprinting between neighborhoods tight on time. Microclimates are real—bring layers, comfortable shoes, and a compact umbrella. I schedule 15‑minute buffers around key sessions to handle inevitable friend‑of‑a‑friend introductions.

    If you need a professional setting for a quick working session, many hotels will extend lobby seating to guests and their visitors. For dedicated space, day passes at coworking operators like Industrious, CANOPY, or Regus are worth it when you’ve got a client briefing or board prep. For a more casual backdrop, Sightglass and Blue Bottle locations typically have reliable Wi‑Fi and just enough outlets if you arrive off‑peak.

    Finally, a word on intent: I set a simple goal for each day—one meaningful connection, one surprising insight, and one concrete action to bring back to my team. ProductCon San Francisco 2025 is a catalyst if you design your experience with the same rigor you apply to your roadmap. If you spot me in a session or at a nearby cafe, say hello—I’m always up for trading notes on product strategy, pricing experiments, and what’s working in the field right now.

    Quick note: restaurants and hours can change quickly—make reservations where possible and double‑check opening times the week of the event.


    Inspired by this post on Product School.


    Book a consult png image
  • How to Build an Outcome-Driven Product Operating Model

    How to Build an Outcome-Driven Product Operating Model

    You have rewritten the roadmap as OKRs, asked teams to focus on outcomes, and changed the titles in the quarterly review. Yet feature requests still arrive as commitments, teams still need approval to change a solution, and leaders still celebrate launches more than customer behavior. The language changed. The operating model did not.

    An outcome-driven product operating model changes who owns the problem, what leaders fund, how teams make decisions, and what evidence can alter the plan. If you are leading that transition, the practical test is simple: can each product team name the behavior it is trying to change, its current baseline, the business result that behavior should influence, its guardrails, and the decisions it can make without escalation?

    Start with an outcome contract, not an outcome slogan

    An outcome-driven model needs more than an outcome-shaped sentence. It needs a clear contract between leadership and the team.

    Leadership defines the strategic direction, the customer or business result that matters, the constraints, and the boundaries of acceptable risk. The team owns discovery, solution choice, sequencing, and the experiments used to find a viable path. This division protects strategic alignment without turning leaders into backlog managers.

    The first source of confusion is usually vocabulary. Outputs are the things a team produces; outcomes are the changes those things are intended to create. A release, migration, redesigned workflow, or pricing page is an output. Activation, retention, conversion, satisfaction, cost, and risk are outcomes when they describe an observable change rather than a work item.

    ElementWhat it doesExample
    ObjectiveSets the direction and explains why it mattersHelp new customers reach value sooner
    OutcomeDescribes the behavior or result that should changeMore new accounts complete the first-value action
    MetricMeasures that changeActivation rate or time-to-first-value
    TargetDefines the desired movement and time horizonThe agreed improvement from the recorded baseline
    BetStates a possible way to create the outcomeGuided setup for the highest-friction step
    OutputNames what the team may build or changeAn in-app guide or revised onboarding flow

    Keeping these elements separate matters. If the objective says “launch onboarding v2,” the solution has already been chosen. Discovery can only validate the predetermined answer. If it says “improve activation,” but there is no segment, baseline, causal explanation, or guardrail, the team has freedom without usable direction.

    A strong outcome contract fits on one page and contains:

    • Target customer and problem: who is affected, where the friction appears, and why resolving it matters now.
    • Primary outcome: the single behavior or business result the team is expected to influence.
    • Baseline and target: the current measurement, desired movement, and decision horizon. If the baseline is unavailable, measurement is the first task rather than an assumption hidden in the plan.
    • Causal chain: the proposed connection from product change to customer behavior to business value.
    • Leading indicators: signals such as completion of a core action or time-to-first-value that can reveal movement before the lagging result is available.
    • Guardrails: measures that must not deteriorate, such as support demand, reliability, performance, satisfaction, privacy, or risk.
    • Constraints: non-negotiable regulatory, security, platform, brand, cost, or commercial boundaries.
    • Decision rights: what the team can decide, what requires consultation, and what requires leadership approval.
    • Evidence standard: what would justify continuing, changing, scaling, or stopping the bet.

    The causal chain is the part most teams skip. “Build a dashboard to improve retention” jumps directly from output to business result. Ask what the customer will do differently because the dashboard exists, why that behavior should affect retention, and which signal would appear first. If no credible behavior connects the feature to the result, the feature is not yet a defensible bet.

    Do not make the outcome so broad that no team can influence it. Company revenue, total churn, and overall customer satisfaction are often shared results shaped by pricing, sales, service, market conditions, and multiple product experiences. A team needs a customer behavior or operating result close enough to its work to guide daily choices, while still having a clear connection to the larger business outcome.

    This is also why outputs should not disappear from planning. Teams still need delivery plans, quality standards, dependencies, and technical milestones. The mistake is treating those items as proof of value. Outputs tell you what changed in the product. Outcomes tell you whether that change mattered.

    Give durable teams a problem and real decision rights

    You cannot hold a team accountable for an outcome while reserving every meaningful decision for someone else. Outcome ownership without authority is delegated blame.

    A durable team should own a customer problem or value area long enough to build context, observe behavior, test alternatives, and learn from the result. A stable product, design, and engineering partnership reduces the handoffs that appear when temporary project teams move from specification to design to implementation.

    Durability does not mean a team owns the same feature forever. It means the team retains responsibility for an outcome space even as its solution changes. An activation team might work on guidance, setup defaults, education, performance, or removing a step entirely. The outcome provides continuity; the outputs remain flexible.

    Make decision rights explicit at each level:

    • Executive leadership: chooses the strategic outcomes, sets material constraints, allocates investment across the portfolio, and resolves conflicts that cross organizational boundaries.
    • Product leadership: translates strategy into outcome spaces, defines evidence and review standards, protects coherent team boundaries, and makes portfolio trade-offs visible.
    • Product teams: investigate opportunities, choose solution hypotheses, decide how to test them, sequence delivery, and recommend whether a bet should continue.
    • Functional leaders: establish engineering, design, data, security, and product-management standards while developing the craft and capability of their people.
    • Stakeholders: contribute customer context, commercial needs, risks, deadlines, and operational knowledge. Their requests are important evidence, but they do not silently become roadmap commitments.

    The wording of the boundary matters. “The team is empowered unless a senior stakeholder disagrees” is not a decision rule. Specify which constraints are binding, who can override a team decision, what evidence an override requires, and who decides which existing commitment will move as a result.

    When a feature request arrives, use a short intake sequence:

    1. Restate the request as a customer problem, business risk, or desired behavior change.
    2. Identify the affected segment, current evidence, urgency, and consequence of doing nothing.
    3. Compare it with the outcomes already assigned to the team.
    4. If it fits, add it as an opportunity or solution hypothesis rather than an automatic commitment.
    5. If it displaces an existing priority, ask the portfolio owner to make that trade-off explicitly and record what is being delayed.

    This prevents the common pattern in which every request is individually reasonable but the combined roadmap is strategically incoherent.

    Enabling work needs equally clear ownership. Reliability, data quality, privacy, scalability, internal tooling, and platform capabilities may not produce an immediate customer behavior change, but they can make an outcome achievable or prevent it from becoming fragile. A Product Tree makes these roots visible alongside customer-facing branches and feature-level leaves.

    Do not force enabling work into a fictional revenue claim. State the operational capability it must improve, the downstream product outcomes it enables, and the risk of postponing it. That gives platform and infrastructure investments a testable rationale without pretending every technical change has a direct, isolated effect on growth.

    Manage a portfolio of bets instead of a feature queue

    A feature roadmap creates the appearance of certainty too early. It commits the organization to solutions before the most important assumptions have been tested. An outcome-driven roadmap still communicates direction and sequencing, but it treats solutions as bets that can earn more investment through evidence.

    Each roadmap item should answer four different questions:

    • Why this problem? The customer pain, strategic relevance, business consequence, and reason it deserves attention now.
    • What should change? The target behavior or result, baseline, leading indicators, and guardrails.
    • How might it change? The current solution hypothesis and the causal assumptions behind it.
    • What happens next? The evidence being gathered and the next continue, change, scale, or stop decision.

    This format changes the roadmap conversation. Stakeholders can challenge the importance of the problem, the logic of the bet, or the quality of the evidence without treating a proposed feature as an irreversible promise.

    Use a lightweight bet brief before substantial delivery begins. It should include:

    • The outcome contract and the strategic objective it supports.
    • The customer opportunity and evidence that the problem is real.
    • The causal chain from proposed change to behavior to business result.
    • The expected reach, frequency of exposure, and direction of behavior change.
    • The solution hypothesis and the riskiest assumptions within it.
    • Confidence, effort, dependencies, privacy implications, data requirements, and technical complexity.
    • The instrumentation, experiment, rollout, and guardrail plan.
    • The evidence that would change the decision.

    A one-page impact brief is usually enough. If a team cannot express the logic concisely, expanding the document will not repair the missing understanding.

    Prioritization frameworks can help compare bets, but they should expose judgment rather than replace it. Reach, impact, confidence, and effort are useful because they force assumptions into view. Cost of delay helps when timing matters. Neither method turns uncertain inputs into objective truth.

    Pressure-test the inputs before trusting the score:

    • Is reach based on actual eligible users or the entire customer base?
    • Does “impact” refer to a behavior that can be measured, or merely to stakeholder enthusiasm?
    • Is confidence supported by behavioral evidence, customer discovery, prior experiments, or only opinion?
    • Does effort include instrumentation, rollout, migration, enablement, support, and dependencies?
    • Would the bet still rank highly if its most optimistic assumption were reduced?

    The portfolio also needs balance. Some bets improve customer behavior directly. Others reduce material risk, strengthen a platform capability, or create the measurement needed to pursue later outcomes responsibly. Make those categories explicit so foundational work is not forced to compete through exaggerated short-term impact claims.

    Set stopping conditions before enthusiasm and sunk cost distort the decision. A stopping condition might be failure to observe the necessary leading behavior, inability to reach the intended segment, unacceptable movement in a guardrail, or evidence that the customer problem is less important than assumed. Stopping a weak bet is not a delivery failure. Continuing it without a credible causal path is.

    Make evidence change plans, funding, and reviews

    The model becomes real only when evidence can change what the organization does. If every bet continues regardless of results, experimentation is theater. If quarterly reviews still focus on release counts, teams will optimize for releases.

    Connect discovery, delivery, and measurement

    Discovery is not a phase that ends when development begins. It is the work of reducing uncertainty throughout the bet. The useful sequence is:

    1. Record the baseline. Confirm that the primary outcome and leading indicators can be measured for the relevant segment.
    2. Map the causal chain. Identify the customer behavior that must change before the business result can move.
    3. Test the riskiest assumption. Learn whether the problem, proposed value, usability, feasibility, or business logic is most uncertain.
    4. Ship the smallest meaningful change. Reduce the scope needed to create observable behavior, not merely the number of tickets in the release.
    5. Monitor leading and guardrail signals. Leading indicators may appear within days, while durable or lagging outcomes can require weeks to assess.
    6. Write the learning memo. Record what happened, what remains uncertain, and whether the evidence supports continuing, changing, scaling, or stopping.

    Instrumentation belongs in the bet, not in a cleanup backlog after launch. Define event names, eligibility rules, segments, exposure, dashboards, and metric ownership before the change reaches customers. Otherwise, the team may ship on time and still be unable to answer whether the intended behavior occurred.

    Match the evidence method to the decision

    Use an A/B test when you need causal confidence and can create valid comparison groups. Set the minimum detectable effect before the test so the team knows whether the available population and duration can detect a change large enough to matter. A test that cannot resolve the decision is activity, not useful evidence.

    Not every change can be randomized. Sequential rollouts, pre-post comparisons, cohort analysis, and synthetic controls can still inform a decision, but their limitations should remain visible. Seasonality, selection effects, concurrent launches, and changes in traffic can produce movement that the product change did not cause. Label the conclusion with the strength of the evidence rather than presenting every dashboard shift as proof.

    Also distinguish a negative result from an inconclusive one. A well-powered test that shows the necessary behavior did not change challenges the hypothesis. A test with weak exposure, broken instrumentation, or insufficient sensitivity says much less. The next decision should reflect that difference.

    Replace status rituals with decision rituals

    Each operating cadence should answer a distinct question:

    • Strategy reviews: Are the chosen outcomes still the right expression of the strategy, given current customer and business evidence?
    • Team reviews: What did the team learn about the problem, causal chain, solution, and metrics, and what will it test next?
    • Portfolio reviews: Which bets deserve more investment, which need to change, and which should stop?
    • Quarterly business reviews: What customer and business results changed, what was learned, and how should allocation change? Releases provide context, not the score.

    A useful review page shows the baseline, current value, target, leading indicators, guardrails, confidence level, latest learning, and next decision. A release list without those fields is a delivery update, even if the slide is labeled “outcomes.”

    Incentives must support the same behavior. Teams should be accountable for the quality of their discovery, the integrity of measurement, the speed with which they resolve material uncertainty, and the decisions they make from evidence. Treating every missed outcome as individual failure encourages conservative targets, favorable metric selection, and reluctance to stop weak bets. Outcomes are influenced, not manufactured on command.

    Introduce the model through a real decision

    A company-wide reorganization is not the safest starting point. Begin with an important product area where the current feature plan contains meaningful uncertainty and leadership is willing to let evidence change the solution.

    1. Select one outcome and record its baseline, causal chain, leading indicators, and guardrails.
    2. Assign it to a durable product trio with written decision boundaries.
    3. Convert the planned initiative into a bet brief with assumptions and stopping conditions.
    4. Change the existing team and portfolio reviews so they require evidence and an explicit decision.
    5. At the end of the planning cycle, inspect where decisions still stalled: unclear strategy, missing data, dependency conflicts, weak skills, incentive mismatch, or executive overrides.
    6. Repair those operating constraints before expanding the model to more teams.

    Treat the operating model itself as a product. Its users are the teams and leaders making decisions. Its outcomes are clearer ownership, lower decision latency, stronger learning, and better allocation of effort. Changing an org chart without changing those behaviors is just another output.

    Key takeaways for your next planning cycle

    • An outcome must name an observable change, not disguise a feature as an OKR.
    • Pair every outcome with a baseline, causal chain, leading indicators, guardrails, constraints, and an evidence standard.
    • Give durable teams authority over discovery and solution choices within explicit strategic and risk boundaries.
    • Manage solutions as bets that can earn, lose, or redirect investment as evidence changes.
    • Keep enabling work visible by naming the capability it improves, the outcomes it unlocks, and the risk of delay.
    • Review customer behavior, business movement, learning, and next decisions. Do not use delivery activity as a substitute for impact.

    At your next roadmap review, take the most expensive planned initiative and rewrite it as an outcome contract and bet brief. If the room cannot agree on the target behavior, baseline, causal link, decision owner, and evidence that would stop the work, the initiative is not ready for a larger commitment. Resolve that uncertainty before adding more scope.

    References

  • Turn Customer Insight Into Messaging That Improves Retention

    Turn Customer Insight Into Messaging That Improves Retention

    Your activation dashboard is weak, support keeps hearing that onboarding is confusing, sales says the story is not landing, and customer success says buyers expected something different. Those can look like four separate problems. They are often four views of the same break between the value customers expect and the value they experience.

    You need a system that connects customer language, product behavior, messaging, and retention. The practical goal is not to collect more feedback or polish more copy. It is to identify an expectation gap, make the product promise more precise, help customers reach the promised outcome, and verify that the outcome lasts.

    Retention problems often begin as promise problems

    Customer insight, product messaging, and retention are usually managed in different rooms. Insight becomes an interview repository. Messaging becomes a launch asset. Retention becomes a dashboard reviewed after customers have already left. That separation hides the causal chain you need to manage.

    A customer arrives with an expectation created by your website, sales conversation, trial, or referral. The product either confirms that expectation or contradicts it. Onboarding determines how quickly the customer can test the promise. Repeated use determines whether the value is durable. Renewal and expansion reveal whether the value is commercially meaningful.

    This is why a messaging problem cannot always be fixed with copy. If the promise is accurate but the path to value is confusing, fix onboarding. If customers reach the advertised outcome once but have no reason to return, fix the recurring value loop. If the product consistently delivers something customers value but your message emphasizes a secondary feature, change the positioning. If the promised outcome is not delivered, the roadmap has to move.

    Start by locating the break in the customer journey. Use this as a diagnostic map, not as a universal scoring model:

    Journey stageEvidence to inspectMessaging questionLeading measureRetention measure
    OnboardingIncomplete steps, early exits, setup questions, and first-run sentimentIs the first promised outcome clear, and does the customer know the next action?Onboarding completion rateEarly cohort retention
    ActivationSetup completed without the behavior that represents first valueWhat observable event proves that the customer received the promised payoff?Activation rate and time-to-valueRetention among activated and non-activated cohorts
    AdoptionInitial success followed by narrow, irregular, or declining useWhich recurring job should bring the customer back?Feature adoption, session frequency, and appropriate stickinessLogo churn and gross revenue retention
    ExpansionRetained accounts asking for an adjacent outcome or broader useDoes the upgrade represent a natural next result, or merely more feature inventory?Adoption of expansion-related capabilitiesExpansion revenue and net revenue retention
    Churn riskDeclining usage, negative sentiment, unresolved tickets, contraction, or downgradesDid the product deliver the original promise to this segment?Customer health, tickets per account, and resolution timeContraction, gross revenue retention, and logo churn

    The most important distinction is between a message that is misunderstood and a promise that is unfulfilled. Both can depress activation, but they require different decisions. Ask what customers thought would happen, what actually happened, and which behavior would demonstrate that the gap has closed.

    Build a customer evidence map before changing the message

    Do not begin with a broad request to understand the customer better. Begin with a decision. For example: should you simplify first-run setup, change the activation message, reposition a capability, or invest in a missing part of the product? A bounded decision tells you which customers, signals, and time period matter.

    Customer sentiment becomes actionable when you connect qualitative feedback with usage, lifecycle, and commercial context. A complaint without behavioral context may be loud but isolated. A usage decline without customer language tells you what happened but not why. The evidence map joins the two.

    1. Select one cohort and one journey stage. Define the segment by a meaningful difference such as customer job, product tier, acquisition path, company profile, or activation status. Avoid blending customers who bought for different reasons.
    2. Define the unit of analysis. Decide whether retention is measured at the user, workspace, account, or revenue level. In a multi-user product, one active user does not necessarily mean the account is healthy.
    3. Join the evidence. Connect interviews, support conversations, reviews, in-app feedback, sales objections, usage events, lifecycle stage, CRM data, and revenue outcomes. Preserve the timestamp so you can tell whether feedback preceded or followed the behavior.
    4. Apply a stable taxonomy. Label the journey stage and a manageable theme such as usability, reliability, pricing, or time-to-value. Keep the original customer language beside the label so a summary never replaces the evidence.
    5. Write an insight as a testable claim. State the observed behavior, the customer language associated with it, your explanation, and the metric that should move if the explanation is correct.

    A useful insight statement has this shape: For [segment] at [journey stage], [observed behavior] occurs alongside [sentiment or recurring language]. Customers appear to expect [outcome] but encounter [barrier]. If that explanation is right, [product or messaging change] should move [leading indicator] and later improve [retention measure].

    The phrase “appear to” matters. Feedback is evidence, not proof of causation. Keep the explanation provisional until a product change, message test, or deeper investigation supports it.

    Read sentiment and behavior together

    Four common patterns lead to different actions:

    • Negative sentiment and failed behavior: customers describe a barrier and telemetry shows that they stop at the same point. This is a strong candidate for product discovery and a focused intervention.
    • Positive sentiment and weak behavior: customers may like the idea, the team, or an isolated capability without depending on the product. Check whether you defined the right value event and whether the expected usage cadence fits the job.
    • High usage and negative sentiment: the product may be useful while still imposing a reliability, usability, pricing, or support cost. Do not dismiss the complaints because engagement looks healthy; the account can still be vulnerable.
    • Positive sentiment and retained behavior: look for the specific outcome customers repeatedly mention and achieve. That combination can become a value pillar and a credible proof point.

    When sentiment and behavior converge, prioritization becomes easier. When they diverge, do not force a confident narrative. Check segmentation, event instrumentation, account-level aggregation, interview sampling, and the natural frequency of the customer’s job before you build.

    Use generative AI for compression, not judgment

    Generative AI can summarize call transcripts, cluster feedback, propose themes, and surface repeated phrases across a large corpus. That makes it useful for triage. It should not become an automatic roadmap-ranking system.

    Keep every generated theme traceable to the underlying records. Sample raw conversations from each important cluster, inspect false classifications, and separate customer wording from model-generated interpretation. Version the taxonomy and prompt when you change them; otherwise a movement in sentiment may reflect a classification change rather than a customer change.

    Apply privacy-by-design and data governance before sending support, CRM, or interview data into a model. Limit access, remove information that is not needed for the decision, and retain provenance. The output should help a product leader find evidence faster, not obscure where a conclusion came from.

    Turn evidence into a promise the whole journey can keep

    A product messaging framework connects the customer, problem, outcome, differentiation, and proof. Its value is operational. Product, design, sales, marketing, support, and customer success can make different artifacts without making different promises.

    For each important customer job, create a value-pillar card with the following fields:

    • Segment: the customer for whom the promise is relevant.
    • Job or problem: the progress the customer is trying to make, in the customer’s language.
    • Outcome: what becomes better when the product works.
    • Mechanism: how the product enables the outcome.
    • Point of parity: the expected capability that establishes category credibility.
    • Differentiation: the meaningful reason to choose this approach over an alternative.
    • Proof: a customer quotation, observed behavior, product demonstration, or supported performance claim.
    • Objection or boundary: where the promise does not apply, what must be true for it to work, and which objection needs an honest response.
    • Success event: the observable behavior showing that the customer reached value.
    • Retention signal: the repeat behavior or commercial outcome that indicates durable value.

    You can compress that card into a working message: For [segment] trying to [job], [product or capability] enables [outcome] through [mechanism]. It meets the category expectation of [parity], differs through [meaningful distinction], and is credible because [proof].

    Do not publish the formula as copy. Use it to expose weak thinking. If the segment is “everyone,” the message is diluted. If the outcome is a feature, the customer value is missing. If the differentiation does not affect the customer’s choice or result, it is decoration. If the proof field is empty, the claim is not ready.

    Carry one promise through different customer moments

    Consistency does not mean repeating the same sentence everywhere. It means preserving the same value logic while giving the customer the information needed at each moment.

    • Company level: define the broad change you exist to create.
    • Product level: explain how the product delivers its part of that change.
    • Segment level: select the job, obstacle, and proof most relevant to a particular customer.
    • Feature level: connect a capability to the outcome it supports instead of announcing functionality in isolation.
    • Acquisition and evaluation: set an accurate expectation, establish the category basics, show differentiation, and provide evidence.
    • Onboarding: restate the outcome the customer chose, identify the first meaningful success, and remove actions that do not help reach it.
    • Activation: make success visible when it occurs, then point to the next behavior that turns first value into repeat value.
    • Adoption: introduce adjacent capabilities when they support the customer’s next job, not simply because they are underused.
    • Renewal and expansion: refer to value the account has actually realized. Position expansion around the next credible outcome rather than a larger bundle alone.
    • Support: use the same names, outcomes, and boundaries as the product and sales experience. Conflicting terminology creates avoidable uncertainty.

    Give the same completed value-pillar card to a salesperson preparing a talk track, a product manager writing a release note, and a designer writing an in-app prompt. The artifacts should differ, but the promised outcome, mechanism, and proof should agree. If they do not, the framework is not yet clear enough to operate.

    Measure whether clearer messaging produces retained value

    A message can increase attention without improving value. That is why click-through rate or onboarding completion cannot be the final success measure. Pair every message or journey experiment with a leading behavioral indicator and a downstream retention indicator.

    Use a written experiment brief before changing the experience:

    • Cohort: who will see the change, and who will not.
    • Journey stage: where the expectation gap appears.
    • Evidence: the behavior and customer language supporting the hypothesis.
    • Change: the product, message, or combined intervention being tested.
    • Leading measure: activation, time-to-value, onboarding completion, feature adoption, or another behavior close to the intervention.
    • Retention measure: cohort retention, logo churn, gross revenue retention, net revenue retention, contraction, or expansion.
    • Guardrails: signals such as support demand, negative sentiment, downgrades, or reliability issues that should not worsen.
    • Minimum detectable effect: the smallest change the test is designed to distinguish, set before results are reviewed.

    A complete retention view combines activation, adoption, customer experience, cohort, and revenue signals. Each one answers a different question:

    • Activation rate asks whether eligible customers reached the defined first-value event.
    • Time-to-value asks how long it took to move from a clearly defined starting event to that first-value event.
    • Feature adoption and usage frequency ask whether customers continue performing the behaviors associated with value. DAU/MAU is only helpful when daily use matches the product’s natural cadence.
    • Cohort retention asks whether customers who started in the same period remain over successive intervals. Segment it when different customer groups buy for different jobs.
    • Logo churn asks what proportion of starting customers left during the period.
    • Gross revenue retention isolates retained recurring revenue before expansion: starting recurring revenue minus churn and contraction, divided by starting recurring revenue.
    • Net revenue retention adds expansion to that revenue view. Because expansion can offset losses, pair NRR with GRR and logo churn instead of reading it alone.
    • Support demand and resolution time help show whether customers are paying an operational cost to realize the promised value.

    Follow the exposed cohorts far enough to observe the retention window you selected. Do not declare success from an early conversion lift when the product decision is about durable use.

    Interpret experiment results without overclaiming

    • Attention rises, but activation does not: the message became more noticeable, not more useful.
    • Onboarding completion rises, but first value does not: the instructions may be clearer while the path still ends at the wrong outcome.
    • Activation rises, but retention falls: the message may attract the wrong expectation, or the activation event may represent task completion rather than customer value.
    • Sentiment improves, but behavior does not: customers may understand the experience better without gaining more utility.
    • Behavior improves, but sentiment remains negative: investigate reliability, effort, pricing, support, and trust rather than assuming usage settles the issue.
    • Activation and later retention improve: the intervention is a candidate for broader rollout. Check segment-level results and guardrails before scaling it.
    • No reliable effect appears: the message may not be the limiting factor, or the test may lack enough information to distinguish the effect. Check the design and evidence before concluding that messaging never matters.

    Use a cadence that matches the speed of the signal

    Review leading indicators such as activation, time-to-value, and feature adoption weekly. Review lagging commercial indicators such as GRR, NRR, and customer lifetime value monthly. Examine cohort retention quarterly to see whether improvements persist rather than merely shifting activity between periods.

    Run the review with the people who can change both the promise and the experience: the product trio and relevant go-to-market leaders. Keep the agenda decision-oriented:

    1. Which cohort and journey stage are under review?
    2. What changed in behavior, sentiment, and commercial outcomes?
    3. Where do those signals agree, and where do they conflict?
    4. Which prior hypothesis did the evidence support or weaken?
    5. Is the next action a product change, a message change, a combined experiment, or further discovery?
    6. Who owns the action, which metric should move, and when will the decision be revisited?
    7. What customer language, objection, proof point, or boundary should be added to the messaging framework?

    This last step closes the loop. New evidence updates the promise. The revised promise shapes acquisition and the product journey. Customer behavior tests whether the promise is true. Retention shows whether the value endures.

    Key takeaways

    • Treat customer insight, messaging, and retention as one operating loop, not three separate workstreams.
    • Diagnose whether the problem is an inaccurate promise, an unclear path, a missing first-value moment, or weak recurring value before changing copy.
    • Join customer language with product behavior, journey stage, account context, and commercial outcomes. Neither sentiment nor telemetry is sufficient alone.
    • Build each value pillar from a specific segment, customer job, outcome, mechanism, point of parity, differentiation, proof, and observable success event.
    • Pair message experiments with both leading indicators and downstream retention measures. An early conversion lift does not establish durable value.
    • Use generative AI to organize and retrieve evidence while preserving raw records, human review, privacy controls, and provenance.
    • Feed experiment results, objections, and customer language back into a living messaging framework so the next customer receives a more accurate promise.

    At your next product review, choose one segment and one journey stage. Bring one observed behavior, one recurring customer phrase, one value promise, and one retention measure. If you cannot name the behavior that proves value, fix the measurement. If you cannot support the promise with evidence, fix the message. If customers understand the promise but cannot realize it, fix the product.

    References

  • How to Build an Evaluation-Driven AI Innovation Strategy

    How to Build an Evaluation-Driven AI Innovation Strategy

    Your team has several credible AI demos, every sponsor sees potential, and no one can answer the question that matters: which idea deserves more engineering time, customer exposure, and operating risk?

    That is not an ideation problem. It is an evidence-design problem. A useful AI innovation strategy makes each investment earn its way forward through customer outcomes, representative evaluations, and explicit kill-or-scale decisions. The result is not less experimentation. It is faster learning with fewer expensive surprises.

    Start every AI bet with a decision contract

    Most AI roadmaps begin too far downstream. The discussion jumps to a model, an assistant, or an agent before the team agrees on the user problem or the evidence required to fund the next stage. The feature then acquires momentum simply because it exists.

    Replace the feature brief with a decision contract. This is a short agreement about what the bet must prove, how it will be evaluated, and what happens when the evidence arrives. It connects vision, portfolio choices, and execution to measurable outcomes before implementation choices harden.

    1. Name the user and the job. Specify who encounters the capability, what they are trying to accomplish, and which situations are out of scope. “Improve support with AI” is not a problem statement. “Help eligible customers resolve account questions without waiting for an agent” is testable.
    2. Choose the business outcome and its baseline. Use resolution rate, time-to-value, activation, retention, revenue lift, or another measure of customer and business value. Record how the existing workflow performs so the AI is compared with a real alternative, not with an empty screen.
    3. State the behavioral hypothesis. Explain how the proposed capability should cause the outcome to move. This exposes weak logic early. A faster response, for example, does not automatically produce a correct resolution.
    4. Define the evidence stack. Identify the offline evaluations needed to establish behavioral confidence and the live experiment needed to validate customer impact. Neither can substitute for the other.
    5. Set constraints and hard guardrails. Include unacceptable failures, privacy boundaries, safe-action requirements, latency expectations, and cost limits. A capability that is accurate but too slow, unsafe, or uneconomic is not ready.
    6. Pre-commit to the decision. Record the minimum detectable effect for the live experiment, the evaluation thresholds that block release, the time at which evidence will be reviewed, and the conditions for killing, refining, or scaling the bet.

    The contract should separate three metric layers. The outcome metric tells you whether customer or business value changed. Behavioral metrics tell you whether the AI performed its assigned job. Guardrails tell you whether that performance remained safe, reliable, responsive, and affordable. This prevents a team from celebrating a model score while the customer experience deteriorates.

    Consider a customer-support assistant. Eligible deflection and first-contact resolution can represent the business outcome. Factuality against the approved knowledge base, helpfulness, tone, retrieval accuracy, and safe CRM actions describe the system’s behavior. Harmful-content rate, unsafe-action rate, response latency, and token cost act as guardrails. A live test can then examine customer satisfaction and resolution instead of merely counting generated replies.

    This is the practical difference between an output and an outcome. Shipping an assistant is an output. Producing more successful resolutions without unacceptable safety, latency, or cost regressions is an outcome. Disciplined evaluation makes that distinction measurable.

    Match the evidence burden to the type and consequence of the bet

    A portfolio needs different kinds of AI innovation, but it should not evaluate every bet in the same way. Core optimization, adjacent expansion, and transformational innovation face different uncertainties. The label determines the strategic question. The consequence of failure determines the rigor.

    Portfolio betQuestion it must answerEvidence that matters mostTypical decision
    Core optimizationCan AI improve an established journey without damaging what already works?A reliable baseline, regression tests, live A/B results, and cost and latency guardrailsAdopt the change only when the improvement survives the existing quality bar
    Adjacent expansionDoes the capability solve a known job for a new segment, channel, or use case?Problem discovery, segment-representative evaluation cases, activation signals, and retention evidenceExpand only after the new audience reaches a meaningful value moment
    Transformational innovationCan a materially different workflow create value and be trusted?Task-completion tests, human review, adversarial testing, safe tool-use checks, and a staged customer pilotIncrease autonomy and exposure only as reliability and business evidence mature

    A core change can have a small strategic scope and still require a high evidence burden. An apparently simple classifier may sit inside a sensitive workflow. Conversely, a transformational concept can begin with a narrow, reversible prototype. Do not use “experimental” as permission to lower the bar for privacy, security, or consequential actions.

    The same discipline improves build, partner, and buy decisions. Generic demonstrations do not reveal how a system will perform on your customers’ language, your knowledge, your policies, or your tools. Run every viable option through the same representative task set. Compare task quality, latency, cost, integration effort, data boundaries, governance fit, and failure recovery. The vendor category matters less than whether the option can satisfy the decision contract.

    Portfolio funding should follow evidence maturity rather than presentation quality. Continue a bet when the team can identify remaining uncertainty and run a proportionate test to reduce it. Pause or kill it when customer value does not materialize, critical failure modes remain unresolved, or the required quality cannot fit inside the operating cost and latency envelope.

    A neutral experiment is not automatically wasted work. It can eliminate a weak hypothesis and release capacity for a better bet. But a poorly instrumented or under-sensitive experiment does not produce a useful neutral result. Set the minimum detectable effect and instrumentation before launch so “no movement” has an interpretable meaning.

    Build an evaluation stack that resembles the real product

    An AI evaluation is useful only when it represents the decisions the product must make under realistic conditions. A polished answer to a convenient prompt is weak evidence. The production system also has to handle ambiguous requests, imperfect retrieval, policy boundaries, long-tail inputs, adversarial behavior, and tool failures.

    Turn the golden dataset into an executable product specification

    Your golden dataset should express product intent through examples. Start with real, properly anonymized inputs from discovery, support, and product usage. Add important edge cases, long-tail situations, and adversarial prompts deliberately; waiting for production to reveal them transfers avoidable risk to customers.

    Each case should carry enough context to diagnose a failure, not just assign a score:

    • The user input and relevant conversation or workflow state
    • The approved information or system state the response may rely on
    • The expected behavior, acceptable answer range, or permitted action
    • A rubric for correctness, helpfulness, tone, and safety
    • A risk label that distinguishes ordinary quality defects from release-blocking failures
    • Metadata for the user segment, use case, input pattern, or workflow stage

    Keep the set versioned. Preserve cases that caught previous regressions, refresh it as customer behavior changes, and hold back examples that are not used for prompt tuning. Otherwise, the team can optimize for a familiar test set while making little progress on the wider product experience.

    Privacy belongs in dataset design. Anonymization, access control, retention rules, and approved data boundaries should be established before customer interactions become test fixtures. Retrofitting those controls after an evaluation pipeline spreads sensitive data is slower and riskier.

    Use several evaluators because each catches a different failure

    No single evaluation method is a complete quality system. Layer methods according to what is being tested:

    • Deterministic tests are appropriate for business rules, schemas, required fields, forbidden actions, exact calculations, and tool arguments. If a rule can be checked directly, do not ask another model to guess whether it passed.
    • Grounded checks compare claims with an approved knowledge base or retrieved context. They are essential when the product promises answers based on company or account information.
    • LLM-as-judge scoring can cover subjective dimensions such as helpfulness, relevance, and tone at useful scale. Define the rubric tightly and calibrate the judge against human decisions. Consistency is not enough if the judge consistently applies the wrong standard.
    • Pairwise preference tests help compare prompt, retrieval, or model variants when an absolute score is hard to interpret. They answer which candidate better satisfies the same rubric.
    • Human review remains necessary for critical, ambiguous, policy-sensitive, or high-consequence cases. It also provides the reference needed to recalibrate automated judges.
    • Red teaming probes manipulation, unsafe requests, policy evasion, and unexpected combinations of otherwise valid instructions.

    Agentic systems need evaluation beyond the final prose. A fluent confirmation can hide a failed or unauthorized action. Measure whether the agent chose the correct tool, supplied valid arguments, respected permissions and confirmation requirements, completed the intended task, and recovered safely when a dependency failed. Task-completion reliability and safe-action rate are more revealing than answer style alone.

    Quality must also be evaluated inside the cost-quality-latency envelope. A larger model can improve a difficult generation task and still be the wrong default for a simple classification step. Test model routing, token budgets, caching, prompt structure, retrieval quality, and function-calling patterns by task. The goal is not to minimize each cost independently; it is to meet the product’s quality bar with an operating profile the business can sustain.

    Turn evaluations into release gates and portfolio decisions

    An evaluation document that lives outside delivery will eventually be skipped. The evaluation suite should run whenever a prompt, model, retrieval pipeline, knowledge source, tool schema, or workflow changes. That makes evaluation part of the release mechanism instead of a launch ceremony.

    Use a gate sequence from discovery through production

    StageEvidence to collectDecision enabled
    Problem discoveryUser problem, current workflow, baseline, value hypothesis, and major risksDecide whether the problem deserves an AI bet
    PrototypeRepresentative golden-set results, failure taxonomy, latency, and estimated operating costDecide whether the capability has a credible path to the product bar
    Pre-releaseRegression suite, calibrated human review, adversarial cases, privacy checks, and safe-action testsBlock, revise, or approve a controlled rollout
    Controlled rolloutPredefined A/B test, value-moment telemetry, satisfaction, guardrails, and incident signalsValidate whether offline quality creates customer and business value
    Production scaleContinuous monitoring, segment-level failures, cost and latency trends, incidents, and refreshed evaluationsScale, route, constrain, roll back, or retire the capability

    Separate hard gates from optimization targets. A prohibited action, a privacy-boundary violation, or a broken business rule should block release. A modest tone improvement or non-critical cost regression may be handled as a tracked trade-off. If every metric is a hard gate, delivery stalls. If none is, the gate is theater.

    I use a simple test for gate quality: if two accountable leaders can read the same result and reach opposite release decisions, the decision rule is incomplete. Define the failing threshold, affected cases, permitted exception process, and rollback action before the result arrives.

    For systems that can change customer data, communicate externally, or trigger another consequential action, start with narrow permissions and human confirmation. Log the proposed action, the tool call, the result, and the reason for escalation. Increase autonomy only when the relevant task and safety evaluations hold under real usage. A human-in-the-loop control is most useful when the escalation path, response owner, and incident procedure are explicit.

    Offline evaluations create confidence to expose the product. They do not prove business impact. A live experiment must test the stated outcome with a predefined minimum detectable effect while watching for novelty bias and segment-specific failures. Instrument the customer’s value moment, not merely clicks on the AI entry point. An assistant can attract curiosity without improving activation, retention, resolution, or satisfaction.

    Production telemetry should feed back into the golden dataset. Add recurring failures, newly observed edge cases, incidents, and examples where users abandon or escalate. This turns customer reality into the next regression suite and prevents evaluation from freezing at the assumptions held before launch.

    Carry one scorecard from the product team to the QBR

    Leadership does not need a separate innovation narrative built from feature updates. Use one scorecard at product reviews, investment reviews, and QBRs. It should contain:

    • The portfolio class and strategic outcome
    • The target user, job, and current baseline
    • The causal hypothesis and non-AI alternative
    • The primary business metric and minimum detectable effect
    • The offline quality measures and live outcome measures
    • The safety, privacy, latency, reliability, and cost guardrails
    • The current evidence, unresolved uncertainty, and confidence level
    • The next test, accountable owner, review point, and kill-or-scale rule

    This creates a common language for product, engineering, design, go-to-market, risk, and executive stakeholders. The conversation becomes: What did the bet need to prove? What evidence changed? Which uncertainty remains? What decision follows? It no longer depends on who presents the most persuasive demonstration.

    The scorecard also protects speed. Teams with explicit boundaries can make routine prompt, retrieval, routing, and interface improvements without reopening the entire strategy. Leadership attention can stay on exceptions, material regressions, capital allocation, and bets whose evidence no longer supports the original thesis.

    Key takeaways for your next AI portfolio review

    • Require a decision contract before an AI idea receives roadmap momentum: user, outcome, hypothesis, evidence, guardrails, and kill-or-scale rule.
    • Classify each bet as core, adjacent, or transformational, but set evaluation rigor according to the consequence of failure.
    • Build a versioned golden dataset from anonymized real inputs, important edge cases, long-tail situations, and adversarial prompts.
    • Layer deterministic checks, grounded tests, calibrated model judging, human review, preference testing, and red teaming.
    • Evaluate agent actions and task completion, not only the fluency of the final response.
    • Run relevant regressions whenever prompts, models, retrieval, knowledge, tools, or workflows change.
    • Use offline evaluation to control release risk and live experimentation to validate customer and business impact.
    • Fund, refine, pause, or kill bets based on evidence maturity rather than demo quality or sunk effort.

    At your next roadmap review, pick one upcoming AI bet and pause the implementation discussion until its decision contract is complete. Then run the current workflow through a representative evaluation set before changing it. That baseline gives every later improvement something honest to beat.

    When each investment has a visible path from user problem to evaluation to decision, AI innovation stops being a contest between plausible demos. It becomes a repeatable way to allocate attention, manage risk, and scale the capabilities that produce durable value.

    References

  • Outcome-Driven Product Discovery: From Ideas to Better Bets

    Outcome-Driven Product Discovery: From Ideas to Better Bets

    You are looking at a roadmap full of plausible ideas, yet nobody can explain which one is most likely to change customer behavior. Sales has requests, support has complaints, leadership has strategic themes, and the product team has solutions waiting for estimates. Everything sounds important because the outcome has not been made precise enough to disqualify anything.

    Outcome-driven product discovery fixes that problem by connecting every roadmap bet to the same chain: business result, customer behavior, opportunity, assumption, experiment, and decision. It gives you a practical way to invest in innovation without turning every interesting idea into a delivery commitment.

    Start with the behavior you need to change

    A launch is an output. Completing a first meaningful workflow is a behavior. Activation is a product outcome. Retained revenue is a business outcome. Those concepts may sit in the same strategy, but they are not interchangeable.

    Start discovery with the product outcome because it is close enough to the customer experience for a team to influence and measure. Then state the business result you expect it to support. That connection is a hypothesis, not an automatic fact. Improving engagement that has no relationship to customer value, retention, conversion, or another meaningful result simply produces a more active feature.

    A useful outcome statement has five parts:

    • Segment: the specific users, accounts, or lifecycle stage whose behavior matters.
    • Behavior: an observable action that represents progress toward value.
    • Baseline and target: the current measurement and the change the team intends to produce.
    • Decision window: when you will review the evidence and decide what to do next.
    • Guardrail: the metric or customer consequence that must not deteriorate while the primary outcome improves.

    Use this template: By [decision date], change [behavior] for [segment] from [baseline] to [target], because that behavior is expected to contribute to [business result], while protecting [guardrail].

    Suppose a SaaS team wants to improve new-account activation. The feature-factory version of the goal is to launch a redesigned onboarding checklist. The outcome-driven version identifies the new-account segment, the value-bearing workflow those users need to complete, the current completion rate, the desired change, the review date, and a guardrail such as downstream retention or support burden. The checklist may become one solution, but it no longer owns the roadmap before discovery begins.

    Keep three measures visible on the same decision page:

    • Primary outcome: the customer behavior you intend to change.
    • Business consequence: the commercial or strategic result that behavior is expected to influence.
    • Guardrail: the cost, quality, trust, or downstream behavior you refuse to sacrifice.

    This is the practical difference between organizing goals around outcomes instead of output and attaching metrics to a feature after it has already been approved. The first approach creates choice. The second decorates a commitment.

    Before accepting an outcome, ask four questions. Can the team observe it? Can the team influence it during the decision window? Does it represent customer progress rather than product activity alone? Is its expected connection to the business result explicit? If any answer is no, revise the outcome before collecting more ideas.

    Key takeaways

    • Begin with a measurable customer behavior, not a feature, project, or launch date.
    • Treat the link between that behavior and the business result as a hypothesis that needs evidence.
    • Map opportunities before comparing solutions, so requests do not become commitments by default.
    • Combine segmented customer evidence with product telemetry; neither is sufficient on its own.
    • Give every experiment a decision rule, a meaningful effect threshold, and guardrails.
    • Judge discovery by the decisions it changes, including decisions to adapt, delay, or stop a bet.

    Map opportunities before you rank solutions

    Once the outcome is clear, resist the urge to run an idea workshop. First map the obstacles, unmet needs, and motivations that could explain why the desired behavior is not happening.

    An opportunity describes a customer condition. A solution describes something you could build. For example, users abandoning setup because they cannot tell which information is required is an opportunity. A setup wizard, template, tooltip, or assisted service is a solution. Keeping those levels separate preserves more than one path to the outcome.

    Translate feature requests with a simple sequence:

    1. Ask which user or account segment is making the request.
    2. Identify the job that person is trying to complete.
    3. Locate the point in the journey where progress breaks down.
    4. Describe the consequence of that breakdown in the customer’s terms.
    5. Connect the problem to the target outcome.
    6. Record the requested feature as one possible solution, not as the opportunity itself.

    This translation matters because a request can be accurate about the pain and wrong about the remedy. It can also be valid for one enterprise account but harmful to the broader value proposition. Segmenting feedback by persona, account tier, lifecycle stage, and job prevents unlike signals from being combined into a misleading vote count. A founder, a new user, a power user, and an account approaching renewal are speaking from different contexts.

    Build the map with a product trio: product management, design, and engineering working on the problem together. Early engineering involvement exposes feasibility constraints and cheaper implementation paths. Design brings the journey and interaction risks into view. Product management connects the opportunity to customer value, strategy, and commercial consequences. The benefit is shared reasoning, not another recurring meeting.

    A practical outcome-driven operating model gives that trio room to investigate opportunities before delivery sequencing hardens. Without it, discovery becomes a product-manager document handed to design and engineering after the consequential decisions have already been made.

    Use the following rubric to compare opportunities. Do not collapse it into a single total score. A tidy score can hide a fatal weakness, such as no evidence that the problem exists for the target segment.

    CriterionDecision questionWarning sign
    Outcome proximityIf this problem is reduced, what customer behavior should change?The connection depends on several untested assumptions.
    Segment evidenceWhich target users experience the problem, and in what context?The evidence comes mainly from unsegmented requests or one loud account.
    Severity and recurrenceDoes the problem block value, repeatedly create friction, or merely inconvenience the user?The team cannot distinguish a recurring obstacle from an isolated preference.
    Strategic coherenceWould solving it strengthen the intended value proposition or differentiation?The solution adds complexity without making the product more valuable to its chosen market.
    Learning valueWhat important uncertainty would pursuing this opportunity resolve?The team is committing substantial delivery capacity without identifying the risky assumption.
    Downside and reversibilityWhat could break, and how easily could the change be contained or reversed?Trust, data, operational, or platform risk is being treated as a post-launch concern.

    The result should be an opportunity map, not a backlog. A backlog asks what can be built. An opportunity map asks where a change could produce the outcome, what evidence supports that belief, and what still needs to be learned.

    Match the strength of evidence to the size of the commitment

    Customer interviews alone do not tell you how widespread a problem is. Product analytics alone do not tell you why a behavior occurs. Strong discovery uses each form of evidence for the question it can answer.

    • Qualitative evidence reveals language, context, motivation, workarounds, and consequences.
    • Behavioral evidence shows where users progress, hesitate, abandon, return, or differ across cohorts.
    • Commercial evidence shows how the opportunity appears in sales, expansion, support, renewal, or churn conversations.
    • Experimental evidence tests whether a specific intervention causes the intended change under defined conditions.

    Start with the journey connected to the outcome. Instrument the important steps, inspect funnels and cohorts, and then use interviews, support conversations, community discussions, and sales or customer-success notes to explain the patterns. This combination of telemetry and customer narrative is more useful than collecting more comments without a decision in mind.

    When qualitative and quantitative evidence disagree, do not average them into a vague conclusion. Investigate the mismatch. The interview sample may represent power users while the funnel includes new users. The telemetry may be missing an offline step. A workflow may be painful but unavoidable, producing high completion despite poor experience. A small segment may have a severe problem hidden by an aggregate rate. Contradiction is often a segmentation or instrumentation clue.

    Create a shared taxonomy so evidence remains usable after the meeting in which it was collected. Tag each item by:

    • problem statement;
    • persona or account segment;
    • job to be done;
    • journey step;
    • lifecycle stage;
    • evidence channel;
    • related outcome;
    • confidence and unresolved uncertainty.

    Then produce a compact evidence packet for each opportunity under active consideration:

    • Outcome: the behavior the team wants to change.
    • Observation: the measured pattern, with its segment and journey context.
    • Customer explanation: the recurring need, obstacle, or workaround found in qualitative evidence.
    • Contrary evidence: what does not fit the current explanation.
    • Current hypothesis: why the opportunity may be causing the behavior.
    • Largest uncertainty: the assumption most capable of invalidating the bet.
    • Next decision: what the team will decide after the next learning step.

    The required evidence should rise with the cost and irreversibility of the commitment. A reversible wording change can justify a lightweight test. A new core workflow, platform dependency, pricing model, or data-access pattern deserves deeper investigation because mistakes create migration cost, operational burden, customer confusion, or trust damage.

    My test is simple: can the team state what evidence would make it change course? If not, the work is advocacy rather than discovery. Evidence is being gathered to support a preferred answer, not to improve the decision.

    Run experiments that force a roadmap decision

    An experiment is useful only when its result can change what happens next. Before choosing a prototype or test method, write the decision the evidence must inform.

    A concise experiment card should contain:

    • Hypothesis: If [segment] receives [intervention] in [context], then [behavior] will change because [reason].
    • Riskiest assumption: the belief that would make the solution unattractive, unusable, infeasible, unviable, or unsafe if false.
    • Method: the least expensive credible way to test that assumption.
    • Primary measure: the signal that directly answers the experiment question.
    • Meaningful effect: the smallest change that would justify a different product decision.
    • Guardrails: the customer, business, quality, or trust measures that must remain acceptable.
    • Decision rule: the conditions for advancing, adapting, stopping, or gathering different evidence.

    Choose the method based on the uncertainty:

    • Use interviews and observation to understand the job, context, current alternative, and consequence of the problem.
    • Use concept tests to learn whether the proposition is understood and relevant.
    • Use clickable prototypes to find comprehension, interaction, and workflow problems before production work.
    • Use a manual or limited implementation to test whether completing the workflow creates enough value to justify automation and scale.
    • Use feature flags and progressive rollouts to contain operational risk and inspect real behavior.
    • Use an A/B test when you need a credible comparison of incremental behavior and have the traffic, instrumentation, and time to run it properly.

    Do not ask one method to prove more than it can. Positive interview reactions do not prove adoption. A usable prototype does not prove retention. A short-term click improvement does not prove durable customer value. Each result should earn the next level of investment, not retroactively validate the entire strategy.

    For A/B tests, define the minimum detectable effect before launch. This is the smallest difference worth reliably detecting for the decision, not the smallest fluctuation visible in a dashboard. Plan the sample around that threshold, avoid repeatedly checking results and stopping when they look favorable, and carry the analysis into downstream behavior where the hypothesis requires it. Statistical discipline and retention analysis prevent short-lived movement from being mistaken for a product win.

    If the available traffic cannot support the planned effect within the decision window, do not run an underpowered test and interpret noise. Reduce the scope, extend the observation period where practical, use a stronger leading indicator, or select a different method. The method should fit the decision environment.

    Guardrails deserve the same pre-commitment as the primary measure. An onboarding change that raises completion but also increases early cancellations, support contacts, errors, or later abandonment may have shifted friction rather than removed it. The team should know in advance which trade-offs are unacceptable.

    End every experiment with one of four explicit decisions:

    • Advance: the evidence supports the assumption strongly enough to justify the next investment.
    • Adapt: the opportunity still matters, but the solution or segment hypothesis needs revision.
    • Stop: the expected outcome no longer justifies the cost, risk, or strategic distraction.
    • Reframe: the test exposed an instrumentation gap, a different opportunity, or an assumption that must be investigated first.

    A failed solution test can still be a successful discovery decision. The value lies in avoiding a larger, poorly justified commitment.

    Turn discovery into the operating system for innovation

    Innovation is not measured by how unfamiliar a solution looks. It is measured by whether the team finds a better way to create and capture value under uncertainty. That requires a learning system, not a separate innovation theater filled with demos that never reach adoption.

    Give every innovation bet a one-page brief:

    • the target segment and job;
    • the behavior and business outcome;
    • the current alternative and why it is insufficient;
    • the opportunity being pursued;
    • the intended value proposition and differentiation;
    • the riskiest value, usability, feasibility, viability, or trust assumption;
    • the next experiment and its decision rule;
    • the owner, review date, and current investment boundary.

    This brief lets leadership compare bets without pretending that early ideas have precise forecasts. Mature work can be judged on measured outcome contribution. Earlier innovation should be judged on the importance of the opportunity, strategic fit, quality of evidence, cost of the next learning step, and whether uncertainty is falling fast enough to justify continued investment.

    Differentiate deliberately. Some capabilities are points of parity that customers expect. Others are candidates for meaningful differentiation. Treating every competitor feature as strategically necessary fragments the product and consumes capacity that could strengthen the chosen value proposition. First-principles reasoning should establish which customer problem matters before competitive comparison influences the solution.

    For AI products, trust belongs inside the outcome

    An AI prototype can appear successful while hiding the operational conditions required for a durable product. Add trust and control questions to discovery from the beginning:

    • What happens when the output is wrong, incomplete, or inappropriate?
    • Which data can the system access, retain, or expose?
    • Where does a person need to review, approve, correct, or override the system?
    • Can the team observe failures and explain consequential actions?
    • Does the workflow create enough customer value after review, exception handling, and operating cost are included?

    Privacy, data governance, transparent controls, and auditability are part of the product proposition, especially when the workflow has meaningful consequences. Moving from an AI demonstration to a durable capability requires evidence about the complete workflow, not just the quality of a favorable output.

    Install a cadence that changes priorities

    Discovery becomes operational when evidence repeatedly changes allocation decisions. A practical cadence is:

    • Weekly product-trio review: examine the target outcome, new evidence, contradictions, largest uncertainty, and next decision for active bets.
    • Monthly cross-functional synthesis: combine themes from product behavior, interviews, sales, support, and customer success; resolve segmentation questions; and identify implications for the roadmap.
    • Quarterly outcome lookback: compare expected and observed changes in activation, adoption, conversion, retention, or the relevant business result; inspect guardrails; and record which assumptions were right or wrong.

    This feedback and synthesis cadence creates organizational memory. It also exposes a hollow process quickly. If repeated discovery reviews never stop, reorder, narrow, or reshape roadmap work, the organization has built a reporting loop rather than a decision loop.

    Represent roadmap items as bets, with the outcome, segment, opportunity, evidence, hypothesis, guardrails, owner, and next decision visible. Delivery milestones still matter, but they sit beneath the reason for the work. That makes stakeholder conversations more precise. Instead of asking whether a requested feature made the roadmap, ask which outcome it supports, what problem it solves, what evidence exists, and what would justify investment.

    Keep a short decision log after each review. Record the decision, evidence considered, assumptions still open, owner, and revisit condition. This prevents the organization from re-litigating old choices after context has disappeared, while allowing a decision to change when genuinely new evidence arrives.

    Take the next substantial item scheduled to enter delivery and try to fill in its outcome statement, opportunity, evidence packet, riskiest assumption, experiment, guardrail, and decision rule. Any field you cannot complete is not paperwork to delegate. It is the uncertainty discovery needs to resolve before the commitment grows.

    Do that with one bet first. When the resulting evidence changes an investment decision, use the same structure for the rest of the roadmap. That is the point at which discovery stops being a phase and starts becoming how innovation is managed.

    References

  • How to Connect Product Activation to Growth Economics

    How to Connect Product Activation to Growth Economics

    Your signup chart is climbing, yet retained revenue and CAC payback are not improving. The usual responses – buy more traffic, add another onboarding tour, or push sales harder – treat the symptoms separately. The real break is often between the promise that earned the signup, the first outcome the customer experiences, and the economic value that follows.

    You can find that break by treating activation as part of a value system, not as an isolated funnel percentage. Define the first value precisely, verify that it predicts repeated value, connect it to revenue quality, and then decide whether acquisition deserves more investment.

    Key takeaways

    • Activation should represent a customer outcome or a credible proxy for one, not merely account creation, onboarding completion, or feature exposure.
    • An activation metric is incomplete without an eligible population, unit of analysis, event, time window, and customer segment.
    • Higher activation is useful only when activated cohorts also show stronger retention, paid conversion, expansion, or another form of durable value.
    • Diagnose activation by ICP, use case, channel, plan, and account type. A blended average can improve because the customer mix changed while the core experience stayed flat.
    • Scale acquisition after the activation-to-economics chain holds. More traffic cannot repair a weak value path; it only sends more people through it.

    Define activation as a contract with the customer

    A signup records intent. Onboarding completion records progress. Activation should record the earliest moment when the customer has evidence that your product can deliver the outcome they came for.

    That distinction matters because product value appears first as a belief and then as an experienced result. Your positioning creates perceived value; the product has to turn it into realized value. Durable growth begins when customers can repeat that result and consider it valuable enough to retain, pay for, or expand. Managing perception, behavior, and economics as connected signals prevents a polished acquisition message from hiding a weak product experience.

    A first campaign launch, a completed core workflow, or a successful CRM connection could be an activation event. The correct choice depends on the promise. Connecting a CRM is meaningful if the connection itself removes an important constraint. If the customer still has to configure several steps before receiving any benefit, the connection is setup, not activation.

    Write an activation specification before asking analysts to build a dashboard:

    1. Choose the value unit. Decide whether value belongs to a user, account, workspace, or team. A collaboration product can show many active users while the customer account remains unactivated.
    2. Name the target customer and job. State which ICP and use case the event represents. Different jobs may require different activation paths, even inside the same product.
    3. Define cohort entry. Specify when the clock starts: account creation, invitation acceptance, trial start, or another unambiguous event.
    4. Define the milestone. Use one observable event or a small, auditable set of conditions. Avoid labels such as engaged user unless every team can calculate them identically.
    5. Set the value window. Measure whether the milestone occurs within a period appropriate to the product’s natural setup and usage cycle. Do not borrow a fashionable first-session or seven-day window if customers cannot reasonably realize value that quickly.
    6. Define the validation behavior. Name the later behavior or economic result that should be stronger among activated customers, such as repeated core usage, retention, paid conversion, or expansion.

    The result should fit into one sentence: An eligible target account activates when it completes a named value event within a defined period after a named starting event. If the sentence contains words such as meaningful, engaged, or successful without an event definition, it is not ready to instrument.

    Capture enough context with the event to diagnose it later: account and user identifiers, role, plan, ICP segment, use case, acquisition channel, and timestamp. Then map the path from cohort entry through required setup, first value, repeated value, monetization, and retention. A clear activation milestone and end-to-end journey give product, marketing, sales, and customer success the same definition of progress.

    Time-to-value belongs beside activation rate. Two cohorts can finish with the same activation percentage while one spends much longer waiting for value. Look at the distribution by segment rather than relying only on one blended average. The long tail will show which customers are technically activating but doing so too late for the experience to feel convincing.

    Connect first value to retention and unit economics

    Activation is a hypothesis about value, not proof of it. You validate that hypothesis by following activated and non-activated cohorts into later behavior and economics. A strong association does not prove that the event caused retention, but it does tell you whether the event is useful as a leading indicator. Controlled experiments can then test whether changing the path to that event produces the expected improvement.

    Use a driver tree that connects qualified demand to first value, repeated value, monetization, and acquisition efficiency. Each stage answers a different management question:

    StageQuestionUseful signalsLikely decision
    Qualified entryAre the right customers entering?ICP-qualified lead rate, qualified lead velocityChange targeting, positioning, channel mix, or the marketing-to-sales handoff
    First valueDo eligible customers reach a credible outcome quickly?Activation rate, time-to-value, critical-path drop-offsRemove setup friction, improve defaults, or clarify the path
    Repeated valueDoes the outcome become part of the customer’s workflow?Retention curves, core feature adoption depth, active teamsStrengthen recurring use cases, habit loops, and proofs of progress
    MonetizationWill customers pay for the value and deepen adoption?Paid conversion, expansion revenue, NRR, gross marginRevisit packaging, pricing, purchase friction, or advanced use cases
    Acquisition efficiencyCan the company fund this growth motion sustainably?CAC by channel, CAC payback, retention-grounded LTV:CACReallocate budget, improve revenue quality, or repair earlier value leaks
    Sales-assisted growthDoes product evidence help qualified opportunities close?Win rate, sales-cycle length, product-qualified account behaviorImprove proof points, positioning, routing, or sales follow-up

    Keep the calculations explicit. Activation rate is activated eligible units divided by eligible units entering the cohort. Time-to-value is the elapsed time from cohort entry to the first-value event. CAC payback asks how many months of gross-margin contribution are required to recover acquisition cost. LTV:CAC compares expected customer value with acquisition cost, but the lifetime assumption must come from observed retention rather than an optimistic spreadsheet.

    There is no universal number that makes these metrics healthy. A tolerable payback period depends on gross margin, cash constraints, contract structure, retention, and the speed at which the company wants to reinvest. The useful comparison is between cohorts and channels calculated consistently under your economic constraints.

    Activation affects more than conversion. Faster value can reduce the amount of explanation and support required before a customer becomes productive. Stronger early value can also improve retention and create room for expansion. That is why activation, time-to-value, channel CAC, payback, and retention-grounded LTV:CAC should appear in the same operating view rather than in separate departmental dashboards.

    For a hybrid product-led and sales-assisted motion, join product events to CRM records using stable account identifiers. You should be able to move from acquisition channel to signup, activation, opportunity, closed revenue, retention, and expansion without changing the cohort definition. This exposes cases where a channel produces inexpensive signups but few valuable customers, or where product-qualified accounts close faster than accounts without value evidence.

    Read the shape of the leak before changing onboarding

    A low activation rate does not automatically mean the onboarding interface is bad. The cause can sit in targeting, the value proposition, required configuration, permissions, product reliability, or the activation definition itself. The pattern across segments and downstream outcomes tells you where to look.

    • Qualified signups are healthy, but activation is weak across the core ICP. Inspect the critical path. Remove unnecessary pre-value work, improve defaults, and find the step where time-to-value expands. If the core outcome requires a complex integration or approval, make that dependency visible before signup rather than surprising the customer inside onboarding.
    • Non-ICP users activate, but the target ICP does not. Do not celebrate the blended rate. The product may be optimized for a simpler use case, or the event may represent value for the wrong customer. Revisit ICP-specific discovery, positioning, and the activation definition.
    • Activation is high, but retention is weak. The milestone may be too shallow, too easy to trigger, or tied to one-time value. Compare behavior immediately before and after activation. Redefine the milestone around a more credible outcome or add a repeated-value measure.
    • Activated customers retain, but paid conversion is weak. The first-value path may be working. Examine packaging, price-to-value alignment, purchase permissions, and the transition from trial value to paid value before redesigning onboarding.
    • Conversion is healthy, but CAC payback deteriorates. Break CAC and gross-margin contribution down by channel and segment. High acquisition cost, a longer sales cycle, heavy implementation work, or high ongoing support cost can weaken economics even when the product converts.
    • The blended metric improves, but every established segment is flat. Customer mix changed. Report both the overall number and stable segment cohorts so a channel shift is not mistaken for a better product experience.

    Run the diagnosis in a fixed order. First, verify event integrity: identifiers, timestamps, duplicate events, eligibility rules, and account-user joins. Second, segment the funnel by ICP, use case, channel, plan, role, and value unit. Third, inspect event sequences and time-to-value around the largest drop-offs. Fourth, use customer interviews and support conversations to understand why the observed step is difficult. Only then choose the intervention.

    This order prevents a common waste pattern: adding a product tour when the customer lacks permissions, adding tooltips when the value proposition attracted the wrong use case, or simplifying an event until the metric rises but its relationship with retention disappears.

    Run experiments that earn the right to scale acquisition

    Start with the three largest losses between entry and first value, then choose the one most concentrated in the target ICP. The biggest percentage drop is not always the best opportunity. Consider how many qualified accounts reach the step, whether the obstacle is within product control, and whether removing it preserves the quality of activation.

    Interventions should match the diagnosed mechanism:

    1. Remove work that is not required for first value. Defer optional fields, preferences, invitations, and integrations until after activation. Keep any dependency that is essential to producing the promised outcome.
    2. Improve the starting state. Use sensible defaults, templates, examples, and preconfigured paths so the customer can act without designing a workflow from an empty screen.
    3. Guide in context. Use in-app guides, product tours, and tooltips at the decision point they support. A tour shown before the customer has relevant context adds completion activity without necessarily shortening time-to-value.
    4. Make progress visible. Show what has been accomplished, what remains, and why the next step matters. Proof of progress is especially useful when setup cannot be compressed into one session.
    5. Personalize by job and role. Route customers to the shortest credible path for their use case instead of forcing every ICP, administrator, and end user through one generic checklist.
    6. Introduce advanced use cases after first value. Templates and higher-order workflows can create expansion, but presenting them too early increases cognitive load before the customer understands the core job.

    Every experiment needs a decision-ready specification: eligible cohort, hypothesis, treatment, primary metric, guardrails, minimum detectable effect, observation window, and decision rule. Setting the minimum detectable effect before an A/B test helps prevent a noisy movement from becoming a declared win. If the available sample cannot detect a change worth acting on, narrow the question, use a larger intervention, or collect more observations rather than repeatedly checking an underpowered result.

    Use activation rate or time-to-value as the leading metric, but keep downstream guardrails. An experiment that increases activation by making the event easier has failed if retained usage or paid conversion falls. An experiment that leaves the final activation rate unchanged may still be valuable if qualified customers reach value sooner without increasing support burden.

    Review the system weekly with product, design, engineering, growth, sales, and customer success owners who can explain the full journey. Keep the review focused on decisions: which segment moved, which part of the driver tree explains it, what the experiment established, and what changes as a result. Shipping a tour is output; improving activation among a defined ICP without weakening retention is an outcome.

    Increase acquisition investment only when the activation event remains associated with later value, the improvement holds in the target ICP, downstream conversion and retention do not weaken, and cohort economics fit the company’s reinvestment constraints. Channel-level CAC matters here: cheap traffic with weak activation and retention is not efficient growth.

    Your next move is small and concrete. Write the one-sentence activation specification, pull the latest cohort old enough to observe the relevant retention behavior, and compare the target ICP’s activators with its non-activators. If the event does not separate later value, fix the definition. If it does, find the largest qualified drop-off on the path to it and test one focused change. Once that link holds through retention and economics, acquisition becomes an accelerator instead of a way to conceal the leak.

    References

  • How to Build a High-Velocity Product Experimentation System

    How to Build a High-Velocity Product Experimentation System

    Your team is shipping more often, yet roadmap debates still drag on and too many releases end without a clear decision. That is not high velocity. It is faster production without faster learning.

    High-velocity product delivery reduces the time between identifying a customer problem, exposing a safe change, reading credible evidence, and deciding what to do next. You get there by treating experimentation and delivery as one operating system, with shared outcomes, explicit decision rules, controlled exposure, reliable instrumentation, and rapid recovery.

    Measure velocity at the decision, not the deployment

    Deployment frequency matters because small, frequent production changes shorten technical feedback loops. It belongs beside lead time for changes, change failure rate, and mean time to recovery as part of a balanced view of delivery performance and reliability. But deployment is only one step in the value chain.

    A deployment puts code into production. A release makes a capability available to users. An experiment exposes a defined population to controlled alternatives so you can answer a question. A product decision uses that evidence to scale, revise, or stop the work. When those actions are treated as one event, teams accumulate large batches, launch cautiously, and struggle to identify what caused the result.

    SignalWhat it tells youWhat it cannot tell you alone
    Deployment frequencyHow often code reaches productionWhether users received value
    Release or exposureWho can use the changeWhether the change caused an outcome
    Experiment decisionWhether evidence changed a product choiceWhether the delivery system is reliable
    Change failure rate and MTTRHow safely the system changes and recoversWhether the product hypothesis was right
    Customer or business outcomeWhether the result that matters movedWhich intervention caused the movement

    I would not call a team high velocity merely because it deploys daily. I would look for a short decision cycle: the elapsed time from accepting a product question to recording an evidence-backed decision. Track that alongside the DORA metrics and the outcome the team owns. This prevents a local improvement in engineering throughput from masquerading as product progress.

    You probably have a decision-flow problem if any of these patterns are common:

    • Features are declared complete at launch, with no owner or date for the readout.
    • Teams run tests but define success after seeing the result.
    • Several unrelated changes enter one release, making attribution difficult and rollback expensive.
    • Product reviews discuss shipped items while customer outcomes remain unchanged or unknown.
    • Deployment frequency rises while change failure rate or recovery time deteriorates.
    • Tests repeatedly end as inconclusive because traffic, detectable effect, or measurement quality was never checked before development.

    Do not respond by setting an experiment quota or a deployment target in isolation. Measure the entire path from question to decision, locate the longest wait state, and remove that constraint. The bottleneck may be test execution, approval, instrumentation, exposure control, analysis, or leadership indecision. More work in progress will only hide it.

    Write the decision before you write the feature

    An experiment should begin with a decision that needs evidence, not with a feature searching for justification. Before implementation starts, write a compact experiment contract. It turns a vague bet into a question the team can actually answer and makes disagreement cheaper because it happens before code is built.

    A reusable experiment contract

    1. Customer problem and population: Name the behavior or friction you are addressing, the eligible segment, and any exclusions. Avoid a target such as all users unless the experience and expected response are genuinely uniform.
    2. Outcome hypothesis: State what behavior should change and why. Use a falsifiable form: If this intervention changes this mechanism for this population, then this outcome should move.
    3. Primary decision metric: Choose the one measure that will decide the test. Diagnostic metrics can explain the result, but they should not become alternate finish lines after the fact.
    4. Minimum detectable effect: Define the smallest effect large enough to change the product decision. Setting the minimum detectable effect before an A/B test begins keeps the team from treating ordinary metric movement as a meaningful win.
    5. Guardrails: Identify customer-experience, reliability, trust, and business measures that must not deteriorate beyond the agreed boundary. A primary metric win is not permission to ignore material harm elsewhere.
    6. Measurement conditions: Record the assignment unit, exposure event, analysis population, start condition, required observation window, and known instrumentation dependencies. If the data cannot distinguish eligibility from actual exposure, fix that before launch.
    7. Decision rule: Specify what will cause the team to scale, iterate, stop, pause, or classify the result as invalid. Name the decision owner and the readout date as part of the same contract.

    The MDE is not the smallest movement you would enjoy seeing. It is the smallest movement worth acting on. It also has to be compatible with baseline behavior, eligible traffic, and the observation window. A tiny MDE may sound rigorous, but if the product cannot gather enough evidence to detect it, the team has designed a waiting period rather than a useful experiment.

    Consider a hypothetical activation test. The problem is that new accounts fail to complete a clearly defined first-value workflow. The proposed intervention is a contextual setup guide shown after first login. The primary metric is completion of the activation event. Reliability errors and a relevant customer-friction signal are guardrails. The team scales only if the primary effect meets the pre-agreed MDE and the guardrails hold. Every field points to a future decision; none merely describes the interface being built.

    Use an A/B test when controlled alternatives, stable assignment, and sufficient eligible traffic can answer the question. Use progressive exposure when the immediate question is operational safety or blast radius. Use discovery methods before either of those when the team still cannot state the customer problem or plausible mechanism. Calling every release an experiment does not make it one.

    If assignment breaks, events are missing, or exposure is contaminated, classify the test as invalid. If the data is valid but the primary metric does not meet the success rule, the hypothesis did not earn further investment in its current form. That distinction protects the team from rerunning weak ideas under the label of a measurement problem.

    Decouple deployment, exposure, and rollback

    High-velocity experimentation needs a delivery system that can put code into production without exposing it to everyone. Feature flags, canary releases, and blue-green deployment make that separation practical. Automated tests, observable pipelines, and fast recovery make it responsible.

    At HighLevel, I have helped products move from a weekly release train toward safe daily and eventually on-demand deployments without increasing incident volume. The important lesson was not to search for one breakthrough tool. Smaller batches, tests that fail when they should, immutable artifacts, flags, progressive delivery, and recovery controls had to work as a system.

    A safe experiment-release path looks like this:

    1. Merge a narrow change through trunk-based development, behind a flag that defaults to off for users.
    2. Build and verify one immutable artifact so the tested artifact is the artifact promoted through the pipeline.
    3. Deploy to production and check technical health before beginning customer exposure.
    4. Expose an internal population, canary cohort, or other deliberately limited group appropriate to the blast radius.
    5. Start experiment assignment only after exposure and measurement checks pass.
    6. Monitor the primary metric and guardrails without rewriting the success rule in response to early movement.
    7. Expand, pause, revert, or stop according to the contract. Preserve the result and rationale in the decision record.
    8. Remove the flag after the rollout or rollback path no longer requires it. Give every flag an owner and cleanup trigger when it is created.

    This sequence separates three kinds of failure that demand different responses:

    • Delivery failure: The change causes errors, incidents, or unacceptable system behavior. Reduce exposure, roll back or disable the path, and restore service before investigating.
    • Measurement failure: Assignment, event capture, or eligibility logic is unreliable. Stop interpretation, repair the measurement path, and rerun only if the decision still matters.
    • Product-hypothesis failure: The system is healthy and the data is valid, but the intervention fails the pre-registered decision rule. Stop or revise the bet instead of blaming the pipeline.

    Large batches make all three failures harder to diagnose. Split work so a change can be deployed, observed, and reversed independently. Long-lived branches and release trains increase the amount of unverified work moving together; fast test feedback, contract testing between services, and preview environments reduce the pressure to accumulate that work.

    A calendar restriction can reduce immediate exposure, but it does not create a safe delivery capability. If the organization cannot tolerate a routine deploy on a particular day, treat that as evidence that detection, rollback, staffing, or blast-radius controls need attention. The goal is not reckless release timing. It is a system in which an ordinary, narrow deployment is uneventful and recovery does not depend on heroics.

    Give empowered teams a learning cadence, not a feature quota

    Technical capability will not create velocity if every decision crosses several management and functional handoffs. Durable product trios should own a customer problem from discovery through delivery and readout. Leaders provide the outcome, strategic context, capacity, and non-negotiable constraints; the trio chooses how to learn and what solution, if any, deserves scale. That is the practical value of empowered teams organized around outcomes rather than output.

    Make the operating contract explicit:

    • Leadership owns direction: Define the few outcomes that matter, the time horizon, material constraints, and where evidence could justify reallocating capacity.
    • The product trio owns the learning loop: Frame the problem, choose the method, write the experiment contract, deliver the change, interpret the evidence, and record the decision.
    • Platform and engineering leadership own the paved road: Provide CI/CD, test infrastructure, feature flags, progressive delivery, observability, and recovery mechanisms that teams can use without bespoke negotiation.
    • Data partners own measurement integrity with the team: Standardize event definitions, validate critical events, and make assignment, eligibility, and exposure auditable.
    • Governance owns clear boundaries: Use privacy-by-design defaults, pre-approved experiment patterns, and a short escalation path for work that changes data use, legal exposure, or customer risk.
    • Portfolio forums own reallocation: Use experiment decisions and outcome movement to continue, stop, or redirect investment. Do not turn the forum into a recital of completed tickets.

    A unified analytics platform helps only when teams can trust and compare its events. For every decision-critical event, record the event name, exact trigger, required properties, owner, and validation status. Review taxonomy changes before launch and inspect live data before starting the experiment clock. Otherwise, the organization gains a shared dashboard but not shared truth.

    Keep one visible record for every active bet. It should show the owned outcome, hypothesis, current state, exposure, decision date, result, and next action. Limit final states to scale, iterate with a stated reason, stop, or invalid. This makes abandoned readouts visible and prevents an endless backlog of tests that technically ran but never influenced a decision.

    Planning and learning operate on different clocks. A roadmap may allocate capacity over a longer horizon, while an experiment can invalidate a bet much sooner. Connect them through regular decision reviews and use QBRs to move resources based on accumulated evidence. Do not force a team to continue a disproven initiative merely because the planning document has not reached its next revision date.

    Judge the system with a balanced scorecard:

    • The customer or business outcome the team is accountable for.
    • Decision cycle time from accepted question to recorded action.
    • The share of launched experiments that reach a decision, separated from invalid tests.
    • Deployment frequency and lead time for changes.
    • Change failure rate and mean time to recovery.
    • Guardrail breaches, rollback quality, and unresolved measurement defects.

    No single number should become a target detached from the rest. Faster deployment with rising failures is not healthy. More experiments with weak decisions is not learning. Better short-term conversion with damaged trust is not value.

    Reset the system in 30 days

    You do not need a company-wide transformation program to begin. Use a four-week reset on one product area and two services. The delivery work follows a practical sequence of baselining, reducing batch size, strengthening the pipeline, and publishing a balanced dashboard; the product work adds an explicit question and decision to that same flow.

    • Week 1: Map the real loop. Baseline production deployments by service, lead time, change failure rate, and MTTR. Trace one recent bet from initial question through release and readout. Mark every queue, approval, handoff, manual step, and missing event. Select one owned outcome and one active question for the pilot.
    • Week 2: Make the work smaller and the decision explicit. Choose two services and cut batch size in half. Enable feature flags for new code paths. Write the pilot experiment contract, including its population, primary metric, MDE, guardrails, exposure event, decision rule, owner, and readout date.
    • Week 3: Prove controlled exposure. Improve the fastest relevant test feedback in the pipeline. Add canary or blue-green delivery for one critical service. Deploy the pilot behind a flag, validate telemetry in production, and begin the smallest safe exposure that can support the test design.
    • Week 4: Close the loop. Publish one dashboard showing deployment frequency beside change failure rate and MTTR, plus the pilot outcome and experiment status. Hold the readout, record a scale, iterate, stop, or invalid decision, and run a retrospective focused on the next constraint to remove.

    At the end of the month, success is not a dramatic improvement in every metric. Success is evidence that the operating loop works: a baseline exists, a narrow change can move independently, exposure is controlled, decision data is trustworthy, one bet reaches an explicit disposition, and the next bottleneck is visible. That is enough to choose the next product area without pretending the system is already mature.

    Key takeaways

    • Define velocity as time to an evidence-backed product decision, then use deployment frequency as one enabling signal rather than the goal.
    • Pre-register the hypothesis, primary metric, MDE, guardrails, measurement conditions, and decision rule before implementation begins.
    • Separate deployment from user exposure with feature flags and progressive delivery so changes can be small, observable, and reversible.
    • Pair delivery speed with change failure rate and MTTR; pair experiment results with customer, reliability, and trust guardrails.
    • Give a durable product trio authority over the full learning loop, while leaders set outcomes and governance supplies clear boundaries.
    • Start with one product area, complete one question-to-decision cycle, and remove the bottleneck that cycle exposes.

    Take one active roadmap bet tomorrow and ask for its decision rule, MDE, guardrails, exposure plan, and readout owner. If the team cannot write them, do not accelerate the build yet. Fix the question first. Then ship the smallest reversible change that can answer it, record the decision, and use what you learn to make the next cycle safer and shorter.

    References

  • Global Product Manager Playbook: Build Borderless Products, Align Teams, Win Every Market

    Global Product Manager Playbook: Build Borderless Products, Align Teams, Win Every Market

    Products without borders are exhilarating—and unforgiving. In my role leading product strategy, I’ve learned that “global” isn’t a launch plan; it’s a system. It’s the discipline of creating one product vision that flexes to many markets without breaking the core experience, the roadmap, or the business.

    Here’s what a Global Product Manager does, key skills, tools, challenges, and how to grow into this high-impact role.

    At its heart, the Global Product Manager role orchestrates product-market fit in multiple regions simultaneously. I translate a unified value proposition into localized realities—aligning product positioning, go-to-market strategy, pricing and packaging, and compliance—while keeping the platform cohesive. That means partnering closely with product trios, regional leaders, sales, customer success, and marketing to drive outcomes vs output OKRs that actually move the business.

    Operationally, I start with deep product discovery across segments and geographies: what pains are universal, and where do we need regional nuance? From there, I map points of parity we must maintain globally and the differentiators we’ll localize—copy, workflows, payments, support models, and integrations. The art is delivering a consistent core with flexible edges so we can scale without fragmenting the codebase or the customer experience.

    Trust is the non-negotiable. I build privacy-by-design into the product and roadmap, and I collaborate early with legal and security on data governance, data residency, and evolving regulations like GDPR. The right guardrails reduce rework later and enable faster regional launches—because compliance is a feature customers feel, even when they don’t see it.

    On the commercial side, I partner on consumption SaaS pricing, product-led growth motions, and country-level market entry. Some markets need lighter onboarding and in-app guides; others demand concierge support or partner-led distribution. I use retention analysis to identify fit and inform sequencing, then adjust messaging and activation flows to shorten time-to-value and improve user activation by region.

    My analytics and enablement stack is intentionally boring—and ruthlessly consistent. A unified analytics platform with Amplitude analytics gives us comparable funnels across countries. For experimentation, I run A/B testing with a clear minimum detectable effect (MDE) and disciplined rollout plans. Pendo powers product tours and in-app guides tailored by locale, while Intercom and CRM integration with HubSpot help me close the loop with GTM and support teams. The outcome is a learning system, not just a dashboard.

    The hardest part isn’t translation—it’s alignment. Time zones, competing priorities, and matrixed ownership test even strong cultures. I rely on stakeholder management, crisp decision records, and product roadmapping and sprint planning rituals that respect regional input without derailing the global plan. When tension rises, I return to first principles decision making and the try do consider framework to make trade-offs transparent and repeatable.

    If you’re growing into this role, start by owning a multi-region initiative end to end: lead localization for a critical workflow, run market-specific A/B testing with clear MDE, and publish a country launch plan that ties discovery insights to OKRs and resourcing. Build your credibility by shipping outcomes, not artifacts—then scale your impact by mentoring peers and creating shared templates for pricing, positioning, and experimentation. That’s how you shift from capable PM to trusted global operator.

    Ultimately, a Global Product Manager is a force multiplier. We reduce complexity for the organization while increasing resonance for customers. If “products without borders” is your mandate, build the systems—analytics, governance, enablement, and decision-making—that make borderless execution reliable, repeatable, and fast.


    Inspired by this post on Product School.


    Book a consult png image
  • From Chaos to Consistency: How I Built a Scalable AI Content Design Agent with RAG

    From Chaos to Consistency: How I Built a Scalable AI Content Design Agent with RAG

    It’s Monday morning, and my Slack and email are already overflowing with content requests: “Can you review this flow?”; “Can you rewrite this screen?”; “Can you name this feature?” I’m not freshly back from holiday—this is just a regular work week kicking off. If you’ve ever been a solo content designer supporting multiple teams, you’ll recognize the pressure. The pipeline for content in product design is always full, and the demand for expertise never stops.

    Fixing this isn’t just a matter of better time management or incremental process tweaks. To truly scale, I needed to extend my reach by bringing AI into the design process—without sacrificing judgment, standards, or quality. That Monday morning, I realized I had to scale my skills, my judgment, and our systems, not just my calendar.

    Building AI is fundamentally about building systems. I wanted to use AI to scale myself without devaluing critical thinking or flooding the product with generic, verbose content. I also knew a useful AI tool must do more than spit out microcopy—it has to plug into a system we can continually shape. As a content designer, the system is always the starting point. Strong design systems create strong content standards; then AI agents can produce content that meets those standards at speed, freeing me from the bulk of standardized work. That’s not a threat—it’s an advantage. To instruct AI well, our systems must be well constructed.

    I often think about this work like a bakery. You need a recipe before you can make a loaf of bread. Most interface content churns out the same loaf, day in and day out. It’s better for the master bakers to focus on the unique, custom bakes—and how the recipe needs to change. With that mindset, I set out to build an AI content design agent.

    Screenshot of a content design assistant interface titled VERBI, showing a chat input field, quick-start prompts like 'Can you write this?', and links to view permissions and agent setup in draft mode.
    Inside the Content Design Agent workspace, a clean chat UI titled VERBI pairs a central prompt box with chips for writing, editing, and reviews, plus clear controls to view permissions and open the agent setup for product teams.

    When I started this project back in May 2025, many LLMs still had frustrating limitations. Google Gemini let me build a custom Gem agent, but I couldn’t share it with other users. ChatGPT could be customized, but only with static files: I couldn’t point it to live, updatable URL sources. I settled on Glean for three simple reasons: everyone at the company had access; Glean could access all internal documentation and treat URLs as sources of truth; and its then-new Agents feature made AI search customizable. Configuring an agent in Glean is straightforward—you choose a trigger, a set of prompts, and a set of actions—but first I needed to get the inputs right.

    AI agents need focus. We had a wealth of internal information at Intercom, but not all of it was current or reliable. I curated exactly what the agent could access and assembled a tightly governed knowledge collection in Glean. Only essential information made the cut: the Intercom style guide—our definitive house style, including regularly-broken rules like “always write in US English” and “use sentence case everywhere”; tone of voice guidance for how we show up across mediums; a product glossary with hundreds of feature names and writing conventions; a monetization glossary for prices, plans, and add-ons; product marketing messaging guides with positioning for every feature and launch; core research insights across the product; and fin.ai and intercom.com/suite as the official, most up-to-date messaging sources.

    This is classic RAG (retrieval-augmented generation) in action, ensuring every answer is grounded in approved sources of truth. With the collection in place, I instructed the agent to prioritize these resources above anything else.

    Screenshot of a no-code workflow builder for a Content Design Agent, with cards for Trigger, Company search, and Respond, plus a sidebar checklist titled The basics to start from scratch.
    Step into a clean, no-code builder that shows how to assemble a Content Design Agent: kick off with a chat-trigger, run a company search, then respond with expert guidance, all guided by a simple starter checklist.

    Then came the fun part—building and branding the agent. “Content Design Assistant” felt bland, so I named it VERBI, a nod to its “verbal” design job. When people interact with VERBI, they usually begin with a question, but the intent varies widely. I defined a set of task prompts to guide expectations and outputs: “Can you write this?”; “Can you edit this?”; “Can you review this?”; “Can you name this?”; “Give me options”; “Give me guidance”; “Give me strategy”; “Give me research.” This mirrors the real breadth of content design, from creation to critique to discovery.

    To manage responses, VERBI needed three things: start with a specific task prompt; understand how to draw on the right resources each time; and connect with other systems. With task prompts defined, I wrote a detailed system prompt covering the essentials. Role: you are a content designer, supporting product designers. Employer: Intercom (consisting of Fin AI Agent and our next-gen Helpdesk). Resources: content design collection, research collection, Storybook design system. Tone of voice: follow a specific tone for our UI, adjust the tone for everything else. Components: for UI, use the specific guidelines in our design system only. Use cases: writing, editing, critiquing, naming, researching, and more.

    One connection mattered most: our design system, recently rebranded as “Surge.” Surge contains detailed content guidelines for every component in our product UI, from accordions and banners to tabs and tooltips. That granularity took months of human effort to codify, and it paid off. Designers no longer guess how to write for a toggle, a button, or a tooltip—and now VERBI understands and enforces those rules, too. A great content design assistant isn’t just a clever system prompt; it needs deep, component-level guidance to retrieve.

    Design system documentation page for a Badge component, with a left navigation of UI elements and a main panel showing content guidelines, examples of statuses, and a color‑coded table of label types.
    UI documentation showcases the Badge component’s content rules, teaching how to name statuses, define types, and apply color so labels read clearly. A handy visual for building a content design agent and ensuring consistent product messaging.

    Accessing the design system wasn’t simple at first. It lives in Storybook, which Glean couldn’t access directly. I started by scraping guidance from Storybook into an HTML file with Cursor and uploading it to VERBI—a functional but clunky workaround that required re-scraping every few days. Then our IT team stepped in. They used the Glean Indexing API to turn Storybook into a live data source. Now VERBI connects to Storybook directly. Ask it something ultra-specific, like the correct date format for Japan, and it returns the right answer. That integration elevated the agent from helpful to indispensable—human-level precision, 24/7, at scale.

    With prompts and resources in place, I launched VERBI and pressure-tested it. It was accurate and well-informed most of the time, but like any AI agent, it had quirks. I needed it to act as a gatekeeper, not a brainstorming partner that might bend rules or invent new ones. So I added a few explicit guardrails to the system prompt. Stopping sycophancy: “Inform, challenge, and assist. Never placate. Don’t agree by default. If something’s wrong, say so. Challenge assumptions.” Halting hallucinations: “If you don’t find the information required in our resources, say you don’t know the answer. Don’t guess and don’t give answers based on general knowledge.” Avoiding verbosity: “Keep answers short and to the point. Cut the fluff. Skip all niceties and social padding. Only give longer answers if the user asks you to.” These constraints keep responses crisp, correct, and consistent. Like any living system, the prompt needs occasional tune-ups, but the maintenance is minor compared to the upside.

    Where we are now: VERBI has been triggered 700+ times since launch. The benefits are tangible. For me, quality scales without constant policing; repetitive questions about naming, style, or punctuation have dropped significantly. I reclaim time because the agent drafts and checks V1 content across teams, enabling me to focus on higher-impact work. For the design team, iteration is faster, confidence is higher, and strategic clarity improves because shared language and grounded guidelines make decisions easier and more consistent.

    I used to spend too much time mopping up basic content mistakes and untangling spaghetti-like UI copy prone to human error. VERBI removes those errors at the source. The real advantage is speed: we get from blank slate to a high-quality first draft quickly, which means we can spend our energy deciding whether the content is right, not just “good enough.” Design is the whole interface—words, visuals, interactions—so reviews now happen with real content, never “copy TBD.” Our principle to sweat the details applies equally whether work is human-made or AI-assisted.

    Knee-jerk critiques of AI-driven content design often assume teams generate content from nothing and ship it. In reality, great AI is the outcome of great human decisions and strong systems. Its value is pulling us together faster—getting us to a complete, standards-compliant design we can review as a team before sharing it with the world. That’s how AI helps us win: by turning chaos into consistency, and consistency into velocity.


    Inspired by this post on The Intercom Blog.


    Book a consult png image
  • What I Learned from Trainline’s Agentic AI: Building a Trusted Travel Assistant at Scale

    What I Learned from Trainline’s Agentic AI: Building a Trusted Travel Assistant at Scale

    Over the past year, I’ve been shipping agentic AI into production and coaching product teams on what it really takes to make these systems trustworthy in the wild. One story that crystallizes the playbook comes from Trainline’s move to an agentic architecture for travel assistance—an approach that mirrors what I’ve seen work in high-stakes, real-time customer experiences.

    Trainline—the world’s leading rail and coach platform—helps millions of travelers get from point A to point B. Now, they’re using AI to make every step of the journey smoother.

    I studied how "David Eason (Principal Product Manager) Billie Bradley (Product Manager), and Matt Farrelly (Head of AI and Machine Learning)" approached the build of "Travel Assistant, an AI-powered travel companion that helps customers navigate disruptions, find real-time answers, and travel with confidence." Their work exemplifies the kind of end-to-end thinking required to move beyond demos into dependable, on-the-go assistance.

    They share how they: Identified underserved traveler needs beyond ticketing; Built a fully agentic system from day one, combining orchestration, tools, and reasoning loops; Designed layered guardrails for safety, grounding, and human handoff; Expanded from 450 to 700,000 curated pages of information for retrieval; Developed LLM-as-judge evals and a custom user context simulator to measure quality in real-time; Balanced latency, UX, and reliability to make AI assistance feel trustworthy on the go.

    I align strongly with their core takeaways: "AI assistants need both scalable reasoning and deep domain context to be useful." "Tool design and guardrails are as critical as prompt design in agent systems." "LLM-as-judge evals make it possible to measure open-ended systems without massive labeling costs." And perhaps most importantly, "Even legacy companies can move fast when they embrace experimentation and tight PM–engineering collaboration."

    From an AI strategy perspective, starting "fully agentic" was the right call. When the problem space is dynamic—disruptions, route changes, fare conditions—reasoning loops and orchestration aren’t luxuries; they’re table stakes. Tool selection becomes product design: you need the right retrieval interfaces, constraint-aware planners, and API contracts that are resilient to partial failures. Layered guardrails for safety, grounding, and human handoff reduce hallucination risk while preserving responsiveness—critical when users are standing on a platform waiting for an answer.

    The retrieval scale-up—"Expanded from 450 to 700,000 curated pages of information for retrieval"—is a classic inflection point. I’ve seen teams stall here when they treat content growth as a pure indexing problem. The winning move is curation and structure: normalize sources, encode policy-level constraints, and align retrieval chunks to decision boundaries the agent actually uses. That’s how you keep precision high while coverage explodes.

    Evaluation is where most open-ended assistants fail quietly, which is why I was encouraged to see "Developed LLM-as-judge evals and a custom user context simulator to measure quality in real-time." In practice, LLM-as-judge gives you scalable, scenario-based scoring without prohibitive labeling, while a user context simulator surfaces regressions tied to persona, itinerary state, and device constraints. The combination closes the loop between model behavior, tool layer changes, and UX outcomes.

    On product delivery, the decision to have the system "Balanced latency, UX, and reliability to make AI assistance feel trustworthy on the go" shows mature prioritization. For travel, trust accrues in seconds: fast-enough responses, graceful degradation when upstream data lags, and explicit handoff when confidence dips. This is where guardrails meet UX writing—clear, bounded language signals competence even when the system defers.

    Finally, the organizational pattern matters. The teams that win in agentic AI are cross-functional, experimentation-driven, and ruthless about instrumentation. Tight PM–engineering collaboration, explicit safety thresholds, and an eval stack that mirrors real user journeys are what turn promising architectures into dependable products.

    It’s a behind-the-scenes look at how an established company is embracing new AI architectures to serve customers at scale.

    If you’re building agentic AI in production, borrow these moves: invest early in tool and guardrail design, scale retrieval with curation not just volume, adopt LLM-as-judge plus context simulation for continuous evaluation, and treat latency and reliability as core product requirements—not afterthoughts. That’s how you ship AI assistance that customers trust when it matters most.


    Inspired by this post on Product Talk.


    Book a consult png image
  • Context Is King: My Playbook to Prep Product Teams for High-Impact AI Collaboration

    Context Is King: My Playbook to Prep Product Teams for High-Impact AI Collaboration

    Context is king in AI-powered product work—and I felt that deeply while digging into “Context is King – All Things Product Podcast with Teresa Torres & Petra Wille.” The conversation affirmed a truth I see daily: AI becomes a powerful teammate only when we give it the right context, just as we do with empowered product teams. When we treat AI like a colleague joining mid-flight—without our company history, industry nuances, or strategy—we instantly unlock better outcomes.

    Listen to this episode on: Spotify | Apple Podcasts

    Here’s what stood out and how I’m applying it. First, most AI outputs fail without proper context. That’s not a model problem; it’s a leadership problem. Thinking of AI like onboarding a new intern is the right mental model—start with the minimum viable context, then iterate. Practical first steps matter: decision logs, clear success metrics, and structured documentation. The art is balancing enough context to guide performance without overloading the system. The parallels are striking: the way we create strategic context for product trios and teams is the same way we’ll empower agentic AI systems.

    In my teams, we prepare for AI collaboration by operationalizing context. We keep decision logs to capture the why behind choices, use outcome-based success metrics (not just output), and maintain machine-readable documentation that LLMs for product managers can parse reliably. We define guardrails up front—constraints, customer segments, privacy-by-design considerations, and the non-goals that often trip up gen ai. This foundation turns AI from a novelty into a force multiplier for product discovery and product roadmapping and sprint planning.

    I use a simple “context pack” to onboard AI agents and teammates alike: 1) business goals and outcomes, 2) constraints and guardrails, 3) canonical artifacts (like PRDs, journey maps, interview notes), 4) domain vocabulary and definitions, and 5) operating procedures (how we make decisions, when to escalate, what good looks like). Start small, then refine as the AI demonstrates capability. This mirrors great onboarding—and it works just as well for agentic AI as it does for humans.

    Not all context is helpful. More isn’t better; the minimum effective context is. I resist the urge to dump our entire Confluence on an AI system. Instead, I progressively reveal relevant details—just like I would with a new PM on a complex problem space. This keeps signals high, noise low, and performance measurable against clear success metrics.

    If your org isn’t adopting AI yet, don’t wait. You can become AI-ready now by documenting strategic intent, decision rationale, and definitions in structured, searchable, machine-readable ways. Treat this as core AI Strategy work that strengthens empowered product teams—regardless of tooling—while building your AI product toolbox for tomorrow.

    For those who want to explore further, these resources and mentions are a strong complement to the episode’s themes.

    Follow Teresa Torres: https://ProductTalk.org

    Follow Petra Wille: https://Petra-Wille.com

    Agentic AI

    Teresa’s new podcast, Just Now Possible in Youtube, Apple Podcast, and Spotify

    Petra’s Coaching Packages

    ChatGPT

    Henrik Kniberg’s talk at Product at Heart on treating AI agents like interns

    Teresa’s webinars on how she built the Product Talk Interview Coach: Behind the Scenes: Building the Product Talk Interview Coach and How I Designed & Implemented Evals for Product Talk’s Interview Coach

    Josh Seiden’s blog series about AI

    Teresa’s new blog posts: 15 Ways to Use AI at Home (and Fill Your AI Product Toolbox) and 21 Ways to Use AI at Work (And Build Your AI Product Toolbox)

    Petra's new blog post: Why Context, Not Just Data, Will Define AI-Ready Product Teams

    Have thoughts on this episode or how you’re preparing your teams to collaborate with AI? Leave a comment below—let’s compare playbooks and level up together.


    Inspired by this post on Product Talk.


    Book a consult png image
  • How to Build an AI-Powered SaaS Customer Lifecycle

    How to Build an AI-Powered SaaS Customer Lifecycle

    You may already have AI in onboarding, a support agent answering questions, a churn score in customer success, and automated upgrade prompts. Yet the customer still experiences four separate systems. They repeat their intent, receive messages that ignore unresolved problems, and get treated as an expansion opportunity before they have realized the value they bought.

    That is not primarily a model problem. It is a lifecycle design problem. The useful goal is not to put AI at every touchpoint. It is to give each lifecycle decision the right evidence, a permitted action, a measurable outcome, and a clear owner.

    Model the lifecycle as customer value states

    Most SaaS lifecycle maps are organized around internal stages: marketing qualified, sold, onboarded, supported, renewed, expanded. Those labels tell you which team owns the account. They do not reliably tell an AI system what the customer is trying to accomplish or what should happen next.

    Start with customer value states instead. A value state is an evidence-based description of the customer’s current relationship with the product. It should be observable in product behavior, account context, or customer conversations. It should also imply a limited set of appropriate actions.

    Customer value stateEvidence to look forDecision the system can supportOutcome to measure
    Seeking first valueThe intended job or role is known, but the account has not completed its activation milestoneChoose the next necessary setup step, guide, or human interventionCompletion of the activation milestone and time to value
    Establishing repeat valueThe first milestone is complete, but the behavior associated with ongoing value is not yet establishedReinforce the next useful workflow without replaying basic onboardingRepeat completion of the value-producing workflow
    BlockedA failed workflow, unresolved ticket, repeated help request, or explicit expression of confusion is presentDiagnose, resolve, or route the obstacle before sending another growth messageResolution of the underlying problem, including reopen and escalation signals
    Deepening valueMore roles, workflows, or relevant capabilities are being adopted after the core job succeedsRecommend education or adjacent capabilities tied to the customer’s demonstrated needUse of the additional capability and continued core-product value
    At risk of losing valueExpected value behavior has weakened and supporting context points to friction or disengagementForm a risk hypothesis, select a recovery action, or ask an owner to investigateRestoration of the value behavior and cohort retention
    Expansion readyThe account has achieved a defined outcome and has evidence of an additional role, capacity, or capability needPresent an offer that addresses the evidenced needAdoption and realized value after expansion, not merely offer acceptance

    These are templates, not universal definitions. Your activation milestone must represent the first meaningful result promised by your product. Your expansion milestone must demonstrate value and a relevant new need. Mapping activation and expansion milestones to the value proposition keeps automation anchored to customer progress rather than internal funnel activity.

    For each state, write a state contract with six parts:

    • Entry evidence: the events, attributes, or conversations that make the state plausible.
    • Exit evidence: what must become true before the customer moves to another state.
    • Disqualifiers: conditions that suppress an action, such as an unresolved blocking issue.
    • Allowed actions: what AI may recommend, draft, or execute while the customer is in that state.
    • Decision owner: the person accountable for the rule and its outcome, even when execution is automated.
    • Success and guardrail metrics: the intended customer result and the signs that the intervention is causing harm.

    A state should not be inferred from one weak signal. A missing login might indicate friction, seasonality, a role change, or successful completion of an infrequent job. Treat it as an observation until supporting evidence changes the recommended action.

    Build a decision system, not a collection of copilots

    A lifecycle agent needs more than a large prompt and access to several applications. It needs an architecture that turns fragmented customer evidence into controlled decisions. I use five layers to make that architecture explicit.

    1. Identity and permissions: resolve the user, account, workspace, role, plan, and data-access boundary before retrieving context.
    2. Signals: assemble relevant product events, CRM attributes, lifecycle milestones, support conversations, tickets, and prior interventions.
    3. Reasoning: classify the value state, cite the evidence, estimate uncertainty, and choose an allowed next action or abstain.
    4. Action: deliver an in-app guide, answer a question, draft outreach, route work, or request approval according to policy.
    5. Feedback: capture the customer outcome, human correction, escalation, and later state transition so the decision can be evaluated.

    The identity layer comes first because customer records rarely share a clean key. A support conversation may identify a person, product analytics may identify a user and workspace, and the CRM may organize the relationship at the account level. If those entities are joined incorrectly, an otherwise capable model can recommend an action using another workspace’s context or attribute one user’s friction to an entire account.

    Do not place every available field into every prompt. Retrieve the minimum context needed for the current decision, and enforce the permissions of the requesting user and the action-taking service. For teams using Intercom with ChatGPT, the available read-only connection can expose conversations, tickets, and user data while respecting existing Intercom permissions. That is a useful pattern for exploration and decision support: broaden access to relevant evidence without silently broadening write authority.

    The reasoning layer should return a structured decision record, not just fluent text. At minimum, store:

    • The proposed customer value state.
    • The specific evidence used and when it was observed.
    • Contradictory or missing evidence.
    • The recommended action and its expected customer outcome.
    • The policy that permits the action.
    • The confidence or abstention reason.
    • The human or system owner.
    • The condition that makes the recommendation stale.

    This record gives you something an operator can inspect and something an evaluation system can score. It also prevents a recommendation from surviving after the facts change. An upgrade prompt prepared before a serious support issue, for example, should expire when that issue appears.

    The feedback layer must record more than whether somebody clicked. Capture whether the customer reached the intended value state, whether a human changed the recommendation, and whether the intervention created a new problem. A unified measurement layer that connects behavior, funnels, cohorts, retention analysis, and CRM context makes those downstream effects visible across teams.

    Automate the next best decision at each lifecycle stage

    The same architecture can serve onboarding, support, retention, and expansion, but the evidence and acceptable actions differ. Design each motion as its own decision loop.

    Onboarding: optimize for first value, not guide completion

    An onboarding system should know the customer’s intended job, current role, completed setup steps, latest product behavior, and activation milestone. Its task is to identify the next necessary step, not to expose every feature.

    A practical decision rule has four parts:

    • Trigger: an eligible account has not yet reached its defined activation milestone.
    • Action: select an in-app guide, explanation, or human handoff based on the missing prerequisite and observed context.
    • Suppression: stop the guide after activation, an opt-out, a conflicting workflow, or evidence of a blocking issue.
    • Measurement: evaluate activation and time to value, with guide completion treated only as a diagnostic signal.

    A personalized tour can still fail if it teaches a workflow unrelated to the customer’s goal. Conversely, a user can skip the tour and activate successfully. That is why the state transition matters more than interaction with the onboarding surface.

    Support: resolve the problem in its product context

    Support is a strong place to begin because the customer’s intent is explicit, the context is relatively rich, and the result can be observed. Contextual in-app help combined with agentic AI can diagnose an issue, retrieve relevant knowledge, and guide the customer without forcing a channel switch.

    The agent should distinguish among an information gap, a product defect, a permissions problem, a configuration problem, and a request for a capability that does not exist. Each requires a different response. A confident but irrelevant answer can lower ticket volume while leaving the customer blocked, so measure resolution of the problem alongside reopen, escalation, and correction signals.

    Give the support agent a clear escalation packet: the customer’s goal, current screen or workflow, relevant recent actions, retrieved evidence, attempted resolution, and reason for escalation. The human should not have to reconstruct the case from a chat transcript.

    Retention: produce a risk hypothesis, not a churn verdict

    Usage decline by itself is ambiguous. A negative conversation by itself may already be resolved. Combine behavioral change with lifecycle expectations, unresolved friction, account context, and previous interventions before deciding that value is at risk.

    The system’s output should explain what changed, why that change matters for this account, which evidence weakens the hypothesis, and what recovery action is appropriate. If the evidence is weak, the next action may be a review task rather than automated outreach.

    Measure whether the expected value-producing behavior returns and whether retention improves for eligible cohorts. Also inspect unnecessary interventions. A message sent to a healthy customer is not harmless merely because it was automated; it can confuse the relationship and consume customer-success attention.

    Expansion: require proof of value and proof of need

    An account reaching a plan limit is not enough to establish expansion readiness. The system should look for two kinds of evidence: the customer has achieved meaningful value with the current product, and an additional role, capacity, workflow, or capability need is now visible.

    Then match the offer to that need. Suppress it when a blocking support issue is open, the account has not reached its prerequisite milestone, or the evidence is too uncertain. Feature adoption, outcomes achieved, and time-to-value can serve as readiness signals, but your product team still has to define what those signals mean for each offer.

    Do not stop measurement at acceptance. Check whether the customer adopts the added capability and continues to receive core value. Otherwise, the system may optimize for short-term conversion while creating future disappointment, downgrade risk, or avoidable support load.

    Measure customer outcomes and decision quality separately

    AI activity metrics are easy to collect: prompts processed, recommendations produced, messages sent, and conversations deflected. None proves that the lifecycle improved. You need two scorecards.

    The first evaluates decision quality before broader release:

    • State accuracy: does the predicted lifecycle state match the available evidence and the review label?
    • Evidence grounding: can each material claim in the decision be traced to retrieved customer context?
    • Action compliance: is the recommended action permitted for this state, user, account, and channel?
    • Abstention quality: does the system pause when identity, evidence, or policy is insufficient?
    • Human correction: what do reviewers change, and do those corrections cluster around a specific state or segment?

    The second evaluates live customer and business outcomes:

    MotionPrimary outcomeUseful diagnosticGuardrail
    OnboardingEligible customers reaching the activation milestoneWhere the activation path stalls by role or use caseAbandonment, blocking support contacts, and unwanted guide exposure
    SupportThe customer’s problem is resolvedRetrieval quality, escalation reasons, and human correctionsReopens, incorrect actions, and negative feedback
    RetentionValue behavior and cohort retention are restoredAccuracy of risk hypotheses and intervention uptakeUnnecessary outreach and healthy accounts incorrectly flagged
    ExpansionThe added capability is adopted and produces valueReadiness evidence and offer relevanceOpen friction, rapid disengagement, downgrade, or increased support burden

    Define the eligible population and denominator before launch. If an onboarding intervention applies only to administrators pursuing a particular use case, evaluate it on that population. Mixing in ineligible users can make a weak intervention appear safe or a useful one appear ineffective.

    When you run an experiment, specify the randomization unit, primary outcome, guardrails, minimum detectable effect, and stopping rule before looking at results. Segmentation and disciplined A/B testing with a defined minimum detectable effect help distinguish a real lifecycle improvement from movement in a convenient proxy.

    Offline evaluations and live experiments answer different questions. An evaluation tells you whether the system follows policy and makes defensible decisions on known cases. An experiment tells you whether exposing eligible customers to those decisions changes outcomes. You need both before granting more autonomy.

    Start with one closed loop and earn autonomy

    Do not begin with an autonomous agent spanning acquisition through renewal. Choose one recurring decision with rich context, a reversible action, an observable outcome, and a named owner. Support or a narrowly defined onboarding obstacle often meets those conditions.

    1. Write the decision specification. Define the value state, eligibility rule, evidence, disqualifiers, permitted actions, success metric, guardrails, and owner.
    2. Assemble read-only context. Resolve identity and permissions, retrieve only the evidence required, and expose citations to the operator.
    3. Run in shadow mode. Let the system produce decisions without contacting customers or changing accounts. Review errors, abstentions, and missing context.
    4. Move to assistive mode. Allow the system to draft or recommend while an authorized person approves the action.
    5. Review the loop regularly. Examine outcomes, overrides, permission failures, stale recommendations, and differences across eligible segments. A weekly digest of customer-conversation highlights can keep frontline evidence present in product and go-to-market decisions.
    6. Grant scoped autonomy. Automate only the action types that have stable performance, reliable outcome capture, and a safe recovery path. Keep monitoring and a kill switch in place.

    Separate access from authority throughout this sequence. The ability to read an account does not authorize the agent to alter it. Use explicit policies for each action and enforce them outside the model.

    • Informational actions: summarizing evidence, classifying a state, retrieving approved knowledge, or preparing a brief can often remain read-only.
    • Assistive actions: drafting outreach, proposing a guide, or recommending a workflow change should remain subject to review until the relevant decision quality is established.
    • Consequential actions: changing access, contracts, pricing, account status, or customer data can create financial, operational, or irreversible harm. Require an authorized human or a separate deterministic approval workflow rather than relying on model confidence.

    Privacy-by-design is part of product quality here. Minimize retrieved data, preserve existing access controls, define retention for prompts and decision records, and log who or what authorized every write. If the system cannot identify the account reliably or explain the evidence behind an action, it should abstain.

    Key takeaways

    • Organize lifecycle AI around observable customer value states, not departmental handoffs.
    • Require every automated decision to include evidence, an allowed action, an owner, an expiry condition, and a measurable customer outcome.
    • Use AI differently across onboarding, support, retention, and expansion because each motion has distinct evidence and risk.
    • Evaluate decision quality offline, then test customer and business impact on a clearly defined eligible population.
    • Begin read-only, move through assisted execution, and grant autonomy one reversible action at a time.

    Your first move is straightforward: pick one lifecycle decision customers encounter repeatedly and write its state contract. If you cannot specify the evidence, disqualifiers, owner, and outcome on one page, the decision is not ready for an agent. Once that contract is clear, AI becomes an implementation choice instead of a substitute for product judgment.

    References