Author: Shivam Tiwari

  • Agentic AI for Incident Response: A Practical Operating Model

    Agentic AI for Incident Response: A Practical Operating Model

    An incident fires. Your responders are not short of data; they are short of a trustworthy path through it. Deployment timelines, service ownership, dashboards, logs, runbooks, and prior incidents live in separate places, while the cost of a wrong action rises by the minute.

    The decision in front of you is not whether AI can summarize the incident channel. It is whether an agent can shorten the investigation without becoming another failure mode. That requires an operating model covering the agent’s job, context, permissions, interface, and evaluation before you give it meaningful authority.

    Give the agent an investigation job before action authority

    An incident-response agent should run a goal-directed investigation loop, not wait for isolated prompts like a chatbot. A credible implementation can collect context, form and test hypotheses, and draft fixes inside Slack. The important product decision is where that loop must stop for human judgment.

    Model the loop on the work a strong responder already performs:

    1. Scope the incident. Identify the affected service, environment, customer surface, start time, and known symptoms. Preserve unknowns instead of filling them with plausible guesses.
    2. Gather relevant context. Retrieve recent changes, service ownership, dependencies, telemetry, runbooks, feature-flag changes, and similar incidents.
    3. Form competing hypotheses. Produce a ranked set rather than locking onto the first convincing explanation. Distinguish observed facts from inferences.
    4. Test each hypothesis. Use read-only tools to query metrics, logs, traces, deployment state, and dependency health. Record what supports or weakens each possibility.
    5. Propose the next best action. Explain the target, expected effect, risk, preconditions, and recovery path. Do not hide uncertainty behind an authoritative tone.
    6. Update the investigation. Incorporate tool results and responder corrections, discard disproven hypotheses, and choose the next check.

    The incident commander remains accountable for priorities and mitigation. The agent acts as an investigation engine: it gathers, tests, organizes, and proposes. This division is more useful than treating human involvement as a final approval click after the AI has already made every material decision.

    Choose the first workflow with care. A good starting point has a bounded service area, dependable read-only signals, known responders, established runbooks, and outcomes you can verify after the incident. A workflow that depends on undocumented tribal knowledge or unrestricted production access is not ready for agentic automation. Fix the operating system around the incident before expecting a model to compensate for it.

    Do not begin with the most dramatic remediation you can automate. Early value usually comes from reducing context switching, locating the correct owner, connecting symptoms to recent changes, and eliminating weak hypotheses. Those tasks consume scarce attention but do not require the agent to mutate production.

    Context quality determines the ceiling of the investigation

    A capable model cannot reason with operational context it cannot find, distinguish, or trust. If a service has three names across the deployment system, observability platform, and incident channel, retrieval becomes unreliable before model reasoning even begins.

    Create a context contract for every service placed within the agent’s scope. At minimum, make these fields explicit:

    • Identity: canonical service name, aliases, repository, runtime, and environment.
    • Ownership: accountable team, current on-call route, and escalation path.
    • Topology: upstream dependencies, downstream consumers, data stores, queues, and shared infrastructure.
    • Change history: deployments, configuration changes, feature flags, migrations, and rollback state.
    • Operational knowledge: current runbooks, known failure modes, dashboards, alerts, and prior incident records.
    • Control policy: tools the agent may call, environments it may inspect, actions it may propose, and actions it may never execute.

    Start retrieval with exact operational signals. Filter by canonical service, environment, incident time window, deployment identifier, alert type, and ownership tag. Then rerank the surviving records for the current question. This deterministic tagging and reranking foundation is easier to debug than making semantic similarity responsible for every retrieval decision.

    Add embeddings where language actually creates ambiguity: matching an unfamiliar symptom to a differently worded historical incident, finding a relevant paragraph inside a long runbook, or connecting terminology used by two teams. Semantic retrieval should widen discovery, not erase exact boundaries such as production versus staging or one tenant versus another.

    Require every retrieved item to carry provenance that a responder can inspect: its system of record, service and environment, creation or update time, incident-time availability, and reason for retrieval. This lets the responder notice four common failures quickly:

    • A runbook is relevant but stale.
    • An ownership record is current but was different when the incident began.
    • A similar incident came from another environment with different dependencies.
    • A historical evaluation accidentally exposed the final root cause before the agent could have known it.

    Treat missing context as an observable product state. The agent should say that it cannot locate a deployment record or dependency map, identify which system was checked, and propose a safe way to continue. A confident answer assembled around a missing record is more dangerous than an explicit gap.

    Scale permissions to reversibility and blast radius

    Autonomy is not one switch. It is a set of permissions attached to particular tools, targets, environments, and action classes. Granting broad credentials because the agent usually behaves conservatively turns a model-quality issue into a production-control issue.

    Action classAppropriate agent roleRequired human control
    Read-only investigationQuery approved telemetry, changes, ownership, and runbooksAudited access with service and environment boundaries
    Recommendation or communicationDraft a diagnostic check, remediation plan, incident update, or escalationA responder reviews customer-facing messages and consequential recommendations
    Bounded, reversible executionInvoke a preapproved runbook against an explicitly named targetApproval bound to the exact action, target, inputs, and current incident
    Irreversible or broad executionExplain the need and prepare a plan, but do not execute during the initial rolloutExisting change controls and accountable operators remain in force

    Do not label an action reversible merely because the interface contains a rollback button. A deployment rollback can still be unsafe after an incompatible schema or data change. A restart can amplify load or destroy useful diagnostic state. Reversibility has to be validated for the specific service state, not inferred from the action name.

    For every executable tool, define guardrails outside the prompt:

    • Use least-privilege credentials scoped by service and environment.
    • Allowlist tools, targets, and input shapes rather than relying on natural-language prohibitions.
    • Preview the exact command or workflow, target, parameters, and expected effect before approval.
    • Bind approval to that exact action so the agent cannot reuse it for a changed target or plan.
    • Use rate limits, idempotency controls, and circuit breakers where repeated calls could cause harm.
    • Route production changes through existing CI/CD or runbook automation when possible.
    • Record retrievals, tool inputs, tool outputs, approvals, denials, and resulting state changes in an audit trail.
    • Provide a direct way to suspend the agent’s tool access without disabling the incident workflow itself.

    The action proposal should be a control artifact, not a conversational suggestion. It needs the evidence supporting the action, the exact target, the expected observable result, the maximum intended scope, known preconditions, and what the responder will do if the result does not appear. If the agent cannot supply those fields, it has not earned execution authority for that action.

    Keep outward communication on a separate permission path. Drafting a status update is low-risk technically but consequential for customers and the business. Human review should verify what is known, what remains uncertain, and whether the message promises a recovery time the evidence cannot support.

    Make evidence and uncertainty legible in the incident room

    Putting the agent inside the collaboration surface where incidents already unfold reduces the friction of opening another product and re-explaining the situation. It also means the agent’s output competes with urgent human messages. Long narrative answers will be skipped, however intelligent they sound.

    Give each investigation update a stable structure:

    • Observed: facts returned by named systems, with timestamps and links where available.
    • Hypotheses: ranked explanations with the supporting and conflicting evidence for each.
    • Changed since the last update: new evidence, rejected hypotheses, and responder corrections.
    • Next check: the read-only query or tool call most likely to distinguish between the remaining possibilities.
    • Proposed action: target, expected effect, blast radius, preconditions, and recovery path.
    • Decision needed: the specific approval, input, or ownership choice required from a human.

    This is not a request to expose a model’s private, free-form chain of thought. Responders need a structured evidence trail: claims, retrieved signals, tool results, rejected alternatives, and action rationale. That artifact is more useful for review because each part can be checked against the operational record.

    Confidence labels are helpful only when they change behavior. Define what the interface does when confidence is low: ask for a missing service identifier, run another safe check, present multiple hypotheses, or escalate to the owner. Do not display a precise-looking score unless you have evaluated whether that score corresponds to actual correctness in your incident set.

    Design human correction as part of the main workflow. A responder should be able to reject a hypothesis, correct the service or environment, mark a retrieved record stale, deny an action, and state why. The agent should preserve that decision in the incident record and replan from it. Repeatedly resurfacing a rejected hypothesis erodes trust even when the underlying model is otherwise capable.

    Watch for a subtle interface failure: polished summaries can make weak investigations look complete. Make unresolved questions and conflicting signals visually prominent in the message structure. The goal is not to make the agent sound certain. It is to help the incident commander see what is known, what is inferred, and what decision comes next.

    Test against past incidents, then expand authority one boundary at a time

    A demo proves that the agent can complete a favorable path. It does not prove that the agent will retrieve the right context, resist a misleading correlation, respect permissions, or propose a safe action when production is ambiguous.

    Use post-incident time-travel evaluations. Reconstruct what the agent could have known at each point in a real incident. Begin with the original trigger and expose deployments, telemetry, messages, and tool results only when they became available. Hide the final root cause, later analysis, and corrected metadata until the corresponding point in the replay. Otherwise, you are testing hindsight rather than incident response.

    Grade the investigation on operational usefulness, not prose quality:

    • Scoping accuracy: Did it identify the correct service, environment, symptoms, and ownership route?
    • Context retrieval: Did it find the relevant change, runbook, dependency, or earlier incident without mixing incompatible records?
    • Hypothesis quality: Where did the eventual cause appear in the ranked set, and what evidence was used to test it?
    • Evidence integrity: Does every factual claim match a retrieved record or tool result? Did the agent invent a signal that was never observed?
    • Tool correctness: Did it select the correct tool, target, environment, and parameters?
    • Action safety: Was the proposed action inside policy, and were its blast radius, preconditions, and recovery path explicit?
    • Calibration: Did expressed certainty track actual correctness, especially when context was incomplete?
    • Time compression: How did the time to a useful hypothesis, correct owner, mitigation decision, and recovery compare with the existing workflow?
    • Human effort: Which searches, handoffs, repeated explanations, and diagnostic checks did the agent remove or add?

    Treat safety failures differently from diagnostic misses. A missed hypothesis is a capability problem. Crossing a permission boundary, inventing evidence, or targeting the wrong environment is a release blocker for that tool path. Averaging all outcomes into one quality score can conceal exactly the failure that matters most.

    A practical rollout sequence

    1. Instrument the human workflow. Capture incident timelines, ownership changes, diagnostic steps, approvals, mitigations, and outcomes. You need a baseline before claiming improvement.
    2. Replay historical incidents. Use time-bounded context and score the agent against known outcomes. Repair retrieval and service metadata before tuning for eloquence.
    3. Run in shadow mode. Let the agent investigate live incidents without posting conclusions or changing systems. Compare its evidence and hypotheses with the responder’s path.
    4. Expose read-only assistance. Allow responders to request context, hypothesis checks, and draft updates. Collect explicit acceptance, correction, and rejection signals.
    5. Add recommendation mode. Let the agent propose remediations using the structured action artifact, while humans continue to execute through established controls.
    6. Enable one bounded action path. Choose a preapproved runbook with a clear target, validated preconditions, observable effect, and recovery procedure. Keep approval attached to the exact invocation.
    7. Expand by tool and service. Grant additional authority only when evaluation evidence supports that particular boundary. Do not treat success on one service as proof of readiness everywhere.

    Re-run the evaluation set after changes to prompts, models, tools, service topology, runbooks, or permissions. An agent can regress even when its general language quality improves. Operational behavior depends on the whole system around the model.

    Key takeaways

    • Start with investigation and context compression; earn execution authority later.
    • Build deterministic service, environment, time, and ownership filters before depending on semantic retrieval.
    • Separate observed facts, hypotheses, and proposed actions in every incident update.
    • Enforce permissions in tools and infrastructure, not only in prompts.
    • Evaluate with historical time travel so the agent never sees facts that were unavailable during the real incident.
    • Expand autonomy one action, tool, service, and environment boundary at a time.

    The next outage is the wrong time to discover that your agent cannot distinguish a plausible explanation from verified evidence. Before it happens, choose one bounded incident workflow, define its context contract and permission envelope, and replay several real investigations without future information. If the agent can make its evidence legible, stay inside policy, and consistently move responders toward the next correct decision, you have a foundation worth expanding.

    References

  • Build a Company You’ll Run Forever: Bootstrapping vs VC, PMF, and the Art of ‘Eating Glass’

    Build a Company You’ll Run Forever: Bootstrapping vs VC, PMF, and the Art of ‘Eating Glass’

    I’ve spent my career building products and teams that I intend to steward for the long haul, and I’m drawn to founders who treat company-building as a craft you can practice forever. In this analysis, I break down a journey that crystallizes what it takes: going from a teenage wholesale hustle to an API-first healthcare clearinghouse, and in the process, learning why execution isn’t a moat, why venture capital is “going pro,” and how “eating glass” can become a durable advantage.

    Here’s the arc that anchored my thinking: a founder who, at 16, turned $2,500 into a wholesale empire; later bootstrapped a wildly profitable auto-parts business; then sold it to tackle “the most complicated problem” he’d ever encountered: business-to-business transaction exchange. He spent years building EDI infrastructure, threw away the entire codebase eight times, and found extraordinary traction in healthcare. The company recently raised a $70M Series B co-led by Stripe and Addition. The throughline is a consistent, high-agency approach to product management and go-to-market strategy, guided by first principles decision making.

    The first customer is often the trickiest—not because demand doesn’t exist, but because the product’s value proposition, points of parity, and competitive differentiation are still coalescing. I push teams to do founder-led GTM early, speak in the user’s language, and orchestrate high-signal conversations that expose real switching costs. That’s how we avoid mistaking polite interest for product-market fit.

    Bootstrapping forces rigor, but it also means being “constrained by capital.” There’s a ceiling to the speed at which you can iterate, validate, and scale. Venture capital, in the right context, is like “going pro”: you trade a bit of optionality for time, talent density, and a faster feedback loop. I often see confusion between ownership vs. control; structurally, you can design for alignment while still moving with the urgency a competitive market demands.

    One theme I return to with my own teams: execution is never actually a moat. Processes can be copied. Culture can be mimicked superficially. What can’t be easily replicated is the willingness to do the unglamorous, compounding work—what the founder here called “eating glass.” It’s the daily discipline of simplifying the system, instrumenting the edge cases, and standing up operational excellence that compounds into true competitive differentiation.

    When product-market fit hits in enterprise infrastructure, it can feel like “the snake swallowing a deer.” Capacity, process, and architecture are stretched to their limits all at once. I’ve experienced the same pattern: everything slows down so the organization can re-architect for scale. The trick is to make those constraints visible—measure service levels, queuing, and error budgets like you would in a production system—so you’re not flying blind.

    Some of the strongest product-management instincts I’ve seen borrow from discount retail and Toyota. From discount retail, we learn to obsess over unit economics, operational throughput, and ruthless simplification. From the Toyota production system, we adopt Kanban / TPS (Toyota), continuous improvement, and respect for constraints. In software terms, this becomes fast deployment frequency, small batch sizes, and defect prevention at the source—because “All software is a cascade of miracles.”

    Scaling decision-making is where most teams stall. I favor clear ownership, lightweight written narratives, and a bias for first principles decision making over committee compromise. That structure lets high-agency individuals move quickly while keeping cross-functional stakeholders aligned on outcomes vs output OKRs. It’s how you build empowered product teams without sacrificing focus.

    Hiring is where philosophy becomes practice. I resonate with the onboarding mantra “everything’s your fault now”—not as blame, but as an invitation to own outcomes end to end. I look for high-agency people who demonstrate systems thinking and the capacity to simplify. Manager hiring should lag role clarity; bring in managers when coordination overhead is the limiting factor, not when it merely feels uncomfortable.

    Longevity comes from founder-approach fit as much as product-market fit. Build a company you don’t want to leave by aligning operating cadence, decision rights, and cultural norms with how you actually work best. Maintain conviction in unconventional practice when the evidence supports it, while remembering that “Reality has a surprising amount of detail.” The more I zoom in on the real work—interfaces, edge cases, workflows—the more the right design emerges.

    In healthcare EDI, that realism matters. HIPAA overview (HHS) sets the compliance baseline. Payer integrations with Aetna, Blue Cross Blue Shield, and Cigna demand reliability and deep domain fidelity. Cloud and back-office ecosystems—from AWS and NetSuite to Slack, Microsoft Teams, Zapier, and Clay—shape the surrounding workflow. Lessons from Amazon, Target, Walmart, and Costco inform operational rigor; supply chain analogies from Ford Motor Company and GM clarify interface contracts. Porter’s five forces helps frame market structure; perspectives from Jeff Bezos and Peter Thiel sharpen strategic posture.

    If you’re building for the long run, here’s the blueprint I use with product leaders: validate painfully specific jobs-to-be-done before you scale; prefer founder-led GTM until messaging closes the intent-to-adoption gap; instrument throughput and quality like a production system; invest in people who treat ambiguity as a chance to lead; and don’t confuse speed with hurry. When the “snake swallowing a deer” moment arrives, re-architect deliberately, protect your margins, and let operational excellence carry you from product discovery to durable product-led growth.

    References and resources: Aetna: https://www.aetna.com/, Amazon: https://www.amazon.com/, AWS: https://aws.amazon.com/, Blue Cross Blue Shield: https://www.bcbs.com/, Change Healthcare: https://www.changehealthcare.com/, Cigna: https://www.cigna.com/, Clay: https://www.clay.com/, Costco: https://www.costco.com/, Ford Motor Company: https://www.ford.com/, GM: https://www.gm.com/, HIPAA overview (HHS): https://www.hhs.gov/hipaa/index.html, Jeff Bezos: https://x.com/JeffBezos, Kanban / TPS (Toyota): https://global.toyota/en/company/vision-and-philosophy/production-system, Microsoft Teams: https://www.microsoft.com/microsoft-teams, NetSuite: https://www.netsuite.com/, O’Reilly Auto Parts: https://www.oreillyauto.com/, Peter Thiel: https://x.com/peterthiel, Porter’s five forces: https://www.isc.hbs.edu/strategy/pages/the-five-forces.aspx, “Reality has a surprising amount of detail”: https://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail, Slack: https://slack.com/, Stedi: https://www.stedi.com/, Summit Racing: https://www.summitracing.com/, Target: https://www.target.com/, Walmart: https://www.walmart.com/, Zapier: https://zapier.com/


    Book a consult png image
  • Turn Claude Code Into a Trusted Teammate: My 3-Layer Memory System You Can Copy

    Turn Claude Code Into a Trusted Teammate: My 3-Layer Memory System You Can Copy

    "Can you critique the landing page for my new Story-Based Customer Interviews course?" That simple ask used to kick off hours of back-and-forth where I fed an AI the same context over and over—only to get generic feedback that wouldn’t land with my audience or fit my products. As a product leader, that inefficiency was unacceptable; as a writer, it was just plain frustrating.

    Not anymore. Today, Claude not only critiques my work, it helps me produce it. It generates marketing copy—in my voice. It helps me write blog posts. It knows what search terms are relevant to my business and helps me optimize my articles for SEO and now AEO. It helps me with competitive research, academic research, and discovery research. And it does all of this with little prompting from me.

    I don’t upload files to a web-based project. I don’t manage elaborate prompt libraries. I don’t repeat myself. I ask for help and Claude knows exactly what to do. The shift happened when I learned how to give Claude Code a memory. Claude now knows who my target customer is, the key value propositions I focus on, the specific opportunities each product addresses, my revenue model, my marketing channels, and so much more.

    Dark-mode slide with monospaced white text outlining an SEO plan: add CLAUDE.md to an AI glossary as the entry point, with bullets on article focus, audience, and search architecture for Give Claude Code a Memory.
    A dark-themed strategy slide for the post Stop Repeating Yourself: Give Claude Code a Memory, showing how to lead with a CLAUDE.md glossary page, write clearly for nontechnical readers, and link glossary and article to boost discovery and engagement.

    With that memory, I consistently get high-quality output tailored to my audience and aligned to my products and services. I don’t retype the same context; Claude just remembers. In this article, I’ll show you exactly how I set up that memory. It relies on Claude Code (which requires a Pro subscription), and it’s worth it. If you’re new to Claude Code, start with "Claude Code: What It Is, How It’s Different, and Why Non-Technical People Should Use It."

    Here’s the underlying problem: with large language models, every conversation starts from scratch. Yes, ChatGPT can remember some things and Claude can search past conversations, but practically speaking each new thread wipes the slate clean. If I were working on a new landing page, I’d normally need to upload target customer context, product details, primary and secondary value propositions, FAQ questions and answers, plus testimonials and logos for social proof—every single time.

    Dark-theme screenshot of the Claude interface with a large prompt field, model selector set to Sonnet 4.5, and quick-action buttons for Write, Learn, Code, Life stuff, and Claude’s choice on the home screen.
    Start fast with Claude’s home screen: Sonnet 4.5 is ready, and quick actions for writing, learning, and coding sit beneath a clean prompt box—ideal for showing how memory cuts repetition and streamlines daily development.

    Projects in web-based tools help a bit, but they introduce a new dilemma. When I move to the next landing page targeting the same customer but a different product and value proposition, do I start a new Project (tedious) or keep expanding the old one (which muddies the context window and degrades output quality)? The good news: Claude Code solves this by giving the model a precise, durable memory without overloading any single conversation.

    Claude Code can read files on my local machine, which is an understated superpower. I use those files to create a persistent, reusable memory that works across all chats and Projects. Files can be mixed and matched, so I give Claude exactly what it needs for the task at hand—and nothing more. For a first landing page, I reference the target customer and the relevant product; for the second, I reuse the same target customer file and point to the new product file.

    Screenshot of a macOS Notes window in dark mode showing an AI-assisted review of producttalk.org, listing Fetch and Read steps and a "Homepage Evaluation" for a first-time B2C visitor.
    Dark-mode Notes screenshot captures Claude Code in action: it fetches producttalk.org, reads context files, and delivers a concise homepage evaluation—showing how memory streamlines repeated analysis tasks.

    When you give an LLM the exact right context, output quality jumps. More context only helps if it’s the right context. For a landing page, Claude needs to know about the current product and perhaps related products for differentiation—but it doesn’t need to know about unrelated offerings. Structure your memory so Claude gets precisely what’s required.

    Once I did this, Claude shifted from “intern who needs handholding” to trusted advisor and capable teammate. It doesn’t guess at my value propositions—I’ve already told it. It writes in my voice because it has my writing guide and samples. It knows who owns which course and which use cases map to which features. The setup takes a bit of upfront work, but it compounds: update a file when something changes and you’re done. Most of this information already lives in your system; the trick is making it easy for Claude to use.

    Diagram of the Claude Code interface with a terminal-style dashboard. Arrows show Global Preferences (~/.claude/CLAUDE.md), Project Preferences (Project/CLAUDE.md), and Custom Files feeding memory into the coding chat.
    See how Claude Code stops repetition: global and project CLAUDE.md files, plus custom reference docs, flow into the editor so the assistant remembers your preferences and context while you code and run commands.

    Because the files live on my machine, I own the system. No vendor or device lock-in. I decide when and who to share with. I can work with Claude on one project and ChatGPT on another—both can rely on the same file-based memory strategy. It’s an AI strategy that scales with product discovery, accelerates go-to-market content, sharpens competitive differentiation, and supports product-led growth.

    Here’s how I design the memory: I use three layers. Claude Code already encourages global preferences and Project-specific instructions, but the third layer—reference context—is where the real power lives.

    Dark-mode screenshot of a macOS editor showing a 'Claude Code Preferences' markdown file with sections on writing conventions, planning protocol, and feedback for collaborating with Claude.
    Peek inside a markdown playbook for Claude Code: concise rules for writing, multi-level planning, and clear feedback that turn repeated reminders into reusable memory and smoother, faster coding sessions.

    Layer 1: Global Preferences (Always on). The first time I launched Claude Code, I created a CLAUDE.md file at ~/.claude/CLAUDE.md. This is where I keep the cross-project rules of engagement—how I like to work with Claude. Mine includes: Always create a plan for me to review before you start any work; Give me direct feedback (no hedging, no gentle suggestions); Use bullet points for summaries; Ask clarifying questions one at a time so I can give complete answers; No emojis unless I explicitly ask for them. Claude Code automatically loads this file at the start of every session, so I never restate my preferences.

    Layer 2: Project-Specific Instructions. Different projects have different rules. In my writing workspace, the Project CLAUDE.md sets the roles (I’m the primary writer; Claude is my thought partner and editor), defines a multi-round review flow (content → structure → accuracy → typos), prioritizes human readability over SEO, and points to my writing style guide. In my task management system, I include how my Trello integration works, file naming conventions for tasks, and how to process research papers into summaries. In my code projects, I specify the technology stack (Node.js vs. Python), testing framework (Jest for Node.js, pytest for Python), code style and conventions, project architecture and directory structure, and which dependencies and libraries to use. Each project directory has its own CLAUDE.md, and Claude automatically loads the relevant file when I’m working there.

    Dark-themed text editor screenshot of a markdown file titled 'Claude Instructions,' featuring sections for session setup, working relationship, editor responsibilities, and research and development guidelines.
    Peek inside a markdown playbook for collaborating with Claude—covering session setup, roles, editorial standards, and research steps—to show how saved instructions create consistent results without repeating yourself.

    Layer 3: Reference Context (Pull as Needed)—the real power. LLMs have a context window—a limit to how much they can process at once. Even within that limit, loading too much degrades performance due to “context rot.” The remedy is ruthless context management: small, targeted files that load only when needed. Keep CLAUDE.md files concise and focused on rules and workflows. For detailed knowledge, create separate reference files and list them in your CLAUDE.md so Claude knows they exist and when to fetch them. When I ask for help creating a landing page, Claude knows to use my business profile, the product file, and my target customers context.

    Here’s what most people miss: you don’t cram everything into global or Project files. You maintain small, reusable reference files that Claude only loads on demand. In my walkthrough, I share exactly which context files I created and why; how I got Claude Code to help me create them; how I break them into small, reusable components so Claude gets precisely what it needs; how I keep everything up to date; and step-by-step instructions so you can set up a similar memory system.

    Diagram of three markdown files (business-profile.md, story-based-customer-interviews.md, target-customers.md) feeding into a Claude Code IDE panel, showing context files powering an AI assistant.
    Three project notes funnel into Claude Code, turning reusable context into working output. This visual shows how saving key docs as memory lets the AI pick up where you left off and skip repetitive prompting across tasks.

    Let’s dive in.


    Inspired by this post on Product Talk.


    Book a consult png image
  • How to Build a Product Positioning and Messaging Strategy

    How to Build a Product Positioning and Messaging Strategy

    Your homepage promises an all-in-one platform. The sales deck leads with automation. The product demo focuses on analytics. Each claim may be true, but together they force the buyer to work out what you are, whom you serve, and why you matter. That is a positioning failure, not a copy problem.

    The way out is to separate the strategic choice from its expression. First decide which customer and buying situation you intend to win. Then build a messaging system that carries that decision from the first impression through sales, onboarding, and product use. The method below gives you the artifacts, tests, and operating rules to do both.

    Separate the strategic decision from the words

    Positioning, messaging, and copy are related, but they solve different problems:

    • Positioning decides the target segment, urgent customer job, category, primary alternative, promised outcome, meaningful difference, and proof.
    • Messaging decides which parts of that position to emphasize, in what order, for each audience and stage of the buying journey.
    • Copy turns the message into a headline, sales talk track, pricing-page explanation, onboarding prompt, or product-tour step.

    This distinction tells you where to intervene. If leaders disagree about the customer or alternative, a headline workshop will only conceal the disagreement. If the position is clear but buyers do not understand it, the messaging hierarchy needs work. If the hierarchy is sound but one page underperforms, you may have a copy or execution problem.

    I use a simple diagnostic: ask the product, marketing, sales, and customer-success owners to complete the following prompts independently. Do not let them discuss wording first.

    • The customer I most want to win is…
    • They look for a solution when…
    • The progress they need is…
    • They would otherwise use, assemble, or tolerate…
    • They should choose this product because…
    • The evidence that makes that claim credible is…

    Compare the nouns and decisions in the answers, not their polish. If one person names agencies, another names sales teams, and another says any growing business, you do not have a shared target. If the alternatives range from a direct competitor to spreadsheets and doing nothing, the team is framing different buying decisions. Resolve those differences before approving new copy.

    The output of this diagnosis should be a short list of strategic questions, each with an owner and an evidence gap. That is far more useful than a document full of compromise language.

    Build the position from evidence, not ambition

    Choose a segment that behaves like a good customer

    A broad market description is not a target segment. Modern teams, small businesses, and enterprises are labels, not choices. A usable segment combines a buyer or user, an operating context, a trigger, and a need that is unusually important in that context.

    Start with behavioral evidence from activation, retention, and expansion. Look for cohorts that reach meaningful value, continue using the product, and deepen their commitment. Then investigate why. A large cohort that requires heavy persuasion and struggles to retain may be a less attractive positioning target than a smaller cohort that recognizes the problem immediately.

    Write a segment brief with four fields:

    • Who: the buying role, user, or accountable leader.
    • Context: the company type, workflow, maturity, or constraint that changes the value of the product.
    • Trigger: the event that turns a background inconvenience into a priority.
    • Exclusion: a plausible customer for whom the product is not the best fit.

    The exclusion is important. If you cannot say who should not buy, the segment is probably still too broad. Specificity does not make the total market disappear. It gives your message a place to land.

    Name the progress, not the product output

    Customers do not wake up wanting a dashboard, an AI assistant, or another system of record. They want to make a decision sooner, remove a risky handoff, create predictable pipeline, reduce manual work, or gain control over an outcome they already own.

    Complete this sentence using the customer’s language: After adopting this product, the customer can do what they could not do reliably before? The answer should describe progress in the customer’s world. A capability belongs in the explanation of how the result happens, not in the result itself.

    Tie the promise to a business result the customer already tracks, but do not add a number merely to make the claim sound concrete. A quantified promise requires evidence that supports the same segment, use case, and conditions. Until you have that evidence, state the direction of value plainly and use verified proof lower in the message.

    Define the category, alternative, difference, and proof

    The buyer needs a familiar frame before your differentiation can matter. A category tells them what kind of decision they are making. Points of parity tell them you meet the minimum conditions for consideration. Differentiation tells them why you should win after you qualify.

    DecisionQuestion to answerCommon failure
    CategoryWhat familiar kind of solution is this?Inventing a label the buyer must decode before understanding the product.
    Points of parityWhat must be true for the product to make the shortlist?Leading with table stakes as if they were differentiation.
    AlternativeWhat would the customer use, assemble, or tolerate without this product?Assuming the only alternative is a named competitor.
    DifferentiationWhich valuable outcome or mechanism is meaningfully better?Using adjectives that any competitor could copy.
    ProofWhat evidence supports the exact claim?Offering confidence, popularity, or technical detail that does not prove the promise.

    The primary alternative may be a competitor, a generic platform, a manual workflow, a collection of tools, or the decision to do nothing. Name the one that appears in the buying situation you are targeting. Your differentiation is meaningful only in relation to that alternative.

    Proof can take several forms: measured customer outcomes, time-to-value evidence, product behavior, implementation evidence, data-governance controls, privacy-by-design, or cybersecurity commitments. Match the proof to the anxiety created by the claim. If you promise speed, prove speed. If you promise control, prove governance. A list of impressive but unrelated facts will not close the credibility gap.

    Use this compact positioning structure once those choices are clear:

    For [specific customer in a defined context] who needs [urgent progress], [product] is a [familiar category] that delivers [customer outcome]. Compared with [primary alternative], it [meaningful difference], supported by [relevant proof].

    Positioning statement template

    Treat every bracket as a decision, not a place for the most flattering phrase. Mark each clause as evidence, assumption, or aspiration. Evidence can enter the approved statement. An assumption becomes a test. An aspiration belongs in product strategy until the product and proof can support it.

    Before moving on, apply six checks:

    • Does the segment exclude anyone you could plausibly sell to?
    • Would the target customer recognize the triggering problem?
    • Does the category reduce the explanation burden?
    • Is the alternative one customers actually consider?
    • Would the difference still matter if a competitor copied the wording?
    • Does the proof establish the claim rather than merely decorate it?

    If a competitor can paste your statement onto its homepage without changing the meaning, you have described the market, not your position.

    Turn one position into a messaging system

    A positioning statement is an internal decision tool. It is rarely the exact sentence that should appear on every customer-facing surface. Buyers need the same strategic story expressed at different levels of depth.

    Build the message in this order:

    1. Category cue: help the buyer place the product on a familiar mental shelf.
    2. Core outcome: state the progress that makes the product worth considering.
    3. Mechanism: explain how the product creates that outcome differently from the alternative.
    4. Proof: supply evidence for the claim and mechanism.
    5. Objection response: address the trade-off, risk, or missing parity point most likely to stop the decision.
    6. Next step: ask for an action that fits the buyer’s current level of intent.

    This order prevents two common errors. Leading with features makes the buyer infer the value. Leading with a grand outcome and no mechanism makes the claim sound ungrounded. The combination of outcome, mechanism, and proof gives the message both relevance and credibility.

    For an intent-data product, a message unit could work like this:

    • Claim: Act on buying intent while it is still useful.
    • Mechanism: Translate live product-usage signals into prioritized opportunities and the appropriate next action.
    • Proof: Insert only verified evidence, such as observed time to value, measured conversion results, documented governance, or customer validation.

    The example does not need faster, smarter, seamless, or revolutionary. Those words add no information unless a mechanism and evidence give them a precise meaning.

    Next, create a message map for each audience that participates in the decision. Use the same position, but change emphasis:

    • Economic buyer: business consequence, strategic fit, financial logic, and adoption risk.
    • Operational user: workflow improvement, usability, time to value, and what changes in the working day.
    • Technical or trust evaluator: integration, data handling, governance, privacy, security, and operational control.

    For each audience, record the trigger, desired outcome, current alternative, core claim, supporting mechanism, accepted proof, likely objection, and appropriate call to action. That becomes the brief for a landing page, demo, campaign, or onboarding flow.

    Do not create a new position for every persona. If an executive hears an efficiency story, an operator hears a feature story, and a technical evaluator hears an infrastructure story with no common outcome, the account receives three products. Keep the strategic claim stable and translate the consequence, mechanism, and proof for the listener.

    Consistency does not mean identical copy. It means every message helps the customer reach the same conclusion about whom the product is for, what it changes, and why it is the better choice.

    Test for customer movement, not internal applause

    A message that wins a leadership vote has passed a preference test. It has not passed a market test. Validation should show whether the intended customer understands the position, believes it, and takes a more valuable next step.

    Write the hypothesis before changing the asset:

    For [target segment] at [journey stage], emphasizing [message decision] instead of [current framing] will improve [customer behavior] because [expected change in understanding or motivation].

    Messaging experiment hypothesis

    Then run the test with the following controls:

    1. Capture the current baseline and the audience definition.
    2. Change one meaningful message decision, not the message, design, offer, and traffic source at the same time.
    3. Choose a primary metric that reflects progress at that stage of the journey.
    4. Add guardrails for downstream quality, retention, or unwanted customer mix.
    5. Set the decision rule before reviewing the result.
    6. Record what changed, what happened, for whom it happened, and what the result does not establish.

    The metric must match the surface:

    • Acquisition page: qualified conversion is more useful than raw visits or attention.
    • Sales conversation: look for clearer problem recognition, fewer category misunderstandings, relevant objections, and progression to the agreed next step.
    • Onboarding: measure activation and completion of the behavior tied to the promised value.
    • In-product message: measure the meaningful action after the prompt, not merely a tooltip click.
    • Expansion motion: look for adoption and commercial movement in the segment the message was intended to reach.

    A higher click-through rate with weaker qualified conversion is not a positioning win. It may mean the new wording creates curiosity but attracts the wrong expectation. Follow the behavior far enough to see whether the message improves customer fit rather than only top-of-funnel volume.

    You can pressure-test messaging across landing pages, onboarding flows, in-app guidance, sales talk tracks, and nurture sequences. Amplitude can help inspect behavioral cohorts; Pendo and Intercom can support in-product delivery and measurement; HubSpot can connect lifecycle messages with funnel behavior. The tool is secondary to a clean hypothesis and a metric that reflects the decision you are trying to improve.

    If traffic is too limited for a reliable A/B test, use customer interviews, comprehension checks, sales-call analysis, and structured message reviews to learn why language works or fails. Treat that evidence as directional. Interview feedback can reveal confusion, relevance, and objection mechanisms, but it should not be relabeled as causal conversion lift.

    Keep a decision log. For every experiment, store the segment, surface, control, variant, hypothesis, primary metric, guardrails, result, interpretation, and next decision. Without that record, teams repeatedly test synonyms while forgetting the strategic assumption underneath them.

    Read results diagnostically. A message that improves acquisition but not activation may be setting an expectation the product does not fulfill. A message that works for one retained cohort but fails for another may reveal that the target segment is too broad. A claim that repeatedly requires explanation may indicate a poor category choice. The purpose of testing is not to defend the original language; it is to improve the decision system.

    Make positioning part of the product operating system

    Positioning decays when it lives only in a launch deck. Sales adapts the story to objections, marketing optimizes individual campaigns, product ships capabilities, and onboarding inherits old promises. Each local choice can seem reasonable while the overall narrative drifts.

    Create one canonical positioning brief with:

    • An accountable owner, version, approval date, and current validation status.
    • The target segment, trigger, and explicit exclusions.
    • The urgent job and customer outcome.
    • The category and required points of parity.
    • The primary alternative and competitive difference.
    • Approved proof for each claim, including any conditions or limits.
    • The message hierarchy and audience-specific message maps.
    • Known objections, prohibited unsupported claims, and open assumptions.
    • Links to experiment results and the decisions they changed.

    The brief should govern product decisions as well as communication. When reviewing roadmap work, ask whether the item strengthens the promised outcome, closes a parity gap that blocks consideration, compounds the reason to choose the product, or creates proof for a claim customers already value. Work that does none of these may still be necessary, but it needs a different strategic justification.

    This prevents differentiation from becoming a slogan unsupported by investment. If the product claims a uniquely fast path to value while roadmap decisions add setup complexity, the market will eventually believe the experience rather than the headline.

    Roll the position through the connected customer journey. Update the homepage, pricing explanation, sales discovery, demo narrative, onboarding, product tours, in-app guidance, customer-success materials, and nurture sequences that rely on the old framing. Prioritize the surfaces where the intended segment makes or validates its decision. A new promise on the homepage paired with an old demo and unrelated onboarding creates more confusion than a controlled, coherent rollout.

    Give one owner authority to maintain the canonical brief, while making product, marketing, sales, and customer success responsible for contributing evidence. Version meaningful changes. A headline iteration does not require a new strategic version; changing the target segment, category, alternative, outcome, or differentiation does.

    Review the position when evidence changes, not merely because the calendar says it is time. Useful triggers include a major product launch, entry into a new segment, a shift in the alternative customers choose, a parity gap that changes shortlist eligibility, new proof that strengthens the promise, or a persistent mismatch between acquisition, activation, retention, and expansion.

    Do not rewrite the position after every losing copy test. A failed expression and a failed strategic premise are different diagnoses. Change the position only when the evidence shows that the customer, problem, category, alternative, promise, or reason to believe has changed.

    Key takeaways

    • Resolve disagreements about the customer, buying trigger, category, and alternative before debating headlines.
    • Choose a segment using activation, retention, and expansion behavior, then document whom the position excludes.
    • Build every major message from an outcome, a distinctive mechanism, and proof that supports the exact claim.
    • Test messaging against meaningful customer behavior and downstream quality, not internal preference or clicks alone.
    • Use the approved position to guide roadmap trade-offs, go-to-market assets, onboarding, and future experiments.

    Take your current positioning statement and label every clause as evidence, assumption, or aspiration. Pick the assumption that would most change the strategy if it proved false, and design the next customer or behavioral test around it. Validate the decision before rewriting every surface. Once it holds, carry the same position all the way into the product experience.

    References

  • The Product Playbook: Measuring Agent Performance with Pendo and Agent Analytics to Drive ROI

    The Product Playbook: Measuring Agent Performance with Pendo and Agent Analytics to Drive ROI

    I treat agent performance analytics as a strategic product lever, not a back-office metric. When I combine Pendo’s product signals with Agent Analytics from our support systems, I get a unified view of where users struggle, how agents intervene, and which in-app experiences accelerate resolution. That visibility lets my team drive product-led growth and improve customer experience while lowering support costs.

    Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

    In practice, I build a clear scorecard that blends both product and support KPIs: first response time, resolution rate, first contact resolution, CSAT, containment/deflection rate, average handle time, ticket volume per active account, onboarding completion, user activation, and time-to-value. This balanced view ensures we reward not just speed, but durable outcomes that reduce repeat contacts and improve retention.

    To make the data actionable, we connect our CRM integration, ticketing events, and Pendo product analytics in a unified analytics platform. That gives me cohort-level clarity—who needed help, what they were doing before opening a ticket, how agents responded, and whether users stayed engaged afterward. With clean instrumentation and consistent taxonomies, Agent Analytics becomes a reliable operating system for both product and support leadership.

    I then use in-app guides, tooltips, and product tours to proactively address the top friction points that drive ticket volume. Through A/B testing, we compare cohorts exposed to guided workflows versus control groups, measuring deflection, faster task completion, and downstream conversion. When a guide meaningfully reduces tickets for a given workflow, we promote it from experiment to standard onboarding, and we feed those learnings back into our roadmap.

    The real unlock comes from tying outcomes to business impact. I track how improvements in resolution quality and self-serve adoption influence expansion revenue, support cost per account, and risk signals like churn propensity. Retention analysis helps us validate whether reduced friction and better agent coaching translate into sustained engagement and healthier accounts.

    Operationally, Agent Analytics helps me coach teams with precision. I spotlight high-performing behaviors, identify knowledge gaps, and standardize winning playbooks directly in the product via in-app guidance. This approach empowers agents, shortens onboarding for new hires, and keeps our best practices current as the product evolves.

    None of this works without trust. We apply privacy-by-design principles and strong data governance, ensuring that analytics, coaching, and automation respect user consent and data minimization standards. With that foundation, we can scale confidently—experiment faster, learn from every interaction, and continuously improve the software experience.

    If you’re getting started, begin by baselining your agent and product KPIs, ship one high-impact guide to deflect a top ticket driver, and review results weekly. Within a quarter, you’ll have a repeatable loop: diagnose friction, test an in-app solution, measure deflection and satisfaction, and reinvest the gains into the next set of improvements.


    Inspired by this post on Pendo – Best Practices.


    Book a consult png image
  • AI at Home, Impact at Work: Experiments That Supercharged My Product Leadership

    AI at Home, Impact at Work: Experiments That Supercharged My Product Leadership

    I recently tuned into an insightful All Things Product episode featuring Teresa Torres and Petra Wille on how experimenting with AI in everyday life sharpens how we build AI-powered products at work. The core premise resonated deeply with my AI Strategy: low-stakes, personal experiments accelerate confidence, clarify limitations, and build an AI product toolbox we can bring into the office with rigor.

    If you want to dive in, you can listen on Spotify or Apple Podcasts. I found the conversation especially relevant for product trios and anyone shaping LLMs for product managers in high-stakes environments.

    The idea is simple but powerful: when I prototype with AI at home—where the stakes are low—I learn faster, make safer mistakes, and internalize critical product patterns. Over time, those patterns transfer directly to work: tighter context management, sharper bias awareness, clearer human-in-the-loop guardrails, and a more nuanced view of when to use AI as a thought partner versus when to consider agentic AI.

    In my own practice, I’ve mirrored many of the scenarios discussed: using ChatGPT by OpenAI to plan meals, analyze public data sets like school budgets, and even sanity-check real estate evaluations. These seemingly mundane tasks are fertile ground for learning about context window limits, hallucination (artificial intelligence), AI bias, and privacy-by-design trade-offs. Each experiment helps me craft better prompts, structure data for clarity, and decide when a human review step is non-negotiable—core habits for AI risk management.

    At work, I treat AI as a thought partner for writing, research synthesis, and contract review. I also explore when and how to responsibly evolve toward agentic AI for repeatable workflows. The distinction matters: a thought partner augments judgment; an agent automates execution. Building the right scaffolding—data governance, auditability, constraints, and escalation paths—ensures we unlock speed without compromising safety.

    Three lines from the episode stayed with me: “I’m trying to write things that only I can write — that’s my guiding writing light right now.” — Teresa. “The more we use AI, the more we learn what it’s good at, what it’s not good at, and where context becomes a limitation.” — Teresa. “It’s a safer playground — we can build our toolbox at home before bringing those lessons to work.” — Petra. These are practical north stars for product management leadership in the GenAI era.

    For anyone getting started, here’s what worked for me: begin with “low-stakes” personal experiments, write down your prompts and outcomes, and reflect on failure modes. Treat each activity as product discovery: What problem am I solving? What outcome matters? What data and context does the model need? Which decisions must stay human-in-the-loop? This discipline builds an AI product toolbox you can confidently apply to real customer problems.

    I also keep a running toolkit of references and tools that inform my practice: Context window as a concept helps me size and sequence information. Visual and video tools like Midjourney and Sora expand how I think about multimodal experiences. I rotate between Claude by Anthropic and ChatGPT by OpenAI depending on task fit, and I’ve used Claude Code when I need structured assistance with code review. For knowledge capture and workflow, Readwise and Ghost help me structure insights and ship content.

    If you want more structured learning paths, I found Josh Seiden’s Learn AI With Me, A 30-Day Sprint to be a practical primer, and the broader community conversation at Product at Heart Conference is invaluable. For a deeper grounding in risk, I recommend reviewing topics like Hallucination (artificial intelligence), AI bias, and Agentic AI—and revisiting the complementary episode, Context is King.

    I’d love to hear how you’re experimenting: Where have you seen AI meaningfully reduce toil? Where does it still struggle? How are you balancing creativity, data safety, and compliance as you scale? Drop a comment below and let’s compare notes—especially on patterns that help product trios move faster without sacrificing trust.

    Bottom line: start small at home, carry lessons into the office, and build with curiosity and intentionality. That’s how we level up our product discovery, sharpen our value proposition, and lead teams confidently through the GenAI transition.


    Inspired by this post on Product Talk.


    Book a consult png image
  • From Engineer to Product Manager: A Practical Transition Plan

    From Engineer to Product Manager: A Practical Transition Plan

    You may already be doing the parts of engineering that sit closest to product management: questioning a requirement, clarifying the user problem, challenging an unnecessary feature, or helping design and product make a difficult trade-off. The uncertainty is whether those moments add up to PM readiness – and whether changing careers means discarding the technical credibility you worked hard to earn.

    They don’t prove that you’re ready, but they give you a strong starting point. The safest path is to test the role before you depend on the title. Own a bounded customer problem, work through discovery and prioritization, ship a small bet, and make the resulting evidence visible. That gives you a transition plan based on demonstrated product judgment rather than potential alone.

    Change the scoreboard from implementation to impact

    Engineering and product management overlap, but they aren’t measured the same way. An engineer is expected to make a solution reliable, maintainable, secure, and feasible. A PM is expected to determine which problem deserves attention, why it matters now, what evidence supports the decision, and how the team will know whether its bet worked.

    The first transition is therefore moving from shipping outputs to driving measurable user or business outcomes. That doesn’t make delivery unimportant. It changes the role delivery plays: a feature becomes a hypothesis about how to create value, not the finish line.

    When you encounter a request such as “build bulk editing,” don’t start by turning it into tickets. Rewrite it as a product decision:

    • User and context: Which segment encounters the problem, and during which workflow?
    • Observed problem: What are people trying to accomplish, and where does the current experience fail them?
    • Current behavior: What workaround or alternative do they use now?
    • Desired outcome: Which user or business measure should change if the problem is solved?
    • Hypothesis: Why should this particular intervention change that measure?
    • Smallest useful test: What can you ship or simulate to reduce the most important uncertainty?
    • Decision rule: What evidence would make you continue, change direction, or stop?

    This framing exposes weak roadmap items quickly. If you can’t identify the affected segment, current behavior, baseline signal, or decision rule, the team doesn’t yet have a product bet. It has a solution looking for justification.

    Technical depth remains useful. You can detect hidden dependencies, challenge unrealistic scope, and understand where platform choices restrict future options. The trap is allowing feasibility to dominate desirability and business value. A solution can be technically elegant, delivered on time, and still leave the customer problem untouched.

    Run a 90-day transition experiment in your current role

    An internal move is usually easier to de-risk because you already understand the product, architecture, delivery process, and organizational context. Instead of asking your manager to approve a permanent career change based on intent, propose a bounded 90-day product experiment with an outcomes dashboard and a weekly stakeholder update.

    Choose a problem that matters but doesn’t require control of the entire roadmap. It should have an identifiable user, an observable pain point, a plausible measure of success, and enough room for a small intervention. Avoid a project whose scope is already fixed. Coordinating predetermined delivery may demonstrate execution, but it gives you little opportunity to show discovery, prioritization, or product judgment.

    PhaseWork to ownEvidence to preserve
    First 30 daysMap the users, workflow, current alternatives, relevant metrics, stakeholders, and decision process. Define the problem boundary and establish the baseline signal.A one-page problem brief, workflow map, initial dashboard, interview plan, and written scope.
    By day 60Run focused discovery, combine interview patterns with quantitative signals, compare possible interventions, and build a hypothesis-led roadmap.Discovery notes, customer language, an opportunity tree, rejected options, trade-offs, and a prioritized experiment.
    By day 90Deliver a thin slice, observe the result, follow up with affected users, and recommend whether to continue, revise, or stop.A before-and-after dashboard, decision log, updated roadmap, outcome narrative, and lessons that change the next decision.

    Set the operating agreement before the trial begins. Write down what you own, which decisions you can make, who remains accountable for the broader roadmap, and how much engineering work you will retain. A minimal engineering contribution can reduce the immediate staffing risk, but minimal must be explicit. Otherwise, you can end up carrying a full engineering workload while attempting a second full-time role.

    Your weekly update should be short enough that leaders will read it and structured enough that they can intervene:

    • The outcome you are trying to influence.
    • What you learned from users or data.
    • Which assumption became stronger or weaker.
    • The decision made and the trade-off accepted.
    • The next uncertainty to reduce.
    • Any decision or support needed from the recipient.

    This cadence does more than report activity. It demonstrates that you can turn incomplete information into a clear decision without hiding uncertainty. It also prevents the trial from becoming invisible work that everyone appreciates but nobody recognizes as product ownership.

    Practice the three skills engineering may not have forced you to build

    Technical competence can help you enter the conversation, but it won’t compensate for weak discovery, vague positioning, or poor stakeholder management. Those are the areas to practice deliberately during the transition.

    Product discovery: investigate behavior before proposing a solution

    Engineers are trained to solve well-defined problems. Product discovery tests whether the apparent problem is real, important, and worth solving for a particular segment. The distinction matters because confident solution design can make a weak assumption look mature.

    Use interviews to reconstruct actual behavior rather than solicit approval for an idea. Useful prompts include:

    • Walk me through the last time you tried to complete this task.
    • What triggered the need?
    • Where did the workflow slow down or break?
    • What did you do next?
    • What workaround have you adopted?
    • What was the consequence of leaving the problem unresolved?

    Avoid leading with a proposed feature or asking whether someone would use it. People can be polite, imaginative, and optimistic about hypothetical behavior. Recent examples, current workarounds, and actual consequences give you firmer evidence.

    Don’t turn each interview into a roadmap vote. Look for repeated situations, motivations, obstacles, and alternatives. Then check those patterns against quantitative signals such as activation, conversion, retention behavior, or support volume. Qualitative evidence explains what may be happening; quantitative evidence helps you understand its reach and movement.

    Product positioning: make the value segment-specific

    A technically capable product can still fail to communicate why anyone should change behavior. Positioning forces you to choose whose problem matters and why your approach is preferable to the status quo.

    Draft a simple statement: For [specific segment] struggling with [observable problem], this capability helps them achieve [meaningful outcome], unlike [current alternative], because [relevant distinction].

    Each bracket requires evidence. If you describe the user as everyone, the segment is too broad. If the outcome is easier or better, it is too vague. If you can’t name the current alternative, you may not understand the real competition, which is often an established workaround rather than another product.

    Stakeholder management: communicate decisions, not activity

    A PM rarely controls every team needed to produce an outcome. You must create alignment through context, evidence, and explicit trade-offs. That is different from satisfying every stakeholder request. Stakeholder agreement can help delivery, but it does not prove customer value.

    Build updates around the decision:

    • What decision is required?
    • Which outcome does it affect?
    • What evidence is relevant?
    • Which viable options were considered?
    • What does each option trade away?
    • What do you recommend, and why?
    • Who owns the next action?

    Remove implementation jargon unless it materially changes the decision. Executives need the consequence of a dependency, not a tour of the dependency graph. Engineers need constraints and reasoning, not a priority handed down without context.

    Practice these skills inside a product trio involving product, design, and engineering. The trio gives you access to different forms of judgment while preventing product discovery from becoming a solo PM exercise. Agree on decision rights and sponsorship at the start so you don’t become an unofficial PM with responsibility but no authority.

    Turn the work into evidence that survives an interview

    A long ticket history doesn’t demonstrate product judgment. Your portfolio has to show how you reduced uncertainty, made a choice under constraints, aligned the people needed to act, and learned from the result.

    Build each case study around a decision rather than a feature:

    • Context: Who was the user, what were they trying to do, and why did the problem matter?
    • Uncertainty: What did the team not know at the beginning?
    • Evidence: Which customer and product signals changed your understanding?
    • Alternatives: What other options were credible, including doing nothing?
    • Choice: What did you prioritize, and what did you deliberately decline?
    • Delivery: How did you reduce scope while preserving a useful test?
    • Outcome: What changed in activation, conversion, support demand, or another relevant measure?
    • Learning: What did the result change about the next roadmap decision?

    Attach the supporting artifacts only after the narrative is clear. Useful evidence includes a one-page problem brief, anonymized discovery notes, customer language, an opportunity solution tree, a hypothesis-led roadmap, an outcomes dashboard, and a before-and-after roadmap snapshot. The artifacts support your judgment; they shouldn’t force the interviewer to reconstruct it.

    Be precise about causality. If several initiatives were running at once, say that your work influenced an outcome rather than claiming it caused the entire change. If the target metric didn’t move, don’t bury the result. Explain which assumption failed, what you stopped doing, and how the evidence improved the next decision. Honest learning is a stronger PM signal than a polished success story with implausibly clean attribution.

    For an internal transfer

    Package your trial as a proposal your manager and product leader can evaluate. Include the problem boundary, success measure, product trio, weekly update rhythm, retained engineering commitment, artifacts you will produce, and the decision to be made at the end of the 90 days. This turns a vague request for a chance into a controlled staffing and product experiment.

    For an external search

    Prepare two deep case studies: one centered on discovery and another on delivery. The discovery case should show how you challenged the initial framing and reduced uncertainty. The delivery case should show how you handled constraints, aligned stakeholders, protected the outcome while reducing scope, and shipped.

    Expect follow-up questions about trade-offs: What did you say no to? Which assumption worried you most? Why was the thin slice sufficient? What evidence would have reversed your decision? What did you do when stakeholders disagreed? If your answer is only that the team completed the roadmap, you are still presenting yourself as a delivery coordinator. The stronger signal is that a decision changed because you understood the customer, business, and system more clearly.

    Key takeaways

    • Your engineering background is an advantage, not proof of PM readiness. Use it to improve decisions, not to dominate the solution.
    • Replace feature completion as your scoreboard with a clearly defined user or business outcome.
    • Build experience before changing titles by owning one bounded problem through a 90-day internal trial.
    • Use a weekly update to expose evidence, assumptions, trade-offs, decisions, and requests for help.
    • Practice discovery, positioning, and stakeholder management deliberately; technical fluency won’t substitute for them.
    • Make your portfolio decision-centered, quantify the outcomes you influenced, and represent causality honestly.
    • Prepare one discovery-led case and one delivery-led case for external interviews.

    Your next move isn’t rewriting your resume. Choose one user pain in a product you already understand. Write a one-page problem brief, identify the product and design partners you need, define the outcome you will track, and ask a sponsor to support a bounded trial. Let the title follow the evidence.

    References

  • Your Ultimate ProductCon San Francisco 2025 Guide: Best Hotels, Eats & Drinks

    Your Ultimate ProductCon San Francisco 2025 Guide: Best Hotels, Eats & Drinks

    Heading to ProductCon San Francisco 2025? I approach conference travel the same way I approach product strategy: optimize for outcomes, reduce friction, and invest in high-signal experiences. Here’s the playbook I use to choose the right hotel, find memorable meals, and make the most of every hour in the city.

    For lodging, I prioritize walkability, safety, and quiet rooms so I can focus during sessions and recover at night. If you want to be steps from most venues and meetups, SoMa and the Yerba Buena corridor are ideal. InterContinental San Francisco, W San Francisco, and The Clancy (Autograph Collection) are reliable, business-friendly picks with strong Wi‑Fi and ample lobby space for impromptu one‑on‑ones. If you prefer classic energy and transit access, Union Square hotels like Hotel Nikko and The Westin St. Francis work well. For waterfront views and a calmer vibe, Hyatt Regency Embarcadero puts you by the Ferry Building with easy BART and Muni access.

    My booking checklist is simple: reserve early, target a high floor away from elevators, and request early check‑in or late checkout around your session schedule. Loyalty programs often unlock better rates and quiet‑room preferences. If you need heads‑down time between talks, ask about day‑use meeting rooms or find a corner of the lobby with stable bandwidth. I also pack a compact power strip and a long USB‑C cable—two small upgrades that routinely save a day.

    Coffee is the fuel of great product conversations. Near SoMa, I rotate between Blue Bottle (Mint Plaza), Sightglass (7th Street), and Philz (Front Street) for pre‑session caffeine and quick stand‑ups. If I’m on the Embarcadero side, the Ferry Building’s roasters are perfect for early starts, and morning lines move faster than you’d expect if you arrive just after opening.

    For efficient lunches, I favor fast‑casual spots that can handle volume without sacrificing quality. Mixt, Souvla, Sweetgreen, Super Duper Burgers, and The Grove are dependable within a short walk of most downtown venues. When I need a higher‑signal lunch with a partner or prospect, I book a table slightly off the main corridor to avoid the rush—think Mourad for elevated Moroccan in SoMa or Boulevard along the Embarcadero for a polished, quiet conversation.

    Dinner is where the best networking often happens, so I plan for atmosphere, acoustics, and a menu that works for mixed dietary needs. Kokkari Estiatorio (FiDi) excels for executive dinners. Liholiho Yacht Club is a creative, memorable choice for cross‑functional teams. Waterbar or Angler near the waterfront pair great food with views that impress visiting colleagues. For something more casual but still conversation‑friendly, Nopa or Sorella deliver consistently.

    When it’s time for drinks, I think in terms of groups and goals. For panoramic views and small group catch‑ups, The View Lounge (Marriott Marquis) is a classic. For wine‑forward conversations with a quiet ambiance, Press Club near Yerba Buena works well. If you’re hosting a more energetic crew, Charmaine’s (SF Proper Hotel), Dirty Habit (Hotel Zelos), or 25 Lusk offer space, good music, and reliable service. For craft cocktails, Pacific Cocktail Haven and ABV are standouts if you don’t mind a short ride.

    Transit and timing matter. From SFO or OAK, BART is often the fastest, most predictable route downtown; rideshare is convenient late at night. I walk whenever possible, but I time routes along well‑lit, busier streets and avoid sprinting between neighborhoods tight on time. Microclimates are real—bring layers, comfortable shoes, and a compact umbrella. I schedule 15‑minute buffers around key sessions to handle inevitable friend‑of‑a‑friend introductions.

    If you need a professional setting for a quick working session, many hotels will extend lobby seating to guests and their visitors. For dedicated space, day passes at coworking operators like Industrious, CANOPY, or Regus are worth it when you’ve got a client briefing or board prep. For a more casual backdrop, Sightglass and Blue Bottle locations typically have reliable Wi‑Fi and just enough outlets if you arrive off‑peak.

    Finally, a word on intent: I set a simple goal for each day—one meaningful connection, one surprising insight, and one concrete action to bring back to my team. ProductCon San Francisco 2025 is a catalyst if you design your experience with the same rigor you apply to your roadmap. If you spot me in a session or at a nearby cafe, say hello—I’m always up for trading notes on product strategy, pricing experiments, and what’s working in the field right now.

    Quick note: restaurants and hours can change quickly—make reservations where possible and double‑check opening times the week of the event.


    Inspired by this post on Product School.


    Book a consult png image
  • How to Build an Outcome-Driven Product Operating Model

    How to Build an Outcome-Driven Product Operating Model

    You have rewritten the roadmap as OKRs, asked teams to focus on outcomes, and changed the titles in the quarterly review. Yet feature requests still arrive as commitments, teams still need approval to change a solution, and leaders still celebrate launches more than customer behavior. The language changed. The operating model did not.

    An outcome-driven product operating model changes who owns the problem, what leaders fund, how teams make decisions, and what evidence can alter the plan. If you are leading that transition, the practical test is simple: can each product team name the behavior it is trying to change, its current baseline, the business result that behavior should influence, its guardrails, and the decisions it can make without escalation?

    Start with an outcome contract, not an outcome slogan

    An outcome-driven model needs more than an outcome-shaped sentence. It needs a clear contract between leadership and the team.

    Leadership defines the strategic direction, the customer or business result that matters, the constraints, and the boundaries of acceptable risk. The team owns discovery, solution choice, sequencing, and the experiments used to find a viable path. This division protects strategic alignment without turning leaders into backlog managers.

    The first source of confusion is usually vocabulary. Outputs are the things a team produces; outcomes are the changes those things are intended to create. A release, migration, redesigned workflow, or pricing page is an output. Activation, retention, conversion, satisfaction, cost, and risk are outcomes when they describe an observable change rather than a work item.

    ElementWhat it doesExample
    ObjectiveSets the direction and explains why it mattersHelp new customers reach value sooner
    OutcomeDescribes the behavior or result that should changeMore new accounts complete the first-value action
    MetricMeasures that changeActivation rate or time-to-first-value
    TargetDefines the desired movement and time horizonThe agreed improvement from the recorded baseline
    BetStates a possible way to create the outcomeGuided setup for the highest-friction step
    OutputNames what the team may build or changeAn in-app guide or revised onboarding flow

    Keeping these elements separate matters. If the objective says “launch onboarding v2,” the solution has already been chosen. Discovery can only validate the predetermined answer. If it says “improve activation,” but there is no segment, baseline, causal explanation, or guardrail, the team has freedom without usable direction.

    A strong outcome contract fits on one page and contains:

    • Target customer and problem: who is affected, where the friction appears, and why resolving it matters now.
    • Primary outcome: the single behavior or business result the team is expected to influence.
    • Baseline and target: the current measurement, desired movement, and decision horizon. If the baseline is unavailable, measurement is the first task rather than an assumption hidden in the plan.
    • Causal chain: the proposed connection from product change to customer behavior to business value.
    • Leading indicators: signals such as completion of a core action or time-to-first-value that can reveal movement before the lagging result is available.
    • Guardrails: measures that must not deteriorate, such as support demand, reliability, performance, satisfaction, privacy, or risk.
    • Constraints: non-negotiable regulatory, security, platform, brand, cost, or commercial boundaries.
    • Decision rights: what the team can decide, what requires consultation, and what requires leadership approval.
    • Evidence standard: what would justify continuing, changing, scaling, or stopping the bet.

    The causal chain is the part most teams skip. “Build a dashboard to improve retention” jumps directly from output to business result. Ask what the customer will do differently because the dashboard exists, why that behavior should affect retention, and which signal would appear first. If no credible behavior connects the feature to the result, the feature is not yet a defensible bet.

    Do not make the outcome so broad that no team can influence it. Company revenue, total churn, and overall customer satisfaction are often shared results shaped by pricing, sales, service, market conditions, and multiple product experiences. A team needs a customer behavior or operating result close enough to its work to guide daily choices, while still having a clear connection to the larger business outcome.

    This is also why outputs should not disappear from planning. Teams still need delivery plans, quality standards, dependencies, and technical milestones. The mistake is treating those items as proof of value. Outputs tell you what changed in the product. Outcomes tell you whether that change mattered.

    Give durable teams a problem and real decision rights

    You cannot hold a team accountable for an outcome while reserving every meaningful decision for someone else. Outcome ownership without authority is delegated blame.

    A durable team should own a customer problem or value area long enough to build context, observe behavior, test alternatives, and learn from the result. A stable product, design, and engineering partnership reduces the handoffs that appear when temporary project teams move from specification to design to implementation.

    Durability does not mean a team owns the same feature forever. It means the team retains responsibility for an outcome space even as its solution changes. An activation team might work on guidance, setup defaults, education, performance, or removing a step entirely. The outcome provides continuity; the outputs remain flexible.

    Make decision rights explicit at each level:

    • Executive leadership: chooses the strategic outcomes, sets material constraints, allocates investment across the portfolio, and resolves conflicts that cross organizational boundaries.
    • Product leadership: translates strategy into outcome spaces, defines evidence and review standards, protects coherent team boundaries, and makes portfolio trade-offs visible.
    • Product teams: investigate opportunities, choose solution hypotheses, decide how to test them, sequence delivery, and recommend whether a bet should continue.
    • Functional leaders: establish engineering, design, data, security, and product-management standards while developing the craft and capability of their people.
    • Stakeholders: contribute customer context, commercial needs, risks, deadlines, and operational knowledge. Their requests are important evidence, but they do not silently become roadmap commitments.

    The wording of the boundary matters. “The team is empowered unless a senior stakeholder disagrees” is not a decision rule. Specify which constraints are binding, who can override a team decision, what evidence an override requires, and who decides which existing commitment will move as a result.

    When a feature request arrives, use a short intake sequence:

    1. Restate the request as a customer problem, business risk, or desired behavior change.
    2. Identify the affected segment, current evidence, urgency, and consequence of doing nothing.
    3. Compare it with the outcomes already assigned to the team.
    4. If it fits, add it as an opportunity or solution hypothesis rather than an automatic commitment.
    5. If it displaces an existing priority, ask the portfolio owner to make that trade-off explicitly and record what is being delayed.

    This prevents the common pattern in which every request is individually reasonable but the combined roadmap is strategically incoherent.

    Enabling work needs equally clear ownership. Reliability, data quality, privacy, scalability, internal tooling, and platform capabilities may not produce an immediate customer behavior change, but they can make an outcome achievable or prevent it from becoming fragile. A Product Tree makes these roots visible alongside customer-facing branches and feature-level leaves.

    Do not force enabling work into a fictional revenue claim. State the operational capability it must improve, the downstream product outcomes it enables, and the risk of postponing it. That gives platform and infrastructure investments a testable rationale without pretending every technical change has a direct, isolated effect on growth.

    Manage a portfolio of bets instead of a feature queue

    A feature roadmap creates the appearance of certainty too early. It commits the organization to solutions before the most important assumptions have been tested. An outcome-driven roadmap still communicates direction and sequencing, but it treats solutions as bets that can earn more investment through evidence.

    Each roadmap item should answer four different questions:

    • Why this problem? The customer pain, strategic relevance, business consequence, and reason it deserves attention now.
    • What should change? The target behavior or result, baseline, leading indicators, and guardrails.
    • How might it change? The current solution hypothesis and the causal assumptions behind it.
    • What happens next? The evidence being gathered and the next continue, change, scale, or stop decision.

    This format changes the roadmap conversation. Stakeholders can challenge the importance of the problem, the logic of the bet, or the quality of the evidence without treating a proposed feature as an irreversible promise.

    Use a lightweight bet brief before substantial delivery begins. It should include:

    • The outcome contract and the strategic objective it supports.
    • The customer opportunity and evidence that the problem is real.
    • The causal chain from proposed change to behavior to business result.
    • The expected reach, frequency of exposure, and direction of behavior change.
    • The solution hypothesis and the riskiest assumptions within it.
    • Confidence, effort, dependencies, privacy implications, data requirements, and technical complexity.
    • The instrumentation, experiment, rollout, and guardrail plan.
    • The evidence that would change the decision.

    A one-page impact brief is usually enough. If a team cannot express the logic concisely, expanding the document will not repair the missing understanding.

    Prioritization frameworks can help compare bets, but they should expose judgment rather than replace it. Reach, impact, confidence, and effort are useful because they force assumptions into view. Cost of delay helps when timing matters. Neither method turns uncertain inputs into objective truth.

    Pressure-test the inputs before trusting the score:

    • Is reach based on actual eligible users or the entire customer base?
    • Does “impact” refer to a behavior that can be measured, or merely to stakeholder enthusiasm?
    • Is confidence supported by behavioral evidence, customer discovery, prior experiments, or only opinion?
    • Does effort include instrumentation, rollout, migration, enablement, support, and dependencies?
    • Would the bet still rank highly if its most optimistic assumption were reduced?

    The portfolio also needs balance. Some bets improve customer behavior directly. Others reduce material risk, strengthen a platform capability, or create the measurement needed to pursue later outcomes responsibly. Make those categories explicit so foundational work is not forced to compete through exaggerated short-term impact claims.

    Set stopping conditions before enthusiasm and sunk cost distort the decision. A stopping condition might be failure to observe the necessary leading behavior, inability to reach the intended segment, unacceptable movement in a guardrail, or evidence that the customer problem is less important than assumed. Stopping a weak bet is not a delivery failure. Continuing it without a credible causal path is.

    Make evidence change plans, funding, and reviews

    The model becomes real only when evidence can change what the organization does. If every bet continues regardless of results, experimentation is theater. If quarterly reviews still focus on release counts, teams will optimize for releases.

    Connect discovery, delivery, and measurement

    Discovery is not a phase that ends when development begins. It is the work of reducing uncertainty throughout the bet. The useful sequence is:

    1. Record the baseline. Confirm that the primary outcome and leading indicators can be measured for the relevant segment.
    2. Map the causal chain. Identify the customer behavior that must change before the business result can move.
    3. Test the riskiest assumption. Learn whether the problem, proposed value, usability, feasibility, or business logic is most uncertain.
    4. Ship the smallest meaningful change. Reduce the scope needed to create observable behavior, not merely the number of tickets in the release.
    5. Monitor leading and guardrail signals. Leading indicators may appear within days, while durable or lagging outcomes can require weeks to assess.
    6. Write the learning memo. Record what happened, what remains uncertain, and whether the evidence supports continuing, changing, scaling, or stopping.

    Instrumentation belongs in the bet, not in a cleanup backlog after launch. Define event names, eligibility rules, segments, exposure, dashboards, and metric ownership before the change reaches customers. Otherwise, the team may ship on time and still be unable to answer whether the intended behavior occurred.

    Match the evidence method to the decision

    Use an A/B test when you need causal confidence and can create valid comparison groups. Set the minimum detectable effect before the test so the team knows whether the available population and duration can detect a change large enough to matter. A test that cannot resolve the decision is activity, not useful evidence.

    Not every change can be randomized. Sequential rollouts, pre-post comparisons, cohort analysis, and synthetic controls can still inform a decision, but their limitations should remain visible. Seasonality, selection effects, concurrent launches, and changes in traffic can produce movement that the product change did not cause. Label the conclusion with the strength of the evidence rather than presenting every dashboard shift as proof.

    Also distinguish a negative result from an inconclusive one. A well-powered test that shows the necessary behavior did not change challenges the hypothesis. A test with weak exposure, broken instrumentation, or insufficient sensitivity says much less. The next decision should reflect that difference.

    Replace status rituals with decision rituals

    Each operating cadence should answer a distinct question:

    • Strategy reviews: Are the chosen outcomes still the right expression of the strategy, given current customer and business evidence?
    • Team reviews: What did the team learn about the problem, causal chain, solution, and metrics, and what will it test next?
    • Portfolio reviews: Which bets deserve more investment, which need to change, and which should stop?
    • Quarterly business reviews: What customer and business results changed, what was learned, and how should allocation change? Releases provide context, not the score.

    A useful review page shows the baseline, current value, target, leading indicators, guardrails, confidence level, latest learning, and next decision. A release list without those fields is a delivery update, even if the slide is labeled “outcomes.”

    Incentives must support the same behavior. Teams should be accountable for the quality of their discovery, the integrity of measurement, the speed with which they resolve material uncertainty, and the decisions they make from evidence. Treating every missed outcome as individual failure encourages conservative targets, favorable metric selection, and reluctance to stop weak bets. Outcomes are influenced, not manufactured on command.

    Introduce the model through a real decision

    A company-wide reorganization is not the safest starting point. Begin with an important product area where the current feature plan contains meaningful uncertainty and leadership is willing to let evidence change the solution.

    1. Select one outcome and record its baseline, causal chain, leading indicators, and guardrails.
    2. Assign it to a durable product trio with written decision boundaries.
    3. Convert the planned initiative into a bet brief with assumptions and stopping conditions.
    4. Change the existing team and portfolio reviews so they require evidence and an explicit decision.
    5. At the end of the planning cycle, inspect where decisions still stalled: unclear strategy, missing data, dependency conflicts, weak skills, incentive mismatch, or executive overrides.
    6. Repair those operating constraints before expanding the model to more teams.

    Treat the operating model itself as a product. Its users are the teams and leaders making decisions. Its outcomes are clearer ownership, lower decision latency, stronger learning, and better allocation of effort. Changing an org chart without changing those behaviors is just another output.

    Key takeaways for your next planning cycle

    • An outcome must name an observable change, not disguise a feature as an OKR.
    • Pair every outcome with a baseline, causal chain, leading indicators, guardrails, constraints, and an evidence standard.
    • Give durable teams authority over discovery and solution choices within explicit strategic and risk boundaries.
    • Manage solutions as bets that can earn, lose, or redirect investment as evidence changes.
    • Keep enabling work visible by naming the capability it improves, the outcomes it unlocks, and the risk of delay.
    • Review customer behavior, business movement, learning, and next decisions. Do not use delivery activity as a substitute for impact.

    At your next roadmap review, take the most expensive planned initiative and rewrite it as an outcome contract and bet brief. If the room cannot agree on the target behavior, baseline, causal link, decision owner, and evidence that would stop the work, the initiative is not ready for a larger commitment. Resolve that uncertainty before adding more scope.

    References

  • Turn Customer Insight Into Messaging That Improves Retention

    Turn Customer Insight Into Messaging That Improves Retention

    Your activation dashboard is weak, support keeps hearing that onboarding is confusing, sales says the story is not landing, and customer success says buyers expected something different. Those can look like four separate problems. They are often four views of the same break between the value customers expect and the value they experience.

    You need a system that connects customer language, product behavior, messaging, and retention. The practical goal is not to collect more feedback or polish more copy. It is to identify an expectation gap, make the product promise more precise, help customers reach the promised outcome, and verify that the outcome lasts.

    Retention problems often begin as promise problems

    Customer insight, product messaging, and retention are usually managed in different rooms. Insight becomes an interview repository. Messaging becomes a launch asset. Retention becomes a dashboard reviewed after customers have already left. That separation hides the causal chain you need to manage.

    A customer arrives with an expectation created by your website, sales conversation, trial, or referral. The product either confirms that expectation or contradicts it. Onboarding determines how quickly the customer can test the promise. Repeated use determines whether the value is durable. Renewal and expansion reveal whether the value is commercially meaningful.

    This is why a messaging problem cannot always be fixed with copy. If the promise is accurate but the path to value is confusing, fix onboarding. If customers reach the advertised outcome once but have no reason to return, fix the recurring value loop. If the product consistently delivers something customers value but your message emphasizes a secondary feature, change the positioning. If the promised outcome is not delivered, the roadmap has to move.

    Start by locating the break in the customer journey. Use this as a diagnostic map, not as a universal scoring model:

    Journey stageEvidence to inspectMessaging questionLeading measureRetention measure
    OnboardingIncomplete steps, early exits, setup questions, and first-run sentimentIs the first promised outcome clear, and does the customer know the next action?Onboarding completion rateEarly cohort retention
    ActivationSetup completed without the behavior that represents first valueWhat observable event proves that the customer received the promised payoff?Activation rate and time-to-valueRetention among activated and non-activated cohorts
    AdoptionInitial success followed by narrow, irregular, or declining useWhich recurring job should bring the customer back?Feature adoption, session frequency, and appropriate stickinessLogo churn and gross revenue retention
    ExpansionRetained accounts asking for an adjacent outcome or broader useDoes the upgrade represent a natural next result, or merely more feature inventory?Adoption of expansion-related capabilitiesExpansion revenue and net revenue retention
    Churn riskDeclining usage, negative sentiment, unresolved tickets, contraction, or downgradesDid the product deliver the original promise to this segment?Customer health, tickets per account, and resolution timeContraction, gross revenue retention, and logo churn

    The most important distinction is between a message that is misunderstood and a promise that is unfulfilled. Both can depress activation, but they require different decisions. Ask what customers thought would happen, what actually happened, and which behavior would demonstrate that the gap has closed.

    Build a customer evidence map before changing the message

    Do not begin with a broad request to understand the customer better. Begin with a decision. For example: should you simplify first-run setup, change the activation message, reposition a capability, or invest in a missing part of the product? A bounded decision tells you which customers, signals, and time period matter.

    Customer sentiment becomes actionable when you connect qualitative feedback with usage, lifecycle, and commercial context. A complaint without behavioral context may be loud but isolated. A usage decline without customer language tells you what happened but not why. The evidence map joins the two.

    1. Select one cohort and one journey stage. Define the segment by a meaningful difference such as customer job, product tier, acquisition path, company profile, or activation status. Avoid blending customers who bought for different reasons.
    2. Define the unit of analysis. Decide whether retention is measured at the user, workspace, account, or revenue level. In a multi-user product, one active user does not necessarily mean the account is healthy.
    3. Join the evidence. Connect interviews, support conversations, reviews, in-app feedback, sales objections, usage events, lifecycle stage, CRM data, and revenue outcomes. Preserve the timestamp so you can tell whether feedback preceded or followed the behavior.
    4. Apply a stable taxonomy. Label the journey stage and a manageable theme such as usability, reliability, pricing, or time-to-value. Keep the original customer language beside the label so a summary never replaces the evidence.
    5. Write an insight as a testable claim. State the observed behavior, the customer language associated with it, your explanation, and the metric that should move if the explanation is correct.

    A useful insight statement has this shape: For [segment] at [journey stage], [observed behavior] occurs alongside [sentiment or recurring language]. Customers appear to expect [outcome] but encounter [barrier]. If that explanation is right, [product or messaging change] should move [leading indicator] and later improve [retention measure].

    The phrase “appear to” matters. Feedback is evidence, not proof of causation. Keep the explanation provisional until a product change, message test, or deeper investigation supports it.

    Read sentiment and behavior together

    Four common patterns lead to different actions:

    • Negative sentiment and failed behavior: customers describe a barrier and telemetry shows that they stop at the same point. This is a strong candidate for product discovery and a focused intervention.
    • Positive sentiment and weak behavior: customers may like the idea, the team, or an isolated capability without depending on the product. Check whether you defined the right value event and whether the expected usage cadence fits the job.
    • High usage and negative sentiment: the product may be useful while still imposing a reliability, usability, pricing, or support cost. Do not dismiss the complaints because engagement looks healthy; the account can still be vulnerable.
    • Positive sentiment and retained behavior: look for the specific outcome customers repeatedly mention and achieve. That combination can become a value pillar and a credible proof point.

    When sentiment and behavior converge, prioritization becomes easier. When they diverge, do not force a confident narrative. Check segmentation, event instrumentation, account-level aggregation, interview sampling, and the natural frequency of the customer’s job before you build.

    Use generative AI for compression, not judgment

    Generative AI can summarize call transcripts, cluster feedback, propose themes, and surface repeated phrases across a large corpus. That makes it useful for triage. It should not become an automatic roadmap-ranking system.

    Keep every generated theme traceable to the underlying records. Sample raw conversations from each important cluster, inspect false classifications, and separate customer wording from model-generated interpretation. Version the taxonomy and prompt when you change them; otherwise a movement in sentiment may reflect a classification change rather than a customer change.

    Apply privacy-by-design and data governance before sending support, CRM, or interview data into a model. Limit access, remove information that is not needed for the decision, and retain provenance. The output should help a product leader find evidence faster, not obscure where a conclusion came from.

    Turn evidence into a promise the whole journey can keep

    A product messaging framework connects the customer, problem, outcome, differentiation, and proof. Its value is operational. Product, design, sales, marketing, support, and customer success can make different artifacts without making different promises.

    For each important customer job, create a value-pillar card with the following fields:

    • Segment: the customer for whom the promise is relevant.
    • Job or problem: the progress the customer is trying to make, in the customer’s language.
    • Outcome: what becomes better when the product works.
    • Mechanism: how the product enables the outcome.
    • Point of parity: the expected capability that establishes category credibility.
    • Differentiation: the meaningful reason to choose this approach over an alternative.
    • Proof: a customer quotation, observed behavior, product demonstration, or supported performance claim.
    • Objection or boundary: where the promise does not apply, what must be true for it to work, and which objection needs an honest response.
    • Success event: the observable behavior showing that the customer reached value.
    • Retention signal: the repeat behavior or commercial outcome that indicates durable value.

    You can compress that card into a working message: For [segment] trying to [job], [product or capability] enables [outcome] through [mechanism]. It meets the category expectation of [parity], differs through [meaningful distinction], and is credible because [proof].

    Do not publish the formula as copy. Use it to expose weak thinking. If the segment is “everyone,” the message is diluted. If the outcome is a feature, the customer value is missing. If the differentiation does not affect the customer’s choice or result, it is decoration. If the proof field is empty, the claim is not ready.

    Carry one promise through different customer moments

    Consistency does not mean repeating the same sentence everywhere. It means preserving the same value logic while giving the customer the information needed at each moment.

    • Company level: define the broad change you exist to create.
    • Product level: explain how the product delivers its part of that change.
    • Segment level: select the job, obstacle, and proof most relevant to a particular customer.
    • Feature level: connect a capability to the outcome it supports instead of announcing functionality in isolation.
    • Acquisition and evaluation: set an accurate expectation, establish the category basics, show differentiation, and provide evidence.
    • Onboarding: restate the outcome the customer chose, identify the first meaningful success, and remove actions that do not help reach it.
    • Activation: make success visible when it occurs, then point to the next behavior that turns first value into repeat value.
    • Adoption: introduce adjacent capabilities when they support the customer’s next job, not simply because they are underused.
    • Renewal and expansion: refer to value the account has actually realized. Position expansion around the next credible outcome rather than a larger bundle alone.
    • Support: use the same names, outcomes, and boundaries as the product and sales experience. Conflicting terminology creates avoidable uncertainty.

    Give the same completed value-pillar card to a salesperson preparing a talk track, a product manager writing a release note, and a designer writing an in-app prompt. The artifacts should differ, but the promised outcome, mechanism, and proof should agree. If they do not, the framework is not yet clear enough to operate.

    Measure whether clearer messaging produces retained value

    A message can increase attention without improving value. That is why click-through rate or onboarding completion cannot be the final success measure. Pair every message or journey experiment with a leading behavioral indicator and a downstream retention indicator.

    Use a written experiment brief before changing the experience:

    • Cohort: who will see the change, and who will not.
    • Journey stage: where the expectation gap appears.
    • Evidence: the behavior and customer language supporting the hypothesis.
    • Change: the product, message, or combined intervention being tested.
    • Leading measure: activation, time-to-value, onboarding completion, feature adoption, or another behavior close to the intervention.
    • Retention measure: cohort retention, logo churn, gross revenue retention, net revenue retention, contraction, or expansion.
    • Guardrails: signals such as support demand, negative sentiment, downgrades, or reliability issues that should not worsen.
    • Minimum detectable effect: the smallest change the test is designed to distinguish, set before results are reviewed.

    A complete retention view combines activation, adoption, customer experience, cohort, and revenue signals. Each one answers a different question:

    • Activation rate asks whether eligible customers reached the defined first-value event.
    • Time-to-value asks how long it took to move from a clearly defined starting event to that first-value event.
    • Feature adoption and usage frequency ask whether customers continue performing the behaviors associated with value. DAU/MAU is only helpful when daily use matches the product’s natural cadence.
    • Cohort retention asks whether customers who started in the same period remain over successive intervals. Segment it when different customer groups buy for different jobs.
    • Logo churn asks what proportion of starting customers left during the period.
    • Gross revenue retention isolates retained recurring revenue before expansion: starting recurring revenue minus churn and contraction, divided by starting recurring revenue.
    • Net revenue retention adds expansion to that revenue view. Because expansion can offset losses, pair NRR with GRR and logo churn instead of reading it alone.
    • Support demand and resolution time help show whether customers are paying an operational cost to realize the promised value.

    Follow the exposed cohorts far enough to observe the retention window you selected. Do not declare success from an early conversion lift when the product decision is about durable use.

    Interpret experiment results without overclaiming

    • Attention rises, but activation does not: the message became more noticeable, not more useful.
    • Onboarding completion rises, but first value does not: the instructions may be clearer while the path still ends at the wrong outcome.
    • Activation rises, but retention falls: the message may attract the wrong expectation, or the activation event may represent task completion rather than customer value.
    • Sentiment improves, but behavior does not: customers may understand the experience better without gaining more utility.
    • Behavior improves, but sentiment remains negative: investigate reliability, effort, pricing, support, and trust rather than assuming usage settles the issue.
    • Activation and later retention improve: the intervention is a candidate for broader rollout. Check segment-level results and guardrails before scaling it.
    • No reliable effect appears: the message may not be the limiting factor, or the test may lack enough information to distinguish the effect. Check the design and evidence before concluding that messaging never matters.

    Use a cadence that matches the speed of the signal

    Review leading indicators such as activation, time-to-value, and feature adoption weekly. Review lagging commercial indicators such as GRR, NRR, and customer lifetime value monthly. Examine cohort retention quarterly to see whether improvements persist rather than merely shifting activity between periods.

    Run the review with the people who can change both the promise and the experience: the product trio and relevant go-to-market leaders. Keep the agenda decision-oriented:

    1. Which cohort and journey stage are under review?
    2. What changed in behavior, sentiment, and commercial outcomes?
    3. Where do those signals agree, and where do they conflict?
    4. Which prior hypothesis did the evidence support or weaken?
    5. Is the next action a product change, a message change, a combined experiment, or further discovery?
    6. Who owns the action, which metric should move, and when will the decision be revisited?
    7. What customer language, objection, proof point, or boundary should be added to the messaging framework?

    This last step closes the loop. New evidence updates the promise. The revised promise shapes acquisition and the product journey. Customer behavior tests whether the promise is true. Retention shows whether the value endures.

    Key takeaways

    • Treat customer insight, messaging, and retention as one operating loop, not three separate workstreams.
    • Diagnose whether the problem is an inaccurate promise, an unclear path, a missing first-value moment, or weak recurring value before changing copy.
    • Join customer language with product behavior, journey stage, account context, and commercial outcomes. Neither sentiment nor telemetry is sufficient alone.
    • Build each value pillar from a specific segment, customer job, outcome, mechanism, point of parity, differentiation, proof, and observable success event.
    • Pair message experiments with both leading indicators and downstream retention measures. An early conversion lift does not establish durable value.
    • Use generative AI to organize and retrieve evidence while preserving raw records, human review, privacy controls, and provenance.
    • Feed experiment results, objections, and customer language back into a living messaging framework so the next customer receives a more accurate promise.

    At your next product review, choose one segment and one journey stage. Bring one observed behavior, one recurring customer phrase, one value promise, and one retention measure. If you cannot name the behavior that proves value, fix the measurement. If you cannot support the promise with evidence, fix the message. If customers understand the promise but cannot realize it, fix the product.

    References

  • How to Build an Evaluation-Driven AI Innovation Strategy

    How to Build an Evaluation-Driven AI Innovation Strategy

    Your team has several credible AI demos, every sponsor sees potential, and no one can answer the question that matters: which idea deserves more engineering time, customer exposure, and operating risk?

    That is not an ideation problem. It is an evidence-design problem. A useful AI innovation strategy makes each investment earn its way forward through customer outcomes, representative evaluations, and explicit kill-or-scale decisions. The result is not less experimentation. It is faster learning with fewer expensive surprises.

    Start every AI bet with a decision contract

    Most AI roadmaps begin too far downstream. The discussion jumps to a model, an assistant, or an agent before the team agrees on the user problem or the evidence required to fund the next stage. The feature then acquires momentum simply because it exists.

    Replace the feature brief with a decision contract. This is a short agreement about what the bet must prove, how it will be evaluated, and what happens when the evidence arrives. It connects vision, portfolio choices, and execution to measurable outcomes before implementation choices harden.

    1. Name the user and the job. Specify who encounters the capability, what they are trying to accomplish, and which situations are out of scope. “Improve support with AI” is not a problem statement. “Help eligible customers resolve account questions without waiting for an agent” is testable.
    2. Choose the business outcome and its baseline. Use resolution rate, time-to-value, activation, retention, revenue lift, or another measure of customer and business value. Record how the existing workflow performs so the AI is compared with a real alternative, not with an empty screen.
    3. State the behavioral hypothesis. Explain how the proposed capability should cause the outcome to move. This exposes weak logic early. A faster response, for example, does not automatically produce a correct resolution.
    4. Define the evidence stack. Identify the offline evaluations needed to establish behavioral confidence and the live experiment needed to validate customer impact. Neither can substitute for the other.
    5. Set constraints and hard guardrails. Include unacceptable failures, privacy boundaries, safe-action requirements, latency expectations, and cost limits. A capability that is accurate but too slow, unsafe, or uneconomic is not ready.
    6. Pre-commit to the decision. Record the minimum detectable effect for the live experiment, the evaluation thresholds that block release, the time at which evidence will be reviewed, and the conditions for killing, refining, or scaling the bet.

    The contract should separate three metric layers. The outcome metric tells you whether customer or business value changed. Behavioral metrics tell you whether the AI performed its assigned job. Guardrails tell you whether that performance remained safe, reliable, responsive, and affordable. This prevents a team from celebrating a model score while the customer experience deteriorates.

    Consider a customer-support assistant. Eligible deflection and first-contact resolution can represent the business outcome. Factuality against the approved knowledge base, helpfulness, tone, retrieval accuracy, and safe CRM actions describe the system’s behavior. Harmful-content rate, unsafe-action rate, response latency, and token cost act as guardrails. A live test can then examine customer satisfaction and resolution instead of merely counting generated replies.

    This is the practical difference between an output and an outcome. Shipping an assistant is an output. Producing more successful resolutions without unacceptable safety, latency, or cost regressions is an outcome. Disciplined evaluation makes that distinction measurable.

    Match the evidence burden to the type and consequence of the bet

    A portfolio needs different kinds of AI innovation, but it should not evaluate every bet in the same way. Core optimization, adjacent expansion, and transformational innovation face different uncertainties. The label determines the strategic question. The consequence of failure determines the rigor.

    Portfolio betQuestion it must answerEvidence that matters mostTypical decision
    Core optimizationCan AI improve an established journey without damaging what already works?A reliable baseline, regression tests, live A/B results, and cost and latency guardrailsAdopt the change only when the improvement survives the existing quality bar
    Adjacent expansionDoes the capability solve a known job for a new segment, channel, or use case?Problem discovery, segment-representative evaluation cases, activation signals, and retention evidenceExpand only after the new audience reaches a meaningful value moment
    Transformational innovationCan a materially different workflow create value and be trusted?Task-completion tests, human review, adversarial testing, safe tool-use checks, and a staged customer pilotIncrease autonomy and exposure only as reliability and business evidence mature

    A core change can have a small strategic scope and still require a high evidence burden. An apparently simple classifier may sit inside a sensitive workflow. Conversely, a transformational concept can begin with a narrow, reversible prototype. Do not use “experimental” as permission to lower the bar for privacy, security, or consequential actions.

    The same discipline improves build, partner, and buy decisions. Generic demonstrations do not reveal how a system will perform on your customers’ language, your knowledge, your policies, or your tools. Run every viable option through the same representative task set. Compare task quality, latency, cost, integration effort, data boundaries, governance fit, and failure recovery. The vendor category matters less than whether the option can satisfy the decision contract.

    Portfolio funding should follow evidence maturity rather than presentation quality. Continue a bet when the team can identify remaining uncertainty and run a proportionate test to reduce it. Pause or kill it when customer value does not materialize, critical failure modes remain unresolved, or the required quality cannot fit inside the operating cost and latency envelope.

    A neutral experiment is not automatically wasted work. It can eliminate a weak hypothesis and release capacity for a better bet. But a poorly instrumented or under-sensitive experiment does not produce a useful neutral result. Set the minimum detectable effect and instrumentation before launch so “no movement” has an interpretable meaning.

    Build an evaluation stack that resembles the real product

    An AI evaluation is useful only when it represents the decisions the product must make under realistic conditions. A polished answer to a convenient prompt is weak evidence. The production system also has to handle ambiguous requests, imperfect retrieval, policy boundaries, long-tail inputs, adversarial behavior, and tool failures.

    Turn the golden dataset into an executable product specification

    Your golden dataset should express product intent through examples. Start with real, properly anonymized inputs from discovery, support, and product usage. Add important edge cases, long-tail situations, and adversarial prompts deliberately; waiting for production to reveal them transfers avoidable risk to customers.

    Each case should carry enough context to diagnose a failure, not just assign a score:

    • The user input and relevant conversation or workflow state
    • The approved information or system state the response may rely on
    • The expected behavior, acceptable answer range, or permitted action
    • A rubric for correctness, helpfulness, tone, and safety
    • A risk label that distinguishes ordinary quality defects from release-blocking failures
    • Metadata for the user segment, use case, input pattern, or workflow stage

    Keep the set versioned. Preserve cases that caught previous regressions, refresh it as customer behavior changes, and hold back examples that are not used for prompt tuning. Otherwise, the team can optimize for a familiar test set while making little progress on the wider product experience.

    Privacy belongs in dataset design. Anonymization, access control, retention rules, and approved data boundaries should be established before customer interactions become test fixtures. Retrofitting those controls after an evaluation pipeline spreads sensitive data is slower and riskier.

    Use several evaluators because each catches a different failure

    No single evaluation method is a complete quality system. Layer methods according to what is being tested:

    • Deterministic tests are appropriate for business rules, schemas, required fields, forbidden actions, exact calculations, and tool arguments. If a rule can be checked directly, do not ask another model to guess whether it passed.
    • Grounded checks compare claims with an approved knowledge base or retrieved context. They are essential when the product promises answers based on company or account information.
    • LLM-as-judge scoring can cover subjective dimensions such as helpfulness, relevance, and tone at useful scale. Define the rubric tightly and calibrate the judge against human decisions. Consistency is not enough if the judge consistently applies the wrong standard.
    • Pairwise preference tests help compare prompt, retrieval, or model variants when an absolute score is hard to interpret. They answer which candidate better satisfies the same rubric.
    • Human review remains necessary for critical, ambiguous, policy-sensitive, or high-consequence cases. It also provides the reference needed to recalibrate automated judges.
    • Red teaming probes manipulation, unsafe requests, policy evasion, and unexpected combinations of otherwise valid instructions.

    Agentic systems need evaluation beyond the final prose. A fluent confirmation can hide a failed or unauthorized action. Measure whether the agent chose the correct tool, supplied valid arguments, respected permissions and confirmation requirements, completed the intended task, and recovered safely when a dependency failed. Task-completion reliability and safe-action rate are more revealing than answer style alone.

    Quality must also be evaluated inside the cost-quality-latency envelope. A larger model can improve a difficult generation task and still be the wrong default for a simple classification step. Test model routing, token budgets, caching, prompt structure, retrieval quality, and function-calling patterns by task. The goal is not to minimize each cost independently; it is to meet the product’s quality bar with an operating profile the business can sustain.

    Turn evaluations into release gates and portfolio decisions

    An evaluation document that lives outside delivery will eventually be skipped. The evaluation suite should run whenever a prompt, model, retrieval pipeline, knowledge source, tool schema, or workflow changes. That makes evaluation part of the release mechanism instead of a launch ceremony.

    Use a gate sequence from discovery through production

    StageEvidence to collectDecision enabled
    Problem discoveryUser problem, current workflow, baseline, value hypothesis, and major risksDecide whether the problem deserves an AI bet
    PrototypeRepresentative golden-set results, failure taxonomy, latency, and estimated operating costDecide whether the capability has a credible path to the product bar
    Pre-releaseRegression suite, calibrated human review, adversarial cases, privacy checks, and safe-action testsBlock, revise, or approve a controlled rollout
    Controlled rolloutPredefined A/B test, value-moment telemetry, satisfaction, guardrails, and incident signalsValidate whether offline quality creates customer and business value
    Production scaleContinuous monitoring, segment-level failures, cost and latency trends, incidents, and refreshed evaluationsScale, route, constrain, roll back, or retire the capability

    Separate hard gates from optimization targets. A prohibited action, a privacy-boundary violation, or a broken business rule should block release. A modest tone improvement or non-critical cost regression may be handled as a tracked trade-off. If every metric is a hard gate, delivery stalls. If none is, the gate is theater.

    I use a simple test for gate quality: if two accountable leaders can read the same result and reach opposite release decisions, the decision rule is incomplete. Define the failing threshold, affected cases, permitted exception process, and rollback action before the result arrives.

    For systems that can change customer data, communicate externally, or trigger another consequential action, start with narrow permissions and human confirmation. Log the proposed action, the tool call, the result, and the reason for escalation. Increase autonomy only when the relevant task and safety evaluations hold under real usage. A human-in-the-loop control is most useful when the escalation path, response owner, and incident procedure are explicit.

    Offline evaluations create confidence to expose the product. They do not prove business impact. A live experiment must test the stated outcome with a predefined minimum detectable effect while watching for novelty bias and segment-specific failures. Instrument the customer’s value moment, not merely clicks on the AI entry point. An assistant can attract curiosity without improving activation, retention, resolution, or satisfaction.

    Production telemetry should feed back into the golden dataset. Add recurring failures, newly observed edge cases, incidents, and examples where users abandon or escalate. This turns customer reality into the next regression suite and prevents evaluation from freezing at the assumptions held before launch.

    Carry one scorecard from the product team to the QBR

    Leadership does not need a separate innovation narrative built from feature updates. Use one scorecard at product reviews, investment reviews, and QBRs. It should contain:

    • The portfolio class and strategic outcome
    • The target user, job, and current baseline
    • The causal hypothesis and non-AI alternative
    • The primary business metric and minimum detectable effect
    • The offline quality measures and live outcome measures
    • The safety, privacy, latency, reliability, and cost guardrails
    • The current evidence, unresolved uncertainty, and confidence level
    • The next test, accountable owner, review point, and kill-or-scale rule

    This creates a common language for product, engineering, design, go-to-market, risk, and executive stakeholders. The conversation becomes: What did the bet need to prove? What evidence changed? Which uncertainty remains? What decision follows? It no longer depends on who presents the most persuasive demonstration.

    The scorecard also protects speed. Teams with explicit boundaries can make routine prompt, retrieval, routing, and interface improvements without reopening the entire strategy. Leadership attention can stay on exceptions, material regressions, capital allocation, and bets whose evidence no longer supports the original thesis.

    Key takeaways for your next AI portfolio review

    • Require a decision contract before an AI idea receives roadmap momentum: user, outcome, hypothesis, evidence, guardrails, and kill-or-scale rule.
    • Classify each bet as core, adjacent, or transformational, but set evaluation rigor according to the consequence of failure.
    • Build a versioned golden dataset from anonymized real inputs, important edge cases, long-tail situations, and adversarial prompts.
    • Layer deterministic checks, grounded tests, calibrated model judging, human review, preference testing, and red teaming.
    • Evaluate agent actions and task completion, not only the fluency of the final response.
    • Run relevant regressions whenever prompts, models, retrieval, knowledge, tools, or workflows change.
    • Use offline evaluation to control release risk and live experimentation to validate customer and business impact.
    • Fund, refine, pause, or kill bets based on evidence maturity rather than demo quality or sunk effort.

    At your next roadmap review, pick one upcoming AI bet and pause the implementation discussion until its decision contract is complete. Then run the current workflow through a representative evaluation set before changing it. That baseline gives every later improvement something honest to beat.

    When each investment has a visible path from user problem to evaluation to decision, AI innovation stops being a contest between plausible demos. It becomes a repeatable way to allocate attention, manage risk, and scale the capabilities that produce durable value.

    References

  • Outcome-Driven Product Discovery: From Ideas to Better Bets

    Outcome-Driven Product Discovery: From Ideas to Better Bets

    You are looking at a roadmap full of plausible ideas, yet nobody can explain which one is most likely to change customer behavior. Sales has requests, support has complaints, leadership has strategic themes, and the product team has solutions waiting for estimates. Everything sounds important because the outcome has not been made precise enough to disqualify anything.

    Outcome-driven product discovery fixes that problem by connecting every roadmap bet to the same chain: business result, customer behavior, opportunity, assumption, experiment, and decision. It gives you a practical way to invest in innovation without turning every interesting idea into a delivery commitment.

    Start with the behavior you need to change

    A launch is an output. Completing a first meaningful workflow is a behavior. Activation is a product outcome. Retained revenue is a business outcome. Those concepts may sit in the same strategy, but they are not interchangeable.

    Start discovery with the product outcome because it is close enough to the customer experience for a team to influence and measure. Then state the business result you expect it to support. That connection is a hypothesis, not an automatic fact. Improving engagement that has no relationship to customer value, retention, conversion, or another meaningful result simply produces a more active feature.

    A useful outcome statement has five parts:

    • Segment: the specific users, accounts, or lifecycle stage whose behavior matters.
    • Behavior: an observable action that represents progress toward value.
    • Baseline and target: the current measurement and the change the team intends to produce.
    • Decision window: when you will review the evidence and decide what to do next.
    • Guardrail: the metric or customer consequence that must not deteriorate while the primary outcome improves.

    Use this template: By [decision date], change [behavior] for [segment] from [baseline] to [target], because that behavior is expected to contribute to [business result], while protecting [guardrail].

    Suppose a SaaS team wants to improve new-account activation. The feature-factory version of the goal is to launch a redesigned onboarding checklist. The outcome-driven version identifies the new-account segment, the value-bearing workflow those users need to complete, the current completion rate, the desired change, the review date, and a guardrail such as downstream retention or support burden. The checklist may become one solution, but it no longer owns the roadmap before discovery begins.

    Keep three measures visible on the same decision page:

    • Primary outcome: the customer behavior you intend to change.
    • Business consequence: the commercial or strategic result that behavior is expected to influence.
    • Guardrail: the cost, quality, trust, or downstream behavior you refuse to sacrifice.

    This is the practical difference between organizing goals around outcomes instead of output and attaching metrics to a feature after it has already been approved. The first approach creates choice. The second decorates a commitment.

    Before accepting an outcome, ask four questions. Can the team observe it? Can the team influence it during the decision window? Does it represent customer progress rather than product activity alone? Is its expected connection to the business result explicit? If any answer is no, revise the outcome before collecting more ideas.

    Key takeaways

    • Begin with a measurable customer behavior, not a feature, project, or launch date.
    • Treat the link between that behavior and the business result as a hypothesis that needs evidence.
    • Map opportunities before comparing solutions, so requests do not become commitments by default.
    • Combine segmented customer evidence with product telemetry; neither is sufficient on its own.
    • Give every experiment a decision rule, a meaningful effect threshold, and guardrails.
    • Judge discovery by the decisions it changes, including decisions to adapt, delay, or stop a bet.

    Map opportunities before you rank solutions

    Once the outcome is clear, resist the urge to run an idea workshop. First map the obstacles, unmet needs, and motivations that could explain why the desired behavior is not happening.

    An opportunity describes a customer condition. A solution describes something you could build. For example, users abandoning setup because they cannot tell which information is required is an opportunity. A setup wizard, template, tooltip, or assisted service is a solution. Keeping those levels separate preserves more than one path to the outcome.

    Translate feature requests with a simple sequence:

    1. Ask which user or account segment is making the request.
    2. Identify the job that person is trying to complete.
    3. Locate the point in the journey where progress breaks down.
    4. Describe the consequence of that breakdown in the customer’s terms.
    5. Connect the problem to the target outcome.
    6. Record the requested feature as one possible solution, not as the opportunity itself.

    This translation matters because a request can be accurate about the pain and wrong about the remedy. It can also be valid for one enterprise account but harmful to the broader value proposition. Segmenting feedback by persona, account tier, lifecycle stage, and job prevents unlike signals from being combined into a misleading vote count. A founder, a new user, a power user, and an account approaching renewal are speaking from different contexts.

    Build the map with a product trio: product management, design, and engineering working on the problem together. Early engineering involvement exposes feasibility constraints and cheaper implementation paths. Design brings the journey and interaction risks into view. Product management connects the opportunity to customer value, strategy, and commercial consequences. The benefit is shared reasoning, not another recurring meeting.

    A practical outcome-driven operating model gives that trio room to investigate opportunities before delivery sequencing hardens. Without it, discovery becomes a product-manager document handed to design and engineering after the consequential decisions have already been made.

    Use the following rubric to compare opportunities. Do not collapse it into a single total score. A tidy score can hide a fatal weakness, such as no evidence that the problem exists for the target segment.

    CriterionDecision questionWarning sign
    Outcome proximityIf this problem is reduced, what customer behavior should change?The connection depends on several untested assumptions.
    Segment evidenceWhich target users experience the problem, and in what context?The evidence comes mainly from unsegmented requests or one loud account.
    Severity and recurrenceDoes the problem block value, repeatedly create friction, or merely inconvenience the user?The team cannot distinguish a recurring obstacle from an isolated preference.
    Strategic coherenceWould solving it strengthen the intended value proposition or differentiation?The solution adds complexity without making the product more valuable to its chosen market.
    Learning valueWhat important uncertainty would pursuing this opportunity resolve?The team is committing substantial delivery capacity without identifying the risky assumption.
    Downside and reversibilityWhat could break, and how easily could the change be contained or reversed?Trust, data, operational, or platform risk is being treated as a post-launch concern.

    The result should be an opportunity map, not a backlog. A backlog asks what can be built. An opportunity map asks where a change could produce the outcome, what evidence supports that belief, and what still needs to be learned.

    Match the strength of evidence to the size of the commitment

    Customer interviews alone do not tell you how widespread a problem is. Product analytics alone do not tell you why a behavior occurs. Strong discovery uses each form of evidence for the question it can answer.

    • Qualitative evidence reveals language, context, motivation, workarounds, and consequences.
    • Behavioral evidence shows where users progress, hesitate, abandon, return, or differ across cohorts.
    • Commercial evidence shows how the opportunity appears in sales, expansion, support, renewal, or churn conversations.
    • Experimental evidence tests whether a specific intervention causes the intended change under defined conditions.

    Start with the journey connected to the outcome. Instrument the important steps, inspect funnels and cohorts, and then use interviews, support conversations, community discussions, and sales or customer-success notes to explain the patterns. This combination of telemetry and customer narrative is more useful than collecting more comments without a decision in mind.

    When qualitative and quantitative evidence disagree, do not average them into a vague conclusion. Investigate the mismatch. The interview sample may represent power users while the funnel includes new users. The telemetry may be missing an offline step. A workflow may be painful but unavoidable, producing high completion despite poor experience. A small segment may have a severe problem hidden by an aggregate rate. Contradiction is often a segmentation or instrumentation clue.

    Create a shared taxonomy so evidence remains usable after the meeting in which it was collected. Tag each item by:

    • problem statement;
    • persona or account segment;
    • job to be done;
    • journey step;
    • lifecycle stage;
    • evidence channel;
    • related outcome;
    • confidence and unresolved uncertainty.

    Then produce a compact evidence packet for each opportunity under active consideration:

    • Outcome: the behavior the team wants to change.
    • Observation: the measured pattern, with its segment and journey context.
    • Customer explanation: the recurring need, obstacle, or workaround found in qualitative evidence.
    • Contrary evidence: what does not fit the current explanation.
    • Current hypothesis: why the opportunity may be causing the behavior.
    • Largest uncertainty: the assumption most capable of invalidating the bet.
    • Next decision: what the team will decide after the next learning step.

    The required evidence should rise with the cost and irreversibility of the commitment. A reversible wording change can justify a lightweight test. A new core workflow, platform dependency, pricing model, or data-access pattern deserves deeper investigation because mistakes create migration cost, operational burden, customer confusion, or trust damage.

    My test is simple: can the team state what evidence would make it change course? If not, the work is advocacy rather than discovery. Evidence is being gathered to support a preferred answer, not to improve the decision.

    Run experiments that force a roadmap decision

    An experiment is useful only when its result can change what happens next. Before choosing a prototype or test method, write the decision the evidence must inform.

    A concise experiment card should contain:

    • Hypothesis: If [segment] receives [intervention] in [context], then [behavior] will change because [reason].
    • Riskiest assumption: the belief that would make the solution unattractive, unusable, infeasible, unviable, or unsafe if false.
    • Method: the least expensive credible way to test that assumption.
    • Primary measure: the signal that directly answers the experiment question.
    • Meaningful effect: the smallest change that would justify a different product decision.
    • Guardrails: the customer, business, quality, or trust measures that must remain acceptable.
    • Decision rule: the conditions for advancing, adapting, stopping, or gathering different evidence.

    Choose the method based on the uncertainty:

    • Use interviews and observation to understand the job, context, current alternative, and consequence of the problem.
    • Use concept tests to learn whether the proposition is understood and relevant.
    • Use clickable prototypes to find comprehension, interaction, and workflow problems before production work.
    • Use a manual or limited implementation to test whether completing the workflow creates enough value to justify automation and scale.
    • Use feature flags and progressive rollouts to contain operational risk and inspect real behavior.
    • Use an A/B test when you need a credible comparison of incremental behavior and have the traffic, instrumentation, and time to run it properly.

    Do not ask one method to prove more than it can. Positive interview reactions do not prove adoption. A usable prototype does not prove retention. A short-term click improvement does not prove durable customer value. Each result should earn the next level of investment, not retroactively validate the entire strategy.

    For A/B tests, define the minimum detectable effect before launch. This is the smallest difference worth reliably detecting for the decision, not the smallest fluctuation visible in a dashboard. Plan the sample around that threshold, avoid repeatedly checking results and stopping when they look favorable, and carry the analysis into downstream behavior where the hypothesis requires it. Statistical discipline and retention analysis prevent short-lived movement from being mistaken for a product win.

    If the available traffic cannot support the planned effect within the decision window, do not run an underpowered test and interpret noise. Reduce the scope, extend the observation period where practical, use a stronger leading indicator, or select a different method. The method should fit the decision environment.

    Guardrails deserve the same pre-commitment as the primary measure. An onboarding change that raises completion but also increases early cancellations, support contacts, errors, or later abandonment may have shifted friction rather than removed it. The team should know in advance which trade-offs are unacceptable.

    End every experiment with one of four explicit decisions:

    • Advance: the evidence supports the assumption strongly enough to justify the next investment.
    • Adapt: the opportunity still matters, but the solution or segment hypothesis needs revision.
    • Stop: the expected outcome no longer justifies the cost, risk, or strategic distraction.
    • Reframe: the test exposed an instrumentation gap, a different opportunity, or an assumption that must be investigated first.

    A failed solution test can still be a successful discovery decision. The value lies in avoiding a larger, poorly justified commitment.

    Turn discovery into the operating system for innovation

    Innovation is not measured by how unfamiliar a solution looks. It is measured by whether the team finds a better way to create and capture value under uncertainty. That requires a learning system, not a separate innovation theater filled with demos that never reach adoption.

    Give every innovation bet a one-page brief:

    • the target segment and job;
    • the behavior and business outcome;
    • the current alternative and why it is insufficient;
    • the opportunity being pursued;
    • the intended value proposition and differentiation;
    • the riskiest value, usability, feasibility, viability, or trust assumption;
    • the next experiment and its decision rule;
    • the owner, review date, and current investment boundary.

    This brief lets leadership compare bets without pretending that early ideas have precise forecasts. Mature work can be judged on measured outcome contribution. Earlier innovation should be judged on the importance of the opportunity, strategic fit, quality of evidence, cost of the next learning step, and whether uncertainty is falling fast enough to justify continued investment.

    Differentiate deliberately. Some capabilities are points of parity that customers expect. Others are candidates for meaningful differentiation. Treating every competitor feature as strategically necessary fragments the product and consumes capacity that could strengthen the chosen value proposition. First-principles reasoning should establish which customer problem matters before competitive comparison influences the solution.

    For AI products, trust belongs inside the outcome

    An AI prototype can appear successful while hiding the operational conditions required for a durable product. Add trust and control questions to discovery from the beginning:

    • What happens when the output is wrong, incomplete, or inappropriate?
    • Which data can the system access, retain, or expose?
    • Where does a person need to review, approve, correct, or override the system?
    • Can the team observe failures and explain consequential actions?
    • Does the workflow create enough customer value after review, exception handling, and operating cost are included?

    Privacy, data governance, transparent controls, and auditability are part of the product proposition, especially when the workflow has meaningful consequences. Moving from an AI demonstration to a durable capability requires evidence about the complete workflow, not just the quality of a favorable output.

    Install a cadence that changes priorities

    Discovery becomes operational when evidence repeatedly changes allocation decisions. A practical cadence is:

    • Weekly product-trio review: examine the target outcome, new evidence, contradictions, largest uncertainty, and next decision for active bets.
    • Monthly cross-functional synthesis: combine themes from product behavior, interviews, sales, support, and customer success; resolve segmentation questions; and identify implications for the roadmap.
    • Quarterly outcome lookback: compare expected and observed changes in activation, adoption, conversion, retention, or the relevant business result; inspect guardrails; and record which assumptions were right or wrong.

    This feedback and synthesis cadence creates organizational memory. It also exposes a hollow process quickly. If repeated discovery reviews never stop, reorder, narrow, or reshape roadmap work, the organization has built a reporting loop rather than a decision loop.

    Represent roadmap items as bets, with the outcome, segment, opportunity, evidence, hypothesis, guardrails, owner, and next decision visible. Delivery milestones still matter, but they sit beneath the reason for the work. That makes stakeholder conversations more precise. Instead of asking whether a requested feature made the roadmap, ask which outcome it supports, what problem it solves, what evidence exists, and what would justify investment.

    Keep a short decision log after each review. Record the decision, evidence considered, assumptions still open, owner, and revisit condition. This prevents the organization from re-litigating old choices after context has disappeared, while allowing a decision to change when genuinely new evidence arrives.

    Take the next substantial item scheduled to enter delivery and try to fill in its outcome statement, opportunity, evidence packet, riskiest assumption, experiment, guardrail, and decision rule. Any field you cannot complete is not paperwork to delegate. It is the uncertainty discovery needs to resolve before the commitment grows.

    Do that with one bet first. When the resulting evidence changes an investment decision, use the same structure for the rest of the roadmap. That is the point at which discovery stops being a phase and starts becoming how innovation is managed.

    References