Month: April 2026

  • Build an AI Toolbox That Improves Product Management

    Build an AI Toolbox That Improves Product Management

    You have an interview transcript waiting to be synthesized, a roadmap debate with more opinions than evidence, and a stakeholder update due before the decisions are settled. A general-purpose chatbot can help with each task. It can also produce a polished version of the wrong answer.

    I’ve evaluated dozens of generative AI products against the work product managers actually do, from discovery through launch. The useful pattern is simple: choose a recurring decision, connect the model to the evidence for that decision, define the human review, and measure whether the workflow improves. The tool is only one part of that system.

    Start with the decision that needs to improve

    If you begin with a tool, its demo will define your use case. You will end up generating summaries, specifications, and slide copy because those outputs are easy to show, not because they remove the most important constraint in your product process.

    Begin with a decision that is slow, inconsistent, or poorly supported. Write the workflow in one sentence:

    When [trigger occurs], [owner] uses [approved evidence] to produce [decision artifact], which [reviewer] checks before [downstream action]. Success is measured by [workflow metric] and [product metric].

    For customer discovery, that might become: when an interview round closes, the product manager uses transcripts, participant metadata, and the research question to produce a theme map and a list of unresolved questions. A research or design partner checks the evidence before the findings enter an opportunity solution tree. Synthesis time, evidence corrections, and the quality of the next research questions show whether the workflow is helping.

    A strong first use case has four properties:

    • It recurs. A workflow used repeatedly gives you enough opportunities to find failure modes and improve the prompt.
    • Its evidence is bounded. You can identify the transcripts, event definitions, strategy documents, or decision logs the model is allowed to use.
    • A qualified person can review it. The reviewer knows what a plausible but unsupported answer looks like.
    • The improvement is observable. You can compare cycle time, rework, evidence quality, or another meaningful measure before and after introducing AI.

    My rule is to start with frequent, evidence-rich work where a mistake is reversible. Interview synthesis, experiment readouts, roadmap option framing, and release communication are usually better learning environments than an autonomous decision that immediately changes customer data or launches an experience.

    Capture a baseline before changing the workflow. Record how long the work takes, where review cycles occur, which errors appear repeatedly, and what downstream decision the artifact supports. Without that baseline, faster drafting can look like progress even when reviewers spend the saved time correcting unsupported claims.

    Build the toolbox in layers, not by brand

    An effective product management toolbox connects LLMs, research synthesis, behavioral analytics, and lightweight automation. These layers solve different problems. Buying several products that all generate text does not create a complete system.

    Tool layerBest PM jobEvidence it needsUseful outputMain failure to catch
    General-purpose LLM workspaceFraming, critique, drafting, and option generationObjective, constraints, definitions, and approved documentsQuestions, alternatives, structured drafts, and decision briefsConfident invention or generic advice detached from product context
    Research synthesisOrganizing customer interviews and qualitative feedbackTranscripts, participant identifiers, segment metadata, and research questionsEvidence-linked themes, contradictions, unmet needs, and follow-up questionsTreating a small sample as market prevalence or erasing minority views
    Behavioral analyticsFinding where behavior changes and sizing an opportunityEvent definitions, entity grain, cohorts, funnels, paths, retention views, and experiment resultsDrop-off patterns, affected segments, anomalies, and testable hypothesesTurning correlation into causation or analyzing an incorrectly defined event
    Knowledge and retrieval layerGrounding answers in current product contextStrategy, decision logs, research, taxonomy, policies, and product documentationTraceable answers with evidence and visible conflictsRetrieving stale, unauthorized, or contradictory material without warning
    Workflow and experience automationMoving an approved decision into repeatable executionApproved copy, segments, triggers, stop conditions, owners, and measurement eventsIn-app guides, product tours, handoffs, checklists, and status updatesPublishing or acting before human approval, measurement, or rollback is ready

    Use the table to expose missing layers. If research synthesis is strong but event definitions are unreliable, another writing assistant will not improve opportunity sizing. If analytics is mature but the model cannot access the current strategy or decision history, its prioritization advice will remain generic. If automation is available but ownership and rollback are unclear, speed will amplify operational risk.

    Evaluate each candidate against the workflow, not against a feature checklist. Ask:

    • Can it work where the approved evidence already lives, or will people create uncontrolled copies?
    • Can a reviewer trace a conclusion back to a transcript, event definition, document, or decision record?
    • Can access, retention, sharing, and deletion follow your data governance rules?
    • Can you test a stable workflow with representative examples instead of judging a polished demo?
    • Can you observe failures, corrections, latency, and cost after rollout?
    • Does the total cost include integration, governance, evaluation, review time, and maintenance rather than only the license?

    A vendor can be impressive and still be wrong for your operating environment. The decisive question is whether it strengthens a specific product decision without weakening evidence quality, privacy, or accountability.

    Turn the tools into repeatable PM workflows

    The prompt is not the workflow. A production workflow includes prepared inputs, an output contract, a review step, a decision owner, and a place to record what happened. The following patterns cover the PM work where AI can create leverage without pretending to replace product judgment.

    Synthesize interviews without manufacturing certainty

    Qualitative synthesis becomes unreliable when the model merges observation, interpretation, and recommendation into one smooth narrative. Preserve those boundaries. Give each participant a stable identifier, retain relevant segment context, and tell the model to cite the evidence behind every theme.

    Copy-paste prompt: Act as a product research analyst. Use only the supplied interviews and research brief. For each theme, return the claim, supporting participant identifiers, contradictory evidence, affected segment, confidence with a reason, and the next unanswered question. Separate direct observations from interpretation and recommendation. Do not infer market prevalence from this interview sample. If a conclusion lacks evidence, label it unsupported.

    Review the output by opening the cited passages, not by judging whether the summary sounds plausible. Look for participants who do not fit the dominant theme. Check whether two different needs have been combined because they use similar words. Confirm that the model has not converted the loudest quotation into the most important opportunity.

    Only then move the findings into your discovery structure. The useful handoff is not a list of themes. It is a set of evidence-backed needs, open questions, affected segments, and assumptions that the product trio can investigate.

    Combine behavioral data with customer evidence

    Behavioral analytics can tell you where users drop out, which segments behave differently, and whether a pattern is large enough to deserve attention. It does not tell you why the behavior occurred. Interviews can reveal possible motivations, but a qualitative sample does not establish how common each motivation is. Use the two evidence types together without asking either to answer the other’s question.

    Before involving an LLM, verify the event name, event meaning, user or account grain, relevant cohort, and analysis window. If instrumentation changed, include that context. Prefer aggregated or appropriately governed data; do not paste raw personal or confidential customer data into an unapproved model.

    Copy-paste prompt: Use the supplied event definitions, cohort table, funnel, and interview themes. Identify the largest observed behavior changes by segment. For each change, distinguish the observed fact from possible explanations. List data quality questions, supporting customer evidence, conflicting customer evidence, and the cheapest analysis or experiment that could reduce uncertainty. Do not claim causation from a correlation.

    Return to the analytics system to validate every material claim. The model is useful for connecting evidence and generating hypotheses; the governed analytics layer remains the place to confirm event behavior, segment definitions, retention patterns, and experiment results.

    Frame roadmap choices as options, not generated certainty

    A roadmap debate rarely fails because nobody can generate feature ideas. It fails when alternatives, assumptions, constraints, and expected outcomes are implicit. AI is most useful here as an argument compiler: it can turn scattered evidence into comparable options and expose what each option requires you to believe.

    Copy-paste prompt: Use the supplied product objective, customer evidence, behavioral evidence, strategic constraints, technical constraints, and decision history. Create a set of distinct options rather than a ranked feature backlog. For each option, state the target outcome, supporting evidence, contradictory evidence, critical assumptions, excluded alternatives, leading indicator, delivery risk, and cheapest test. Flag any recommendation that lacks a traceable source. Do not make the final priority decision.

    This format makes outcome-versus-output confusion visible. An option such as build a new onboarding checklist is an output. Improve successful first-time setup for a defined customer segment is an outcome. The first can support the second, but the relationship is still a hypothesis. Keep that hypothesis visible in the roadmap and in the experiment plan.

    The human decision owner should record the selected option, why it won, what evidence mattered, which assumption remains unresolved, and when the decision should be revisited. That decision log becomes grounding material for later planning instead of forcing the next model session to reconstruct context from scattered documents.

    Move an approved launch into an observable experience

    Once the decision is approved, AI can reduce the mechanical work of adapting positioning into release notes, support context, product tours, and in-app guides. The risky part is not drafting the words. It is allowing generated content to reach the wrong segment, appear at the wrong moment, or launch without a measurement and stop condition.

    Copy-paste prompt: Using only the approved positioning, UX terminology, target segment, trigger event, and product constraints, draft an in-app sequence. For each message, state its purpose, trigger, target user, action requested, dismissal behavior, stop condition, and measurement event. Preserve the approved claim boundaries. Flag any copy that introduces a benefit, capability, or promise not present in the supplied material.

    Review the experience in context. Confirm that the audience definition matches the analytics definition, the trigger can actually be observed, the requested action exists in the current interface, and users can dismiss or complete the sequence. Keep experiment design and success analysis outside the copy generator. Fluent wording cannot declare the launch successful.

    Make every output inspectable before it becomes operational

    The difference between a useful personal assistant and a dependable organizational workflow is inspectability. A reviewer must be able to see which evidence was available, which instructions shaped the answer, what the model produced, what a person changed, and which decision followed.

    Use a retrieval-first pipeline grounded in product documents and decision logs. Do not rely on model memory for current strategy, naming, policy, or product behavior. Define an authority order for conflicting material. A current approved decision record should not silently lose to an older planning document simply because the older document contains more text.

    Your grounding layer should preserve permissions. Retrieval is not an excuse to expose every document to every workflow. Record the owner and freshness of important material, remove obsolete versions from the approved collection, and instruct the model to show conflicts instead of resolving them invisibly.

    Treat each repeated prompt as a small product surface with a contract:

    • Goal: the decision or artifact the workflow must support.
    • Allowed evidence: the documents, data, and tools the model may use.
    • Definitions: the product terms, entities, events, segments, and metrics that must remain consistent.
    • Method constraints: what the model must separate, preserve, cite, or avoid inferring.
    • Output contract: the required fields, order, labels, and evidence links.
    • Uncertainty behavior: when to flag missing context, conflicting inputs, or unsupported conclusions.
    • Review and stop conditions: who approves the output and what prevents it from moving downstream.

    Then create an evaluation set from representative work. Include ordinary inputs, ambiguous cases, conflicting documents, incomplete evidence, sensitive-data traps, and previously observed failures. A good evaluation checks groundedness, traceability, coverage, decision usefulness, confidentiality, and consistency. Writing quality matters, but polish is not evidence.

    Re-run the evaluation whenever the model, prompt, connector, knowledge collection, event taxonomy, or output schema changes. A workflow that passed yesterday’s cases can regress when one dependency changes. This is why eval-driven development, observability, privacy-by-design, and AI risk management belong in the product manager’s toolbox rather than in a separate governance document.

    For each operational run, retain enough information to diagnose failure: workflow name, input sources, prompt or configuration version, output, reviewer corrections, final decision, latency, and cost where available. The record should support improvement without retaining sensitive data longer than your policy permits.

    A screenshot checklist can make the workflow easier to teach and audit. Capture the approved input location, relevant access setting, prompt configuration, evidence-linked output, human edits, final decision record, and measurement view. Screenshots do not replace logs or documentation, but they give PMs and stakeholders the same operating picture during onboarding and review.

    Scale adoption through gates and measurable outcomes

    Do not roll an AI tool out to every product manager and hope good practices emerge. Move one workflow through explicit gates:

    1. Baseline the current workflow. Record cycle time, review effort, recurring errors, and the downstream outcome it supports.
    2. Run in shadow mode. Produce the AI-assisted artifact without allowing it to drive the real decision. Compare it with the normal process and save failure cases.
    3. Introduce assisted use. Let a named human owner use and edit the output. Require evidence checks before it reaches stakeholders or customers.
    4. Standardize the operating pattern. Publish the input rules, prompt contract, evaluation set, owner, storage location, escalation path, and fallback process.
    5. Expand only after the workflow holds up. Add users, data sources, or automation after quality, privacy, and review behavior remain dependable.

    Measure the workflow at more than one level. Cycle time tells you whether work moves faster. Correction rate and review effort show whether speed is hiding rework. Evidence coverage shows whether claims can be defended. The linked product metric shows whether the artifact supports a meaningful outcome. Total cost tells you whether licenses, integration, evaluations, governance, and human review are worth the saved effort.

    Do not count prompts submitted, words generated, summaries created, or seats assigned as product impact. Those are activity measures. A workflow is valuable when it shortens a real decision cycle, improves the evidence behind a decision, reduces preventable rework, or helps the team learn about an outcome sooner.

    Pause or roll back the workflow when material claims cannot be traced, confidential data crosses an unapproved boundary, reviewers begin rubber-stamping output, small configuration changes cause unpredictable recommendations, or the review and governance burden cancels the useful gain. A graceful fallback to the previous process is part of the design, not an admission that AI failed.

    Key takeaways

    • Choose a recurring product decision before choosing an AI product.
    • Combine LLMs, research synthesis, behavioral analytics, grounded knowledge, and automation only where the workflow needs them.
    • Require bounded evidence, visible uncertainty, traceable claims, and a named human decision owner.
    • Turn repeated prompts into governed contracts with evaluations, observability, and clear stop conditions.
    • Judge the toolbox by cycle time, evidence quality, rework, product learning, and total cost rather than by generated output.

    This week, select one recurring PM decision and write its workflow sentence. Baseline the current process, run the AI-assisted version in shadow mode, and save every failure as an evaluation case. Your toolbox becomes valuable when it improves a decision you can defend, not when it produces more material to review.

    References

  • How to Design an AI Customer Agent for Sales Qualification

    How to Design an AI Customer Agent for Sales Qualification

    A prospect reaches your pricing page with a real buying question. The form promises a reply, but the reply arrives after the prospect has moved on, chosen a competitor, or forgotten why the question mattered.

    An AI customer agent can remove that delay, but speed is only the entry requirement. The harder product problem is deciding whom to qualify, what evidence to collect, which next step to offer, and how to preserve enough context that a salesperson can continue the conversation without starting over.

    Start with routing decisions, not chatbot dialogue

    The purpose of a sales qualification agent is not to produce a pleasant conversation or a high lead score. Its job is to make a defensible next-step decision while the buyer’s intent is still active.

    That distinction matters because conversational fluency can hide weak commercial logic. An agent may sound helpful while booking low-fit meetings, sending strong prospects down a generic self-serve path, or marking inferred information as confirmed. Those failures make the pipeline look larger before they make it less trustworthy.

    Define the available outcomes before you write prompts. Most inbound motions need some version of these routes:

    RouteMinimum evidenceAgent action
    Sales-readyThe problem fits the product, the buyer needs sales involvement, and the timing or buying process satisfies your acceptance rule.Offer an appropriate meeting, create or update the CRM record, and send the qualification evidence.
    Self-serveThe use case is viable, but the buyer can select a plan, begin a trial, or complete signup without a salesperson.Recommend the relevant path, help the buyer take the next action, and preserve the conversation for later use.
    Promising but not readyThere is plausible fit, but intent, timing, authority, or requirements remain unresolved.Provide the useful resource or follow-up path defined by your policy without manufacturing urgency.
    Not a fitA hard requirement conflicts with the product’s supported scope or the request belongs elsewhere.State the limitation clearly and redirect the person without placing an unqualified meeting on a seller’s calendar.
    Human exceptionThe request involves an existing account, a sensitive claim, a complex commercial exception, or information the agent cannot verify.Escalate with the context already collected and identify the unresolved question.

    Keep fit and readiness as separate dimensions. A large, recognizable account can be a strong fit and still be months away from a decision. A highly motivated buyer can be ready to act and still need a capability you do not provide. Combining both dimensions into one opaque score conceals the reason behind the route and makes mistakes difficult to diagnose.

    Separate hard constraints from soft signals as well. A required capability that does not exist may be a hard stop. A vague timeline is usually uncertainty to resolve, not automatic disqualification. Firmographic enrichment can help prioritize an account, but it cannot confirm what a buyer has not actually said.

    For every consequential route, require three outputs: a reason code, the evidence behind it, and the next action. If the agent cannot produce all three, it has not completed qualification. It has merely assigned a label.

    Turn your qualification policy into an executable conversation

    A natural-language playbook makes sales policy easier to express, and current customer agents can be instructed to follow qualification rules, address approved objections, and move buyers toward defined outcomes. Natural language does not remove ambiguity, however. If two experienced salespeople would interpret a rule differently, the agent will not reliably repair the policy for you.

    Ask only what changes the route

    Traditional lead forms collect fields because the CRM has columns. A conversation should be more selective. Every question should either help the buyer, determine fit, resolve readiness, or select the correct action.

    1. Open from observable context, such as the plan, feature, or integration the person is exploring.
    2. Answer the buyer’s current question before turning the exchange into discovery.
    3. Ask the smallest useful question that could change the route.
    4. Branch from the answer instead of walking every prospect through the same questionnaire.
    5. Confirm the important facts before treating them as qualification evidence.
    6. Explain the recommended next step and let the buyer act while still in the conversation.

    If someone asks whether a specific integration is available, answer that question first. Then ask how the integration fits the intended workflow if the answer would affect plan selection or sales involvement. Leading with budget, company size, or phone number when none of those details helps answer the immediate question makes the agent feel like a form with typing animation.

    A useful qualification schema usually covers the following areas, but the agent should collect only the fields relevant to the current branch:

    • The problem or use case the buyer is trying to address.
    • The capabilities, integrations, or constraints that determine product fit.
    • The consequence of leaving the problem unsolved, when that affects urgency or route.
    • The buyer’s role in evaluation and the remaining buying process.
    • The intended timing and any event driving it.
    • Commercial expectations or budget when those facts genuinely affect the path.
    • Identity and account context, with a clear distinction between what was stated, enriched, or inferred.

    Do not ask about budget merely because a familiar qualification framework includes it. If pricing is public and the buyer can start without sales assistance, the better action may be to explain plan fit and help the buyer proceed. If commercial terms require human involvement, budget or purchasing process may become relevant later in the branch.

    Preserve provenance instead of filling blanks with guesses

    Store each material qualification field with its provenance. Buyer-stated, externally enriched, model-inferred, and unknown are different states. Treating them as interchangeable creates false confidence in the CRM.

    An enriched company size may help prioritize a conversation, but it is not buyer-confirmed budget. A page visit may indicate interest in a feature, but it is not a confirmed requirement. An enthusiastic phrase may indicate intent, but it is not a purchasing timeline. Keep those distinctions visible to the routing logic and the salesperson receiving the lead.

    I would not allow the agent to write a final qualification status unless every required field is either supported by evidence or explicitly marked unknown. Unknown information is operationally useful: it tells the seller what still needs to be resolved. Fabricated completeness does the opposite.

    Evaluate decisions, not just responses

    Build a scenario set from the situations that cause real routing disagreements. Include a high-fit account with low intent, a small account with urgent intent, an existing customer asking a sales-shaped question, a buyer requiring an unsupported capability, a returning visitor, a pricing objection, conflicting information, and a request the agent should escalate.

    For each scenario, define the expected answer, acceptable route, required CRM writes, escalation condition, and forbidden behavior. Then test the complete journey. A correct answer followed by the wrong calendar, duplicated CRM record, or context-free handoff is still a failed qualification experience.

    Build trust into answers, memory, and handoffs

    A sales agent cannot qualify reliably if its product knowledge is unreliable. It needs approved information about pricing, packages, capabilities, integrations, plan eligibility, trial paths, and common objections. Current implementations can draw on an existing product knowledge base while combining that knowledge with playbooks, enrichment, and memory, which reduces duplicated setup but does not eliminate ownership.

    Assign a business owner to every consequential knowledge area. When pricing, packaging, an integration, or an eligibility rule changes, update the canonical material and rerun the scenarios affected by that change. A polished answer based on stale commercial information is more dangerous than an explicit handoff because the buyer has little reason to question it.

    Define the agent’s boundaries in the same system. It should know when it may explain published pricing, when it must avoid inventing discounts, when roadmap questions need human confirmation, and when a security, legal, or contractual claim requires escalation. The safe fallback is not a vague non-answer. It is a clear statement of what remains unverified and a context-rich route to someone authorized to answer.

    Use memory as buyer state, not as an unlimited transcript

    Memory is valuable when a returning visitor does not have to repeat the use case, plan under consideration, or unresolved objection. A customer agent can recognize returning context and continue the buying journey, but old information should not silently override new facts.

    Store a compact buyer state: confirmed needs, important constraints, questions already answered, current route, unresolved items, and the last agreed next step. Keep timestamps and provenance so the system can notice when a current statement conflicts with an earlier one. Ask for confirmation when a material fact may have changed.

    Enrichment deserves the same discipline. Use it to improve context and routing, not to pretend the agent knows the buyer personally. Record where enriched data came from, apply your privacy and retention controls, and give buyer-stated information precedence when the two conflict.

    Make the handoff a product deliverable

    Booking a meeting is not the end of qualification. It is the beginning of a human handoff. Passing only a name, email address, and transcript forces the salesperson to reconstruct the conversation under time pressure.

    The handoff package should contain:

    • Identity and account information, including the provenance of enriched fields.
    • The buyer’s problem and intended outcome in the buyer’s own terms.
    • Confirmed requirements, constraints, timing, and buying-process details.
    • Questions answered and the approved information used to answer them.
    • Objections, unresolved questions, and any claim requiring human confirmation.
    • The selected route, its reason code, and the evidence that supported it.
    • The next action already promised to the buyer.
    • A link to the full conversation for detail or audit.

    Modern customer agents can book through scheduling tools, sync structured context into the CRM, and pass both conversation history and an AI-generated summary. The summary should reduce reading effort, while the structured fields should support routing, reporting, and workflow automation. Neither should replace access to the original conversation.

    Test the first minute of the seller’s follow-up. Can the seller see why the lead was routed, what the buyer already knows, and what must happen next? If the opening question repeats discovery the agent just completed, the handoff has broken the continuity you used AI to create.

    Measure whether the agent creates incremental pipeline

    Meeting count is an attractive but incomplete success metric. Bookings can rise because the agent reaches previously unattended demand, because it diverts buyers who would have booked with a human anyway, or because it lowers the qualification bar. Only the first outcome is unambiguously additive.

    Instrument the full decision funnel rather than the chat interface alone:

    1. Reach: eligible visitors, conversations initiated, response latency, and coverage by channel or time window.
    2. Conversation quality: questions answered, unresolved-answer rate, corrections, escalations, and abandonment before a useful action.
    3. Qualification quality: completion of required evidence, unknown-field rate, route distribution, seller acceptance, and rejection reasons.
    4. Handoff quality: meeting attendance, repeated discovery, missing CRM context, reassignment, and follow-up delay.
    5. Commercial outcome: accepted qualified opportunities, pipeline created, trial or self-serve conversion, win rate, contract value, sales-cycle progression, and cost per accepted opportunity.

    Audit both error directions. False positives waste seller time and inflate forecasts. False negatives are quieter: a strong buyer is sent away, mislabeled as self-serve, or blocked by an unanswered question. Review unsuccessful routes as well as booked meetings, because the most expensive qualification error may never appear in the sales team’s queue.

    Early deployment data shows why coverage and incrementality need separate analysis. In a vendor-reported overnight rollout, Fellow booked 18 January meetings that its human team would not otherwise have reached, with around 48% converting, while the human booking rate held. That is evidence of an additive channel in that deployment, not a universal conversion benchmark.

    Volume alone tells a different and incomplete story. During a vendor-reported three-month deployment, Attio’s agent handled more than 1,600 visitor conversations, qualified more than 50 leads for sales, and routed more than 30 applicants into a startup program. Those figures show multiple useful outcomes from the same inbound surface, but they do not establish causal lift for another company’s funnel.

    Establish your own baseline separately for hours and pages with human coverage and those without it. If feasible, use a randomized holdout among otherwise eligible sessions. If randomization would create an unacceptable buyer experience, compare matched cohorts by page, channel, segment, visitor status, and time window. Do not compare an overnight agent cohort with daytime human coverage and call the difference an AI effect.

    A controlled rollout can begin on a high-intent surface or during a coverage gap. First run routing in shadow mode and compare the proposed decisions with qualified human judgment. Then enable one consequential action at a time, such as self-serve guidance before autonomous meeting booking. Keep a human review path for exceptions and expand only when answer quality, routing precision, CRM completeness, and buyer outcomes remain acceptable together.

    Your north-star measure should reflect accepted commercial value, such as incremental qualified opportunities or incremental pipeline per eligible visitor. Pair it with guardrails for incorrect claims, seller rejection, buyer complaints, CRM errors, and missed high-fit leads. An agent that creates more records while reducing trust has not improved the sales system.

    Key takeaways

    • Treat the AI customer agent as a decision system, not a conversational layer placed in front of a lead form.
    • Define sales-ready, self-serve, not-ready, not-fit, and human-exception routes before writing dialogue.
    • Keep fit separate from readiness, and preserve whether each field was buyer-stated, enriched, inferred, or unknown.
    • Ask only questions that help the buyer or change the route; answer the buyer’s immediate question before running discovery.
    • Make structured qualification evidence, unresolved issues, and the promised next action part of every human handoff.
    • Measure incremental accepted opportunities and pipeline, not chat volume, MQL count, or booked meetings in isolation.

    Start with one high-intent entry point. Write its route contract, connect only approved knowledge, test the difficult scenarios, and compare shadow decisions with the people who currently qualify those leads. Give the agent authority gradually. The goal is not to automate the most conversation; it is to make the right buying path available at the moment the buyer is ready to take it.

    References

  • How to Build Agentic AI for Product Analytics and Support

    How to Build Agentic AI for Product Analytics and Support

    Your support bot can tell a customer where a setting lives, yet leave that customer to diagnose the problem, change the setting, and hope it worked. Your product team then receives a chat transcript without knowing whether the interaction improved activation, feature adoption, or retention.

    If you are deciding how to connect AI, product analytics, and support, do not start with the model. Design the closed loop first: assemble trustworthy context, choose an allowed action, verify the resulting product state, and measure the user outcome. The model is one component inside that system.

    Treat product analytics as the agent’s control plane

    A useful standard is an assistant that understands the user’s context, can complete an allowed action, and measures whether the action helped. Remove any one of those capabilities and the experience degrades quickly. Context without action produces advice. Action without context creates risk. Action without measurement creates an impressive demo that cannot earn a durable place on the roadmap.

    Product analytics supplies the behavioral context and outcome signals for this loop. It can show where the user is in a journey, which features have been adopted, which step failed, and whether the expected success event eventually occurred. It should not be treated as a warehouse-sized attachment to the prompt.

    Define a support context contract

    Create a small, governed context object for each supported workflow. Give the agent only the fields required to understand and resolve that workflow:

    • Actor and access: the authenticated user, account, role, entitlements, and permissions relevant to the requested action.
    • Journey state: the onboarding step, feature-adoption state, experiment assignment, or other stage that explains what the user is trying to complete.
    • Current product state: the relevant configuration from the operational system of record, including whether required prerequisites are satisfied.
    • Friction evidence: recent failed events, validation results, repeated attempts, and known errors connected to this workflow.
    • Desired outcome: the product state and behavioral event that will count as successful resolution.

    Resolve analytics events and tool calls to the same stable user and account identifiers. Preserve timestamps and the origin of each field. For a live action decision, let the operational system of record determine current state; use analytics to explain the journey and measure the outcome. An event stream can be delayed or incomplete, so it should not overrule a current configuration read.

    Behavior is also evidence, not intent. Repeated visits to a setup screen could indicate confusion, careful verification, or an advanced workflow. When those interpretations require different actions, the agent should ask one targeted question instead of turning a behavioral pattern into a confident diagnosis.

    Apply data minimization at this boundary. Do not place secrets, payment information, unrelated conversation history, or an account’s entire event history into the model context. Filter fields before the model sees them, and enforce the filter in code rather than relying on a prompt instruction.

    Give the analytics agent a metric contract

    An internal analytics agent has a different job from a customer-facing support agent. It may translate a product question into metrics, cohorts, funnels, or retention views, but a fluent answer is not enough. Require every analysis to return:

    • the product question it interpreted;
    • the metric definition and success event it used;
    • the cohort, filters, and observation window;
    • the analysis or query reference needed to reproduce the result;
    • known data-quality limitations and unresolved ambiguity; and
    • a clear distinction between observed association and demonstrated causal lift.

    This turns the analytics agent into a traceable decision aid. It also prevents two agents from using the same metric name while silently applying different event definitions, account filters, or windows.

    Design one closed loop from signal to verified outcome

    The core unit of agentic support is not the conversation. It is a resolution attempt with a beginning, an authorized action, and a verifiable end state. Use the following loop for every workflow:

    1. Observe the trigger. Capture the user’s request or a product signal that indicates likely friction.
    2. Assemble scoped context. Load only the identity, permission, journey, state, and error fields defined in the context contract.
    3. Diagnose the next constraint. Determine which prerequisite, configuration, permission, or knowledge gap is blocking progress. If the evidence is ambiguous, ask rather than assume.
    4. Select an approved playbook. Match the constraint to a versioned workflow with explicit eligibility rules, allowed tools, and prohibited actions.
    5. Obtain the required authorization. Show the proposed change and its consequence whenever the action changes product state or affects other people.
    6. Execute through a narrow tool. Use a typed, allowlisted operation. Make retryable actions idempotent so a repeated call does not create duplicate changes.
    7. Verify the result. Read the resulting product state and look for the defined success event. Tool completion alone does not prove customer resolution.
    8. Record the outcome. Log the context version, playbook, model, policy decision, tool call, result, success signal, and any escalation or user reversal.

    The loop supports two related products without collapsing their permissions. An internal analytics agent can identify an affected cohort, inspect a funnel, or surface a recurring failure pattern. A customer-facing support agent can use the approved finding to help one authenticated user, but it should see only that user’s permitted context and tools. A human support operator should receive the same trace when the agent escalates.

    Keep the shared layer deliberately small: stable identities, canonical metric definitions, governed context fields, outcome events, and versioned playbooks. The analytics agent and support agent can then improve the same system while retaining separate access policies and evaluation criteria.

    Do not automatically convert every observed correlation into a new support action. Let analytics generate a candidate playbook, review the causal logic and risk, test it against known cases, and release it through a controlled experiment. The learning unit is the reviewed playbook, not an unexamined prompt change.

    Choose a first workflow that can prove its own value

    The first pilot should be easy to verify, not merely easy to demonstrate. A conversational answer looks polished even when it does not change the user’s outcome. A narrow configuration or onboarding workflow is usually a better proving ground because eligibility, allowed actions, and success can be defined before launch.

    Score candidate workflows against these criteria:

    • Repeated demand: the same intent or failure appears often enough to justify a reusable playbook.
    • Observable state: the agent can read the prerequisites and current configuration instead of guessing from the user’s description.
    • Clear success: one product state or behavioral event can verify that the problem was resolved.
    • Safe execution: the initial actions are reversible, user-scoped, and unlikely to affect billing, security, data retention, or other users.
    • Short feedback: the primary outcome appears soon enough to support iteration, even if retention is monitored later.
    • Enough eligible traffic: the workflow can support a meaningful experiment rather than a handful of anecdotes.

    Write the pilot contract before the prompt

    A pilot contract forces the product, analytics, support, engineering, and risk decisions into one inspectable artifact. It should specify:

    • the user problem and eligible cohort;
    • the trigger that starts the workflow;
    • the context fields and systems of record;
    • the approved diagnostic branches;
    • the allowed and prohibited actions;
    • the point at which confirmation is required;
    • the precondition and postcondition for each tool call;
    • the success event and observation window;
    • known failure modes and the human handoff rule; and
    • the primary outcome, guardrail metrics, experiment design, and minimum detectable effect.

    Consider an onboarding configuration workflow. The trigger might be a user repeatedly reaching setup without completing it. The context could include entitlement, current configuration, prerequisite status, and the latest validation result. The agent may be allowed to run validation, explain a missing prerequisite, prefill a reversible setting, or launch the next approved step. Resolution requires both the expected configuration state and its corresponding success event. If validation continues to fail, the handoff should include the exact state, error, playbook branch, and actions already attempted.

    Avoid starting with data deletion, broad permission changes, security recovery, billing adjustments, or external communications. Those workflows combine difficult authorization questions with high consequences. Prove context quality, tool reliability, verification, and measurement on a narrower action set before expanding the blast radius.

    Set the minimum detectable effect before the experiment. If the eligible population cannot detect an outcome change that would justify the investment, narrow the claim, combine additional time periods, or choose a more observable workflow. Do not call an underpowered neutral result proof that the agent has no effect.

    Instrument the agent like a product surface, not a transcript

    Conversation volume, message count, and thumbs-up feedback are diagnostic signals. They are not sufficient outcome measures. A customer can like an explanation and still remain blocked; another can dislike the wording even though the configuration was fixed.

    Measurement layerQuestion it answersUseful signals
    Operational reliabilityDid the system execute as designed?Tool success, validation failure, retry, latency, rollback, and escalation
    Verified resolutionDid the requested product state become true?Verified resolution rate, time to resolution, repeat attempt, and repeat contact
    Product outcomeDid the user progress in the journey?Activation, feature adoption, workflow completion, and later retention
    Support outcomeDid the workflow reduce avoidable support effort?Eligible ticket rate, escalation reason, handle-time impact, and handoff quality
    Safety and trustDid the agent stay within policy and user intent?Permission block, wrong-action review, user reversal, policy violation, and privacy incident

    Define the denominators as carefully as the numerators. Verified resolution rate should use eligible support sessions as its denominator and require the success state defined in the pilot contract. Action completion rate should use authorized action attempts, not every conversation. Time to resolution should begin with the original request and stop only when the postcondition is verified, not when the agent finishes generating text.

    Do not optimize ticket deflection or containment in isolation. The absence of a ticket can represent resolution, abandonment, or a user working around the problem. Pair support-efficiency measures with product success, repeat contact, and safety guardrails.

    Use evaluations and experiments for different questions

    A disciplined AI product rhythm connects eval-driven development, A/B testing, minimum detectable effect, activation, retention analysis, and data governance. Each mechanism answers a different question:

    • Pre-release evaluations: Can the system interpret known intents, select the right context, follow policy, choose an allowed tool, handle tool errors, and verify the expected postcondition? Run the relevant suite whenever the model, prompt, context contract, policy, tool, or playbook changes.
    • Shadow operation: What would the agent have proposed in real traffic without being allowed to change state? Review mismatched diagnoses, unsupported context, unsafe actions, and missed escalation conditions.
    • Controlled experiments: Does the agent improve the predefined outcome compared with the existing support experience for the eligible population? Record assignment before the interaction and preserve it through outcome analysis.
    • Production monitoring: Are errors, reversals, escalations, latency, or policy blocks changing by journey, user role, entitlement, playbook, or release version?

    Be careful with naive correlation. Users who invoke support are often already struggling, so their outcomes may look worse than those of users who never needed help. Random assignment among eligible users gives you a defensible counterfactual. When randomization is not possible, describe the result as observational and avoid claiming that the agent caused the change.

    Log enough version information to reproduce a decision: model, prompt, policy, context schema, playbook, experiment assignment, tool version, input identifiers, authorization result, and postcondition. Do not place raw secrets or unrestricted personal data in that trace. A metric change is actionable only when you can connect it to the system version that produced it.

    Set action boundaries before the model receives tool access

    Model confidence is not authority. A highly confident response must never expand a user’s permissions, bypass confirmation, or convert a prohibited action into an allowed one. Authorization belongs in deterministic policy and tool infrastructure outside the model.

    Action classTypical scopeRequired controlsVerification
    Read and explainShow relevant state, explain an error, or recommend a next stepUser-scoped reads, field filtering, and visible uncertainty when evidence conflictsConfirm that the response used current state and an approved knowledge path
    Reversible changeUpdate a non-sensitive preference, run validation, or trigger a recoverable workflowPreview, confirmation when needed, typed input, idempotency, and rollbackRead the resulting state and observe the workflow’s success event
    Consequential changeAlter billing, permissions, security, external communication, or retained dataStrong confirmation or human review, separation of duties, and a complete audit trailVerify every postcondition and provide a safe recovery or escalation route

    Implement the boundary with controls the agent cannot negotiate away:

    • Least-privilege credentials: issue short-lived, user-scoped authorization rather than a general service credential wherever the architecture permits it.
    • Allowlisted tools: expose narrow actions with typed parameters, explicit preconditions, and constrained targets. Do not give a customer-facing agent arbitrary database or shell access.
    • Policy before execution: validate identity, permission, data sensitivity, action class, and confirmation status outside the model before any state-changing call.
    • Postcondition checks: require the agent to read the resulting state. A successful API response can still produce the wrong business outcome.
    • Safe retries: attach idempotency controls to operations that might be repeated after a timeout or interrupted conversation.
    • Complete handoffs: send the human operator the intent, relevant context, diagnosis, attempted action, tool result, and unresolved condition so the customer does not have to start over.
    • Controlled release: use feature flags, cohort restrictions, action-level limits, and an immediate disable path while a workflow is being validated.

    Evaluate build-versus-buy decisions at the system boundary

    Conversation quality is easy to demonstrate and difficult to use as a purchasing criterion. Evaluate an agent platform on whether it can operate inside your context, permission, observability, and experimentation model.

    • Can you define and inspect the context contract for each workflow?
    • Can the platform use user-scoped credentials and enforce tool permissions outside the prompt?
    • Can every decision, action, version, and outcome be exported to your unified analytics platform?
    • Can you separate aggregate analytics access from individual customer support access?
    • Can you run offline evaluations, shadow traffic, controlled experiments, and cohort rollouts?
    • Can you configure confirmation, rollback, handoff, retention, and data-residency policies?
    • Can you change the model, tool, or support system without losing metric definitions and historical outcome traces?

    A platform that generates excellent dialogue but cannot expose its action trace or connect to verified outcomes will make governance and product measurement harder. A less theatrical system with clear contracts may be the more useful product foundation.

    Key takeaways

    • Start with a governed context contract, not a larger prompt or model.
    • Connect product analytics and support through shared identities, metric definitions, outcome events, and versioned playbooks.
    • Give customer-facing agents user-scoped context and a small set of reversible, allowlisted actions.
    • Count a resolution only when the intended product state or success event is verified.
    • Use offline evaluations for capability and policy, controlled experiments for causal impact, and production monitoring for drift and safety.
    • Expand autonomy only after context accuracy, tool reliability, outcome lift, and guardrails have all been demonstrated.

    At your next roadmap review, ask for one pilot contract rather than a broad AI support initiative. Choose one recurring journey, name its verified success event, define the smallest safe action set, and make the owner show how every action will be authorized, observed, and reversed. That is enough to move from a chatbot concept to an agentic product you can manage.

    References

  • Auditable AI Code Review: A Practical Operating Model

    Auditable AI Code Review: A Practical Operating Model

    You are not deciding whether an AI model can find bugs in a pull request. You are deciding whether an automated reviewer can participate in a production control without leaving your team unable to explain, challenge, or reverse its decision.

    If the only evidence behind an approval is a bot comment that says the change looks safe, keep the system advisory. An auditable AI reviewer needs a bounded mandate, a deterministic approval policy, traceable evidence, and a feedback loop tied to production outcomes. Build those controls first, and faster review becomes a consequence rather than a gamble.

    Start with a decision contract, not a model prompt

    An approval is a policy decision. The model can supply findings, evidence, and a recommendation, but it should not define the conditions under which its own recommendation becomes authoritative.

    Write a decision contract before selecting a model or tuning a prompt. It should answer five questions:

    • What may the system decide? Typical outcomes are approve, request changes, provide non-blocking comments, or escalate to a person.
    • Which changes are eligible? Eligibility should be determined by explicit repository, path, change-type, test, ownership, and reversibility rules.
    • Which checks are mandatory? An eligible pull request should not be approved if a required review lens failed to run, returned incomplete evidence, or produced an unresolved blocking finding.
    • When must the system abstain? Missing context, conflicting findings, unavailable tools, excessive scope, low-confidence evidence, and protected code paths should cause escalation rather than optimistic approval.
    • Who owns the result? Name the engineer accountable for the change, the owner of the review policy, and the person or group authorized to change the automation boundary.

    The core approval rule can be expressed plainly: the change is eligible, every mandatory check completed, no blocking issue remains, the evidence record is complete, and no human-review requirement was triggered. Encode that rule in a controller your team can inspect and test. Do not bury it inside natural-language instructions to the model.

    This separation gives you a clean control plane. Review agents analyze the change. A policy engine evaluates their structured results. A narrowly permissioned service performs the approved action. The model never gets to reinterpret the boundary at the moment it encounters a difficult pull request.

    Auditability does not require a future model run to reproduce the same words. Model endpoints, retrieved context, and dependencies can change. It requires the original decision to remain reconstructable from preserved inputs, outputs, policies, tool results, and versions. A skeptical engineer should be able to determine why the pull request was approved without trusting the personality or reputation of the bot.

    Split the review into specialist checks with explicit evidence

    A single prompt asking whether a pull request is safe compresses several different judgments into one opaque answer. Decompose the review so that each judgment has a clear purpose, input set, output schema, and failure mode.

    A practical review pipeline can include these specialist lenses:

    • Problem-definition quality: Is the requested behavior specific enough to review, and are the acceptance conditions testable?
    • Intent alignment: Does the diff implement the stated change without silently expanding or contradicting it?
    • Scope and dependency impact: Which callers, data flows, interfaces, jobs, or services can the change affect outside the edited files?
    • Logical correctness: Do the changed execution paths handle expected states, boundary conditions, and failure paths?
    • Test adequacy: Do the tests exercise the behavior that changed, and did the required checks actually run against the reviewed commit?
    • Security and privacy: Does the change alter trust boundaries, permissions, authentication, secrets, sensitive data handling, or externally controlled inputs?
    • Local engineering guidance: Does the implementation comply with versioned repository conventions, architectural constraints, and known anti-patterns?
    • Deployment and recovery: Can the change be observed, disabled, or rolled back without creating a second unsafe operation?

    Every specialist should return the same minimum structure: a check identifier, pass/fail/escalate status, a concise claim, evidence tied to files or tool output, the applicable rule version, severity, and a recommended action. A finding such as a possible regression is not auditable. A finding that identifies the affected path, explains the conflicting behavior, points to the relevant code, and names the violated policy is.

    Run independent checks before aggregation, and preserve every result even when the final decision is approval. The aggregator may deduplicate findings, but it should not erase dissent. If the intent checker says the change is aligned while the execution-path checker finds a contradiction, route the conflict to a person.

    Review context must extend beyond the visible diff. A seemingly harmless one-line copy change was once found to contradict validation behavior elsewhere in the codebase. That is the kind of defect a diff-only reviewer is structurally unlikely to see. Give relevant checks controlled access to callers, validators, schemas, tests, ownership metadata, and versioned internal guidance, then record exactly which context each check used.

    More context is not automatically better. Retrieval should be targeted and attributable. When a finding depends on an internal rule, capture the rule identifier and version. When it depends on a test, capture the command, commit, status, and output reference. When it depends on inferred execution flow, record the relevant path so a maintainer can inspect it.

    Treat pull-request text, code comments, test fixtures, generated files, and documentation on the changed branch as untrusted data. They can contain instructions designed to redirect an agent. Load approval policy from a protected service or the trusted base branch, not from files the pull request can rewrite. Run proposed code in an isolated environment, mediate tool calls through an allowlist, and keep approval or merge credentials outside the model’s reach. The policy controller should translate a valid decision into an action; the model should never hold the credential that performs it.

    Set the automation boundary with hard eligibility gates

    Do not begin by assigning every pull request a risk score and approving anything below a convenient number. A composite score can hide a disqualifying condition: a tiny authorization change may receive a low size score even though its blast radius is high. Apply hard gates first. Use scoring only to route changes that remain eligible after those gates.

    Common reasons to require human review include:

    • Authentication, authorization, permissions, cryptography, secrets, or trust-boundary changes.
    • Payments, billing, entitlements, destructive data operations, or irreversible migrations.
    • Public API contracts, shared schemas, release infrastructure, or broadly consumed dependencies.
    • A pull request that changes its own review policy, test requirements, ownership rules, or deployment controls.
    • Missing required tests, failed or stale CI results, unavailable analysis tools, or a mismatch between the reviewed commit and the tested commit.
    • Changes spanning too many concerns, components, or execution paths for the approved review envelope.
    • An active incident, an unclear rollback path, or a direct request for human review.

    There is no universal line-count threshold for a small pull request. Derive limits from your architecture and incident history, then version them. A change to a central permission function may be riskier than a much larger isolated test refactor. Scope should include dependency reach and behavior change, not just added and deleted lines.

    A staged authority model keeps the boundary legible:

    ModeWhat the AI reviewer may doWho decides the mergeAppropriate use
    ShadowProduce a private decision record without affecting the pull requestHuman reviewerBaseline evaluation and policy tuning
    AdvisoryPost evidence-backed, non-blocking findingsHuman reviewerMeasuring usefulness and false alarms in normal work
    BlockingRequest changes for narrow, testable policy violationsHuman reviewer after resolutionStable rules with clear evidence and an appeal path
    Bounded approvalApprove only changes that pass every eligibility and review conditionPolicy controller within its delegated scopeValidated low-risk change classes with complete audit records
    Mandatory escalationSummarize evidence and route the changeNamed human ownerSensitive paths, conflicting findings, missing evidence, or any requested human review

    Do not turn bounded approval into an auto-approval quota. Coverage is a result of demonstrated safety, not a target that should pressure teams to weaken eligibility rules.

    One high-frequency engineering environment reports that more than 93% of pull requests across two main codebases are agent-driven and more than 19% are approved without a human reviewer. Its reported median merge time fell from 75.8 minutes with human review to 14.6 minutes with AI approval, while downtime from breaking changes declined 35% as deployments doubled. Those organization-level results show that bounded automation can coexist with high deployment frequency and improving safety outcomes. They do not prove that AI approval caused the downtime reduction, and they should not be imported as another team’s launch threshold.

    Keep the escape hatch explicit. Any engineer should be able to request a human review without defending the choice. The accountable engineer should still watch the change in production and be ready to roll it back. Automated approval changes who performs a review step; it does not transfer ownership of the production outcome to a model.

    Preserve the evidence, then earn autonomy through evaluation

    Build a decision record that survives model and policy changes

    Create an append-only decision event for every review attempt, including abstentions and failed runs. At minimum, retain:

    • Repository, pull-request identifier, base commit, reviewed head commit, author, accountable owner, and timestamps.
    • The pull-request description and acceptance criteria as they existed when the decision was made.
    • Eligibility rules, protected-path rules, ownership data, prompt-template identifiers, and policy versions.
    • Model provider and model identifier, relevant runtime settings, retrieval configuration, and tool versions.
    • The context each specialist received, including immutable references or preserved snapshots for mutable material.
    • Structured specialist outputs, supporting evidence, tool invocations, CI results, conflicts, and failures.
    • The deterministic rule evaluation that produced approve, block, comment, or escalate.
    • Subsequent human overrides, appeals, edits, approvals, merges, rollbacks, hotfixes, and linked incidents.

    Store concise decision rationale and inspectable evidence, not hidden chain-of-thought. An auditor needs to know which claim was made, what supported it, which rule applied, and how the controller reached the outcome. Private internal reasoning is neither necessary nor a reliable substitute for those artifacts.

    Apply the same security discipline to review logs that you apply to source code. Minimize captured secrets and personal data, control access, define retention, and log policy changes. If a model or retrieval service cannot handle the code under your data-governance requirements, that repository is not eligible for the workflow.

    Evaluate decisions, not polished comments

    A review can sound thoughtful and still approve the wrong change. Build an evaluation set around decisions and evidence rather than writing quality.

    1. Assemble representative cases. Include clean pull requests, valuable historical human findings, escaped defects, incident-causing changes, incomplete requirements, sensitive paths, cross-component changes, and attempts to manipulate the reviewer through repository content.
    2. Label the expected control outcome. For each case, identify whether the correct action is approve, request changes, or escalate. Record the evidence that an acceptable review must surface.
    3. Separate clear cases from disputed ones. Known incident causes and explicit policy violations can provide strong labels. Ambiguous architectural judgments need maintainer adjudication, and disagreement should remain visible rather than being forced into false certainty.
    4. Freeze a holdout set. Use one portion to improve prompts, retrieval, and policy. Keep another portion unseen until release evaluation so repeated tuning does not create a misleading score.
    5. Compare equivalent cohorts. Evaluate AI and human review on the same risk classes and change types. Comparing AI-approved low-risk changes with all human-reviewed pull requests confounds reviewer quality with task difficulty.

    Track metrics that expose different failure modes:

    • Decision accuracy: How often did the system choose the expected approve, block, or escalate outcome?
    • False auto-approval rate: How often did it approve a labeled case that should have been blocked or escalated? Break this out by severity and risk class.
    • Blocking precision: Of the findings that stopped a change, how many maintainers judged valid and actionable?
    • Known-defect recall: Which seeded or historically verified defects did the review catch? Label this carefully; it is not recall over every defect that might exist.
    • Evidence completeness: Can every decision be traced to required checks, immutable inputs, policy versions, and supporting artifacts?
    • Abstention and override rates: Where is the system uncertain, and where do engineers reverse it? Investigate patterns by repository and change class.
    • Delivery performance: Measure review latency and merge time, but only alongside quality metrics.
    • Production outcomes: Track rollbacks, hotfixes, escaped defects, incidents, downtime, and customer impact for comparable risk cohorts.

    Comment helpfulness is useful feedback, but it is not a safety metric. Engineers may like a concise reviewer that misses a critical defect, or dislike a strict reviewer that correctly blocks an unsafe change. Keep usefulness, correctness, and production impact as separate measures.

    Roll out by change class and turn escapes into regression tests

    Move from shadow mode to advisory comments, then to narrow blocking rules, and only then to bounded approval. Start with one repository and one low-risk, reversible change class. Write the exit criteria before the pilot begins, including acceptable false-approval and false-block rates, required audit completeness, escalation behavior, and production guardrails.

    Canary each expansion. Maintain a kill switch that disables new automated approvals without removing the accumulated evidence. If a required service, model, retrieval index, test runner, or policy store is unavailable, fail closed and return the pull request to the human path.

    When an approved change causes a production problem, diagnose the control layer that failed:

    • Was the change wrongly eligible?
    • Did retrieval omit relevant code or guidance?
    • Did a specialist miss or misclassify the defect?
    • Did the aggregator suppress a conflict?
    • Did the policy permit approval despite the evidence?
    • Did CI test a different commit or an incomplete environment?
    • Did production monitoring fail to surface the effect promptly?

    Add the case to the regression suite, version the corrective policy or guidance, rerun the holdout evaluation, and preserve the relationship between the incident and the updated control. That is eval-driven development applied to governance: every escape should make a specific layer harder to fail in the same way again.

    Key takeaways

    • AI output is an input to approval, not the approval policy itself.
    • Use deterministic eligibility gates before any model-based risk judgment.
    • Decompose review into specialist checks that return claims, evidence, rule versions, and explicit pass/fail/escalate states.
    • Keep policy and credentials outside the pull request and outside the model’s control.
    • Preserve enough evidence to reconstruct the original decision even when the model, repository, or internal guidance later changes.
    • Expand autonomy only when evaluation and comparable production cohorts support it; never optimize for auto-approval coverage by itself.

    Your first useful milestone is not an AI-approved pull request. It is a shadow decision that a maintainer can reconstruct, dispute, and improve. Once that record is dependable, grant the smallest reversible slice of authority, watch what reaches production, and make every expansion earn its place.

    References

  • A Practical Scenario Planning System for AI Product Strategy

    A Practical Scenario Planning System for AI Product Strategy

    Your leadership team wants a firm answer: which AI bets belong on the roadmap, and which ones are expensive distractions? The difficult part is not generating ideas. It is deciding what to fund when customer adoption, interfaces, economics, and enterprise constraints could each develop differently.

    A useful scenario plan does not hide that uncertainty behind a confident forecast. It converts uncertainty into conditional commitments: what you fund now, what you preserve as an option, what evidence would change the decision, and what you refuse to scale until the right signal appears.

    Frame the exercise around a decision, not the future of AI

    Broad questions such as “What will AI look like?” produce interesting conversations and weak strategy. Nobody has to choose anything at the end. Start with a decision that has an owner, a planning window, and a meaningful consequence if the underlying assumptions prove wrong.

    A usable decision statement looks like this: “For this customer and workflow, should AI become the primary experience, remain an embedded assistant, or stay in discovery while the existing product carries the outcome?”

    Write down five elements before discussing scenarios:

    • The decision: State the product, investment, or sequencing choice that must be made.
    • The planning window: Define when the choice needs to be made and when it can be reconsidered.
    • The expensive assumption: Identify what must be true for the proposed strategy to work.
    • The reversal cost: Separate choices that can be changed cheaply from commitments involving substantial architecture, hiring, migration, or go-to-market work.
    • The decision owner: Name the person accountable for changing the plan when the evidence changes.

    A scenario belongs in the exercise only if it could change that decision. If two imagined futures lead to the same investment, combine them. More narrative does not create more strategic value.

    Next, identify the customer need that should remain valid across the plausible futures. Customers may still need to complete a workflow with less effort, avoid costly rework, understand what the system did, or retain control over a consequential action. Those durable needs become the anchor. They keep an empowered product team focused on outcomes when a particular interface, model, or market prediction stops holding up.

    Build scenarios from uncertainties that can change the plan

    Do not begin by writing a best case, a base case, and a worst case. That format encourages everyone to treat the base case as the forecast. Instead, find uncertainties that are both unresolved and capable of changing your product choice.

    Useful uncertainty prompts include:

    • Customer behavior: Does repeated AI use spread through the intended market, or remain concentrated among enthusiasts and specialists?
    • Interaction model: Does AI become the main way customers initiate work, or does it operate inside a familiar interface?
    • Scope of autonomy: Do customers delegate complete tasks, or accept AI only for bounded assistance and recommendations?
    • Product economics: Does the value created support the cost and operational burden of delivering the experience?
    • Enterprise constraints: Do security, privacy, compliance, procurement, and change-management requirements permit the proposed workflow?
    • Organizational readiness: Can the company evaluate, support, govern, and improve the product after launch?

    Select the uncertainties with the greatest decision impact, then push competing possibilities to useful extremes. Extremes expose assumptions that a comfortable middle case can conceal. “Graphical interfaces disappear” and “AI remains an invisible utility” should not be treated as predictions. They are boundary conditions for examining what your product would need in very different environments.

    For example, crossing the pattern of customer adoption with AI’s place in the workflow creates the following set of hypothetical futures:

    Adoption patternAI’s place in the workflowPlausible futureDecision it tests
    Use broadens across intended segmentsAI becomes the primary interactionCustomers increasingly start and complete the workflow through AIWhether to redesign the core experience around an AI-first path
    Use broadens across intended segmentsAI remains embeddedAI becomes valuable infrastructure inside a familiar productWhether intelligence, context, and workflow integration matter more than a new interface
    Use remains unevenAI becomes primary for specialistsA smaller group wants an AI-native experience while the broader market retains existing habitsWhether to support distinct experiences instead of forcing one migration
    Use remains unevenAI remains embeddedBounded assistance improves parts of the workflow without replacing itWhether focused augmentation is a better investment than broad transformation

    The point is not to choose your favorite quadrant. Develop each one far enough to reveal its product implications. For every scenario, describe the target customer’s behavior, the job that still matters, the role AI plays, the constraints that become binding, and the most likely way the strategy fails.

    Keep the scenarios plausible rather than theatrical. A future that cannot affect a real decision is entertainment. A future that merely restates the current roadmap is confirmation bias.

    Turn each scenario into signals, triggers, and stop conditions

    A scenario without observable signals is just a story. A decision-ready scenario tells you what to watch, how that evidence relates to an assumption, and what action follows if the signal appears.

    The common mistake is to monitor whatever is easiest to count. Trial starts, demo enthusiasm, and requests from technically confident customers can show interest, but they do not establish broad adoption. Early adopters cannot stand in for the whole market. Segment the evidence so that enthusiasm in one cohort does not silently become a claim about every customer.

    Build a signal set that answers distinct questions:

    • Adoption: Are intended customers returning to the AI workflow after the initial trial, and is use spreading beyond opt-in enthusiasts?
    • Customer value: Is the core outcome improving through less effort, less rework, fewer avoidable errors, or another measure that matters for this workflow?
    • Trust and control: How often do customers accept, modify, override, or abandon the result, and what reason do they give?
    • Enterprise viability: Are security, compliance, procurement, or change-management reviews blocking deployment or narrowing the acceptable use case?
    • Operational viability: Are reliability, latency, support demand, and cost-to-serve compatible with the value being delivered?
    • Interface behavior: Do customers initiate work through AI, or invoke AI at specific points inside an established process?

    Pair every confirming signal with a disconfirming signal. This prevents the team from collecting only evidence that supports the roadmap it already wants.

    Strategic assumptionConfirming evidenceDisconfirming evidenceProduct response
    AI should become the primary interfaceIntended customers repeatedly initiate and complete the core workflow through AICustomers retreat to the familiar interface for consequential workKeep the AI-first experience as an option while improving embedded assistance and control
    Broader autonomy will create more valueOutcome quality improves while customer intervention and rework declineEscalations, corrections, or abandonment persist as scope expandsNarrow the delegated task and strengthen evaluation, permissioning, and fallback behavior
    Adoption can expand through the target marketRepeated use spreads across intended cohorts and survives enterprise reviewUse remains concentrated among specialists or stalls during approval and rolloutPreserve a dual experience and address the blocking constraint before funding broad migration

    Write the trigger before launching the bet. A practical format is: “When this signal persists in the target segment and this guardrail remains acceptable, move this investment from an option to a commitment. If this disconfirming signal appears, stop, narrow, or redesign the bet.”

    The exact threshold will depend on your product, baseline, risk, and decision cost. What matters is agreeing on it before stakeholders can reinterpret ambiguous results. If nobody can say what evidence would reduce or end the investment, the roadmap contains a belief, not a testable strategy.

    Convert the scenarios into a portfolio, roadmap, and sprint choices

    Once the scenarios and signals are explicit, separate the portfolio by commitment type. This is where scenario planning becomes operating discipline rather than an occasional workshop.

    • No-regret bets: Investments that support the durable customer outcome across several scenarios. Depending on the product, these may include better evaluation, permissions, observability, fallback paths, data governance, or clearer measurement. Do not label generic platform work “no regret” unless it supports a named customer outcome.
    • Option bets: Bounded, reversible work that buys information or preserves a future choice. A prototype, limited workflow, architecture seam, or controlled release can test an assumption without committing the entire product.
    • Contingent bets: Investments that make sense only after a defined signal appears. Keep the entry condition beside the roadmap item so it cannot become committed work through inertia.
    • High-regret commitments: Expensive moves that are difficult to reverse, such as a forced workflow migration or a large architecture and hiring commitment. Require stronger support across scenarios before making them.

    This creates a roadmap with different funding postures, not a backlog pretending every item has equal certainty.

    Roadmap laneWhy it existsWhat earns progressWhat removes it
    Durable outcomesAdvance needs that remain important across plausible futuresEvidence that the customer outcome is improvingThe need or outcome no longer matters
    Evidence betsReduce uncertainty between competing scenariosA learning milestone tied to a strategic assumptionThe assumption is resolved or the evidence cannot affect a decision
    Triggered scaleExpand an option after its entry condition is metThe agreed confirming signal appears while guardrails remain acceptableA stop condition appears or the economics no longer support expansion
    Deferred commitmentsPreserve ideas that are valid only in a narrower futureA named scenario becomes more plausible through observable evidenceThe relevant scenario is disconfirmed

    At sprint planning, ask what kind of item is entering delivery. Work should either create durable customer value or buy decision-relevant information. “Build an AI assistant” is an output. “Determine whether target customers will delegate this bounded task while retaining acceptable control” is a learning goal that can change a strategic choice.

    For each evidence bet, require the product trio to answer:

    • Which scenario and assumption does this work test?
    • Which customer segment must provide the evidence?
    • What behavior or outcome will be observed?
    • What result would justify more investment?
    • What result would stop, narrow, or redirect the work?
    • Which part of the work remains valuable if the favored scenario is wrong?

    This also changes the stakeholder conversation. Replace “Which prediction do you believe?” with “Which commitments are justified across plausible futures, which ones are options, and what signal unlocks the next level of funding?” The latter question makes uncertainty governable.

    Review the decision when a trigger fires, a critical assumption changes, or the next expensive commitment approaches. A recurring calendar review can help, but elapsed time alone is not evidence. Keep a short decision record containing the active scenarios, current signals, funding posture, stop conditions, owner, and next decision point.

    Key takeaways for your next AI roadmap review

    • Start with a product or investment decision that could genuinely change. Do not try to describe the entire future of AI.
    • Build scenarios from unresolved uncertainties with high decision impact, including customer behavior and real-world constraints.
    • Anchor the strategy in customer needs and outcomes that remain valuable across several plausible futures.
    • Separate confirming evidence from disconfirming evidence, and segment adoption so enthusiasts do not masquerade as the whole market.
    • Predefine the signal, guardrail, trigger, and stop condition for every material option bet.
    • Fund no-regret moves now, use reversible work to buy information, and hold contingent commitments until their entry conditions appear.
    • Connect every sprint item to either durable value or a decision-relevant uncertainty.

    At your next roadmap review, choose the most expensive assumption behind one AI initiative. Write the opposite plausible scenario, identify the customer need shared by both futures, and name the signal that would change the funding decision. If the roadmap still makes sense, the strategy is more resilient. If it does not, you have found the adaptation point before the market finds it for you.

    References

  • AI-Native Startup Execution: A Practical Operating System

    AI-Native Startup Execution: A Practical Operating System

    Your startup can produce impressive demos every week and still learn slowly. If model behavior, customer evidence, and deployment feedback move through separate queues, faster shipping only creates more uncertainty.

    If you are deciding how to run an AI-native startup, use one standard: how quickly can your team turn a real customer artifact into an evaluated product change and a measurable customer outcome? That standard should shape your wedge, ideal customer profile, team design, sales motion, and operating cadence.

    Build the smallest closed learning loop

    An AI-enabled company can add a model to an existing product process. An AI-native company has to organize the process around intelligence itself: the data entering the system, the judgment the model makes, the action produced, the evaluation of that action, and the feedback that improves the next decision.

    That makes a long AI feature list the wrong starting point. Begin with the smallest end-to-end loop that proves the product’s central claim. In a security product, for example, the loop might start with suspicious activity, produce a risk judgment, stop or downgrade a threat, and capture enough evidence to assess whether the intervention was correct. A polished dashboard without that closed loop is packaging, not proof.

    Use three gates before committing to a wedge:

    1. The customer can quantify the problem. Ask for frequency, severity, operational burden, or another observable consequence in the customer’s own workflow. General concern about AI is not enough.
    2. The wedge can create value within one buying cycle. If proving value requires a broad platform rollout, several integrations, and organizational change across multiple functions, you have probably selected a destination rather than an entry point.
    3. Product use improves your data advantage. Determine what feedback the product will capture, whether you can legitimately use it, how quickly it arrives, and which evaluation or model decision it improves. Data volume alone does not create an advantage.

    Failing any one of these gates weakens the loop. Urgent pain without accessible data leaves the model blind. Useful data without a near-term outcome produces an experiment that customers may admire but will not adopt. A fast result that teaches you nothing can become a services engagement disguised as software.

    Your first version should feel narrower and less polished than the roadmap in your head. It may require manual review, a rough operator interface, or close founder involvement. That is acceptable when the roughness is visible and recoverable. It is not permission to hide model failure, skip evaluations, or make consequential actions impossible to inspect. Cut breadth before you cut control.

    Define completion around the loop rather than the interface. You are done with the first proof when a target customer can supply the relevant input, receive the intended outcome, show whether it helped, and feed the result back into a repeatable evaluation. Everything else competes with learning speed.

    Choose an ICP that maximizes learning density

    The largest market is rarely the most useful starting market. Your early ideal customer profile should concentrate urgency, usable data, workflow access, and a reachable decision maker. Those conditions allow one customer interaction to improve both the product and the go-to-market motion.

    Create a one-page ICP card with four evidence fields:

    • Urgency: What is happening now that makes the buyer act rather than merely agree that the problem matters?
    • Data pathway: Which alerts, cases, decisions, errors, or other operational artifacts can enter the product and its evaluation process?
    • Workflow position: Where will your output appear, who will act on it, and what existing step must change?
    • Success signal: What observable customer outcome would justify adoption, renewal, or expansion?

    Rate each field as high, medium, or low, but require an evidence note beside the rating. A founder’s conviction is not evidence. A buyer who can describe the cost of the problem, provide representative artifacts, identify the operator who owns the workflow, and agree on a success signal gives you something testable.

    Keep adjacent customer segments out of the first learning loop when they require different data, integrations, evaluation criteria, or buying logic. A larger pipeline can make progress look healthier while mixing incompatible requirements into the roadmap. The practical test is simple: if two prospects would judge the same model behavior by different definitions of success, they should not share an early product plan merely because they could buy from the same company.

    Founder-market fit helps here, but it is a compression mechanism rather than proof of product-market fit. Deep domain experience can sharpen the problem statement, reduce translation with buyers, and establish credibility. It cannot substitute for evidence that customers will change a workflow around the product.

    Founder-led sales should therefore operate as a structured discovery system. For every lost or stalled opportunity, record the exact objection, the stage at which it appeared, the evidence offered by the buyer, and the suspected root cause. Do not turn each objection into a feature request. Cluster the objections, identify the two most common root causes, and turn those into experiments within a sprint.

    Look beyond compliments when judging early product-market fit. Stronger signals appear when customers escalate a problem to your team, volunteer data that can improve the system, or invite you into the workflow where decisions are made. None is conclusive alone, but each requires the customer to spend trust, attention, or organizational effort. That makes them more informative than enthusiasm during a demo.

    Organize the team around evaluation and deployment

    A conventional feature organization separates product definition, engineering delivery, model work, implementation, and customer support. That creates handoffs precisely where an AI-native startup needs rapid feedback. Organize durable units around a customer outcome, with the people who can inspect data, change the system, deploy it, and evaluate the result.

    The exact titles can vary, but the unit needs four capabilities:

    • Product judgment: Choose the scenario, define the customer outcome, and decide which uncertainty deserves the next experiment.
    • Model and data judgment: Design evaluations, diagnose failure patterns, and understand whether a change improves one scenario while damaging another.
    • Product engineering: Turn model behavior into a reliable workflow with usable interfaces, permissions, integrations, and operational controls.
    • Frontline deployment: Work safely in live customer environments, resolve implementation friction, and return generalizable learning to the product.

    Forward deployed engineers and solutions engineers are especially valuable when the product has to enter complicated workflows. They can shorten the distance between a live failure and a product decision. They can also become an unofficial custom-development queue. Prevent that by attaching three fields to every customer-specific change: the broader pattern it tests, the product owner responsible for reviewing the result, and the condition under which the work will be standardized, stopped, or kept outside the core product.

    AI fluency should also change your hiring process. Model vocabulary on a resume tells you little about how someone will reason under uncertainty. Give candidates a work sample built around a realistic failure: model performance changes across scenarios, the data distribution is moving, and customer trust is at risk.

    Ask the candidate to define the evaluation, explain the consequences of false positives and false negatives, diagnose a precision-recall tradeoff, and translate the technical result into a product decision. The strongest response will not jump straight to a model change. It will clarify the scenario, inspect the evidence, identify what the current evaluation misses, and explain the tradeoff in terms an operator or buyer can use.

    Use the same discipline for founder and leadership disagreement. Write down the principle in dispute, the evidence each side is using, the decision owner, the time box, and the condition that will trigger a review. Debate the principle rather than multiplying competing implementation plans. Once the decision is made, commit until the agreed review point. This preserves speed without pretending that an uncertain decision has become permanently correct.

    Run model and business evidence on the same cadence

    An AI-native startup needs two connected scoreboards. One shows whether the system is becoming more reliable. The other shows whether customers are receiving enough value to adopt, retain, and expand. Reviewing them separately makes it easy to celebrate a model improvement that does not change customer behavior, or a short-term commercial win built on fragile product performance.

    Evidence layerQuestionMeasures to inspectDecision it should inform
    Scenario qualityDoes the system make the right judgment in each important situation?Precision and recall by scenario; recurring misclassification patternsEvaluation coverage, data work, model changes, and acceptable operating thresholds
    Change detectionDoes the system recognize when its environment is moving?Drift, data freshness, and time to first signal for an emerging patternRefresh cadence, investigation priority, and new evaluation cases
    Operational experienceCan the customer use the output without creating a new burden?Alert-fatigue reduction and latency under loadWorkflow design, prioritization, and deployment constraints
    Business valueIs reliable behavior turning into durable adoption?Activation, expansion, and Net Recurring RevenueICP focus, onboarding, positioning, and roadmap investment

    Do not collapse these measures into one composite score. Aggregation hides the failure mode you need to fix. A model can improve overall precision while degrading a high-consequence scenario. Activation can remain flat even when evaluation results improve because the output arrives too late, lacks an escalation path, or does not fit the operator’s workflow. The broken link between the two scoreboards is often the most valuable finding in the review.

    Make the weekly customer debrief evidence-based. Bring actual alerts, misclassifications, escalations, and deployment failures rather than a verbal account of how the customer feels. For each material artifact, record the hypothesis it challenges, the evaluation that can reproduce it, the next bet, and the owner. That written log becomes organizational memory when priorities change quickly.

    Use an evaluation gate for every consequential release:

    • Run the relevant scenario-level evaluations, not only an aggregate benchmark.
    • Inspect newly introduced failure modes as well as improvements to the target case.
    • Check latency and operational behavior under the conditions the customer will actually use.
    • Name the owner of monitoring, escalation, and rollback before deployment.

    This is not process for its own sake. In an AI product, observability is the control center for trust. It tells your team whether the model still deserves the authority the workflow gives it. Demo speed cannot compensate for an inability to see, explain, and correct failure in production.

    Key takeaways for your next operating cycle

    • Define the product as a closed loop from customer signal to verified outcome, not as a list of AI capabilities.
    • Commit to a wedge only when the problem is quantified, value can appear within one buying cycle, and legitimate product use improves the data advantage.
    • Select the first ICP for urgency, data access, workflow access, and a clear success signal. Keep adjacent segments separate when they require different definitions of success.
    • Turn founder-led sales into a discovery system by logging every rejection, clustering root causes, and testing the two most common causes within a sprint.
    • Build durable customer-facing units that combine product, model, engineering, and deployment judgment.
    • Review scenario-level model evidence and business outcomes together, then investigate the link whenever one improves without the other.

    For your next planning cycle, choose one ICP, one end-to-end loop, one scenario-based evaluation set, and one customer outcome. Give a single owner responsibility for showing the connection between them. If the team cannot trace that chain with production evidence, the roadmap is still too broad. Narrow it before adding another feature, segment, or model.

    References

    • Shivam.Consulting Blog – Inside Artemis’ AI vs AI Security War: Hiring at Speed, PMF Signals, and Founder-Led Sales
  • From 70 Employees to Dominance: My Playbook for Hypergrowth, Focus, and Top-Down Goals

    From 70 Employees to Dominance: My Playbook for Hypergrowth, Focus, and Top-Down Goals

    Scaling a real-world marketplace from scrappy to dominant takes a different kind of product leadership. Reflecting on Christopher Payne’s decade leading DoorDash as President and COO — growing from roughly 70 employees to the dominant food delivery platform in the US — I’m struck by how much of that success hinged on mastering an atoms-based business while still operating with software-level rigor. As a VP of Product Management, I see the same patterns in my own work: relentless clarity on inputs, a bias for builder-executives, and a cadence that keeps leaders close to product details without becoming bottlenecks.

    Running an atoms-based business versus a pure software company forces you to obsess over operational physics: unit economics, quality control, on-time reliability, and dense local liquidity. It’s precisely where traditional “bits” executives can stumble. What’s worked for me is a simple “plate spinning” framework for executive attention: identify the five or six plates that must never stop — customer experience, marketplace health, quality and safety, product velocity, platform reliability, and P&L — then schedule recurring deep dives to keep those plates spinning. If a plate wobbles, I drop in, fix the root cause, re-instrument the inputs, and zoom back out.

    Hiring at hypergrowth speed only works when you bias toward a “builder mentality.” I look for executives who run toward fuzzy problems, write clearly, and can prove they’ve shipped value with incomplete information. Prior industry experience can be a liability when you’re reinventing the market; first-principles thinkers outlearn domain experts who try to port yesterday’s playbooks. In executive hiring, I’ve found structured work samples and narrative memos far more predictive than marathon interview loops — companies routinely spend too much time on job interviews and too little time evaluating how candidates think and execute.

    Great executives never outgrow the details. Staying close doesn’t mean micromanaging — it means sampling the customer journey and instrumenting the system so you can feel where it hurts. In my own practice, I rotate through frontline touchpoints weekly: support transcripts, NPS verbatims, failed checkout sessions, and reliability dashboards. Small signals often reveal systemic issues. A single ciabatta bread moment — the kind of edge-case substitution that seems trivial — can expose broken handoffs, unclear policies, and misaligned incentives across the marketplace.

    Top-down goal setting beats bottom-up when you’re aiming for category leadership. Bottom-up targets tend to regress to comfort; they calibrate to today’s constraints, not tomorrow’s possibilities. I set ambitious, top-down outcomes (not output), frame the non-negotiables, and map driver trees to clarify the input metrics that matter. Then I ask empowered product teams to pressure-test the plan, propose approaches, and own the how. This preserves ambition while unlocking creativity — a practical balance of clarity and autonomy that outcomes vs output OKRs were designed to achieve.

    One-size-fits-all management is a myth. Early-stage teams need hands-on coaching and fast decisions; later-stage teams need mechanisms that scale: crisp PRDs, pre-mortems, and operating cadences that separate strategy, planning, and execution. The mark of a high-functioning executive team is not uniform style — it’s high candor, fast escalation paths, and visible commitment after debate. In tough moments, a little charisma goes a long way; in practice, that’s not theatrics, it’s steady optimism, simple language, and consistent follow-through that keeps people moving forward.

    The hypergrowth skill stack for executives is surprisingly learnable: ruthless prioritization under uncertainty, narrative writing that aligns cross-functionally, structured delegation with clear “inspection points,” and a weekly rhythm that protects maker time. I leverage a cadence of business reviews (inputs > outputs), customer-scent checks, and decision logs so we can move fast without losing the thread. CEO and executive time management is the ultimate forcing function — if we can’t show where our attention maps to goals, the team won’t either.

    Some of my enduring lessons echo the best of Amazon and eBay: customer obsession beats competitor obsession, input metrics beat lagging vanity metrics, and simple mechanisms beat heroics. From Jeff Bezos’s playbook I borrow the insistence on written narratives, single-threaded ownership, and clarity on what will not change. Those principles remain the backbone of platform scalability and resilient product strategy, especially when markets get noisy.

    AI is about to flatten organizations. With agentic AI, retrieval-first pipelines, and AI workflows embedded into product development, managers can widen their span without losing fidelity. I see LLMs for product managers accelerating discovery, PRD drafting, and experiment analysis — while raising the bar on decision quality. The implication for leadership: fewer layers, more transparency, and even greater pressure to define sharp, top-down outcomes that teams can autonomously pursue.

    If I had to compress this into a playbook, it’s this: set audacious, top-down goals; keep your “plate spinning” calendar sacred; write more than you talk; hire builders, not resume archetypes; sample the customer journey every week; and build mechanisms that make the right thing easier than the heroic thing. That’s how you scale product management leadership from dozens to thousands — in atoms, in bits, and in the messy, exhilarating space where they meet.

    Book a consult png image
  • Outcome-Driven Product Development: A Practical Operating Model

    Outcome-Driven Product Development: A Practical Operating Model

    Your team has hit its release dates, the roadmap is moving, and the launch calendar is full. Then someone asks the question that exposes the problem: what changed for the customer or the business? If the answer is a list of shipped features, activity is visible but progress is still uncertain.

    Outcome-driven product development closes that gap. It gives a team a measurable problem to solve, enough freedom to change its solution as evidence changes, and a clear point at which exploration should become committed delivery. The result is not less shipping. It is less investment in work that has never earned the right to scale.

    Replace the feature request with an outcome contract

    A feature describes what the team will produce. An outcome describes the change the team intends to cause. That distinction sounds simple, but it changes planning, discovery, measurement, and accountability.

    Suppose the roadmap says, “Launch an AI onboarding assistant.” The statement has a deliverable, but it leaves the important questions unanswered. Which customers are struggling? What behavior needs to change? How will the assistant cause that change? What evidence would justify continued investment? What must not get worse?

    Rewrite the request as an outcome contract before discussing scope. A useful contract contains:

    • Target customer and context: the specific user or account segment experiencing the problem, plus the situation in which it occurs.
    • Problem evidence: the interviews, behavioral data, support patterns, or other observations showing that the problem is real.
    • Behavioral outcome: the customer action that should become more frequent, successful, or efficient if the problem is solved.
    • Business connection: the reason that behavior matters to activation, conversion, retention, revenue, cost, or another strategic result.
    • Baseline, target, and measurement window: the current state, the intended direction or threshold, and the period over which the change will be assessed.
    • Guardrails: the customer, operational, financial, ethical, or reliability measures that must not deteriorate while the primary metric improves.
    • Decision point: the evidence that will trigger scaling, iteration, another experiment, or stopping the work.

    The rewritten AI onboarding bet might be: “For new workspace administrators who struggle to complete setup, increase the share who reach their first useful workflow during onboarding, without increasing support demand or incorrect automations.” The assistant is now one possible solution, not the definition of success.

    That wording gives the team room to discover that a checklist, better defaults, a guided setup flow, clearer copy, or a narrower AI capability solves the problem more effectively. It also prevents a common failure mode: launching the requested feature and retroactively selecting whichever metric happened to move.

    Use three tests before accepting an outcome:

    • If the proposed feature disappeared, would the stated customer and business result still matter?
    • Can the team observe the target behavior with enough precision to distinguish exposure, use, and successful completion?
    • Does the team have permission to change the solution while preserving the outcome and agreed constraints?

    If any answer is no, you probably have a feature commitment decorated with outcome language. Fix that before the roadmap item absorbs a delivery team.

    Use an evidence gate between learning and earning

    Outcome-driven teams still build. The important choice is what they are building for at each moment.

    In build-to-learn mode, the objective is to reduce uncertainty cheaply. The team is trying to understand the problem, test whether a solution is desirable and usable, and expose delivery or business risks before making a large commitment. Customer interviews, lightweight prototypes, assumption mapping, an opportunity solution tree, and narrowly scoped experiments belong here.

    In build-to-earn mode, the objective changes to dependable value capture. The team has enough evidence to invest in production quality, integration, operational readiness, adoption, and scale. Acceptance criteria, sprint planning, release discipline, observability, and post-launch measurement become central.

    These are modes of investment, not separate departments. A product trio can move between them as confidence changes. The practical goal is to learn only until the evidence supports conviction, then move decisively into value capture while keeping discovery alive.

    Make every learning activity answer a decision

    Discovery becomes theater when teams collect feedback without specifying what the feedback will change. Give each experiment a compact brief:

    • Hypothesis: what the team currently believes about the customer, problem, solution, or expected behavior.
    • Riskiest assumption: the belief most likely to invalidate the bet if it is wrong.
    • Method: the cheapest credible way to test that assumption.
    • Evidence: the observable result that would strengthen or weaken confidence.
    • Timebox: the boundary that prevents exploration from continuing without a decision.
    • Next action: scale, revise, run a different test, or stop.

    Match the method to the question. Interviews can reveal whether a problem exists and how customers describe it, but they do not prove that a shipped solution will change behavior. A prototype can expose comprehension and usability problems, but it does not establish durable adoption. An A/B test can quantify incremental impact when exposure, instrumentation, and analysis are suitable, but it cannot rescue a weak problem definition.

    An opportunity solution tree helps keep those questions connected. Start with the outcome, map the customer opportunities that could influence it, attach possible solutions to those opportunities, and place experiments under the assumptions they test. This makes it easier to notice when a favored solution has become detached from the original problem.

    Define the evidence gate before enthusiasm takes over

    There is no universal discovery duration or confidence score. The appropriate threshold depends on the cost, reversibility, operational risk, and strategic importance of the decision. A narrow, reversible change should not face the same gate as a platform migration or a customer-facing AI system with meaningful risk.

    The team is usually ready to enter build-to-earn mode when it can answer yes to these questions:

    • Is the target problem supported by evidence from the intended customer segment?
    • Does the proposed solution reliably address that problem in the contexts tested so far?
    • Has the team observed a credible leading signal connected to the desired behavior?
    • Can the outcome, baseline, primary measure, and guardrails be instrumented?
    • Are delivery, operational, and business risks understood well enough to make a commitment?
    • Is the expected value sufficient to justify production investment and opportunity cost?

    Do not wait for certainty; product decisions never get it. But do not confuse executive sponsorship, customer enthusiasm, a polished prototype, or engineering progress with evidence that the bet will produce the intended result. If the gate is not met, fund the next learning step rather than pretending the full solution is ready to scale.

    The transition does not have to happen all at once. A team can productionize a narrow use case while continuing to test adjacent opportunities. What matters is that learning work and scaling work are identified honestly, funded deliberately, and judged by different standards.

    Run discovery and delivery as one product system

    An outcome will not survive if strategy, discovery, delivery, and stakeholder management operate as separate handoffs. It needs an operating model in which the same people retain context from the problem through the impact review.

    Organize the work around an empowered product trio: product management, design, and engineering jointly own the outcome and the evidence behind the solution. That does not erase specialist responsibilities. It removes the pattern in which product writes requirements, design decorates them, and engineering discovers the hard constraints after the decision has already been presented as final.

    Discovery and delivery should run as coordinated tracks. While validated work moves through implementation, release, and adoption, the trio continues testing the assumptions behind upcoming bets and watching what released behavior teaches them. This preserves learning without making committed delivery wait for every future question to be answered.

    Maintain a compact bet record for every meaningful investment. It should show:

    • the customer problem and strategic outcome;
    • the current solution hypothesis;
    • the evidence gathered so far;
    • the assumptions that remain unresolved;
    • the current mode: learning or earning;
    • the next decision and the evidence required to make it;
    • the primary metric and guardrails;
    • the delivery constraints and dependencies that materially affect the bet.

    This record should travel into roadmap and sprint decisions. A roadmap then becomes a sequence of outcome bets, ordered by expected leverage, evidence, dependencies, and strategic fit. Sprint work remains concrete, but each significant task can be traced to the hypothesis it supports. That traceability exposes orphan features: work with no defined problem, no testable belief, and no measurable result.

    Turn stakeholder reviews into decision reviews

    Stakeholders usually ask for feature certainty because feature status is what the operating system gives them. Change the review format and the conversation changes with it.

    For each bet, report the outcome, the evidence gained since the previous review, the decision that evidence supports, the largest remaining risk, and the delivery forecast when the work is in earn mode. Then ask:

    • Has the customer problem or strategic priority changed?
    • Did the latest evidence increase or reduce confidence?
    • Is the team still testing the highest-risk assumption?
    • Has new scope appeared without a corresponding outcome or risk reduction?
    • Does the next investment buy learning, value capture, or neither?

    This is also how you control scope without turning every discussion into a negotiation. New work must improve the expected outcome, address a documented risk, or satisfy an explicit constraint. If it does none of those things, it belongs outside the current bet.

    A stakeholder can still impose a solution for regulatory, contractual, architectural, or strategic reasons. Record that constraint plainly. Then preserve the team’s responsibility for validating the problem, minimizing unnecessary scope, instrumenting the result, and reporting whether the mandated solution actually changed behavior.

    Instrument the decision loop, not just the launch dashboard

    A release is evidence that the team produced something. It is not evidence that customers encountered it, understood it, used it successfully, or changed their behavior because of it.

    Build a metric chain before production work becomes expensive:

    • Business result: the strategic effect the company ultimately cares about.
    • Customer behavior: the action expected to contribute to that result.
    • Product signal: the observable event or state that indicates the behavior occurred successfully.
    • Capability: the shipped solution intended to influence the signal.

    If retention is the business result, successful setup or repeated use might be an earlier behavioral signal. That relationship is still a hypothesis in your product. Do not assume a leading indicator matters merely because it is easy to move. Validate its connection to the downstream result over an appropriate measurement window.

    Before release, define who is eligible, how exposure will be recorded, what successful use means, which segment will be analyzed, what baseline or comparison will be used, and which guardrails could stop expansion. Verify the instrumentation itself. Otherwise, a missing event can be mistaken for missing adoption, while duplicate or ambiguous events can create fictional progress.

    Where traffic and exposure permit, an A/B test can estimate whether the capability caused an incremental change. Where randomization is not practical, a staged release, cohort comparison, or carefully monitored pilot can still provide directional evidence, provided its limitations are stated. In either case, observability should cover both customer behavior and the operational health of the released system.

    Use the post-launch review to make a decision, not to celebrate a dashboard. Work through the causal chain in order:

    • No meaningful exposure: investigate eligibility, distribution, rollout, onboarding, or instrumentation before judging the solution.
    • Exposure without use: examine relevance, comprehension, trust, discoverability, and usability.
    • Use without the intended behavior: challenge the solution mechanism and the definition of successful use.
    • Behavior change without the business result: reassess the assumed link, segmentation, and measurement window.
    • Primary improvement with guardrail damage: pause expansion and address the tradeoff rather than averaging the harm away.

    These are diagnostic starting points, not automatic conclusions. Their value is that they direct the next investigation. The review must end with an explicit choice: expand, continue observing, revise the solution, test a different opportunity, or stop investing.

    AI changes the cost of learning, not the evidence standard

    Generative AI can make prototypes, interface variants, and qualitative synthesis much faster. That is useful because it lowers the cost of testing assumptions. It also makes it easier to produce a convincing solution before anyone has established that the problem deserves investment.

    Treat an AI-generated prototype as a learning instrument, not customer validation. A plausible interface or polished response does not show that customers will trust it, incorporate it into their workflow, or achieve a better result. Those questions still require evidence from the intended users and context.

    For an AI-powered feature, separate system quality from product outcome. Prompt changes, model changes, response quality, and task-level evaluations can explain how the capability behaves. The customer outcome tells you whether that capability improved the workflow that matters. A better model-level result can coexist with a flat customer outcome if the feature addresses the wrong problem, appears at the wrong moment, or creates too much friction around the generated output.

    Keep risk guardrails beside the primary outcome from the beginning. The relevant guardrails depend on the use case, but they should capture the ways an apparently successful AI feature could create unacceptable customer, operational, ethical, or business consequences. Faster experimentation is valuable only when the decision loop can detect both value and harm.

    Outcome-driven product development FAQ

    What if an executive has already specified the feature?

    Treat the feature as a proposed or mandated solution, then ask what result makes it worth building. Document any non-negotiable constraint and write the outcome contract around it. Offer alternatives when discovery shows a cheaper or more effective route, but do not hide the original decision. The team should still instrument the feature and report whether it delivered the intended behavior.

    How long should discovery take?

    Long enough to cross the evidence gate for the decision being made, and no longer. Timebox each experiment, not the truth of the entire opportunity. A reversible, limited bet may need modest evidence; an expensive or risky commitment warrants more. If discovery continues without changing confidence or informing a decision, narrow the question or stop the activity.

    Can an outcome-driven team still commit to a delivery date?

    Yes, once the work is in build-to-earn mode and the important delivery unknowns are bounded. Keep the delivery forecast separate from outcome confidence. A team may be highly confident about when a capability will ship while remaining uncertain about the behavior it will cause. Reporting both prevents schedule confidence from being mistaken for product confidence.

    What should happen when the outcome does not move?

    Start with exposure, then use, then behavior, then the business connection. Identify where the chain broke before adding scope. If the evidence weakens the underlying hypothesis, revise or stop the bet. Outcome accountability means changing course when the mechanism fails, not punishing a team for refusing to manufacture a favorable story.

    At your next roadmap review, take the highest-investment item and replace its feature description with an outcome contract. If the problem evidence, behavioral measure, guardrails, or decision rule is blank, fund the missing learning before you fund scale. That single change will reveal whether the roadmap is managing customer and business progress or merely scheduling production.

    References

  • Churn Prediction: A Practical Build-Versus-Buy Framework

    Churn Prediction: A Practical Build-Versus-Buy Framework

    You need a churn score soon. Customer success wants a prioritized account list, engineering wants requirements, and finance wants to know whether it is funding a vendor contract or a permanent internal capability. A polished model can still leave all three teams waiting if nobody has decided what happens after an account is flagged.

    Start with the retention decision, not the algorithm. Once you know who will act, what they will do, and how you will measure the result, the build-versus-buy choice becomes much clearer.

    Decide which capability you actually need to own

    Churn prediction is often discussed as if it were a single model. In practice, it is an operating loop with several layers:

    1. Define the outcome. Specify which customers can churn, what event counts as churn, and the prediction window that gives your team enough time to intervene.
    2. Assemble the signals. Connect product usage, account attributes, engagement, support, billing, and other permitted data to a consistent customer identity.
    3. Estimate risk. Produce a score, category, or ranking that separates accounts requiring attention from the rest of the portfolio.
    4. Activate the prediction. Route the result into the CRM, customer-success workflow, lifecycle message, or in-product experience where somebody can respond.
    5. Learn from the intervention. Measure whether the action changed retention, adoption, engagement, or Net Recurring Revenue rather than assuming that a plausible score created value.

    You do not necessarily need to own every layer. A vendor might provide behavioral analytics, scoring, in-app guides, and CRM integration while you retain ownership of the churn definition, intervention policy, and experiment design. Conversely, you might build a specialized risk model but continue using commercial tools to collect events and deliver treatments.

    My default is to separate model ownership from outcome ownership. Your company must own the definition of success, the permitted uses of the score, and the learning loop. It only needs to own the model code when that ownership creates a strategic advantage.

    Before evaluating an architecture or vendor, complete this sentence:

    When a customer in [defined population] crosses [risk condition], [named owner] will take [specific action] through [named system], and I will judge the intervention using [business outcome].

    If you cannot complete it, pause the model decision. You have an intervention-design problem. Buying software will automate the ambiguity, while building will make the ambiguity more expensive.

    Run six decision gates before choosing a path

    The right answer depends on more than whether your team can train a model. Use these gates to expose the constraint that should control the decision.

    Decision gateEvidence to inspectWhat pushes you toward a path
    Time to valueDecision deadline, current churn visibility, and readiness of the first interventionUrgent activation favors buying; a longer strategic horizon makes building more viable
    Data readinessOutcome labels, identity resolution, event consistency, signal freshness, and usable historyImmature data favors a packaged baseline while you repair foundations; reliable proprietary data strengthens the case to build
    Strategic differentiationSignals or decisions competitors and general-purpose vendors cannot reproduceA must-have retention capability favors buying; a defensible product advantage favors building
    Operating talentNamed owners for data pipelines, production scoring, monitoring, governance, and intervention designMissing ownership favors buying; durable cross-functional capacity makes building credible
    Activation fitCRM, customer-success, messaging, analytics, and in-product delivery requirementsStandard integrations favor buying; specialized actions or product-embedded scoring may require a build or hybrid approach
    Risk and explainabilityPrivacy, access, retention, audit, explanation, and regulatory requirementsStandard controls may fit a vendor platform; domain-specific constraints can justify owning selected layers

    Time to value: is speed useful, or merely urgent?

    A short deadline only matters when an intervention is ready. If customer success already knows what it will do with a high-risk account, buying can put usable signals into existing workflows sooner. If the team has not agreed on an action, a fast score simply creates a faster queue of unanswered alerts.

    Ask for the date on which a real user must receive the first actionable score. Then work backward through integration, workflow design, governance review, enablement, and experiment setup. This prevents a vendor demonstration or model prototype from being mistaken for operational readiness.

    Data readiness: can your records support the decision?

    A custom model cannot rescue an unstable churn definition or inconsistent customer identity. Inspect whether product events can be joined to the correct account, whether the churn outcome is recorded consistently, whether important segments have comparable coverage, and whether signals arrive early enough to support action.

    Do not interpret weak data as an automatic reason to buy. A vendor cannot manufacture missing labels or repair every instrumentation gap. It can, however, give you a practical baseline using the signals already available while your team improves the data foundation.

    Differentiation: would model ownership change your product advantage?

    Build when proprietary context can materially improve the decision. That may include distinctive behavioral signals, domain-specific anomaly detection, specialized explanations, or a risk score embedded directly into your product. These are stronger reasons than a general preference to own technology.

    If competitors could buy an equivalent capability and churn prediction mainly helps customer success prioritize outreach, ownership is unlikely to be the differentiator. Put product and engineering attention into the intervention, customer experience, and learning loop instead.

    Talent: can you operate the system after launch?

    Having someone who can train a model is not the same as having an operating team. A production capability also needs data engineering, scoring infrastructure, monitoring for drift, feature maintenance, incident ownership, governance, and a product owner who connects model changes to retention outcomes.

    Put a name beside every continuing responsibility. An empty cell is not a future hiring plan; it is part of the build cost. If the same scarce people are also responsible for your core product, include the opportunity cost of redirecting them.

    Activation: can the score reach the moment of action?

    A prediction trapped in a dashboard has little retention value. Confirm that a score can create the right CRM task, customer-success play, lifecycle message, product tour, contextual tooltip, or in-app nudge. The recipient also needs enough explanation to choose an appropriate response.

    Evaluate activation with a concrete scenario, not a feature checklist. Give a candidate vendor or internal team one representative account and ask it to show the full path from new behavior to updated risk, reason, assigned owner, intervention, and measured outcome. Any manual handoff in that path belongs in the decision record.

    Governance: what must remain controlled and explainable?

    Document which data may be used, who may see the result, how long inputs and scores are retained, what explanations users need, and how a customer could be affected by a mistaken classification. Privacy-by-design, data governance, regulatory compliance, and AI risk management apply whether the prediction is purchased or built.

    Building gives you more design control, but it also transfers the burden of evidence, monitoring, and remediation to your organization. Buying transfers implementation work, not accountability. Require the same governance review for both paths.

    The pattern is straightforward: buy when speed, standard coverage, and workflow activation dominate; build when proprietary signals, specialized explanations, or product differentiation dominate; blend when you need results now but have a credible reason to own selected layers later. A useful default is to buy a working baseline and build only where your context can create an outsized advantage.

    Compare the full economics, not a license and a prototype

    The most common cost comparison is structurally wrong: an annual software license is placed beside the effort required to train an initial model. One is closer to an operating capability; the other is an experiment. Compare both options across the same time horizon and include four cost classes: starting, running, changing, and exiting.

    What belongs in the buy case

    • License, usage, seat, and service costs that apply to the intended customer population.
    • Implementation work for event collection, identity mapping, historical data, and system integrations.
    • Security, privacy, legal, regulatory, and procurement review.
    • Internal administration, score interpretation, workflow ownership, and user enablement.
    • Configuration or services needed for segments, reason codes, guides, alerts, and experiments.
    • Limits on data access, exports, custom features, scoring frequency, and downstream activation.
    • Migration effort if the vendor no longer fits, including preservation of historical scores and experiment records.

    What belongs in the build case

    • Instrumentation, data quality, identity resolution, label construction, and feature pipelines.
    • Exploration, training, evaluation, explanation design, and production validation.
    • Batch or real-time scoring, storage, APIs, access control, and reliability engineering.
    • CRM, messaging, customer-success, analytics, and in-product integrations.
    • Monitoring for drift, broken inputs, coverage gaps, and unexpected segment behavior.
    • Retraining, feature maintenance, documentation, incident response, and ongoing product ownership.
    • Privacy controls, audit evidence, risk review, retention rules, and regulatory compliance.
    • Replacement or migration work when the architecture, churn definition, or business workflow changes.

    Add cost of delay to both cases. Buying may carry a visible contract cost, but waiting for a custom capability can defer retention experiments and leave customer-success capacity poorly targeted. Building may require more internal investment, but a vendor that cannot express your signals or deliver the required intervention can delay learning in a different way.

    Keep benefit assumptions separate from cost estimates. The model’s theoretical accuracy is not a financial return. Estimate value only through an intervention that can plausibly affect customer behavior, then validate that assumption with an experiment.

    Your comparison should therefore show three views for each path:

    • Capability: which parts of the signal-to-action loop will actually work?
    • Economics: what will it cost to start, operate, change, and exit?
    • Evidence: what experiment will determine whether the capability improves retention or NRR?

    If one option looks cheaper only because a row is blank, resolve the missing responsibility before approving it.

    Use a hybrid path without creating two disconnected systems

    A hybrid strategy is more than running a vendor score and an internal score at the same time. Done well, it sequences the work: buy the common layers needed for speed and activation, learn which proprietary signals matter, and build only the components that earn their continuing cost.

    Phase one: establish a usable baseline

    Choose one defined customer population, one churn outcome, and one intervention. Configure the purchased capability to produce a risk signal and a usable reason, then route both into the workflow where the named owner can act.

    Record three different kinds of evidence:

    • Prediction evidence: coverage, signal freshness, ranking or precision, stability across relevant segments, and the usefulness of explanations.
    • Operational evidence: whether scores arrive in time, whether users understand them, and whether a flagged account reliably receives the intended treatment.
    • Business evidence: whether the intervention changes retention, adoption, engagement, or NRR.

    Do not use prediction quality to claim business impact. It is possible to identify high-risk accounts accurately and still deliver an ineffective intervention. It is also possible for a broad model to create value because it reaches the right team at the right moment. These are different questions and need different measures.

    Phase two: test where proprietary context adds value

    Use retention analysis to identify behaviors that appear meaningfully connected to continued use or churn. Focus on information a general-purpose platform cannot represent well, such as domain-specific sequences, unusual account structures, specialized failure states, or product-specific anomalies.

    Introduce one material improvement at a time. Compare the resulting decisions with the baseline: which accounts move, whether the reason becomes more actionable, and whether the intervention performs better. A more complex score is not automatically a better product.

    Use A/B testing or another appropriate controlled rollout to evaluate the intervention. Set the minimum detectable effect before the test so the team agrees on the smallest change worth detecting and whether the experiment can support the decision. Where withholding an intervention is inappropriate, compare credible treatments or use a phased rollout rather than treating measurement as optional.

    Phase three: build only the layer that proved distinctive

    The result may not be a complete vendor replacement. You might own a proprietary feature pipeline, domain-specific anomaly detector, custom explanation layer, or specialized risk score while retaining commercial analytics and activation. That is often a cleaner boundary than recreating collection, dashboards, integrations, guides, and workflow delivery.

    Before moving a custom component into production, require evidence that:

    • The proprietary signal changes a meaningful decision rather than merely changing a score.
    • The resulting intervention has a credible path to measurable retention or NRR impact.
    • A named team owns data quality, production reliability, drift monitoring, governance, and retraining.
    • The migration preserves the activation loop instead of sending users to a separate dashboard.
    • The added value justifies both the continuing cost and the engineering capacity displaced by the work.

    Create a canonical risk contract before two systems coexist. Define the eligible population, outcome, prediction window, score meaning, reason codes, refresh expectations, owner, permitted actions, and measurement plan. Without that contract, teams will compare incompatible scores and select whichever one confirms their prior belief.

    Run the custom component beside the baseline before switching interventions. Inspect coverage, stability, explanations, workflow behavior, and segment differences without changing several parts of the retention program at once. This makes the eventual migration a product decision supported by evidence, not an infrastructure milestone searching for a justification.

    Key takeaways

    • Buy when your immediate need is dependable coverage, rapid activation, and standard integrations for customer success or product-led growth.
    • Build when proprietary signals, domain-specific risk scoring, specialized explainability, or product differentiation can create material value and you can fund continuing operations.
    • Blend when you need a working baseline now and have a testable hypothesis about where your data or context can outperform a general-purpose capability.
    • Do not approve any path until every score has a named recipient, a defined action, a delivery system, and a business outcome.
    • Compare equivalent total costs, including data work, integrations, monitoring, governance, activation, opportunity cost, and migration.
    • Measure the model and the intervention separately. Prediction quality can prioritize attention; only an effective action can improve retention.

    Take a one-page decision memo into your next review. It should name the churn definition, first population, intervention, deadline, available signals, proprietary advantage, workflow, operating owners, governance constraints, total-cost boundary, and experiment. End the meeting with a selected path and an explicit condition for reconsidering it.

    Start with the smallest path that closes the loop from behavior to action to measured outcome. Earn the right to build more by proving that your own data changes the decision and that the decision changes retention.

    References

  • From Brain Dump to Done: How Todoist’s Ramble Captures Tasks in Real Time with AI

    From Brain Dump to Done: How Todoist’s Ramble Captures Tasks in Real Time with AI

    Turning a rambling stream of consciousness into a clean task list while someone is still talking has been a longtime product dream of mine. With Ramble, Todoist brought that dream to life by using live audio AI to capture tasks in real time—no transcription step required. The result is a voice-to-task flow that feels natural, fast, and surprisingly disciplined.

    As I listened to the Doist team—Ernesto Garcia (Front-end Product Engineer), Thomas Jost (Backend Software Engineer), and Hugo Fauquenoi (Product Manager)—walk through their approach, I heard a blueprint for building pragmatic GenAI features. What began as a two-to-three month AI exploration became one of their most technically deliberate releases: a “Gemini-powered pipeline that makes tool calls while the user is still speaking, surfacing tasks on screen in real time without any text output from the model.”

    The breakthrough started with user research. People weren’t merely dictating tasks; they were doing a “brain dump” first—often into pen and paper or even ChatGPT voice—and only then committing items to Todoist. Meeting users where they already are reframed the problem: don’t force structure upfront; capture fluid thought and translate it into actionable tasks instantly.

    That insight led to a bold architectural choice: skip transcription entirely and process raw audio directly with a Gemini live audio model. By removing the brittle middleman of text, the team reduced latency and kept the model focused on one job—turning intent into structured actions. It’s a crisp example of AI workflows designed for reliability over novelty.

    The real magic is in the real-time “tool calls.” As the user speaks, the model triggers add task, edit task, and delete task operations immediately. For high-friction contexts like driving, they paired visual task cards with subtle sound effects as confirmation cues. It’s thoughtful conversation design that respects attention and safety without sacrificing speed.

    Teaching the model to capture tasks literally—without over-interpreting or trying to complete the work—required careful prompt engineering for voice and temperature tuning. Drawing a bright line between “capture versus do” kept the experience trustworthy. In my own AI Strategy work, I’ve found that establishing explicit agentic guardrails early prevents unintended autonomy later.

    Dates were the sleeper challenge. The team had to inject the current date, normalize to days vs. months, and always output dates in English for the natural language parser—while preserving the user’s original language for everything else. If you’ve ever shipped date handling across locales, you’ll appreciate how many edge cases hide in “Taming Dates and Time.”

    Quality didn’t hinge on intuition alone. They built an LLM-judge eval system using real employee recordings from 100+ people across 35 countries in 20+ languages to catch prompt regressions. That’s eval-driven development done right: representative data, repeatable scoring, and tight feedback loops as models and prompts evolve.

    For project and label matching, they chose direct context injection over RAG. Instead of building a retrieval pipeline, they injected the full project/label list into the system prompt. With smart context window management and a sharply constrained task schema, this was both simpler and more accurate. Sometimes the fastest path to product-market fit is removing moving parts, not adding them.

    One product principle stood out: easy correction beats perfect first-time accuracy. Natural language interfaces earn trust when users can fix misfires in a tap or two. That bias toward quick recovery over false precision is how you ship AI that feels useful from day one.

    Looking ahead, the roadmap is compelling: multimodal task capture from images and text blobs, Apple Watch support, and automation integrations. As voice AI agent patterns mature, this “tool-only architecture” sets a solid foundation for going from capture to coordinated execution—without losing the simplicity that makes Ramble shine.

    If you want to hear the full conversation, you can listen on Spotify or Apple Podcasts. It’s a masterclass in building focused GenAI features that trade cleverness for clarity—and still delight.

    Resources & Links: Todoist • Doist • Google Vertex AI (Gemini)

    Inspired by this post on Product Talk.

    Book a consult png image
  • How to Build a Trusted AI Product Platform That Scales

    How to Build a Trusted AI Product Platform That Scales

    Your teams have AI pilots that work in a demo. Then the questions start. Security wants to know what data the system can reach. Product wants to know whether the answers are dependable. Support wants a fallback when the model fails. Executives want evidence that the investment is changing a customer or business outcome.

    You do not need another impressive model response. You need a product platform that makes AI behavior understandable, controllable, and repeatable across use cases. That requires a trust architecture, a path from prototype to production, and metrics that expose failure instead of averaging it away.

    Trust fails where an AI output crosses a decision boundary

    Most teams discuss AI trust as if it were a property of the model. It is better understood as a property of the whole product system. A capable model can still create an untrustworthy experience if it uses the wrong context, hides a consequential assumption, calls an unauthorized tool, or leaves the user unable to correct an action.

    The important moment is the handoff from generation to decision. Before that handoff, the output is a possibility. After it, someone may use it to answer a customer, change a record, prioritize work, or trigger another system. The controls you need depend on what crosses that boundary.

    A practical way to classify AI use cases is by the authority you give the system:

    • Inform: The system summarizes, explains, retrieves, or drafts. A person still interprets the result.
    • Recommend: The system ranks options or proposes a next action. Its framing can materially influence a decision.
    • Act: The system invokes tools, changes state, communicates externally, or starts a workflow.
    Use modePrimary trust failureRequired product controlEvidence needed before release
    InformAn incorrect, incomplete, or untraceable answerVisible scope, supporting evidence, uncertainty, and an easy correction pathAn evaluator can reproduce the evidence path and identify known limitations
    RecommendA hidden assumption, weak comparison, or recommendation that ignores the user’s constraintsExplicit assumptions, alternatives, decision criteria, and user-editable constraintsRepresentative cases show whether the recommendation applies the intended rubric
    ActAn unauthorized, excessive, or difficult-to-reverse changeLeast-privilege access, previews, confirmation, audit records, and reversal where the underlying system supports itAuthorized reviewers validate simulated actions, denied actions, failure recovery, and a limited production path

    This classification prevents a common planning error: giving every AI feature the same review process. A summarizer and an autonomous account-management agent should not pass through identical gates. The second system needs stronger identity, permission, confirmation, and recovery controls because its mistakes can propagate beyond the conversation.

    For each proposed use case, ask five questions before discussing a model:

    <!– wp:list {
  • Never Lose Your AI Superpowers: How I Sync Context and Skills Across Every Device

    Never Lose Your AI Superpowers: How I Sync Context and Skills Across Every Device

    I spend a meaningful portion of my week helping teams operationalize AI workflows, and one theme comes up over and over: how to share context files and skills seamlessly across devices and with colleagues. Hosting Claude Code office hours has only reinforced it—sharing context and skills is the single biggest blocker to reliable, repeatable outcomes.

    I hear from leaders driving AI adoption who have built robust, high-signal context systems and carefully crafted skills. Their challenge isn’t creating value—it’s distributing it. They need a way to make the same trusted workflows available to teammates and to keep everything in sync across laptops, desktops, and phones.

    I hit the same wall myself. I work across multiple devices (a Mac Mini for day-to-day, a MacBook Air on the road, and an iPhone) and I collaborate with a full-time admin. I wanted my context and skills to be consistent everywhere, for both of us. In this piece, I’ll share my setup—what I store where, how I share it across devices and with my team, the trade-offs of each option, and how I keep everything current. We’ll cover four different syncing services: git/GitHub, Obsidian Sync, Dropbox and iCloud.

    If you’re new to this series, this is the eighth installment. Earlier pieces provide foundational context: Claude Code: What It Is, How It's Different, and Why Non-Technical People Should Use It; Stop Repeating Yourself: Give Claude Code a Memory; How to Use Claude Code Safely: A Non-Technical Guide to Managing Risk; How to Choose Which Tasks to Automate with AI (+50 Real Examples); How to Build AI Workflows with Claude Code (Even If You're Not Technical); How to Use Claude Code: A Guide to Slash Commands, Agents, Skills, and Plug-ins; and Context Rot: Why AI Gets Worse the Longer You Chat (And How to Fix It).

    The day it really hit me was right before my interview with Claire Vo on How I AI. I was staying in an AirBnB with only my laptop, and I planned to demo my /today command along with my context file structure. Minutes before the session, I realized the latest version of my /today command wasn’t on that machine. I was able to remote into my Mac Mini and grab it—crisis averted—but it was a wake-up call. I needed a more reliable, shareable approach for syncing context and skills across devices and with my admin.

    I started by testing the tools I already used—Dropbox, iCloud, and GitHub—to see what might fit. Each got me partway there, but each also introduced friction that mattered in daily use.

    First, absolute file paths don’t travel well. I began with Dropbox but quickly ran into cross-linking headaches. Good context systems rely on rich interlinking—index files point to other context files, and those context files link to each other. When Claude creates a link from one context file to another, it tends to use the full file path: /Users/ttorres/Library/CloudStorage/Dropbox. That worked on my Mac Mini and MacBook (same user name), but not on my phone—and not for my admin. I tried to force relative links (~/Dropbox), but couldn’t get Claude to do it consistently, which led to broken links. This isn’t unique to Dropbox; Claude prefers full paths because they’re reliable on a single machine, but they’re brittle across devices and useless when sharing with colleagues. Claude is trained to use relative file paths when working within a git repository, but I struggled to get it to work reliably in Dropbox.

    Second, skills live in a user directory by default. By default, skills live in ~/.claude/skills. Most sync services aren’t designed to share your ~/ folder. iCloud is the exception, but then you’re limited to Apple devices—no Windows or Android. There is a workaround: set up a claude folder in Dropbox and create a symlink from ~/.claude to your synced claude folder, so all skills, commands, and settings live in Dropbox. Then, on each device (yours or a colleague’s), you set up a symlink to that folder so Claude can find the files. This works, but I was running into another limitation that made Dropbox a poor fit.

    Third, Obsidian on iOS doesn’t sync cleanly with Dropbox. I rely on Obsidian’s file browser alongside my notes to navigate context quickly. Storing vaults in Dropbox gave me parity across my Mac Mini and MacBook Air, but I couldn’t get the iOS Obsidian app to reliably load my Dropbox vaults. That friction was a dealbreaker for on-the-go work.

    At that point, I explored git/GitHub. GitHub is cloud storage for git repositories. A git repository is a folder of shared files used so engineers can collaborate on the same code base. Each person clones a local copy, works locally, then pushes changes back to the hosted repo on GitHub; others pull to update. Git’s merge and conflict tooling is excellent. Git is the powerhouse of file syncing and version control. It easily handles syncing context and skills, Claude behaves better with relative links in a git repo, and I can open the repo in my IDE with a clean file browser. For me, that checked all the boxes—until I factored in my admin. Git has a learning curve, requires manual pull/push hygiene, and often assumes an IDE workflow. That overhead was too heavy for a non-technical collaborator.

    The turning point was Obsidian Sync. A colleague suggested it, and it ended up being the sweet spot. Obsidian is a markdown reader; files are stored locally in a normal folder you can open in Finder or File Explorer. There’s no proprietary format—you can read files with any text editor, and Claude can access them via bash commands. Obsidian Sync is simpler than git: open a note and it syncs in the background. I can access the same vaults across my Mac Mini, MacBook Air, and iPhone, and I can share a vault with my admin so we can both create and access notes.

    Because we’re in different time zones and rarely edit the same note simultaneously, limited conflict handling hasn’t been an issue. Obsidian’s internal link notation also means one note can link to another and those links just work across devices. Claude can follow these links, so the brittle file path problem disappears.

    Here’s where I landed. After a lot of trial and error, I have a setup that works across my devices and for my admin, who uses both a Windows desktop and a Mac laptop. I keep my core context in Obsidian vaults synced with Obsidian Sync, which preserves portability, link integrity, and ease of use. For skills, I avoid scattering files in machine-specific locations and instead centralize what Claude needs to reference in shared, human-readable folders. If you require advanced version control with branching and reviews, git/GitHub is excellent. If your priority is low-friction, cross-device access for non-technical teammates, Obsidian Sync is a practical, reliable choice. And if you must use Dropbox or iCloud, consider symlinks and be vigilant about relative paths—just know that absolute paths won’t travel well.

    Inspired by this post on Product Talk.

    Book a consult png image