Category: AI Strategy

How to Design an AI Customer Agent for Sales Qualification

A prospect reaches your pricing page with a real buying question. The form promises a reply, but the reply arrives after the prospect has moved on, chosen a competitor, or forgotten why the question mattered.

An AI customer agent can remove that delay, but speed is only the entry requirement. The harder product problem is deciding whom to qualify, what evidence to collect, which next step to offer, and how to preserve enough context that a salesperson can continue the conversation without starting over.

Start with routing decisions, not chatbot dialogue

The purpose of a sales qualification agent is not to produce a pleasant conversation or a high lead score. Its job is to make a defensible next-step decision while the buyer’s intent is still active.

That distinction matters because conversational fluency can hide weak commercial logic. An agent may sound helpful while booking low-fit meetings, sending strong prospects down a generic self-serve path, or marking inferred information as confirmed. Those failures make the pipeline look larger before they make it less trustworthy.

Define the available outcomes before you write prompts. Most inbound motions need some version of these routes:

Route	Minimum evidence	Agent action
Sales-ready	The problem fits the product, the buyer needs sales involvement, and the timing or buying process satisfies your acceptance rule.	Offer an appropriate meeting, create or update the CRM record, and send the qualification evidence.
Self-serve	The use case is viable, but the buyer can select a plan, begin a trial, or complete signup without a salesperson.	Recommend the relevant path, help the buyer take the next action, and preserve the conversation for later use.
Promising but not ready	There is plausible fit, but intent, timing, authority, or requirements remain unresolved.	Provide the useful resource or follow-up path defined by your policy without manufacturing urgency.
Not a fit	A hard requirement conflicts with the product’s supported scope or the request belongs elsewhere.	State the limitation clearly and redirect the person without placing an unqualified meeting on a seller’s calendar.
Human exception	The request involves an existing account, a sensitive claim, a complex commercial exception, or information the agent cannot verify.	Escalate with the context already collected and identify the unresolved question.

Keep fit and readiness as separate dimensions. A large, recognizable account can be a strong fit and still be months away from a decision. A highly motivated buyer can be ready to act and still need a capability you do not provide. Combining both dimensions into one opaque score conceals the reason behind the route and makes mistakes difficult to diagnose.

Separate hard constraints from soft signals as well. A required capability that does not exist may be a hard stop. A vague timeline is usually uncertainty to resolve, not automatic disqualification. Firmographic enrichment can help prioritize an account, but it cannot confirm what a buyer has not actually said.

For every consequential route, require three outputs: a reason code, the evidence behind it, and the next action. If the agent cannot produce all three, it has not completed qualification. It has merely assigned a label.

Turn your qualification policy into an executable conversation

A natural-language playbook makes sales policy easier to express, and current customer agents can be instructed to follow qualification rules, address approved objections, and move buyers toward defined outcomes. Natural language does not remove ambiguity, however. If two experienced salespeople would interpret a rule differently, the agent will not reliably repair the policy for you.

Ask only what changes the route

Traditional lead forms collect fields because the CRM has columns. A conversation should be more selective. Every question should either help the buyer, determine fit, resolve readiness, or select the correct action.

Open from observable context, such as the plan, feature, or integration the person is exploring.
Answer the buyer’s current question before turning the exchange into discovery.
Ask the smallest useful question that could change the route.
Branch from the answer instead of walking every prospect through the same questionnaire.
Confirm the important facts before treating them as qualification evidence.
Explain the recommended next step and let the buyer act while still in the conversation.

If someone asks whether a specific integration is available, answer that question first. Then ask how the integration fits the intended workflow if the answer would affect plan selection or sales involvement. Leading with budget, company size, or phone number when none of those details helps answer the immediate question makes the agent feel like a form with typing animation.

A useful qualification schema usually covers the following areas, but the agent should collect only the fields relevant to the current branch:

The problem or use case the buyer is trying to address.
The capabilities, integrations, or constraints that determine product fit.
The consequence of leaving the problem unsolved, when that affects urgency or route.
The buyer’s role in evaluation and the remaining buying process.
The intended timing and any event driving it.
Commercial expectations or budget when those facts genuinely affect the path.
Identity and account context, with a clear distinction between what was stated, enriched, or inferred.

Do not ask about budget merely because a familiar qualification framework includes it. If pricing is public and the buyer can start without sales assistance, the better action may be to explain plan fit and help the buyer proceed. If commercial terms require human involvement, budget or purchasing process may become relevant later in the branch.

Preserve provenance instead of filling blanks with guesses

Store each material qualification field with its provenance. Buyer-stated, externally enriched, model-inferred, and unknown are different states. Treating them as interchangeable creates false confidence in the CRM.

An enriched company size may help prioritize a conversation, but it is not buyer-confirmed budget. A page visit may indicate interest in a feature, but it is not a confirmed requirement. An enthusiastic phrase may indicate intent, but it is not a purchasing timeline. Keep those distinctions visible to the routing logic and the salesperson receiving the lead.

I would not allow the agent to write a final qualification status unless every required field is either supported by evidence or explicitly marked unknown. Unknown information is operationally useful: it tells the seller what still needs to be resolved. Fabricated completeness does the opposite.

Evaluate decisions, not just responses

Build a scenario set from the situations that cause real routing disagreements. Include a high-fit account with low intent, a small account with urgent intent, an existing customer asking a sales-shaped question, a buyer requiring an unsupported capability, a returning visitor, a pricing objection, conflicting information, and a request the agent should escalate.

For each scenario, define the expected answer, acceptable route, required CRM writes, escalation condition, and forbidden behavior. Then test the complete journey. A correct answer followed by the wrong calendar, duplicated CRM record, or context-free handoff is still a failed qualification experience.

Build trust into answers, memory, and handoffs

A sales agent cannot qualify reliably if its product knowledge is unreliable. It needs approved information about pricing, packages, capabilities, integrations, plan eligibility, trial paths, and common objections. Current implementations can draw on an existing product knowledge base while combining that knowledge with playbooks, enrichment, and memory, which reduces duplicated setup but does not eliminate ownership.

Assign a business owner to every consequential knowledge area. When pricing, packaging, an integration, or an eligibility rule changes, update the canonical material and rerun the scenarios affected by that change. A polished answer based on stale commercial information is more dangerous than an explicit handoff because the buyer has little reason to question it.

Define the agent’s boundaries in the same system. It should know when it may explain published pricing, when it must avoid inventing discounts, when roadmap questions need human confirmation, and when a security, legal, or contractual claim requires escalation. The safe fallback is not a vague non-answer. It is a clear statement of what remains unverified and a context-rich route to someone authorized to answer.

Use memory as buyer state, not as an unlimited transcript

Memory is valuable when a returning visitor does not have to repeat the use case, plan under consideration, or unresolved objection. A customer agent can recognize returning context and continue the buying journey, but old information should not silently override new facts.

Store a compact buyer state: confirmed needs, important constraints, questions already answered, current route, unresolved items, and the last agreed next step. Keep timestamps and provenance so the system can notice when a current statement conflicts with an earlier one. Ask for confirmation when a material fact may have changed.

Enrichment deserves the same discipline. Use it to improve context and routing, not to pretend the agent knows the buyer personally. Record where enriched data came from, apply your privacy and retention controls, and give buyer-stated information precedence when the two conflict.

Make the handoff a product deliverable

Booking a meeting is not the end of qualification. It is the beginning of a human handoff. Passing only a name, email address, and transcript forces the salesperson to reconstruct the conversation under time pressure.

The handoff package should contain:

Identity and account information, including the provenance of enriched fields.
The buyer’s problem and intended outcome in the buyer’s own terms.
Confirmed requirements, constraints, timing, and buying-process details.
Questions answered and the approved information used to answer them.
Objections, unresolved questions, and any claim requiring human confirmation.
The selected route, its reason code, and the evidence that supported it.
The next action already promised to the buyer.
A link to the full conversation for detail or audit.

Modern customer agents can book through scheduling tools, sync structured context into the CRM, and pass both conversation history and an AI-generated summary. The summary should reduce reading effort, while the structured fields should support routing, reporting, and workflow automation. Neither should replace access to the original conversation.

Test the first minute of the seller’s follow-up. Can the seller see why the lead was routed, what the buyer already knows, and what must happen next? If the opening question repeats discovery the agent just completed, the handoff has broken the continuity you used AI to create.

Measure whether the agent creates incremental pipeline

Meeting count is an attractive but incomplete success metric. Bookings can rise because the agent reaches previously unattended demand, because it diverts buyers who would have booked with a human anyway, or because it lowers the qualification bar. Only the first outcome is unambiguously additive.

Instrument the full decision funnel rather than the chat interface alone:

Reach: eligible visitors, conversations initiated, response latency, and coverage by channel or time window.
Conversation quality: questions answered, unresolved-answer rate, corrections, escalations, and abandonment before a useful action.
Qualification quality: completion of required evidence, unknown-field rate, route distribution, seller acceptance, and rejection reasons.
Handoff quality: meeting attendance, repeated discovery, missing CRM context, reassignment, and follow-up delay.
Commercial outcome: accepted qualified opportunities, pipeline created, trial or self-serve conversion, win rate, contract value, sales-cycle progression, and cost per accepted opportunity.

Audit both error directions. False positives waste seller time and inflate forecasts. False negatives are quieter: a strong buyer is sent away, mislabeled as self-serve, or blocked by an unanswered question. Review unsuccessful routes as well as booked meetings, because the most expensive qualification error may never appear in the sales team’s queue.

Early deployment data shows why coverage and incrementality need separate analysis. In a vendor-reported overnight rollout, Fellow booked 18 January meetings that its human team would not otherwise have reached, with around 48% converting, while the human booking rate held. That is evidence of an additive channel in that deployment, not a universal conversion benchmark.

Volume alone tells a different and incomplete story. During a vendor-reported three-month deployment, Attio’s agent handled more than 1,600 visitor conversations, qualified more than 50 leads for sales, and routed more than 30 applicants into a startup program. Those figures show multiple useful outcomes from the same inbound surface, but they do not establish causal lift for another company’s funnel.

Establish your own baseline separately for hours and pages with human coverage and those without it. If feasible, use a randomized holdout among otherwise eligible sessions. If randomization would create an unacceptable buyer experience, compare matched cohorts by page, channel, segment, visitor status, and time window. Do not compare an overnight agent cohort with daytime human coverage and call the difference an AI effect.

A controlled rollout can begin on a high-intent surface or during a coverage gap. First run routing in shadow mode and compare the proposed decisions with qualified human judgment. Then enable one consequential action at a time, such as self-serve guidance before autonomous meeting booking. Keep a human review path for exceptions and expand only when answer quality, routing precision, CRM completeness, and buyer outcomes remain acceptable together.

Your north-star measure should reflect accepted commercial value, such as incremental qualified opportunities or incremental pipeline per eligible visitor. Pair it with guardrails for incorrect claims, seller rejection, buyer complaints, CRM errors, and missed high-fit leads. An agent that creates more records while reducing trust has not improved the sales system.

Key takeaways

Treat the AI customer agent as a decision system, not a conversational layer placed in front of a lead form.
Define sales-ready, self-serve, not-ready, not-fit, and human-exception routes before writing dialogue.
Keep fit separate from readiness, and preserve whether each field was buyer-stated, enriched, inferred, or unknown.
Ask only questions that help the buyer or change the route; answer the buyer’s immediate question before running discovery.
Make structured qualification evidence, unresolved issues, and the promised next action part of every human handoff.
Measure incremental accepted opportunities and pipeline, not chat volume, MQL count, or booked meetings in isolation.

Start with one high-intent entry point. Write its route contract, connect only approved knowledge, test the difficult scenarios, and compare shadow decisions with the people who currently qualify those leads. Give the agent authority gradually. The goal is not to automate the most conversation; it is to make the right buying path available at the moment the buyer is ready to take it.

References

Intercom – Fin for Sales: Instantly Engage, Qualify, and Close High-Intent Leads with an AI Customer Agent

April 22, 2026

How to Build Agentic AI for Product Analytics and Support

Your support bot can tell a customer where a setting lives, yet leave that customer to diagnose the problem, change the setting, and hope it worked. Your product team then receives a chat transcript without knowing whether the interaction improved activation, feature adoption, or retention.

If you are deciding how to connect AI, product analytics, and support, do not start with the model. Design the closed loop first: assemble trustworthy context, choose an allowed action, verify the resulting product state, and measure the user outcome. The model is one component inside that system.

Treat product analytics as the agent’s control plane

A useful standard is an assistant that understands the user’s context, can complete an allowed action, and measures whether the action helped. Remove any one of those capabilities and the experience degrades quickly. Context without action produces advice. Action without context creates risk. Action without measurement creates an impressive demo that cannot earn a durable place on the roadmap.

Product analytics supplies the behavioral context and outcome signals for this loop. It can show where the user is in a journey, which features have been adopted, which step failed, and whether the expected success event eventually occurred. It should not be treated as a warehouse-sized attachment to the prompt.

Define a support context contract

Create a small, governed context object for each supported workflow. Give the agent only the fields required to understand and resolve that workflow:

Actor and access: the authenticated user, account, role, entitlements, and permissions relevant to the requested action.
Journey state: the onboarding step, feature-adoption state, experiment assignment, or other stage that explains what the user is trying to complete.
Current product state: the relevant configuration from the operational system of record, including whether required prerequisites are satisfied.
Friction evidence: recent failed events, validation results, repeated attempts, and known errors connected to this workflow.
Desired outcome: the product state and behavioral event that will count as successful resolution.

Resolve analytics events and tool calls to the same stable user and account identifiers. Preserve timestamps and the origin of each field. For a live action decision, let the operational system of record determine current state; use analytics to explain the journey and measure the outcome. An event stream can be delayed or incomplete, so it should not overrule a current configuration read.

Behavior is also evidence, not intent. Repeated visits to a setup screen could indicate confusion, careful verification, or an advanced workflow. When those interpretations require different actions, the agent should ask one targeted question instead of turning a behavioral pattern into a confident diagnosis.

Apply data minimization at this boundary. Do not place secrets, payment information, unrelated conversation history, or an account’s entire event history into the model context. Filter fields before the model sees them, and enforce the filter in code rather than relying on a prompt instruction.

Give the analytics agent a metric contract

An internal analytics agent has a different job from a customer-facing support agent. It may translate a product question into metrics, cohorts, funnels, or retention views, but a fluent answer is not enough. Require every analysis to return:

the product question it interpreted;
the metric definition and success event it used;
the cohort, filters, and observation window;
the analysis or query reference needed to reproduce the result;
known data-quality limitations and unresolved ambiguity; and
a clear distinction between observed association and demonstrated causal lift.

This turns the analytics agent into a traceable decision aid. It also prevents two agents from using the same metric name while silently applying different event definitions, account filters, or windows.

Design one closed loop from signal to verified outcome

The core unit of agentic support is not the conversation. It is a resolution attempt with a beginning, an authorized action, and a verifiable end state. Use the following loop for every workflow:

Observe the trigger. Capture the user’s request or a product signal that indicates likely friction.
Assemble scoped context. Load only the identity, permission, journey, state, and error fields defined in the context contract.
Diagnose the next constraint. Determine which prerequisite, configuration, permission, or knowledge gap is blocking progress. If the evidence is ambiguous, ask rather than assume.
Select an approved playbook. Match the constraint to a versioned workflow with explicit eligibility rules, allowed tools, and prohibited actions.
Obtain the required authorization. Show the proposed change and its consequence whenever the action changes product state or affects other people.
Execute through a narrow tool. Use a typed, allowlisted operation. Make retryable actions idempotent so a repeated call does not create duplicate changes.
Verify the result. Read the resulting product state and look for the defined success event. Tool completion alone does not prove customer resolution.
Record the outcome. Log the context version, playbook, model, policy decision, tool call, result, success signal, and any escalation or user reversal.

The loop supports two related products without collapsing their permissions. An internal analytics agent can identify an affected cohort, inspect a funnel, or surface a recurring failure pattern. A customer-facing support agent can use the approved finding to help one authenticated user, but it should see only that user’s permitted context and tools. A human support operator should receive the same trace when the agent escalates.

Keep the shared layer deliberately small: stable identities, canonical metric definitions, governed context fields, outcome events, and versioned playbooks. The analytics agent and support agent can then improve the same system while retaining separate access policies and evaluation criteria.

Do not automatically convert every observed correlation into a new support action. Let analytics generate a candidate playbook, review the causal logic and risk, test it against known cases, and release it through a controlled experiment. The learning unit is the reviewed playbook, not an unexamined prompt change.

Choose a first workflow that can prove its own value

The first pilot should be easy to verify, not merely easy to demonstrate. A conversational answer looks polished even when it does not change the user’s outcome. A narrow configuration or onboarding workflow is usually a better proving ground because eligibility, allowed actions, and success can be defined before launch.

Score candidate workflows against these criteria:

Repeated demand: the same intent or failure appears often enough to justify a reusable playbook.
Observable state: the agent can read the prerequisites and current configuration instead of guessing from the user’s description.
Clear success: one product state or behavioral event can verify that the problem was resolved.
Safe execution: the initial actions are reversible, user-scoped, and unlikely to affect billing, security, data retention, or other users.
Short feedback: the primary outcome appears soon enough to support iteration, even if retention is monitored later.
Enough eligible traffic: the workflow can support a meaningful experiment rather than a handful of anecdotes.

Write the pilot contract before the prompt

A pilot contract forces the product, analytics, support, engineering, and risk decisions into one inspectable artifact. It should specify:

the user problem and eligible cohort;
the trigger that starts the workflow;
the context fields and systems of record;
the approved diagnostic branches;
the allowed and prohibited actions;
the point at which confirmation is required;
the precondition and postcondition for each tool call;
the success event and observation window;
known failure modes and the human handoff rule; and
the primary outcome, guardrail metrics, experiment design, and minimum detectable effect.

Consider an onboarding configuration workflow. The trigger might be a user repeatedly reaching setup without completing it. The context could include entitlement, current configuration, prerequisite status, and the latest validation result. The agent may be allowed to run validation, explain a missing prerequisite, prefill a reversible setting, or launch the next approved step. Resolution requires both the expected configuration state and its corresponding success event. If validation continues to fail, the handoff should include the exact state, error, playbook branch, and actions already attempted.

Avoid starting with data deletion, broad permission changes, security recovery, billing adjustments, or external communications. Those workflows combine difficult authorization questions with high consequences. Prove context quality, tool reliability, verification, and measurement on a narrower action set before expanding the blast radius.

Set the minimum detectable effect before the experiment. If the eligible population cannot detect an outcome change that would justify the investment, narrow the claim, combine additional time periods, or choose a more observable workflow. Do not call an underpowered neutral result proof that the agent has no effect.

Instrument the agent like a product surface, not a transcript

Conversation volume, message count, and thumbs-up feedback are diagnostic signals. They are not sufficient outcome measures. A customer can like an explanation and still remain blocked; another can dislike the wording even though the configuration was fixed.

Measurement layer	Question it answers	Useful signals
Operational reliability	Did the system execute as designed?	Tool success, validation failure, retry, latency, rollback, and escalation
Verified resolution	Did the requested product state become true?	Verified resolution rate, time to resolution, repeat attempt, and repeat contact
Product outcome	Did the user progress in the journey?	Activation, feature adoption, workflow completion, and later retention
Support outcome	Did the workflow reduce avoidable support effort?	Eligible ticket rate, escalation reason, handle-time impact, and handoff quality
Safety and trust	Did the agent stay within policy and user intent?	Permission block, wrong-action review, user reversal, policy violation, and privacy incident

Define the denominators as carefully as the numerators. Verified resolution rate should use eligible support sessions as its denominator and require the success state defined in the pilot contract. Action completion rate should use authorized action attempts, not every conversation. Time to resolution should begin with the original request and stop only when the postcondition is verified, not when the agent finishes generating text.

Do not optimize ticket deflection or containment in isolation. The absence of a ticket can represent resolution, abandonment, or a user working around the problem. Pair support-efficiency measures with product success, repeat contact, and safety guardrails.

Use evaluations and experiments for different questions

A disciplined AI product rhythm connects eval-driven development, A/B testing, minimum detectable effect, activation, retention analysis, and data governance. Each mechanism answers a different question:

Pre-release evaluations: Can the system interpret known intents, select the right context, follow policy, choose an allowed tool, handle tool errors, and verify the expected postcondition? Run the relevant suite whenever the model, prompt, context contract, policy, tool, or playbook changes.
Shadow operation: What would the agent have proposed in real traffic without being allowed to change state? Review mismatched diagnoses, unsupported context, unsafe actions, and missed escalation conditions.
Controlled experiments: Does the agent improve the predefined outcome compared with the existing support experience for the eligible population? Record assignment before the interaction and preserve it through outcome analysis.
Production monitoring: Are errors, reversals, escalations, latency, or policy blocks changing by journey, user role, entitlement, playbook, or release version?

Be careful with naive correlation. Users who invoke support are often already struggling, so their outcomes may look worse than those of users who never needed help. Random assignment among eligible users gives you a defensible counterfactual. When randomization is not possible, describe the result as observational and avoid claiming that the agent caused the change.

Log enough version information to reproduce a decision: model, prompt, policy, context schema, playbook, experiment assignment, tool version, input identifiers, authorization result, and postcondition. Do not place raw secrets or unrestricted personal data in that trace. A metric change is actionable only when you can connect it to the system version that produced it.

Set action boundaries before the model receives tool access

Model confidence is not authority. A highly confident response must never expand a user’s permissions, bypass confirmation, or convert a prohibited action into an allowed one. Authorization belongs in deterministic policy and tool infrastructure outside the model.

Action class	Typical scope	Required controls	Verification
Read and explain	Show relevant state, explain an error, or recommend a next step	User-scoped reads, field filtering, and visible uncertainty when evidence conflicts	Confirm that the response used current state and an approved knowledge path
Reversible change	Update a non-sensitive preference, run validation, or trigger a recoverable workflow	Preview, confirmation when needed, typed input, idempotency, and rollback	Read the resulting state and observe the workflow’s success event
Consequential change	Alter billing, permissions, security, external communication, or retained data	Strong confirmation or human review, separation of duties, and a complete audit trail	Verify every postcondition and provide a safe recovery or escalation route

Implement the boundary with controls the agent cannot negotiate away:

Least-privilege credentials: issue short-lived, user-scoped authorization rather than a general service credential wherever the architecture permits it.
Allowlisted tools: expose narrow actions with typed parameters, explicit preconditions, and constrained targets. Do not give a customer-facing agent arbitrary database or shell access.
Policy before execution: validate identity, permission, data sensitivity, action class, and confirmation status outside the model before any state-changing call.
Postcondition checks: require the agent to read the resulting state. A successful API response can still produce the wrong business outcome.
Safe retries: attach idempotency controls to operations that might be repeated after a timeout or interrupted conversation.
Complete handoffs: send the human operator the intent, relevant context, diagnosis, attempted action, tool result, and unresolved condition so the customer does not have to start over.
Controlled release: use feature flags, cohort restrictions, action-level limits, and an immediate disable path while a workflow is being validated.

Evaluate build-versus-buy decisions at the system boundary

Conversation quality is easy to demonstrate and difficult to use as a purchasing criterion. Evaluate an agent platform on whether it can operate inside your context, permission, observability, and experimentation model.

Can you define and inspect the context contract for each workflow?
Can the platform use user-scoped credentials and enforce tool permissions outside the prompt?
Can every decision, action, version, and outcome be exported to your unified analytics platform?
Can you separate aggregate analytics access from individual customer support access?
Can you run offline evaluations, shadow traffic, controlled experiments, and cohort rollouts?
Can you configure confirmation, rollback, handoff, retention, and data-residency policies?
Can you change the model, tool, or support system without losing metric definitions and historical outcome traces?

A platform that generates excellent dialogue but cannot expose its action trace or connect to verified outcomes will make governance and product measurement harder. A less theatrical system with clear contracts may be the more useful product foundation.

Key takeaways

Start with a governed context contract, not a larger prompt or model.
Connect product analytics and support through shared identities, metric definitions, outcome events, and versioned playbooks.
Give customer-facing agents user-scoped context and a small set of reversible, allowlisted actions.
Count a resolution only when the intended product state or success event is verified.
Use offline evaluations for capability and policy, controlled experiments for causal impact, and production monitoring for drift and safety.
Expand autonomy only after context accuracy, tool reliability, outcome lift, and guardrails have all been demonstrated.

At your next roadmap review, ask for one pilot contract rather than a broad AI support initiative. Choose one recurring journey, name its verified success event, define the smallest safe action set, and make the owner show how every action will be authorized, observed, and reversed. That is enough to move from a chatbot concept to an agentic product you can manage.

References

April 21, 2026

Auditable AI Code Review: A Practical Operating Model

You are not deciding whether an AI model can find bugs in a pull request. You are deciding whether an automated reviewer can participate in a production control without leaving your team unable to explain, challenge, or reverse its decision.

If the only evidence behind an approval is a bot comment that says the change looks safe, keep the system advisory. An auditable AI reviewer needs a bounded mandate, a deterministic approval policy, traceable evidence, and a feedback loop tied to production outcomes. Build those controls first, and faster review becomes a consequence rather than a gamble.

Start with a decision contract, not a model prompt

An approval is a policy decision. The model can supply findings, evidence, and a recommendation, but it should not define the conditions under which its own recommendation becomes authoritative.

Write a decision contract before selecting a model or tuning a prompt. It should answer five questions:

What may the system decide? Typical outcomes are approve, request changes, provide non-blocking comments, or escalate to a person.
Which changes are eligible? Eligibility should be determined by explicit repository, path, change-type, test, ownership, and reversibility rules.
Which checks are mandatory? An eligible pull request should not be approved if a required review lens failed to run, returned incomplete evidence, or produced an unresolved blocking finding.
When must the system abstain? Missing context, conflicting findings, unavailable tools, excessive scope, low-confidence evidence, and protected code paths should cause escalation rather than optimistic approval.
Who owns the result? Name the engineer accountable for the change, the owner of the review policy, and the person or group authorized to change the automation boundary.

The core approval rule can be expressed plainly: the change is eligible, every mandatory check completed, no blocking issue remains, the evidence record is complete, and no human-review requirement was triggered. Encode that rule in a controller your team can inspect and test. Do not bury it inside natural-language instructions to the model.

This separation gives you a clean control plane. Review agents analyze the change. A policy engine evaluates their structured results. A narrowly permissioned service performs the approved action. The model never gets to reinterpret the boundary at the moment it encounters a difficult pull request.

Auditability does not require a future model run to reproduce the same words. Model endpoints, retrieved context, and dependencies can change. It requires the original decision to remain reconstructable from preserved inputs, outputs, policies, tool results, and versions. A skeptical engineer should be able to determine why the pull request was approved without trusting the personality or reputation of the bot.

Split the review into specialist checks with explicit evidence

A single prompt asking whether a pull request is safe compresses several different judgments into one opaque answer. Decompose the review so that each judgment has a clear purpose, input set, output schema, and failure mode.

A practical review pipeline can include these specialist lenses:

Problem-definition quality: Is the requested behavior specific enough to review, and are the acceptance conditions testable?
Intent alignment: Does the diff implement the stated change without silently expanding or contradicting it?
Scope and dependency impact: Which callers, data flows, interfaces, jobs, or services can the change affect outside the edited files?
Logical correctness: Do the changed execution paths handle expected states, boundary conditions, and failure paths?
Test adequacy: Do the tests exercise the behavior that changed, and did the required checks actually run against the reviewed commit?
Security and privacy: Does the change alter trust boundaries, permissions, authentication, secrets, sensitive data handling, or externally controlled inputs?
Local engineering guidance: Does the implementation comply with versioned repository conventions, architectural constraints, and known anti-patterns?
Deployment and recovery: Can the change be observed, disabled, or rolled back without creating a second unsafe operation?

Every specialist should return the same minimum structure: a check identifier, pass/fail/escalate status, a concise claim, evidence tied to files or tool output, the applicable rule version, severity, and a recommended action. A finding such as a possible regression is not auditable. A finding that identifies the affected path, explains the conflicting behavior, points to the relevant code, and names the violated policy is.

Run independent checks before aggregation, and preserve every result even when the final decision is approval. The aggregator may deduplicate findings, but it should not erase dissent. If the intent checker says the change is aligned while the execution-path checker finds a contradiction, route the conflict to a person.

Review context must extend beyond the visible diff. A seemingly harmless one-line copy change was once found to contradict validation behavior elsewhere in the codebase. That is the kind of defect a diff-only reviewer is structurally unlikely to see. Give relevant checks controlled access to callers, validators, schemas, tests, ownership metadata, and versioned internal guidance, then record exactly which context each check used.

More context is not automatically better. Retrieval should be targeted and attributable. When a finding depends on an internal rule, capture the rule identifier and version. When it depends on a test, capture the command, commit, status, and output reference. When it depends on inferred execution flow, record the relevant path so a maintainer can inspect it.

Treat pull-request text, code comments, test fixtures, generated files, and documentation on the changed branch as untrusted data. They can contain instructions designed to redirect an agent. Load approval policy from a protected service or the trusted base branch, not from files the pull request can rewrite. Run proposed code in an isolated environment, mediate tool calls through an allowlist, and keep approval or merge credentials outside the model’s reach. The policy controller should translate a valid decision into an action; the model should never hold the credential that performs it.

Set the automation boundary with hard eligibility gates

Do not begin by assigning every pull request a risk score and approving anything below a convenient number. A composite score can hide a disqualifying condition: a tiny authorization change may receive a low size score even though its blast radius is high. Apply hard gates first. Use scoring only to route changes that remain eligible after those gates.

Common reasons to require human review include:

Authentication, authorization, permissions, cryptography, secrets, or trust-boundary changes.
Payments, billing, entitlements, destructive data operations, or irreversible migrations.
Public API contracts, shared schemas, release infrastructure, or broadly consumed dependencies.
A pull request that changes its own review policy, test requirements, ownership rules, or deployment controls.
Missing required tests, failed or stale CI results, unavailable analysis tools, or a mismatch between the reviewed commit and the tested commit.
Changes spanning too many concerns, components, or execution paths for the approved review envelope.
An active incident, an unclear rollback path, or a direct request for human review.

There is no universal line-count threshold for a small pull request. Derive limits from your architecture and incident history, then version them. A change to a central permission function may be riskier than a much larger isolated test refactor. Scope should include dependency reach and behavior change, not just added and deleted lines.

A staged authority model keeps the boundary legible:

Mode	What the AI reviewer may do	Who decides the merge	Appropriate use
Shadow	Produce a private decision record without affecting the pull request	Human reviewer	Baseline evaluation and policy tuning
Advisory	Post evidence-backed, non-blocking findings	Human reviewer	Measuring usefulness and false alarms in normal work
Blocking	Request changes for narrow, testable policy violations	Human reviewer after resolution	Stable rules with clear evidence and an appeal path
Bounded approval	Approve only changes that pass every eligibility and review condition	Policy controller within its delegated scope	Validated low-risk change classes with complete audit records
Mandatory escalation	Summarize evidence and route the change	Named human owner	Sensitive paths, conflicting findings, missing evidence, or any requested human review

Do not turn bounded approval into an auto-approval quota. Coverage is a result of demonstrated safety, not a target that should pressure teams to weaken eligibility rules.

One high-frequency engineering environment reports that more than 93% of pull requests across two main codebases are agent-driven and more than 19% are approved without a human reviewer. Its reported median merge time fell from 75.8 minutes with human review to 14.6 minutes with AI approval, while downtime from breaking changes declined 35% as deployments doubled. Those organization-level results show that bounded automation can coexist with high deployment frequency and improving safety outcomes. They do not prove that AI approval caused the downtime reduction, and they should not be imported as another team’s launch threshold.

Keep the escape hatch explicit. Any engineer should be able to request a human review without defending the choice. The accountable engineer should still watch the change in production and be ready to roll it back. Automated approval changes who performs a review step; it does not transfer ownership of the production outcome to a model.

Preserve the evidence, then earn autonomy through evaluation

Build a decision record that survives model and policy changes

Create an append-only decision event for every review attempt, including abstentions and failed runs. At minimum, retain:

Repository, pull-request identifier, base commit, reviewed head commit, author, accountable owner, and timestamps.
The pull-request description and acceptance criteria as they existed when the decision was made.
Eligibility rules, protected-path rules, ownership data, prompt-template identifiers, and policy versions.
Model provider and model identifier, relevant runtime settings, retrieval configuration, and tool versions.
The context each specialist received, including immutable references or preserved snapshots for mutable material.
Structured specialist outputs, supporting evidence, tool invocations, CI results, conflicts, and failures.
The deterministic rule evaluation that produced approve, block, comment, or escalate.
Subsequent human overrides, appeals, edits, approvals, merges, rollbacks, hotfixes, and linked incidents.

Store concise decision rationale and inspectable evidence, not hidden chain-of-thought. An auditor needs to know which claim was made, what supported it, which rule applied, and how the controller reached the outcome. Private internal reasoning is neither necessary nor a reliable substitute for those artifacts.

Apply the same security discipline to review logs that you apply to source code. Minimize captured secrets and personal data, control access, define retention, and log policy changes. If a model or retrieval service cannot handle the code under your data-governance requirements, that repository is not eligible for the workflow.

Evaluate decisions, not polished comments

A review can sound thoughtful and still approve the wrong change. Build an evaluation set around decisions and evidence rather than writing quality.

Assemble representative cases. Include clean pull requests, valuable historical human findings, escaped defects, incident-causing changes, incomplete requirements, sensitive paths, cross-component changes, and attempts to manipulate the reviewer through repository content.
Label the expected control outcome. For each case, identify whether the correct action is approve, request changes, or escalate. Record the evidence that an acceptable review must surface.
Separate clear cases from disputed ones. Known incident causes and explicit policy violations can provide strong labels. Ambiguous architectural judgments need maintainer adjudication, and disagreement should remain visible rather than being forced into false certainty.
Freeze a holdout set. Use one portion to improve prompts, retrieval, and policy. Keep another portion unseen until release evaluation so repeated tuning does not create a misleading score.
Compare equivalent cohorts. Evaluate AI and human review on the same risk classes and change types. Comparing AI-approved low-risk changes with all human-reviewed pull requests confounds reviewer quality with task difficulty.

Track metrics that expose different failure modes:

Decision accuracy: How often did the system choose the expected approve, block, or escalate outcome?
False auto-approval rate: How often did it approve a labeled case that should have been blocked or escalated? Break this out by severity and risk class.
Blocking precision: Of the findings that stopped a change, how many maintainers judged valid and actionable?
Known-defect recall: Which seeded or historically verified defects did the review catch? Label this carefully; it is not recall over every defect that might exist.
Evidence completeness: Can every decision be traced to required checks, immutable inputs, policy versions, and supporting artifacts?
Abstention and override rates: Where is the system uncertain, and where do engineers reverse it? Investigate patterns by repository and change class.
Delivery performance: Measure review latency and merge time, but only alongside quality metrics.
Production outcomes: Track rollbacks, hotfixes, escaped defects, incidents, downtime, and customer impact for comparable risk cohorts.

Comment helpfulness is useful feedback, but it is not a safety metric. Engineers may like a concise reviewer that misses a critical defect, or dislike a strict reviewer that correctly blocks an unsafe change. Keep usefulness, correctness, and production impact as separate measures.

Roll out by change class and turn escapes into regression tests

Move from shadow mode to advisory comments, then to narrow blocking rules, and only then to bounded approval. Start with one repository and one low-risk, reversible change class. Write the exit criteria before the pilot begins, including acceptable false-approval and false-block rates, required audit completeness, escalation behavior, and production guardrails.

Canary each expansion. Maintain a kill switch that disables new automated approvals without removing the accumulated evidence. If a required service, model, retrieval index, test runner, or policy store is unavailable, fail closed and return the pull request to the human path.

When an approved change causes a production problem, diagnose the control layer that failed:

Was the change wrongly eligible?
Did retrieval omit relevant code or guidance?
Did a specialist miss or misclassify the defect?
Did the aggregator suppress a conflict?
Did the policy permit approval despite the evidence?
Did CI test a different commit or an incomplete environment?
Did production monitoring fail to surface the effect promptly?

Add the case to the regression suite, version the corrective policy or guidance, rerun the holdout evaluation, and preserve the relationship between the incident and the updated control. That is eval-driven development applied to governance: every escape should make a specific layer harder to fail in the same way again.

Key takeaways

AI output is an input to approval, not the approval policy itself.
Use deterministic eligibility gates before any model-based risk judgment.
Decompose review into specialist checks that return claims, evidence, rule versions, and explicit pass/fail/escalate states.
Keep policy and credentials outside the pull request and outside the model’s control.
Preserve enough evidence to reconstruct the original decision even when the model, repository, or internal guidance later changes.
Expand autonomy only when evaluation and comparable production cohorts support it; never optimize for auto-approval coverage by itself.

Your first useful milestone is not an AI-approved pull request. It is a shadow decision that a maintainer can reconstruct, dispute, and improve. Once that record is dependable, grant the smallest reversible slice of authority, watch what reaches production, and make every expansion earn its place.

References

Intercom — AI Now Approves Our Pull Requests—Safely: Inside an Agentic, Auditable Review Engine

April 21, 2026

A Practical Scenario Planning System for AI Product Strategy

Your leadership team wants a firm answer: which AI bets belong on the roadmap, and which ones are expensive distractions? The difficult part is not generating ideas. It is deciding what to fund when customer adoption, interfaces, economics, and enterprise constraints could each develop differently.

A useful scenario plan does not hide that uncertainty behind a confident forecast. It converts uncertainty into conditional commitments: what you fund now, what you preserve as an option, what evidence would change the decision, and what you refuse to scale until the right signal appears.

Frame the exercise around a decision, not the future of AI

Broad questions such as “What will AI look like?” produce interesting conversations and weak strategy. Nobody has to choose anything at the end. Start with a decision that has an owner, a planning window, and a meaningful consequence if the underlying assumptions prove wrong.

A usable decision statement looks like this: “For this customer and workflow, should AI become the primary experience, remain an embedded assistant, or stay in discovery while the existing product carries the outcome?”

Write down five elements before discussing scenarios:

The decision: State the product, investment, or sequencing choice that must be made.
The planning window: Define when the choice needs to be made and when it can be reconsidered.
The expensive assumption: Identify what must be true for the proposed strategy to work.
The reversal cost: Separate choices that can be changed cheaply from commitments involving substantial architecture, hiring, migration, or go-to-market work.
The decision owner: Name the person accountable for changing the plan when the evidence changes.

A scenario belongs in the exercise only if it could change that decision. If two imagined futures lead to the same investment, combine them. More narrative does not create more strategic value.

Next, identify the customer need that should remain valid across the plausible futures. Customers may still need to complete a workflow with less effort, avoid costly rework, understand what the system did, or retain control over a consequential action. Those durable needs become the anchor. They keep an empowered product team focused on outcomes when a particular interface, model, or market prediction stops holding up.

Build scenarios from uncertainties that can change the plan

Do not begin by writing a best case, a base case, and a worst case. That format encourages everyone to treat the base case as the forecast. Instead, find uncertainties that are both unresolved and capable of changing your product choice.

Useful uncertainty prompts include:

Customer behavior: Does repeated AI use spread through the intended market, or remain concentrated among enthusiasts and specialists?
Interaction model: Does AI become the main way customers initiate work, or does it operate inside a familiar interface?
Scope of autonomy: Do customers delegate complete tasks, or accept AI only for bounded assistance and recommendations?
Product economics: Does the value created support the cost and operational burden of delivering the experience?
Enterprise constraints: Do security, privacy, compliance, procurement, and change-management requirements permit the proposed workflow?
Organizational readiness: Can the company evaluate, support, govern, and improve the product after launch?

Select the uncertainties with the greatest decision impact, then push competing possibilities to useful extremes. Extremes expose assumptions that a comfortable middle case can conceal. “Graphical interfaces disappear” and “AI remains an invisible utility” should not be treated as predictions. They are boundary conditions for examining what your product would need in very different environments.

For example, crossing the pattern of customer adoption with AI’s place in the workflow creates the following set of hypothetical futures:

Adoption pattern	AI’s place in the workflow	Plausible future	Decision it tests
Use broadens across intended segments	AI becomes the primary interaction	Customers increasingly start and complete the workflow through AI	Whether to redesign the core experience around an AI-first path
Use broadens across intended segments	AI remains embedded	AI becomes valuable infrastructure inside a familiar product	Whether intelligence, context, and workflow integration matter more than a new interface
Use remains uneven	AI becomes primary for specialists	A smaller group wants an AI-native experience while the broader market retains existing habits	Whether to support distinct experiences instead of forcing one migration
Use remains uneven	AI remains embedded	Bounded assistance improves parts of the workflow without replacing it	Whether focused augmentation is a better investment than broad transformation

The point is not to choose your favorite quadrant. Develop each one far enough to reveal its product implications. For every scenario, describe the target customer’s behavior, the job that still matters, the role AI plays, the constraints that become binding, and the most likely way the strategy fails.

Keep the scenarios plausible rather than theatrical. A future that cannot affect a real decision is entertainment. A future that merely restates the current roadmap is confirmation bias.

Turn each scenario into signals, triggers, and stop conditions

A scenario without observable signals is just a story. A decision-ready scenario tells you what to watch, how that evidence relates to an assumption, and what action follows if the signal appears.

The common mistake is to monitor whatever is easiest to count. Trial starts, demo enthusiasm, and requests from technically confident customers can show interest, but they do not establish broad adoption. Early adopters cannot stand in for the whole market. Segment the evidence so that enthusiasm in one cohort does not silently become a claim about every customer.

Build a signal set that answers distinct questions:

Adoption: Are intended customers returning to the AI workflow after the initial trial, and is use spreading beyond opt-in enthusiasts?
Customer value: Is the core outcome improving through less effort, less rework, fewer avoidable errors, or another measure that matters for this workflow?
Trust and control: How often do customers accept, modify, override, or abandon the result, and what reason do they give?
Enterprise viability: Are security, compliance, procurement, or change-management reviews blocking deployment or narrowing the acceptable use case?
Operational viability: Are reliability, latency, support demand, and cost-to-serve compatible with the value being delivered?
Interface behavior: Do customers initiate work through AI, or invoke AI at specific points inside an established process?

Pair every confirming signal with a disconfirming signal. This prevents the team from collecting only evidence that supports the roadmap it already wants.

Strategic assumption	Confirming evidence	Disconfirming evidence	Product response
AI should become the primary interface	Intended customers repeatedly initiate and complete the core workflow through AI	Customers retreat to the familiar interface for consequential work	Keep the AI-first experience as an option while improving embedded assistance and control
Broader autonomy will create more value	Outcome quality improves while customer intervention and rework decline	Escalations, corrections, or abandonment persist as scope expands	Narrow the delegated task and strengthen evaluation, permissioning, and fallback behavior
Adoption can expand through the target market	Repeated use spreads across intended cohorts and survives enterprise review	Use remains concentrated among specialists or stalls during approval and rollout	Preserve a dual experience and address the blocking constraint before funding broad migration

Write the trigger before launching the bet. A practical format is: “When this signal persists in the target segment and this guardrail remains acceptable, move this investment from an option to a commitment. If this disconfirming signal appears, stop, narrow, or redesign the bet.”

The exact threshold will depend on your product, baseline, risk, and decision cost. What matters is agreeing on it before stakeholders can reinterpret ambiguous results. If nobody can say what evidence would reduce or end the investment, the roadmap contains a belief, not a testable strategy.

Convert the scenarios into a portfolio, roadmap, and sprint choices

Once the scenarios and signals are explicit, separate the portfolio by commitment type. This is where scenario planning becomes operating discipline rather than an occasional workshop.

No-regret bets: Investments that support the durable customer outcome across several scenarios. Depending on the product, these may include better evaluation, permissions, observability, fallback paths, data governance, or clearer measurement. Do not label generic platform work “no regret” unless it supports a named customer outcome.
Option bets: Bounded, reversible work that buys information or preserves a future choice. A prototype, limited workflow, architecture seam, or controlled release can test an assumption without committing the entire product.
Contingent bets: Investments that make sense only after a defined signal appears. Keep the entry condition beside the roadmap item so it cannot become committed work through inertia.
High-regret commitments: Expensive moves that are difficult to reverse, such as a forced workflow migration or a large architecture and hiring commitment. Require stronger support across scenarios before making them.

This creates a roadmap with different funding postures, not a backlog pretending every item has equal certainty.

Roadmap lane	Why it exists	What earns progress	What removes it
Durable outcomes	Advance needs that remain important across plausible futures	Evidence that the customer outcome is improving	The need or outcome no longer matters
Evidence bets	Reduce uncertainty between competing scenarios	A learning milestone tied to a strategic assumption	The assumption is resolved or the evidence cannot affect a decision
Triggered scale	Expand an option after its entry condition is met	The agreed confirming signal appears while guardrails remain acceptable	A stop condition appears or the economics no longer support expansion
Deferred commitments	Preserve ideas that are valid only in a narrower future	A named scenario becomes more plausible through observable evidence	The relevant scenario is disconfirmed

At sprint planning, ask what kind of item is entering delivery. Work should either create durable customer value or buy decision-relevant information. “Build an AI assistant” is an output. “Determine whether target customers will delegate this bounded task while retaining acceptable control” is a learning goal that can change a strategic choice.

For each evidence bet, require the product trio to answer:

Which scenario and assumption does this work test?
Which customer segment must provide the evidence?
What behavior or outcome will be observed?
What result would justify more investment?
What result would stop, narrow, or redirect the work?
Which part of the work remains valuable if the favored scenario is wrong?

This also changes the stakeholder conversation. Replace “Which prediction do you believe?” with “Which commitments are justified across plausible futures, which ones are options, and what signal unlocks the next level of funding?” The latter question makes uncertainty governable.

Review the decision when a trigger fires, a critical assumption changes, or the next expensive commitment approaches. A recurring calendar review can help, but elapsed time alone is not evidence. Keep a short decision record containing the active scenarios, current signals, funding posture, stop conditions, owner, and next decision point.

Key takeaways for your next AI roadmap review

Start with a product or investment decision that could genuinely change. Do not try to describe the entire future of AI.
Build scenarios from unresolved uncertainties with high decision impact, including customer behavior and real-world constraints.
Anchor the strategy in customer needs and outcomes that remain valuable across several plausible futures.
Separate confirming evidence from disconfirming evidence, and segment adoption so enthusiasts do not masquerade as the whole market.
Predefine the signal, guardrail, trigger, and stop condition for every material option bet.
Fund no-regret moves now, use reversible work to buy information, and hold contingent commitments until their entry conditions appear.
Connect every sprint item to either durable value or a decision-relevant uncertainty.

At your next roadmap review, choose the most expensive assumption behind one AI initiative. Write the opposite plausible scenario, identify the customer need shared by both futures, and name the signal that would change the funding decision. If the roadmap still makes sense, the strategy is more resilient. If it does not, you have found the adaptation point before the market finds it for you.

References

Product Talk – Predicting the Future: All Things Product Podcast with Teresa Torres and Petra Wille

April 21, 2026

AI-Native Startup Execution: A Practical Operating System

Your startup can produce impressive demos every week and still learn slowly. If model behavior, customer evidence, and deployment feedback move through separate queues, faster shipping only creates more uncertainty.

If you are deciding how to run an AI-native startup, use one standard: how quickly can your team turn a real customer artifact into an evaluated product change and a measurable customer outcome? That standard should shape your wedge, ideal customer profile, team design, sales motion, and operating cadence.

Build the smallest closed learning loop

An AI-enabled company can add a model to an existing product process. An AI-native company has to organize the process around intelligence itself: the data entering the system, the judgment the model makes, the action produced, the evaluation of that action, and the feedback that improves the next decision.

That makes a long AI feature list the wrong starting point. Begin with the smallest end-to-end loop that proves the product’s central claim. In a security product, for example, the loop might start with suspicious activity, produce a risk judgment, stop or downgrade a threat, and capture enough evidence to assess whether the intervention was correct. A polished dashboard without that closed loop is packaging, not proof.

Use three gates before committing to a wedge:

The customer can quantify the problem. Ask for frequency, severity, operational burden, or another observable consequence in the customer’s own workflow. General concern about AI is not enough.
The wedge can create value within one buying cycle. If proving value requires a broad platform rollout, several integrations, and organizational change across multiple functions, you have probably selected a destination rather than an entry point.
Product use improves your data advantage. Determine what feedback the product will capture, whether you can legitimately use it, how quickly it arrives, and which evaluation or model decision it improves. Data volume alone does not create an advantage.

Failing any one of these gates weakens the loop. Urgent pain without accessible data leaves the model blind. Useful data without a near-term outcome produces an experiment that customers may admire but will not adopt. A fast result that teaches you nothing can become a services engagement disguised as software.

Your first version should feel narrower and less polished than the roadmap in your head. It may require manual review, a rough operator interface, or close founder involvement. That is acceptable when the roughness is visible and recoverable. It is not permission to hide model failure, skip evaluations, or make consequential actions impossible to inspect. Cut breadth before you cut control.

Define completion around the loop rather than the interface. You are done with the first proof when a target customer can supply the relevant input, receive the intended outcome, show whether it helped, and feed the result back into a repeatable evaluation. Everything else competes with learning speed.

Choose an ICP that maximizes learning density

The largest market is rarely the most useful starting market. Your early ideal customer profile should concentrate urgency, usable data, workflow access, and a reachable decision maker. Those conditions allow one customer interaction to improve both the product and the go-to-market motion.

Create a one-page ICP card with four evidence fields:

Urgency: What is happening now that makes the buyer act rather than merely agree that the problem matters?
Data pathway: Which alerts, cases, decisions, errors, or other operational artifacts can enter the product and its evaluation process?
Workflow position: Where will your output appear, who will act on it, and what existing step must change?
Success signal: What observable customer outcome would justify adoption, renewal, or expansion?

Rate each field as high, medium, or low, but require an evidence note beside the rating. A founder’s conviction is not evidence. A buyer who can describe the cost of the problem, provide representative artifacts, identify the operator who owns the workflow, and agree on a success signal gives you something testable.

Keep adjacent customer segments out of the first learning loop when they require different data, integrations, evaluation criteria, or buying logic. A larger pipeline can make progress look healthier while mixing incompatible requirements into the roadmap. The practical test is simple: if two prospects would judge the same model behavior by different definitions of success, they should not share an early product plan merely because they could buy from the same company.

Founder-market fit helps here, but it is a compression mechanism rather than proof of product-market fit. Deep domain experience can sharpen the problem statement, reduce translation with buyers, and establish credibility. It cannot substitute for evidence that customers will change a workflow around the product.

Founder-led sales should therefore operate as a structured discovery system. For every lost or stalled opportunity, record the exact objection, the stage at which it appeared, the evidence offered by the buyer, and the suspected root cause. Do not turn each objection into a feature request. Cluster the objections, identify the two most common root causes, and turn those into experiments within a sprint.

Look beyond compliments when judging early product-market fit. Stronger signals appear when customers escalate a problem to your team, volunteer data that can improve the system, or invite you into the workflow where decisions are made. None is conclusive alone, but each requires the customer to spend trust, attention, or organizational effort. That makes them more informative than enthusiasm during a demo.

Organize the team around evaluation and deployment

A conventional feature organization separates product definition, engineering delivery, model work, implementation, and customer support. That creates handoffs precisely where an AI-native startup needs rapid feedback. Organize durable units around a customer outcome, with the people who can inspect data, change the system, deploy it, and evaluate the result.

The exact titles can vary, but the unit needs four capabilities:

Product judgment: Choose the scenario, define the customer outcome, and decide which uncertainty deserves the next experiment.
Model and data judgment: Design evaluations, diagnose failure patterns, and understand whether a change improves one scenario while damaging another.
Product engineering: Turn model behavior into a reliable workflow with usable interfaces, permissions, integrations, and operational controls.
Frontline deployment: Work safely in live customer environments, resolve implementation friction, and return generalizable learning to the product.

Forward deployed engineers and solutions engineers are especially valuable when the product has to enter complicated workflows. They can shorten the distance between a live failure and a product decision. They can also become an unofficial custom-development queue. Prevent that by attaching three fields to every customer-specific change: the broader pattern it tests, the product owner responsible for reviewing the result, and the condition under which the work will be standardized, stopped, or kept outside the core product.

AI fluency should also change your hiring process. Model vocabulary on a resume tells you little about how someone will reason under uncertainty. Give candidates a work sample built around a realistic failure: model performance changes across scenarios, the data distribution is moving, and customer trust is at risk.

Ask the candidate to define the evaluation, explain the consequences of false positives and false negatives, diagnose a precision-recall tradeoff, and translate the technical result into a product decision. The strongest response will not jump straight to a model change. It will clarify the scenario, inspect the evidence, identify what the current evaluation misses, and explain the tradeoff in terms an operator or buyer can use.

Use the same discipline for founder and leadership disagreement. Write down the principle in dispute, the evidence each side is using, the decision owner, the time box, and the condition that will trigger a review. Debate the principle rather than multiplying competing implementation plans. Once the decision is made, commit until the agreed review point. This preserves speed without pretending that an uncertain decision has become permanently correct.

Run model and business evidence on the same cadence

An AI-native startup needs two connected scoreboards. One shows whether the system is becoming more reliable. The other shows whether customers are receiving enough value to adopt, retain, and expand. Reviewing them separately makes it easy to celebrate a model improvement that does not change customer behavior, or a short-term commercial win built on fragile product performance.

Evidence layer	Question	Measures to inspect	Decision it should inform
Scenario quality	Does the system make the right judgment in each important situation?	Precision and recall by scenario; recurring misclassification patterns	Evaluation coverage, data work, model changes, and acceptable operating thresholds
Change detection	Does the system recognize when its environment is moving?	Drift, data freshness, and time to first signal for an emerging pattern	Refresh cadence, investigation priority, and new evaluation cases
Operational experience	Can the customer use the output without creating a new burden?	Alert-fatigue reduction and latency under load	Workflow design, prioritization, and deployment constraints
Business value	Is reliable behavior turning into durable adoption?	Activation, expansion, and Net Recurring Revenue	ICP focus, onboarding, positioning, and roadmap investment

Do not collapse these measures into one composite score. Aggregation hides the failure mode you need to fix. A model can improve overall precision while degrading a high-consequence scenario. Activation can remain flat even when evaluation results improve because the output arrives too late, lacks an escalation path, or does not fit the operator’s workflow. The broken link between the two scoreboards is often the most valuable finding in the review.

Make the weekly customer debrief evidence-based. Bring actual alerts, misclassifications, escalations, and deployment failures rather than a verbal account of how the customer feels. For each material artifact, record the hypothesis it challenges, the evaluation that can reproduce it, the next bet, and the owner. That written log becomes organizational memory when priorities change quickly.

Use an evaluation gate for every consequential release:

Run the relevant scenario-level evaluations, not only an aggregate benchmark.
Inspect newly introduced failure modes as well as improvements to the target case.
Check latency and operational behavior under the conditions the customer will actually use.
Name the owner of monitoring, escalation, and rollback before deployment.

This is not process for its own sake. In an AI product, observability is the control center for trust. It tells your team whether the model still deserves the authority the workflow gives it. Demo speed cannot compensate for an inability to see, explain, and correct failure in production.

Key takeaways for your next operating cycle

Define the product as a closed loop from customer signal to verified outcome, not as a list of AI capabilities.
Commit to a wedge only when the problem is quantified, value can appear within one buying cycle, and legitimate product use improves the data advantage.
Select the first ICP for urgency, data access, workflow access, and a clear success signal. Keep adjacent segments separate when they require different definitions of success.
Turn founder-led sales into a discovery system by logging every rejection, clustering root causes, and testing the two most common causes within a sprint.
Build durable customer-facing units that combine product, model, engineering, and deployment judgment.
Review scenario-level model evidence and business outcomes together, then investigate the link whenever one improves without the other.

For your next planning cycle, choose one ICP, one end-to-end loop, one scenario-based evaluation set, and one customer outcome. Give a single owner responsibility for showing the connection between them. If the team cannot trace that chain with production evidence, the roadmap is still too broad. Narrow it before adding another feature, segment, or model.

References

Shivam.Consulting Blog – Inside Artemis’ AI vs AI Security War: Hiring at Speed, PMF Signals, and Founder-Led Sales

April 21, 2026

Churn Prediction: A Practical Build-Versus-Buy Framework

You need a churn score soon. Customer success wants a prioritized account list, engineering wants requirements, and finance wants to know whether it is funding a vendor contract or a permanent internal capability. A polished model can still leave all three teams waiting if nobody has decided what happens after an account is flagged.

Start with the retention decision, not the algorithm. Once you know who will act, what they will do, and how you will measure the result, the build-versus-buy choice becomes much clearer.

Decide which capability you actually need to own

Churn prediction is often discussed as if it were a single model. In practice, it is an operating loop with several layers:

Define the outcome. Specify which customers can churn, what event counts as churn, and the prediction window that gives your team enough time to intervene.
Assemble the signals. Connect product usage, account attributes, engagement, support, billing, and other permitted data to a consistent customer identity.
Estimate risk. Produce a score, category, or ranking that separates accounts requiring attention from the rest of the portfolio.
Activate the prediction. Route the result into the CRM, customer-success workflow, lifecycle message, or in-product experience where somebody can respond.
Learn from the intervention. Measure whether the action changed retention, adoption, engagement, or Net Recurring Revenue rather than assuming that a plausible score created value.

You do not necessarily need to own every layer. A vendor might provide behavioral analytics, scoring, in-app guides, and CRM integration while you retain ownership of the churn definition, intervention policy, and experiment design. Conversely, you might build a specialized risk model but continue using commercial tools to collect events and deliver treatments.

My default is to separate model ownership from outcome ownership. Your company must own the definition of success, the permitted uses of the score, and the learning loop. It only needs to own the model code when that ownership creates a strategic advantage.

Before evaluating an architecture or vendor, complete this sentence:

When a customer in [defined population] crosses [risk condition], [named owner] will take [specific action] through [named system], and I will judge the intervention using [business outcome].

If you cannot complete it, pause the model decision. You have an intervention-design problem. Buying software will automate the ambiguity, while building will make the ambiguity more expensive.

Run six decision gates before choosing a path

The right answer depends on more than whether your team can train a model. Use these gates to expose the constraint that should control the decision.

Decision gate	Evidence to inspect	What pushes you toward a path
Time to value	Decision deadline, current churn visibility, and readiness of the first intervention	Urgent activation favors buying; a longer strategic horizon makes building more viable
Data readiness	Outcome labels, identity resolution, event consistency, signal freshness, and usable history	Immature data favors a packaged baseline while you repair foundations; reliable proprietary data strengthens the case to build
Strategic differentiation	Signals or decisions competitors and general-purpose vendors cannot reproduce	A must-have retention capability favors buying; a defensible product advantage favors building
Operating talent	Named owners for data pipelines, production scoring, monitoring, governance, and intervention design	Missing ownership favors buying; durable cross-functional capacity makes building credible
Activation fit	CRM, customer-success, messaging, analytics, and in-product delivery requirements	Standard integrations favor buying; specialized actions or product-embedded scoring may require a build or hybrid approach
Risk and explainability	Privacy, access, retention, audit, explanation, and regulatory requirements	Standard controls may fit a vendor platform; domain-specific constraints can justify owning selected layers

Time to value: is speed useful, or merely urgent?

A short deadline only matters when an intervention is ready. If customer success already knows what it will do with a high-risk account, buying can put usable signals into existing workflows sooner. If the team has not agreed on an action, a fast score simply creates a faster queue of unanswered alerts.

Ask for the date on which a real user must receive the first actionable score. Then work backward through integration, workflow design, governance review, enablement, and experiment setup. This prevents a vendor demonstration or model prototype from being mistaken for operational readiness.

Data readiness: can your records support the decision?

A custom model cannot rescue an unstable churn definition or inconsistent customer identity. Inspect whether product events can be joined to the correct account, whether the churn outcome is recorded consistently, whether important segments have comparable coverage, and whether signals arrive early enough to support action.

Do not interpret weak data as an automatic reason to buy. A vendor cannot manufacture missing labels or repair every instrumentation gap. It can, however, give you a practical baseline using the signals already available while your team improves the data foundation.

Differentiation: would model ownership change your product advantage?

Build when proprietary context can materially improve the decision. That may include distinctive behavioral signals, domain-specific anomaly detection, specialized explanations, or a risk score embedded directly into your product. These are stronger reasons than a general preference to own technology.

If competitors could buy an equivalent capability and churn prediction mainly helps customer success prioritize outreach, ownership is unlikely to be the differentiator. Put product and engineering attention into the intervention, customer experience, and learning loop instead.

Talent: can you operate the system after launch?

Having someone who can train a model is not the same as having an operating team. A production capability also needs data engineering, scoring infrastructure, monitoring for drift, feature maintenance, incident ownership, governance, and a product owner who connects model changes to retention outcomes.

Put a name beside every continuing responsibility. An empty cell is not a future hiring plan; it is part of the build cost. If the same scarce people are also responsible for your core product, include the opportunity cost of redirecting them.

Activation: can the score reach the moment of action?

A prediction trapped in a dashboard has little retention value. Confirm that a score can create the right CRM task, customer-success play, lifecycle message, product tour, contextual tooltip, or in-app nudge. The recipient also needs enough explanation to choose an appropriate response.

Evaluate activation with a concrete scenario, not a feature checklist. Give a candidate vendor or internal team one representative account and ask it to show the full path from new behavior to updated risk, reason, assigned owner, intervention, and measured outcome. Any manual handoff in that path belongs in the decision record.

Governance: what must remain controlled and explainable?

Document which data may be used, who may see the result, how long inputs and scores are retained, what explanations users need, and how a customer could be affected by a mistaken classification. Privacy-by-design, data governance, regulatory compliance, and AI risk management apply whether the prediction is purchased or built.

Building gives you more design control, but it also transfers the burden of evidence, monitoring, and remediation to your organization. Buying transfers implementation work, not accountability. Require the same governance review for both paths.

The pattern is straightforward: buy when speed, standard coverage, and workflow activation dominate; build when proprietary signals, specialized explanations, or product differentiation dominate; blend when you need results now but have a credible reason to own selected layers later. A useful default is to buy a working baseline and build only where your context can create an outsized advantage.

Compare the full economics, not a license and a prototype

The most common cost comparison is structurally wrong: an annual software license is placed beside the effort required to train an initial model. One is closer to an operating capability; the other is an experiment. Compare both options across the same time horizon and include four cost classes: starting, running, changing, and exiting.

What belongs in the buy case

License, usage, seat, and service costs that apply to the intended customer population.
Implementation work for event collection, identity mapping, historical data, and system integrations.
Security, privacy, legal, regulatory, and procurement review.
Internal administration, score interpretation, workflow ownership, and user enablement.
Configuration or services needed for segments, reason codes, guides, alerts, and experiments.
Limits on data access, exports, custom features, scoring frequency, and downstream activation.
Migration effort if the vendor no longer fits, including preservation of historical scores and experiment records.

What belongs in the build case

Instrumentation, data quality, identity resolution, label construction, and feature pipelines.
Exploration, training, evaluation, explanation design, and production validation.
Batch or real-time scoring, storage, APIs, access control, and reliability engineering.
CRM, messaging, customer-success, analytics, and in-product integrations.
Monitoring for drift, broken inputs, coverage gaps, and unexpected segment behavior.
Retraining, feature maintenance, documentation, incident response, and ongoing product ownership.
Privacy controls, audit evidence, risk review, retention rules, and regulatory compliance.
Replacement or migration work when the architecture, churn definition, or business workflow changes.

Add cost of delay to both cases. Buying may carry a visible contract cost, but waiting for a custom capability can defer retention experiments and leave customer-success capacity poorly targeted. Building may require more internal investment, but a vendor that cannot express your signals or deliver the required intervention can delay learning in a different way.

Keep benefit assumptions separate from cost estimates. The model’s theoretical accuracy is not a financial return. Estimate value only through an intervention that can plausibly affect customer behavior, then validate that assumption with an experiment.

Your comparison should therefore show three views for each path:

Capability: which parts of the signal-to-action loop will actually work?
Economics: what will it cost to start, operate, change, and exit?
Evidence: what experiment will determine whether the capability improves retention or NRR?

If one option looks cheaper only because a row is blank, resolve the missing responsibility before approving it.

Use a hybrid path without creating two disconnected systems

A hybrid strategy is more than running a vendor score and an internal score at the same time. Done well, it sequences the work: buy the common layers needed for speed and activation, learn which proprietary signals matter, and build only the components that earn their continuing cost.

Phase one: establish a usable baseline

Choose one defined customer population, one churn outcome, and one intervention. Configure the purchased capability to produce a risk signal and a usable reason, then route both into the workflow where the named owner can act.

Record three different kinds of evidence:

Prediction evidence: coverage, signal freshness, ranking or precision, stability across relevant segments, and the usefulness of explanations.
Operational evidence: whether scores arrive in time, whether users understand them, and whether a flagged account reliably receives the intended treatment.
Business evidence: whether the intervention changes retention, adoption, engagement, or NRR.

Do not use prediction quality to claim business impact. It is possible to identify high-risk accounts accurately and still deliver an ineffective intervention. It is also possible for a broad model to create value because it reaches the right team at the right moment. These are different questions and need different measures.

Phase two: test where proprietary context adds value

Use retention analysis to identify behaviors that appear meaningfully connected to continued use or churn. Focus on information a general-purpose platform cannot represent well, such as domain-specific sequences, unusual account structures, specialized failure states, or product-specific anomalies.

Introduce one material improvement at a time. Compare the resulting decisions with the baseline: which accounts move, whether the reason becomes more actionable, and whether the intervention performs better. A more complex score is not automatically a better product.

Use A/B testing or another appropriate controlled rollout to evaluate the intervention. Set the minimum detectable effect before the test so the team agrees on the smallest change worth detecting and whether the experiment can support the decision. Where withholding an intervention is inappropriate, compare credible treatments or use a phased rollout rather than treating measurement as optional.

Phase three: build only the layer that proved distinctive

The result may not be a complete vendor replacement. You might own a proprietary feature pipeline, domain-specific anomaly detector, custom explanation layer, or specialized risk score while retaining commercial analytics and activation. That is often a cleaner boundary than recreating collection, dashboards, integrations, guides, and workflow delivery.

Before moving a custom component into production, require evidence that:

The proprietary signal changes a meaningful decision rather than merely changing a score.
The resulting intervention has a credible path to measurable retention or NRR impact.
A named team owns data quality, production reliability, drift monitoring, governance, and retraining.
The migration preserves the activation loop instead of sending users to a separate dashboard.
The added value justifies both the continuing cost and the engineering capacity displaced by the work.

Create a canonical risk contract before two systems coexist. Define the eligible population, outcome, prediction window, score meaning, reason codes, refresh expectations, owner, permitted actions, and measurement plan. Without that contract, teams will compare incompatible scores and select whichever one confirms their prior belief.

Run the custom component beside the baseline before switching interventions. Inspect coverage, stability, explanations, workflow behavior, and segment differences without changing several parts of the retention program at once. This makes the eventual migration a product decision supported by evidence, not an infrastructure milestone searching for a justification.

Key takeaways

Buy when your immediate need is dependable coverage, rapid activation, and standard integrations for customer success or product-led growth.
Build when proprietary signals, domain-specific risk scoring, specialized explainability, or product differentiation can create material value and you can fund continuing operations.
Blend when you need a working baseline now and have a testable hypothesis about where your data or context can outperform a general-purpose capability.
Do not approve any path until every score has a named recipient, a defined action, a delivery system, and a business outcome.
Compare equivalent total costs, including data work, integrations, monitoring, governance, activation, opportunity cost, and migration.
Measure the model and the intervention separately. Prediction quality can prioritize attention; only an effective action can improve retention.

Take a one-page decision memo into your next review. It should name the churn definition, first population, intervention, deadline, available signals, proprietary advantage, workflow, operating owners, governance constraints, total-cost boundary, and experiment. End the meeting with a selected path and an explicit condition for reconsidering it.

Start with the smallest path that closes the loop from behavior to action to measured outcome. Earn the right to build more by proving that your own data changes the decision and that the decision changes retention.

References

Pendo – Build vs. Buy for Churn Prediction: My Proven Playbook for Faster Retention and ROI

April 16, 2026

From Brain Dump to Done: How Todoist’s Ramble Captures Tasks in Real Time with AI

Turning a rambling stream of consciousness into a clean task list while someone is still talking has been a longtime product dream of mine. With Ramble, Todoist brought that dream to life by using live audio AI to capture tasks in real time—no transcription step required. The result is a voice-to-task flow that feels natural, fast, and surprisingly disciplined.

As I listened to the Doist team—Ernesto Garcia (Front-end Product Engineer), Thomas Jost (Backend Software Engineer), and Hugo Fauquenoi (Product Manager)—walk through their approach, I heard a blueprint for building pragmatic GenAI features. What began as a two-to-three month AI exploration became one of their most technically deliberate releases: a “Gemini-powered pipeline that makes tool calls while the user is still speaking, surfacing tasks on screen in real time without any text output from the model.”

The breakthrough started with user research. People weren’t merely dictating tasks; they were doing a “brain dump” first—often into pen and paper or even ChatGPT voice—and only then committing items to Todoist. Meeting users where they already are reframed the problem: don’t force structure upfront; capture fluid thought and translate it into actionable tasks instantly.

That insight led to a bold architectural choice: skip transcription entirely and process raw audio directly with a Gemini live audio model. By removing the brittle middleman of text, the team reduced latency and kept the model focused on one job—turning intent into structured actions. It’s a crisp example of AI workflows designed for reliability over novelty.

The real magic is in the real-time “tool calls.” As the user speaks, the model triggers add task, edit task, and delete task operations immediately. For high-friction contexts like driving, they paired visual task cards with subtle sound effects as confirmation cues. It’s thoughtful conversation design that respects attention and safety without sacrificing speed.

Teaching the model to capture tasks literally—without over-interpreting or trying to complete the work—required careful prompt engineering for voice and temperature tuning. Drawing a bright line between “capture versus do” kept the experience trustworthy. In my own AI Strategy work, I’ve found that establishing explicit agentic guardrails early prevents unintended autonomy later.

Dates were the sleeper challenge. The team had to inject the current date, normalize to days vs. months, and always output dates in English for the natural language parser—while preserving the user’s original language for everything else. If you’ve ever shipped date handling across locales, you’ll appreciate how many edge cases hide in “Taming Dates and Time.”

Quality didn’t hinge on intuition alone. They built an LLM-judge eval system using real employee recordings from 100+ people across 35 countries in 20+ languages to catch prompt regressions. That’s eval-driven development done right: representative data, repeatable scoring, and tight feedback loops as models and prompts evolve.

For project and label matching, they chose direct context injection over RAG. Instead of building a retrieval pipeline, they injected the full project/label list into the system prompt. With smart context window management and a sharply constrained task schema, this was both simpler and more accurate. Sometimes the fastest path to product-market fit is removing moving parts, not adding them.

One product principle stood out: easy correction beats perfect first-time accuracy. Natural language interfaces earn trust when users can fix misfires in a tap or two. That bias toward quick recovery over false precision is how you ship AI that feels useful from day one.

Looking ahead, the roadmap is compelling: multimodal task capture from images and text blobs, Apple Watch support, and automation integrations. As voice AI agent patterns mature, this “tool-only architecture” sets a solid foundation for going from capture to coordinated execution—without losing the simplicity that makes Ramble shine.

If you want to hear the full conversation, you can listen on Spotify or Apple Podcasts. It’s a masterclass in building focused GenAI features that trade cleverness for clarity—and still delight.

Resources & Links: Todoist • Doist • Google Vertex AI (Gemini)

Inspired by this post on Product Talk.

April 16, 2026

How to Build a Trusted AI Product Platform That Scales

Your teams have AI pilots that work in a demo. Then the questions start. Security wants to know what data the system can reach. Product wants to know whether the answers are dependable. Support wants a fallback when the model fails. Executives want evidence that the investment is changing a customer or business outcome.

You do not need another impressive model response. You need a product platform that makes AI behavior understandable, controllable, and repeatable across use cases. That requires a trust architecture, a path from prototype to production, and metrics that expose failure instead of averaging it away.

Trust fails where an AI output crosses a decision boundary

Most teams discuss AI trust as if it were a property of the model. It is better understood as a property of the whole product system. A capable model can still create an untrustworthy experience if it uses the wrong context, hides a consequential assumption, calls an unauthorized tool, or leaves the user unable to correct an action.

The important moment is the handoff from generation to decision. Before that handoff, the output is a possibility. After it, someone may use it to answer a customer, change a record, prioritize work, or trigger another system. The controls you need depend on what crosses that boundary.

A practical way to classify AI use cases is by the authority you give the system:

Inform: The system summarizes, explains, retrieves, or drafts. A person still interprets the result.
Recommend: The system ranks options or proposes a next action. Its framing can materially influence a decision.
Act: The system invokes tools, changes state, communicates externally, or starts a workflow.

Use mode	Primary trust failure	Required product control	Evidence needed before release
Inform	An incorrect, incomplete, or untraceable answer	Visible scope, supporting evidence, uncertainty, and an easy correction path	An evaluator can reproduce the evidence path and identify known limitations
Recommend	A hidden assumption, weak comparison, or recommendation that ignores the user’s constraints	Explicit assumptions, alternatives, decision criteria, and user-editable constraints	Representative cases show whether the recommendation applies the intended rubric
Act	An unauthorized, excessive, or difficult-to-reverse change	Least-privilege access, previews, confirmation, audit records, and reversal where the underlying system supports it	Authorized reviewers validate simulated actions, denied actions, failure recovery, and a limited production path

This classification prevents a common planning error: giving every AI feature the same review process. A summarizer and an autonomous account-management agent should not pass through identical gates. The second system needs stronger identity, permission, confirmation, and recovery controls because its mistakes can propagate beyond the conversation.

For each proposed use case, ask five questions before discussing a model:

<!– wp:list {

April 15, 2026

Never Lose Your AI Superpowers: How I Sync Context and Skills Across Every Device

I spend a meaningful portion of my week helping teams operationalize AI workflows, and one theme comes up over and over: how to share context files and skills seamlessly across devices and with colleagues. Hosting Claude Code office hours has only reinforced it—sharing context and skills is the single biggest blocker to reliable, repeatable outcomes.

I hear from leaders driving AI adoption who have built robust, high-signal context systems and carefully crafted skills. Their challenge isn’t creating value—it’s distributing it. They need a way to make the same trusted workflows available to teammates and to keep everything in sync across laptops, desktops, and phones.

I hit the same wall myself. I work across multiple devices (a Mac Mini for day-to-day, a MacBook Air on the road, and an iPhone) and I collaborate with a full-time admin. I wanted my context and skills to be consistent everywhere, for both of us. In this piece, I’ll share my setup—what I store where, how I share it across devices and with my team, the trade-offs of each option, and how I keep everything current. We’ll cover four different syncing services: git/GitHub, Obsidian Sync, Dropbox and iCloud.

If you’re new to this series, this is the eighth installment. Earlier pieces provide foundational context: Claude Code: What It Is, How It's Different, and Why Non-Technical People Should Use It; Stop Repeating Yourself: Give Claude Code a Memory; How to Use Claude Code Safely: A Non-Technical Guide to Managing Risk; How to Choose Which Tasks to Automate with AI (+50 Real Examples); How to Build AI Workflows with Claude Code (Even If You're Not Technical); How to Use Claude Code: A Guide to Slash Commands, Agents, Skills, and Plug-ins; and Context Rot: Why AI Gets Worse the Longer You Chat (And How to Fix It).

The day it really hit me was right before my interview with Claire Vo on How I AI. I was staying in an AirBnB with only my laptop, and I planned to demo my /today command along with my context file structure. Minutes before the session, I realized the latest version of my /today command wasn’t on that machine. I was able to remote into my Mac Mini and grab it—crisis averted—but it was a wake-up call. I needed a more reliable, shareable approach for syncing context and skills across devices and with my admin.

I started by testing the tools I already used—Dropbox, iCloud, and GitHub—to see what might fit. Each got me partway there, but each also introduced friction that mattered in daily use.

First, absolute file paths don’t travel well. I began with Dropbox but quickly ran into cross-linking headaches. Good context systems rely on rich interlinking—index files point to other context files, and those context files link to each other. When Claude creates a link from one context file to another, it tends to use the full file path: /Users/ttorres/Library/CloudStorage/Dropbox. That worked on my Mac Mini and MacBook (same user name), but not on my phone—and not for my admin. I tried to force relative links (~/Dropbox), but couldn’t get Claude to do it consistently, which led to broken links. This isn’t unique to Dropbox; Claude prefers full paths because they’re reliable on a single machine, but they’re brittle across devices and useless when sharing with colleagues. Claude is trained to use relative file paths when working within a git repository, but I struggled to get it to work reliably in Dropbox.

Second, skills live in a user directory by default. By default, skills live in ~/.claude/skills. Most sync services aren’t designed to share your ~/ folder. iCloud is the exception, but then you’re limited to Apple devices—no Windows or Android. There is a workaround: set up a claude folder in Dropbox and create a symlink from ~/.claude to your synced claude folder, so all skills, commands, and settings live in Dropbox. Then, on each device (yours or a colleague’s), you set up a symlink to that folder so Claude can find the files. This works, but I was running into another limitation that made Dropbox a poor fit.

Third, Obsidian on iOS doesn’t sync cleanly with Dropbox. I rely on Obsidian’s file browser alongside my notes to navigate context quickly. Storing vaults in Dropbox gave me parity across my Mac Mini and MacBook Air, but I couldn’t get the iOS Obsidian app to reliably load my Dropbox vaults. That friction was a dealbreaker for on-the-go work.

At that point, I explored git/GitHub. GitHub is cloud storage for git repositories. A git repository is a folder of shared files used so engineers can collaborate on the same code base. Each person clones a local copy, works locally, then pushes changes back to the hosted repo on GitHub; others pull to update. Git’s merge and conflict tooling is excellent. Git is the powerhouse of file syncing and version control. It easily handles syncing context and skills, Claude behaves better with relative links in a git repo, and I can open the repo in my IDE with a clean file browser. For me, that checked all the boxes—until I factored in my admin. Git has a learning curve, requires manual pull/push hygiene, and often assumes an IDE workflow. That overhead was too heavy for a non-technical collaborator.

The turning point was Obsidian Sync. A colleague suggested it, and it ended up being the sweet spot. Obsidian is a markdown reader; files are stored locally in a normal folder you can open in Finder or File Explorer. There’s no proprietary format—you can read files with any text editor, and Claude can access them via bash commands. Obsidian Sync is simpler than git: open a note and it syncs in the background. I can access the same vaults across my Mac Mini, MacBook Air, and iPhone, and I can share a vault with my admin so we can both create and access notes.

Because we’re in different time zones and rarely edit the same note simultaneously, limited conflict handling hasn’t been an issue. Obsidian’s internal link notation also means one note can link to another and those links just work across devices. Claude can follow these links, so the brittle file path problem disappears.

Here’s where I landed. After a lot of trial and error, I have a setup that works across my devices and for my admin, who uses both a Windows desktop and a Mac laptop. I keep my core context in Obsidian vaults synced with Obsidian Sync, which preserves portability, link integrity, and ease of use. For skills, I avoid scattering files in machine-specific locations and instead centralize what Claude needs to reference in shared, human-readable folders. If you require advanced version control with branching and reviews, git/GitHub is excellent. If your priority is low-friction, cross-device access for non-technical teammates, Obsidian Sync is a practical, reliable choice. And if you must use Dropbox or iCloud, consider symlinks and be vigilant about relative paths—just know that absolute paths won’t travel well.

Inspired by this post on Product Talk.

April 15, 2026
Cracking the Hardest Percentages: Turn Complex Support into Scalable, Trust-Building Automation

I’ve learned that the smallest slice of your support queue often dictates the majority of your operating cost, customer memory, and automation ceiling. In product reviews and CX ops deep-dives, I see the same pattern: the “easy” tickets pad your resolution counts, but the complex, multi-step queries quietly own your handle time and your brand trust. If you care about compounding impact, your customer support AI strategy has to target that hardest percentage first.

Complex queries are a small percentage of your queue, but they consume a disproportionate share of your team’s time.

Take a typical queue: password resets outnumber refund disputes ten to one, but a reset takes five minutes and a dispute takes thirty. The “rare” query accounts for over a third of total handling time. The same pattern holds for account investigations, subscription changes, and billing disputes.

How you handle complex queries is also what customers actually remember about their support experience. When someone is dealing with a damaged order or a billing dispute, the stakes are higher, and a fast, good resolution is what separates a forgettable interaction from one that builds lasting trust.

Most AI Agents automate the easy, informational queries well. The question for your automation rate is whether they can handle the hard ones. That’s where agentic AI and robust AI workflows make or break your outcomes.

We’ve gotten really good at informational queries – the hard part is what comes next. I’ve seen teams invest deeply here, and for good reason: it lifts containment quickly and cheaply. But to break through the plateau, you have to execute actions across systems, not just answer with text.

We’ve invested deeply in informational Q&A. We built Apex, a specialized customer service model trained on billions of support interactions, as Fin’s core answering engine. Beneath that sits a custom retrieval model, a purpose-built reranker, and a unified RAG pipeline, all trained specifically for customer service. Fin resolves issues at a higher rate than general-purpose frontier models, with fewer hallucinations and at lower cost.

But informational Q&A only covers queries where text is the answer. Most Agents can handle that. Far fewer let you configure complex, multi-step actions without a forward-deployed engineer setting it up for you, which creates a gap.

Every query your team handles falls into one of three categories:

Informational: “Can you ship transatlantic by priority next day?” Answered with text from your knowledge base.

Personalized: “Where is my order?” Requires data unique to that user.

Action-led: “My order arrived damaged, I need a refund.” Requires doing something: checking a return window, cross-referencing transaction data, making a judgment call – reading from multiple systems and acting across them.

From Jan to Apr 2026, the trend moves steadily upward, pausing briefly before a sharp late surge. A clear snapshot of momentum for customer service KPIs, finance results, and the impact of new procedures.

These complex queries, the ones that require multi-step processes across systems, aren’t edge cases; they’re the reason your support team exists. This is the gap Fin Procedures was built to close.

It works in practice, and the trajectory matters for product strategy and ops planning.

Procedures is live, it’s scaling, and the results are clear. Since launching in managed availability, Procedures has handled over 1.5 million conversations, and volume is doubling month over month across hundreds of apps in fintech, e-commerce, gaming, healthcare, and SaaS.

When customers hit complex, multi-step queries, the experience is dramatically better when Fin can do the work end-to-end. We tested this with a randomized 5% holdout – conversations where Procedures would normally run, but didn’t. CSAT was 28.93% higher when Procedures ran, a statistically significant result.

A product, not a services engagement. I’ve sat through too many “automation” projects that were really solutions engineering gigs: workshops, custom scripts, then a queue of change requests when policies shift. It’s fragile and slow.

The B2B AI industry has a consultingware problem. It’s not databases being forked anymore, it’s prompts. The economics of maintaining bespoke setups per customer don’t work. Either the application falls behind new models, or the vendor changes the model and quality degrades invisibly.

In my view, an agentic AI platform should be a product your team owns end to end: a natural language editor – literally paste your existing SOPs – branching logic, data connectors, and AI-powered simulations for testing. Your CX ops team configures this, iterates on it, owns it. If you need help, a forward-deployed team can assist, but they’re optional, not a dependency. You always have control.

And because it’s a unified product, improvement compounds. When the vendor optimizes a prompt, every customer’s Procedures get better. When they upgrade the model, they can A/B test across the entire customer base and know it’s better before rolling out. You can’t do that when every customer has a bespoke prompt. The consulting model isn’t just expensive, it’s structurally unable to compound.

Today, Fin Procedures is available to every Intercom customer – no waitlist or managed rollout, ready for all 8,000+ customers.

We’re iterating fast based on real customer feedback. Here’s what’s landed since the last major update, and why it matters for reliability and governance:

AI-powered Procedure review: Flags broken logic, missing references, and unreachable conditions before you deploy.

Kick off your journey with the #1 Agent—an AI partner designed to turn resolutions into real outcomes. Tap “Start a free trial” to explore faster, smarter customer service and see how Fin delivers value from day one.

Procedure failure reporting: A new reporting dimension that lets you drill into conversations where Procedures failed, so you can diagnose and fix.

Version history with rollback: Track every change, compare versions, roll back if needed.

Data connector health monitoring: See at a glance if your integrations are healthy, degraded, or failing.

Optional data connector parameters: Fin only asks customers for information when it’s actually needed, instead of prompting for every field.

Email Simulation support: Test how your Procedures behave across chat and email before going live.

Agent in the Loop (Beta) unlocks the next tranche of automation. Even with Procedures, two things hold teams back from automating their most complex queries: missing integrations and policies that require a human sign-off on sensitive decisions.

“Agent in the Loop” is built for both. Need Fin to check your internal admin tools but haven’t built a data connector yet? Put a human checkpoint at that step. Fin handles the conversation, gathers context, and pauses, surfacing a structured summary for a human agent to verify or act, then resumes. You get automation on the 80% that doesn’t need the integration.

For compliance – identity verification, high-value refunds – Fin does the legwork, a human makes the final call and then hands it back to Fin. This works natively in the Intercom Inbox and via Slack. Some competitors don’t have an inbox-native variant at all, meaning humans need to leave their primary workspace to review AI actions.

Procedures are also built to let you collaborate with all your teammates – both human agents and AI Agents. Fin can work with them directly inside a Procedure, using APIs and webhooks to loop in another teammate mid-flow, hand off context, and pick back up once they’re done.

Making it easier, faster. Procedures is already self-serve, but the next step is making Procedure creation, testing, and maintenance significantly more streamlined and easy to do, with less manual editing and more AI-assisted building and debugging. There’s a lot coming in this space over the next few months – and it aligns perfectly with a retrieval-first pipeline and stronger governance at scale.

The hardest percentages matter the most. The biggest unlock for your automation rate won’t be answering more FAQs, it will be handling the complex, multi-step queries that consume your team’s time and define what customers remember about their experience with you.

That means working with an Agent that goes beyond answering questions and executes processes. A product your team owns and configures, not a service you buy and hope gets maintained. And a platform where every improvement compounds across every customer. That’s Procedures. Available now, for everyone.

Inspired by this post on The Intercom Blog.

April 14, 2026
Product Work Is Relationship Work: How I Align Stakeholders Faster and Cut Team Politics

Lately, I keep hearing a familiar question: with AI making it so easy to generate ideas and build products, do we still need product managers? My answer is unequivocal—yes. Tools accelerate delivery, but they don’t build trust, reconcile competing incentives, or create the shared understanding teams need to ship outcomes. Product work is relationship work.

I recently listened to “Product Work Is Relationship Work – All Things Product with Teresa & Petra,” and it echoed what I see every day in high-performing product organizations. If you prefer to watch, here’s the episode on YouTube: https://www.youtube.com/embed/d-0f8uAfc8w?feature=oembed

Listen to this episode on: Spotify | Apple Podcasts

While AI can help build things faster, it can’t replace the relationship work required to align stakeholders, navigate competing priorities, and create shared understanding across teams. That’s the hard, human part of product management—and it’s not going away.

In my experience, product teams stall when collaboration becomes transactional. We jump to negotiation (“What can you commit by Friday?”) before establishing context (“What problem are we solving and why now?”). When I slow down to get curious—about constraints, incentives, and assumptions—momentum actually increases because we’re rowing in the same direction.

Stakeholder alignment often breaks down when we conflate advocacy with exploration. We argue our viewpoint as if it were the only lens that matters, rather than making space to surface how others see the system. I’ve found the distinction between “dialogue vs. discussion,” rooted in work by Chris Argyris and elaborated in The Fifth Discipline by Peter Senge, to be a powerful reset. Dialogue builds shared understanding; discussion decides. You need both, in the right order.

Language matters in the room. The improv principle “Yes, and” is deceptively simple but transformative. When a designer, engineer, or executive feels heard (“Yes”) and we build on their idea (“and”), we create psychological safety without sacrificing critical thinking. I use “Yes, and” to explore perspectives before we converge on decisions—especially with product trios and senior stakeholders.

Here are the moves I rely on to keep collaboration relational and outcomes-focused. First, we align on outcomes before solutions. I explicitly separate outcomes vs output OKRs so we’re clear on what success looks like, independent of the features we ship. That clarity reduces rework and speeds up decision-making later.

Second, we operationalize curiosity with continuous discovery. I schedule recurring, lightweight touchpoints with customers and internal stakeholders so insights compound. When learning is continuous, debates quiet down—evidence does the heavy lifting.

Third, we invest in relationship rituals. Regular 1:1s with key partners, stakeholder maps that capture motivations, and pre-reads that frame trade-offs all prevent misalignment from surfacing in the last mile. These small habits pay huge dividends in trust and speed.

Fourth, I’m explicit about mode-switching in meetings: are we advocating a position or exploring perspectives? Calling the mode out loud prevents people from mistaking questions for opposition and keeps the conversation productive.

Fifth, we use “Yes, and” to move from possibility to practicality. We explore generously, then converge rigorously—ranking options by impact, effort, and risk so decisions are transparent and fair.

If stakeholder alignment, team dynamics, or product “politics” slow your team down, this conversation offers a practical reframe. You’ll move faster when you build the relational tissue first—because alignment is an accelerant, not a tax.

Resources & Links:

Follow Teresa Torres: https://ProductTalk.org

Follow Petra Wille: https://Petra-Wille.com

Mentioned in this episode:

Petra’s Coaching Packages

Work by Chris Argyris on organizational learning and dialogue vs. discussion

The Fifth Discipline: The Art and Practice of the Learning Organization by Peter Senge

Improv principle “Yes, and”: Saying “Yes, and” — A principle for improv, business & life and Yes, and …

Have thoughts on this episode or examples from your team? Leave a comment below—I’d love to learn what’s working (and what’s not) in your stakeholder landscape.

Inspired by this post on Product Talk.

April 14, 2026
How We Taught Agentic AI to Speak Product Analytics—and Unlocked Actionable Insights

I set out to solve a deceptively simple problem: help our teams ask product questions in plain English and get trustworthy, analysis-grade answers—fast. That required more than a powerful model; it demanded agents that genuinely understand the language of product analytics, from behavioral analytics nuances to the messy reality of event taxonomies, funnels, and cohorts. In this post, I share how we engineered agentic AI that speaks our domain fluently and turns questions into decisions.

The core challenge wasn’t data volume or dashboard sprawl; it was semantics. Different teams said “activation,” “onboarding,” or “first value” and meant overlapping but distinct things. Our PMs, analysts, and engineers navigated a maze of synonyms across Amplitude analytics, Pendo, and our unified analytics platform. Generic LLMs stumbled on these nuances, so we built a shared ontology—driver trees anchored to a clear North Star—with canonical definitions for activation, retention, and conversion, plus consistent event naming and cohort logic.

We started with a rigorous metric catalog: every KPI linked to its drivers, exact formulas, cohorts, and time windows; every event mapped to a product taxonomy; every dashboard and SQL snippet versioned with ownership and lineage. That catalog became the ground truth for agents. We embedded data governance and privacy-by-design from the start—permissioning for fields and queries, PII redaction, and scoped access that reflected how product teams actually work.

Next, we built a retrieval-first pipeline to ground the agents in our corpus before generation. We indexed metric definitions, dashboards, experiment readouts, runbooks, and high-signal Slack threads so the agent could cite relevant artifacts, not just predict plausible text. With careful context window management and prompt engineering, the agent retrieves definitions and prior analyses, then plans multi-step actions: run a query, compare cohorts, check “minimum detectable effect (MDE)” for an A/B test, and summarize findings with references.

Architecturally, we treated this as “Agent Analytics”: an orchestrator that selects tools based on intent—querying Amplitude analytics or Pendo for behavioral paths and funnels, hitting our warehouse for cohort tables, or pulling experiment metadata and anomaly detection alerts. Tool use is permission-aware, auditable, and designed to fail safe. The agent’s outputs include citations back to the exact definitions, dashboards, and SQL used, so reviewers can validate and iterate.

Quality came from eval-driven development, not intuition. We built a gold set of representative product questions (activation inflections, retention analysis by segment, funnel drop-offs after feature launches) and scored the agent on faithfulness to definitions, numerical accuracy, latency, and actionability. We incorporated regression checks to catch drifts after schema changes, and we tuned prompts to reduce overconfident answers and push for clarifying questions when context was missing.

Safety and reliability were non-negotiable. We layered AI risk management with role-based access, guardrails that block destructive queries, and risk scoring for unfamiliar joins or sudden spikes in metric deltas. The agent logs every step—what it retrieved, which tools it called, and why—so analysts can replay and refine the chain of thought with transparent provenance.

The payoff: product teams now self-serve nuanced questions in minutes instead of days, and our analysts spend more time on discovery than report wrangling. Retention analysis improved as the agent standardized cohort logic; conversion investigations accelerated thanks to consistent funnel definitions; and cross-functional decisions aligned around the same driver trees and shared language. Most importantly, the agent turned ambiguous asks into structured analyses that stand up to scrutiny.

For fellow product leaders, my lesson is simple: start with semantics, not models. A crisp ontology, disciplined taxonomy, and clear ownership will outperform a flashy stack riddled with ambiguity. Avoid technology FOMO; favor retrieval-first grounding, small sharp tools, and continuous discovery with your product trios. When your organization speaks a common analytics language, agents can finally think with you, not just for you.

Next, we’re extending the agent’s planning skills to recommend experiment designs, estimate power and “minimum detectable effect (MDE),” and propose driver-tree-informed bet sizing. We’re also tightening feedback loops so every accepted answer, edit, or override strengthens the retrieval corpus and evaluations. The vision: a calm, reliable layer that makes rigorous product analytics feel conversational—and helps teams move from questions to confident action.

Inspired by this post on Amplitude – Best Practices.

April 13, 2026