Tag: agentic AI

  • From Internal FinOps Agents to Customer-Embedded Optimization

    From Internal FinOps Agents to Customer-Embedded Optimization

    Your cloud-cost agent can identify the line item that moved and still fail to change a single decision. The gap appears after the diagnosis: the recommendation arrives without the product, pricing, ownership, and risk context needed to act.

    If you are taking an internal FinOps capability into the customer experience, design for a closed decision loop. The goal is not autonomous cost cutting. It is a governed system that connects spend to customer value, recommends the next move, and proves whether the move worked.

    Design a decision loop, not another cost dashboard

    Start by naming the decision your product will improve. A broad promise such as optimize cloud spend gives the agent no useful boundary. A better contract is: detect a material change in workload cost, identify the most plausible driver, propose one permitted response, route it to the right owner, and verify the effect.

    Draw the product boundary around an outcome

    The operating loop is simple to describe: observe, explain, propose, authorize, execute, and verify. A dashboard normally stops at observe or explain. An agentic FinOps workflow carries evidence into a recommendation and then closes the loop with an approved action and post-action telemetry.

    Agentic does not mean unrestricted. It means the agent can select the next permitted step based on context. Deterministic services should still perform calculations, enforce policies, check permissions, and execute infrastructure changes. Use the model where interpretation is valuable: reconciling signals, building a driver narrative, identifying missing context, explaining tradeoffs, and routing a decision.

    That distinction matters in FinOps. A model should not improvise a billing calculation, invent a price, or bypass a commitment policy. If a calculation has one correct result, compute it in code and give the result to the agent as evidence.

    Build four layers with explicit responsibilities

    • Evidence layer: Billing exports, usage metering, observability, product telemetry, pricing logic, feature flags, deployment activity, environment metadata, customer segmentation, and ownership records.
    • Reasoning layer: Driver trees, anomaly triage, competing explanations, confidence evidence, and recommendation selection.
    • Action layer: Policy checks, approval routing, change preparation, execution, rollback, and escalation.
    • Learning layer: Post-action telemetry, realized outcomes, agent evaluations, customer feedback, and recurring patterns that belong in the product roadmap.

    A retrieval-first pipeline that combines billing, usage, observability, product, and go-to-market context is more useful than a large prompt containing a monthly cost export. Retrieve the records needed for the current decision and preserve their lineage. Every recommendation should reveal which records were used, when they were updated, which pricing assumptions applied, and what the agent could not retrieve.

    Customer-facing retrieval adds another non-negotiable boundary: tenant isolation must be enforced before context reaches the model. Do not rely on a prompt to prevent cross-customer disclosure. Access control belongs in the retrieval and service layers, with the resulting access decision recorded in the audit trail.

    Start with one anomaly and one reversible response

    Your first release does not need to optimize every cloud service. A practical thin slice is anomaly detection plus one high-leverage remediation path. For example, the agent might detect a change in non-production workload cost, connect it to a schedule change, prepare a schedule correction, request approval from the workload owner, and monitor the next usage window.

    Choose a first action that is bounded and reversible. A scheduling correction is easier to inspect and undo than a long-term financial commitment or a production capacity change. The purpose of the thin slice is to prove the whole operating loop, not merely the anomaly model.

    Make every recommendation safe enough to act on

    A recommendation without an execution envelope is an opinion. It may be correct, but the recipient still has to reconstruct the evidence, find the owner, assess the downside, and decide how to validate it. That is where apparently intelligent systems create more work than they remove.

    Use a recommendation contract

    Treat every agent recommendation as a structured product object. At minimum, require these fields:

    1. Decision: The exact choice the recipient is being asked to make.
    2. Scope: The account, workload, service, environment, and time window affected.
    3. Owner: The person or role accountable for the workload and the person authorized to approve the action.
    4. Evidence: Links to the billing, usage, observability, deployment, and product records that support the diagnosis, including their freshness.
    5. Driver path: The causal chain the agent believes explains the change, plus material alternative explanations it considered.
    6. Proposed action: The change, its expected mechanism, and any assumptions behind an estimated effect. If the effect cannot be estimated reliably, say that it is unknown.
    7. Confidence and unknowns: Evaluation-backed confidence evidence, missing context, and conditions that would invalidate the recommendation.
    8. Execution envelope: Policy checks, blast radius, approver, expiration, rollback procedure, and escalation path.
    9. Verification plan: The telemetry, observation window, success condition, and stop condition used after the action.

    The expiration field is easy to overlook. Cloud state changes quickly enough that an old recommendation can remain plausible after its evidence has gone stale. Expire the recommendation when its pricing, topology, deployment, or usage assumptions are no longer current. Force a fresh retrieval before execution.

    Grant autonomy by action class

    Do not give an agent one global autonomy setting. Earn autonomy independently for each action class:

    1. Observe: Detect and organize a possible anomaly.
    2. Explain: Build a driver tree and expose supporting evidence without proposing a change.
    3. Recommend: Propose an action while a human retains approval and execution.
    4. Prepare: Generate a change plan or dry run, but require an authorized owner to apply it.
    5. Execute within policy: Apply a reversible, bounded action only when the policy engine, permissions, evidence freshness, and rollback checks all pass.

    Purchasing a cloud commitment or altering production resources can create real financial or availability exposure. Keep finance and service owners in the approval path until confidence evidence and post-action telemetry demonstrate reliable performance for that specific intervention. Good results on anomaly explanations do not establish that the same agent is safe to execute infrastructure changes.

    Governance should be visible in the product, not left in a policy document. Show the approver which data was accessed, which rules passed, who changed the recommendation, what action ran, and what happened afterward. Privacy-by-design, data controls, and transparent decision logs are part of the user experience when the system influences money and production infrastructure.

    Evaluate the decision loop, not the prose

    A polished explanation is not evidence of a useful agent. Build evaluations around the failure modes that can block or distort a decision:

    • Did the recommendation use the correct customer, workload, environment, price, and time window?
    • Can each material claim be traced to an underlying record?
    • Does the driver path match known cases, including cases with several plausible causes?
    • Does the agent abstain when ownership, telemetry, or pricing context is missing?
    • Did approval routing and policy enforcement behave correctly?
    • Can the recipient perform the proposed action without reconstructing missing steps?
    • Did post-action telemetry confirm the expected direction of change without creating an unacceptable operational tradeoff?

    Put retrieval changes, prompts, policies, and tools through the same delivery discipline as application code. Eval-driven development, CI/CD, and a weekly shipping cadence make regressions visible before a persuasive but poorly grounded recommendation reaches an operator or customer.

    Embed the capability with customers before scaling it

    The first customer version should not be a general-purpose cost chatbot. It should be a narrow, product-assisted engineering motion in which a Forward Deployed Engineer, or FDE, helps the customer connect product usage, cloud architecture, and cost-to-value.

    Choose a small pod and customers that can teach you

    A sensible starting shape is one FDE pod focused on two or three high-potential customers. High potential should not mean merely the largest cloud bill. Select customers where the team can access the necessary evidence, an accountable sponsor can authorize changes, the problem is likely to recur, and the customer agrees to clear data and governance boundaries.

    • Evidence readiness: Billing, metering, observability, pricing, and deployment context can be joined without weeks of manual reconciliation.
    • Decision access: An engineering, product, or finance owner can approve an intervention and explain the operational constraints.
    • Learning value: The problem represents a pattern that may apply beyond one account.
    • Measurability: The customer and FDE can agree on a cost-to-value measure before making a change.
    • Governance fit: Data access, retention, tenant isolation, approvals, and audit expectations are explicit.

    If any of these conditions is absent, the engagement may still be commercially important, but it is a weak environment for deciding whether the agentic product works. Separate account urgency from product-learning quality.

    Run a customer optimization loop that produces reusable knowledge

    1. Define the value unit. Agree on what an active workload or valuable unit of product usage means. Total spend alone cannot distinguish efficient growth from contraction.
    2. Establish the baseline. Record current cost per active workload, time-to-first-value, relevant deployment behavior, and the constraints the customer will not trade away.
    3. Build the driver tree. Connect the spend change to services, environments, releases, product behavior, and customer usage. Surface gaps instead of filling them with assumptions.
    4. Select one intervention. Prefer the smallest action that can test the diagnosis. Document the expected mechanism, approver, risk, and rollback before execution.
    5. Verify the outcome. Compare post-action telemetry with the agreed baseline. Record savings, unit-economics movement, performance effects, adoption effects, and unintended consequences separately.
    6. Codify the pattern. Capture the inputs, decision rule, action, exceptions, safeguards, and evidence required to repeat the intervention.
    7. Send a weekly learning packet to product. Include successful patterns, failed diagnoses, missing platform capabilities, customer language, and recommendations that still depend on FDE judgment.

    Within a quarter, this loop should make it possible to distinguish interventions that can be automated, patterns that should become native product features, and problems that still require deeper solutions engineering. The point is not to eliminate the FDE. It is to reserve that scarce judgment for cases where ambiguity and customer context remain material.

    Make the commercial incentive legible

    Customer-embedded optimization creates an obvious trust question for a consumption business: does the vendor want the customer to spend less or consume more? The clean answer is to optimize cost-to-value rather than either number in isolation.

    A customer’s total cloud cost can rise while cost per active workload improves because valuable usage is growing. Total cost can also fall because the customer is using less of the product, which is not an optimization success. Label the outcome precisely: lower total spend, lower unit cost, avoided waste, shifted commitment, higher useful consumption, or reduced operational risk. Do not collapse these different effects into a generic savings claim.

    The FDE is also a trust boundary. The role should explain the recommendation, expose assumptions, and represent the customer’s constraints. It should not become a human interface for repetitive exports and one-off queries that the platform ought to handle.

    Turn field work into a roadmap, not permanent custom service

    A strong FDE can make a weak product look successful by solving every gap manually. That is useful for an individual customer and dangerous for product strategy. You need an explicit test for moving work from the field into an agent workflow or native platform capability.

    Apply a productization test to every recurring intervention

    1. Can the same signal be retrieved reliably across the intended customer segment?
    2. Can the decision logic be expressed without undocumented customer-specific knowledge?
    3. Can the action be bounded by a stable policy, approval path, and rollback procedure?
    4. Can the outcome be measured with telemetry that exists before and after the change?
    5. Do the likely exceptions fit a review workflow, or do they fundamentally change the decision?

    If the signal, decision, action, and measurement are repeatable, make the pattern a native feature or automated playbook. If the evidence is repeatable but judgment varies, keep an agentic workflow with human review. If the action carries high financial or availability risk, keep the FDE and accountable owner in the loop. If the pattern is a one-off, document it but resist turning it into product scope.

    Use a scorecard that reveals where the loop is breaking

    DimensionMeasureDecision it informs
    Insight speedTime-to-insight from a material spend changeIs the system finding the issue early enough to change an engineering decision?
    Action qualityRecommendations with evidence, an owner, a permitted action, and a verification planIs the agent producing executable decisions or polished commentary?
    EconomicsRealized savings per recommendation and cost per active workloadDid the intervention improve spend or unit economics for the intended value unit?
    ReliabilityPost-action effects, abstentions, rollbacks, and policy failures by action classWhich interventions have earned more autonomy, and which need tighter controls?
    Customer outcomeTime-to-first-value and NRR movement on FDE-supported accountsIs the motion improving adoption and durable account value? NRR is directional evidence, not proof of causation.
    Product leverageRecurring field patterns converted into features, guardrails, or in-product guidanceIs customer work compounding into a scalable product?

    Recommendation volume, prompt length, and agent activity are operating diagnostics, not business outcomes. A quiet system that changes a few high-value decisions can be more useful than an active system that produces hundreds of unactioned findings.

    Make build versus buy a component decision

    Do not treat the choice as one monolithic platform decision. Separate commodity capabilities from the context and workflow that create differentiation. Evaluate billing ingestion, normalization, anomaly detection, the context model, pricing logic, recommendation policy, approval routing, execution, and agent analytics independently.

    • Does the capability require knowledge of your architecture, pricing model, feature flags, customer usage, or deployment behavior?
    • Can an external component preserve evidence lineage, tenant isolation, and decision logs at the level your customers require?
    • Is the capability a generic input to the product, or is it where your product makes a differentiated decision?
    • Can your team evaluate and operate the component continuously, including regressions after model, prompt, policy, or data changes?
    • Will the component reduce time-to-value without trapping critical customer and pricing context in an opaque workflow?

    Unique architecture, pricing, and growth loops can justify building the context and decision layers. But weak tagging, unclear ownership, and missing observability undermine either path. Fix those foundations before expecting an in-house or purchased agent to produce precise optimization decisions.

    Give the core product to a product trio spanning product management, engineering, and FinOps. Bring FDE, customer success, SRE, finance, and security into discovery and evaluation where their decisions are affected. Field requests should enter the roadmap with evidence of recurrence, strategic importance, or platform leverage rather than becoming an informal side door to custom development.

    Key takeaways

    • Define the product as observe, explain, propose, authorize, execute, and verify. Diagnosis alone is not an agentic outcome.
    • Retrieve billing, usage, observability, pricing, product, and ownership context for each decision, with lineage and tenant boundaries enforced outside the prompt.
    • Represent every recommendation as a governed contract containing evidence, owner, action, risk, approval, rollback, expiration, and verification.
    • Grant autonomy by action class. Keep humans in the loop for commitments and production changes until that intervention has reliable post-action evidence.
    • Start customer delivery with one FDE pod and two or three customers that offer evidence access, decision access, measurable value, and reusable learning.
    • Measure time-to-insight, realized outcomes, unit economics, reliability, customer value, and productized patterns instead of counting recommendations.

    This week, choose one recurring cost anomaly and map the complete path from underlying records to a verified action. Name the owner, approval rule, rollback, and success telemetry before improving the prompt. Do not add a second workflow until the first can explain what changed, why the action was allowed, and whether it improved customer cost-to-value.

    References

  • How to Run AI-Assisted Feature Launches That Drive Growth

    How to Run AI-Assisted Feature Launches That Drive Growth

    You are days from releasing a feature. Engineering needs a rollout decision. Go-to-market teams need a clear promise. Support needs to know what could go wrong. Leadership wants to know whether the release changed customer behavior. Dropping an AI bot into the launch channel will not resolve those tensions. If the metrics, authority, and escalation rules are vague, the bot will only answer ambiguity faster.

    The useful model is a closed loop: define the behavior you want to change, instrument exposure and value, operate the rollout from one shared channel, let agents handle repeatable retrieval and synthesis, and reserve consequential decisions for accountable people. Done well, AI reduces the coordination tax around a launch while making the growth decision more disciplined.

    Define the growth decision before you automate the launch

    A feature being available is an output. A customer reaching value is an outcome. Your launch plan has to connect the two before anyone writes an agent prompt or schedules a readout.

    A durable growth plan translates the product North Star into activation and retention signals, then defines the minimum detectable effect before experimentation. The North Star provides direction, but it is often too distant to diagnose a new feature. A launch needs an earlier behavioral signal that can tell you whether eligible users encountered the feature, understood it, and reached its intended value.

    Write a short launch contract with these fields:

    1. Target user and moment: Name the user or account segment, the situation that makes the feature relevant, and any eligibility rules. A feature intended for a new administrator solving an initial setup problem should not be evaluated across every user in the product.
    2. Behavioral hypothesis: State the current behavior, the desired behavior, and why the feature should cause the change. If the causal link cannot be written plainly, the team is not ready to interpret the launch data.
    3. Measurement chain: Instrument eligibility, actual exposure, meaningful engagement, the activation action, and the downstream value event. If you record engagement but not exposure, low adoption could mean either that users ignored the feature or that they never saw it.
    4. Primary signal: Choose the behavior closest to customer value that can mature within the launch window. Do not promote every available metric to equal status. That turns a decision into a search for whichever chart looks most favorable.
    5. Guardrails: Name the operational and customer signals that can stop a rollout, such as degraded performance, errors, support burden, privacy concerns, or a harmful shift in another important behavior. Define the actual acceptable bounds in your internal contract before launch; do not negotiate them after a concerning result appears.
    6. Minimum detectable effect: Decide what change would be large enough to matter to the product and business. This keeps the team from celebrating meaningless movement or waiting indefinitely for certainty that the planned test cannot provide.
    7. Decision rule and authority: Specify what evidence permits a ramp, what requires a hold, what triggers investigation, and who can pause or roll back the feature. An agent may assemble the evidence, but it should not invent the rule during the incident.

    The contract should also distinguish a growth signal from a health signal. Activation, conversion, or repeated use may tell you whether the feature is producing value. Latency, error rates, complaints, and anomalous segment behavior tell you whether it is safe to continue. A healthy system with an immature growth signal may justify holding the rollout. Broken instrumentation or a material guardrail breach calls for a different response.

    This distinction prevents a common category error: treating an inconclusive experiment as a failed feature, or treating early adoption as proof of durable value. The launch decision should always answer the same question: given trustworthy exposure data, the primary signal, and the guardrails, should you ramp, hold, investigate, or roll back?

    Turn the launch channel into a decision system

    A launch channel becomes useful when it preserves context and decisions, not merely conversation. A practical setup is one channel named #launch-[feature], with its scope, service expectations, success metrics, dashboards, and rollout plan pinned. Product, engineering, data, support, and go-to-market stakeholders can then work from the same operational record.

    Set up the channel before rollout begins:

    1. Pin the launch contract: Include the hypothesis, eligible population, event definitions, primary signal, guardrails, rollout stages, owners, and links to live dashboards. A screenshot becomes stale; a governed dashboard remains inspectable.
    2. Create stable work lanes: Use separate parent threads for metrics, incidents, enablement, and customer feedback. This gives each agent and human responder a predictable place to work without fragmenting the overall launch record.
    3. Publish response expectations: State which questions the agent can answer immediately, which require a human owner, and how urgent operational issues are escalated. The agent should never make an urgent request look handled merely because it produced a fluent reply.
    4. Keep a decision ledger: For every ramp, hold, pause, or rollback, record the timestamp, evidence considered, decision, rationale, approver, and next review point. This matters later when a stakeholder asks why exposure changed or when the team compares the result with the original hypothesis.
    5. Require channel-visible handoffs: If a question moves to a data, engineering, or privacy owner, the agent should post the handoff and preserve the relevant query, definitions, filters, and context. Do not let direct messages become a shadow operating system.

    Give every automated data answer a consistent shape:

    • The direct answer, including the population and time window.
    • The metric definition and denominator.
    • The relevant cohort, segment, environment, and experiment variant.
    • A link to the approved underlying data or dashboard.
    • An as-of timestamp so readers know how fresh the result is.
    • Any missing data, definition conflict, or limitation that changes interpretation.
    • The named human owner when judgment or investigation is required.

    An activation rate without its denominator, an environment, or a timestamp is not decision-grade evidence. A polished answer should not receive more trust than the data lineage beneath it. Make uncertainty visible instead of prompting the agent to conceal it behind a confident summary.

    Give agents narrow jobs and humans explicit authority

    The safest launch architecture separates three jobs: retrieving data, operating rollout controls, and interpreting evidence. Combining them in one broadly empowered agent creates unnecessary risk. It also makes failures harder to diagnose because you cannot tell whether the problem came from a bad query, a bad recommendation, or an unauthorized action.

    Use a data agent for retrieval and first-pass synthesis

    Connect the data agent only to approved sources and metric definitions. It can answer repeatable questions such as activation by cohort, conversion by segment, latency by region, exposure by variant, or the movement of a named guardrail. It should provide citations and timestamps, then route questions requiring nuance to an owner while keeping the context in the thread.

    Write the escalation boundary into its operating instructions. Escalate when metric definitions conflict, required data is unavailable, a query touches restricted information, the request asks for a causal conclusion that descriptive data cannot establish, or the answer would materially change rollout. The best response in those cases is not a guess. It is a precise statement of what is missing and who must resolve it.

    Keep the feature-flag agent read-only by default

    A flag agent can safely expose status by environment, current rollout allocation, and change history. That alone removes many repetitive questions. Write access is different: an incorrect production change can expose an unready experience, expand an incident, or remove access unexpectedly.

    When you permit flag mutations, require an explicit sequence:

    1. The requester names the feature, environment, target population, requested action, and reason.
    2. The agent shows the current flag state and summarizes the evidence relevant to the request.
    3. The authorized approver confirms the exact change. Approval cannot be inferred from an emoji, an ambiguous reply, or the agent’s own recommendation.
    4. The integration performs only the approved action through constrained permissions.
    5. The agent posts the resulting state, timestamp, requester, approver, rationale, and change-history link.

    Do not give the agent a broad production credential merely because the chat interface is convenient. Restrict its access by environment and role, preserve an audit trail, and keep a manual rollback path available to the responsible engineer.

    Use a readout agent to maintain the launch narrative

    Scheduled summaries prevent the team from rebuilding the same analysis for every stakeholder. A useful default is to publish readouts at T+1 hour, T+24 hours, and T+7 days, while adapting the questions to the product’s actual usage cycle:

    • T+1 hour: Confirm that exposure is occurring as intended. Check instrumentation, operational performance, obvious anomalies, and incident status. This checkpoint is primarily about measurement and safety, not declaring growth success.
    • T+24 hours: Review adoption and activation by the planned cohorts, early conversion movement where applicable, support themes, and any uneven behavior across important segments.
    • T+7 days: Evaluate experiment results that have had time to mature, retention or repeated-value signals when the product cycle makes them observable, significant outliers, and the follow-up work needed to harden or revise the experience.

    These checkpoints are operating cadences, not guarantees of statistical maturity. A feature used on a longer cycle may not produce a meaningful retention signal by the final checkpoint. The readout should say that plainly instead of treating missing maturity as neutral evidence.

    Every readout should end with a decision or an explicit statement that no decision is yet warranted. It should also name the evidence still needed, the owner, and the next review point. A summary that lists charts but does not clarify the decision state creates more reading without reducing uncertainty.

    Make the accountability map visible

    • Product owns the behavioral hypothesis, the primary growth signal, and the recommendation to ramp, hold, or change direction.
    • Engineering owns operational health, flag implementation, incident response, and safe rollback execution.
    • Data owns metric definitions, instrumentation validity, experiment design, and interpretation limits.
    • Support and go-to-market owners contribute customer feedback, readiness concerns, enablement status, and communication needs.
    • Agents retrieve, summarize, route, and perform narrowly preauthorized steps. They do not approve their own consequential recommendations.

    The governance layer is part of the product, not a final compliance check. Apply role-based access, protect personally identifiable information, require source citations, and retain transparent logs. Then monitor response accuracy, deflection, and time-to-answer through Agent Analytics. Deflection alone is a poor success metric: a confidently wrong response may reduce human questions while increasing decision risk. Review incorrect answers, unnecessary escalations, missed escalations, and stale data as carefully as response speed.

    Run the rollout as a sequence of evidence gates

    A feature flag is not merely a switch. It lets you separate deployment from exposure and turn a large release decision into a sequence of smaller, inspectable decisions. The appropriate rollout stages depend on the feature’s operational, privacy, and customer risk, so define them in advance rather than copying a universal percentage.

    Use this operating sequence:

    1. Preflight the measurement: Verify eligibility, exposure, activation, value, and guardrail events in the intended environment. Confirm that dashboards use the launch contract’s definitions and that the agent can retrieve the same governed numbers.
    2. Release to the defined cohort: Use the flag to control who can receive the experience. Confirm actual exposure before interpreting engagement. Eligibility and exposure are different facts.
    3. Inspect evidence at the scheduled gates: Start with instrumentation and safety, then move to activation, conversion, retention, and other downstream value signals as they become observable. Review the preselected segments before exploring unexpected cuts of the data.
    4. Choose a named decision state: Ramp, hold, investigate, pause, or roll back. Record the evidence and rationale. Avoid vague states such as looking good, because they do not tell engineering what to do or stakeholders what has been decided.
    5. Feed the learning back into the journey: Update onboarding, in-product guidance, targeting, positioning, or the feature itself based on the observed friction. A winning test becomes a growth mechanism only when the trigger, experience, and value-producing behavior can be repeated reliably.

    Use a clear decision ladder:

    • Ramp when measurement is trustworthy, guardrails remain inside the pre-agreed bounds, the evidence meets the decision rule, and customer-facing teams are ready for broader exposure.
    • Hold when the system is healthy but the outcome has not matured enough to support a decision. State what evidence is pending and when it can reasonably be reviewed.
    • Investigate when an anomaly, segment divergence, definition conflict, or instrumentation gap makes the aggregate result unreliable.
    • Pause when continued exposure could obscure an incident, contaminate the test, or expand a customer problem while the team diagnoses it.
    • Roll back when a material operational, privacy, safety, or customer guardrail crosses the boundary defined in the launch contract. Do not wait for the primary growth metric to mature before acting on a serious downside.

    If the feature itself uses AI, measure the product experience separately from the operational agents supporting its launch. AI can provide intelligent nudges, next-best actions, and adaptive experiences while applying privacy-by-design and strong data governance. That creates at least four distinct questions: Was the user eligible? Was the AI experience delivered? Did the user engage with it? Did that engagement lead to value without violating a guardrail?

    Logging only the final conversion makes those questions impossible to separate. A delivery problem, poor recommendation, confusing interaction, and weak value proposition can all produce the same downstream result. Preserve the path from eligibility through value, including the experience variant the user received. If targeting or adaptive behavior changes during the test, log the change and account for it in the interpretation.

    Do not confuse high initial use with a durable growth loop. Novelty can produce engagement without retained value. Look for the sequence that matters to your product: activation, repeated value, and then the relevant expansion, collaboration, or retention behavior. If the product has no natural invitation or sharing mechanic, do not force a viral story onto it. Build the loop around the behavior customers already have a reason to repeat.

    Key takeaways

    • Start with a launch contract that names the user, behavioral hypothesis, measurement chain, primary signal, guardrails, minimum detectable effect, decision rules, and accountable owners.
    • Use one launch channel as the shared operational record, but separate metrics, incidents, enablement, and feedback into stable threads.
    • Split agent responsibilities across data retrieval, flag operations, and scheduled readouts. Keep consequential actions approval-gated and auditable.
    • Treat T+1 hour, T+24 hours, and T+7 days as decision checkpoints, not automatic declarations of success.
    • Use feature flags to move through evidence gates. Ramp, hold, investigate, pause, or roll back according to rules written before the data arrives.
    • Measure AI-powered experiences from eligibility through delivery, engagement, value, and guardrails so you can diagnose why growth did or did not move.

    For your next launch, begin with a narrow operating slice: a completed launch contract, a structured channel, a data agent limited to approved queries, read-only flag visibility, and scheduled readouts. Review every wrong answer, escalation, and decision after the rollout. Expand the agent’s authority only when the evidence shows that the control system is trustworthy.

    References

  • Outcome-Based Pricing That Delivers: Pay $10 Only for Qualified Leads with Fin for Sales

    Outcome-Based Pricing That Delivers: Pay $10 Only for Qualified Leads with Fin for Sales

    Our outcome-based pricing model hinges on one principle: you pay when Fin delivers value.

    As Fin takes on new roles, that principle doesn’t change, but the definition of value does.

    Fin for Sales qualifies leads, engages prospects, and routes high-intent buyers to your sales team. The value it creates isn’t a resolved query, but a pipeline of qualified opportunities. So we price accordingly: $10 per qualified lead. And you, the customer, define what “qualified” means, not Fin.

    This is the first outcome-based pricing model for an AI Agent for sales. Here’s why I believe it’s the right approach and how I’ve seen it change the way teams think about SaaS pricing and ROI.

    Over the years, I’ve learned that the fastest way to earn trust with sales and finance leaders is to align pricing with outcomes they actually report on. The core finding from our research was unambiguous: zero buyers preferred paying for activity. They wanted to pay for results.

    That insight shaped how we priced Fin for its service role, $0.99 per resolution, where a resolution means the customer’s issue is fully solved without human intervention. More recently, we evolved that model to outcomes, reflecting the broader ways Fin delivers value across complex workflows. We believe pricing should be aligned with value delivery, and the vendor should carry risk when the product doesn’t perform. In sales, the best unit of value is pipeline.

    Most sales teams today are overwhelmed by leads. Early in my career, I watched reps spend hours chasing form fills that looked promising but went nowhere. That experience cemented a lesson I still use: volume is vanity; qualification is sanity.

    Ensuring the right opportunities promptly reach your sales team is what makes a difference. When a prospect visits your site, engages with Fin, answers qualifying questions, and is directed to a sales rep, Fin is identifying whether the opportunity is worth your team’s time and delivering value.

    Charging per conversation would penalize businesses for every curious visitor who asks a question but isn’t a buyer. And charging per token, well, that’s always been a model that protects the vendor, not the customer.

    We needed a metric that captures the actual value Fin creates in a sales context: qualified leads.

    The purest version of outcome-based pricing for Fin’s sales role would be a percentage of closed revenue. Fin qualifies the lead, a rep closes the deal, and we take a cut. On paper, it looks elegant; in practice, I found it breaks down for two reasons that matter to operators.

    First, attribution. Between the moment Fin qualifies a lead and the moment a deal closes, dozens of things can impact the final result. The quality of human-led demos can differ, products can have outages, prospects’ budgets can get cut. Tying Fin’s price to the final outcome holds it accountable for variables entirely outside its control.

    Second, measurement. To track closed revenue, we’d need deep integration into every customer’s CRM, tracking each opportunity from qualification through to close. That’s a significant implementation burden that slows time to value, which is the opposite of what we want.

    So we asked: what’s the most honest proxy for the value Fin delivers, where Fin is clearly the one creating it?

    A qualified lead is that proxy. It represents the moment Fin has done its job. It has engaged the prospect, gathered the relevant information, evaluated them against your criteria, and determined they’re qualified. Everything up to that point is Fin’s work. Everything after it is the rep’s. At $10 per qualified lead, the pricing reflects this boundary.

    There are two key components to how this pricing model works.

    First, the customer defines success. With Fin’s sales role, the customer sets their own qualification criteria based on their business context. A company with high average contract values might set a lower bar because they can’t afford to miss anyone. A company where rep time is scarce and deal sizes are smaller might set a much higher bar, filtering aggressively to only surface the most promising prospects. The criteria flex to match the business.

    Second, the economics are different by design. As a Customer Agent, Fin can switch between roles like sales and service. So if you’ve deployed Fin for Sales, it can still handle support queries like prospects asking a product question. Those queries are charged at $1 per resolution, consistent with our service pricing. Disqualifications, where Fin determines a prospect doesn’t meet the criteria, are also $1. The $10 price point for qualified leads reflects the higher value of pipeline creation compared to issue resolution.

    The ROI speaks for itself. Early customers are reporting significant returns using Fin for Sales. One shared a perspective that mirrors what I hear in executive QBRs:

    “I would say it’s at least 10 times the value. You’re now giving the business exactly what it needs as opposed to just activity. We say this expression in sales leadership all the time – ‘I don’t pay my sales team for activity. I pay them for results.’ I want my AI engine to be the same way.”

    When you compare the cost of a qualified lead from Fin against the fully loaded cost of an SDR—salary, benefits, tooling, ramp time—the economics are compelling. For many businesses, particularly those that never had SDRs in the first place, Fin for Sales isn’t just replacing headcount, but creating an entirely new capability that wasn’t economically viable before.

    This pricing model came from extensive customer research—qualitative interviews and quantitative studies—exploring how buyers want to pay for AI in a sales context. We tested multiple concepts: per-conversation, per-token, per-seat, revenue share, and per-qualified-lead. The research consistently pointed to outcome-aligned pricing as the preferred model, with the qualified lead emerging as the metric that best balances value alignment, measurability, and practical implementation.

    Outcome-based pricing is still rare in AI, but we think that will change. For Sales Agents, we’re the first to do it. Transparency is part of the model. If you understand why we price the way we do, you can evaluate whether it works for your business.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • How to Evaluate a Shopify-Native AI Shopping Agent

    How to Evaluate a Shopify-Native AI Shopping Agent

    You’ve probably been asked a deceptively simple question: should your Shopify store add an AI shopping agent? The hard part isn’t installing another chat widget. It is deciding whether the agent can help an uncertain shopper choose correctly without recommending unavailable products, misreading policy, or making a costly order change.

    Treat this as a commerce-system decision, not a chatbot decision. A useful agent must connect conversation, live product data, cart behavior, checkout, and post-purchase service. The evaluation framework below will help you separate a persuasive demo from a system you can trust with customers and revenue.

    The native test: can the agent read, reason, and act?

    “Shopify-native” should describe an architecture, not a distribution channel. Being listed in an app marketplace or embedded in a storefront does not make an agent native. The meaningful test is whether Shopify remains the operational source of truth while the agent uses its data and APIs in the customer’s current context.

    A concrete implementation shows how high that bar can be: a Shopify connection can expose products, variants, content, live inventory, order data, policies, and transactional APIs to the same customer-facing agent. That combination matters because a correct product description is still a bad answer if the relevant size is unavailable, and an accurate return policy is insufficient if the customer must start over somewhere else to use it.

    I would evaluate an agent at four capability levels. A product that stops at the first level may still be useful, but it should not be presented internally as an autonomous commerce agent.

    CapabilityWhat the agent needsWhat you should ask it to prove
    AnswerProduct content, store policies, and current catalog factsAnswer a precise product or policy question and identify the relevant item, variant, or rule
    RecommendCatalog relationships, inventory, conversation context, and the shopper’s constraintsTurn an ambiguous request into a short, reasoned shortlist instead of returning generic search results
    TransactCart or order APIs, authentication, permissions, and confirmation controlsUpdate a test cart or prepare an order change while showing exactly what will happen before execution
    RecoverShared state across shopping, service, and human escalationResolve a support interruption and resume the customer’s original shopping task without asking for the same context again

    Freshness is part of correctness. During a test, change the availability of a variant in Shopify and repeat the same shopping request. The agent should stop recommending that variant once the change is reflected in the connected system. Run a similar test with a policy update. A polished answer based on yesterday’s state is not a small quality defect; it can create a promise your operations team must later unwind.

    Actions deserve an even stricter test. Ask the vendor to demonstrate the complete chain: customer identification, authorization, interpretation of the request, action preview, explicit confirmation, API execution, and a visible result. If any step is simulated, ask which one. A fast setup can reduce implementation effort, but it does not prove that the agent is accurate, observable, or safe.

    Design the shopping dialogue around decisions, not keywords

    Traditional ecommerce search works best when the customer already knows the product vocabulary. A shopping agent earns its place when the request is incomplete: a gift for a partner, a mattress for a particular sleep preference, or shoes that must work across road and trail conditions. The agent’s first job is not to produce an answer. It is to discover which answer would be useful.

    A strong product-discovery dialogue follows a repeatable decision sequence:

    1. Restate the customer’s job in plain language so a misunderstanding becomes visible early.
    2. Identify hard constraints first, such as an in-stock variant, required use case, compatibility, budget boundary, or delivery requirement.
    3. Ask only for information that could change the recommendation. A question that does not alter ranking or eligibility is conversational overhead.
    4. Present a small shortlist and tie each option to the constraints the customer supplied.
    5. Explain the meaningful tradeoff between the options instead of declaring a universal winner.
    6. Offer the next useful action: compare details, select a variant, update the cart, or continue narrowing the choice.

    This sequence turns conversation design into a product requirement. For every recommended item, your evaluation record should capture the customer’s stated need, the product facts used, the reason the item fits, the tradeoff disclosed, current variant availability, and the next question or action. That record gives your team something concrete to inspect when a recommendation is challenged.

    Test whether the reasoning is responsive rather than decorative. Change one important answer while holding the rest of the conversation constant. If the customer switches from occasional use to daily use, removes a budget constraint, or requires an available color, the shortlist should change when that information is material. If the products remain identical regardless of the customer’s answers, the experience is probably search with conversational packaging.

    The agent should also know when not to narrow further. Once the customer has enough information to choose, another question adds friction. Conversely, confidence should not be manufactured when catalog data cannot resolve the request. A safe response identifies the missing fact, asks for clarification, or hands the conversation to a person with the constraints already summarized.

    Product cards can accelerate the final step, but the interface should preserve the reasoning that produced them. An image, name, and price answer “what is this?” The conversation must also answer “why does this fit me?” That is the difference between displaying inventory and assisting a decision.

    Make shopping and support one customer state machine

    A shopper does not experience your sales and support departments as separate funnels. The same person can compare products, ask about shipping, check an existing order, correct a variant, and return to buying in one session. Routing each intent to a separate tool forces the customer to reconstruct context at every boundary.

    Model the journey as one state machine: discover, decide, transact, service, and resume. The agent can move between those states, but it should retain the customer’s goal, constraints, products considered, cart state, relevant order, completed actions, and unresolved question. That shared state is more important than whether the organization labels the current message “sales” or “support.”

    This is where a connected agent can do more than answer FAQs. Current Shopify-oriented implementations can handle tracking, returns, exchanges, refunds, order changes, shipping questions, and subscription updates through connected procedures and APIs. Each additional action increases usefulness, but it also increases the consequence of a misunderstanding.

    Use different controls for different action classes:

    • Read-only actions, such as showing order status, should still require appropriate customer identification but do not change commercial state.
    • Reversible shopping actions, such as adding or removing a cart item, should be immediately visible and easy for the customer to undo.
    • Financially consequential actions, including refunds, paid order changes, and subscription updates, should require authentication, an exact action summary, explicit confirmation, and a durable result or receipt.
    • Ambiguous or unsupported actions should stop safely and transfer to a person. The agent must not treat conversational enthusiasm, silence, or an inferred preference as consent.

    That last distinction protects both the customer and the business. A mistaken recommendation can usually be reconsidered. An executed refund or subscription change creates financial and operational consequences. If the system cannot preview and verify the exact action, keep that workflow read-only and let a trained person execute it.

    The transition back to shopping also needs deliberate design. After resolving a delivery problem or order correction, the agent should restore the prior context and offer a relevant path forward. It should not force an upsell into every service interaction. The next best action after a serious order problem is often confirmation that the problem is resolved. Commercial momentum comes from reducing friction, not from ignoring the customer’s immediate priority.

    When escalation is necessary, pass a structured handoff rather than a transcript dump. Include the detected intent, verified identity state, constraints already collected, products or orders involved, actions attempted, results returned by Shopify, and the unresolved decision. A human agent should be able to continue with the next question, not repeat the first one.

    Measure incremental commerce value and operational risk

    Chat conversion is an attractive metric and an easy one to misread. People who open a shopping conversation may already have higher intent than people who do not. Comparing those groups directly can credit the agent for demand it did not create.

    Ninja Transfers reported that 10% of its conversations converted to orders with values 20% above the store’s average order value. That is a useful customer result, but it is a vendor-supplied case from one merchant, not a universal benchmark or proof that the agent caused the full difference. Your business case should depend on your own incremental test.

    Where traffic permits, randomize eligible storefront sessions between an agent experience and the existing experience. Measure the result across all eligible sessions, not only the visitors who choose to chat. That intention-to-treat view reduces self-selection bias and answers the executive question: what changed when the store made the agent available?

    Use a balanced scorecard rather than a single conversion target:

    • Business outcomes: completed orders per eligible session, revenue per eligible session, average order value, checkout completion, and assisted revenue.
    • Decision outcomes: recommendation engagement, product-detail visits after a recommendation, add-to-cart actions, and successful comparison flows.
    • Downstream quality: cancellations, returns, exchanges, and contacts caused by a poor recommendation or incorrect expectation.
    • Service outcomes: successful action completion, repeat contact for the same problem, human escalation, and time to a confirmed resolution.
    • Agent quality: use of current catalog facts, in-stock recommendation rate, policy accuracy, clarification behavior, safe refusal, and correct tool execution.
    • Risk outcomes: unauthorized or incorrect actions, failed confirmations, customer complaints, policy exceptions, and cases requiring operational repair.

    Conversion and average order value belong beside returns and cancellations. An agent can raise the initial basket by recommending a more expensive option while reducing customer fit. Without the downstream view, the dashboard rewards the sale and hides the repair.

    Your event model should make the journey reconstructable. Useful events include agent opened, intent classified, clarification answered, recommendation shown, product selected, cart changed, checkout started, order completed, support action requested, confirmation received, action completed, and human handoff. Join these events through an appropriately governed session or order identifier so the team can inspect both funnel movement and individual failure paths.

    Build the evaluation before the vendor demo

    Create scenarios from your real catalog shape, policies, and failure modes. For each scenario, write the expected outcome and the failures that would make deployment unacceptable. Include:

    • An ambiguous shopping request that requires clarification.
    • Two products that appear similar but differ on a constraint customers care about.
    • An unavailable variant that would otherwise be the best match.
    • A question whose answer depends on store policy rather than product copy.
    • A conversation that moves from shopping to order support and back again.
    • A request that sounds actionable but lacks required authentication or confirmation.
    • An unsupported request that should trigger a safe handoff.
    • A catalog or policy change made after the agent’s initial synchronization.

    Run the same scenarios repeatedly and record the underlying catalog state each time. You are testing consistency, grounding, and recovery, not literary elegance. A shorter answer that uses the correct variant and policy is more valuable than a fluent answer that improvises.

    Expand autonomy in the order of consequence

    A staged rollout lets evidence determine how much authority the agent receives:

    1. Evaluate offline with approved scenarios and representative catalog data.
    2. Launch read-only product discovery and policy answers with an obvious human fallback.
    3. Add visible, reversible cart actions after recommendation quality is stable.
    4. Introduce authenticated order-support workflows with previews and confirmations.
    5. Enable financially consequential actions only after tool execution, auditability, exception handling, and operational ownership have been tested end to end.

    Define the owner of each failure before launch. Product should own the intended customer behavior and success measures. Commerce or operations should own policy and workflow correctness. Engineering should own integration reliability and observability. Customer support should own escalation quality and emerging failure patterns. The exact reporting lines can vary; an unowned failure queue cannot.

    Review both aggregates and conversation-level traces after release. Aggregate metrics tell you whether the experience is moving the business. Traces tell you why. A small cluster of incorrect variant recommendations or failed order actions can disappear inside a healthy conversion average while creating disproportionate customer harm.

    Key takeaways

    • A Shopify-native agent should use live commerce data and governed APIs; storefront placement alone is not enough.
    • The agent’s product-discovery job is to uncover decision criteria, apply hard constraints, explain tradeoffs, and lead to the next useful action.
    • Shopping and support should share customer context, but the agent’s permissions must become stricter as actions become harder to reverse.
    • Conversation conversion is a diagnostic metric, not causal proof. Measure incremental results across eligible traffic whenever possible.
    • Pair conversion and average order value with returns, cancellations, incorrect actions, and operational repair costs.
    • Begin with read-only assistance and expand autonomy only after each workflow proves accurate, observable, recoverable, and properly owned.

    Before you approve a purchase, bring a vague shopping request, a policy edge case, an unavailable variant, a mixed sales-and-support conversation, and a consequential order action into a live test store. If the agent cannot show where its answer came from, what it will change, and how it fails safely, you have found the next product requirement, not a detail to defer until launch.

    References

  • 5 Proven Agent Skills I Use to Automate Weekly Product Reviews with Claude, Cursor, and Codex

    5 Proven Agent Skills I Use to Automate Weekly Product Reviews with Claude, Cursor, and Codex

    Weekly product reviews are where strategy meets execution, and over the past year I’ve turned them into a high-signal, low-friction ritual by leaning on agentic AI. As VP of Product Management at HighLevel, Inc., I’ve standardized a set of agent skills that compress preparation time, surface the right insights, and keep PMs, engineers, and designers focused on decisions—not document wrangling.

    "Learn how our teams use agent skills with claude, cursor and codex to run product reviews as PMs, engineers, and designers. Here are 5 killer use cases for builder."

    Below, I walk through the five skills I rely on most in our weekly cadence—each one mapped to a clear product management outcome. They’re simple to set up, easy to govern, and aligned with core practices like continuous discovery, product roadmapping and sprint planning, and eval-driven development.

    Skill 1 — Backlog triage with signal extraction: I point an agent at fresh tickets, customer notes, and experiment results to cluster themes, tag impact, and flag regressions. Using a retrieval-first pipeline and Agent Analytics, the assistant ranks items by value, effort, and risk so our meeting starts with a prioritized, explainable shortlist instead of a raw queue.

    Skill 2 — PRD and spec synthesizer: Ahead of the review, an agent drafts a one-page PRD update from design diffs, git history, and decision logs. With Claude Code and Cursor, it highlights interface changes, acceptance criteria, and open questions, linking back to sources. The result is a crisp, auditable brief that keeps product trios aligned without re-litigating context.

    Skill 3 — Experiment and metrics analyzer: An analytics agent pulls A/B testing readouts, checks minimum detectable effect assumptions, and annotates anomalies. It turns raw telemetry into a narrative: what moved, by how much, and whether we trust it. This makes our discussion about tradeoffs, not spreadsheets, and speeds commitments on next steps.

    Skill 4 — Voice-of-customer synthesizer: The assistant clusters interviews, support threads, and NPS verbatims into jobs-to-be-done and pain themes. It proposes opportunity solution tree updates and calls out places where our roadmap diverges from customer signal. That keeps continuous discovery alive in the room—even when time is tight.

    Skill 5 — Roadmap and sprint planning co-pilot: After decisions, an agent converts outcomes into scoped backlog items, engineering tasks, and stakeholder updates. It drafts sprint goals, flags dependency risks, and aligns work to objectives. Because it’s grounded in the meeting record, it preserves intent while removing ambiguity.

    Under the hood, prompt engineering patterns and guardrails keep these workflows predictable: a retrieval-first pipeline for context, eval-driven development for quality checks, and role-specific prompts for PMs, engineers, and designers. With Claude Code I generate structured diffs and test scaffolds; with Cursor I accelerate code-review summaries; and with codex I bootstrap utility scripts that keep the loop tight between insights and implementation.

    The payoff is tangible: higher decision velocity, fewer meetings to “re-clarify,” and clearer accountability across the product organization. Just as important, governance and privacy-by-design are built in—every agent logs rationale, cites sources, and respects data boundaries—so leaders can scale AI workflows confidently.

    If you’re looking to level up your product reviews, start with these five skills, measure impact with Agent Analytics, and iterate. Small automations compound quickly, and the more consistently you run them, the more your team’s attention shifts from preparing content to making better product decisions.

    Inspired by this post on Amplitude – Perspectives.

    Book a consult png image
  • How to Build a Reliable WhatsApp AI Ordering Agent

    How to Build a Reliable WhatsApp AI Ordering Agent

    You are not really deciding whether an LLM can chat about a menu. You are deciding whether it can turn a messy WhatsApp exchange into a correct, payable order without making the customer or venue staff repair its work.

    That distinction changes the product. The hard parts are structured order state, deterministic commerce operations, response time, failure recovery, and venue-specific evaluation. Get those right and WhatsApp can become a genuine ordering channel. Get them wrong and you have a fluent chatbot sitting in front of an unreliable transaction.

    Key takeaways

    • Define success as a confirmed, recoverable order in the system of record, not a conversation that sounded helpful.
    • Let the model interpret customer language, but keep menu data, prices, modifiers, delivery eligibility, payment state, and order commits behind deterministic tools.
    • Store the current order as structured state outside the transcript. A conversation is evidence of intent, not an order ledger.
    • Measure useful response time across the complete WhatsApp-to-POS path, then remove tool round trips and parallelize safe read operations.
    • Make item identification accuracy the primary trust metric, supported by guardrails for modifiers, payments, duplicate submissions, handoffs, and latency.
    • Evaluate every venue against its real menu and rules, then turn recurring configuration, tests, and operating procedures into reusable templates.

    Define the product around a completed order

    WhatsApp is the interface, not the product boundary. The product boundary should run from the customer’s first request to an order state that the venue can fulfill and the customer can verify.

    A useful benchmark is the end-to-end flow implemented by AITropos: recommendations, item modifiers, delivery-zone checks, payment links, and status updates inside WhatsApp. Covering the whole journey matters because every missing step creates a handoff. A bot that recommends a meal but cannot resolve its required modifiers is a discovery feature. A bot that drafts an order but cannot verify submission is an assistant. Neither is yet an autonomous ordering agent.

    Write an order contract before choosing models or orchestration frameworks. The contract is the minimum structured state required to fulfill, charge for, recover, and audit an order. It will usually include:

    • The venue and the applicable menu version.
    • Canonical item identifiers, quantities, and customer-facing item names.
    • Required and optional modifier selections, represented by identifiers rather than prose alone.
    • Fulfillment method, such as pickup or delivery.
    • The validated delivery result when delivery is requested.
    • A system-generated quote, including the values the customer must approve before payment or submission.
    • Payment-link and payment states, without treating a generated link as proof of payment.
    • Customer confirmation state, POS submission state, and the resulting order identifier.
    • The current owner of the interaction: agent, venue staff, or a defined recovery process.

    The contract gives product, engineering, operations, and venue teams the same definition of done. It also exposes where autonomy is not yet safe. If the integration cannot validate a delivery zone, for example, the agent should collect the address and hand the order to a person. It should not infer eligibility from a conversational guess.

    Order stageThe agent’s jobCondition before proceeding
    DiscoverMap natural language to menu candidates and explain relevant options.One supported item is identified, or the agent asks a specific clarifying question.
    ConfigureCapture quantity, required modifiers, exclusions, and additions.Every required choice is present and valid for that item.
    FulfillmentResolve pickup or delivery and call the applicable eligibility checks.The requested fulfillment method is supported for this order.
    Quote and paymentRetrieve the authoritative quote and create the approved payment flow.Prices and payment state come from the commerce system, not generated text.
    CommitPresent the structured summary and submit the confirmed order once.The customer has confirmed the current version and the POS returns a result.
    Status and recoveryReport system-backed status or transfer the interaction with its context intact.The response is tied to an order identifier or an explicit handoff owner.

    Pay particular attention to the acceptance boundary. A friendly message such as “your order is being prepared” is an operational commitment. It must only appear after the system of record has accepted the order. If submission times out or returns an ambiguous result, the safe response is that confirmation is still pending, followed by a status check or human recovery. Guessing success can create duplicate orders, missed orders, and payment disputes.

    You can still launch with partial automation, but name it accurately. Menu search, order drafting, and staff-assisted submission can deliver value while the integrations mature. The mistake is allowing the customer to believe the order was accepted when the product has only generated a plausible summary.

    Keep the order deterministic even when the conversation is not

    Customers do not speak in schemas. They change quantities, refer to items by incomplete names, add a second request before answering the first question, and revise earlier choices. Your architecture has to translate that non-deterministic conversation into structured, POS-compatible data without losing which version the customer actually approved.

    My rule is simple: the model may interpret intent and propose an order-state change, but deterministic services must validate and commit it. The transcript should never be the only place where the current order exists.

    A reliable turn can follow this sequence:

    1. Load the current structured order, venue configuration, and relevant menu context.
    2. Interpret the latest message as a proposed change: add, remove, replace, modify, confirm, cancel, pay, or request status.
    3. Resolve referenced items and modifiers to canonical identifiers.
    4. Call read-only tools for availability, configuration, fulfillment rules, or quotes as needed.
    5. Validate the proposed change against required modifiers and venue rules.
    6. Write a new order-state version and generate the next response from that validated state.
    7. Use a separate, idempotent write operation when the customer confirms submission.

    This design makes corrections much safer. If the customer says, “Make the second one large and remove the fries,” the agent should apply a state delta to the identified lines, validate the revised configuration, and show the updated summary. It should not regenerate the entire order from memory and hope that unrelated details remain intact.

    Tool contracts should be narrow and explicit. Menu search should return canonical candidates and the information needed to distinguish them. Item detail should return valid modifier groups. A quote tool should return authoritative values. A payment tool should return a system-created link or a structured error. An order-submission tool should return an accepted identifier, a definite rejection, or an unresolved state that triggers recovery.

    Do not let the model invent a price, payment URL, availability claim, delivery decision, or order status. These are business facts with financial and operational consequences. The response composer can explain them in natural language, but the underlying values must come from an approved system.

    Separate reads from writes in the architecture. Independent menu and item lookups can often run in parallel. Writes should be serialized against a known order-state version. Every commit operation should accept an idempotency key so a retry cannot create a second order. If the state changed after the customer saw the summary, require confirmation of the new version rather than silently committing it.

    The same discipline applies to human handoff. Transfer the structured cart, unresolved question, relevant tool results, and submission state along with the transcript. A handoff that forces staff to reread the entire conversation and reconstruct the order is not graceful degradation; it is deferred manual work.

    Choose the orchestration pattern from the service objective, not from architectural fashion. Under tight response constraints, AITropos chose direct tool calls instead of MCP or a multi-stage pipeline to reduce orchestration overhead. That is not a universal argument against MCP. It is a reason to benchmark the actual path. Compare end-to-end latency, traceability, schema governance, failure isolation, and engineering cost using representative ordering turns. If an abstraction adds useful control, keep it. If it only adds another round trip, remove it.

    Manage latency as part of the customer experience

    The model’s inference time is only one part of latency. From the customer’s perspective, the clock starts when the message is sent and stops when a useful next action arrives. Context retrieval, menu search, validation, payment calls, POS submission, message delivery, retries, and overloaded queues all sit inside that interval.

    Instrument the complete path before optimizing it. Capture timestamps for message receipt, context assembly, model execution, every tool call, state validation, response creation, and outbound delivery. Report median and tail latency by turn type. A single average can hide a checkout path that is consistently slower than menu questions.

    At minimum, separate these turn classes:

    • Menu discovery and recommendation.
    • Item identification and configuration.
    • Cart edits and corrections.
    • Delivery or fulfillment validation.
    • Quote and payment-link creation.
    • Order confirmation and POS submission.
    • Order-status retrieval.
    • Human escalation and recovery.

    Set a service objective for each class from observed channel behavior and the operational risk of delay. There is no useful universal number. A status lookup and a multi-item order edit do different work. What matters is that the team can see which component consumes the budget and what happens when that component times out.

    Optimize in the order that removes uncertainty as well as delay:

    1. Remove unnecessary model and tool round trips. Load the active order and venue configuration before asking the model what to do.
    2. Parallelize independent read operations, such as resolving multiple products mentioned in one message.
    3. Prefetch likely item context so the agent does not discover basic menu facts one call at a time.
    4. Inject only the context needed for the current turn. An oversized prompt moves latency rather than eliminating it.
    5. Keep deterministic validation outside the model when a rule or schema check can answer immediately.
    6. Give every external dependency a timeout, an observable error state, and a safe recovery path.
    7. Use concise responses that advance the order. Extra prose increases reading time and can obscure the decision you need from the customer.

    A useful implementation pattern is already visible in production: multiple product searches run in parallel, product context is prefetched, and smaller, faster components prepare the relevant context for each turn. The product lesson is not to create a swarm of agents. It is to move predictable preparation out of the critical reasoning loop while preserving one coherent order state.

    Watch the failure mode on the other side of aggressive optimization. Cached menu metadata can reduce retrieval work, but stale availability or price data can create a wrong commitment. Define which fields are stable enough to cache, how they are invalidated, and which values must be retrieved at quote or submission time. Speed is valuable only when the answer remains authoritative.

    When a slow operation cannot be avoided, use an honest progress message and preserve the pending state. Do not fill the wait with repeated acknowledgements that imply completion. If the customer sends another message while the tool is running, the state machine should know whether to queue the change, cancel the pending operation, or ask the customer to wait for its result.

    Evaluate each venue, then template what repeats

    Make item accuracy precise enough to govern decisions

    Item identification accuracy deserves to be the primary trust metric. If the agent resolves the wrong item, every later component can behave perfectly and still produce the wrong order. AITropos treats order item identification accuracy as its most important KPI, giving model, prompt, retrieval, and fallback decisions a common objective.

    Define the metric before building a dashboard. I would count an attempted line item as correct only when the canonical item, quantity, and required modifier interpretation match the customer’s resolved intent. A necessary clarification is not automatically an error; it should count against a separate clarification-burden metric. Otherwise, the team may improve apparent accuracy by asking the customer to confirm every obvious detail.

    Do not let the primary KPI hide transaction failures. Pair it with guardrails for:

    • Unsupported substitutions or invented items.
    • Missing and invalid required modifiers.
    • Customer corrections after the agent presents a summary.
    • Quote, payment-link, and POS tool failures.
    • False confirmations, unresolved submissions, and duplicate commits.
    • Order completion and abandonment by journey stage.
    • Human handoff rate, reason, and time to recovery.
    • End-to-end latency by turn class and venue.

    Link corrections back to the original decision. If the customer changes an item because the agent misunderstood it, label the item-resolution turn rather than treating the correction as an unrelated edit. That is how production behavior becomes useful evaluation data instead of a collection of support anecdotes.

    Simulate failures before customers encounter them

    A venue-specific evaluation suite should use that venue’s menu identifiers, modifiers, availability behavior, delivery rules, payment flow, and POS adapter. A generic restaurant benchmark can test language understanding, but it cannot tell you whether the agent knows that a particular size requires a particular modifier or that two similar menu names map to different SKUs.

    Build test families for:

    • Incomplete names, colloquial references, and ambiguous matches.
    • Several products requested in one message.
    • Required modifiers, exclusions, additions, and invalid combinations.
    • Quantity changes, replacements, removals, and cancellation.
    • Unavailable items and acceptable alternatives.
    • Pickup, delivery, and addresses that cannot be validated.
    • Quote changes before confirmation.
    • Payment failure, delayed payment state, and an abandoned payment flow.
    • Tool timeouts, malformed tool results, retries, and uncertain POS submission.
    • Interrupted conversations that resume with an existing cart.
    • Requests that require staff judgment rather than autonomous execution.

    Generate the expected structured order independently from the agent being tested. Otherwise, the same model can reproduce its own misunderstanding in both the answer and the grade. Keep a small, human-reviewed set of critical conversations alongside the larger generated suite, and add every material production failure to the permanent regression set.

    Scale matters when menus contain many combinations. Before each new venue goes live, AITropos runs thousands of simulated customer conversations overnight. The number alone is not the release gate. Coverage, a trustworthy expected answer, and clear failure categories are what make simulation useful.

    Simulation also cannot reproduce every production condition. Follow it with a staff sandbox and a controlled production phase. Use only redacted, properly authorized customer conversations in evaluation systems, and retain no more personal data than the test requires.

    I would treat any path that invents a price or payment state, falsely confirms an order, or can duplicate a commit as release-blocking. Other thresholds should reflect the venue’s menu complexity, existing human baseline, handoff capacity, and the cost of a wrong order. Record those thresholds before the final test run so launch pressure cannot redefine success afterward.

    Roll out autonomy in observable stages

    Start with a venue that is operationally manageable but representative enough to expose real modifiers, fulfillment rules, and integration behavior. An unusually simple pilot may produce a clean demo while postponing the problems that determine whether the product can scale.

    1. Configuration: ingest and normalize the menu, map canonical identifiers, mark required modifiers, connect fulfillment and payment rules, and produce a completeness report. No customer-facing ordering is enabled.
    2. Sandbox: let venue staff run realistic conversations while write tools remain disabled or point to a test environment.
    3. Approval mode: allow the agent to prepare a structured order, but require a person to approve the commit. Measure how often the person changes it and why.
    4. Constrained production: enable autonomous submission for the supported venue, fulfillment modes, and order types, with a staffed handoff path and rapid rollback.
    5. Expansion: widen scope only after production traces confirm the accuracy, latency, recovery, and operational workload expected by the release criteria.

    For every stage, decide who can pause the agent, how staff take over an active conversation, how the customer learns that a person has taken over, and how an uncertain submission is reconciled before another order is created. These are product requirements, not post-launch operating notes.

    Once one venue works, resist copying its prompt and integrations into a new branch. Make venue differences configuration wherever possible: normalized menu schemas, modifier patterns, fulfillment policies, tool mappings, escalation contacts, evaluation packs, and dashboard dimensions. Keep truly distinct behavior explicit rather than burying it in prompt prose.

    The scalability payoff can be substantial. AITropos reduced new-venue onboarding from three months to a few weeks, while domain templates are being used to shorten it further. Track your own onboarding work by category: configuration, data cleanup, integration, prompt or policy changes, evaluation, venue training, and launch support. If every venue still requires bespoke code and a rewritten conversation flow, the product has not yet separated its platform from its implementations.

    Your next step should be concrete. Choose one representative venue and create three artifacts: the canonical order contract, a failure-and-recovery matrix for every tool, and a venue-specific evaluation set built from redacted, authorized scenarios. If those artifacts cannot show what happens when item resolution, a modifier, delivery validation, payment, or POS submission fails, the agent is not ready to accept orders. Once those states are explicit, model and architecture choices become testable decisions rather than matters of confidence.

    References

  • Amplitude AI Product Analytics: A Practical Agent Playbook

    Amplitude AI Product Analytics: A Practical Agent Playbook

    You are deciding whether Amplitude Agents deserve a place in your product operating system. A fluent answer or polished insight is easy to admire. The harder question is whether the agent helps someone make a better decision, complete a valuable task, or change user behavior.

    That distinction determines how you should instrument, evaluate, and roll out the experience. Treat Amplitude as the measurement spine connecting agent activity to funnels, cohorts, experiments, retention, and product outcomes. Otherwise, you will know that the agent was used without knowing whether it was useful.

    Pick a workflow with an observable finish line

    Do not begin with a broad ambition such as helping everyone understand the data. It cannot be measured cleanly, and it gives the agent too much room to produce plausible output without resolving a real job.

    The useful standard is that AI product management remains accountable for helping teams build better products. The agent response is therefore an intermediate output, not the outcome. A strong starting point is one narrowly scoped, high-signal workflow with an unambiguous done state.

    Write a workflow contract before configuring dashboards or prompts:

    • User: Name the role doing the work, such as a product manager investigating onboarding friction.
    • Trigger: Describe the event that makes the job necessary, such as a drop in activation or an unexpected cohort difference.
    • Bounded job: State exactly what the agent should help accomplish.
    • Required evidence: Identify the events, funnels, segments, or cohorts that should support the output.
    • Done state: Define the observable action that marks useful completion.
    • Fallback: Decide what happens when the inputs are missing, the evidence conflicts, or the agent cannot complete the task reliably.

    For an onboarding investigation, the contract might ask the agent to help identify where a defined cohort leaves the activation journey and produce evidence-backed hypotheses for the product manager to review. The task is not complete when text appears. It is complete when the user reviews the relevant evidence and records a decision, launches a follow-up analysis, or creates an experiment.

    Use a simple outcome ladder to keep the team honest: eligible users see the experience, some start it, some reach the workflow’s done state, some act on the result, and the intended product outcome changes. Each level answers a different question. Collapsing them into an agent usage metric hides the point at which value disappears.

    Instrument the agent journey, not just the final answer

    Your event design should let you reconstruct the journey from opportunity to outcome. The names below are examples, not an official Amplitude schema. Adapt them to your existing naming convention and governance rules.

    Journey stageQuestion it answersSuggested event
    EligibleWho could reasonably use this workflow?agent_workflow_eligible
    ExposedWho actually saw an entry point?agent_entry_viewed
    StartedWho chose to begin?agent_run_started
    Evidence reviewedWho engaged with the information needed to judge the output?agent_evidence_viewed
    CompletedWho reached the workflow-specific done state?agent_task_completed
    ActionedWho used the output in a downstream decision or action?agent_output_applied
    Handed offWhere did the experience require a deterministic flow or human review?agent_handoff_triggered
    ReturnedWho came back when the job occurred again?agent_run_started, segmented by prior successful completion

    Add properties that explain why behavior differs: workflow identifier, product surface, user role, account cohort, journey stage, agent version, prompt or instruction version, completion reason, handoff reason, and error class. Version properties are essential. Without them, a release can change output quality while the dashboard incorrectly treats the experience as one stable product.

    If prompts may contain customer or company data, do not log the raw text by default. Prefer derived classifications, structured outcome fields, or properly redacted samples governed by your retention and access policies. Product analytics should increase observability without creating an unnecessary copy of sensitive input.

    Build each metric with an explicit denominator:

    • Discovery rate: exposed eligible users divided by eligible users.
    • Start rate: users who start divided by users exposed to the entry point.
    • Completion rate: users reaching the workflow-specific done state divided by users who start.
    • Action rate: users taking the defined downstream action divided by users who complete.
    • Retained use: previously successful users who return when the job recurs divided by previously successful users who had another opportunity.

    The eligibility and opportunity conditions matter as much as the numerator. A user cannot retain to a workflow that has not recurred, and someone who never saw the entry point should not be treated as a failed starter.

    In Amplitude, separate the views rather than forcing everything into one chart. Use an exposure funnel for discoverability, a workflow funnel for completion, cohorts for segment differences, retention analysis for repeat behavior, and a guardrail view for errors, retries, and handoffs. Use Agent Analytics for the execution signals available from the agent, then connect those signals to the behavioral events that represent product value.

    Keep output quality and product impact on separate scorecards

    Behavioral analytics cannot tell you whether an answer was correct. An evaluation set cannot tell you whether customers changed their behavior. You need both views because they fail in different ways.

    Before widening access, create an evaluation set drawn from the workflow contract. Include ordinary cases, incomplete inputs, ambiguous requests, conflicting evidence, and cases that should trigger a handoff. Grade the output against criteria that can be reviewed consistently:

    • Correctness: Does the conclusion match the available evidence?
    • Grounding: Can the user see which events, funnels, cohorts, or other inputs support it?
    • Task adherence: Did the agent solve the bounded job rather than produce a generic analysis?
    • Uncertainty handling: Does it distinguish supported conclusions from hypotheses?
    • Handoff behavior: Does it stop or redirect appropriately when required evidence is unavailable?
    • Actionability: Can the intended user make the next decision without reconstructing the analysis?

    Record pass or fail for non-negotiable criteria such as unsupported conclusions and failed handoffs. Keep graded usefulness criteria separate. A high average score should not conceal a smaller set of serious failures.

    Run the same evaluation set when you change instructions, tools, model configuration, retrieval behavior, or the data made available to the agent. This is the practical value of eval-driven development: a fast release becomes a controlled product change rather than an untraceable shift in behavior.

    Your online scorecard should then contain distinct layers:

    • Primary outcome: the workflow-specific completion or downstream action that represents value.
    • Adoption diagnostics: eligibility, exposure, start rate, and first successful completion.
    • Quality diagnostics: evaluation results, user corrections, retries, and unsupported-output flags.
    • Operational guardrails: errors, latency appropriate to the workflow, abandonment, and handoffs.
    • Product impact: the activation, feature adoption, retention, or other behavioral outcome the workflow is intended to influence.

    Choose one primary outcome before launch. The other measures explain why it moved or protect against a misleading win. If every metric is primary, the team can always find one that improved after the fact.

    User ratings can help diagnose tone, relevance, or missing context, but they are not a substitute for observed outcomes. A response can feel impressive and still produce no action. It can also look concise while helping an expert complete the job quickly. Pair stated feedback with completion, downstream action, and return behavior.

    Run an experiment that can survive executive scrutiny

    Do not compare enthusiastic agent adopters with everyone who ignored it. Those groups selected themselves, so their product outcomes may have differed before the agent appeared. Establish a baseline and create a controlled comparison wherever the workflow and traffic permit it.

    1. Write the hypothesis in behavioral terms. Name the user, workflow, expected action, and product outcome.
    2. Measure the current workflow before introducing the agent. Capture completion, abandonment, downstream action, and relevant guardrails.
    3. Define eligibility before assignment so the comparison includes people with the same underlying job.
    4. Choose the assignment unit that matches how the workflow spreads. Use an account-level unit when teammates share agent output; use a user-level unit only when experiences are genuinely independent.
    5. Expose the treatment through a feature flag or controlled rollout, while keeping the existing path available as the comparison and fallback.
    6. Evaluate the primary outcome and guardrails together. Do not call a faster workflow successful if output quality, error handling, or downstream behavior deteriorates.
    7. Inspect cohorts to understand a credible result, not to search endlessly for a segment that happens to look positive.

    The metric pattern often tells you where to investigate next:

    • High exposure with low starts can indicate weak positioning, poor timing, or an irrelevant eligible population.
    • Healthy starts with low completion can indicate that the promise is attractive but the workflow, inputs, or output quality is failing.
    • High completion with low downstream action can indicate that your done state is too shallow or the output is not trusted enough to use.
    • Strong agent engagement without movement in the product outcome can indicate a locally pleasant experience that does not change the broader journey.
    • Strong first use with weak return behavior can indicate novelty, unreliable value, or a job that simply occurs infrequently. Check opportunity before interpreting it as churn.
    • Good aggregate results with concentrated handoffs in one cohort can indicate missing context, permissions, or data for that segment.

    Guardrails should be operational, not aspirational. Validate required inputs. Make the agent’s task and evidence boundaries clear. Route the user to a deterministic flow or human review when observable conditions show that the task cannot be completed. Missing data, failed tool calls, validation failures, and unsupported claims are stronger handoff triggers than an agent merely describing itself as confident.

    Scale only when value repeats under real conditions

    A spike in usage after launch mainly proves that people noticed something new. Scale when the complete chain repeats: eligible users discover the workflow, finish it, act on the result, and return when the same job appears again.

    Segment that chain by role, account cohort, use case, journey, and agent version. A workflow that helps an experienced product analyst may confuse a first-time manager. An onboarding investigation may need different evidence and handoffs from a retention investigation. Aggregate adoption can hide both realities.

    Expand the rollout when the primary outcome improves, evaluation quality remains stable across relevant cohorts, guardrail failures stay controlled, and repeat use matches the natural frequency of the job. Redesign when successful users cannot find the entry point, retries cluster around the same step, completed outputs rarely lead to action, or results depend on one unusually capable cohort.

    Pause expansion when the agent does not improve the existing workflow, important outputs cannot be audited back to evidence, or failures cannot be routed safely. More exposure only creates more ambiguous data when the workflow contract itself is weak.

    Key takeaways

    • Define one bounded workflow and an observable done state before measuring adoption.
    • Connect agent execution signals to exposure, completion, downstream action, and product outcomes in Amplitude.
    • Use evaluation sets for output quality and behavioral analytics for real-world impact; neither replaces the other.
    • Compare the agent with the existing workflow among equally eligible users.
    • Treat retries, errors, unsupported outputs, and handoffs as product signals, not merely engineering logs.
    • Scale repeatable value across cohorts and versions, not a launch-driven usage spike.

    Your next move should fit on one page: the workflow contract, event map, evaluation criteria, experiment metric, and fallback path. If those elements are clear, Amplitude can show where the agent creates value and where it merely creates activity. If they are not clear, narrow the workflow before you widen the rollout.

    References

  • Unleashing Inbound Sales with AI: My Playbook for Launching and Scaling Sales Agents Fast

    Unleashing Inbound Sales with AI: My Playbook for Launching and Scaling Sales Agents Fast

    Inbound leads shouldn’t wait for a rep’s calendar. When we first launched The Service Agent Blueprint, support leaders finally had a clear AI path. Go-to-market and revenue teams are now facing similar uncertainty, so I’m introducing The Sales Agent Blueprint—a practical map for launching and scaling AI for sales with confidence.

    For most sales teams, inbound motions require a lot of manual work. I’ve watched leads pile up in queues, waiting for availability rather than being prioritized by buyer intent. That delay costs meetings, pipeline, and momentum—and it’s exactly where a modern AI Strategy can transform your go-to-market strategy.

    Agents can run sales conversations end to end – engaging buyers, qualifying leads, and routing high-intent opportunities to the right team to move prospective buyers forward quickly. Humans will still be involved, but will move their focus to the consultative conversations and higher-value work they did not have time to focus on before. In practice, this shift enables cleaner AI workflows, better conversation design, and a healthier balance between sales-led growth and product-led growth.

    The questions many go-to-market and revenue leaders are facing now are where do you start? What should success look like? How do you actually test and deploy these solutions? These are the right questions—and the ones I hear most often when teams weigh build vs buy decisions, evaluation frameworks, and CRM integration nuances.

    The Sales Agent Blueprint answers those questions. It’s designed to be a strategic guide for sales, revenue, and AI transformation leaders who want to deploy AI for inbound sales fast, prove value, and build momentum. If you’re aiming for eval-driven development, this will help you define success up front and operationalize it.

    What’s inside is simple by design yet deep enough to take you from zero to value. The Sales Agent Blueprint is structured around two tracks that reflect how high-performing teams adopt agentic AI: first, launch for quick wins; next, scale for durable growth.

    Minimal blue banner for Introducing the Sales Agent Blueprint with a bold 'Scale it' headline, abstract halftone device graphic, subtle crop marks, and a 'Coming Soon' badge in the upper-right corner.
    Coming soon: Sales Agent Blueprint. A sleek, blueprint-inspired teaser with the call to 'Scale it' signals tools, playbooks, and workflows to grow revenue, streamline operations, and scale teams with confidence.

    Today, I’m releasing the first part of the Blueprint: “Launch it.” It’s a practical guide for getting your Agent live and seeing real results. You’ll learn how to deploy a Sales Agent that runs inbound sales conversations end to end, engaging buyers, qualifying leads, and routing high-intent opportunities to the right outcome in real time—without disrupting your current CRM integration or pipeline processes.

    By the end of the “Launch it” track, you’ll be ready to execute with clarity. Here’s how I frame the essential steps, based on what consistently works in the field.

    Understand what a Sales Agent is: Discover why they’re different from chatbots and how they work. Build a business case: Prove the basic economics of AI, decide whether to buy or build, and get the buy-in and budget you need to move forward.

    Evaluate an Agent: Learn how to define success, choose the right evaluation criteria, and run a focused, high-impact assessment with our five-step framework.

    Deploy with confidence: Build a deployment plan that gets your Agent live quickly to engage buyers at peak intent. Learn what to expect at each stage.

    Vector-style 'Blueprint' title on a light grid with Bézier points, plus a royal-blue panel reading '1 Launch it' next to a satellite icon; footer shows FIN.AI/BLUEPRINT/SALES promoting the Sales Agent Blueprint.
    Introducing the Sales Agent Blueprint. This crisp, grid-based graphic spotlights step 1—Launch it—signaling day-one activation for an AI sales agent. Explore the framework and get started at fin.ai/blueprint/sales.

    Continuously improve performance: After launch, your Agent becomes a system to manage. We’ll show you how to implement a repeatable process to train, test, deploy, and optimize.

    The second track, “Scale it” (coming soon), focuses on the organizational and systems design work that unlocks compounding gains. Launching AI is only the beginning. To unlock its full potential, you need to rewire your inbound sales motion—redesigning the buyer journey, building AI-first systems and ownership models, and rethinking how pipeline is generated and scaled. This is where governance, measurement, and team roles evolve to support sustainable growth.

    I’ll be building this Blueprint in public as I navigate the same challenges—sharing what works, what to avoid, and how to accelerate time-to-value without sacrificing quality or trust. If you’re ready to turn intent into revenue with agentic AI, this is your head start.

    The Sales Agent Blueprint is live now. Explore the full guide at fin.ai/blueprint/sales and start your “Launch it” sprint today.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • How to Design an AI Customer Agent for Sales Qualification

    How to Design an AI Customer Agent for Sales Qualification

    A prospect reaches your pricing page with a real buying question. The form promises a reply, but the reply arrives after the prospect has moved on, chosen a competitor, or forgotten why the question mattered.

    An AI customer agent can remove that delay, but speed is only the entry requirement. The harder product problem is deciding whom to qualify, what evidence to collect, which next step to offer, and how to preserve enough context that a salesperson can continue the conversation without starting over.

    Start with routing decisions, not chatbot dialogue

    The purpose of a sales qualification agent is not to produce a pleasant conversation or a high lead score. Its job is to make a defensible next-step decision while the buyer’s intent is still active.

    That distinction matters because conversational fluency can hide weak commercial logic. An agent may sound helpful while booking low-fit meetings, sending strong prospects down a generic self-serve path, or marking inferred information as confirmed. Those failures make the pipeline look larger before they make it less trustworthy.

    Define the available outcomes before you write prompts. Most inbound motions need some version of these routes:

    RouteMinimum evidenceAgent action
    Sales-readyThe problem fits the product, the buyer needs sales involvement, and the timing or buying process satisfies your acceptance rule.Offer an appropriate meeting, create or update the CRM record, and send the qualification evidence.
    Self-serveThe use case is viable, but the buyer can select a plan, begin a trial, or complete signup without a salesperson.Recommend the relevant path, help the buyer take the next action, and preserve the conversation for later use.
    Promising but not readyThere is plausible fit, but intent, timing, authority, or requirements remain unresolved.Provide the useful resource or follow-up path defined by your policy without manufacturing urgency.
    Not a fitA hard requirement conflicts with the product’s supported scope or the request belongs elsewhere.State the limitation clearly and redirect the person without placing an unqualified meeting on a seller’s calendar.
    Human exceptionThe request involves an existing account, a sensitive claim, a complex commercial exception, or information the agent cannot verify.Escalate with the context already collected and identify the unresolved question.

    Keep fit and readiness as separate dimensions. A large, recognizable account can be a strong fit and still be months away from a decision. A highly motivated buyer can be ready to act and still need a capability you do not provide. Combining both dimensions into one opaque score conceals the reason behind the route and makes mistakes difficult to diagnose.

    Separate hard constraints from soft signals as well. A required capability that does not exist may be a hard stop. A vague timeline is usually uncertainty to resolve, not automatic disqualification. Firmographic enrichment can help prioritize an account, but it cannot confirm what a buyer has not actually said.

    For every consequential route, require three outputs: a reason code, the evidence behind it, and the next action. If the agent cannot produce all three, it has not completed qualification. It has merely assigned a label.

    Turn your qualification policy into an executable conversation

    A natural-language playbook makes sales policy easier to express, and current customer agents can be instructed to follow qualification rules, address approved objections, and move buyers toward defined outcomes. Natural language does not remove ambiguity, however. If two experienced salespeople would interpret a rule differently, the agent will not reliably repair the policy for you.

    Ask only what changes the route

    Traditional lead forms collect fields because the CRM has columns. A conversation should be more selective. Every question should either help the buyer, determine fit, resolve readiness, or select the correct action.

    1. Open from observable context, such as the plan, feature, or integration the person is exploring.
    2. Answer the buyer’s current question before turning the exchange into discovery.
    3. Ask the smallest useful question that could change the route.
    4. Branch from the answer instead of walking every prospect through the same questionnaire.
    5. Confirm the important facts before treating them as qualification evidence.
    6. Explain the recommended next step and let the buyer act while still in the conversation.

    If someone asks whether a specific integration is available, answer that question first. Then ask how the integration fits the intended workflow if the answer would affect plan selection or sales involvement. Leading with budget, company size, or phone number when none of those details helps answer the immediate question makes the agent feel like a form with typing animation.

    A useful qualification schema usually covers the following areas, but the agent should collect only the fields relevant to the current branch:

    • The problem or use case the buyer is trying to address.
    • The capabilities, integrations, or constraints that determine product fit.
    • The consequence of leaving the problem unsolved, when that affects urgency or route.
    • The buyer’s role in evaluation and the remaining buying process.
    • The intended timing and any event driving it.
    • Commercial expectations or budget when those facts genuinely affect the path.
    • Identity and account context, with a clear distinction between what was stated, enriched, or inferred.

    Do not ask about budget merely because a familiar qualification framework includes it. If pricing is public and the buyer can start without sales assistance, the better action may be to explain plan fit and help the buyer proceed. If commercial terms require human involvement, budget or purchasing process may become relevant later in the branch.

    Preserve provenance instead of filling blanks with guesses

    Store each material qualification field with its provenance. Buyer-stated, externally enriched, model-inferred, and unknown are different states. Treating them as interchangeable creates false confidence in the CRM.

    An enriched company size may help prioritize a conversation, but it is not buyer-confirmed budget. A page visit may indicate interest in a feature, but it is not a confirmed requirement. An enthusiastic phrase may indicate intent, but it is not a purchasing timeline. Keep those distinctions visible to the routing logic and the salesperson receiving the lead.

    I would not allow the agent to write a final qualification status unless every required field is either supported by evidence or explicitly marked unknown. Unknown information is operationally useful: it tells the seller what still needs to be resolved. Fabricated completeness does the opposite.

    Evaluate decisions, not just responses

    Build a scenario set from the situations that cause real routing disagreements. Include a high-fit account with low intent, a small account with urgent intent, an existing customer asking a sales-shaped question, a buyer requiring an unsupported capability, a returning visitor, a pricing objection, conflicting information, and a request the agent should escalate.

    For each scenario, define the expected answer, acceptable route, required CRM writes, escalation condition, and forbidden behavior. Then test the complete journey. A correct answer followed by the wrong calendar, duplicated CRM record, or context-free handoff is still a failed qualification experience.

    Build trust into answers, memory, and handoffs

    A sales agent cannot qualify reliably if its product knowledge is unreliable. It needs approved information about pricing, packages, capabilities, integrations, plan eligibility, trial paths, and common objections. Current implementations can draw on an existing product knowledge base while combining that knowledge with playbooks, enrichment, and memory, which reduces duplicated setup but does not eliminate ownership.

    Assign a business owner to every consequential knowledge area. When pricing, packaging, an integration, or an eligibility rule changes, update the canonical material and rerun the scenarios affected by that change. A polished answer based on stale commercial information is more dangerous than an explicit handoff because the buyer has little reason to question it.

    Define the agent’s boundaries in the same system. It should know when it may explain published pricing, when it must avoid inventing discounts, when roadmap questions need human confirmation, and when a security, legal, or contractual claim requires escalation. The safe fallback is not a vague non-answer. It is a clear statement of what remains unverified and a context-rich route to someone authorized to answer.

    Use memory as buyer state, not as an unlimited transcript

    Memory is valuable when a returning visitor does not have to repeat the use case, plan under consideration, or unresolved objection. A customer agent can recognize returning context and continue the buying journey, but old information should not silently override new facts.

    Store a compact buyer state: confirmed needs, important constraints, questions already answered, current route, unresolved items, and the last agreed next step. Keep timestamps and provenance so the system can notice when a current statement conflicts with an earlier one. Ask for confirmation when a material fact may have changed.

    Enrichment deserves the same discipline. Use it to improve context and routing, not to pretend the agent knows the buyer personally. Record where enriched data came from, apply your privacy and retention controls, and give buyer-stated information precedence when the two conflict.

    Make the handoff a product deliverable

    Booking a meeting is not the end of qualification. It is the beginning of a human handoff. Passing only a name, email address, and transcript forces the salesperson to reconstruct the conversation under time pressure.

    The handoff package should contain:

    • Identity and account information, including the provenance of enriched fields.
    • The buyer’s problem and intended outcome in the buyer’s own terms.
    • Confirmed requirements, constraints, timing, and buying-process details.
    • Questions answered and the approved information used to answer them.
    • Objections, unresolved questions, and any claim requiring human confirmation.
    • The selected route, its reason code, and the evidence that supported it.
    • The next action already promised to the buyer.
    • A link to the full conversation for detail or audit.

    Modern customer agents can book through scheduling tools, sync structured context into the CRM, and pass both conversation history and an AI-generated summary. The summary should reduce reading effort, while the structured fields should support routing, reporting, and workflow automation. Neither should replace access to the original conversation.

    Test the first minute of the seller’s follow-up. Can the seller see why the lead was routed, what the buyer already knows, and what must happen next? If the opening question repeats discovery the agent just completed, the handoff has broken the continuity you used AI to create.

    Measure whether the agent creates incremental pipeline

    Meeting count is an attractive but incomplete success metric. Bookings can rise because the agent reaches previously unattended demand, because it diverts buyers who would have booked with a human anyway, or because it lowers the qualification bar. Only the first outcome is unambiguously additive.

    Instrument the full decision funnel rather than the chat interface alone:

    1. Reach: eligible visitors, conversations initiated, response latency, and coverage by channel or time window.
    2. Conversation quality: questions answered, unresolved-answer rate, corrections, escalations, and abandonment before a useful action.
    3. Qualification quality: completion of required evidence, unknown-field rate, route distribution, seller acceptance, and rejection reasons.
    4. Handoff quality: meeting attendance, repeated discovery, missing CRM context, reassignment, and follow-up delay.
    5. Commercial outcome: accepted qualified opportunities, pipeline created, trial or self-serve conversion, win rate, contract value, sales-cycle progression, and cost per accepted opportunity.

    Audit both error directions. False positives waste seller time and inflate forecasts. False negatives are quieter: a strong buyer is sent away, mislabeled as self-serve, or blocked by an unanswered question. Review unsuccessful routes as well as booked meetings, because the most expensive qualification error may never appear in the sales team’s queue.

    Early deployment data shows why coverage and incrementality need separate analysis. In a vendor-reported overnight rollout, Fellow booked 18 January meetings that its human team would not otherwise have reached, with around 48% converting, while the human booking rate held. That is evidence of an additive channel in that deployment, not a universal conversion benchmark.

    Volume alone tells a different and incomplete story. During a vendor-reported three-month deployment, Attio’s agent handled more than 1,600 visitor conversations, qualified more than 50 leads for sales, and routed more than 30 applicants into a startup program. Those figures show multiple useful outcomes from the same inbound surface, but they do not establish causal lift for another company’s funnel.

    Establish your own baseline separately for hours and pages with human coverage and those without it. If feasible, use a randomized holdout among otherwise eligible sessions. If randomization would create an unacceptable buyer experience, compare matched cohorts by page, channel, segment, visitor status, and time window. Do not compare an overnight agent cohort with daytime human coverage and call the difference an AI effect.

    A controlled rollout can begin on a high-intent surface or during a coverage gap. First run routing in shadow mode and compare the proposed decisions with qualified human judgment. Then enable one consequential action at a time, such as self-serve guidance before autonomous meeting booking. Keep a human review path for exceptions and expand only when answer quality, routing precision, CRM completeness, and buyer outcomes remain acceptable together.

    Your north-star measure should reflect accepted commercial value, such as incremental qualified opportunities or incremental pipeline per eligible visitor. Pair it with guardrails for incorrect claims, seller rejection, buyer complaints, CRM errors, and missed high-fit leads. An agent that creates more records while reducing trust has not improved the sales system.

    Key takeaways

    • Treat the AI customer agent as a decision system, not a conversational layer placed in front of a lead form.
    • Define sales-ready, self-serve, not-ready, not-fit, and human-exception routes before writing dialogue.
    • Keep fit separate from readiness, and preserve whether each field was buyer-stated, enriched, inferred, or unknown.
    • Ask only questions that help the buyer or change the route; answer the buyer’s immediate question before running discovery.
    • Make structured qualification evidence, unresolved issues, and the promised next action part of every human handoff.
    • Measure incremental accepted opportunities and pipeline, not chat volume, MQL count, or booked meetings in isolation.

    Start with one high-intent entry point. Write its route contract, connect only approved knowledge, test the difficult scenarios, and compare shadow decisions with the people who currently qualify those leads. Give the agent authority gradually. The goal is not to automate the most conversation; it is to make the right buying path available at the moment the buyer is ready to take it.

    References

  • How to Build Agentic AI for Product Analytics and Support

    How to Build Agentic AI for Product Analytics and Support

    Your support bot can tell a customer where a setting lives, yet leave that customer to diagnose the problem, change the setting, and hope it worked. Your product team then receives a chat transcript without knowing whether the interaction improved activation, feature adoption, or retention.

    If you are deciding how to connect AI, product analytics, and support, do not start with the model. Design the closed loop first: assemble trustworthy context, choose an allowed action, verify the resulting product state, and measure the user outcome. The model is one component inside that system.

    Treat product analytics as the agent’s control plane

    A useful standard is an assistant that understands the user’s context, can complete an allowed action, and measures whether the action helped. Remove any one of those capabilities and the experience degrades quickly. Context without action produces advice. Action without context creates risk. Action without measurement creates an impressive demo that cannot earn a durable place on the roadmap.

    Product analytics supplies the behavioral context and outcome signals for this loop. It can show where the user is in a journey, which features have been adopted, which step failed, and whether the expected success event eventually occurred. It should not be treated as a warehouse-sized attachment to the prompt.

    Define a support context contract

    Create a small, governed context object for each supported workflow. Give the agent only the fields required to understand and resolve that workflow:

    • Actor and access: the authenticated user, account, role, entitlements, and permissions relevant to the requested action.
    • Journey state: the onboarding step, feature-adoption state, experiment assignment, or other stage that explains what the user is trying to complete.
    • Current product state: the relevant configuration from the operational system of record, including whether required prerequisites are satisfied.
    • Friction evidence: recent failed events, validation results, repeated attempts, and known errors connected to this workflow.
    • Desired outcome: the product state and behavioral event that will count as successful resolution.

    Resolve analytics events and tool calls to the same stable user and account identifiers. Preserve timestamps and the origin of each field. For a live action decision, let the operational system of record determine current state; use analytics to explain the journey and measure the outcome. An event stream can be delayed or incomplete, so it should not overrule a current configuration read.

    Behavior is also evidence, not intent. Repeated visits to a setup screen could indicate confusion, careful verification, or an advanced workflow. When those interpretations require different actions, the agent should ask one targeted question instead of turning a behavioral pattern into a confident diagnosis.

    Apply data minimization at this boundary. Do not place secrets, payment information, unrelated conversation history, or an account’s entire event history into the model context. Filter fields before the model sees them, and enforce the filter in code rather than relying on a prompt instruction.

    Give the analytics agent a metric contract

    An internal analytics agent has a different job from a customer-facing support agent. It may translate a product question into metrics, cohorts, funnels, or retention views, but a fluent answer is not enough. Require every analysis to return:

    • the product question it interpreted;
    • the metric definition and success event it used;
    • the cohort, filters, and observation window;
    • the analysis or query reference needed to reproduce the result;
    • known data-quality limitations and unresolved ambiguity; and
    • a clear distinction between observed association and demonstrated causal lift.

    This turns the analytics agent into a traceable decision aid. It also prevents two agents from using the same metric name while silently applying different event definitions, account filters, or windows.

    Design one closed loop from signal to verified outcome

    The core unit of agentic support is not the conversation. It is a resolution attempt with a beginning, an authorized action, and a verifiable end state. Use the following loop for every workflow:

    1. Observe the trigger. Capture the user’s request or a product signal that indicates likely friction.
    2. Assemble scoped context. Load only the identity, permission, journey, state, and error fields defined in the context contract.
    3. Diagnose the next constraint. Determine which prerequisite, configuration, permission, or knowledge gap is blocking progress. If the evidence is ambiguous, ask rather than assume.
    4. Select an approved playbook. Match the constraint to a versioned workflow with explicit eligibility rules, allowed tools, and prohibited actions.
    5. Obtain the required authorization. Show the proposed change and its consequence whenever the action changes product state or affects other people.
    6. Execute through a narrow tool. Use a typed, allowlisted operation. Make retryable actions idempotent so a repeated call does not create duplicate changes.
    7. Verify the result. Read the resulting product state and look for the defined success event. Tool completion alone does not prove customer resolution.
    8. Record the outcome. Log the context version, playbook, model, policy decision, tool call, result, success signal, and any escalation or user reversal.

    The loop supports two related products without collapsing their permissions. An internal analytics agent can identify an affected cohort, inspect a funnel, or surface a recurring failure pattern. A customer-facing support agent can use the approved finding to help one authenticated user, but it should see only that user’s permitted context and tools. A human support operator should receive the same trace when the agent escalates.

    Keep the shared layer deliberately small: stable identities, canonical metric definitions, governed context fields, outcome events, and versioned playbooks. The analytics agent and support agent can then improve the same system while retaining separate access policies and evaluation criteria.

    Do not automatically convert every observed correlation into a new support action. Let analytics generate a candidate playbook, review the causal logic and risk, test it against known cases, and release it through a controlled experiment. The learning unit is the reviewed playbook, not an unexamined prompt change.

    Choose a first workflow that can prove its own value

    The first pilot should be easy to verify, not merely easy to demonstrate. A conversational answer looks polished even when it does not change the user’s outcome. A narrow configuration or onboarding workflow is usually a better proving ground because eligibility, allowed actions, and success can be defined before launch.

    Score candidate workflows against these criteria:

    • Repeated demand: the same intent or failure appears often enough to justify a reusable playbook.
    • Observable state: the agent can read the prerequisites and current configuration instead of guessing from the user’s description.
    • Clear success: one product state or behavioral event can verify that the problem was resolved.
    • Safe execution: the initial actions are reversible, user-scoped, and unlikely to affect billing, security, data retention, or other users.
    • Short feedback: the primary outcome appears soon enough to support iteration, even if retention is monitored later.
    • Enough eligible traffic: the workflow can support a meaningful experiment rather than a handful of anecdotes.

    Write the pilot contract before the prompt

    A pilot contract forces the product, analytics, support, engineering, and risk decisions into one inspectable artifact. It should specify:

    • the user problem and eligible cohort;
    • the trigger that starts the workflow;
    • the context fields and systems of record;
    • the approved diagnostic branches;
    • the allowed and prohibited actions;
    • the point at which confirmation is required;
    • the precondition and postcondition for each tool call;
    • the success event and observation window;
    • known failure modes and the human handoff rule; and
    • the primary outcome, guardrail metrics, experiment design, and minimum detectable effect.

    Consider an onboarding configuration workflow. The trigger might be a user repeatedly reaching setup without completing it. The context could include entitlement, current configuration, prerequisite status, and the latest validation result. The agent may be allowed to run validation, explain a missing prerequisite, prefill a reversible setting, or launch the next approved step. Resolution requires both the expected configuration state and its corresponding success event. If validation continues to fail, the handoff should include the exact state, error, playbook branch, and actions already attempted.

    Avoid starting with data deletion, broad permission changes, security recovery, billing adjustments, or external communications. Those workflows combine difficult authorization questions with high consequences. Prove context quality, tool reliability, verification, and measurement on a narrower action set before expanding the blast radius.

    Set the minimum detectable effect before the experiment. If the eligible population cannot detect an outcome change that would justify the investment, narrow the claim, combine additional time periods, or choose a more observable workflow. Do not call an underpowered neutral result proof that the agent has no effect.

    Instrument the agent like a product surface, not a transcript

    Conversation volume, message count, and thumbs-up feedback are diagnostic signals. They are not sufficient outcome measures. A customer can like an explanation and still remain blocked; another can dislike the wording even though the configuration was fixed.

    Measurement layerQuestion it answersUseful signals
    Operational reliabilityDid the system execute as designed?Tool success, validation failure, retry, latency, rollback, and escalation
    Verified resolutionDid the requested product state become true?Verified resolution rate, time to resolution, repeat attempt, and repeat contact
    Product outcomeDid the user progress in the journey?Activation, feature adoption, workflow completion, and later retention
    Support outcomeDid the workflow reduce avoidable support effort?Eligible ticket rate, escalation reason, handle-time impact, and handoff quality
    Safety and trustDid the agent stay within policy and user intent?Permission block, wrong-action review, user reversal, policy violation, and privacy incident

    Define the denominators as carefully as the numerators. Verified resolution rate should use eligible support sessions as its denominator and require the success state defined in the pilot contract. Action completion rate should use authorized action attempts, not every conversation. Time to resolution should begin with the original request and stop only when the postcondition is verified, not when the agent finishes generating text.

    Do not optimize ticket deflection or containment in isolation. The absence of a ticket can represent resolution, abandonment, or a user working around the problem. Pair support-efficiency measures with product success, repeat contact, and safety guardrails.

    Use evaluations and experiments for different questions

    A disciplined AI product rhythm connects eval-driven development, A/B testing, minimum detectable effect, activation, retention analysis, and data governance. Each mechanism answers a different question:

    • Pre-release evaluations: Can the system interpret known intents, select the right context, follow policy, choose an allowed tool, handle tool errors, and verify the expected postcondition? Run the relevant suite whenever the model, prompt, context contract, policy, tool, or playbook changes.
    • Shadow operation: What would the agent have proposed in real traffic without being allowed to change state? Review mismatched diagnoses, unsupported context, unsafe actions, and missed escalation conditions.
    • Controlled experiments: Does the agent improve the predefined outcome compared with the existing support experience for the eligible population? Record assignment before the interaction and preserve it through outcome analysis.
    • Production monitoring: Are errors, reversals, escalations, latency, or policy blocks changing by journey, user role, entitlement, playbook, or release version?

    Be careful with naive correlation. Users who invoke support are often already struggling, so their outcomes may look worse than those of users who never needed help. Random assignment among eligible users gives you a defensible counterfactual. When randomization is not possible, describe the result as observational and avoid claiming that the agent caused the change.

    Log enough version information to reproduce a decision: model, prompt, policy, context schema, playbook, experiment assignment, tool version, input identifiers, authorization result, and postcondition. Do not place raw secrets or unrestricted personal data in that trace. A metric change is actionable only when you can connect it to the system version that produced it.

    Set action boundaries before the model receives tool access

    Model confidence is not authority. A highly confident response must never expand a user’s permissions, bypass confirmation, or convert a prohibited action into an allowed one. Authorization belongs in deterministic policy and tool infrastructure outside the model.

    Action classTypical scopeRequired controlsVerification
    Read and explainShow relevant state, explain an error, or recommend a next stepUser-scoped reads, field filtering, and visible uncertainty when evidence conflictsConfirm that the response used current state and an approved knowledge path
    Reversible changeUpdate a non-sensitive preference, run validation, or trigger a recoverable workflowPreview, confirmation when needed, typed input, idempotency, and rollbackRead the resulting state and observe the workflow’s success event
    Consequential changeAlter billing, permissions, security, external communication, or retained dataStrong confirmation or human review, separation of duties, and a complete audit trailVerify every postcondition and provide a safe recovery or escalation route

    Implement the boundary with controls the agent cannot negotiate away:

    • Least-privilege credentials: issue short-lived, user-scoped authorization rather than a general service credential wherever the architecture permits it.
    • Allowlisted tools: expose narrow actions with typed parameters, explicit preconditions, and constrained targets. Do not give a customer-facing agent arbitrary database or shell access.
    • Policy before execution: validate identity, permission, data sensitivity, action class, and confirmation status outside the model before any state-changing call.
    • Postcondition checks: require the agent to read the resulting state. A successful API response can still produce the wrong business outcome.
    • Safe retries: attach idempotency controls to operations that might be repeated after a timeout or interrupted conversation.
    • Complete handoffs: send the human operator the intent, relevant context, diagnosis, attempted action, tool result, and unresolved condition so the customer does not have to start over.
    • Controlled release: use feature flags, cohort restrictions, action-level limits, and an immediate disable path while a workflow is being validated.

    Evaluate build-versus-buy decisions at the system boundary

    Conversation quality is easy to demonstrate and difficult to use as a purchasing criterion. Evaluate an agent platform on whether it can operate inside your context, permission, observability, and experimentation model.

    • Can you define and inspect the context contract for each workflow?
    • Can the platform use user-scoped credentials and enforce tool permissions outside the prompt?
    • Can every decision, action, version, and outcome be exported to your unified analytics platform?
    • Can you separate aggregate analytics access from individual customer support access?
    • Can you run offline evaluations, shadow traffic, controlled experiments, and cohort rollouts?
    • Can you configure confirmation, rollback, handoff, retention, and data-residency policies?
    • Can you change the model, tool, or support system without losing metric definitions and historical outcome traces?

    A platform that generates excellent dialogue but cannot expose its action trace or connect to verified outcomes will make governance and product measurement harder. A less theatrical system with clear contracts may be the more useful product foundation.

    Key takeaways

    • Start with a governed context contract, not a larger prompt or model.
    • Connect product analytics and support through shared identities, metric definitions, outcome events, and versioned playbooks.
    • Give customer-facing agents user-scoped context and a small set of reversible, allowlisted actions.
    • Count a resolution only when the intended product state or success event is verified.
    • Use offline evaluations for capability and policy, controlled experiments for causal impact, and production monitoring for drift and safety.
    • Expand autonomy only after context accuracy, tool reliability, outcome lift, and guardrails have all been demonstrated.

    At your next roadmap review, ask for one pilot contract rather than a broad AI support initiative. Choose one recurring journey, name its verified success event, define the smallest safe action set, and make the owner show how every action will be authorized, observed, and reversed. That is enough to move from a chatbot concept to an agentic product you can manage.

    References

  • Auditable AI Code Review: A Practical Operating Model

    Auditable AI Code Review: A Practical Operating Model

    You are not deciding whether an AI model can find bugs in a pull request. You are deciding whether an automated reviewer can participate in a production control without leaving your team unable to explain, challenge, or reverse its decision.

    If the only evidence behind an approval is a bot comment that says the change looks safe, keep the system advisory. An auditable AI reviewer needs a bounded mandate, a deterministic approval policy, traceable evidence, and a feedback loop tied to production outcomes. Build those controls first, and faster review becomes a consequence rather than a gamble.

    Start with a decision contract, not a model prompt

    An approval is a policy decision. The model can supply findings, evidence, and a recommendation, but it should not define the conditions under which its own recommendation becomes authoritative.

    Write a decision contract before selecting a model or tuning a prompt. It should answer five questions:

    • What may the system decide? Typical outcomes are approve, request changes, provide non-blocking comments, or escalate to a person.
    • Which changes are eligible? Eligibility should be determined by explicit repository, path, change-type, test, ownership, and reversibility rules.
    • Which checks are mandatory? An eligible pull request should not be approved if a required review lens failed to run, returned incomplete evidence, or produced an unresolved blocking finding.
    • When must the system abstain? Missing context, conflicting findings, unavailable tools, excessive scope, low-confidence evidence, and protected code paths should cause escalation rather than optimistic approval.
    • Who owns the result? Name the engineer accountable for the change, the owner of the review policy, and the person or group authorized to change the automation boundary.

    The core approval rule can be expressed plainly: the change is eligible, every mandatory check completed, no blocking issue remains, the evidence record is complete, and no human-review requirement was triggered. Encode that rule in a controller your team can inspect and test. Do not bury it inside natural-language instructions to the model.

    This separation gives you a clean control plane. Review agents analyze the change. A policy engine evaluates their structured results. A narrowly permissioned service performs the approved action. The model never gets to reinterpret the boundary at the moment it encounters a difficult pull request.

    Auditability does not require a future model run to reproduce the same words. Model endpoints, retrieved context, and dependencies can change. It requires the original decision to remain reconstructable from preserved inputs, outputs, policies, tool results, and versions. A skeptical engineer should be able to determine why the pull request was approved without trusting the personality or reputation of the bot.

    Split the review into specialist checks with explicit evidence

    A single prompt asking whether a pull request is safe compresses several different judgments into one opaque answer. Decompose the review so that each judgment has a clear purpose, input set, output schema, and failure mode.

    A practical review pipeline can include these specialist lenses:

    • Problem-definition quality: Is the requested behavior specific enough to review, and are the acceptance conditions testable?
    • Intent alignment: Does the diff implement the stated change without silently expanding or contradicting it?
    • Scope and dependency impact: Which callers, data flows, interfaces, jobs, or services can the change affect outside the edited files?
    • Logical correctness: Do the changed execution paths handle expected states, boundary conditions, and failure paths?
    • Test adequacy: Do the tests exercise the behavior that changed, and did the required checks actually run against the reviewed commit?
    • Security and privacy: Does the change alter trust boundaries, permissions, authentication, secrets, sensitive data handling, or externally controlled inputs?
    • Local engineering guidance: Does the implementation comply with versioned repository conventions, architectural constraints, and known anti-patterns?
    • Deployment and recovery: Can the change be observed, disabled, or rolled back without creating a second unsafe operation?

    Every specialist should return the same minimum structure: a check identifier, pass/fail/escalate status, a concise claim, evidence tied to files or tool output, the applicable rule version, severity, and a recommended action. A finding such as a possible regression is not auditable. A finding that identifies the affected path, explains the conflicting behavior, points to the relevant code, and names the violated policy is.

    Run independent checks before aggregation, and preserve every result even when the final decision is approval. The aggregator may deduplicate findings, but it should not erase dissent. If the intent checker says the change is aligned while the execution-path checker finds a contradiction, route the conflict to a person.

    Review context must extend beyond the visible diff. A seemingly harmless one-line copy change was once found to contradict validation behavior elsewhere in the codebase. That is the kind of defect a diff-only reviewer is structurally unlikely to see. Give relevant checks controlled access to callers, validators, schemas, tests, ownership metadata, and versioned internal guidance, then record exactly which context each check used.

    More context is not automatically better. Retrieval should be targeted and attributable. When a finding depends on an internal rule, capture the rule identifier and version. When it depends on a test, capture the command, commit, status, and output reference. When it depends on inferred execution flow, record the relevant path so a maintainer can inspect it.

    Treat pull-request text, code comments, test fixtures, generated files, and documentation on the changed branch as untrusted data. They can contain instructions designed to redirect an agent. Load approval policy from a protected service or the trusted base branch, not from files the pull request can rewrite. Run proposed code in an isolated environment, mediate tool calls through an allowlist, and keep approval or merge credentials outside the model’s reach. The policy controller should translate a valid decision into an action; the model should never hold the credential that performs it.

    Set the automation boundary with hard eligibility gates

    Do not begin by assigning every pull request a risk score and approving anything below a convenient number. A composite score can hide a disqualifying condition: a tiny authorization change may receive a low size score even though its blast radius is high. Apply hard gates first. Use scoring only to route changes that remain eligible after those gates.

    Common reasons to require human review include:

    • Authentication, authorization, permissions, cryptography, secrets, or trust-boundary changes.
    • Payments, billing, entitlements, destructive data operations, or irreversible migrations.
    • Public API contracts, shared schemas, release infrastructure, or broadly consumed dependencies.
    • A pull request that changes its own review policy, test requirements, ownership rules, or deployment controls.
    • Missing required tests, failed or stale CI results, unavailable analysis tools, or a mismatch between the reviewed commit and the tested commit.
    • Changes spanning too many concerns, components, or execution paths for the approved review envelope.
    • An active incident, an unclear rollback path, or a direct request for human review.

    There is no universal line-count threshold for a small pull request. Derive limits from your architecture and incident history, then version them. A change to a central permission function may be riskier than a much larger isolated test refactor. Scope should include dependency reach and behavior change, not just added and deleted lines.

    A staged authority model keeps the boundary legible:

    ModeWhat the AI reviewer may doWho decides the mergeAppropriate use
    ShadowProduce a private decision record without affecting the pull requestHuman reviewerBaseline evaluation and policy tuning
    AdvisoryPost evidence-backed, non-blocking findingsHuman reviewerMeasuring usefulness and false alarms in normal work
    BlockingRequest changes for narrow, testable policy violationsHuman reviewer after resolutionStable rules with clear evidence and an appeal path
    Bounded approvalApprove only changes that pass every eligibility and review conditionPolicy controller within its delegated scopeValidated low-risk change classes with complete audit records
    Mandatory escalationSummarize evidence and route the changeNamed human ownerSensitive paths, conflicting findings, missing evidence, or any requested human review

    Do not turn bounded approval into an auto-approval quota. Coverage is a result of demonstrated safety, not a target that should pressure teams to weaken eligibility rules.

    One high-frequency engineering environment reports that more than 93% of pull requests across two main codebases are agent-driven and more than 19% are approved without a human reviewer. Its reported median merge time fell from 75.8 minutes with human review to 14.6 minutes with AI approval, while downtime from breaking changes declined 35% as deployments doubled. Those organization-level results show that bounded automation can coexist with high deployment frequency and improving safety outcomes. They do not prove that AI approval caused the downtime reduction, and they should not be imported as another team’s launch threshold.

    Keep the escape hatch explicit. Any engineer should be able to request a human review without defending the choice. The accountable engineer should still watch the change in production and be ready to roll it back. Automated approval changes who performs a review step; it does not transfer ownership of the production outcome to a model.

    Preserve the evidence, then earn autonomy through evaluation

    Build a decision record that survives model and policy changes

    Create an append-only decision event for every review attempt, including abstentions and failed runs. At minimum, retain:

    • Repository, pull-request identifier, base commit, reviewed head commit, author, accountable owner, and timestamps.
    • The pull-request description and acceptance criteria as they existed when the decision was made.
    • Eligibility rules, protected-path rules, ownership data, prompt-template identifiers, and policy versions.
    • Model provider and model identifier, relevant runtime settings, retrieval configuration, and tool versions.
    • The context each specialist received, including immutable references or preserved snapshots for mutable material.
    • Structured specialist outputs, supporting evidence, tool invocations, CI results, conflicts, and failures.
    • The deterministic rule evaluation that produced approve, block, comment, or escalate.
    • Subsequent human overrides, appeals, edits, approvals, merges, rollbacks, hotfixes, and linked incidents.

    Store concise decision rationale and inspectable evidence, not hidden chain-of-thought. An auditor needs to know which claim was made, what supported it, which rule applied, and how the controller reached the outcome. Private internal reasoning is neither necessary nor a reliable substitute for those artifacts.

    Apply the same security discipline to review logs that you apply to source code. Minimize captured secrets and personal data, control access, define retention, and log policy changes. If a model or retrieval service cannot handle the code under your data-governance requirements, that repository is not eligible for the workflow.

    Evaluate decisions, not polished comments

    A review can sound thoughtful and still approve the wrong change. Build an evaluation set around decisions and evidence rather than writing quality.

    1. Assemble representative cases. Include clean pull requests, valuable historical human findings, escaped defects, incident-causing changes, incomplete requirements, sensitive paths, cross-component changes, and attempts to manipulate the reviewer through repository content.
    2. Label the expected control outcome. For each case, identify whether the correct action is approve, request changes, or escalate. Record the evidence that an acceptable review must surface.
    3. Separate clear cases from disputed ones. Known incident causes and explicit policy violations can provide strong labels. Ambiguous architectural judgments need maintainer adjudication, and disagreement should remain visible rather than being forced into false certainty.
    4. Freeze a holdout set. Use one portion to improve prompts, retrieval, and policy. Keep another portion unseen until release evaluation so repeated tuning does not create a misleading score.
    5. Compare equivalent cohorts. Evaluate AI and human review on the same risk classes and change types. Comparing AI-approved low-risk changes with all human-reviewed pull requests confounds reviewer quality with task difficulty.

    Track metrics that expose different failure modes:

    • Decision accuracy: How often did the system choose the expected approve, block, or escalate outcome?
    • False auto-approval rate: How often did it approve a labeled case that should have been blocked or escalated? Break this out by severity and risk class.
    • Blocking precision: Of the findings that stopped a change, how many maintainers judged valid and actionable?
    • Known-defect recall: Which seeded or historically verified defects did the review catch? Label this carefully; it is not recall over every defect that might exist.
    • Evidence completeness: Can every decision be traced to required checks, immutable inputs, policy versions, and supporting artifacts?
    • Abstention and override rates: Where is the system uncertain, and where do engineers reverse it? Investigate patterns by repository and change class.
    • Delivery performance: Measure review latency and merge time, but only alongside quality metrics.
    • Production outcomes: Track rollbacks, hotfixes, escaped defects, incidents, downtime, and customer impact for comparable risk cohorts.

    Comment helpfulness is useful feedback, but it is not a safety metric. Engineers may like a concise reviewer that misses a critical defect, or dislike a strict reviewer that correctly blocks an unsafe change. Keep usefulness, correctness, and production impact as separate measures.

    Roll out by change class and turn escapes into regression tests

    Move from shadow mode to advisory comments, then to narrow blocking rules, and only then to bounded approval. Start with one repository and one low-risk, reversible change class. Write the exit criteria before the pilot begins, including acceptable false-approval and false-block rates, required audit completeness, escalation behavior, and production guardrails.

    Canary each expansion. Maintain a kill switch that disables new automated approvals without removing the accumulated evidence. If a required service, model, retrieval index, test runner, or policy store is unavailable, fail closed and return the pull request to the human path.

    When an approved change causes a production problem, diagnose the control layer that failed:

    • Was the change wrongly eligible?
    • Did retrieval omit relevant code or guidance?
    • Did a specialist miss or misclassify the defect?
    • Did the aggregator suppress a conflict?
    • Did the policy permit approval despite the evidence?
    • Did CI test a different commit or an incomplete environment?
    • Did production monitoring fail to surface the effect promptly?

    Add the case to the regression suite, version the corrective policy or guidance, rerun the holdout evaluation, and preserve the relationship between the incident and the updated control. That is eval-driven development applied to governance: every escape should make a specific layer harder to fail in the same way again.

    Key takeaways

    • AI output is an input to approval, not the approval policy itself.
    • Use deterministic eligibility gates before any model-based risk judgment.
    • Decompose review into specialist checks that return claims, evidence, rule versions, and explicit pass/fail/escalate states.
    • Keep policy and credentials outside the pull request and outside the model’s control.
    • Preserve enough evidence to reconstruct the original decision even when the model, repository, or internal guidance later changes.
    • Expand autonomy only when evaluation and comparable production cohorts support it; never optimize for auto-approval coverage by itself.

    Your first useful milestone is not an AI-approved pull request. It is a shadow decision that a maintainer can reconstruct, dispute, and improve. Once that record is dependable, grant the smallest reversible slice of authority, watch what reaches production, and make every expansion earn its place.

    References

  • Cracking the Hardest Percentages: Turn Complex Support into Scalable, Trust-Building Automation

    Cracking the Hardest Percentages: Turn Complex Support into Scalable, Trust-Building Automation

    I’ve learned that the smallest slice of your support queue often dictates the majority of your operating cost, customer memory, and automation ceiling. In product reviews and CX ops deep-dives, I see the same pattern: the “easy” tickets pad your resolution counts, but the complex, multi-step queries quietly own your handle time and your brand trust. If you care about compounding impact, your customer support AI strategy has to target that hardest percentage first.

    Complex queries are a small percentage of your queue, but they consume a disproportionate share of your team’s time.

    Take a typical queue: password resets outnumber refund disputes ten to one, but a reset takes five minutes and a dispute takes thirty. The “rare” query accounts for over a third of total handling time. The same pattern holds for account investigations, subscription changes, and billing disputes.

    How you handle complex queries is also what customers actually remember about their support experience. When someone is dealing with a damaged order or a billing dispute, the stakes are higher, and a fast, good resolution is what separates a forgettable interaction from one that builds lasting trust.

    Most AI Agents automate the easy, informational queries well. The question for your automation rate is whether they can handle the hard ones. That’s where agentic AI and robust AI workflows make or break your outcomes.

    We’ve gotten really good at informational queries – the hard part is what comes next. I’ve seen teams invest deeply here, and for good reason: it lifts containment quickly and cheaply. But to break through the plateau, you have to execute actions across systems, not just answer with text.

    We’ve invested deeply in informational Q&A. We built Apex, a specialized customer service model trained on billions of support interactions, as Fin’s core answering engine. Beneath that sits a custom retrieval model, a purpose-built reranker, and a unified RAG pipeline, all trained specifically for customer service. Fin resolves issues at a higher rate than general-purpose frontier models, with fewer hallucinations and at lower cost.

    But informational Q&A only covers queries where text is the answer. Most Agents can handle that. Far fewer let you configure complex, multi-step actions without a forward-deployed engineer setting it up for you, which creates a gap.

    Every query your team handles falls into one of three categories:

    Informational: “Can you ship transatlantic by priority next day?” Answered with text from your knowledge base.

    Personalized: “Where is my order?” Requires data unique to that user.

    Action-led: “My order arrived damaged, I need a refund.” Requires doing something: checking a return window, cross-referencing transaction data, making a judgment call – reading from multiple systems and acting across them.

    Dark-themed line chart of percentages from Jan 2026 to Apr 2026. An orange line with circular markers climbs steadily, pauses briefly mid‑period, then spikes sharply to a new high near the end of the timeline.
    From Jan to Apr 2026, the trend moves steadily upward, pausing briefly before a sharp late surge. A clear snapshot of momentum for customer service KPIs, finance results, and the impact of new procedures.

    These complex queries, the ones that require multi-step processes across systems, aren’t edge cases; they’re the reason your support team exists. This is the gap Fin Procedures was built to close.

    It works in practice, and the trajectory matters for product strategy and ops planning.

    Procedures is live, it’s scaling, and the results are clear. Since launching in managed availability, Procedures has handled over 1.5 million conversations, and volume is doubling month over month across hundreds of apps in fintech, e-commerce, gaming, healthcare, and SaaS.

    When customers hit complex, multi-step queries, the experience is dramatically better when Fin can do the work end-to-end. We tested this with a randomized 5% holdout – conversations where Procedures would normally run, but didn’t. CSAT was 28.93% higher when Procedures ran, a statistically significant result.

    A product, not a services engagement. I’ve sat through too many “automation” projects that were really solutions engineering gigs: workshops, custom scripts, then a queue of change requests when policies shift. It’s fragile and slow.

    The B2B AI industry has a consultingware problem. It’s not databases being forked anymore, it’s prompts. The economics of maintaining bespoke setups per customer don’t work. Either the application falls behind new models, or the vendor changes the model and quality degrades invisibly.

    In my view, an agentic AI platform should be a product your team owns end to end: a natural language editor – literally paste your existing SOPs – branching logic, data connectors, and AI-powered simulations for testing. Your CX ops team configures this, iterates on it, owns it. If you need help, a forward-deployed team can assist, but they’re optional, not a dependency. You always have control.

    And because it’s a unified product, improvement compounds. When the vendor optimizes a prompt, every customer’s Procedures get better. When they upgrade the model, they can A/B test across the entire customer base and know it’s better before rolling out. You can’t do that when every customer has a bespoke prompt. The consulting model isn’t just expensive, it’s structurally unable to compound.

    Today, Fin Procedures is available to every Intercom customer – no waitlist or managed rollout, ready for all 8,000+ customers.

    We’re iterating fast based on real customer feedback. Here’s what’s landed since the last major update, and why it matters for reliability and governance:

    AI-powered Procedure review: Flags broken logic, missing references, and unreachable conditions before you deploy.

    Promotional banner reading "Get started with the #1 Agent today" over a dark, aurora-like gradient background, featuring a white button labeled "Start a free trial"; marketing graphic for an AI support agent.
    Kick off your journey with the #1 Agent—an AI partner designed to turn resolutions into real outcomes. Tap “Start a free trial” to explore faster, smarter customer service and see how Fin delivers value from day one.

    Procedure failure reporting: A new reporting dimension that lets you drill into conversations where Procedures failed, so you can diagnose and fix.

    Version history with rollback: Track every change, compare versions, roll back if needed.

    Data connector health monitoring: See at a glance if your integrations are healthy, degraded, or failing.

    Optional data connector parameters: Fin only asks customers for information when it’s actually needed, instead of prompting for every field.

    Email Simulation support: Test how your Procedures behave across chat and email before going live.

    Agent in the Loop (Beta) unlocks the next tranche of automation. Even with Procedures, two things hold teams back from automating their most complex queries: missing integrations and policies that require a human sign-off on sensitive decisions.

    “Agent in the Loop” is built for both. Need Fin to check your internal admin tools but haven’t built a data connector yet? Put a human checkpoint at that step. Fin handles the conversation, gathers context, and pauses, surfacing a structured summary for a human agent to verify or act, then resumes. You get automation on the 80% that doesn’t need the integration.

    For compliance – identity verification, high-value refunds – Fin does the legwork, a human makes the final call and then hands it back to Fin. This works natively in the Intercom Inbox and via Slack. Some competitors don’t have an inbox-native variant at all, meaning humans need to leave their primary workspace to review AI actions.

    Procedures are also built to let you collaborate with all your teammates – both human agents and AI Agents. Fin can work with them directly inside a Procedure, using APIs and webhooks to loop in another teammate mid-flow, hand off context, and pick back up once they’re done.

    Making it easier, faster. Procedures is already self-serve, but the next step is making Procedure creation, testing, and maintenance significantly more streamlined and easy to do, with less manual editing and more AI-assisted building and debugging. There’s a lot coming in this space over the next few months – and it aligns perfectly with a retrieval-first pipeline and stronger governance at scale.

    The hardest percentages matter the most. The biggest unlock for your automation rate won’t be answering more FAQs, it will be handling the complex, multi-step queries that consume your team’s time and define what customers remember about their experience with you.

    That means working with an Agent that goes beyond answering questions and executes processes. A product your team owns and configures, not a service you buy and hope gets maintained. And a platform where every improvement compounds across every customer. That’s Procedures. Available now, for everyone.

    Inspired by this post on The Intercom Blog.

    Book a consult png image