Month: May 2026

  • An AI Operating Model That Measures Outcomes, Not Activity

    An AI Operating Model That Measures Outcomes, Not Activity

    Your AI team is shipping, dashboards are filling up, and executives are still asking the uncomfortable question: what changed for the customer or the business?

    The answer is rarely another model metric. You need an operating model that connects AI quality to customer behavior, workflow performance, commercial results, and risk. When that chain is visible, you can decide what to scale, what to repair, and what to stop.

    Key takeaways

    • Give every AI initiative an outcome contract that names the target behavior, business result, guardrails, and decision owner.
    • Measure four linked layers: AI quality, user behavior, workflow results, and business outcomes.
    • Preserve the context behind each interaction so you can compare outcomes by customer, workflow, model version, and acquisition path.
    • Run one recurring evidence review where teams make explicit scale, fix, hold, or stop decisions.
    • Use the first 90 days to prove a reusable learning system, not merely a functioning AI experience.

    Start each initiative with an outcome contract

    A feature brief tells a team what to build. An outcome contract tells it why the work exists, how evidence will be interpreted, and who can act on that evidence. It is the smallest practical unit of an outcome-led AI portfolio.

    Write the contract before choosing a model or polishing a prompt. Keep it to one page and require six fields:

    • Target workflow: Name the repeated job being changed, such as resolving a support request or preparing a sales follow-up.
    • Target user behavior: Describe what a person should do differently. Adoption alone is weak; successful completion, accepted recommendations, or reduced rework is stronger.
    • Business outcome: Connect the behavior to retention, expansion, qualified demand, service capacity, or another commercial result.
    • Quality floor: Define the task-level evaluation the AI must pass before exposure expands.
    • Guardrails: Name the safety, privacy, latency, reliability, and cost conditions that must remain acceptable.
    • Decision rule: State what evidence will trigger a scale, fix, hold, or stop decision, and name the person accountable for making it.

    A driver tree makes the logic inspectable. Start with the business result, work backward to the customer behavior that can influence it, then identify the product and AI capabilities that can change that behavior. This prevents a model improvement from being mistaken for business progress.

    The contract also gives empowered teams useful boundaries. Leaders align the portfolio around outcomes and constraints; teams retain room to change prompts, retrieval methods, interaction design, or even the proposed solution. That is the practical connection between AI strategy, continuous discovery, evaluation, delivery, and value capture.

    Build one scorecard across four layers

    AI outcome analytics is not a single north-star metric. It is a chain of evidence. If you measure only the beginning of the chain, you learn whether the system produced an answer. If you measure only the end, you may see revenue move without knowing why.

    Measurement layerQuestion it answersUseful examplesTypical decision
    AI qualityDid the system perform the intended task?Task success, groundedness, safety failures, response varianceChange prompts, context, retrieval, model, or fallback
    User behaviorDid a person trust and use the result?Acceptance, correction, abandonment, repeat use, human escalationChange the interaction, explanation, or moment of assistance
    Workflow outcomeDid the job become meaningfully better?Successful completion, rework, cycle time, resolution qualityExpand, narrow, or redesign the workflow
    Business and risk outcomeDid the change create durable value within constraints?Retention, expansion, qualified leads, cost per successful outcome, incidentsScale, repackage, hold, or stop

    Read the layers from left to right. Good AI quality with weak behavior usually points to product design, trust, or workflow placement. Strong usage with no workflow improvement may indicate novelty rather than value. Workflow gains with poor economics mean the experience works but the architecture or packaging does not.

    Use the workflow attempt as the basic unit of analysis whenever possible. A generic session can contain several unrelated intentions. A workflow attempt lets you connect the user request, retrieved context, model and prompt version, response, correction, completion, and downstream result.

    Persist the properties needed to reconstruct that journey. Customer segment, acquisition context, workflow type, entitlement, experiment group, model version, retrieval version, and human-handoff status often matter more than another page-view event. Carrying critical context across visits lets you trace behavior from early exploration to conversion and expansion instead of losing the causal story at signup.

    Keep the event taxonomy small enough to govern. Instrument decisions and state changes, not every interface movement. For each event, document its owner, trigger, required properties, prohibited sensitive data, and validation method. A dashboard built on ambiguous events creates confidence without clarity.

    Run a weekly loop from evidence to decision

    Analytics creates value only when it changes a decision. Give each AI initiative a recurring evidence review attended by the product trio and the engineering, data, risk, operations, or go-to-market partners needed for that workflow.

    1. Check the contract. Reconfirm the target workflow, primary outcome, evaluation floor, and guardrails. If the goal has changed, update the contract before interpreting the data.
    2. Inspect the scorecard. Review AI quality, behavior, workflow, business, risk, and cost in that order. Look for breaks in the chain rather than averaging them into one health score.
    3. Segment the result. Compare the cohorts that could conceal a failure: new and experienced users, customer tiers, workflow types, channels, experiment groups, and system versions.
    4. Review failure cases. Sample unsuccessful attempts and classify the reason: missing context, poor retrieval, incorrect generation, confusing interaction, policy restriction, latency, or a problem outside the AI system.
    5. Make one portfolio decision. Choose scale, fix, hold, or stop. Record the evidence, owner, next test, and condition for revisiting the decision.

    Do not let offline evaluations and online analytics compete. Offline evaluations test whether a candidate change can handle representative tasks and known edge cases. Online measures show whether the released experience changes real behavior under real conditions. A candidate should clear the evaluation floor before broader exposure, then earn expansion through customer and business evidence.

    When you run an experiment, agree on the hypothesis, primary outcome, guardrails, minimum detectable effect, and stopping rule before looking at results. Feature flags and progressive rollout keep the decision reversible. If the result is ambiguous, improve the test or narrow the population; do not promote the most flattering proxy.

    This rhythm makes learning rate operational. The useful question is not how many experiments ran. It is how many consequential uncertainties were resolved and converted into a product, portfolio, or go-to-market decision. Testable decisions, behavioral analytics, and guarded rollouts make speed credible because the evidence can survive scrutiny.

    Assign decision rights before the dashboard turns red

    AI products cross boundaries that ordinary feature teams can often ignore. Product owns the customer and business outcome. Engineering owns service behavior and remediation. Data or AI teams own evaluation integrity and model observability. Risk, security, legal, and operations own constraints that cannot be traded away informally.

    Write those responsibilities into the operating model. For each risk tier, specify who can approve an initial release, expand exposure, pause the system, change a model or retrieval source, accept a temporary exception, and communicate an incident. A named decision owner is more useful than a large committee with shared accountability.

    Governance should begin during discovery. The team can then choose acceptable data, design consent, build traceability, create fallbacks, and define escalation paths before those choices become expensive. Model cards, data records, evaluation results, release history, and incident decisions should form one audit trail rather than separate compliance paperwork.

    The same principle applies to commercial decisions. Product, finance, sales, and customer success need a shared definition of value. Measure inference and support costs against successful workflow outcomes, not raw requests or tokens. Packaging can then reflect delivered value while protecting margins and avoiding incentives for wasteful usage.

    Use simple decision tests:

    • Scale when the primary outcome improves, quality and safety floors hold, economics remain acceptable, and the result repeats in the intended cohorts.
    • Fix when the chain reveals a local weakness, such as adequate AI quality but low acceptance, or strong adoption but excessive rework.
    • Hold when the evidence is inconclusive, the measurement is unreliable, or a guardrail is close enough to its limit that broader exposure would create avoidable risk.
    • Stop when only proxy metrics improve, the target workflow does not change, or the value depends on manual intervention that cannot be sustained.

    Use 90 days to prove the operating system

    Your first 90 days should produce more than a working use case. They should leave behind a repeatable contract, event model, evaluation set, rollout path, governance record, and decision cadence that the next team can reuse.

    1. Weeks 1–2: choose the workflow. Audit available content and data, map the highest-value repeatable workflows, and select one where behavior and business impact can be observed. Write the outcome contract and assign decision rights.
    2. Weeks 3–4: define the evidence. Build the driver tree, establish the four-layer scorecard, create representative offline evaluations, classify risk, and document the release and stop conditions.
    3. Weeks 5–8: build and instrument. Create the retrieval and prompt baseline, capture lineage and version context, validate events, implement observability, and test graceful fallbacks. Rehearse how the team will diagnose a failed attempt.
    4. Weeks 9–12: release and learn. Ship behind a feature flag, begin with limited exposure, compare behavior and outcomes, inspect failure cohorts, and make explicit scale, fix, hold, or stop decisions.

    At the end, ask for three forms of proof. Can the team explain which customer behavior changed? Can it connect that behavior to a workflow and business result without hand-waving? Can another team reuse the operating artifacts without rebuilding them from scratch?

    If any answer is no, keep the rollout narrow and repair the system of learning. If all three are yes, fund the next workflow using the same operating model. The goal is not a larger collection of AI features. It is an organization that can turn uncertain AI capabilities into measurable outcomes, repeatedly and responsibly.

    References

  • AI-Ready Customer Support: An Operating Model That Works

    AI-Ready Customer Support: An Operating Model That Works

    You may already have an AI agent, a vendor shortlist, or pressure to automate more tickets. But if your policies conflict, ownership is unclear, and agents routinely rely on tribal knowledge, adding AI will expose those weaknesses at customer speed.

    The practical goal is not to make every ticket autonomous. It is to build a support operation in which AI can resolve the right issues, recognize when it lacks authority or information, and help your team improve the system after every failure.

    Key takeaways

    • Start with a bounded customer problem, not a general mandate to automate support.
    • Treat knowledge as a controlled production input with owners, audience rules, and review triggers.
    • Define acceptable outcomes, prohibited actions, and escalation conditions before configuring the agent.
    • Preserve a middle path where a human can unblock the AI without taking over the entire conversation.
    • Expand automation only when evaluation results, live outcomes, and operational ownership support it.

    Start with the queue, not the model

    An AI-ready operation begins with a resolvable job. “Handle customer support” is too broad. “Help authenticated customers update their billing details under the current policy” is something you can document, test, monitor, and constrain.

    Choose an initial queue where demand is meaningful, the desired outcome is clear, and the governing policy is reasonably stable. Avoid starting with cases that depend on negotiation, undocumented exceptions, or several teams making judgment calls behind the scenes. Those cases may become suitable later, but they are poor places to learn basic operational control.

    Review a representative slice of conversations from that queue. For each one, record the customer’s intent, the information required, the systems touched, the policy applied, the final outcome, and any human judgment that changed the path. This turns a pile of tickets into a resolution map.

    Pay special attention to cases that look identical at first but require different actions. A refund request may depend on plan type, purchase date, account state, or a regulatory restriction. These branches are where a fluent answer can still be operationally wrong.

    You also need to decide where AI will sit. In most established operations, the safer path is to work through the support systems, queues, and reporting practices your team already uses. Replacing the help desk and automating the work at the same time creates two migrations and makes failures harder to diagnose.

    Turn knowledge into a controlled production input

    Your help center is only one part of the answer set. Reliable support may also depend on internal runbooks, policy clarifications, troubleshooting steps, approved reply snippets, product limitations, escalation instructions, and information held by product or customer success teams.

    Bring those materials into a governed knowledge inventory. Every record should answer seven operational questions:

    <!– wp:list {
  • How to Design a Dependable CLI Agent Users Can Trust

    How to Design a Dependable CLI Agent Users Can Trust

    Your CLI agent can look impressive in a controlled demo and still feel unsafe in a real repository. The moment it can edit files, invoke tools, or use credentials, users need to understand what it will do before they let it proceed.

    The dependable design is rarely the one with the most capabilities. It is the one with the smallest clear promise, predictable execution, visible controls, and evidence that it succeeds repeatedly.

    Define the boundary before you define the features

    Start by writing an operating contract for the agent. This is a product decision, not a prompt-writing exercise. A useful contract answers five questions:

    • What job does the agent complete?
    • Which resources and tools may it use?
    • What must it never do?
    • Which actions require explicit approval?
    • What observable result counts as success?

    Keep the job narrow enough to explain in one sentence. If the description needs a collection of exceptions, the interface is already carrying too much ambiguity. Split the work into a clearly named subcommand or make the advanced behavior opt-in.

    Treat every flag, tool, and permission as an increase in blast radius. A new option does not merely add flexibility. It creates another state the agent can misunderstand, another path you must test, and another behavior the user must learn. Reducing the surface area can improve repeatability and trust because both the agent and the user have fewer possible paths to reason about.

    When reviewing a proposed capability, ask whether it makes the mental model smaller. If it does not, remove it, defer it, or isolate it behind progressive disclosure. Safe, fast defaults should handle the common case without demanding that a new user understand the entire system.

    Design one boring, observable execution path

    A dependable run should feel like a transaction with recognizable stages. The model can help interpret intent, but it should not invent the execution contract as it goes.

    • Capture intent: Ask only for information required to resolve the task. If a missing choice would materially change the result, stop and ask.
    • Retrieve context: Fetch the smallest relevant set of files, facts, or records. More context can introduce conflicting instructions and distract the agent from the requested change.
    • Show the plan: Present a compact description of the intended actions, affected targets, and likely side effects.
    • Preview when useful: Provide a dry run for operations whose effects the user should inspect before execution.
    • Execute through narrow tools: Give each tool a deterministic input and output contract. Reject malformed responses instead of guessing what they meant.
    • Verify the result: Check the resulting state and tell the user what changed, what did not, and whether any step failed.

    The agent should stop when the requested scope changes, required context is unavailable, or a tool returns an unexpected result. A visible stop is easier to recover from than confident improvisation.

    Favor idempotent operations wherever you can. Repeating an idempotent action produces the intended state without duplicating or compounding its effects. That property matters in a CLI because interrupted runs and retries are normal operating conditions. Test the second run as deliberately as the first.

    Put human control at the blast-radius boundary

    Do not ask for approval at every step. Constant prompts train users to approve without reading. Place confirmation gates where the consequence or scope changes.

    • Read-only work: Make inspection and planning the default where possible.
    • Scoped writes: Request access only to the specific project, service, or resource needed for the task.
    • Destructive actions: Require a separate confirmation that names the target and explains the consequence.
    • Credentials: Use narrowly scoped, time-bounded access rather than broad credentials that persist beyond the run.
    • Expanded capability: Let users opt into advanced tools instead of quietly enabling them for every session.

    A confirmation message should help the user make a decision. Replace a generic question such as “Continue?” with a concrete statement of what will be changed and whether it can be undone.

    Reversibility should shape the underlying implementation as well. Prefer changes that can be represented as a patch, show the proposed difference before applying it, and preserve enough information to explain how to undo the operation. When reversal is impossible, make that fact visible before execution.

    Use a simple review question for each workflow: can a user predict the maximum consequence of saying yes? If the answer is unclear, the permission boundary is too broad or the confirmation arrives too late.

    Prove reliability before expanding the roadmap

    Do not use capability count as the measure of progress. Before adding a feature, define the task it should complete, the success threshold it must meet, and the smallest interface needed to test it. This turns roadmap discussions into observable product decisions.

    Evaluate at least three outcomes: task completion, time to first successful result, and stability when the same operation is run again. A capability that succeeds once but behaves differently on a retry is not ready merely because the first demonstration worked.

    Instrument each run with Agent Analytics. Capture the input, tools selected, duration, outcome, and error pattern. Review those signals to find where the agent asks unnecessary questions, repeats tool calls, loses users, or encounters the same failure. The response may be a smaller prompt, a tighter tool contract, a safer default, or the removal of a confusing option.

    Documentation belongs in this reliability loop. Keep runnable examples alongside the code and make them reflect the golden path. Treat any mismatch between documented behavior and actual behavior as a product defect. If the workflow cannot be explained and demonstrated simply, it is not yet a dependable workflow.

    Use these evaluations as promotion gates. Add power only after the current path is measurable, understandable, and stable. That discipline earns you the right to expand without turning the CLI into a collection of loosely related agent behaviors.

    Key takeaways

    • Write the agent’s operating contract before choosing its tools or refining its prompt.
    • Keep the default workflow narrow, safe, fast, and explainable in one sentence.
    • Retrieve minimal context, show a compact plan, execute through deterministic contracts, and verify the result.
    • Place explicit approval at destructive, irreversible, or scope-expanding boundaries.
    • Measure completion, time to first success, and rerun stability before adding another capability.
    • Use run telemetry and executable documentation to decide what to simplify next.

    Choose one golden-path task and write its operating contract now. Then run it twice: once normally and once as a retry. Every surprise you find is a reliability requirement to resolve before you broaden the agent’s reach.

    References

  • Analytics-Led Growth Engineering: A Practical Operating Model

    Analytics-Led Growth Engineering: A Practical Operating Model

    Your team has dashboards, event data, and a backlog of growth ideas. Yet decisions still come down to whoever has the strongest opinion, and experiment results rarely change the roadmap.

    The missing piece is usually not another analytics tool. It is an operating model that connects user behavior to a decision, a controlled release, and a measurable business result. Here is how to build one.

    Start with a growth constraint, not a dashboard

    Analytics-led growth begins with a constraint you want to remove. A broad instruction such as improve onboarding gives your team too much room to produce activity without progress. Frame the problem as a break in the user journey instead: qualified users reach the setup screen but fail to complete the action associated with first value.

    Connect that problem to your North Star metric through a driver tree. If the North Star depends on retained active accounts, its drivers might include the number of activated accounts, how frequently they return, and how deeply they use the product. Each driver can then be decomposed into observable behaviors.

    This prevents a common mistake: optimizing the easiest metric to move rather than the metric that matters. More tooltip clicks are not useful if they do not increase successful setup. Higher setup completion is still questionable if those users never return.

    Before opening your analytics platform, write down four things: the user segment, the behavior that is breaking, the outcome it should influence, and the decision you will make if the signal changes. If you cannot name the decision, you are probably requesting a report rather than investigating a growth opportunity.

    Build an evidence chain you can trust

    A growth team needs to trace the path from exposure to durable value. That requires more than counting page views. Instrument the events that represent intent, progress, successful value delivery, and return behavior.

    For every important event, define who triggered it, what object it affected, where it occurred, and whether it represents an attempt or a successful outcome. A generic event such as integration clicked cannot tell you whether the connection worked. Separate the attempt, completion, failure, and first successful use.

    Then inspect the journey through three complementary views. Funnel analysis shows where users stop progressing. Cohorts reveal whether the problem is concentrated among particular acquisition channels, plans, roles, or use cases. Retention analysis tests whether an apparent activation gain survives after the initial session.

    Behavior alone will not explain motivation. Pair the quantitative signal with customer interviews, support conversations, or session-level evidence. If a funnel shows that users abandon a configuration step, qualitative evidence can distinguish confusing language from missing permissions, weak intent, or a technical failure.

    Treat instrumentation defects as product defects. An event that fires twice, changes meaning, or omits a critical property can send engineering effort toward the wrong problem. Assign an owner to each decision-critical event and verify it across the full journey before using it to approve a rollout. Reliable behavioral analytics, cohorting, and funnel analysis are the foundation of this operating model, not a reporting layer added after release.

    Turn every growth idea into an experiment contract

    An experiment should begin with a falsifiable claim. Use this structure: for a defined user segment, changing a specific part of the experience should change a target behavior because it removes an identified barrier.

    Complete the contract before implementation. Name the primary success metric, the guardrails that must not deteriorate, the expected direction of change, and the minimum detectable effect. The MDE forces a useful product decision: what is the smallest improvement that would justify shipping and maintaining this change?

    Power considerations belong in planning, not in the explanation written after results arrive. If the eligible audience cannot produce a credible read on the effect that matters, change the experiment. You can target a higher-signal segment, test a stronger intervention, choose a more responsive leading indicator, or treat the release as a qualitative learning exercise rather than claiming a statistical win.

    Pre-commit to the decision rules as well. A positive primary metric with damaged guardrails should not become an automatic launch. A neutral result can still eliminate a weak theory. A surprising segment difference should become a new hypothesis, not an invitation to search repeatedly for a favorable slice of the data.

    This discipline changes backlog quality. Ideas compete on the strength of their evidence, the importance of the driver they address, and the clarity of the learning they can produce. The roadmap becomes a portfolio of testable growth mechanisms rather than a list of requested features.

    Use staged releases to separate learning from risk

    Feature flags let you control exposure without tying every decision to a new deployment. Start with internal validation, expose the change to an eligible cohort, watch technical and user guardrails, and widen access only when the evidence supports it.

    Keep three decisions distinct. The first is whether the change works as designed. The second is whether it improves the intended user behavior. The third is whether that behavior produces a lasting outcome. Passing the first decision does not answer the other two.

    Onboarding illustrates the difference. A clearer tooltip may increase interaction with a setup control. An in-app guide may increase completion of the setup flow. Neither result proves that users reached value or formed a durable habit. Follow the exposed cohort through the activation event and into retention before declaring the intervention successful.

    Small, reversible changes are especially useful here. Progressive disclosure, revised UX writing, a better default, or guidance at a predictable stall point can isolate a mechanism more clearly than a full onboarding redesign. When several elements change together, you may see movement without learning what caused it.

    Make the product trio accountable for learning

    Growth engineering is not an analytics team handing insights to a delivery team. Product, engineering, and design should jointly own the hypothesis, the intervention, the instrumentation, and the interpretation.

    Product connects the opportunity to the growth model and defines the decision. Design identifies the user friction and shapes the smallest credible intervention. Engineering validates event behavior, controls exposure, and protects reliability. All three inspect the outcome together.

    Close each experiment with a short decision record. Capture what you believed, what changed, which users were exposed, what happened to the primary metric and guardrails, what you decided, and which assumption changed. Record neutral and negative results as carefully as wins. Otherwise, old ideas return with new wording and consume another cycle.

    Leaders should review the quality of this learning system, not just the number of tests shipped. Notice whether teams are testing consequential hypotheses, whether events remain trustworthy, whether results lead to explicit decisions, and whether short-term activation gains are being checked against retention. Experiment volume without decision quality is another output metric.

    Key takeaways

    • Define the broken user behavior and the decision it affects before opening a dashboard.
    • Connect activation, depth, and frequency to your North Star through a driver tree.
    • Specify the hypothesis, primary metric, guardrails, MDE, and decision rules before implementation.
    • Use feature flags and staged exposure to manage risk while preserving a valid learning loop.
    • Validate leading indicators against retention, and store every result in a reusable decision record.

    Choose one important journey this week and trace it from first intent to retained value. If the events, ownership, or decision rules break anywhere along that path, fix that link before adding another growth experiment. Compounding growth begins with compounding clarity.

    References

  • Speed-to-Lead Is Dead: How AI Agents End the Wait and Rebuild a High-Velocity Sales Org

    Speed-to-Lead Is Dead: How AI Agents End the Wait and Rebuild a High-Velocity Sales Org

    A prospect lands on our site, skims pricing, watches a demo, and clicks “contact sales.” For years, that’s where momentum died. They waited, and we built entire sales motions around managing that delay.

    We optimized for “speed-to-lead,” made it the hallmark of a high-performing sales development org, hired more SDRs, tuned routing rules, added shift coverage, and stared at response-time dashboards. Typical SLA targets were one hour for best-fit leads, four hours for core MQLs, forty-eight hours for everyone else. Those were considered good numbers.

    No one questioned the premise because the lag felt structural—shift scheduling, routing delays, and humans working 9–5. The fastest teams could only shrink the gap; nobody could remove it.

    An AI Agent closes it completely.

    When a prospect arrives today, the conversation can begin immediately. That single change reshapes how I design a sales org—how we staff it, what our team prioritizes, and the metrics we hold ourselves accountable for.

    Step outside our dashboards and look at the buyer experience. We spend heavily to drive traffic, then push visitors into forms and queues that add friction precisely when purchase intent peaks.

    Intent is highest the moment someone seeks out our product. If an SDR follows up two or three hours later, that buyer’s in another meeting, the urgency has faded, and the moment is gone. We still call it a lead; the buyer has already moved on.

    What AI changes

    Agents eliminate the structural constraints that made speed-to-lead a problem—shift scheduling, routing delays, CRM batch processing, the SDR being on another call. None of it applies anymore because every single lead can be engaged immediately, at any hour and in any language.

    The impact goes beyond response time. When an Agent engages at peak intent, qualification, discovery, and even an initial demo moment can unfold in a single, continuous conversation. The gated funnel collapses. There’s no reason to qualify someone today, schedule discovery for Thursday, and demo the following week when the conversation is already happening.

    The constraint the industry built around simply isn’t there anymore. We’re already seeing it with Fin, a Customer Agent. As sales leaders, we need to frame this differently.

    If speed-to-lead is no longer the constraint, the knock-on effects reach every part of the org.

    Minimalist hero graphic with the headline 'Add Fin to your sales team today,' a glossy 3D blue spiral at center, and a black 'Start free trial' button, promoting Fin for Sales as an AI customer agent.
    Introduce Fin for Sales to your team with this clean hero banner: bold headline, signature blue spiral, and a clear 'Start free trial' call to action—inviting readers to explore an AI customer agent built for revenue.

    SDRs focus on moving deals forward. Instead of frontline triage, they double down on phone-based selling and relationship building, complex deal navigation, and multi-threaded engagement across stakeholders—the high-leverage work that used to get crowded out by the inbox.

    Pipeline gets more relevant. The old model rewarded volume: capture as many form fills as possible, respond fast, and sort quality later. When an Agent engages at the moment of intent, it qualifies during the conversation. Low-fit leads get filtered out before they reach the team, and high-fit prospects arrive with context—needs, timeline, stakeholders—instead of just a name and email.

    You measure outcomes, not response time. When first response is instant, different metrics matter. I anchor on three questions:

    1) Is the Agent doing the work? Completion rate, qualification rate, and contact capture rate indicate whether conversations reach clear outcomes and produce usable handoffs to the team.

    2) Is the work producing pipeline? Meetings booked and pipeline created through Agent-handled conversations are the leading indicators of revenue, not how fast someone followed up.

    3) Are buyers having a good experience? Conversation-level satisfaction matters more than ever because the Agent is the first interaction prospects have with your company. The experience it delivers is the first impression you make.

    These three questions reveal whether the motion is working. Time-to-first-response can’t.

    Sales orgs built hiring plans, workflows, and performance metrics around beating intent decay. That made sense when the lag was unavoidable. It isn’t anymore.

    An Agent is always on. It engages the moment a prospect arrives on your site, qualifies them in real time, and routes them to the right outcome without waiting for someone to be free. The lag the industry built itself around doesn’t exist when the conversation starts immediately.

    The companies leaning into this are investing in what happens after the conversation starts: how well the Agent qualifies, where it creates pipeline, and what SDRs should actually spend time on. What matters now is not how fast you respond, but what the conversation produces.

    Speed-to-lead made sense when the delay was structural. It isn’t anymore. If you’re re-architecting go-to-market, instrument Agent Analytics, revisit SDR charters, and tighten CRM integration so every qualified handoff is instant, traceable, and revenue-linked.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • A Product Leader’s Playbook for Humane, Sustainable Growth

    A Product Leader’s Playbook for Humane, Sustainable Growth

    Your growth dashboard can be green while your product is becoming less valuable to the people who use it. Activation rises. Engagement deepens. Revenue follows. Yet customers feel pressured, workers absorb hidden costs, or automation removes the human contact that made the experience trustworthy.

    You don’t have to choose between humane technology and commercial performance. You do need an operating model that treats human outcomes as product outcomes, exposes harmful trade-offs early, and rewards durable value rather than extraction.

    Start with the harm your growth model could create

    Most growth models describe the path from acquisition to revenue. A humane growth model also describes who could be worse off if that path succeeds.

    Map the product’s intended value first: the problem a person wants to solve, the moment they receive a useful result, and the reason they would return. Then examine the same journey from the perspective of people who may not appear in your analytics. That can include a customer’s employees, contractors who deliver the service, family members affected by the product, local businesses, or people excluded by the design.

    Create an impact ledger for the growth surface you are reviewing. Keep it beside the business case, not in a separate ethics document that nobody consults during prioritization.

    Impact areaQuestion to answerSignal to monitor
    User agencyCan people understand the choice, refuse it, reverse it, and leave?Overrides, cancellations, reversals, and interview evidence
    Well-beingDoes additional use help people finish their intended task, or merely keep them present?Successful outcomes, passive time, and expressions of regret
    Economic fairnessWho captures the value, and who absorbs the labor, risk, or cost?Complaints, payout concerns, and changes in burden across participants
    Human connectionDoes the experience strengthen useful relationships or replace them unnecessarily?Human handoffs and feedback from affected communities
    Trust and safetyDo people know when automation is involved and what happens to their data?Escalations, corrections, safety reports, and trust feedback

    The ledger is not an attempt to predict every consequence. It is a way to make foreseeable trade-offs visible before a team becomes committed to a launch. This matters commercially as well as ethically: extractive growth can weaken trust and retention while increasing regulatory and reputational exposure.

    Pair every growth metric with a human countermetric

    A metric becomes dangerous when the team can improve it while making the customer’s life worse. Engagement is the familiar example. More time in a product may indicate value, confusion, dependency, or difficulty leaving. The number alone cannot tell you which.

    Give each primary growth metric a countermetric that protects the outcome you actually intend. The pair should appear in the same experiment brief and the same review meeting.

    Growth metricHuman countermetricDecision it improves
    ActivationCompletion of the customer’s intended outcomeWhether setup creates value or only reaches an internal milestone
    EngagementIntentional task completionWhether additional use is productive or merely prolonged
    RetentionTrust, voluntary continuation, and ease of exitWhether customers stay because the product remains useful
    ConversionComprehension of price, consent, and commitmentWhether revenue depends on informed choice
    Automation rateCorrection, reversal, and human-escalation successWhether efficiency survives real-world exceptions

    Do not combine the pair into a single score too quickly. A blended score can conceal the exact trade-off leaders need to see. Review both trends and ask whether the business result would still be desirable if the countermetric deteriorated further.

    Set the stopping condition before running an experiment. Decide which trust, safety, fairness, or agency signal would block rollout even if the primary metric improves. A guardrail invented after seeing strong conversion is rarely a real guardrail.

    Expand discovery beyond the people who already love the product

    Power users are good at explaining how to improve the experience they have accepted. They are less able to represent people who abandoned it, avoided it, could not access it, or carry costs without being the buyer.

    Add an outside-in lane to continuous discovery. Include customers who reduced usage or left, people who encountered a failed automation, front-line workers affected by the workflow, and community members who experience consequences without controlling the purchase. Treat these conversations as product discovery, not public relations.

    Ask questions that reveal displacement and dependency: What became easier? What became harder? What did this replace? When did you feel unable to make a meaningful choice? Who else had to change their behavior so you could receive the benefit? What would a responsible version of this experience preserve?

    Bring the evidence into roadmap decisions in its original shape. A complaint about loss of control should not be translated into a generic request for better usability. A contractor describing unfair risk is not reporting a minor service defect. Name the underlying impact so the team can address the product model rather than polish its interface.

    Put humane constraints inside the experiment

    Principles have little effect if they enter the process after pricing, interaction design, and technical architecture are settled. Put them into the experiment before the team writes production code.

    1. State the human outcome. Describe what should become better in the person’s life or work, not merely what behavior should increase.
    2. Name the affected groups. Include non-users who supply labor, absorb risk, or experience downstream effects.
    3. Define meaningful choice. Specify how people will understand automation, decline it, correct it, and reverse important actions.
    4. Design the failure path. Decide how a person reaches human help when the system is uncertain, unsafe, or wrong.
    5. Pre-commit to a stopping rule. Record which negative signal pauses expansion regardless of the growth result.

    For AI products, this is where risk management becomes part of product management. Give users enough information to understand when AI is acting. Preserve review for consequential outputs. Build correction and escalation into the main workflow. Apply privacy-by-design while deciding what data the product needs, rather than after collecting everything that might be useful.

    The product trio should own these decisions. Legal, security, trust, and policy partners can strengthen the work, but they cannot compensate for a roadmap whose incentives reward harm. The product leader remains accountable for the whole system being optimized.

    Choose durable depth over indiscriminate scale

    Scale is not proof of value. It is an amplifier. If the operating model depends on weak consent, hidden costs, unfair labor, or the removal of every human interaction, scale magnifies those weaknesses.

    A narrower product can create a stronger business when the team understands a community deeply enough to solve its full problem. A locally focused mobility service, for example, could optimize for rider safety, driver economics, and neighborhood usefulness rather than treating every participant as an interchangeable unit of supply or demand. The market is smaller by design, but the value proposition can be clearer and trust can become part of the product’s advantage.

    Test the durability of your strategy with a simple question: if customers become better informed and cultural expectations become stricter, does the growth model become stronger or weaker? A group of German primary-school parents collectively chose to delay smartphones until age 11 or 12. Product leaders should expect social norms to change, sometimes in direct opposition to adoption assumptions embedded in a forecast.

    At the next roadmap review, challenge any initiative that needs customers to misunderstand a choice, remain dependent, or accept worsening treatment as the company grows. If removing that mechanism destroys the economics, you have found a strategy problem, not an optimization problem.

    Key takeaways

    • Document who could be harmed by a successful growth initiative, including people who never appear in the customer database.
    • Pair activation, engagement, retention, conversion, and automation metrics with measures of outcomes, agency, trust, and recovery.
    • Include former users, affected workers, and non-buyers in continuous discovery.
    • Define consent, correction, escalation, and stopping conditions before launching an experiment.
    • Prefer a focused market with durable value over scale that depends on hidden human costs.

    Start with the growth initiative carrying the greatest human risk. Add its impact ledger and countermetric to the next decision meeting, assign an owner, and make expansion conditional on both business value and human value holding up.

    References

  • Prompt Engineering for Amplitude Global Agent That Holds Up

    Prompt Engineering for Amplitude Global Agent That Holds Up

    You ask Amplitude Global Agent why activation fell. It returns a plausible explanation, but you still can’t tell which events it examined, whether the comparison was valid, or what your product team should do next.

    The fix is to treat the prompt as an analysis specification. Define the decision, provide the relevant analytics context, constrain unsupported conclusions, and make the agent show its work. You will get an answer that is easier to verify and more useful in a product review.

    Start with the decision, not a broad request for insights

    Requests such as “analyze activation” leave several decisions unresolved. The agent must guess what activation means, which users belong in the analysis, which period matters, and what kind of answer you expect. Even a polished response may answer the wrong question.

    Before writing the prompt, complete this sentence: “After reading the answer, we need to decide whether to…” Your ending might be “change the onboarding sequence,” “investigate a recent release,” or “prioritize one segment for discovery.” That decision gives the analysis a destination.

    Then assign a role that matches the work. “You are a product analyst investigating activation performance” is more useful than “You are a helpful assistant.” Add the audience as well. An executive needs the size and business relevance of a change; a product trio also needs the affected steps, segments, and follow-up questions.

    A strong opening contains three elements:

    • Role: the analytical perspective the agent should take.
    • Decision: what the team will choose or investigate after reading the result.
    • Success criteria: what the answer must establish before it is useful.

    For example: “You are a product analyst helping the onboarding team decide whether to redesign a weak activation step. Identify the largest meaningful drop-off, show which defined segment is most affected, and separate measured findings from possible explanations.”

    Give the agent a compact analytics contract

    The most reliable prompt names the data the agent may use. Include the relevant event names, property names, segment definitions, filters, and timeframe. If activation has an internal definition, write it out rather than relying on the agent to infer it.

    This is a retrieval-first approach: put authoritative definitions, dashboard context, and prior query logic into the request before asking for interpretation. Concrete grounding reduces room for invented assumptions and makes repeated analyses easier to compare. A structured prompt can also specify the role, business objective, allowed data, and output fields.

    Prompt elementWhat to provideWhat it prevents
    Metric definitionThe exact event sequence or outcome that countsA different interpretation of activation or retention
    PopulationIncluded users or accounts and explicit exclusionsComparisons across unlike populations
    SegmentsNamed properties and the values to compareArbitrary segmentation
    TimeframeThe analysis period and comparison periodHidden or inconsistent date choices
    Evidence boundaryThe events, properties, definitions, and dashboards allowedUnsupported claims presented as measured facts
    Output contractRequired sections, fields, ordering, and lengthA long narrative that cannot be reviewed quickly

    Do not dump every available definition into the context. Include only what the question requires. More context is useful when it resolves ambiguity; irrelevant context competes for attention and makes the prompt harder for a teammate to audit.

    Use a reusable prompt that exposes uncertainty

    You can adapt the following structure for activation, retention, anomaly investigation, or another behavioral analysis:

    1. Role and audience: “Act as a product analyst. Write for the product manager and analytics lead responsible for [area].”
    2. Decision: “Help us decide whether to [decision].”
    3. Question: “Determine [specific analytical question].”
    4. Definitions: “For this analysis, [metric] means [explicit event or outcome definition].”
    5. Data context: “Use these events: [names]. Use these properties: [names]. Compare these segments: [definitions]. Analyze [timeframe] against [comparison period]. Apply [filters and exclusions].”
    6. Constraints: “Use only the supplied Amplitude analytics events, properties, and definitions. Do not treat an unmeasured explanation as a finding.”
    7. Output: “Return the metric result, segment comparison, timeframe, evidence, interpretation, confidence or limitation, and recommended next check.”
    8. Fallback: “If the available data cannot answer the question, state what is missing and provide the smallest follow-up query needed.”

    The fallback matters. Without it, the agent has an incentive to complete the requested narrative even when the evidence is incomplete. A useful failure is specific: it identifies a missing event, undefined property, absent comparison, or ambiguous metric. Your team can fix that. A confident guess is harder to detect.

    Ask for measured findings, interpretations, and recommendations as separate fields. A measured drop-off is evidence. A claim that users were confused is an interpretation unless the supplied data establishes it. A recommendation to inspect session replay or conduct customer interviews is a next step, not proof of the cause. Keeping those layers separate makes the result safer to use in prioritization.

    Turn prompt quality into a small product evaluation

    Do not judge a prompt by whether one response sounds intelligent. Save the prompt version, input context, and output. Then test it against a question whose answer your team already knows. This gives you a reference point for accuracy before you use the template on an ambiguous problem.

    Score each version on three dimensions:

    • Accuracy: Did the answer use the supplied definitions, filters, segments, and timeframe correctly?
    • Clarity: Can a reviewer distinguish evidence, interpretation, limitations, and next steps?
    • Actionability: Does the result support the stated decision or name the next query required?

    Change one meaningful element at a time. You might compare a broad objective with a decision-specific objective, a narrative response with a fixed output contract, or an unrestricted answer with an explicit evidence boundary. Run the same test question through each variant. Otherwise, you will not know which change improved the result.

    Commit to two or three prompt iterations for one critical workflow. Review the failures, tighten the ambiguous instruction, and keep the better-performing version. Within a sprint, that process can produce a reusable template for a recurring analysis such as activation, retention, or anomaly detection.

    Store winning prompts with their required inputs and known limitations. A template without those notes becomes cargo cult: teammates copy the wording but omit the definitions that made it work. Treat the prompt, context requirements, evaluation question, and scoring criteria as one asset.

    Key takeaways

    • State the product decision before requesting analysis.
    • Define the metric, population, segments, filters, and timeframe explicitly.
    • Restrict conclusions to the analytics evidence you supplied.
    • Separate measured findings from interpretations and recommended actions.
    • Require a specific fallback when the data is insufficient.
    • Version and score prompts for accuracy, clarity, and actionability.

    Start with the recurring Amplitude question that currently creates the most debate. Write its decision, definitions, evidence boundary, and output contract. Run two or three scored iterations, then give the winning template to another product manager. If they can obtain a defensible answer without you translating the prompt, it is ready to become part of the team’s operating system.

    References

  • Beyond Accuracy: How I Evaluate AI Customer Service Agents That Delight and Scale

    When teams evaluate AI Agent options for customer service, I often see the rigor aimed at the wrong subset of criteria. After leading and observing dozens of proof of concept (POC) efforts with our customers and prospects, I understand why performance—accuracy scores, resolution rates, and benchmark tests on curated datasets—soaks up most of the attention. But those indicators alone won’t guarantee success once you leave the sandbox and face real customers.

    If your POC only proves that the AI “works,” you’re missing the bigger picture. Here’s what else I look for to make the best long-term decision.

    How does it handle your real-world setup?

    Performance is table stakes, but it has to reflect the messiness of an actual support environment. The best-performing Agents don’t just get answers right—they exhibit resilient, human-like behavior under pressure. I watch how the Agent behaves when it doesn’t know an answer: does it recover or spiral? Does it stay on track through multi-step requests, and how gracefully does it hand off to human agents? If your knowledge base depends on a retrieval-first pipeline, test cross-source retrieval and grounding—not just single-document lookups.

    When I build evaluation scenarios, I put the Agent through its paces with a broad, realistic mix:

    • Multi-turn queries that require the Agent to carry context across a conversation, not just answer isolated questions.
    • Vague or fragmented inputs, like typos, grammatical errors, and incomplete questions, because that’s how customers actually write.
    • Edge cases and sensitive scenarios, like billing disputes, frustrated customers, and questions that sit at the boundary of what the Agent is trained on.
    • Different phrasings of the same question. An Agent that handles one version well but fails on a rephrasing has a knowledge problem, not a performance problem.
    • Queries that require pulling from multiple knowledge sources. Real issues are rarely answered by a single help article, and an Agent that can only handle single-source questions will hit a ceiling fast.
    • Multilingual conversations, if your customer base requires it. Performance can vary significantly across languages and it’s better to discover that in testing than in production.

    This preparation is worth the effort. Any Agent can look impressive in a demo; what matters is how it holds up as part of your team, serving your customers in production.

    What does it feel like to interact with the Agent?

    Two AI Agents can post the same quantitative scores—resolution rates, containment rate, and more—and still deliver very different customer experiences. Resolution rate tells me whether the Agent finishes conversations; it says nothing about how customers felt during them. I deliberately assess the experience, not just the outcome, because conversation design shapes trust and brand perception.

    Here’s what I look for to ensure the AI Agent is enjoyable to interact with:

    • Is the tone natural and on-brand, or does it feel robotic and generic?
    • Does it build trust early in the conversation, or does it create friction that makes customers want to immediately request a human?
    • When it doesn’t know the answer, does it handle that gracefully?
    • When it hands off to a human, is that transition seamless, or does the customer feel abandoned?

    As George Dilthey at Clay put it when evaluating their AI setup: “Keep what’s important to your business up front and center. For us, that was transparency and control over the customer experience.”

    That framing is exactly right. The Agent represents your brand in every conversation. Customers don’t experience “accuracy,” they experience conversations. An Agent that’s technically accurate but tonally off-brand will erode customer trust over time.

    I make the experience dimension explicit in my POCs. I have people on my team—and when possible, a small cohort of real customers—interact with the Agent under realistic conditions. Then I ask how it felt, not just whether it worked.

    Can you keep improving it after launch?

    This is the dimension most teams don’t evaluate at all, and it’s possibly the most important one. Choosing an Agent that works today and ensures you can continuously improve the customer experience over time requires more than a functional demo. You’re buying a system that must get better every week, not just during the first sprint.

    The feedback loop

    Can your team easily review conversations and identify where the Agent is underperforming? Can you pinpoint specific gaps (missing knowledge, incorrect tone, poor handoff decisions) and act on them quickly? The faster the loop between “something isn’t working” and “we’ve fixed it,” the more value compounds over time. In practice, that means instrumenting conversations, leveraging Agent Analytics, tagging misroutes and tone slips, and running targeted evals on known failure modes.

    The speed of iteration

    When you identify a gap, how quickly can you address it? This is partly a question of tooling (how easy is it to update knowledge, refine guidance, adjust behavior?) and partly a question of team capability. The teams getting the most out of AI are the ones that have changed how they operate and made continuous improvement a part of their everyday work. They’ve committed to going all-in for the long term, not just the first few weeks when launching their AI Agent. We treat this as eval-driven development: automate evaluations that mirror real tickets, tighten prompt engineering and retrieval settings, and ship small fixes daily.

    The vendor partnership

    The vendor behind the Agent matters just as much as the solution itself. You’re choosing a partner for transformation that will help you evolve how your business delivers customer experience. Ask:

    • How does customer feedback influence the product roadmap, and can they show you examples?
    • If you have feedback on limitations or weaknesses, do they engage transparently or get defensive?
    • What kind of support will you get post-launch?
    • Are they shaping where AI customer experience is going, or reacting to what others are building?

    How a vendor responds to those questions tells you more about the long-term relationship than any benchmark result.

    What a good POC proves

    If your POC only proves “the AI works,” you haven’t done enough. A strong proof of concept tests performance in realistic conditions, evaluates the experience from the customer’s perspective, and validates the system that will support continuous improvement after launch. Done well, it sets you up for long-term operational success and builds organizational AI readiness—not just a flashy demo.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • Product Analytics for Retention: A Practical Operating System

    Product Analytics for Retention: A Practical Operating System

    You have a retention chart and a familiar problem: the curve is falling, the segments disagree, and every team has a different explanation. Another dashboard will not tell you what to build.

    You need a decision loop that connects retained value to observable behavior. Define the outcome, instrument the journey, locate the behavioral gap, and test the smallest change that could close it. That turns retention analytics into a product operating system rather than a monthly reporting exercise.

    Start with a retention contract, not a dashboard

    Before opening your analytics tool, finish this sentence: “For users who first do [starting action], retention means completing [valuable action] again within [return window].” If your team cannot agree on the blanks, it is not ready to interpret a retention curve.

    The starting action should identify a meaningful cohort. Account creation is often too weak because it combines curious visitors, evaluators, invited teammates, and serious users. Prefer the moment a person begins the journey you intend to improve, such as creating a project, starting an agent, or completing an initial workflow.

    The return action must represent delivered value, not convenient activity. Opening the app, viewing a page, or receiving a notification may be easy to count but weakly connected to the reason someone adopted the product. Choose an action that would make a customer notice if the product disappeared.

    Set the return window around the product’s natural use cycle. A daily workflow and an occasional administrative task should not share the same definition. Document the window, the qualifying action, the excluded users, and whether retention is measured at the user or account level. This is your retention contract.

    Next, build a driver tree connecting the retention outcome to measurable inputs. Put retained value at the top. Beneath it, map activation, repeated value-producing behavior, and the friction that can interrupt either one. This separates the lagging outcome you care about from leading signals a team can move sooner.

    For every leading signal, add a guardrail. If a change increases sessions but reduces task completion, it has created activity rather than value. If it improves first-session completion but does not affect return behavior, treat it as an onboarding improvement until the retention evidence catches up.

    Instrument the journey so the data can survive a decision

    Retention analysis breaks when event names mirror the interface instead of the customer’s progress. A click on “Continue” becomes meaningless after the button moves. An event such as workflow_started or task_completed remains interpretable across interface changes.

    For each critical event, record enough context to reconstruct what happened:

    • The user and, for collaborative products, the account.
    • The channel, surface, or entry point that started the journey.
    • The use case or object involved.
    • The event timestamp and relevant status.
    • The experiment assignment, when the experience is being tested.
    • The event version when its meaning or properties change.

    Give every retention-critical event a plain-language definition and an owner. The definition should state when the event fires, when it must not fire, which properties are required, and how duplicate or failed actions are handled. Keep cohort definitions centralized for the same reason. Product, marketing, and customer success cannot compare decisions if each team silently defines “activated” or “retained” differently.

    Validate the journey before trusting the curve. Trace real test accounts from the starting action through the value event and return action. Compare the interface state, raw events, and resulting cohort membership. Check identity transitions such as anonymous-to-signed-in usage, invitations, account switching, and merged profiles. A polished retention chart built on broken identity resolution is still broken.

    Treat the taxonomy like a product surface. Changes need review, backward compatibility, documentation, and monitoring. This work feels slower than building a dashboard, but it prevents teams from spending an entire planning cycle acting on instrumentation defects.

    Diagnose the behavioral gap before proposing a feature

    A retention curve tells you where return behavior weakens. It does not explain why. Use a fixed analysis sequence so the team does not jump from an interesting segment to a preferred solution.

    1. Inspect the curve shape. An early drop points you toward expectation-setting, onboarding, or initial value. A later decline points you toward repeat value, changing needs, or workflow friction.
    2. Segment with a hypothesis. Compare acquisition source, device, channel, use case, or customer type only when you can explain why that dimension might change the experience.
    3. Compare retained and non-retained cohorts. Look for behaviors that differ in sequence, completion, or repetition, not merely events with high volume.
    4. Build a funnel around the strongest candidate behavior. Find the step where the cohorts separate and inspect how users arrive there.
    5. Review session replay, conversation transcripts, or journey detail at that step. Look for hesitation, repeated attempts, unclear choices, missing context, and premature exits.

    This sequence moves you from outcome to segment, behavior, moment, and observable friction. Stop if the evidence cannot support that chain. A behavior that correlates with retention is a place to investigate, not proof that forcing the behavior will retain users.

    AI products make this distinction especially important. A generic greeting may produce a response without moving the user toward a task. If people hesitate, test a concise follow-up that clarifies the agent’s scope, offers two or three concrete choices, and still accepts free-form input. Measure the chain from continuation to task start, task completion, and return across the first three to five sessions. Do not optimize for extra conversation turns if users remain stuck.

    Pair behavioral evidence with continuous discovery. Analytics identifies the moment worth investigating; interviews and direct observation help explain the need, expectation, or constraint behind it. That combination produces a testable problem statement instead of a feature request decorated with data.

    Turn retention signals into controlled product bets

    Write the opportunity before discussing solutions: “When [cohort] reaches [moment], [observable friction] prevents [valuable behavior], which is associated with lower [retention outcome].” The wording forces you to name the user, the moment, the evidence, and the outcome without pretending you have already established causality.

    Then create an experiment card with:

    • A hypothesis linking the proposed change to a specific behavior.
    • The eligible cohort and trigger moment.
    • One primary retention outcome.
    • A leading indicator that can move earlier.
    • Guardrails for completion quality, errors, or unintended friction.
    • The minimum detectable effect and planned evaluation window.
    • A decision rule for stopping, iterating, rolling out, or reversing the change.

    Choose a change small enough to isolate the mechanism. If the suspected problem is uncertainty at the start of an AI interaction, test the opening sequence rather than redesigning the agent, onboarding flow, and navigation together. A smaller bet makes the result easier to interpret and cheaper to reverse.

    Review experiments on a regular product cadence. Begin with data quality, then evaluate the leading indicator, guardrails, and retention outcome in that order. Inspect the segments named in the original hypothesis rather than searching every possible cut for a favorable result. Record what the team decided, why it decided it, and what evidence would change the decision.

    Your roadmap should name the retention outcome and the behavioral driver, not promise a feature prematurely. “Increase repeat task completion for newly activated accounts” leaves room to test messaging, workflow design, defaults, or assistance. “Build a new onboarding wizard” locks the team into an answer before it has earned confidence in the problem.

    Key takeaways

    • Define retention as a cohort, a value-producing action, and a return window before interpreting any chart.
    • Use a driver tree to connect the lagging retention outcome to behaviors a product team can influence.
    • Standardize event and cohort definitions, then validate identity and journey data with real test accounts.
    • Move from curve to segment, behavior, moment, and friction before proposing a solution.
    • Use controlled, reversible experiments to distinguish a useful behavioral signal from a causal retention lever.

    Start with one journey that matters this week. Write its retention contract, trace the events, and identify the first point where retained and non-retained users behave differently. That single decision-ready path is more valuable than a broad analytics program nobody trusts.

    References

  • Old-School Selling Beats PLG in the AI Era: My GTM Playbook for 8‑Day Enterprise Deals

    Old-school, in-person selling is having a renaissance in the AI era, and I’ve seen why up close. From leading product and go-to-market teams through hypergrowth, I keep returning to one lesson: enterprise buyers still reward the teams who show up, orchestrate change management, and own outcomes end-to-end. The tech has changed; the human dynamics haven’t.

    Has the sales playbook changed in the AI era? The tools are faster and the surface area is bigger, but the core motion remains the same: “showing up” beats letting the marketplace decide. That’s why in-person enterprise rollouts still beat product-led motions, especially when the stakes include security, governance, and cross-functional adoption. You win by reducing organizational risk, not by assuming free trials will do the heavy lifting.

    Great enterprise sellers collapse silos. They sell to engineers and executives in one motion, pairing deeply technical validation with crisp business narratives. In my org, that means every high-velocity pilot has a dual thread: hands-on, eval-driven proof for the builders and a value architecture for the budget owners. When those motions run in parallel, time-to-value plummets and procurement friction fades.

    Selling to AI-native buyers who grew up on ChatGPT changes tempo, not fundamentals. The same seller, different tempo: 8 weeks vs. 8 business days. These buyers evaluate fast, expect clear ROI, and push for automation-first workflows. How AI-native buyers handle build vs. buy decisions comes down to build for differentiation and buy for acceleration. If you make procurement feel like product—frictionless, instrumented, and transparent—you’ll meet their bar.

    Process matters, but humanity wins. Building a robust sales process that still leaves room for unscripted moments is where trust is formed. I’ll never forget the story of the rep who taught a champion’s son guitar over Zoom—an unscripted moment that cemented a partnership. The lesson: raise the floor without capping the ceiling. Equip every rep with repeatable plays, then celebrate the creative instincts that make champions out of customers.

    In early GTM, why the three highest-leverage early sales hires aren’t sellers at all resonates with my experience. I prioritize a solutions engineer who can de-risk integration, a forward-deployed operator who can run the first rollout like a product manager, and a customer success lead who designs adoption paths from day zero. Together, they compress the value journey from proof to production.

    Compensation design shapes your talent market. The case for outsized commission accelerators for star sellers — and the kind of person they attract is real: magnets for competitors who close complex, multi-threaded deals and thrive with ownership. But beware: why too much process narrows the kind of seller you attract. Over-script it and you filter out the very people who can navigate ambiguity with customers.

    Under the hood, instrumenting the funnel from stage zero to close keeps the system honest. I track intent signals before pipeline, conversion by persona and use case, proof milestones, and time-to-value in production. The three pillars of GTM excellence for me are repeatable discovery, referenceable outcomes, and relentless enablement. And inside the leadership team, building peers who are 80% aligned, not 100% preserves healthy tension while keeping execution fast.

    AI is expanding the definition of enablement—whether AI is changing what good enablement looks like isn’t a theoretical question anymore. I see world-class teams arming reps with retrieval-first knowledge bases, sandbox environments, and objection libraries that evolve weekly. Meanwhile, selling against direct and implied competitors at once is the norm: your battlecard must cover “do nothing,” internal tools, adjacent categories, and new AI entrants—while you still remember why in-person enterprise rollouts still beat product-led motions for durable adoption.

    Planning horizons tighten in AI markets. How far out should a GTM leader be planning? I work a dual cadence: a rolling 6-week operating plan that’s ruthlessly tactical and a 2–3 quarter roadmap for coverage, enablement, and category storytelling. What a normal week looks like in hypergrowth blends customer time, pipeline triage, onboarding and enablement, deal engineering, and process tuning—always with one or two high-conviction bets that could bend the curve.

    References: Ahead: https://www.ahead.com; Amazon: https://www.amazon.com; Anthropic: https://www.anthropic.com; Attio: https://www.attio.com; Augment Code: https://www.augmentcode.com/; Cognition: https://cognition.ai; Cursor: https://cursor.com; Dani McCabe: https://www.linkedin.com/in/danielle-mccabe/; Datadog: https://www.datadoghq.com; GitHub Copilot: https://github.com/features/copilot; HubSpot: https://www.hubspot.com; Jeremy Powers: https://www.linkedin.com/in/jeremypowers/; JPMorgan: https://www.jpmorgan.com; Matt McClernan: https://www.linkedin.com/in/mattmcclernan/; MongoDB: https://www.mongodb.com; Nicole Rettinger: https://www.linkedin.com/in/nicole-rettinger-23b20465/; Notion: https://www.notion.com; OpenAI: https://openai.com; Parag Agrawal: https://www.linkedin.com/in/paragagr/; Parallel: https://parallel.ai; Snowflake: https://www.snowflake.com; University of Chicago: https://www.uchicago.edu; Windsurf: https://windsurf.com

    If you’re scaling an AI product today, pair a disciplined sales-led growth engine with the best of product-led growth: fast paths to proof, hands-on validation for builders, executive-level value mapping, and human moments that turn customers into advocates. That’s how you compress an eight-week cycle into five business days—and keep the expansion flywheel spinning.

    Book a consult png image
  • Supercharge Core Web Vitals with Amplitude’s Global Agent: Faster Rankings, Happier Users

    Supercharge Core Web Vitals with Amplitude’s Global Agent: Faster Rankings, Happier Users

    I measure product health by a simple equation: speed plus clarity equals trust. That’s why I prioritize Core Web Vitals and search performance together—because the fastest path to better UX and higher rankings is a closed loop between measurement, diagnosis, and action. Standardizing on Amplitude’s Global Agent with Amplitude AI Agents let my teams compress that loop from weeks to hours, and in many cases, to minutes.

    Learn how to track your web vitals and page rankings faster with Amplitude AI Agents and improve your site’s user experience and SEO rankings. That goal sounds ambitious, but with the right instrumentation and analytics workflow, it becomes a repeatable operating rhythm rather than a one-off project.

    Here’s what changed for us with Amplitude’s Global Agent: a single, consistent way to capture performance signals across pages and journeys, unified context for every session, and a lightweight footprint that doesn’t get in the way of speed. By centralizing measurement, we eliminated blind spots and gave product, growth, and engineering one shared truth for Core Web Vitals and behavioral analytics.

    My practical playbook is straightforward: 1) Establish a performance baseline for Core Web Vitals on key templates and critical user paths. 2) Segment results by device, location, acquisition channel, and content type to surface where users actually feel the friction. 3) Connect those vitals to downstream behaviors—scroll depth, engagement, and conversion—so we prioritize fixes that move business outcomes, not just lab scores. 4) Use feature flags and A/B testing to ship improvements safely and quantify uplift. 5) Close the loop with Agent Analytics to keep learnings visible and actionable.

    Operationally, we rely on anomaly detection to flag regressions early, CI/CD guardrails to prevent performance slips at deploy time, and observability plus session replay to accelerate root-cause analysis. This combination reduces mean time to resolution, protects page experience during fast iteration cycles, and helps us avoid trading UX for speed—or vice versa.

    The strategic benefit is compounding: better Core Web Vitals improve user perception and increase engagement, which strengthens SEO signals and, ultimately, page rankings. With a unified analytics platform in place, we can spotlight the few improvements that create outsized gains, then scale those patterns across the site with confidence.

    If your roadmap includes faster pages, stronger rankings, and happier users, align your teams around this simple loop: measure precisely, diagnose quickly, experiment safely, and learn continuously. Amplitude’s Global Agent and Amplitude AI Agents give you the instrumentation and insight to make that loop your competitive advantage.

    Inspired by this post on Amplitude – Best Practices.

    Book a consult png image
  • How to Operate Always-On AI Agents Without Losing Control

    How to Operate Always-On AI Agents Without Losing Control

    You want an AI agent to keep work moving after you close your laptop. The difficult part is not getting one successful overnight run. It is making the hundredth run predictable enough that you do not wake up to an embarrassing email, a corrupted task queue, or an unexplained usage bill.

    The right operating model looks less like a clever prompt and more like a small, well-managed operations team. Give each agent a narrow job, an inspectable queue, limited tools, a clear definition of done, and an explicit place to stop. That is how you gain useful autonomy without surrendering control.

    Start with a delegation contract, not a general-purpose assistant

    An always-on agent should not begin with a broad instruction such as “manage my sales work.” That leaves the model to decide what managing means, which systems it may change, and when it has enough evidence to act. The ambiguity is tolerable during an interactive session because you can correct it. It becomes operational risk when the agent runs unattended.

    Start by defining a job that produces a recognizable artifact. A sales-admin agent can prepare a briefing before a scheduled call and create proposed follow-up tasks afterward. A podcast-manager agent can assemble interview context, prepare a transcript-review document, and queue a reminder to share it. A coding-manager agent can review prior sessions and identify recurring mistakes. These are bounded responsibilities with visible outputs, not vague mandates to “help.” Three specialized agents handling podcast, sales, and coding workflows demonstrate how cleanly this pattern can separate unrelated work.

    Write the delegation contract in an identity file that the agent reads at the beginning of every run. It should answer seven questions:

    1. Who are you? Name the role, not the underlying model: sales admin, podcast manager, coding manager, or another function a person would recognize.
    2. What outcome do you own? Describe the recurring deliverable and the event that makes it useful.
    3. Where may you work? Name the exact task, output, and script folders the agent can use.
    4. What inputs may you trust? Identify the calendar, task file, transcript, session log, or other allowed input for the job.
    5. What may you change? Separate reading, drafting, creating internal files, updating tasks, and acting in external systems.
    6. What counts as complete? Specify the artifact, required fields, location, and status update expected at the end.
    7. When must you stop? Define what the agent should do when information conflicts, a tool fails, permission is missing, or the next step would affect another person.

    The last question matters most. A useful agent does not need permission to improvise its way through every obstacle. It needs a reliable way to say, “I could not complete this safely; here is the missing decision.” Treat a well-documented block as a successful operational outcome, not as agent failure.

    Keep consequential decisions outside the unattended role. The agent can prepare a customer email without sending it. It can propose changes to a deal record without changing the commercial commitment. It can summarize a coding pattern without modifying a production system. Moving from preparation to execution should be a deliberate permission decision, not an accidental side effect of adding another tool.

    Build an inspectable operating loop around four components

    The prompt is only one part of the system. Reliable agent operations need four components with distinct responsibilities: identity, scheduling, tasks, and scripts. Keeping them separate makes failures easier to locate and changes easier to review.

    Identity defines responsibility

    The identity file is the stable operating policy. It tells the agent what role it is playing, where its work lives, what it may do, and what completion looks like. Do not overload it with the details of one assignment. If the identity changes every time a task arrives, you no longer have a stable agent; you have an unreviewed prompt generator.

    The scheduler supplies a heartbeat

    The scheduler should wake the agent, point it to the correct identity and queue, and capture the result. It should not contain the business logic for podcast preparation or sales follow-up. That logic belongs in inspectable task instructions and small scripts.

    A Mac that remains online can use macOS LaunchAgents as this heartbeat. LaunchAgents run with the user’s permissions, which is operationally convenient but also defines the risk boundary: the agent may be able to reach anything the scheduled process and its tools can reach. Running scheduled agents on an always-on Mac Mini therefore makes permission design part of the architecture, not a setting to revisit later.

    Make the schedule explicit and easy to disable. Each job should have a known trigger, whether that is a recurring interval, a calendar-related event, or a periodic review. If you cannot quickly answer why an agent ran at a particular time, the scheduler is already too opaque.

    Tasks hold durable state

    Use a dedicated task folder for each agent. A Markdown file with frontmatter is enough to represent a work item while remaining readable by both a person and a tool. The frontmatter can hold machine-readable state; the body can hold the request, context, acceptance criteria, and eventual run notes.

    Choose a small lifecycle and apply it consistently. For example: queued, in progress, blocked, completed, and failed. The exact labels matter less than the transition rules:

    • A queued task is eligible to be claimed.
    • An in-progress task records which run claimed it, preventing another run from silently doing the same work.
    • A blocked task names the missing input or decision and preserves all useful partial work.
    • A completed task links to its output and records what changed.
    • A failed task records the failed operation and whether retrying it is safe.

    Give each recurring event a stable identifier. Before creating a meeting brief, transcript-review document, or follow-up task, the agent should check whether that event has already been processed. This idempotency check prevents a retry or overlapping schedule from creating duplicates.

    Do not treat chat history as the task database. Conversations are useful working context, but durable state belongs in a file or system you can inspect independently. Saving identities, task files, and scripts in a shared knowledge workspace such as Obsidian also makes the operating model portable across devices and coding assistants. Changing the model runner should not require rebuilding the job.

    Scripts expose narrow capabilities

    Scripts should perform small, deterministic operations: fetch an allowed input, create a document in a known location, normalize a transcript, or update a task field. Keep the judgement in the agent and the mechanics in scripts with explicit inputs and outputs.

    A small script is easier to inspect than a broad instruction to use the terminal however the model sees fit. It also gives you one place to add validation, duplicate checks, and error handling. When an agent repeatedly constructs the same command or edits the same file shape, promote that operation into a reviewed script rather than relying on the model to reproduce it perfectly on every run.

    Design the overnight failure path before the happy path

    Unattended automation changes the cost of a mistake. During an interactive session, a confusing output costs a correction. Overnight, the same confusion can trigger repeated work, alter several systems, or contact someone before you see it. Your design should limit the consequence of a wrong interpretation, not merely improve the probability of a correct one.

    Use a permission ladder

    Classify capabilities by consequence and grant them one level at a time:

    1. Read: inspect approved calendars, task files, transcripts, logs, or documents.
    2. Prepare: create drafts, summaries, reports, and proposed tasks inside a bounded workspace.
    3. Update: change internal records whose history can be inspected and reversed.
    4. Act externally: send messages, share files, update customer-facing systems, or invoke paid services.
    5. Perform destructive or privileged work: delete data, change access, alter infrastructure, or execute an irreversible operation.

    Most new agents should prove themselves at the read and prepare levels. Promotion should be capability-specific. An agent that reliably prepares a sales brief has not thereby earned permission to send customer communication. Reliability does not transfer automatically from one action class to another.

    For external actions, use a pending-approval state that contains the exact proposed action. You should be able to review the recipient, content, destination, and relevant context without reopening the entire run. Destructive or privileged actions should remain outside unattended execution unless you have an explicit recovery path and have deliberately accepted the consequence of failure.

    Treat external text as data, not authority

    Calendar descriptions, transcripts, web pages, emails, and documents may contain instructions that conflict with the agent’s job. The identity and task contract must outrank text found inside those inputs. An interview guest’s biography can inform a briefing; it cannot expand the podcast agent’s permissions. A meeting note can identify a follow-up; it cannot authorize the agent to send one.

    Keep credentials out of identity and task files. Give scripts access only to the credentials required for their operation, and avoid handing an agent a general browser, terminal, file system, and credential store merely because each tool is useful in isolation. The dangerous capability is often the combination.

    Make retries selective

    A retry is appropriate when the failure is plausibly temporary and repeating the operation is safe. A network timeout during a read may qualify. Ambiguous recipient identity, conflicting meeting details, missing share settings, or an unclear customer commitment do not. Retrying an ambiguity only asks the model to make the same unsupported decision again.

    Before enabling automatic retries, require the operation to pass three tests: it can detect whether it already succeeded, a duplicate would not create harm, and the number of attempts is capped. Otherwise, mark the task blocked and surface it for review.

    Put hard boundaries around usage

    Always-on does not mean continuously reasoning. It means the system is available to process eligible work on a known schedule. A run should inspect the queue, process a bounded amount of work, record its result, and exit.

    Set limits at several layers: eligible task types, work accepted per run, retries per task, tools available to the role, and provider-side spending or usage controls where available. Record usage beside the task outcome so you can distinguish an expensive valuable job from an agent that consumes resources while circling an ambiguity. Surprise charges are not only a pricing problem; they usually indicate that the operating loop lacks a stopping rule.

    Finally, maintain a kill switch you can use without asking the agent to cooperate. Disabling the schedule or revoking the narrow credential should stop future work. If stopping the system requires the same model and scripts that may be malfunctioning, it is not an independent control.

    Measure whether the agent is reducing work or relocating it

    A completed status is not proof of value. An agent can close every task while leaving you to verify facts, repair formatting, remove duplicates, and reconstruct why it made a decision. That is work relocation, not delegation.

    Evaluate the operation with measures tied to the job:

    • Usable completion rate: the share of eligible tasks that produce an output meeting the acceptance criteria without substantive rework.
    • Correction rate: how often you must change facts, recipients, permissions, status, or next steps before using the output.
    • Duplicate or false-action rate: how often the agent repeats a job or creates an action that the triggering event did not require.
    • Blocked rate by cause: which missing inputs, permissions, or unclear rules repeatedly prevent completion.
    • Time to review: the human attention required to approve, repair, or understand the result.
    • Usage per usable outcome: the model or service consumption attached to work you actually keep.

    These measures tell you what to change. A high blocked rate caused by missing context points to an input problem. Frequent factual corrections point to retrieval or acceptance-criteria problems. Duplicate work points to task identity and idempotency. High review time with otherwise correct output often means the evidence and change log are poorly presented.

    Require every run to leave a compact receipt: the task it claimed, inputs it used, scripts it invoked, files or records it changed, output location, completion status, and reason for any block. You should not need to replay hidden reasoning. You need enough evidence to verify the operation and diagnose the next failure.

    Review early runs closely and review again after changing an identity, script, tool, model, or input source. A stable task can become unstable when any one of those dependencies changes. Plain-text identities, tasks, and scripts make that change surface inspectable and versionable.

    Your agents can also improve the operating system itself. A periodic coding-manager workflow, for example, can review prior coding sessions, identify recurring dead ends, and propose changes in how future sessions are run. The important separation is that the agent proposes an improvement with evidence; the operating policy changes only after review. Self-observation is useful. Unreviewed self-modification is a different risk class.

    Expand only when the current job has earned more autonomy

    Adding agents is easy once the scheduler and folder structure exist. That convenience can tempt you to automate work whose boundaries are not ready. Scale based on operational evidence, not on the number of possible use cases you can imagine.

    A job is a strong candidate for always-on operation when it has a recurring trigger, stable inputs, an observable deliverable, clear acceptance criteria, bounded permissions, and enough repetition to justify maintaining the workflow. Preparation, follow-up capture, document setup, and periodic retrospectives fit because a person can inspect their artifacts and correct them before higher-consequence decisions are made.

    Keep work interactive when the task depends on novel judgement, unresolved organizational context, sensitive negotiation, or irreversible action. An agent may still prepare evidence and options, but the decision should remain with the person who owns the consequence.

    Before expanding an existing agent’s permissions or creating another role, check five gates:

    1. The current output is regularly usable without substantial reconstruction.
    2. Common failure modes are visible and end in safe states.
    3. Duplicate prevention and retry behavior have been exercised.
    4. Usage is attributable to tasks and bounded by stopping rules.
    5. The next capability has its own acceptance criteria and consequence review.

    Do not create one agent per application. Create one per coherent responsibility. A podcast manager may use a calendar, a document system, and a task list while retaining one outcome. Conversely, sales administration and coding retrospectives should not share an identity merely because they use the same model. Role boundaries should follow accountability, not tooling.

    Key takeaways

    • Begin with one recurring job that produces an inspectable artifact, not a general instruction to manage a function.
    • Give the agent a durable identity, a dedicated task queue, an explicit schedule, and small reviewed scripts.
    • Use task states, stable event identifiers, and completion receipts so retries and overlapping runs do not create invisible duplication.
    • Keep new agents at read-and-prepare permissions until their outputs and failure modes are consistently understandable.
    • Route ambiguity and consequential external actions to approval instead of asking the model to guess.
    • Cap eligible work, retries, tools, and usage; always-on availability should still produce finite runs.
    • Measure usable outcomes, corrections, blocks, duplicates, review effort, and usage before granting more autonomy.

    Pick one task you already repeat and write its delegation contract before choosing more tools. If you cannot define the input, output, permission boundary, completion test, and safe stopping condition on one page, the job is not ready to run while you are offline. Tighten the job first. The agent can earn broader responsibility after the operating evidence is there.

    References