Category: Product Management Leadership

  • Unlocking Impact: What Amplitude’s MCP server and experimentation platform teach product leaders

    Unlocking Impact: What Amplitude’s MCP server and experimentation platform teach product leaders

    In my role leading product management at HighLevel, I study the architectures and operating models behind high-velocity learning. I often reference "Amplitude's MCP server and its experimentation platform" as a benchmark for how to operationalize scale, reliability, and speed of insight across complex product ecosystems. That lens informs how I design processes, data flows, and decision loops that turn ambiguity into measurable outcomes.

    Experimentation is the heartbeat of eval-driven development. In practice, that means running disciplined A/B testing, deploying targeted feature flags to de-risk rollouts, and sizing experiments with a clear minimum detectable effect (MDE) so we avoid vanity wins. When teams internalize these habits, we shift from opinion-led debates to evidence-led decisions—and that’s where product-led growth compounds.

    I'm an AI enthusiast, so I think a lot about how experimentation accelerates AI roadmaps. The same rigor that validates UI changes should govern prompts, retrieval strategies, and policy settings for LLM-backed features. By treating AI behaviors as first-class experiment surfaces—and tying them to user activation, retention analysis, and value proposition metrics—we move faster without compromising safety, privacy-by-design, or customer trust.

    Making this work in production demands clean instrumentation and a unified analytics platform. I look for stacks that combine Amplitude analytics with robust observability and CI/CD to ensure we can ship, measure, and iterate continuously. When platform scalability and data governance are baked in from the start, product trios can focus on product discovery rather than firefighting pipelines or reconciling metrics.

    My playbook is straightforward: define decision-worthy questions, map them to crisp success metrics, run right-sized experiments with feature flags, and use consistent analytics to close the loop. Do this well, and you create a durable advantage—faster learning cycles, sharper product positioning, and a culture that lives by outcomes over output. That’s the real lesson I take from platforms that execute experimentation at scale: process and technology are table stakes; what wins is the discipline to learn relentlessly.

    Inspired by this post on Amplitude – Perspectives.

    Book a consult png image
  • Apex Arrives: Vertical AI That Beats GPT-5.4 on Customer Service Speed, Accuracy, and Cost

    Apex Arrives: Vertical AI That Beats GPT-5.4 on Customer Service Speed, Accuracy, and Cost

    I just watched one of the most significant leaps in customer service AI in years. Last week, a quiet but seismic release landed in CX: Fin introduced Apex, a vertical model purpose-built for support that raises the bar on speed, accuracy, and cost. As a product leader, this is exactly the kind of breakthrough that changes roadmaps, vendor strategies, and what customers can expect from modern service operations.

    It’s a brand new model for Fin called Apex, and it’s objectively the highest performing, fastest, and cheapest model for customer service. It beats the very best models in the industry including GPT-5.4 and Opus 4.5.

    In this analysis, I’ll unpack why the launch matters for the customer service agent category, what it signals for frontier labs and open‑weight ecosystems, and how leaders should rethink their AI Strategy, build vs buy decisions, and eval-driven development roadmaps.

    Fin was already the highest performing and most sophisticated agent in the customer service space, consistently beating impressive competitors like Decagon and Sierra at an average win rate in the 70s. It operates at tremendous scale, now resolving almost 2M customer issues per week, a number that’s growing at an exponential clip. In its short life it’s grown to nearly $100M in recurring revenue.

    As of last week, ~100% of all (English language, chat and email) customer conversations are now running on Apex. Since day 1, the Fin engine has comprised a system of models, and last year the team began replacing off‑the‑shelf models with custom ones trained on proprietary data. The core answering model had been a frontier labs offering—initially versions of GPT and more recently Sonnet 4.0. Now, that core answering model is Apex 1.0.

    This model resolves customer issues at a materially higher rate than any other model available. One of their largest customers in the gaming space saw the resolution rate improve overnight from 68% to 75% (i.e. a reduction in unresolved conversations of 22%). The team notes they had never seen a jump this large from a single improvement since they started Fin.

    Just as important, it’s dramatically faster, has fewer hallucinations, and is far cheaper than other available models—exactly the attributes operations leaders weigh most when deploying agents at scale. In practice, these are the levers that unlock higher CSAT, tighter SLAs, and better unit economics.

    Achieving all three simultaneously is extraordinarily hard. Credit goes to foundational research from a 60‑person AI group run by Fergal Reid, and, crucially, to domain‑specific proprietary evals drawn from billions of human and agent interactions produced by the Fin resolution engine—already hand‑tuned to be the most effective in the category. That creates a flywheel: an eval‑driven development loop that trains models to keep improving at the edge of the system’s abilities. In other words, Apex 1.0 looks like the tip of the iceberg.

    Zooming out, service is one of the few categories where generative AI has already delivered commercial impact at scale (alongside coding, and arguably the legal industry). With TAMs measured in the hundreds of billions, competition is intense and well capitalized. The pattern I’ve seen repeatedly is clear: winners in these spaces must become full‑stack AI companies. As features become ~free to build, durable competitive differentiation shifts under the hood—to proprietary data, post‑training, inference efficiency, and the quality of the eval loop.

    Dual bar charts showcasing Fin Apex 1.0 with -65% hallucination reduction and a 3.7s time to first token, benchmarked against Sonnet 4.6, Opus 4.5, and GPT-5.4 on a clean, light background.
    Fin Apex raises the bar for finance-ready AI, highlighting a -65% cut in hallucinations and a quicker first token at 3.7s (0.6s faster), compared with Sonnet 4.6, Opus 4.5, and GPT-5.4 in side-by-side charts.

    That’s why competitors will need to release their own models. Many appear to be just starting to hire the talent to do so, which likely gives Fin at least a year of head start. For product leaders, this is a strong signal to revisit build vs buy assumptions, and to quantify when owning your post‑training pipeline and evals becomes the rational move.

    Honestly, 2–3 years ago I expected AI application differentiation to live mostly in what we built around third‑party models. The AI game humbles all of us; today it’s obvious that vertical models paired with proprietary evals create compounding moats.

    In a podcast interview last week, Andrej Karpathy said:

    "I do think we should expect more speciation in the intelligences. The animal kingdom is extremely [diverse] in the brains that exist. And there’s lots of different niches of nature… And I think we should be able to see more speciation. And you don’t need this oracle that knows everything. You kind of speciate it. And then you put it on a specific task. And we should be seeing some of that because you should be able to have much smaller models that still have the cognitive core."

    The frontier labs still have the very best models, but open‑weight models aren’t far behind—making pre‑training look increasingly like a commodity. The frontier is moving to post‑training, which is precisely what we see with Apex (and Cursor’s Composer 2), and what we should expect to dominate going forward.

    Labs now face a dual reality. On one hand, horizontal general‑purpose models can over‑serve specific verticals (e.g., customer service doesn’t need an oracle that knows everything). On the other, open‑weight models are good enough that high‑quality, domain‑specific post‑training can produce superior models for special‑purpose jobs—and in the ways that matter for those jobs. In service, soft factors like judgement, pleasantness, and attentiveness matter alongside hard factors like resolution effectiveness, speed, and cost.

    I’m still bullish on the labs. Many organizations remain heavy customers of Anthropic—whether as part of multi‑model systems or through deep usage of Claude Code in engineering teams (see this example of Claude Code adoption). Yet classic disruption (à la the late, great Clay Christensen) is now at their door. The way out is to disrupt themselves by building cheaper specialized models too, which likely requires acquiring the evals—or the companies with the evals—needed for each task. Expect creative data partnerships, M&A consolidation, and a wave of hyper‑specific model providers that compete head‑to‑head with the labs.

    In the meantime, Fin appears to be the only vendor in its space with a custom model that’s also objectively superior to everything else out there. I’m excited to see it deployed broadly for end customers, and I’m watching closely for the next announcement that will accelerate that rollout. For product leaders, the message is clear: the age of vertical models and agentic AI is here—bring your evals, or bring your checkbook.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • From Engineer to CEO: Hard-Won Lessons on GTM, Cloud-First Bets, and Must-Do Focus

    From Engineer to CEO: Hard-Won Lessons on GTM, Cloud-First Bets, and Must-Do Focus

    Making the leap from engineer to CEO demands an almost entirely new skillset. I’ve felt that jolt firsthand: the tools that serve you as an IC or even a product leader—system design, crisp PRDs, elegant roadmaps—only get you about 20% of the way. The rest is learning to orchestrate go-to-market strategy, finance, hiring, culture, and product positioning with just enough depth to make sound, fast decisions while empowering true experts to execute.

    My operating heuristic is the 80% rule. As CEO or GM, I don’t need to be the best marketer, seller, or finance leader; I need to understand 80% of each function well enough to set a compelling product strategy, ask the right questions, and catch the second-order effects. That breadth unlocks speed, quality of judgment, and the conviction to say no when the organization is tempted by what it can do rather than what it must do.

    The clearest illustration comes from the journey that turned Apache Kafka—originally built at LinkedIn—into Confluent, a publicly traded enterprise software company. The technical insight was powerful, but the real lift came from translating that insight into a repeatable go-to-market engine. That required building new muscles: founder-led GTM, enterprise sales orchestration, and open source monetization without alienating the community that fueled adoption.

    Early on, the product was “embarrassing” by enterprise standards—thin features, sharp edges, and a long tail of operational gaps. Shipping anyway was the point. A thin vertical slice into the market created learning loops with real customers, not hypotheticals. That uncomfortable speed became a superpower, especially when the company decided to push toward a cloud-first business in the face of widespread opposition.

    The messaging challenge was just as hard as the technical one. Most marketing fails because it starts with what we built, not what customers must achieve. A simple product marketing pyramid—vision at the top, category framing and points of parity in the middle, crisp value props and proof at the base—helped explain Kafka to the world in customer language. When the narrative snaps into place, adoption accelerates. In Kafka’s case, one well-timed blog post clarified the “why now” and unlocked a step-change in community and enterprise pull.

    There’s a pivotal distinction leaders underestimate: the gap between what a company can do and what it must do. I use a must-do filter before every planning cycle: What moves are non-discretionary for durable product-market fit? For Kafka and Confluent, that meant ruthless prioritization on managed cloud services, reliability, and platform scalability—even when it jeopardized short-term revenue or required retooling how engineering, sales, and support worked.

    Fundraising strategy mirrored this clarity. Planning to raise before building the full product wasn’t about hype; it was about matching capital to the physics of the problem. If your category requires enterprise credibility, global infrastructure, and 24/7 SRE, you finance those table stakes early. That’s first principles decision making: instrument the constraints, then design the sequence that gets you to scale with the fewest irreversible mistakes.

    In the early years, every product decision felt like a trade between polish and learning. The team essentially bludgeoned its way into a cloud-first posture—less because the initial product was ready, and more because the market’s must-do was obvious. That’s the essence of founder-led GTM: get into the field, close lighthouse customers, and use their arcs to shape the roadmap. It’s also where open source monetization matures from downloads into durable, enterprise value.

    As the organization scales, excellence often erodes—the Chipotle problem. Process hardens; quality blurs; the magic decays. The antidotes are simple but hard: a few non-negotiable product quality bars, a short set of product-market fit metrics that everyone can recite, and empowered product teams who own outcomes over output. This is where organizational development matters as much as code: design clear interfaces between product, sales, and success, and you’ll keep velocity without losing standards.

    Contrary to popular lore, founder optimism is overrated. Constructive realism wins. I try to model “probabilistic optimism”: assume we will win, but instrument the journey like an SRE runs an incident. Set leading indicators, rehearse failure modes, and make pre-commitments to the must-do path so you’re not swayed by the latest anecdote. It keeps the team out of a failure mindset while making room for rigorous course correction.

    Giving up the right things at the right time is a CEO superpower. As complexity grows, I hand off decisions that benefit from specialization and keep only those tied to company narrative, must-do prioritization, and talent bar. CEO time management becomes a portfolio problem: ensure each week contains deep product time, frontline customer exposure, and one compounding systems fix (hiring loop, pricing rubric, or GTM enablement) that pays back for quarters.

    If you’re moving from IC or PM into a GM/CEO role, here’s a practical playbook: build your product marketing pyramid; write the one-page must-do memo for the next six quarters; ship a narrow, managed cloud slice early; pick three product-market fit metrics (usage, time-to-value, retention) and publish them company-wide; and architect an enablement engine that turns field learnings into roadmap changes within one quarter. That’s how you transform technical advantage into a category-defining business.

    The Kafka-to-Confluent arc reminds me that technology can open a door—but clarity of narrative, sequencing, and must-do focus determines whether you walk through it. When in doubt, bias toward shipping, talking to customers, and tightening the loop between what you learn and what you build. That’s the work of product management leadership at scale.

    Book a consult png image
  • Inside My Product Marketing Playbook: Amplitude Analytics Tactics That Drive PLG Wins

    Inside My Product Marketing Playbook: Amplitude Analytics Tactics That Drive PLG Wins

    I’ve curated a focused set of product marketing insights that zero in on what actually moves the needle—turning data into decisions. You’ll find a special emphasis on Amplitude Analytics, because its behavioral analytics foundation makes it easier to translate product usage into clear messaging, sharper positioning, and measurable growth.

    In my day-to-day as a product leader, I’m constantly bridging the gap between product discovery and go-to-market strategy. The best outcomes come when we connect quantitative signals to narrative: using behavioral analytics to inform the value proposition, refining product positioning with cohort trends, and driving product-led growth with activation and retention insights.

    Here’s how I put this into practice. I start with user activation and retention analysis to identify the few behaviors that predict long-term value. Then I run tightly scoped A/B testing to validate messaging and in-product prompts that nudge those behaviors. When the numbers move, I translate wins into a consistent story—one that sales, success, and marketing can all rally around.

    One pattern keeps repeating: clarity beats complexity. Instead of piling on more features, I focus on the minimum, verifiable set of behaviors that correlate with outcomes. That discipline makes it easier to craft a crisp value proposition, streamline go-to-market strategy, and accelerate feedback loops between product, design, and marketing.

    As you explore this collection, expect practical playbooks over platitudes. You’ll see how to apply Amplitude Analytics to uncover hidden friction, validate hypotheses faster, and operationalize product-led growth motions that compound over time. My goal is to help you move from interesting dashboards to decisive actions that strengthen your roadmap and your revenue.

    If you care about building empowered product teams that learn continuously, you’ll feel at home here. Dive in, borrow what works, and adapt the rest to your context—then measure it, iterate, and share the wins with your team.

    Inspired by this post on Amplitude – Best Practices.

    Book a consult png image
  • How to Scale AI Customer Experience Without Losing Quality

    How to Scale AI Customer Experience Without Losing Quality

    Your AI agent can resolve more conversations while customer experience quietly becomes harder to trust. A ticket marked resolved can still contain an inaccurate answer, a skipped process step, a repetitive loop, or an escalation that arrived too late.

    As the product leader, you don’t have to choose between automation and judgment. You need an operating system that identifies which conversations matter, defines what good looks like, routes exceptions to the right owner, verifies fixes, and connects support quality to customer behavior.

    Measure outcomes, execution quality, and coverage separately

    Resolution rate is a throughput metric. It tells you how often the operation reached a terminal state, but not whether the answer was correct or the customer was treated appropriately. CSAT has a similar limitation: it captures sentiment, not conformance. Customer sentiment and adherence to your standards answer different questions, so neither should stand in for the other.

    Build the dashboard around four layers. Keeping them separate prevents a good aggregate number from concealing a weak customer experience.

    Measurement layerQuestion it answersSignals to trackDecision it supports
    Customer outcomeDid the customer get useful help?Resolution outcome, recontact, escalation outcome, sentimentWhether the interaction solved the customer’s problem
    Execution qualityDid the conversation meet your operating standard?Accuracy, process adherence, clarification, escalation ease, efficiencyWhether the agent behaved correctly
    Evaluation coverageHow much of the operation did you actually inspect?Eligible, automatically evaluated, human-reviewed, and still-unreviewed conversationsHow much confidence to place in the quality result
    Product impactDid better help change customer behavior?Activation, feature adoption, retention, and journey completion by cohortWhether the CX improvement created durable value

    Read the layers together, but don’t merge them into one executive number. High sentiment with low accuracy can mean the agent sounded helpful while giving the wrong answer. A strong scorecard result with poor sentiment can expose a technically correct but difficult experience. Good conversation quality with repeated contacts may mean the product, policy, or documentation still creates the underlying problem.

    The product-impact layer is especially important. A support answer may pass every conversational check without improving the journey that matters. Connect CX data to activation, adoption, and retention behavior so you can distinguish a better answer from a better customer outcome.

    A simple driver tree makes that connection explicit. Start with the business result, trace it to the customer behavior that produces it, identify the journey friction blocking that behavior, and then define the AI behavior that should remove the friction. If you can’t trace a proposed quality criterion through that chain, it may be a preference rather than a requirement.

    Design monitoring as a portfolio, not a random sample

    Monitoring begins with selection. A precisely calculated score from the wrong population creates false confidence. Small random samples are useful for trend stability, but they are unlikely to expose every high-risk edge case, complex escalation, or early sign of drift.

    Use four complementary monitoring layers:

    1. A stable benchmark cohort. Evaluate a repeatable sample on a consistent schedule. Preserve the same eligibility rules and segmentation so changes in pass rate represent changes in performance rather than changes in the sample.
    2. Risk-targeted monitors. Select conversations with signals that deserve deliberate review. Examples include a customer showing signs of financial vulnerability, an agent repeating essentially the same answer, a required escalation that did not happen, or a sensitive process being handled without its required checks.
    3. Change-specific monitors. Create cohorts for a new model, prompt, workflow, tool, policy, or knowledge release. Version every relevant component so a quality change can be traced to the production change that preceded it.
    4. Journey monitors. Group conversations by customer intent and journey stage, not just channel or queue. This exposes recurring friction in onboarding, adoption, billing, account management, and other product journeys that an operation-wide average will flatten.

    Instrument every eligible conversation, but don’t assume every conversation needs human review. Automated evaluation is appropriate for high-volume, clearly defined criteria. Human judgment belongs on critical failures, ambiguous cases, disputed scores, new scenarios, and calibration cohorts. The scalable design is broad automated visibility with concentrated human attention.

    Each monitor should report more than a pass rate:

    • Coverage rate: completed evaluations divided by eligible conversations. Never present pass rate without this denominator.
    • Pass rate: completed evaluations that met the scorecard standard divided by all completed evaluations.
    • Critical failure rate: completed evaluations that failed at least one critical criterion divided by all completed evaluations.
    • Unreviewed queue: qualifying conversations that have not received their required review. Break this out by risk and age rather than showing only a total.
    • Evaluator overturn rate: dual-reviewed cases in which a human changed the automated judgment. A rising rate can signal an ambiguous rubric, a weak evaluator, or a new conversation pattern.
    • Failure recurrence: previously addressed failure modes that appear again after a fix. This distinguishes completed work from effective work.

    Segment these measures by AI versus human handling, journey, intent, customer segment, channel, and deployed version. One shared quality system can compare AI and human conversations against the same customer outcomes, while still assigning different operational responsibilities. Unified review across automated and human conversations also makes handoff failures visible; otherwise, each side can look healthy while the transition between them breaks.

    Be deliberate about the data attached to monitoring. Store the fields needed for segmentation, diagnosis, and audit, and restrict access to sensitive conversation content. A monitoring program creates risk if it spreads personal or regulated data into dashboards that were designed only for aggregate analytics.

    Turn scorecards into executable product requirements

    A monitor determines what enters review. A scorecard determines how the conversation is judged. That distinction is operationally important: targeted selection and custom evaluation criteria work as two separate controls.

    I treat a scorecard as an executable product requirement. Each criterion should be observable in the conversation, interpretable by two independent reviewers, and connected to a specific action when it fails. Vague criteria such as helpful, natural, or on-brand produce arguments rather than reliable signals.

    Build each criterion with the following fields:

    1. Intent: the customer or business outcome the criterion protects.
    2. Pass condition: the observable behavior that must be present.
    3. Failure condition: the observable behavior that makes the conversation fail.
    4. Evidence rule: the part of the transcript, tool trace, policy, or approved knowledge that supports the judgment.
    5. Applicability: when the criterion is required and when it is not applicable.
    6. Severity: whether failure contributes to a weighted score or overrides the entire evaluation.
    7. Reviewer: automated evaluation, human evaluation, or both.
    8. Remediation owner: the person or function expected to act on failure.

    A practical CX scorecard usually needs criteria such as:

    • Accuracy: the answer is supported by approved knowledge, customer context, and tool results. Unsupported claims fail even if the customer accepts them.
    • Resolution and next step: the agent answers the request, clearly states what remains, or routes the customer to the correct next action.
    • Process adherence: required verification, disclosure, permission, and workflow steps are completed in the correct context.
    • Clarification: the agent asks for missing information when intent or account context is too ambiguous to answer safely.
    • Escalation: the agent recognizes defined handoff conditions, escalates without unnecessary resistance, and transfers the context the next agent needs.
    • Conversation efficiency: the agent avoids repetition, irrelevant steps, and loops while preserving the information necessary for a correct outcome.
    • Communication quality: the response is clear, appropriately direct, and consistent with the brand’s communication standard.

    Weights help express relative importance, but a weighted average must not wash away a consequential failure. Mark accuracy, safety, security, required process steps, or other non-negotiable controls as critical where appropriate. A polite answer that gives a harmful instruction should not pass because it accumulated enough points elsewhere.

    For legal, financial, safety, account-security, or regulated decisions, follow the applicable organizational policy and require human review or escalation where that policy demands it. An aggregate quality score is not authorization for an AI agent to make a decision outside its approved scope.

    Automated evaluators also need calibration. They are measurement components, not ground truth. Build an adjudicated set of clear passes, clear failures, difficult edge cases, and not-applicable examples. Have human reviewers score the same cases independently, compare their reasoning with the automated result, and rewrite any criterion that allows materially different interpretations. Repeat calibration after changes to the model, evaluator, prompt, tools, knowledge, policy, or conversation mix.

    Keep the evidence behind every automated judgment. A score without the relevant transcript excerpt or trace is difficult to challenge and nearly impossible to improve. Reviewers should be able to see what failed, why it failed, and which requirement governed the decision.

    Make every failure end in a product decision

    Quality monitoring creates value only when it changes the system. The review queue should represent work, not a museum of bad conversations. A useful workflow moves each case through explicit states such as Not reviewed, Reviewed, Needs a fix, and Fix complete.

    Require the following information before a failed case leaves review:

    • The customer intent and journey stage.
    • The failed criterion and supporting evidence.
    • The failure class, not merely the visible symptom.
    • The owner responsible for the correction.
    • The proposed change and the signal expected to improve.
    • The regression case, deployment version, and monitor that will verify the correction.

    A consistent failure taxonomy keeps teams from treating every bad answer as a prompt problem:

    • Knowledge failure: the approved information is missing, stale, contradictory, or too difficult to interpret.
    • Retrieval or context failure: the right information exists, but the system did not retrieve, rank, or apply it.
    • Policy or workflow failure: the operating rule is wrong, incomplete, or impossible for the agent to execute.
    • Model behavior failure: the system ignored instructions, made an unsupported inference, or produced an otherwise defective response despite receiving adequate context.
    • Conversation-design failure: the interaction collected the wrong information, asked an unclear question, or sequenced the exchange poorly.
    • Tool or handoff failure: an integration, action, routing rule, or transfer prevented the correct outcome.
    • Product-friction failure: the support interaction is a recurring symptom of something the product itself should make clearer or eliminate.

    This taxonomy changes prioritization. If the same onboarding question keeps passing through support, improving the answer may reduce handling friction but preserve the underlying product problem. Journey mapping, behavioral analytics, and in-product guidance can reveal whether the better fix belongs in the interface, workflow, documentation, or agent.

    Close each failure through a controlled loop:

    1. Reproduce it. Confirm the failure and preserve the relevant inputs, context, versions, and tool behavior.
    2. Diagnose it. Assign a cause from the shared taxonomy and identify the owner with authority to change that component.
    3. Correct it. Update the product, knowledge, retrieval logic, prompt, workflow, policy, integration, or escalation rule that caused the defect.
    4. Test it. Run the original failure and nearby cases that should remain unchanged. Add the sanitized case to the regression set where appropriate.
    5. Deploy it with versioning. Preserve enough release context to compare behavior before and after the change.
    6. Verify it in production. Watch the targeted monitor, baseline quality, and associated customer behavior. Close the case only when the expected signal improves without creating a new failure elsewhere.

    Bring this loop into a regular operating review. Inspect coverage first, then critical failures, the largest changes by segment and version, queue health, recurring causes, completed fixes, and downstream customer behavior. The meeting should end with product decisions: roll back a change, revise knowledge, alter a workflow, strengthen an escalation, change the interface, expand monitoring, or explicitly accept a known limitation.

    Assign ownership before volume grows. CX and support leaders can define service standards; product leaders can connect recurring friction to roadmap decisions; AI and engineering owners can maintain instrumentation, evaluators, and regression tests; analytics can connect interactions to behavioral outcomes; and security, legal, or compliance owners can approve critical controls in their domains. Your organization may divide the roles differently, but every critical criterion and failure class needs a named decision owner.

    Key takeaways

    • Resolution rate measures throughput, not whether an AI interaction was accurate, compliant, or useful.
    • Track customer sentiment, execution quality, evaluation coverage, and product impact as separate layers.
    • Combine a stable benchmark cohort with risk-targeted, change-specific, and journey-based monitoring.
    • Show pass rate with coverage and critical failure rate. A strong score over thin or biased coverage is weak evidence.
    • Write scorecard criteria as executable requirements with pass conditions, failure evidence, severity, reviewer type, and remediation ownership.
    • Use automated evaluation for breadth and human judgment for calibration, ambiguity, disputes, and consequential failures.
    • Don’t close a quality issue when a document or prompt changes. Close it after the deployed fix passes regression checks and improves the intended production signal.
    • Route recurring conversation failures into product discovery. Sometimes the best CX fix is removing the reason customers need to ask.

    Start with one journey that combines meaningful volume with meaningful consequence. Give it an eligibility rule, a scorecard, explicit coverage, a review queue, a failure taxonomy, named owners, regression cases, and a downstream product outcome. Run that chain until failures reliably produce verified changes, then extend the same control loop to the next journey. Scalable quality comes from repeating a dependable operating system, not from adding another dashboard.

    References

  • Bad Advice from Your AI Clone? Ethics, IP, and How Product Leaders Protect Quality

    Bad Advice from Your AI Clone? Ethics, IP, and How Product Leaders Protect Quality

    What happens when an AI starts giving advice in your voice—advice you’d never actually give? I’ve been thinking a lot about that question, and this conversation hit home for me as a product leader navigating the fast-evolving reality of AI “clones.”

    Listen to this episode on: https://open.spotify.com/episode/7DNDIlIimwbbMOytArewRp?ref=producttalk.org | https://podcasts.apple.com/kh/podcast/bad-advice/id1794203808?i=1000756914818&ref=producttalk.org. Prefer video? Watch on YouTube: https://www.youtube.com/embed/RF4BwaeMMlg?feature=oembed

    The episode examines AI “clones” built from podcast transcripts and public content—where the experimentation feels exciting, where it crosses ethical lines, and what happens when mediocre AI outputs get attributed to real people. The tension is real: when a bot confidently answers in your style but misses the nuance, “it’s not me” becomes more than a disclaimer—it’s a reputational defense.

    We dig into the messy parts: IP ownership of open-sourced transcripts, the role of pirated books in LLM training sets, rising inference costs, and the uncomfortable economic question: if anyone can prompt “act like Teresa,” how do creators make a living? In my own decision-making, I look for clear consent, guardrails that prevent impersonation, and transparent UX that never confuses a synthetic perspective with a human expert.

    This isn’t anti-AI. It’s a nuanced conversation about quality, consent, and remembering there are real humans behind the ideas.

    Here’s how I translate the key takeaways into practice. Using AI for perspective is fine—equating it to the real person isn’t. Free-feeling AI outputs still rely on someone’s work. Expertise is more than past content—it’s context, judgment, and evolution. If someone’s work influences you, find a way to support them. These principles help teams benefit from gen ai without eroding trust or the creator ecosystem.

    “Technically possible” doesn’t mean “ethically okay.” My AI Strategy playbook includes privacy-by-design, clear data governance on training materials, and a bright line between inspiration and impersonation. When we ship AI features, we label synthetic outputs, avoid mimicking living experts without permission, and create paths to compensate or promote the humans whose thinking underpins the experience.

    I’ve also tested the “act like X” pattern to stress-test product quality. Even when outputs sound plausible, they rarely capture the expert’s mental models, trade-offs, or the evolution of their thinking—especially in complex product discovery work. That gap is the difference between average AI text and expert product management leadership.

    If you listen, consider a few reflection prompts: Have you ever used AI to “act like” someone you admire? Could you tell whether the output matched that person’s actual thinking? How do you decide what’s ethically okay when using public content in LLMs? And how can we support creators while still embracing new tools?

    Resources & Links you may find helpful: Follow Teresa Torres: https://ProductTalk.org; Follow Petra Wille: https://Petra-Wille.com; Delphi.ai (AI bot platform discussed): https://www.delphi.ai/?ref=producttalk.org; Lenny’s Podcast: https://www.lennysnewsletter.com/podcast?ref=producttalk.org; ChatGPT: https://chatgpt.com/?ref=producttalk.org; Petra’s Coaching Packages: https://www.petra-wille.com/coaching-packages?ref=producttalk.org; Teresa’s Product Talk: https://www.producttalk.org/; Teresa’s book Continuous Discovery Habits: https://www.producttalk.org/continuous-discovery-habits/; Lenny’s open-sourced podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&e=1&st=ahz0fj11&dl=0&ref=producttalk.org

    Have thoughts on this episode or practices that have worked in your org? Share them below—I’m keen to learn how other teams are balancing innovation with integrity.

    Inspired by this post on Product Talk.

    Book a consult png image
  • How to Build Scalable, AI-Ready Product Documentation

    How to Build Scalable, AI-Ready Product Documentation

    Your AI assistant gives a confident but outdated setup answer. Search returns three pages with slightly different instructions. Support knows the real workaround, but the documentation owner does not know the product changed. This is usually described as an AI problem. It is more often a knowledge-system problem.

    You do not need a second documentation estate written for machines. You need one governed source of product truth that a customer can follow, a support engineer can trust, and an AI system can retrieve without reconstructing the answer from conflicting fragments.

    Key takeaways

    • Organize documentation around the questions and tasks users bring to it, not only around your product navigation or internal team structure.
    • Give every important section a clear answer, scope, procedure, expected result, and permanent link so it remains useful when retrieved on its own.
    • Control terminology, versions, ownership, and deprecation explicitly. An AI assistant cannot reliably resolve contradictions that your organization has left unresolved.
    • Put documentation changes through version control, review, automated checks, and release gates so the published truth keeps pace with the product.
    • Measure successful task completion and grounded answer quality, not page views alone. Use failures to decide whether to fix the content, retrieval layer, assistant behavior, or product itself.

    Start with an answer contract, not a page inventory

    A documentation redesign often begins with a list of existing pages. That tells you what you publish, but not what customers need to accomplish. It also preserves accidental boundaries: a feature may have five pages because five teams touched it, while the customer still sees one task.

    Begin with an intent register for one product area. Capture the questions that appear during activation, onboarding, routine use, escalation, and renewal. Include the language people actually use in search queries and support requests, even when it differs from your preferred product terminology.

    For each intent, record:

    • The user’s question in their own language.
    • The task they are trying to complete or the decision they need to make.
    • The relevant audience or role, such as administrator, developer, or analyst.
    • The product version, plan, permission, integration, or prerequisite that changes the answer.
    • The canonical page and section that should answer the question.
    • The person accountable for keeping that answer current.
    • The consequence of a wrong or missing answer, such as failed activation, an unnecessary escalation, or use of a deprecated workflow.

    This register exposes three different problems that page counts conceal. Some important questions have no answer. Some have several competing answers. Others have an answer that exists but cannot stand on its own because the conditions or expected result appear somewhere else.

    Turn each priority intent into an answer contract. A complete unit should state what the user can accomplish, when the instructions apply, what must already be true, what to do, what success looks like, and where to go next. If any of those elements are missing, a human has to infer them and an AI system may invent the bridge.

    The opening of a page should therefore name the job, not advertise the feature. “Configure routing for inbound leads” gives the reader a destination. “About lead routing” merely names a subject. This small distinction also gives retrieval systems a stronger match between a real question and the section intended to answer it.

    Build retrieval units that still make sense alone

    A person may enter through a search result, while an AI application may retrieve only a passage from the middle of a page. In both cases, the selected section has to survive separation from the surrounding document.

    That does not mean chopping every page into tiny fragments. Atomic content is complete enough to answer one intent and bounded enough to avoid unrelated material. A fragment that says “click Save” without naming the object, required permission, or expected result is short, but it is not atomic.

    Use a repeatable section pattern

    For a task-oriented section, use this sequence:

    1. Write a heading that reflects the question or task.
    2. Give the direct answer or outcome before background material.
    3. State who the instructions are for and when they apply.
    4. List permissions, inputs, and prerequisites before the procedure.
    5. Use numbered steps with one observable action in each step.
    6. State the expected result and how the reader can verify it.
    7. Separate exceptions, limitations, and failure states from the main path.
    8. Link to the next likely task rather than a generic documentation landing page.

    Keep interface labels, API parameters, status values, and error messages verbatim. If the product displays “Connection expired,” do not rewrite it as “Your integration is no longer active.” The second phrase may read naturally, but it weakens exact search, obscures the product state, and makes support instructions harder to match.

    Examples should expose inputs, outputs, and constraints. A useful example says which role is acting, what value is supplied, what the system returns, and which condition would make the result different. A screenshot without that context is evidence of appearance, not a durable explanation of behavior.

    Make boundaries and links dependable

    Use one primary topic per page, semantic H1-H3 hierarchy, descriptive slugs, and stable section anchors. These practices make pages easier to scan and create smaller, linkable units that retrieval systems can identify precisely.

    A stable anchor is part of the content contract. If an implementation guide links directly to the authentication prerequisite, changing that anchor silently breaks more than navigation. It breaks the path by which customers, support macros, release notes, and AI responses reach the authoritative answer.

    Do not copy the same procedure into several pages to make each page self-contained. Keep one canonical procedure and give adjacent pages enough context to explain why the reader needs it, followed by a precise link. Duplication feels convenient at publication time and becomes a contradiction risk at the next product change.

    Control vocabulary without ignoring customer language

    Choose one canonical term for each product concept across the interface, API, documentation, and support material. Put accepted synonyms and older names in a glossary or metadata field so search can recognize them, but keep the explanation anchored to the current term.

    This is the difference between supporting natural language and allowing synonym sprawl. “Workspace,” “account,” “tenant,” and “organization” may sound interchangeable inside a company. If they represent different objects in the product, casual substitution creates false equivalence. If they represent the same object, choosing one term removes needless translation work for every reader and retrieval pipeline.

    Protect the current truth with metadata and delivery controls

    Good prose cannot compensate for missing scope. Two instructions can each be correct for a different version, role, or integration and still produce a wrong answer when retrieved together. Metadata makes those boundaries explicit before retrieval begins.

    Define a required metadata contract for every governed page or content unit. At minimum, include:

    • A stable content ID and canonical URL.
    • A descriptive title and short task-oriented description.
    • The product area and content type.
    • The intended audience or role.
    • The applicable version or version status.
    • The lifecycle state, such as current or deprecated.
    • The accountable owner.
    • The last-updated or last-reviewed date.

    Use the fields as controls, not decoration. Audience metadata should allow an assistant to distinguish administrator instructions from end-user instructions. Version metadata should prevent a current answer from silently incorporating an obsolete step. Ownership should route a failed evaluation to someone who can resolve it.

    Deprecation needs more than a warning banner. State what is deprecated, which users or versions are affected, what replaces it, and how to move forward. Preserve old URLs with redirects when a current replacement exists. Removing the old page without a forward path turns bookmarks and deep links into dead ends; leaving it searchable without a clear status lets obsolete guidance continue to circulate.

    Ship documentation as part of the product change

    Scalability depends on the delivery system behind the content. Version control, peer review, and CI/CD give documentation the same traceability and release discipline used for software changes.

    For each product change, the release workflow should answer:

    • Which user intents and canonical sections are affected?
    • Do interface labels, parameters, permissions, errors, examples, or screenshots change?
    • Does the change introduce a new term or alter an existing definition?
    • Do version boundaries, redirects, or deprecation notices need updating?
    • Which retrieval evaluations must pass before release?
    • Who approves the content and owns follow-up corrections?

    Automate the checks that have unambiguous pass or fail conditions: broken links, missing required metadata, duplicate IDs, invalid internal references, and orphaned pages. Use human review for semantic accuracy, task completeness, terminology, and whether an image still reflects the current workflow. Automation can detect that a screenshot file exists; it cannot reliably decide that the image teaches the correct behavior.

    Set update expectations according to consequence. Instructions tied to a product release need to be correct when the change reaches users. A deprecated workflow needs a forward path before the old path disappears. Lower-risk explanatory material can follow a review schedule. One blanket service level treats cosmetic drift and activation-breaking errors as if they carry the same cost.

    Measure answer quality, then migrate in risk order

    Page views tell you that someone arrived. They do not tell you whether the person completed the task or whether an AI answer was accurate, grounded, and current. Pair human behavior with retrieval evaluations so each signal leads to a plausible corrective action.

    SignalWhat it can revealLikely action
    Repeated searches or rapid returns to resultsThe answer is hard to find, uses mismatched language, or does not resolve the intentImprove the title, intent mapping, vocabulary, or section completeness
    Low task completion after readingThe procedure may omit prerequisites, verification, or a failure pathTest the instructions against the actual workflow and repair the answer contract
    Support escalation after a documentation visitThe content may be incomplete, untrusted, outdated, or describing product frictionInspect the escalation reason before assuming more content is the solution
    Low answer accuracy or groundingThe wrong passage was retrieved, the selected passage conflicts with another, or the assistant exceeded the evidenceSeparate retrieval, content, and answer-generation failures
    Current and deprecated guidance in one answerVersion metadata, lifecycle labels, or retrieval filters are insufficientStrengthen version boundaries and remove obsolete material from current-answer paths
    High response latencyThe retrieval or answer path may be doing unnecessary workInspect the pipeline without trading away accuracy or grounding

    Build the evaluation set from the same intent register used to design the documentation. For each test question, define the expected canonical page or section, the claims a correct answer must contain, the audience and version it applies to, and any deprecated claim that must not appear. Include questions that should not be answered when the documentation lacks enough evidence. A reliable assistant must be able to stop at the boundary of the known answer.

    When a test fails, classify the failure before editing anything:

    • If retrieval selected the wrong section, inspect information architecture, headings, metadata, vocabulary, and chunk boundaries.
    • If retrieval selected the correct section but the answer distorted it, inspect the assistant’s instructions and answer-generation behavior.
    • If two selected sections disagree, resolve the underlying ownership, versioning, or duplication problem.
    • If no section answers the question, add the missing knowledge or make the limitation explicit.
    • If the answer is correct but users still fail, inspect the procedure and the product experience. Documentation should not be used to disguise avoidable product friction.

    You do not need to rebuild the entire knowledge base before learning whether this operating model works. Migrate in this order:

    1. Choose one product area with meaningful activation, support, or deprecation risk.
    2. Collect its real user intents and map each one to an accountable answer.
    3. Resolve duplicate, contradictory, and missing guidance before changing the retrieval system.
    4. Restructure priority answers into self-contained, linkable sections.
    5. Add the required metadata, ownership, version, and lifecycle controls.
    6. Put those sections through the product release workflow and automated checks.
    7. Run human task checks and retrieval evaluations, classify the failures, and repair the responsible layer.
    8. Expand only after the pattern is repeatable for another product area.

    Your first useful deliverable is not an AI documentation strategy deck. It is one high-value customer question with one canonical, current, owned answer that survives retrieval and changes alongside the product.

    Start with the question that creates the most expensive ambiguity today. Make its answer complete, linkable, versioned, testable, and part of the release path. That single vertical slice will show you where the larger system actually needs work.

    References

  • A Practical Strategy for Foundational AI Platforms and Analytics

    A Practical Strategy for Foundational AI Platforms and Analytics

    Your AI roadmap is filling up, but every new use case seems to require another custom retrieval pipeline, another evaluation method, and another analytics implementation. The pilots may work. The portfolio does not yet compound.

    You do not solve that problem by declaring an AI platform initiative and assembling a long infrastructure backlog. You solve it by identifying the decisions your products must support, defining the contracts every AI workflow must honor, and connecting evaluation to real product behavior. The result is a foundation that makes the next useful experiment easier to ship, safer to operate, and easier to learn from.

    Make critical use cases define the platform

    A platform team can spend months building abstractions that application teams neither understand nor need. The usual cause is starting with components: a model gateway, vector storage, a feature store, an evaluation tool, or a new analytics stack. Each may be useful, but none tells you what the platform must make easier.

    Start with the highest-value AI decisions in your product and internal operations. A use case is specific enough when you can describe the person making the decision, the context the system requires, the action the AI is allowed to take, the unacceptable failure modes, and the outcome you expect to change.

    Write a short capability brief for each critical use case:

    • User and moment: Who is trying to do what, and where does the AI enter the workflow?
    • Decision or action: Is the system recommending, drafting, classifying, retrieving, or acting autonomously?
    • Required context: Which behavioral events, account data, documents, permissions, and prior actions affect the answer?
    • Quality definition: What makes an output acceptable, and which errors matter most?
    • Release evidence: What must pass before the change reaches users?
    • Product outcome: Which user behavior or business result should move if the capability works?
    • Operating constraints: What must be logged, redacted, approved, monitored, or reversible?
    • Owner: Who owns the outcome after launch, rather than merely delivering the component?

    Now compare the briefs. Repeated needs are platform candidates. If several use cases require permission-aware retrieval, consistent experiment assignment, traceable prompt versions, or the same release evaluation, those capabilities deserve shared interfaces. A requirement that appears once may belong inside the application until reuse is proven.

    This distinction prevents two expensive mistakes. The first is premature generalization: building a universal service before you understand the variation. The second is copy-and-paste scaling: allowing every team to create incompatible versions of a capability that is already clearly common.

    Prioritize the platform backlog by friction removed from critical use cases, not by architectural elegance. A useful backlog item should complete a sentence such as: After this ships, a product team can run the standard evaluation suite on every retrieval change without building its own runner. If the item cannot name the team behavior it changes, it is probably still an implementation idea rather than a platform outcome.

    Build the foundation as four enforceable contracts

    A foundational AI platform is not one product or one technical layer. It is a set of contracts connecting data, evaluation, delivery, and analytics. The contracts matter more than whether every component comes from the same vendor. They let application teams move independently without giving up consistency where consistency is valuable.

    1. The data and context contract

    This contract defines what context an AI workflow can request and what comes back. It should cover identity, permissions, event definitions, document metadata, freshness, provenance, and retention. Retrieval should enforce access rules before context reaches the model, not rely on the model to decide what a user may see.

    Keep the interface narrow. An application should be able to request approved context for a known user, account, and task without understanding every underlying data system. The response should carry enough metadata to explain where the context came from and which version was used.

    A feature store belongs here when several predictive or real-time workflows need the same feature definitions at training and serving time. It should not become a mandatory platform component merely because feature stores appear on AI architecture diagrams. Add one when inconsistency is a demonstrated problem.

    2. The evaluation contract

    This contract defines the evidence required to change a model, prompt, retrieval configuration, tool, or policy. It should include representative test cases, expected behavior, failure labels, scoring rules, comparison baselines, and release gates.

    Do not reduce evaluation to one average score. A harmless wording variation and a permission leak cannot cancel each other out. Track important failure classes separately and make critical failures blocking. Keep examples that exposed production problems so the evaluation set becomes a memory of what the system has learned.

    The evaluation harness should produce a result that a product manager can interpret and an automated delivery pipeline can enforce. That is the core of eval-driven development: model and workflow changes move through repeatable evidence rather than one-off demonstrations.

    3. The delivery and operations contract

    This contract governs how a tested change reaches production and how you recover when it behaves badly. It should preserve the versions of the model, prompt, retriever, tools, policies, and relevant data configuration associated with each release.

    Use controlled rollout, feature flags, rollback paths, and CI/CD gates for AI changes just as you would for other consequential product changes. Observability should connect a production trace to the exact configuration that produced it. Otherwise, a team can detect a quality drop without being able to isolate whether the model, context, prompt, tool call, or upstream data changed.

    Record enough to diagnose the workflow, but do not treat raw prompts and conversations as ordinary telemetry. Apply privacy-by-design: minimize collection, redact sensitive fields where appropriate, control access, and make retention a deliberate policy. The default path should be the governed path. If secure operation requires every application team to remember a separate checklist, the platform has not removed the risk.

    4. The product analytics contract

    This contract maps AI interactions to product behavior. Define events for exposure, acceptance, correction, rejection, fallback, task completion, and abandonment where those states apply. Include stable identifiers that connect an interaction to its release and experiment assignment without copying sensitive content into the analytics layer.

    The contract should also specify which product metric each AI capability is expected to influence. That keeps teams from declaring success because response quality improved in an offline test while users ignore the feature or fail to complete the task.

    Turn analytics into the AI learning and control loop

    Traditional product analytics asks what users did. AI evaluation asks how a probabilistic workflow performed. You need both views joined at the interaction level. Without that connection, model teams optimize scores, product teams optimize funnels, and neither can explain why a release helped or hurt the user.

    Use four layers of measurement, and do not blend them into one health score:

    <!– wp:list {
  • A Product Leadership System for Faster, Clearer Execution

    A Product Leadership System for Faster, Clearer Execution

    Your roadmap is full, every function has a planning ritual, and experienced people are working hard. Yet decisions still wait, priorities keep reopening, and substantial work reaches customers later than anyone expected. Adding another process layer will not solve that problem.

    You need an execution system: explicit ownership, small batches, a dependable decision cadence, direct customer feedback, and a scorecard that distinguishes progress from activity. When those elements reinforce one another, your teams can move faster without lowering the quality bar or routing every judgment through you.

    Give each team an operating contract, not just a roadmap

    A roadmap identifies intended destinations. It rarely tells a team how to make the decisions required to reach them. That gap is where autonomy turns into ambiguity: product believes it owns the sequence, engineering waits for scope to stabilize, design explores a wider problem, and an executive assumes a requested feature is already committed.

    Before an initiative becomes active work, give the team a short operating contract. It should fit on one page and answer these questions:

    • Whose problem are you solving, and in what specific scenario does it occur?
    • What observable customer or business outcome should change if the work succeeds?
    • Who is accountable for the initiative and its sequence?
    • Which constraints are fixed, and which assumptions remain open?
    • What is explicitly outside the current scope?
    • What is the smallest end-to-end slice that can produce useful evidence?
    • What evidence will support the next decision?
    • When will that decision be made, and who has the right to make it?

    The owner is not the person who approves every task. The owner keeps the problem, outcome, sequence, and unresolved decisions coherent. Engineering, design, research, and product still make solution decisions together inside the stated boundaries.

    This contract also protects the team from executive requests that arrive as solutions without context. When someone asks for a feature, do not turn the request directly into a backlog item. Translate it into a problem entry first: the affected customer, the workflow that breaks, the evidence behind the request, the relevant constraint, and the result the requester expects. A commercially important request can remain urgent after that translation, but the team can now evaluate it rather than merely obey it.

    Set escalation boundaries at the same time. A team should escalate when decision rights are unclear, two constraints conflict, a priority change affects another team, or the work crosses an agreed risk boundary. It should not need escalation merely because a solution choice is consequential. If every consequential choice travels upward, the team is not autonomous; it is a queue feeding a senior leader.

    Finally, maintain one prioritized backlog for the team. Separate executive, product, engineering, and sales backlogs create hidden competition. The operating contract establishes the logic, and the single backlog makes the resulting sequence visible.

    Run a weekly loop around decisions and customer learning

    Many product cadences organize meetings while leaving decisions to happen unpredictably. A useful cadence does the opposite. Every recurring touchpoint should help the organization choose, learn, or remove a constraint.

    A workable leadership week looks like this:

    • Monday: Confirm the few priorities that matter, identify decisions that could block progress, and resolve changes in sequencing. Do not reread the entire roadmap.
    • Midweek: Review selected product requirements, design flows, research findings, and engineering readiness. Concentrate on ambiguity, batch size, and untested assumptions.
    • Thursday: Spend time with customers and partners. Put working slices in front of them when possible, and bring the resulting evidence back to the team.
    • Friday: Write down what changed in your understanding. Update the backlog, decision log, and operating contracts where the evidence warrants it.

    The sequence matters. Monday establishes intent. Midweek exposes execution risk while there is still time to change course. Customer contact tests the team’s reasoning. Friday turns scattered observations into organizational memory. Without the synthesis step, customer conversations can become interesting anecdotes that never alter a decision.

    Make the weekly demo the heartbeat of the team. A good demo starts with the user scenario and the intended outcome, shows the smallest working behavior, states what the team learned, and ends with the next decision. A tour of completed tickets is not a substitute. For platform or infrastructure work, demonstrate working behavior, operational evidence, or a retired technical risk rather than manufacturing a customer-facing screen.

    When a team repeatedly has nothing meaningful to demonstrate, inspect the system before questioning effort. The batch may be too large. A dependency may lack an owner. Decisions may be waiting in an approval queue. The team may be building several disconnected components before completing one testable path. The correction is usually to narrow the slice, clarify the decision, or remove the dependency.

    A thin slice is not an arbitrary reduction in scope. It must preserve one coherent scenario, reach a state where someone can evaluate it, and create evidence for a consequential next choice. Backend, frontend, and enablement tasks can all be necessary, but completing them separately does not create a feedback loop.

    Put product and revenue in the same operating loop

    Product and revenue drift apart when they maintain different versions of the customer. Product sees research themes and usage behavior. Revenue sees active deals, objections, urgency, and willingness to pay. Neither view is sufficient on its own.

    Use one customer narrative, one shared pipeline of problems worth solving, and one scorecard. Review them together every week. Each proposed problem should carry the customer segment, affected workflow, available evidence, commercial context, expected outcome, and complexity the solution could add.

    Then make the sequencing decision explicit:

    • Solve now: The problem is important enough, supported well enough, and compatible with the current strategy.
    • Stage for scale: The need is credible, but the team must first validate the pattern, build a reusable foundation, or resolve a dependency.
    • Do not add: The request is too narrow, conflicts with the product direction, or creates complexity that its value does not justify.
    • Sunset: Existing functionality consumes attention without contributing enough customer value or strategic leverage.

    This turns product-versus-sales conflict into a visible portfolio decision. Revenue contributes evidence and urgency. Product protects coherence and long-term defensibility. Both functions see why an item moved, waited, or stopped.

    Measure outcomes, flow, and quality as separate signals

    A team can ship frequently without improving a customer outcome. It can also improve an outcome temporarily while accumulating quality problems that make the pace unsustainable. Your scorecard needs to keep those conditions separate.

    For each important bet, review three signal groups:

    • Outcome: The observable customer or business result the team is trying to change, supported by current evidence rather than a list of releases.
    • Flow: Deployment frequency and the age or state of the current thin slice. These signals reveal whether value and learning can move through the system.
    • Quality: Change failure rate and the recurring friction exposed in customer feedback, support conversations, or postmortems.

    Use the scorecard to direct attention, not to automate judgment. If deployment frequency is healthy but the intended outcome is not moving, inspect the hypothesis, target customer, and value proposition. More releases may simply deliver the wrong idea faster. If deployment frequency falls, examine batch size, dependencies, and delayed decisions. If change failure rate worsens, narrow the slice and strengthen readiness or recovery before asking the team to accelerate.

    Do not rank unlike teams by raw deployment counts. Use trends within the relevant product and technical context. The point is to find constraints and make decisions, not to turn a diagnostic signal into a performance contest.

    Write outcome-focused OKRs with enough precision to guide a trade-off. A useful structure is: for a named user and scenario, improve an observable result from its current baseline toward an agreed target by the review point, without damaging a stated guardrail. Establish the baseline before debating the target. If the team cannot observe the result, say that plainly and make instrumentation or customer evidence part of the initial slice.

    Feature count, roadmap completion, tickets closed, and activity volume can help with local planning. They are not proof of customer value. Treat them as operational context, not as the headline definition of success.

    Keep the executive view compact. Each team should be able to present its intended outcome, current evidence, deployment-frequency trend, change-failure trend, most important customer learning, and next unresolved decision. If a metric never changes a question, a priority, or an intervention, remove it from the review.

    Stay close to the work without taking the work away

    Product leaders lose judgment when they only consume summaries. They become bottlenecks when they join every working session. The useful middle ground is deliberate sampling: inspect enough real work to calibrate your view, then give feedback that strengthens the team’s next decision.

    Each week, sample a rotating set of artifacts such as a product requirements document, a design flow, customer research notes, a postmortem, or a customer thread. You are not trying to approve every artifact. You are checking whether the operating system is producing clear thinking.

    Use questions that reveal decision quality:

    • Does the requirement name a user scenario and a problem, or does it begin with a predetermined feature?
    • Does the design expose a complete path that can be tested, or only polished fragments?
    • Do the research notes separate what customers did and said from the team’s interpretation?
    • Does the postmortem change an operating mechanism, or merely remind people to be careful?
    • Does the customer thread reveal a pattern, an important exception, or one loud request?
    • Can the team state the next decision this artifact is meant to support?

    Feedback should create motion. Name the user scenario, identify the friction or ambiguity, state the decision principle, propose a smaller testable slice when appropriate, and clarify the next decision. A vague comment such as make this more strategic forces the team to guess what you mean and then wait for another review.

    I use a simple leadership boundary: push hard on problem clarity, sequencing, and the quality bar; leave room on solution design and implementation. That boundary keeps accountability with leadership without converting senior judgment into remote-control product management.

    Exemplars make this boundary easier to scale. Keep a small, current library of strong problem statements, concise narrative memos, useful research syntheses, clear acceptance criteria, and honest postmortems. Show why each example is effective. Teams learn a quality bar faster from visible work than from an expanding rulebook.

    Create short paths for decisions and uncomfortable information

    Open office hours give anyone a direct route for a difficult escalation, unfinished design, customer insight, or cross-team conflict. Run them as a decision forum, not an extra status meeting. Capture the decision, owner, rationale, and follow-up so people who were not present can still act consistently.

    Keep weekly one-to-ones with your leaders as well. Office hours expose work across the organization; one-to-ones develop judgment, surface recurring constraints, and help a leader notice when someone is absorbing ambiguity on behalf of the system.

    Fast feedback from leadership matters because waiting expands batches. When teams expect a long approval cycle, they tend to gather more material and seek approval for more decisions at once. Publish clear decision rights and a dependable response path. If you do not need to make the decision, say so immediately and return it to the named owner.

    Spend unscripted time with individual contributors, too. Formal reporting lines filter information. Direct exposure to the people building, researching, designing, supporting, and selling the product helps you hear where the written process and actual work have diverged.

    Install the system without reorganizing first

    You do not need a company-wide transformation program to test this operating model. Start with one important initiative that is moving slowly or generating repeated disagreement. Keep the current reporting structure and change the mechanics around the work.

    1. Capture the current friction. Identify where the initiative waits, where priorities conflict, which decisions keep reopening, and where work returns for avoidable clarification.
    2. Write the operating contract. Name the problem, outcome, owner, constraints, non-goals, initial thin slice, required evidence, and next decision.
    3. Collapse the work into one sequence. Bring product, engineering, executive, and commercial requests into one prioritized backlog. Preserve their context rather than preserving separate queues.
    4. Run the weekly loop. Set priorities on Monday, inspect selected artifacts midweek, expose work and assumptions to customers, and synthesize the learning on Friday.
    5. Publish the compact scorecard. Show the intended outcome, deployment frequency, change failure rate, newest customer evidence, and next decision. Do not wait for a perfect dashboard.
    6. Inspect the mechanism after a full loop. Remove one gate that added waiting without adding learning, divide one oversized batch, and clarify one decision right that caused an escalation.

    During the review, ask concrete questions: What waited for a decision? What was redone because the original problem was unclear? Which customer signal changed the plan? Which metric caused an intervention? Which request arrived without enough context? Where did leadership provide useful boundaries, and where did it take ownership away from the team?

    Expand the model only after the team can explain how it changed actual work. Copying the ceremonies without the decision rights, customer exposure, and scorecard will create more meetings, not a stronger execution system.

    Key takeaways

    • Start each important initiative with a one-page operating contract that connects a real customer problem to an owner, outcome, constraints, thin slice, and next decision.
    • Protect autonomy with explicit boundaries. Escalate conflicting priorities and constraints, not every consequential solution choice.
    • Organize the week around decisions, working evidence, customer contact, and written synthesis rather than status reporting.
    • Read outcomes, deployment frequency, and change failure rate together. No single signal can tell you whether the team is delivering sustainable value.
    • Sample real artifacts and give specific feedback, while leaving solution and implementation ownership with the team.
    • Give product and revenue one customer narrative, one problem pipeline, and one scorecard so trade-offs become visible sequencing decisions.

    At your next Monday priority review, choose one live initiative and write its operating contract before discussing another roadmap change. The missing answers will show you where execution is actually slowing down. Fix that mechanism, run the loop, and let evidence determine where the system should expand next.

    References

    • Shivam.Consulting Blog — The CPO Playbook I Wish I’d Had: Ditch Bad Wisdom, Ship Faster, and Lead with Clarity
  • Outcomes vs Outputs: How I Stopped the Feature Factory and Drove Real Product Impact

    Outcomes vs Outputs: How I Stopped the Feature Factory and Drove Real Product Impact

    “Outcomes over outputs” is the right mantra—and one I’ve championed across product teams—but turning it into daily practice is where most teams stumble.

    It’s simple in theory: focus on the impact of what we build, not just shipping features. In reality, it’s rarely black and white because most teams are asked to do both—hit outcomes and deliver specific outputs—at the same time.

    In a benchmark survey, 20% of product teams claim to be outcome-focused, nearly half describe themselves as working in a mix of outcomes and outputs, and about 30% are still primarily working with outputs. I’ve seen versions of this in my own org: we aspire to outcomes, but our rituals, roadmaps, and reporting still reward shipping.

    Here’s how I draw the line clearly, coach my teams to avoid common traps, and negotiate better, more actionable outcomes that unlock genuine product discovery and business results.

    Simple definitions we live by

    An output is something you build or produce—a feature, a project, an initiative. It’s something your team ships.

    An outcome is the impact of that output—a change in customer behavior or a business result.

    Josh Seiden puts it well in his book Outcomes Over Output: “An outcome is a change in human behavior that drives business results.”

    Infographic comparing outputs vs outcomes in product management: outputs are what you ship—feature, project, integration; outcomes are what changes—customer behavior and business results; arrow notes where value happens.
    Shift from shipping to shaping results. This graphic clarifies outputs vs outcomes, revealing that value emerges between deliverables and impact—when features change customer behavior and move business results.

    I distinguish business outcomes from product outcomes. Business outcomes are typically financial metrics that measure the health of the business (e.g. increase revenue or reduce costs) while product outcomes measure a customer behavior in the product or a sentiment about the product.

    Here’s a simple example I’ve used with platform teams. Many B2B companies support a number of integrations. Integrations are outputs. Having integrations alone doesn’t create value. Customers using and finding value in those integrations—that’s an outcome. If those customers retain their subscriptions longer because of the integrations—that’s also an outcome.

    Building something isn’t the same as creating value. That’s the core of this distinction, and it’s what separates empowered product teams from feature factories.

    Why this distinction matters for empowered product teams

    When we task teams with delivering outputs, they’re done when the software ships. When we task teams with delivering outcomes, they aren’t done until the software ships and has the expected impact.

    That small shift changes almost everything about how a team works: what we measure (impact, not just delivery), how we know we’re done (measurable behavior change, not release notes), the autonomy we grant (told what to achieve, not what to build), and the planning artifacts we use (an opportunity solution tree beats a feature roadmap when we’re exploring the best path to an outcome).

    When I assign outcomes, I’m giving the team latitude—and responsibility—to figure out the best path to success. That’s what opens the door for real product discovery and continuous discovery habits.

    Infographic comparing output-driven vs outcome-driven teams, covering metrics measured, team autonomy, definition of done, and planning artifacts: feature roadmap vs opportunity solution tree.
    Shift your lens from shipping features to achieving impact. This side-by-side visual explains how outcome-driven teams measure success, grant more autonomy, define 'done' by results, and plan with an opportunity solution tree.

    Examples: spotting outputs disguised as outcomes

    Clear-cut example: “Our outcome is to deliver an Android app.” An Android app is something we build and ship. It’s clearly an output.

    To get to an outcome, I ask, “What’s the value of having an Android app?” or “How will we know the Android app is successful?”

    We might answer: “Having an Android app will allow us to engage more users. We’ll know it’s successful when people engage with the app on a regular basis.”

    This answer uncovers the hidden outcome: engage more people. Now we can set the right scope: increase the percentage of engaged users across any platform; increase the percentage of engaged mobile users; or increase the percentage of engaged Android users.

    Any of these outcomes gives us more room to explore than a fixed output. Maybe we don’t need a native app at all. We could deliver the same engagement through a mobile web experience, notifications, or email. And we’re not done when we ship—we’re done when the right people are actually engaged.

    Tricky example 1: measure the value creation moment (hires, not applicants)

    Infographic showing shift from output to outcome: build an Android app -> ask when it is successful -> increase engaged users. Highlights value, goals, and accountability in product management.
    Move beyond shipping features to the impact that matters. This visual maps the path from build an Android app to the real goal, increase engaged users, by asking why, defining value, and owning results.

    When setting outcomes, it’s tempting to choose the easiest-to-measure metric. But a good outcome measures the customer’s value creation moment.

    I worked at a company that helped new college grads find their first job. When I started working there, the primary outcome was “increase job applications.” This technically is an outcome—it measures a specific behavior in the product.

    But it doesn’t measure the value creation moment. A job seeker doesn’t get value when they apply for a job. They only get value when they get the job. Similarly, employers don’t get value from any job applicant, they get value when the right job applicant applies.

    Many job boards try to measure qualified applicants—instead of counting any applicant, they compare the credentials of the applicant to the job description and only count qualified applicants. This is better. But it still doesn’t measure the value creation moment. Both the job seeker and the employer get value when an open job is successfully filled. The right metric is hires.

    Yes, “hires” can be hard to instrument because it happens off-platform and incentives misalign. Measure it anyway, even with proxies. The easy metric isn’t always the right outcome.

    Tricky example 2: measure impact, not user-generated output (the course reviews trap)

    I worked with a team that helped students choose university courses. They set their outcome as: “Increase the number of course reviews on our platform.”

    Infographic titled '4 Outcome Traps to Avoid' for product teams, highlighting wrong moment, output in disguise, traction trap, and sentiment alone with concise guidance.
    Confusing activity with impact? This visual breaks down four common outcome traps—measuring at the wrong moment, mistaking outputs, chasing adoption, and relying on sentiment—so teams focus on real value.

    Sounds like an outcome, right? It’s a metric. You can measure it. It’s an action users take on the site—writing a review. But it’s actually an output in disguise.

    Reviews are valuable when they help a student evaluate a course. They don’t create any value if a student never sees them. More reviews aren’t always better, especially if they’re clustered where nobody looks.

    A better outcome is “Increase the number of course views that include reviews.” Now we’re measuring impact on the decision moment, not just the production of content.

    If you can hit your metric without helping customers, you’re tracking an output, not an outcome.

    Tricky example 3: measure success, not just adoption (the traction metric trap)

    “Increase the percentage of users who viewed the performance report.”

    This looks like a good outcome. It measures a specific behavior in the product. It’s within the team’s control. But it’s what I call a traction metric—it measures adoption of a single feature, not value to the customer.

    Infographic 'Why Teams Stay Stuck on Outputs' with a trust cycle—manager micromanages, team reports features, manager stays in details—and an accountability trap about safe targets and disguised outputs.
    Why teams get trapped in shipping features: a vicious trust cycle fuels micromanagement, while performance-linked outcomes push safe targets. Break the loop and refocus on customer outcomes that truly move the needle.

    Two problems arise. First, people can view the report and still not find what they need. Second, we might have perfectly happy customers who don’t need the report at all. Driving usage of an unneeded feature wastes time and erodes trust.

    Measure the value creation moment, not just feature adoption.

    Tricky example 4: pair sentiment with behavior

    I define a product outcome as a metric that measures either 1. a specific behavior in the product or 2. a sentiment about the product. But sentiment metrics—like CSAT or NPS—can be tricky on their own.

    Sentiment metrics are outcomes, but they aren’t directional. They don’t tell us where to explore or set guardrails for what to avoid. So I pair a behavior with a sentiment, for example: “Increase engagement without negatively impacting satisfaction.” I use sentiment as a counterweight.

    Facebook and Instagram illustrate why this matters. Meta is exceptional at driving engagement—but to a fault. Many of us don’t like these addictive products. Pairing engagement with a satisfaction guardrail prevents “engagement at all costs.”

    Why getting this right is hard (and how I counter it)

    Infographic, 'How to Make the Shift,' shows five steps to move teams from outputs to outcomes: translate metrics, negotiate with teams, expect iteration, watch for traps, and go deeper.
    Ready to move from shipping features to creating impact? This visual playbook shares five practical moves—translate metrics, partner with teams, iterate, avoid traps, and dig deeper—to turn outputs into measurable outcomes.

    The trust cycle. Managers don’t trust that teams can reach outcomes on their own. So managers micromanage the outputs. Teams, in turn, don’t communicate their progress toward outcomes—they communicate their progress on features. This reinforces the manager’s belief that they need to stay involved in the details. It’s a vicious cycle.

    I break it by asking teams to show their work—share assumptions, research, opportunity solution trees, and evidence behind choices—and by giving feedback on the thinking, not just the solutions.

    The accountability trap. When performance reviews are tied to hitting outcomes, teams play it safe. They sandbag their targets. They disguise outputs as outcomes to guarantee “success.”

    I treat outcomes as learning opportunities first. When we start on a new outcome, I set a learning goal—“learn what moves the needle on this metric”—before a performance goal—“increase X by Y%.” This creates space to explore without fear.

    How I get teams started with better outcomes

    Translate business outcomes to product outcomes. Business outcomes like revenue, retention, and market share are lagging indicators—by the time you see them, it’s too late to act. Product outcomes measure behavior changes within the product that lead to those business results. They’re leading indicators within the team’s control.

    Negotiate outcomes with your team. Outcome-setting should be a two-way conversation. Leadership brings the cross-company context. The team brings customer insight and technical realities. Neither side dictates; we co-own the target and the constraints.

    Infographic on outcomes vs outputs in product management: side-by-side panels show Feature Factory (measure what you ship) versus Product Team (measure what it changes), highlighting the shift to impact.
    Stop celebrating shipped features and start celebrating change. This visual contrasts a feature factory mindset with a true product team, urging teams to track impact, not output, and define success by outcomes.

    Expect to iterate on your metrics. Your first outcome metric probably won’t be right. That’s normal. Sonja at tails.com went through four iterations—from 90-day retention to 30-day to 5-day to behavior-based metrics—before landing on something actionable. Thomas at Bluestone Analytics iterated three or four times before finding the right metric. Iteration is the work.

    Watch for common mistakes. Outputs disguised as outcomes. Traction metrics masquerading as product outcomes. Sentiment metrics without direction. Business outcomes assigned directly to product teams without translating to behavior change.

    Use the right artifacts. Replace feature roadmaps with an opportunity solution tree to explore multiple paths, test assumptions, and sequence bets explicitly against a clear outcome.

    Align OKRs with outcomes. If your company uses OKRs, make sure the “KR”s are true product outcomes (behavior change and value creation), not a list of features to ship.

    The bottom line

    When we shift from an output-first mindset to an outcome-first mindset, it doesn’t mean that outputs stop mattering. Product teams will always ship features, and the ability to do so quickly and with quality still matters. This shift simply ensures those features achieve the intended impact. We aren’t done when we ship—we’re done when what we shipped has the intended impact.

    Measure success by the impact of what you ship and you’ll build a product team that learns, adapts, and creates real value. Measure success by what you ship and you’ll get a feature factory.

    Quick self-check: is your “outcome” really an outcome?

    Ask yourself: 1) Does it measure a behavior change or a sentiment tied to value creation? 2) Could we hit it without helping customers? 3) Is it adoption of a single feature (a traction metric) or a result that customers and the business care about? 4) Do we have a counter-metric to prevent unintended harm? If you stumble on any of these, refine it before you commit.

    Inspired by this post on Product Talk.

    Book a consult png image
  • Staying Sane as a Product Leader: Practical Strategies I’m Using from Teresa Torres & Petra Wille

    Staying Sane as a Product Leader: Practical Strategies I’m Using from Teresa Torres & Petra Wille

    The world can feel like it’s spinning, and as a product leader, I feel that pressure acutely—juggling customer needs, stakeholder expectations, and the relentless news cycle. I recently listened to a powerful conversation with Teresa Torres and Petra Wille about staying grounded when everything feels “bonkers,” and it offered a practical, human way to keep showing up without losing yourself.

    What resonated most was the invitation to live my values through small, consistent actions. Rather than waiting for grand gestures or perfect solutions, I’m leaning into the mindset of “Something is better than nothing.” It’s the same spirit we bring to continuous improvement in product: make a change, evaluate impact, iterate.

    “Create the world you want to live in” has become a daily prompt for me. I’m applying it to how I spend my attention, time, and platform—three scarce resources for any product management leader. I’m not going to do everything perfectly, but I can make better trade-offs this week than I did last week, and I can keep improving.

    Practically, that looks like reconsidering which speaking invites I accept, especially when representation is skewed. If a stage is heavily male, I now ask organizers about their plan for balance before committing. I also question travel expectations for short talks when a high-quality virtual experience is possible—good for sustainability, budgets, and energy. These choices compound, just like product roadmapping and sprint planning decisions.

    Petra’s “under-complexity” lens was a wake-up call. In product, oversimplified narratives—whether a single KPI, a vanity metric, or a forced binary—usually increase fear and bad decisions. The same is true in civic discourse. To counter that, I’m seeking more nuance on purpose: reading multiple sources on the same story, listening for who’s not in the room, and noticing how the same facts can carry different meanings depending on who’s telling it.

    One simple habit helps: I’ll read The New York Times and The Wall Street Journal on a headline, then follow up with Tangle by Isaac Saul, which lays out “what the left says / what the right says / editor’s take,” sometimes including perspectives from affected communities. It’s a lightweight form of personal knowledge management that improves my product judgment and my citizenship.

    Another idea that stuck with me is swapping media proxies for human connection. In product, we don’t ship based on secondhand opinions—we run customer interviews, co-create with users, and build empowered product teams. The same principle applies in community: talk to someone directly affected, ask real questions, and stay curious. When conversations get heated, I try to build bridges, reduce proxies, and look people in the eye.

    I’m also reflecting on platform responsibility. Even a “small” platform can snowball through weak ties inside a company or community. I’m asking: When should I speak up? Where should I draw lines? And when is “staying in your lane” actually a way to avoid necessary leadership? These are the same stakeholder management questions we navigate in product strategy—assess impact, clarify intent, and act with integrity.

    Local grounding matters, too. I’ve found energy and clarity in community-level action: voting, attending public protests when it feels right, mentoring, and supporting nonprofits like World Pulse. I love the framing of “don’t mess with my neighbors”—it keeps me focused on tangible care when the internet starts to feel like reality. I’ve also seen leaders use angel investing in agriculture-related efforts as a counterbalance to “internet reality,” channeling resources into durable, real-world outcomes.

    If you want to experiment this week, pick one small lever you control: where you spend money, time, attention, or your platform. Add nuance by reading at least two different perspectives before reacting. Replace proxies with people by talking to someone with lived experience. Reduce polarization by asking, “what shaped that view?” before judging it. And go local—connect with neighbors or a community group and let small actions compound.

    If you’d like to hear the full conversation that inspired these reflections, you can listen on Spotify or Apple Podcasts. Here are the direct links: Spotify: https://open.spotify.com/episode/1sxEFquu73ZB9fL9gGk6Om and Apple Podcasts: https://podcasts.apple.com/kh/podcast/staying-sane/id1794203808?i=1000755696295

    Resources I’m exploring and recommend: World Pulse (https://www.worldpulse.org/), The New York Times (https://www.nytimes.com/), The Wall Street Journal (https://www.wsj.com/), and Tangle by Isaac Saul (https://www.readtangle.com/ and https://www.readtangle.com/author/isaac-saul/). For builders and writers, I also appreciate Ghost (https://ghost.org/) as an open-source publishing platform. If you work in or with the MENA ecosystem, take a look at MENA Product Summit ’26 (https://www.prdkt.plus/summit26). Colleagues like Jeff Merrell (https://jeffdmerrell.com/) and grassroots efforts such as No Kings Protest (https://www.nokings.org/) offer additional perspectives and ways to get involved.

    If this resonates, share it with a teammate who’s been feeling the weight of the world. I’d love to hear one small, values-aligned action you’re taking this month—what “something” will you try next?

    Inspired by this post on Product Talk.

    Book a consult png image
  • How We Automated 81% of Customer Support with AI—While Uplifting CX, Speed, and ROI

    How We Automated 81% of Customer Support with AI—While Uplifting CX, Speed, and ROI

    Leading the Support function for a company that builds a leading Agent and AI-forward customer service platform has been, for me, unique, exciting, and yes—daunting. It’s where product ambition meets operational reality, and where every decision I make is immediately tested by customers who expect excellence.

    It’s unique because we use the same technology as our customers. We live in the product every day, which puts us in a privileged position to be the voice of the customer across the organization. That tight feedback loop has shaped how I prioritize, what I build next, and how I measure success.

    It’s exciting because we get to try all of the new features and capabilities of Fin and the Intercom helpdesk. With a relentless focus on AI innovation, I’ve had access to remarkable tools that help us deliver an incredible customer experience—and I’ve seen firsthand how the right workflows and guardrails turn those tools into outcomes.

    And it’s daunting because expectations for our own Customer Support (CS) team are sky high. If we can’t deliver incredible support using our own technology, we undermine its value proposition. That imperative has kept me honest, focused, and fast.

    In our new research, “The 2026 Customer Service Transformation Report,” we’ve been sharing how forward-looking teams use AI to transform their support models. If you’d like to get straight to the report, download it here.

    When Intercom changed its focus in late 2022 to prioritize the customer service use case, we undertook a critical review of the support experience we were delivering and committed to driving meaningful change under an AI-first framework. That was a turning point: I aligned product strategy and operations around a single north star—automate with quality, and elevate humans to higher-value work.

    Three years on, Fin now resolves over 81% of all our customer support volume, delivering immediate and high-quality resolutions. We have absorbed a 300%+ increase in customer demand since 2022 without proportional headcount growth. Without Fin, we would have needed at least 100 additional CS team members to meet that demand and our improved service levels – a net saving to Intercom of between $7.5M–$9M annually.

    Throughout this work, we drew on research from the 2026 Customer Service Transformation Report and applied the lessons directly to our own org design, knowledge management, and AI workflows. What follows is our story of transformation and how we achieved a mature deployment of Fin.

    The problems we set out to solve

    Back in 2022, our challenges looked familiar to any modern support organization, and I knew we needed a step-change—not incremental tweaks.

    We faced increased support demand from new and existing customers: Intercom was launching major features and changes at speed, driving up overall customer conversation volume and requiring additional headcount for the CS team. I could see we were scaling people faster than processes—unsustainable without automation.

    Our support policy (as defined by our service level objectives) was not based on a high bar: In most cases, we were only committed to “business hours” coverage for the majority of our customers, impacting first response times. Even with SLOs that were not considered best in class, we were struggling to meet our commitments. I wanted 24/7 coverage and faster first responses without sacrificing quality.

    We wanted to do more: As we pivoted our strategy, we wanted to open new routes to our support team, such as providing support to website visitors with technical questions and to trial customers. That meant meeting customers earlier in their journey with accurate, on-brand responses—at scale.

    What we did

    We made a very conscious decision to become our own best reference customer. As Intercom embraced the opportunity that generative AI presented to transform customer service, we intentionally moved to an AI-first strategy for our Customer Support team. I set a simple operating principle: ship value quickly, measure relentlessly, and let evidence guide the next bet.

    We started with the highest-volume, informational queries and saw our resolution rates climb quickly. With that foundation in place, we pushed Fin further, training it on deeper documentation and internal procedures, and eventually giving it the ability to take actions on behalf of customers. As Fin took on more complex work, our results started to compound—and trust in the system grew across the organization.

    Early adoption and building trust. When “AI Assist” features came to the Intercom Inbox, the CS team got early exposure to AI and were empowered to provide feedback directly to our product teams. This built awareness and trust across the team about what we were trying to achieve with AI, and helped shape the product roadmap. We were also the first beta customer for Fin, rolling it out to a subset of customers to watch sentiment and outcomes closely. With no adverse reaction and an initial resolution rate of over 25%, we deployed Fin to most customer segments within weeks. I’ll never forget the first week we put Fin in front of real customers—the silence of issues that never reached humans was the loudest signal of success.

    Knowledge management as a product. We recognized quickly that time spent tuning our help center and knowledge assets for Fin would pay dividends. We transitioned our Help Center Manager into a “Knowledge Manager,” with a dedicated remit to optimize content for Fin. We embedded knowledge creation into our “New Product Introduction” (NPI) process, targeting that Fin would resolve at least 50% of customer issues at every new product and feature launch. Over time, we added new sources, including “Developer Documents,” enabling Fin to handle increasingly complex issues. We built a culture of continuous improvement—allocating “out of the inbox” time so every teammate could close content gaps and raise the bar.

    Conversation design end-to-end. To ensure a consistent, high-quality customer experience, we created a new “Conversation Designer” role that owns the journey across automation and human handoffs. Using Intercom’s Workflows, we introduced “skills-based routing” so that when a customer asks for a human, the conversation reaches someone with the right expertise quickly. This is now handled by Fin directly using a feature called “Attributes.” The result: a seamless, on-brand experience regardless of channel or escalation path.

    Neon green hero graphic reading 'The 2026 Customer Service Transformation Report', with subhead 'The AI deployment gap is widening' and a black 'Get the report' button over a bar-chart pattern.
    Leaders are racing ahead with real AI in support. Explore the 2026 Customer Service Transformation Report to see where deployment is stalling, benchmark your team, and get practical steps to scale automation that delights.

    Organization changes that unlocked leverage. As we scaled Fin, we stood up a dedicated AI Support team under a senior CS leader to continuously optimize automation and define our AI adoption strategy across the journey. We restructured human roles into “Technical Support Specialist” and “Technical Support Engineer” to better align with the complexity of incoming work. We also expanded Support Operations to focus on optimization—using AI to uplevel Enablement, Workforce Management, QA, Process Management, and Data Insights. Just as important, we reset expectations about the balance between time spent supporting customers directly versus improving AI. That mindset shift created compounding returns.

    Pushing Fin further with new capabilities. As capabilities matured, we were early adopters and saw measurable wins:

    Fin Guidance: Multiple Guidance rules provide additional controls and a more personalized, targeted experience for customers.

    Fin Tasks and Procedures: Enables Fin to carry out activities such as updating customers on incident status and deep troubleshooting for technical issues.

    Insights: AI-driven dashboards provide deep insight into Fin’s performance and surface recommendations for further optimization. Insights also provides a Customer Experience (CX) Score for every customer interaction, enabling more targeted improvement efforts and opening up new ways to close the loop with customers who have had a poor experience.

    What we achieved

    What started as a focused effort to improve our customer support experience became the strongest proof point for what’s possible when you fully embrace AI. Fin now resolves over 81% of all our customer support volume and has allowed us to absorb a 300%+ increase in demand without proportional headcount growth. Over 90% of our customers now benefit from improved first response performance, 24/7 coverage, and outbound phone support.

    What the numbers don’t fully capture is the shift in how our team operates. With volume absorbed by Fin, our CS teammates now deliver consultative support—guiding next best actions, deepening product adoption, and contributing directly to retention and expansion. Customers that receive these engagements adopt Fin at a much deeper level and achieve greater support success. What was once a reactive, volume-driven team is now a function that generates significant revenue.

    What’s next

    Customer expectations are always rising, so we’re building on our progress by embracing the Fin Flywheel—an actionable framework for ongoing improvement and optimization. This keeps us honest about the discipline required to sustain AI performance at scale.

    Train: Teach Fin to resolve even the most complex queries with Procedures, knowledge, and policies.

    Test: Run fully simulated customer conversations from start to finish to see exactly how Fin will behave before going live.

    Deploy: Set Fin live across every channel – voice, email, chat, and social – for consistent support wherever customers reach out.

    Analyze: Use AI-powered Insights to analyze and improve Fin’s performance and deliver better customer experiences.

    We are also investing in our support teammates so they can adjust to the new world of AI—taking on more complex work and being valued for the subject matter expertise, consultative engagement, and empathy they bring to the role. That human layer is where differentiation shines.

    We will continue to develop and share best practices for deploying an Agent, based on our own experience with Fin and the lessons learned from our most forward-looking customers. These are captured and continually evolving in The Agent Blueprint.

    Transformation takes commitment

    The most successful teams aren’t bolting AI onto old processes; they’re rebuilding support around it—investing in knowledge and people alongside technology, and treating AI as a continuous discipline rather than a one-time deployment. That’s the real change required. For support teams willing to make it, there’s a rare opportunity to redefine what customer service can deliver—higher CSAT, faster resolution, and durable ROI.

    Inspired by this post on The Intercom Blog.

    Book a consult png image