Category: Product Management

Build CX Scores You Can Defend: My 5-step playbook for transparent, trustworthy AI metrics

“You don’t have to trust the algorithm; you can see exactly why a conversation earned the score it did.”

We recently shared how we redesigned CX Score to deliver deeper, more actionable insights across every conversation. The most common follow-up from support leaders was simpler and incredibly important: “Can I trust it?” It’s the right question—and it’s the one I use as my own bar for whether a metric is ready for the C‑suite.

CS teams are the subject matter experts on customer experience. They understand the nuance of what customers feel, the context behind every interaction, and the difference between a technically resolved issue and a genuinely satisfied customer. I’ve learned, conversation by conversation, that any metric we ship has to capture that nuance at scale—or it doesn’t deserve to be used.

We built CX Score to give support teams a complete view of how their customers feel across every conversation. It surfaces what’s working, what’s not, and why—so leaders can communicate impact clearly and drive change across support, product, and the wider business.

A CX Score in action: repeated CSV export failures trigger a low score and customer frustration, while the AI agent clarifies next steps and gathers details—turning raw signals into actionable support insights.

Here’s exactly how I approached building a trustworthy metric that support leaders can inspect, explain, and defend.

1) It’s grounded in how support teams define quality. I started with how experienced support professionals actually evaluate conversations—collecting real examples of strong, mixed, and poor interactions across industries, identifying the specific factors that shape overall experience, and writing plain-English rules for each. The result: CX Score applies the same criteria a trained support professional would use, not generic LLM assumptions.

2) It’s aligned with human judgment. We created a dataset of thousands of real customer conversations spanning multiple industries, languages, channels, and agent types. Each was manually reviewed by experienced support professionals—with two reviewers per conversation where possible and disagreement resolution to create stable consensus labels. The result: CX Score is trained and tested to behave like an expert reviewer, not a language model making broad guesses.

A modern CX analytics view shows how conversations flow from chat, email, and mobile into AI assistance, then to resolutions and sentiment outcomes—turning messy support data into a single, defensible CX Score.

3) It’s engineered by AI specialists. CX Score isn’t a prompt attached to an LLM. It’s a production system built by Intercom’s AI Group: 37 ML scientists and 350 engineers whose full-time focus is AI for customer service. The system includes specialized handling for long transcripts, model configuration tailored for support language and subtle sentiment, prompt engineering designed to default to neutral when evidence is weak, and a multi-stage evaluation pipeline that checks for precision, consistency, and reliability. The result: A metric built by a team that understands LLM behavior in production support environments, where accuracy and consistency matter most.

4) It’s validated statistically, not qualitatively. Trust requires measurement, not vibes. We tested CX Score across standard ML metrics: Precision (when the model flags a negative experience, how often do humans agree?), recall (how many human-identified issues does it catch?), and F1 score (the balance between both). We set an explicit bar: F1 above 0.8, representing high agreement with human judgment. We reran these evaluations through every revision, checking for regressions or biases, and I focused especially on negative experiences, because a false negative hides a real problem. The result: CX Score meets a measurable standard before it ships—not a gut check, a statistical requirement.

5) It was battle-tested with real customers. Lab accuracy isn’t enough. Customer environments are messy: Varied ticket types, mixed languages, unpredictable edge cases. Before release, we ran a multi-phase field test—shadow-scoring conversations with both old and new models, validating sensible behavior across agent type and conversation length, then rolling out to a controlled customer group who confirmed the scores felt right, reasons were clear, and insights were actionable. The result: CX Score shipped because real teams told us it made sense in practice, not because it passed internal tests.

From conversation to clarity: this visual maps the drivers behind a CX Score. Explore how policy feedback, answer quality, and effort combine to produce defendable insights support leaders can act on.

The importance of explainability. One of the most critical choices I made was ensuring CX Score isn’t a black box. Every score comes with clear reasons, concrete excerpts, and a short explanation of what influenced the rating. This turns the metric into something you can inspect, audit, and explain to executives. You don’t have to trust the algorithm. You can see exactly why a conversation earned the score it did.

A metric that evolves with your business. Customer expectations shift. Products change. AI improves. A trustworthy metric can’t be static. CX Score evolves with the same commitments that shaped its redesign: Evaluate the real signals that shape customer experience, keep the logic simple and interpretable, and ensure leaders can make clear decisions from it. It’s built to be a durable source of truth across every conversation.

The takeaway. In a world where products look the same and AI can generate any interaction, customer experience is one of the few differentiators that actually matters. Support leaders have built that expertise conversation by conversation. What they’ve lacked is a measurement system that could validate it at scale—one that’s reliable enough to report to the C-suite, explainable enough to defend in strategy meetings, and rigorous enough to drive real decisions. That’s what CX Score is designed to be: A metric that reflects the reality support leaders see every day, backed by the technical rigor to make it credible everywhere else.

Want to see CX Score in your workspace? Ask your admin to enable it for your team, and start using explainable AI insights to improve customer experience and coach with confidence.

Inspired by this post on The Intercom Blog.

February 9, 2026
AI Agent Deployment Mastery: My Proven Checklist to Ship Safely, Faster, and at Scale

Shipping AI agents is not like shipping a typical feature. The system learns, reasons, and takes action in unpredictable environments, and when it’s customer-facing, the stakes are high. Over the past few years, I’ve refined a practical checklist that helps my teams move quickly without breaking trust. It balances speed with safety, and ambition with accountability—exactly what you need to scale agentic AI in production.

This checklist was forged in real launches—some smooth, some humbling. Early on, I watched an otherwise brilliant agent confidently offer a refund policy we didn’t have. That one incident made it clear: AI agents require a higher bar for guardrails, evals, and observability. Today, I won’t greenlight an AI rollout without these steps being explicit, owned, and testable.

Start with outcomes, not output. I define the job-to-be-done, the target users, and the measurable business impact using outcomes vs output OKRs and driver trees. Success is not “ship an agent,” it’s “reduce first-response time by 40% with no drop in CSAT,” or “increase qualified demo bookings by 20% at a lower cost per acquisition.” Clear outcomes give the agent a purpose and the team a north star.

Prepare the knowledge the agent will use. A retrieval-first pipeline beats raw prompting for most enterprise cases. I inventory sources of truth, set access controls, and enforce data governance from day one. That includes PII handling, redaction, retention policies, and privacy-by-design. If the agent can’t reliably retrieve the right fact at the right time, the rest doesn’t matter.

Choose models and prompts with discipline. I align model selection with context window management, cost, latency, and tool-use requirements. Then I build prompts and tools together, not in isolation, and I keep temperature, stop conditions, and function-calling explicit. Most importantly, I use eval-driven development: golden datasets, task-specific metrics (accuracy, helpfulness, latency, cost), and target thresholds that must be met before widening rollout.

Manage AI risk upfront. I treat jailbreaks, toxicity, and data leakage as product risks, not just security issues. I implement layered defenses—input/output filtering, policy checks, rate limits, and abuse monitoring—and define escalation paths and human-in-the-loop handoffs for ambiguous cases. Every risky capability needs an owner, a playbook, and a test.

Build the pipeline that lets you iterate safely. Prompts, tools, policies, and retrieval configs go through the same CI/CD rigor as code. I use feature flags for progressive delivery, canary cohorts to limit blast radius, and clear rollback procedures. Observability isn’t optional; I track latency, token usage, cost, failure modes, and user outcomes. I also watch DORA metrics and deployment frequency to ensure we’re improving the engine, not just the output.

Constrain autonomy intentionally. Agent behavior design matters as much as model choice. I set step limits, define tool whitelists, separate read vs write permissions, and specify decision checkpoints. When the agent is uncertain or confidence drops below a threshold, it hands off to a human or a deterministic workflow. Guardrails aren’t barriers; they’re bumpers that keep you on the track.

Instrument what users experience, not just what models produce. I track activation, task success, self-serve completion rates, and time-to-value. I pair Agent Analytics with journey analytics so I can see where the agent helps or hurts. I also invest in UX trust cues—transparent explanations, undo paths, and in-app guides—so users feel in control. When the agent changes behavior through learning, the interface should make that understandable.

If you’re shipping a voice AI agent, test in realistic conditions. I set targets for ASR accuracy, barge-in responsiveness, TTS prosody, and end-to-end latency. I predefine safe transfer logic for complex calls and ensure compliance for call recording and data retention. Voice amplifies both the magic and the mistakes; operational excellence is non-negotiable.

Plan the business rollout like a product, not a press release. I align pricing (often consumption SaaS pricing), packaging, and SLAs with actual unit economics—tokens, inference, and retrieval. I equip solutions engineering with playbooks and reference architectures, wire up CRM integration for attribution, and put feedback loops into Intercom or the support stack so we learn from every interaction.

Run operations like an SRE team. I define incident severity for AI-specific failures (e.g., harmful output, runaway cost, degraded retrieval), add alerting, and keep runbooks current. I schedule postmortems that feed directly into eval baselines and backlog priorities. Continuous discovery isn’t a ceremony; it’s the safety net that keeps improvements compounding.

Close the loop on compliance and governance. From day zero, I document data flows, vendor scopes, and audit logs. I verify regulatory compliance and adopt privacy-by-design so I’m not retrofitting later. Transparency, user consent, and opt-outs aren’t just legal checkboxes; they’re trust-building tools that differentiate your product.

The result of this checklist is speed with confidence. It gives my teams a common language to debate trade-offs, a clear path to production, and the guardrails to scale safely. If you’re preparing to deploy an agent, adapt these steps to your stack and your customers. Your future self—and your users—will thank you.

Inspired by this post on Product School.

February 9, 2026
Vibe Coding Unleashed: How Parallel Agents Build KPI Driver Trees in Under Two Hours

I’ve been exploring what I call the next level of vibe coding: orchestrating agentic AI to build complex product artifacts in minutes, not days. The breakthrough comes from ditching linear handoffs and embracing true parallelism—letting specialized agents tackle the work simultaneously while I steer the orchestration. In product management contexts where speed and clarity matter, this shift changes everything.

Building a KPI Driver Tree in two hours becomes possible when you stop building sequentially and start building with parallel agents.

For product leaders, a KPI Driver Tree is the fastest way to make strategy legible. It ties high-level outcomes to the levers we can actually pull—features, channels, pricing, onboarding, activation, and retention mechanics—so we can prioritize with confidence. Done well, it connects outcomes vs output OKRs, clarifies measurement, and aligns the team around a shared, testable model of growth.

Here’s how I operationalize it with agentic AI and AI workflows. I spin up a small team of specialized parallel agents: a Metrics Librarian (taxonomy and definitions), a Data Modeler (event and table design), a Research Synthesizer (voice of customer and causal hypotheses), a UX Prototyper (visualizing the tree and flows), and a QA/Evaluator (logic and consistency checks). An Orchestrator coordinates these agents, resolves conflicts, and composes outputs into a single, production-ready artifact—while I set constraints, review deltas, and decide.

In a typical two-hour sprint, all agents run at once. While the Metrics Librarian finalizes the KPI ontology, the Data Modeler validates instrumentable events and joins, and the UX Prototyper renders an interactive driver tree for a unified analytics platform. Meanwhile, the Synthesizer maps qualitative insights to quantitative levers, and the Evaluator stress-tests assumptions. Because we’re not waiting for sequential handoffs, we converge on a coherent driver tree and its initial measurement plan in one pass.

The payoff isn’t just speed—it’s higher-quality decisions. Parallel agents reduce context loss, expose trade-offs earlier, and allow me to compare multiple viable paths side-by-side. This accelerates continuous discovery, aligns with product strategy, and gives product managers and LLMs for product managers a clear, living map of how inputs roll up to outcomes. It’s the closest I’ve found to running a product trio at machine speed.

Guardrails matter. I pair this approach with strong data governance, privacy-by-design, and eval-driven development so every agent’s output is testable and auditable. Clear prompts, scoped corpora, and consistent acceptance criteria keep the Orchestrator honest, while lightweight Agent Analytics helps me see where reasoning falters and where to improve the system.

If your team is still tackling analytics artifacts sequentially—requirements, then instrumentation, then visualization—consider switching mental models. Treat the driver tree as the backbone, empower parallel agents to co-create around it, and reserve human judgment for the critical calls. This is vibe coding for product management: creative, fast, and grounded in measurable outcomes.

Inspired by this post on Pendo – Best Practices.

February 5, 2026
Amplitude’s AI Visibility Upgrade: Content Generation, Chat Segmentation, Sleeker UI—Why It Matters

I look for analytics upgrades that meaningfully compress time-to-insight for product teams. The newest expansion of Amplitude AI Visibility stands out because it improves how we explore user behavior, automate insight creation, and translate data into action across product-led growth motions.

Explore the most recent updates to Amplitude AI Visibility, including content generation, AI chat-driven segmentation, better UI, and improved reliability.

Here’s how I’m thinking about the impact. Content generation can turn raw events into ready-to-share narratives—experiment summaries for A/B testing, cohort deep-dives for retention analysis, and executive briefs that tie outcomes to roadmap decisions. For leaders and ICs alike, this trims the manual lift in Amplitude analytics while keeping the human in the loop to verify context and nuance.

AI chat-driven segmentation is another meaningful unlock. Instead of clicking through complex filters, I can describe the cohort I want in natural language and iterate quickly. That speeds up continuous segmentation work—spotting activation bottlenecks, isolating churn precursors, or defining cohorts for product-led growth experiments—and keeps the team focused on hypotheses and decisions, not interface friction. With LLMs for product managers, the key is pairing this speed with clear guardrails and validation steps.

The updated UI matters more than aesthetic polish. A clearer, more consistent experience reduces cognitive load, improves adoption across cross-functional partners, and reinforces a unified analytics platform approach. Improved reliability, paired with strong observability, increases trust in the stack—critical when insights drive roadmap priorities and high-visibility launches.

Operationally, I’d roll this out with a simple playbook: identify 2–3 high-value use cases (e.g., activation funnel analysis, churn cohort exploration, experiment reporting), define success metrics (time-to-insight, stakeholder adoption, decision velocity), and establish basic AI risk management and data governance guardrails (prompt templates, access policies, and review steps). The goal is to turn AI workflows into a durable capability rather than a one-off novelty.

Bottom line: these enhancements remove friction between questions and answers. If your team relies on Amplitude analytics, the combination of content generation, AI chat-driven segmentation, a cleaner UI, and stronger reliability should accelerate discovery cycles and help you translate insight into action with greater confidence.

Inspired by this post on Amplitude – Best Practices.

February 5, 2026
Two People, Zero Waste: How Earmark’s Agentic AI Turns Meetings into Finished Work

I care about meetings only insofar as they create momentum and outcomes. What if your meetings could actually produce the artifacts you need—specs, tickets, slides—before the call even ends?

I recently listened to an episode of Just Now Possible where Teresa Torres talks with Mark Barbir (CEO) and Sanden Gocka (Co-Founder), the co-founders of Earmark, about building a productivity suite that turns unstructured conversations into finished work in real time. As a product leader, this premise hits the sweet spot of agentic AI, real-time AI workflows, and ruthless focus on outcomes over output.

Listen to this episode on: Spotify | Apple Podcasts

Unlike generic AI notetakers that produce summaries nobody reads, Earmark runs multiple agents in parallel during your meetings—translating engineering jargon, drafting product specs, even spinning up prototypes in Cursor or V0 while you're still talking. That’s the bar I want from AI in the room: finished work, not notes.

What impressed me most was the clarity of their pivot. They moved from an Apple Vision Pro presentation coaching tool to a web-based meeting assistant. I’ve made similar calls: when the distribution path and daily workflow are obvious, you follow the user’s gravity. This shift unlocked a broader surface area—PMs, engineers, design partners—and made agentic workflows useful where work actually happens.

They also turned a technical constraint into a commercial advantage. Their ephemeral (no-storage) architecture became a feature for enterprise sales. I’ve seen this repeatedly in AI risk management: privacy-by-design and clear data governance reduce friction with security reviewers and accelerate procurement. For many enterprises, “we don’t store your data” is the win condition.

Cost discipline was another standout. They tackled the hard problem of making real-time AI affordable—from $70 per meeting down to under a dollar through prompt caching. That’s not just optimization; it’s product strategy. Choices like model selection, context window management, and retrieval-first pipeline design determine whether a feature can scale to every meeting or remains a demo.

On capability design, the team leaned into templates and simulated stakeholders to ship value fast. Template-based agents: Engineering Translator, Make Me Look Smart, Acronym Explainer. Personas that simulate absent team members (security architect, legal, accessibility). This is exactly how I frame early AI workflows: remove friction for the product trio, anticipate blockers, and let the agent do the tedious, error-prone first pass.

They were refreshingly pragmatic about models. Why GPT 4.1 still beats newer models for prose quality in their use case is a reminder that “best” is contextual. When the job-to-be-done is precise prose and production-grade artifacts, consistent quality trumps leaderboard buzz. Of course, they also invest in guardrails to ensure quality and manage hallucinations—another non-negotiable for enterprise adoption.

Search and analysis across time is where many AI products stumble. They explained the limits of vector search for analysis questions across meetings and how they’re building agentic search with multiple retrieval tools (RAG, BM25, metadata queries, bespoke summaries). I couldn’t agree more: analysis requires reasoning over structure, time, and purpose—not just semantic proximity. Layered retrieval with stateful agents beats a single embedding call.

They also articulated a crisp user thesis: design for product managers as the extreme user to solve for everyone. In my experience, if you satisfy the PM’s bar for clarity, traceability, and actionability, engineers, designers, and go-to-market teams benefit immediately. That’s how you earn daily active use, not once-a-week novelty.

For builders curious about the stack and comparables, they discuss services and tools like Assembly AI for speech-to-text, OpenAI API with prompt caching support, and build integrations with Cursor and V0 by Vercel. They also reference Granola as a comparison point and nod to ProductPlan, where both founders previously worked. If you want to try the product, here’s Earmark—a productivity suite where the work completes itself.

If you're a PM drowning in follow-up work or a builder curious about real-time AI architectures, this conversation offers a detailed look at what it takes to ship an AI product that people can't imagine working without. Personally, I see this as a credible path toward an AI chief of staff—their vision goes beyond automating deliverables to orchestrating judgment, compliance signals, and cross-functional readiness.

The episode covers the founder backstory, what Earmark does, comparisons to competitors, unique features, templates and personas, technical decisions, early versions and challenges, optimizing transcript summarization, managing multiple tools and costs, challenges with context and reasoning models, innovative search and retrieval techniques, creating actionable artifacts from meetings, ensuring quality and managing hallucinations, and the future vision for an AI chief of staff. It’s a full-spectrum look at building with agentic AI, not just talking about it.

Podcast transcripts are only available to paid subscribers.

Inspired by this post on Product Talk.

February 5, 2026
Mastering 30,000-Foot Vision and Ground-Level Execution: Systems That Decide Without You

Executive function, for me, is the art and discipline of building systems that make high-quality decisions without my constant involvement. The real unlock isn’t personal heroics; it’s institutionalizing judgment. When I do my job well, teams move faster, ambiguity shrinks, and the organization compounds learning even when I’m not in the room.

Operating simultaneously at 30,000 feet and ground level is the defining muscle of executive leadership. I deliberately switch altitudes. At 30,000 feet, I obsess over strategy, architecture, and resourcing. On the ground, I validate core assumptions with firsthand data, listen for weak signals, and spot process cracks before they widen. Altitude changes are not random; they’re triggered by variance from plan, critical customer moments, or leading indicators that deviate from expected ranges.

The leap from frontline manager to manager of managers is where many rising leaders stall. As a manager of managers, my primary value shifts from personal execution to system design. I move from answering questions to installing mechanisms that ensure questions get answered well by others. This includes clear decision rights, shared metrics, and repeatable, lightweight rituals that scale across teams.

What is an executive actually accountable for? Outcomes over output, talent density, and the clarity of the operating system. That means defining strategy, aligning resources, creating a cadence of review that exposes truth, and ensuring incentives reward the behaviors we want. My barometer: if I step away, do priorities hold, do metrics behave as expected, and do tradeoffs land where I would have landed?

Knowing when to dive deep versus when to step back is a craft. I dive deep when risks are existential, when metrics have no credible owner, or when narrative and numbers diverge. I step back when leaders demonstrate consistent judgment, metrics sit inside control limits, and learnings are documented. The principle I return to again and again: context is everything. Senior leaders operate on context, not control.

To scale judgment, I teach people how I think. I externalize my mental models: how I construct decision trees, how I stress-test assumptions, and how I weigh time horizons. I rely heavily on driver trees for metrics because they force causal clarity. If we can’t map how a top-line goal decomposes into controllable levers, we’re managing by hope, not design.

Creating a shared language across the business is a force multiplier. I standardize definitions for our core metrics, codify what “good” looks like, and make it easy to repeat the system. We align around outcomes versus output, and we use cadences like MBRs and QBRs to unify narrative and numbers. Shared language makes decisions legible across functions and reduces rework.

My COO playbook emphasizes owning the full customer experience end to end. When marketing rolls up under a COO in certain stages, the upside is coherence: one narrative from awareness to activation to expansion, one set of metrics, one growth engine. The point isn’t org charts; it’s removing seams customers can feel.

Demanding and supportive is not a contradiction. I set ambitious, unambiguous bars and back them with coaching, resourcing, and fast feedback. The combination builds trust: expectations are clear, and help is immediate. I expect leaders to bring problems paired with proposed solutions and to escalate early, not perfectly.

Inside my executive interview process, I’m assessing altitude agility, operating cadence, and taste in metrics. I use structured interviews and live case workshops to see how candidates frame ambiguous problems, build driver trees, and prioritize tradeoffs. The best prompts are simple and revealing: design the operating system for a 3x scale scenario; diagnose a broken funnel with incomplete data; align two teams with conflicting incentives. The workshop prompts that reveal everything surface thinking speed, humility, and the instinct to make context legible.

The common thread in failed executive hires is a mismatch between the company’s operating system and the leader’s default mode. Some leaders can’t stop doing the work themselves. Others stay too abstract and never build mechanisms. I look for demonstrated ability to change systems, not just run them—leaders who can both author and evolve the playbook.

On metrics, I practice the driver tree philosophy. I begin with the North Star, decompose it into controllable levers, instrument each node, and assign single-threaded owners. We design review cadences where deviations trigger targeted diagnostics, not thrash. Each tree has documented assumptions, data sources, and thresholds that prompt action. This is how teams learn to anticipate, not react.

High-functioning executive teams are visibly collaborative. We clarify decision rights, disagree and commit quickly, and conduct post-decisions to harvest learnings without blame. My favorite litmus test is simple: can 30 people operate as one team when it matters? When we get this right, information flows, execution accelerates, and customers feel consistency.

One of the most counterintuitive leadership lessons is working yourself out of a job. If the system cannot run without you, you have a key-man risk, not a leadership strength. I aim to build successors, codify judgment, and design mechanisms that make good decisions the default state. That’s how you create durable, compounding advantage.

And the review feedback you can’t unhear? Mine was brutally honest: my bar was high, but my mechanisms were implicit. Once I wrote them down—how I decide, what I expect, where I dive deep—the organization moved faster, and I actually became less central. If there’s a throughline to extraordinary leadership, it’s this: make your judgment teachable and your systems inevitable.

February 5, 2026
From Idea to Impact: My PM-Friendly Blueprint to Building Your First AI Agent Fast

AI agents are quickly moving from novelty to necessity, and the fastest way to capture value is to approach them like any other high-stakes product initiative. In this guide, I share how I plan, build, and launch production-grade agents with a product mindset—balancing ambition with risk, speed with governance, and innovation with measurable outcomes.

I start by getting crisp on the outcome. Who is the primary user, what job are they hiring the agent to do, and how will we know it’s working? I translate this into outcomes vs output OKRs, such as resolution rate, time-to-value, cost-to-serve, or qualified pipeline influenced—anchoring the roadmap before a single line of code or prompt is written.

Next, I map the agent’s scope and boundaries. I write a simple capability canvas: the tasks the agent must perform, the tools it can use, the data it can access, and the constraints it must respect. Most successful builds follow a retrieval-first pipeline: connect trusted knowledge sources, enrich with metadata, and manage a lean context window to keep responses relevant and cost-efficient. From the start, I bake in privacy-by-design, data governance, and AI risk management so compliance isn’t an afterthought.

Model selection comes after the workflow is clear. I choose an LLM for the job (latency, cost, multilingual needs, and tool-use fidelity) and pair it with the right connectors and actions—think CRM integration, ticketing, search, or internal APIs. For voice experiences, I define a voice AI agent persona, turn-taking rules, and barge-in behavior. This is where agentic AI patterns shine: structured planning, tool invocation, and verification loops create a resilient, goal-directed system.

Prompt design is product design. I write system prompts that define role, tone, constraints, data sources, and success criteria. I add few-shot examples that mirror my top use cases and edge cases, then apply prompt engineering best practices to control style, limit speculation, and encourage citations. For voice, I include prompt engineering for voice to optimize brevity, warmth, and disfluency handling without sacrificing accuracy.

Before launch, I build an eval-driven development workflow. I curate golden datasets from real user intents, add adversarial cases, and automate evals for accuracy, safety, grounding, and tool-use success. I set a minimum detectable effect (MDE) so A/B testing can validate improvements with confidence, and I define go/no-go thresholds to prevent regression. This becomes my continuous discovery loop for the agent.

Instrumentation is non-negotiable. I wire up Agent Analytics to track task success, containment/deflection rate, handoff quality, cost per task, and user satisfaction. I supplement with a unified analytics platform and session replays to observe failure patterns. These signals feed prioritization and help me decide when to expand scope versus harden reliability.

For delivery, I rely on CI/CD with feature flags to gate risky capabilities, plus canary releases for new tools and prompts. I monitor DORA metrics to maintain deployment frequency without trading off quality. When incidents happen, I treat them like production issues: incident management playbooks, rollbacks, and clear postmortems.

Trust is earned through safety and transparency. I enforce least-privilege access, structured logging, and red-teaming for jailbreaks, prompt injection, and data exfiltration. Threat detection and response plus clear user disclosures keep the experience responsible and compliant with regulatory requirements.

GTM is product-led. I use in-app guides, product tours, and onboarding checklists to drive user activation and early wins. I define success moments, turn them into habit loops, and run retention analysis to find where users stall. This tight loop of messaging, measurement, and iteration accelerates product-market fit.

Common high-ROI use cases I prioritize include customer support ai strategy (automated resolution and augmented agent assist), sales and success workflows (lead qualification, QBR prep), and internal knowledge copilots (policy, process, engineering runbooks). Each starts narrow, ships fast, and scales with proven evidence from analytics and experiments.

If you’re skimming, here’s the blueprint: clarify outcomes, design AI workflows with a retrieval-first pipeline, select the right LLM and tools, engineer robust prompts, institutionalize evals and A/B testing, instrument Agent Analytics, ship with CI/CD and feature flags, and iterate with discipline. In the walkthrough video above, I go deeper on templates, prompts, and experiments you can use to build your first agent with confidence.

Inspired by this post on Product School.

February 5, 2026
Becoming AI Native: A Practical Playbook to Transform Strategy, Teams, Data, and Tech

AI Native is more than a feature set—it’s an operating system for the entire business. In my role leading product, I’ve seen that companies win when they treat AI as a first-class citizen across strategy, architecture, workflows, and go-to-market. In this narrative, I unpack what “AI Native: What It Means and How to Get There” looks like in practice, sharing the frameworks I use to align vision, technology, and teams around measurable customer outcomes.

When I say AI Native, I mean a company where core value creation, customer experience, and internal operations are powered by AI end-to-end. It’s not just bolting on a chatbot. It’s rethinking product strategy, data foundations, and execution so we can deliver differentiated experiences faster, at lower cost, and with higher reliability. This shift demands clarity on where AI truly creates leverage—and the courage to say no where it doesn’t.

The starting point is strategy. I ground teams in outcomes vs output OKRs and a crisp value proposition: Which customer jobs-to-be-done benefit most from generative AI? Where can we unlock 10x improvements in speed, accuracy, or personalization? We prioritize a small number of high-signal use cases, size impact, and design Minimum Viable Experiments (MVEs) to de-risk assumptions before scaling. This is where build vs buy decisions matter—use foundation models and platforms for commodity needs, and invest your scarce engineering time where differentiation lives.

Next comes architecture and data. AI Native products thrive on a retrieval-first pipeline, strong context window management, and model-agnostic abstraction so we can swap providers as needs evolve. I emphasize privacy-by-design, robust data governance, and observability across prompts, embeddings, latency, and cost. These guardrails let us move quickly without compromising trust, especially in regulated or enterprise settings.

Execution shifts as well. I organize empowered product teams and product trios around the highest-value workflows, not components. Continuous discovery pairs with CI/CD, feature flags, and telemetry so we can test safely in production. Eval-driven development is non-negotiable: we design offline and online evaluations that mirror real user success criteria—accuracy, helpfulness, safety, and business outcomes—then wire those evals into the build pipeline to prevent regressions.

On the intelligence layer, we increasingly rely on AI workflows and agentic AI to orchestrate multi-step tasks—retrieval, reasoning, tool use, and verification—with human-in-the-loop where appropriate. Clear system prompts, tool definitions, and fallbacks keep behavior predictable. This is where product craft meets prompt engineering and LLMs for product managers: the best teams codify patterns, share prompts in a living library, and standardize on a lightweight AI product toolbox.

Risk and reliability are part of the product, not an afterthought. I run AI risk management as a continuous program spanning red teaming, content filters, PII handling, audit trails, and incident response. We tie policies to concrete controls and create simple dashboards leaders can trust. The goal is to ship boldly with safety, maintainability, and scale in mind.

Becoming AI Native also changes how we grow. We lean into product-led growth with clear in-app guides, product tours, and activation paths that teach users where AI shines. CRM integration ensures sales and success teams have context to coach customers. Pricing experiments—often usage- or value-based—align revenue with the impact customers feel, while retention analysis helps us double down on the use cases that drive compounding value.

To make this real, I use a 90-day plan. Days 0–30: align on strategy, top use cases, and risk posture; stand up data pipelines and a basic retrieval-first stack; define evaluation metrics. Days 31–60: ship MVEs behind feature flags, run head-to-head evals, and instrument observability; start a cross-functional community of practice. Days 61–90: scale the winning use cases, formalize governance, and publish a roadmap tied to outcomes—not just features—with clear SLAs and success metrics.

The destination is a durable advantage: faster iteration cycles, smarter experiences, and a product strategy that compounds with every interaction. If you’re ready to make the leap, start small, measure obsessively, and build the muscle to ship, learn, and adapt. That’s the heart of becoming AI Native—and it’s well within reach.

Inspired by this post on Product School.

February 5, 2026
How We Built Rock-Solid AI Infrastructure: Lessons From Scaling AI Visibility and Reliability

Scaling AI Visibility pushed me to rethink what “reliable” really means for AI infrastructure. As my team expanded usage across more datasets, models, and workflows, we uncovered unexpected sources of report failure and built the guardrails, observability, and processes that now anchor our stability strategy.

In practice, the surprising failure modes were rarely the loud ones. We saw report failure triggered by small schema drift from non-deterministic LLM outputs, silent permission changes in upstream data sources, token-limit truncation that broke downstream parsing, third-party API rate limits that surfaced only under bursty load, and clock skew that confused idempotent writes. Individually these issues looked minor; together they created reliability debt.

Our first move was deep observability. We instrumented the end-to-end pipeline with structured logs, distributed tracing, and high-signal metrics mapped to SLOs and error budgets. That visibility let us separate symptom from cause, quantify impact by segment, and prioritize fixes that moved business outcomes, not just vanity thresholds. It also gave product managers and SREs a shared, real-time view to make tradeoffs explicit.

Next, we hardened the runtime with resilience patterns: circuit breakers on flaky dependencies, timeouts tuned to p95 behavior, retries with jittered backoff, idempotent processing for at-least-once delivery, and backpressure-aware queues. We enforced schema contracts at ingestion with JSON validation and added feature flags to decouple deploys from releases, so we could roll forward or back within minutes when signals degraded.

On the product side, we adopted eval-driven development for model and prompt changes, shifting risky modifications behind canaries and staged rollouts. CI/CD gates required evaluation baselines to hold or improve before promotion. We tracked DORA metrics to keep deployment frequency high without sacrificing change failure rate, and we used P95 latency and budget burn as the forcing functions for prioritization.

Culture mattered as much as code. We formalized incident management with clear ownership, lightweight runbooks, and blameless reviews that produced crisp, automatable actions. We partnered early with SRE on SLO design, integrated privacy-by-design and PII scanning into the pipeline, and treated AI risk management as an ongoing product constraint rather than a checkbox.

The net effect: fewer flaky reports, faster recovery when things do break, and far more confidence to ship improvements to AI Visibility at pace. If you’re scaling similar capabilities, start with observability, make resilience patterns non-negotiable, and let SLOs guide your product roadmap. Reliability is not a phase—it’s the product.

Inspired by this post on Amplitude – Best Practices.

February 4, 2026
Inside Amplitude’s AI Playbook: Lessons from Leo Jiang on Ask Amplitude, Agents, and Visibility

I continually study how high-velocity teams turn AI ambition into shipped product, and Amplitude’s approach stands out. "Leo Jiang is the Head of Engineering, AI Products at Amplitude, focused on building new AI and marketing products. He has helped build Ask Amplitude, Agents, and AI Visibility." From a product management leadership lens, that portfolio signals a clear AI strategy: enable insight (Ask Amplitude), drive action (Agents), and ensure trust and observability (AI Visibility).

What I appreciate most is the sequencing: start with user-facing value, build agentic AI capabilities where tasks repeat and outcomes can be evaluated, and layer AI workflows with robust governance. For PMs and LLMs for product managers, the implication is to define success via eval-driven development—quantitative rubrics, offline test sets, and real-time feedback loops—before scaling automation. This also hints at an emerging discipline of Agent Analytics: instrument prompts, tool calls, and outcome quality so we can tune performance like we tune a funnel.

Ask Amplitude gives a relatable example: natural-language questions lower the activation barrier for product and growth teams inside an Amplitude analytics environment. When agents turn answers into next-best actions, product-led growth becomes measurable—from hypothesis to change to impact—inside a unified decision loop. That tight loop is where product strategy, design, and reliability meet to create compounding value.

Operationally, I organize a product trio around each capability and pair it with forward deployed engineers to accelerate discovery with customers. I also invest in privacy-by-design and data governance early, ensuring marketing use cases respect compliance while keeping iteration speed high. The goal is a repeatable path from prototype to scale that preserves momentum without compromising safety.

My takeaway for peers: pick one high-frequency workflow, define clear agent boundaries, ship a narrow slice, and measure relentlessly. Use retrieval-first pipeline patterns for grounding, add human-in-the-loop checkpoints, and close the loop with qualitative insights from in-app guides. When that works, expand capabilities—not just features—and let outcomes vs output OKRs steer prioritization.

Inspired by this post on Amplitude – Best Practices.

February 4, 2026
12 MCP prompts that rally your whole company around product data and drive adoption

I’ve seen first-hand how quickly a company aligns when product data becomes everyone’s common language. To make that happen at scale, I rely on MCP prompts inside Pendo to turn raw behavioral signals into clear, cross-functional actions. When we give people precise questions to ask of the data, engineering, product, marketing, customer success, and sales move in lockstep—and outcomes follow.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

What follows are the 12 MCP prompts I use to help teams across the business make better, faster decisions from product analytics, in-app guides, and customer feedback. They’re battle-tested, easy to adapt to your stack, and intentionally written to drive product-led growth and clearer accountability.

Prompt 1: Show me the activation funnel by segment (SMB, MM, ENT) for the last 90 days, highlight the biggest drop-off steps, and quantify which change would yield the largest absolute lift in activated users.

Prompt 2: Rank features by adoption velocity over the past 30 days, identify underutilized high-value features by persona, and recommend the top three in-app guide placements to increase engagement.

Prompt 3: Plot 30/60/90-day retention curves for new users by plan type and persona, flag statistically significant gaps, and suggest two experiments to improve week-two retention.

Prompt 4: Cluster qualitative feedback (NPS verbatims, support tickets, and in-app survey responses) by theme and feature, summarize the top friction points in one paragraph per theme, and propose fixes ordered by impact and effort.

Prompt 5: Analyze common user paths after onboarding, surface where users stall or loop, and recommend targeted product tours or tooltips to reduce time-to-first-value.

Prompt 6: Evaluate the impact of a specific in-app guide on activation rate using an A/B test, report lift with confidence intervals, and include the minimum detectable effect (MDE) assumptions used in the analysis.

Prompt 7: Identify accounts at churn risk based on declining feature usage, login frequency, and support sentiment; produce a prioritized list with the top three customer success plays for each account.

Prompt 8: Generate a weekly list of product-qualified leads (PQLs) based on usage thresholds, map them to opportunities in our CRM, and recommend the best follow-up message for sales based on feature interest.

Prompt 9: Analyze usage distribution across pricing tiers, highlight features driving upgrades, and suggest one packaging change and one in-app nudge to improve conversion to the next plan.

Prompt 10: Measure time-to-value by persona for a key action, compare pre/post tutorial launch, and quantify the impact of our in-app guides on reducing time-to-first-value.

Prompt 11: For our last three releases, summarize adoption, top feedback themes, and any regressions; recommend one quick win and one strategic bet for the next sprint.

Prompt 12: Produce a weekly executive summary with the top three product insights, the KPIs they influence, and clear owner-action pairs across Product, CS, and Marketing.

When teams start their day with these MCP prompts, product data stops being a report and becomes a decision engine. That’s how we drive adoption, run better experiments, reduce churn, and keep everyone focused on outcomes instead of opinions. If you adapt even a few of these prompts to your context, you’ll feel the shift—more clarity, tighter cycles, and a company moving as one.

Inspired by this post on Pendo – Best Practices.

February 4, 2026
Stop Groupthink in Hiring: Proven Product-Led Tactics to Make Faster, Fairer Decisions

Is hiring broken—or just badly designed? I’ve been sitting with that question after a recent conversation that crystallized what I see across product organizations: AI-fueled application overload, sprawling interview loops, and fuzzy criteria that invite groupthink at exactly the wrong moments. If you’ve ever watched a promising candidate stall out late in the process, you’re not alone. Listen to this episode on: Spotify | Apple Podcasts.

Here’s the reality I’m observing in the market: Layoffs and hiring freezes have flooded the funnel, while AI tools make it trivial to submit hundreds of applications. Companies are overwhelmed, so they respond by adding more interviews and more stakeholders, hoping more touchpoints equal better signal. In practice, that complexity often dilutes accountability and increases noise—especially for product management leadership roles where clarity, not consensus theater, determines success.

I’ve seen too many offers derailed by “one last step.” A candidate clears every structured interview, then a casual lunch or unframed panel suddenly becomes the deciding factor. The team isn’t briefed on what to evaluate, one lukewarm comment lands, and group dynamics cascade into a no-hire. That’s not rigor—it’s randomness masked as prudence.

Groupthink ≠ good hiring decisions. When everyone has veto power, risk-averse no-decisions become the default. Focus-group-style interviews create bias, not signal, and “culture fit” often becomes a proxy for stereotyping or personal preference. As product leaders, we’d never ship a feature based on vibes; we shouldn’t make high-stakes hiring calls that way either.

There’s a better way—and it mirrors how we run great product discovery. Define who you’re hiring before writing the job description. Set clear success metrics for the role. Assign each interviewer specific criteria to evaluate. Treat hiring like product discovery: intentional, structured, and evidence-based. In my teams, that looks like tight scorecards, interviewer calibration, and a decision owner who synthesizes evidence—not a popularity contest where the loudest voice wins.

Chemistry checks still matter, but only when we define what collaboration actually means for the role. Introversion, debate style, or lunch-table small talk are not performance indicators. I look for behaviors we value in empowered product teams—clarity of thinking, healthy dissent, co-creation under constraints—often via a real working session with the future product trio. Diverse teams outperform homogenous ones, even if not everyone “vibes,” so I optimize for complementary strengths over sameness.

If you’re a candidate, remember: When a process feels broken, it’s often not about you. Ask how you’re being evaluated to gauge process maturity; a thoughtful team will happily walk you through their rubric and what great looks like. For structure and support, I’ve seen “Who: The A Method for Hiring” help leaders clarify requirements; “Never Search Alone” and joining a Job Search Council (JSC) can give you peer accountability and sharper narratives. For current openings, I regularly point PMs to Scott Baldwin’s PM job postings on LinkedIn.

My challenge to fellow product leaders: Audit your hiring process the way you’d audit your roadmap. Where are decisions getting stuck? Where are you over-indexing on consensus and under-indexing on evidence? Tighten the criteria, streamline stakeholders, and instrument the funnel so you can learn and improve. The payoff is faster, fairer, more confident decisions—and teams that reflect the rigor we expect in product strategy and stakeholder management.

What’s one change you can make this week—reworking the scorecard, calibrating interviewers, or replacing an unstructured lunch with a real collaboration exercise? Small improvements compound. Let’s build hiring systems that are worthy of the talent we’re trying to attract.

Inspired by this post on Product Talk.

February 3, 2026