I’m sharing a focused set of insights on analytics, experimentation, and personalization designed to help teams ship smarter, reduce risk, and accelerate outcomes. Drawing on years of leading product teams, I translate complex data practices into practical playbooks you can apply immediately to improve user activation, conversion, and retention.
My approach starts with a strong measurement foundation. I lean on a unified analytics platform—often powered by tools like Amplitude analytics—to centralize product, marketing, and customer success signals. With clear event taxonomies, consistent governance, and trustworthy dashboards, teams gain a single source of truth to prioritize the right problems and sequence roadmap bets with confidence.
Experimentation turns insight into evidence. I emphasize A/B testing discipline, including minimum detectable effect (MDE), guardrail metrics, and pre-registered hypotheses. This repeatable system lifts decision quality, shortens feedback loops, and aligns cross-functional partners around what actually moves the needle, not what merely sounds promising.
Personalization compounds the value of experimentation by delivering the right value to the right segment at the right moment. Thoughtful in-app guides and product tours—rooted in behavioral signals—nudge users through friction points and increase the likelihood of early wins. The result is a more intuitive path to first value, stronger user activation, and healthier long-term engagement.
Retention is the ultimate scoreboard. I rely on retention analysis, cohorting, and leading-indicator metrics to connect feature usage to durable outcomes. When paired with product-led growth motions, teams can identify activation thresholds, build habit loops, and scale what works without overextending sales or support capacity.
If you’re getting started, begin with a crisp instrumentation plan, shared definitions, and a lightweight review ritual. Use continuous discovery practices, opportunity solution tree mapping, and driver trees to tie data signals to real user problems. From there, iterate: test small, learn fast, and scale what is proven. Over time, this system becomes a flywheel for product strategy—fewer debates, more evidence, better products.
In this series, I distill the frameworks, templates, and real-world lessons that have consistently improved outcomes for product teams: how to structure experiment backlogs, how to read funnel breakpoints, how to detect false positives quickly, and how to operationalize analytics for day-to-day decisions. Expect practical guidance you can copy, adapt, and run with immediately.
Inspired by this post on Amplitude – Perspectives.
For years, I chased the elusive goal of delivering a perfect customer experience. Today, with AI embedded in our support operations, that standard is finally within reach—and it’s reshaping how we prioritize, design, and scale service.
In “The 2026 Customer Service Transformation Report,” teams report early, tangible wins from AI: faster responses, higher efficiency, and consistent coverage across languages and time zones. Those gains create the capacity we’ve always needed. The more we push the technology, the more quality improvements we unlock.
This marks a fundamental shift. As AI takes on more, our focus can finally move from firefighting to crafting the customer experience. When the AI is working, the measure of success becomes how well it’s working—across accuracy, tone, resolution, and end-to-end journey quality.
I’ve seen this transformation firsthand. Mature AI deployment gives my team “breathing room,” so we can design for consistently excellent outcomes rather than obsess over deflection. That means widening access to support, removing friction on the path to resolution, and anticipating customer needs before they escalate.
In our own support organization, we opened support to trial customers, accelerated first response times, and added consultative sessions during onboarding. We absorbed a 300% increase in total demand without adding headcount—made possible by deep integration of an AI Agent and a disciplined AI strategy.
Teams with mature customer service deployments are nearly three times likelier to say they always meet increasing expectations—27% vs 9% at initial rollout—highlighted by bold orange and gray comparison bubbles.
Across the industry, the pattern is similar. When teams initially deploy AI, only 9% say they can always meet customer expectations. That number triples as teams reach a mature level of deployment. Even as expectations rise, the organizations that deeply integrate AI—complete with clear ownership, robust instrumentation, and continuous improvement loops—are the ones most likely to meet (and exceed) the bar.
Looking ahead to 2026, I expect omnichannel consistency to become a key differentiator. The data shows planned investment is distributed nearly equally across chat, email, and social messaging (36% each), closely followed by phone/voice (31%). The question is no longer “Which channel should we optimize?” but “How do we deliver a consistent, AI-powered experience everywhere our customers are?”
Teams that solve for omnichannel consistency will bridge the long-standing gap between what customers expect and what support can deliver. Every touchpoint becomes an opportunity to exceed expectations and build durable trust.
Consider Clay, a team that scaled support without sacrificing quality. Support is one of their main growth drivers, and as their customer base expanded, ticket volume surged. Early on, they concentrated much of their effort in Slack, cultivating close, transparent community relationships. But relying on a single channel created friction as they grew; customers wanted the flexibility of email and in-app chat, and Clay needed to deliver the same high standard everywhere.
Where AI investment is headed for customer service in 2026: chat, social, and email lead at 36%, with phone/voice close behind at 31%. A bold visual snapshot of shifting channel priorities in CX.
By unifying their support experience with an AI Agent, Clay brought consistency across channels. Today, AI is involved in 90% of all queries and handles half of Clay’s total volume, upwards of 7,000 queries a month. First response rates improved significantly, freeing the team to focus on proactive, high-impact work.
That work includes identifying content gaps for education and content marketing, reaching customers before they need to ask for help, and surfacing feature requests and recurring challenges to product teams. Clay proves that when support is truly great, it becomes a competitive edge.
So how do you build a superior customer experience with an AI Agent? Here are five principles I use when scaling toward mature deployment.
1) Treat customer experience like a product. Treating support as a product means designing, building, and managing the support experience with the same rigor as your core product. You define goals (faster onboarding, higher CSAT or CX Score, lower churn). You map flows (AI starts the conversation, human handovers, proactive nudges). You instrument the journey (track handoffs, drop-offs, success states). You run tests and ship improvements (tone tweaks, fallback paths, training updates). You own the outcomes (gather feedback, measure performance, use insights to continuously improve the system).
Leaders are racing ahead with real AI in support. Explore the 2026 Customer Service Transformation Report to see where deployment is stalling, benchmark your team, and get practical steps to scale automation that delights.
2) Lead with AI, back with humans. AI isn’t replacing the human touch. It’s redefining when, where, and how it’s most valuable. In a scaled model, AI is the first responder and the end point for most conversations. Humans step in where they add the most value—particularly during high-stakes issues—and those handoffs should feel seamless. Meanwhile, your team focuses on improving AI performance and optimizing the end-to-end journey.
3) Be proactive. Use AI to anticipate needs, guide customers before problems arise, and nudge them toward successful outcomes. This is where customer support AI strategy shines—moving from reactive triage to journey orchestration that protects momentum and builds trust.
4) Build for trust. Many customers still carry the legacy of clunky chatbots that delivered vague answers and dead ends. You earn trust by showing that your system works. Don’t hide your AI Agent behind layers of “choose an option.” Get customers to the AI quickly, demonstrate real problem-solving, and ensure that when a human is needed, they join with full context to resolve complex issues efficiently.
5) Make it feel personal. Your AI Agent represents your brand. The way it speaks, follows policies, and responds matters. Use tone control, fallback logic, and language preferences to align the experience to your standards. Consistency builds trust; personality builds connection and loyalty.
Perfect really is possible. With deep AI implementation, you can scale comprehensive, fast, and personal support across channels—so customers feel supported not just when they reach out, but throughout their journey. That’s the promise of modern AI workflows in support, and it’s what will separate leaders from laggards in the years ahead.
For years, I’ve watched high-performing product teams run into the same wall: the gap between insight and action. Dashboards multiply, yet decisions stall. That final mile—where we interpret trends, prioritize tradeoffs, and ship changes—remains the last bottleneck. It’s not a data problem; it’s a bandwidth and focus problem.
Amplitude's AI Analytics Platform takes the next step: agents that investigate, monitor, and act so your team can build what actually matters.
From my seat leading product at HighLevel, I see “agentic AI” as a structural upgrade to the product operating system. Instead of waiting on human cycles to discover anomalies, craft hypotheses, and trigger the next experiment, Agent Analytics can continuously investigate user behavior, monitor mission-critical metrics, and initiate actions—closing the loop from observation to outcome. That shift transforms analytics from a passive reference layer into an active, decision-making teammate.
Practically, this matters because empowered product teams win on speed and focus, not on the volume of reports. When agents surface the most material opportunities—say, a sudden drop in activation for a high-value cohort or a retention dip tied to a recent release—we compress time-to-insight and, more importantly, time-to-action. The result is fewer context switches, fewer meetings, and more cycles invested in building meaningful value.
The most compelling use cases are those that compound: continuous discovery that highlights friction in onboarding flows, proactive retention analysis on at-risk segments, automated experiment prioritization aligned to outcomes vs output OKRs, and closed-loop alerts that trigger workflows in your CRM or in-app guides to accelerate product-led growth. With a unified analytics platform feeding these agents, we can move from reactive analytics to anticipatory product strategy.
Of course, leverage requires guardrails. I anchor adoption in three pillars: clear decision rights for agents (what they can autonomously act on vs. recommend), transparency in reasoning (so PMs can audit how conclusions were reached), and explicit alignment to key outcomes (activation, retention, expansion). Done right, this is not a replacement for product judgment—it’s an amplifier for it.
If I were rolling this out today, I’d set a success dashboard that tracks: time-to-insight, time-to-action, percentage of initiatives initiated by agents, impact on North Star metrics, and the reduction in manual analysis hours. I’d also implement lightweight prompts and playbooks—LLMs for product managers—that standardize how we ask better questions and interpret agent outputs.
The promise here is simple but profound: eliminate the last bottleneck by giving your teams a partner that never sleeps, never tires, and never loses the plot. When agents investigate, monitor, and act, we spend less time arguing about the data and more time building the right things, faster.
Inspired by this post on Amplitude – Best Practices.
I rely on Amplitude analytics and Figma Make to turn real user insights into high-fidelity prototypes in hours, not weeks. This pairing compresses our continuous discovery loop and helps my team prioritize what truly moves the needle for customers and the business.
Design smarter with Amplitude and Figma Make. Use AI and product analytics together to prototype, test, and learn faster.
Here’s how I put that into practice: I start with product analytics to isolate a measurable opportunity—often around user activation, conversion drop‑offs, or retention analysis. Amplitude cohorts and funnels surface where friction hides; I translate those signals into design prompts and flows in Figma Make, so we can visualize and validate potential solutions before a single line of production code is written.
Once a promising direction emerges, I convene the product trio—design, engineering, and product—around a clear outcome metric, not output. We build a lightweight driver tree, align on a hypothesis, and define the minimum detectable effect (MDE) so our A/B testing has enough statistical power to be decision‑worthy. From there, we create a small set of Figma Make variations that reflect distinct value hypotheses, not cosmetic tweaks.
On the experimentation front, I gate risky changes behind feature flags and ship via our CI/CD pipeline to limit blast radius and accelerate feedback. I monitor the experiment with a unified analytics platform mindset: the same definitions and segments in Amplitude power both pre‑launch discovery and post‑launch evaluation. That continuity lets us compare prototype expectations against production reality with far fewer translation errors.
A few principles keep this workflow sharp and responsible: I use privacy-by-design patterns, apply data governance guardrails to keep datasets consent‑aligned, and set AI risk management standards so generated designs respect accessibility and brand constraints. Critically, I avoid vanity metrics—I measure learning speed, decision quality, and downstream impact on activation or retention, which are what sustain product-led growth.
If you’re looking for a playbook, try this cadence: 1) define the customer outcome and success metric; 2) map a simple driver tree to narrow the solution space; 3) explore multiple flows in Figma Make; 4) validate quickly with concept tests and usability checks; 5) run A/B testing with a clearly defined MDE; 6) ship iteratively behind feature flags; 7) close the loop in Amplitude with cohort‑level retention analysis; 8) refine copy and UX writing to reinforce the core value proposition. Repeat until the signal is undeniable.
Blending Amplitude analytics with Figma Make has become my fastest path from insight to impact. It keeps my team focused on learning that compounds, features that matter, and outcomes customers can feel—so we truly make what matters.
Inspired by this post on Amplitude – Best Practices.
What happens when you treat an AI agent not as a chatbot, but as a full teammate on your sales team – one that can jump on video calls, demo your product, make phone calls, and follow up over days?
I recently dug into this question with the team behind ShowMe, an AI-native startup building digital sales reps for inbound teams. Founded in April 2025, ShowMe has engineered a multi‑agent system that combines conversation agents for live voice and video interactions, evaluator agents that score every call for quality and sentiment, and creator agents that ingest customer documentation to build tailored playbooks. A workflow layer orchestrates the entire lead‑to‑close journey across days, not minutes—exactly the kind of agentic AI approach I expect to see become standard in revenue workflows.
What stood out to me first was the origin story: a glaring conversion gap on a previous website, and the realization that a purpose‑built AI could fill it. The initial MVP was refreshingly pragmatic—start with a voice agent, pair it with product videos, and back it with a simple RAG knowledge base. That retrieval‑first pipeline let the team ship quickly, validate real user behavior, and then scale sophistication where it mattered.
Then came a pivotal affordance shift: adding a realistic avatar via HeyGen. It wasn’t just eye candy; it changed how prospects engaged. The video-call UX established trust and made the AI’s capabilities legible at a glance. Prospects behaved as if they were with a human rep—interrupting, probing, and asking for demos—because the surface area invited that behavior.
On the architecture side, the team decomposed a single sales conversation into multiple specialized sub‑agents—greeting, qualifying, pitching—to manage latency, memory constraints, and model limitations. Deterministic workflows handle the happy paths reliably, while a smart orchestrator is emerging to break out of rigid paths when context demands it. Confidence scoring and frustration detection kick in for real‑time human handoff decisions, a must for revenue‑critical moments where a missed nuance can cost pipeline.
Training the system to sell like your team is where it gets powerful. ShowMe ingests sales transcripts and training materials to teach company‑specific sales skills, then uses creator agents to assemble tailored playbooks. Conversation agents stay focused on live interactions, while evaluator agents continuously score calls for quality and sentiment. The result: repeatable, compliant, and brand‑consistent selling—without flattening personalization.
Quality isn’t an afterthought—it’s operationalized. Early deployments run with customer-driven evaluation loops where 100% of conversations are reviewed, tapering to about 5% over time as confidence increases. Feedback becomes automated tests to prevent prompt regression, and production quality is proven with POCs, A/B rollouts, dashboards, and CRM logging. This is eval-driven development applied to go‑to‑market: measurable, auditable, and continuously improving.
I also appreciate how they treat the agent as a coworker, not a widget. Onboarding happens via Slack, weekly reporting aligns with sales leadership rhythms, and tight CRM integration keeps data flowing both ways. That mindset unlocks adoption because it fits how sales teams actually operate—and it creates real Agent Analytics you can manage.
From a product perspective, several pragmatic details matter. Real‑time voice and avatar demos rely on latency tricks and a library of video clips to keep interactions snappy. The conversation agent evolved from a basic Q&A bot into guided sales discovery, balancing personalization with the ever-present risks of hallucination. Guardrails, human‑in‑the‑loop, and clearly defined handoff rules are non‑negotiables in high‑stakes sales workflows.
Looking ahead, the roadmap makes sense: move toward self‑serve PLG setup, add smarter orchestration that adapts beyond deterministic flows, and expand into adjacent roles like customer success. For product leaders building in gen ai, the pattern here is instructive: start with inbound value, design AI workflows that align to proven sales motions, and use rigorous evals to earn the right to automate more.
If you want to go deeper into the build, the live demos, and the full multi‑agent orchestration, listen to this episode on: Spotify | Apple Podcasts. For more on the stack, explore ShowMe and the avatar platform HeyGen.
I’ve been reflecting on why so many revenue leaders are at risk of falling behind, and the conclusion is stark: fewer than 10% of current CROs will thrive by 2028. That isn’t hyperbole—it’s a wake-up call for how quickly go-to-market strategy, organizational design, and AI-driven execution are evolving. From my seat leading product, I see the pressure building on the CRO role to orchestrate the entire revenue system, not just run a sales team.
One story that crystallizes this reality comes from the journey of Stevie Case, the CRO of Vanta, the trust management platform serving everyone from founders to Fortune 100 CISOs. A former pro-video gamer who stumbled into sales through a mentor’s bet, she exemplifies how unconventional paths can drive unconventional insight. Her trajectory underscores a bigger truth I’ve witnessed across companies: the best revenue leaders aren’t just great sellers—they’re builders who understand product, process, and people at scale.
Why do early revenue hires fail? In my experience, it’s rarely about raw talent. It’s about fit, scope, and time horizon. Early-stage teams often hire coin-operated closers to sprint for this quarter’s number, when what they actually need are long-term builders who can shape ICP clarity, pipeline math, and repeatable motion. The trap is simple: you hire for momentum before you’ve validated the motion. That misalignment shows up at 00:00 Why early revenue hires fail and again at 04:16 Coin-operated sellers vs. long-term builders—two ideas every founder-led GTM team should internalize before the first half-dozen sales hires.
What separates a VP of Sales from a top 1% CRO is scope and systems thinking. A true CRO owns the full revenue engine—marketing, sales, solutions engineering, customer success, pricing, channels, and post-sale activation—not just the new-business line. It’s a role defined by precision around 07:44 Metrics, confidence, and velocity and the courage to decide when to centralize vs. decentralize capabilities as you grow. Should CROs lead sales? At 12:04 Should CROs lead sales?, the nuance is clear: yes, if the motion is still coalescing; not necessarily, once the machine is humming and specialization unlocks scale. My rule of thumb: start consolidated for speed of learning; split functions only when interlocks are provably robust.
There’s a humbling lesson in 16:36 Learning to scale at Twilio and 19:58 Stevie’s scaling mistake at Vanta: copying another company’s operating system, even a world-class one, is an easy way to blunt your edge. Context is king. What worked at Twilio won’t automatically work at a trust management business. That’s why the line at 17:44 “There is no CRO playbook” resonates so deeply. There are principles—org design, segmentation, enablement, compensation, customer activation—but your playbook must be bespoke to your product, pricing, cycle time, and buyer power map.
22:16 Why Vanta stays 100% sales-led is a reminder that not every high-growth motion demands product-led growth. In categories where compliance, security, and risk shape buying behavior, a consultative, sales-led approach builds trust and shortens time to value—especially when solutions engineering, onboarding, and customer success are tightly choreographed. I’ve seen teams chase PLG headlines while ignoring the higher-ROI path right in front of them: nailing the sales-led experience, from first touch to first value.
Top CROs plan 24–26 months ahead. 23:16 The value of planning 24-26 months ahead isn’t about creating perfect forecasts; it’s about designing optionality. That means hiring with stage gates, building enablement before you feel “ready,” instrumenting activation and retention early, and pressure-testing your pricing and packaging quarterly. In my org reviews, I push for scenario modeling: what breaks at 2x volume, what centralizes again at 600 headcount, and what competencies must be grown vs. bought.
On judgment and decision quality, 29:54 When trusting intuition was the wrong call is a familiar leadership tax. Pattern recognition is powerful—until it isn’t. I’ve learned to pair intuition with a data backstop and a lightweight pre-mortem: what would have to be true for this to fail? It’s the same posture I take with AI in GTM. At 30:49 Do humans still have a place in the future of GTM? and AI vs. humans in go-to-market, the answer is yes—but augmented. Humans set narrative, negotiate ambiguity, and build trust; AI accelerates research, writing, discovery, and coaching. The winning motion fuses both.
I’m often asked which tools materially shift outcomes. For revenue intelligence and operational rigor, I look to systems that compound learning: Gong: https://www.gong.io/, Salesforce: https://www.salesforce.com/, and Cursor: https://cursor.sh/. To study benchmark operating models and developer-led growth infrastructure, Twilio: https://www.twilio.com/ remains instructive. And to understand why trust, security, and compliance can define the entire GTM architecture, Vanta: https://www.vanta.com/ is a useful case study.
Leadership non-negotiables matter more as you scale. 33:33 Stevie’s leadership non-negotiables reminded me to be explicit about standards: clarity over activity, customer outcomes over internal wins, and auditability over anecdotes. 36:36 The myth of hiring for industry expertise shows up again and again—I’d rather hire for learning velocity, systems thinking, and builder DNA than narrow domain familiarity. And at 40:00 What stays centralized in a 600-person company, remember: centralize what must be consistent (data, tooling, pricing guardrails, core enablement), decentralize what benefits from speed and context (segment plays, partner motions, field marketing).
If you prefer a structured digest, here’s the operating checklist I use with revenue and product peers: define your ICP and value proposition crisply; hire builders over coin-operated sellers; instrument the first 30 days post-sale (47:09 The hidden leverage of a customer’s first 30 days); align pricing, packaging, and onboarding to activation; model capacity and hiring plans on 24–26 month horizons; decide early what stays centralized; use AI to amplify discovery, coaching, and content while keeping humans front-and-center for trust-building; and cultivate an unvarnished CEO–CRO pact (01:02:30 Unpacking the CEO-CRO dynamic) that aligns on strategy, segmentation, and sequencing.
For those who want a few timeline highlights: 00:00 Why early revenue hires fail; 02:23 Who to hire at $5M in revenue; 05:57 What excellence looks like in the CRO role; 17:44 “There is no CRO playbook”; 22:16 Why Vanta stays 100% sales-led; 23:16 The value of planning 24-26 months ahead; 47:09 The hidden leverage of a customer’s first 30 days; 53:42 Why the CRO role will face enormous changes by 2028; 58:42 What leaders must do now to stay relevant.
The throughline is simple and urgent. 53:42 Why the CRO role will face enormous changes by 2028 isn’t a forecast—it’s a present-tense mandate. 58:42 What leaders must do now to stay relevant: build a revenue system, not a sales team; plan further out while executing faster; let AI handle the mechanical so your people can master the human. Those who internalize this shift will be the fewer than 10% of current CROs who thrive by 2028. The rest will be outpaced by change they could have anticipated—and designed for.
Over the past few years, I’ve led cross-functional teams to deploy agentic AI in production, and I’ve learned that success rarely hinges on the model alone. It comes from methodically designing the right workflows, instrumenting every step, and building a feedback loop that compounds. Learn how companies like Replit are consolidating workflows, creating one-person departments, and building systems for scale with Amplitude.
When I talk about AI agents, I’m describing software that behaves like a focused teammate—owning a clear job to be done end-to-end. In practice, that means consolidating fragmented tasks into a single accountable “one-person department,” then giving it the context, tools, and analytics to perform reliably. This is how agentic AI moves beyond demos into durable business impact.
I start with outcomes, not algorithms. I map a driver tree from business goals (e.g., lower response time, higher activation, better retention) to the specific moments an agent can influence. This outcome-first alignment keeps scope tight, informs guardrails, and grounds the value proposition in measurable change instead of vanity metrics.
Next, I define the workflow the agent will fully own. I look for high-volume, rules-adjacent processes—think lead qualification, support triage, or billing inquiries—where clear decision criteria already exist but human time is the bottleneck. I document triggers, inputs, decision points, and handoffs, then design the ideal-state flow the agent will run autonomously, with transparent escalation paths to humans.
On architecture, I favor a retrieval-first pipeline to keep responses accurate and current. I scope the knowledge base, implement context window management, and standardize tools the agent can call (search, CRM actions, ticket updates). For teams new to this, I coach “LLMs for product managers” fundamentals so we make sensible trade-offs between speed and reliability rather than chasing model-of-the-week headlines.
Instrumentation is where the system becomes self-improving. I use Amplitude analytics and an Agent Analytics schema to track intent detection, tool usage, resolution rate, time-to-resolution, deflection, and escalation causes. A unified analytics platform lets me connect agent outcomes to core product metrics—activation, retention, and conversion—so we can see the real revenue and experience impact, not just local efficiency gains.
To validate impact, I run A/B testing when traffic allows, setting a minimum detectable effect (MDE) upfront to avoid inconclusive reads. In lower-volume scenarios, I lean on eval-driven development: curated test sets for edge cases, scenario-based regression suites, and error taxonomies that accelerate iteration. Feature flags let us stage capabilities safely (shadow mode, assistive, autonomous) while we monitor deltas before full rollout.
Reliability and trust are designed in from the start. I apply AI risk management practices—privacy-by-design, data governance, and policy-aligned prompt templates—paired with observability to trace decisions. Clear escalation policies, incident management runbooks, and human-in-the-loop checkpoints ensure the agent fails safe, not silently.
Shipping cadence matters. I use CI/CD to increase deployment frequency, keep prompts and tools versioned, and gate risky changes with targeted rollouts. As patterns stabilize, we scale horizontally to new use cases, sharing core capabilities (retrieval, analytics, guardrails) as a platform. This is how “one-person departments” multiply without multiplying overhead.
Change management closes the loop. I partner with product trios and frontline teams to co-design prompts, set acceptance criteria, and define what “good” looks like in plain language. In-app guides and product tours introduce the agent’s role and limits, and structured feedback channels feed directly into our discovery and iteration rhythm.
The throughline of this playbook is simple: treat agents like real teammates with a job description, operating procedures, and performance reviews. With disciplined workflow design, a retrieval-first pipeline, and outcome-level instrumentation in Amplitude, agentic AI stops being a science project and starts compounding into durable product-led growth.
Inspired by this post on Amplitude – Perspectives.
Most product teams—and especially well-run product trios—know they should be interviewing customers. More teams than ever are actually doing it. That’s the good news.
The bad news? Many teams still struggle with what comes next. Turning raw recordings into a structured opportunity space that truly guides product discovery can feel overwhelming.
In my experience, interview synthesis is cognitively demanding work. You have to extract the key moments from each conversation, translate those moments into clear opportunities, and then organize those opportunities into a coherent view of your opportunity space. It’s no surprise I hear teams say, "We need to stop interviewing so we can catch up on what we’ve already learned." Too often, they pause—and never start again.
Recordings pile up. Maybe there are scattered notes. But nothing gets turned into an opportunity solution tree. The team hasn’t synthesized what they’ve learned, so the research isn’t actionable. That’s the gap I want to help close.
What if you could go from 3 interviews to a draft OST in minutes?
My AI goals are straightforward: 1) build tools that help you learn discovery and 2) build tools that help you do discovery. The learning tools are coming through on-demand courses. Today, I’m excited to share the first big step on the "do" side.
I’m excited to see an expanded partnership with Vistaly—the opportunity solution tree tool many of you already use—to bring AI-powered discovery tools directly into their platform.
Great synthesis happens in two steps: first, you synthesize each interview separately; then you synthesize across interviews. Most AI tools skip the first step and jump straight to cross-interview analysis—exactly how teams lose the nuance and context that make research actionable.
This approach does both. You upload three interviews for the same product outcome. The AI extracts the key moments and opportunities from each one separately. Then it synthesizes across those interviews and generates a first draft of your opportunity solution tree for you. Three interviews in. A draft OST out.
Here’s what this is—and what it isn’t. You’ve probably heard criticism of tools that promise "one-click opportunity solution trees." Those tools ask you to describe your market, click a button, and get a tree. The point of an opportunity solution tree is not to have one—it’s to synthesize what you’re learning from real customers so your team can align on the best path forward. A one-click tree built from made-up data is useless.
Turn interviews into insights in minutes with Vistaly. This hero screen invites you to enroll in beta and showcases an opportunity solution tree that maps outcomes to opportunities and actionable solutions.
This approach is fundamentally different. It starts with your real customer interviews. The AI does the heavy lifting of extracting key moments and opportunities from those conversations and organizing them into a draft opportunity solution tree. But it’s a draft—you review it, refine it, and reorganize it. You bring your judgment and context to the work.
My vision for AI-aided cross-interview synthesis is simple: AI identifies common opportunities across interviews, suggests a tree structure, and facilitates the team’s review. Historically, it’s been hard to give AI access to an opportunity solution tree in a way that preserves structure and context. The integration with Vistaly solves that problem by building this capability directly into the tool where your tree already lives.
In my own experiments using Claude, the AI surfaced opportunities I missed—and I caught things it missed. The highest-quality synthesis came from combining both perspectives. Research (see here and here) backs this up: Experts working with AI outperform both experts working alone and AI working alone. That’s the model we’re building toward—AI generates the draft, you bring the expertise.
I have mixed feelings about AI doing discovery work for us because there is real value in doing the synthesis yourself. But I also know that a draft OST you actually refine is better than a perfect process you never get to. This is about raising the floor—helping more teams get to a structured opportunity space, even if they aren’t doing every step manually.
We’re looking for a small group of alpha partners to help shape this product. To apply, sign up for a free Vistaly account and upload three customer interviews for the same outcome or product space.
We’ll select alpha partners from the applicants. We want a range of interview styles, experience levels, and product spaces. Selected partners will get access to the AI-powered synthesis tools and will work closely with the team to shape the product. Even if you aren’t selected for the alpha, your application puts you at the front of the line when we enter beta.
A few things to know as you apply: Your three interviews should be for the same outcome, goal, or product space, so the tool can generate a meaningful OST. You don’t need to be a Vistaly user today—the account is free. You don’t need to be an expert interviewer either; we’re looking for a range of experience levels, though we’re particularly interested in story-based customer interviews.
This is just the beginning. The vision is a full AI-powered discovery suite inside Vistaly—from interview analysis to complete interview snapshots to opportunity solution trees and beyond. We’ll learn alongside our alpha partners and share what we discover as we go.
If you’ve been looking to bridge the gap between your customer interviews and your opportunity space, this is your chance to help shape how that works. Apply for the alpha today.
When people ask me about "LLM vs AI Agents: What Product Teams Must Get Right," I start with a simple truth: an LLM is a powerful prediction engine, while an AI agent is a productized workflow that plans, takes actions with tools, remembers, and closes the loop on an outcome. That difference sounds academic until you’re on the hook for reliability, cost, and customer trust.
In my role, I’ve shipped LLM copilots that delight users and piloted agents that automate complex workflows. The pattern that never fails is this: start assistive, then graduate to autonomy. Copilots accelerate people; agents own outcomes. When we respect that gradient, adoption climbs, incidents fall, and we earn the right to expand scope.
The first decision point is use-case fit. If the task benefits from human judgment, high-context nuance, or brand voice, I frame it as a copilot with strong guardrails and crisp UX. If the task is well-bounded, tool-heavy, and verify‑able, I consider an agent—but only after we can measure end‑to‑end task success with eval-driven development.
Architecture matters. I reach for a retrieval-first pipeline to keep responses grounded in authoritative data, then add tool use for actions (search, write, schedule, transact) with deterministic scaffolding to prevent thrashing. Good prompt engineering is table stakes, but context window management and a clean memory strategy (short‑term scratchpad, long‑term facts, and policy) separate demos from durable systems.
Agents amplify both value and risk. I build safety in layers: role and scope definition, tool whitelists, unit limits, human‑in‑the‑loop checkpoints at irreversible steps, and privacy-by-design data governance. We log every decision token-for-token because auditability isn’t optional once agents touch customers, money, or data.
Measurement is non‑negotiable. For LLM features, I track time‑to‑first‑token, response latency, groundedness, and user satisfaction. For agents, I add Agent Analytics: task success rate, number of steps per task, tool error rate, loop detection, guardrail triggers, escalation to human, cost per successful task, and containment rate. If we can’t see it, we can’t ship it.
My delivery playbook mirrors modern software ops. We use feature flags, gated betas, and canary rollouts; we version prompts like code; we set incident management paths for model outages and tool drift; and we rehearse fallbacks so the experience degrades gracefully, not catastrophically. Dull operations build dazzling products.
On roadmapping, I thin‑slice value. We introduce a minimal viable copilot that handles a single, frequent job-to-be-done with high success. Only after continuous discovery confirms product‑market fit do we grant more autonomy, one capability at a time. Outcomes vs output OKRs keep us honest: if the customer’s job gets done faster, cheaper, and with fewer errors, we scale; if not, we fix fundamentals before adding scope.
Build vs buy is rarely binary. I tend to buy the undifferentiated heavy lifting—observability, prompt versioning, red‑teaming, and policy enforcement—while building the proprietary workflows, data modeling, and UX that encode our defensible advantage. The litmus test: if it’s part of our unique value proposition, we own it; if not, we integrate the best‑in‑class and move.
Go‑to‑market must be as rigorous as the tech. We position clearly (assistant vs agent), price to value with transparent consumption SaaS pricing, and communicate risk posture in plain language. Customers don’t buy models; they buy confidence that a job gets done reliably within their constraints.
Common failure modes repeat: shipping autonomy before instrumentation, treating prompts as magic instead of software, skipping data governance, and ignoring the human experience. The antidote is disciplined AI Strategy rooted in empowered product teams, tight feedback loops, and relentless evaluation.
If you take nothing else: choose the right paradigm for the job (copilot first, agent when proven), ground with a retrieval-first pipeline, instrument with eval-driven development and Agent Analytics, and operationalize like a mission‑critical system. Do that, and you’ll turn LLM capabilities into durable product outcomes.
Every week, product and data leaders ask me the same question: can AI agents truly shoulder enterprise analytics without sacrificing trust, governance, or speed? I’ve spent the past year putting agentic AI through its paces in real product workflows, and I’ve distilled what works into a practical, task-driven evaluation approach you can adopt immediately.
Learn how to evaluate AI analytics agents with a task-based framework across analytics tasks. See how Amplitude’s Global Agent scores.
When I say “enterprise analytics,” I’m talking about far more than chatty dashboards. The bar includes consistent metric definitions, privacy-by-design, RBAC and data governance, audit trails, low-latency decision support, and repeatable outcomes across retention analysis, funnels, cohorts, A/B testing, instrumentation planning, and anomaly detection—ideally within a unified analytics platform.
My task-based framework evaluates eight capability pillars I expect from an enterprise-ready Agent Analytics solution: task coverage and depth across common product analytics workflows; data fidelity and governance (lineage, access controls, PII handling); instruction-following and reasoning transparency; evaluation rigor and reliability (repeatability, error modes, regressions); security and compliance posture; latency and cost efficiency; integration into existing product strategy workflows (e.g., CRM integration, CI/CD-linked instrumentation, experiment platforms); and human-in-the-loop controls for approvals and guardrails.
Operationally, I define canonical tasks that reflect day-to-day product management: codify a North Star metric; perform retention analysis by cohort; generate and explain a funnel with drop-off drivers; recommend an event taxonomy and tracking plan; analyze an A/B test with minimum detectable effect (MDE) considerations; and propose a driver tree that maps inputs to outcomes. Each task comes with ground-truth datasets, acceptance criteria, and edge cases to stress the agent—an eval-driven development practice I’ve found indispensable.
I then score maturity across four levels. L0: a pure chat UI that summarizes existing charts. L1: a retrieval-first pipeline that grounds responses in your analytics catalog and metric store. L2: a tool-using agent that is schema-aware, can write safe SQL, and reconciles results to canonical definitions. L3: a governance-aware autonomous workflow that executes analytics tasks end-to-end with approvals, audit logs, feature flags, and rollback plans. Most teams discover they’re between L1 and L2; reaching L3 requires serious investment in data governance and eval automation.
Risk management is non-negotiable. I require strict data governance and privacy-by-design controls, including scoped credentials, PII redaction, policy-aware retrieval, and comprehensive observability (query traces, prompt/response logs, lineage). Feature flags and approval gates prevent unintended metric redefinitions. Red-teaming tasks expose prompt injection, schema drift, and hallucination failure modes before they hit production stakeholders.
Where do agents shine today? Rapid exploration, SQL generation from schema context, summarizing experimentation results, and turning natural-language questions into actionable charts. Where do they struggle? Ambiguous metric semantics, under-specified experiment designs, and edge-case-heavy analyses where ground truth depends on organizational nuance. The cure is disciplined product management: codify definitions, maintain a living analytics taxonomy, and continuously harden your eval suite.
In the context of product analytics stacks, Amplitude analytics is a common anchor for product teams, and many are evaluating “Amplitude’s Global Agent” to accelerate insight generation. In my framework, I look for how well it grounds to canonical metrics, handles retention and funnel tasks, explains trade-offs, and respects governance boundaries—before I consider expanded autonomy. I share the full task matrix and scoring rubric so you can replicate the assessment in your environment.
If you’re getting started, pick your top ten high-frequency analytics tasks and define crisp success metrics for each (accuracy, explainability, latency, and reusability). Build a small eval harness with golden datasets, assertions, and regression tests. Favor a retrieval-first pipeline tied to your taxonomy and metric store, add human-in-the-loop approvals for sensitive actions, then pilot with a cross-functional tiger team. Measure time-to-insight, analyst hours saved, and stakeholder trust—then iterate.
Enterprise analytics isn’t a single feature; it’s a system of definitions, workflows, and governance. With a task-based, eval-driven approach, agentic AI can become a reliable partner—not just a novel interface. If you’re evaluating options, apply this framework first, then expand scope as reliability and trust climb.
Inspired by this post on Amplitude – Best Practices.
I write from a place many product leaders know well—the moment when the data you need to make decisions simply doesn’t exist, and you have to build the capability from the ground up. That firsthand experience with gaps in analytics shaped how I think about product strategy, product discovery, and the relentless pursuit of product-market fit lessons.
In my work, I lean on continuous discovery to surface the most meaningful problems, then translate those insights into outcomes vs output OKRs that keep teams focused on impact. When we anchor roadmaps to real user behavior and business results, we avoid vanity metrics and create a durable plan that compounds learning over time.
Execution matters just as much as insight. I rely on rigorous A/B testing, clear minimum detectable effect (MDE) thresholds, and retention analysis to separate signal from noise. This discipline ensures that every iteration—whether it’s a small UX nudge or a bold bet—moves us closer to measurable value for customers and the business.
None of this works without empowered product teams. I build around product trios that partner tightly across design, engineering, and product, and I foster a product-led growth mindset so we earn activation, engagement, and expansion through the experience itself. The goal is to create a system where learning is fast, ownership is clear, and the user’s job-to-be-done stays front and center.
On the tooling side, I favor a unified analytics platform so insights are consistent from discovery to deployment. Whether I’m instrumenting funnels with Amplitude analytics or stitching together qualitative and quantitative inputs, the principle is the same: give teams trustworthy, real-time visibility so they can make better decisions, faster.
If you’re looking to operationalize these practices, you’ll find practical playbooks, decision frameworks, and real-world examples here—built for leaders who want clarity, speed, and confidence in how they discover, ship, and scale products.
Inspired by this post on Amplitude – Best Practices.
I hear the same refrain from product leadership peers everywhere: we’re overwhelmed. Shrinking headcount, constant AI disruption, economic uncertainty, and relentless context switching make it feel like we’re carrying two jobs—setting strategy while shielding our teams. I recently listened to an episode of All Things Product that zeroes in on what a real support system for product leaders looks like, and it resonated deeply with my day-to-day.
Want to listen to the conversation yourself? Find it on Spotify or Apple Podcasts.
Here’s the core tension I see (and felt early in my own leadership journey): product leaders tend to underinvest in themselves. We hold onto work because it feels faster, safer, or “just easier if I do it.” But that pattern quietly taxes strategy, slows learning, and caps team throughput. The hidden cost of “doing it all yourself” is real.
Early in my tenure leading product, I tried to keep every plate spinning—roadmap reviews, stakeholder prep, user research, executive updates—while protecting my team’s focus. I was busy and useful, but not maximally valuable. The turning point came when I started building a lightweight support stack: a few hours of executive assistant help each week, targeted research support for bet sizing, and a personal cadence with a leadership coach. The result wasn’t just more time; it was better time.
One provocative point that landed hard: product leaders rarely have executive assistants—and that’s a problem. If your calendar is your operating system, an EA is an extension of your leverage. Mine now handles scheduling, meeting hygiene, prep packets, and post-meeting artifacts. That shift moved me from “calendar triage” to “strategic curation.” It also reinforced a core principle: delegation is a leadership skill, not a weakness. When I delegate outcomes (not just tasks), my team learns, ownership grows, and we ship decisions faster.
Support for strategy work shouldn’t stop at the calendar. Research and data enable better bets. Lightweight research ops, access to product analytics, and brief synthesis sprints keep me anchored in evidence without drowning in artifacts. Paired with a strong community of practice, I get a steady stream of comparative patterns—how other leaders delegate, scope advisory boards, or run decision reviews—which short-circuits trial-and-error.
Coaches were framed as shortcuts for clarity, accountability, and skill-building—and I agree. A good coach compresses cycles, sharpens decision quality, and holds the mirror up when you drift into doer mode. Two quotes captured the mindset perfectly: “You are a pro athlete. It makes sense to think about how you scale your impact without adding more to your calendar.” — Petra Wille. “As you get busier, it becomes more important to focus on the value only you can bring.” — Teresa Torres.
There’s also a helpful nudge to let go of perfectionism: “80% done by someone else is 100% awesome.” — Dan Martell (quoted). In practice, that means I accept great drafts from others, then add the 10–20% only I can contribute—context, narrative, and the sharp edges of the decision.
What about AI? The conversation hits a practical middle ground I share: use AI where it compounds leverage—meeting summaries, research synthesis starters, doc outlines, and backlog triage. But keep humans where judgment, alignment, and context truly matter—strategy framing, stakeholder management, and the final decision-making loops. In other words, apply an AI Strategy that respects product leadership’s uniquely human work.
Key themes I took away: why product leaders struggle to scale themselves; the true cost of “doing it all yourself”; why not having executive assistants limits impact; delegation as a core leadership capability; how to identify and protect the work only you can uniquely do; using research and data to inform strategy; coaches as accelerators for clarity and accountability; communities of practice as a force multiplier; adopting a “professional athlete” mindset; when AI helps—and when humans still matter; and the liberating mantra that “80% done by someone else is 100% awesome.”
If you’re wondering where to begin, start small and practical. Audit your time: what work truly requires you? Experiment with small amounts of support (even a few hours a week). Delegate outcomes, not just tasks. Keep the hands-on work you love—but be intentional. Use peers, coaches, and communities to learn how others delegate. Don’t wait until burnout to build your support system.
Resources mentioned if you want to go deeper: Follow Teresa Torres: https://ProductTalk.org. Follow Petra Wille: https://Petra-Wille.com. Petra’s Coaching for Product Leaders: https://www.petra-wille.com/coaching-packages. Dan Martell’s book Buy Back Your Time: https://www.buybackyourtime.com.
I’m curious: what’s one outcome you’ll delegate this week, and what support would make it stick? Share your thoughts in the comments—your playbook might be exactly what another product leader needs right now.
