Tag: A/B testing

How I Use ChatGPT to Supercharge Product Management: Workflows, Prompts, and PM Playbooks

I treat ChatGPT as a force multiplier across the entire product lifecycle—from discovery and strategy to delivery and growth. Unlock workflows, prompts, and real PM tips showing how ChatGPT quietly reshapes product management behind the scenes.

My goal is pragmatic: turn generative AI into repeatable, measurable leverage for product discovery, product roadmapping and sprint planning, stakeholder management, and product-led growth without sacrificing quality, privacy-by-design, or judgment. This is how I apply LLMs for product managers in a way that strengthens customer empathy and speeds up decision cycles.

In discovery, I use ChatGPT to synthesize interviews, categorize sentiment, and surface emergent themes faster than a manual pass. I’ll feed it anonymized notes and ask for Jobs-to-be-Done statements, contradictory signals to validate, and the top three risks to our hypotheses. When the corpus gets large, I pair it with a retrieval-first pipeline and apply context window management so outputs stay grounded in real customer data.

On strategy and positioning, I draft and refine a crisp value proposition, clarify points of parity, and identify competitive differentiation. I ask ChatGPT to convert inputs into outcomes vs output OKRs, pressure-test assumptions, and produce a one-page narrative that even non-technical stakeholders can engage with. The result is faster alignment and fewer meetings to get to the same level of clarity.

For planning and delivery, I use ChatGPT to accelerate PRD outlines, user stories, and acceptance criteria, while explicitly requesting edge cases, failure states, and non-functional requirements. I’ll have it map risks to mitigations and suggest simple instrumentation aligned to DORA metrics and incident management readiness—useful when we’re iterating within a CI/CD cadence.

In experimentation, ChatGPT helps me frame strong A/B testing plans, calculate a minimum detectable effect (MDE), and sanity-check sample sizes. I also use it to translate metrics into plain language updates for the team, connect learnings to the next experiment, and propose follow-up analyses for retention analysis or activation bottlenecks.

For growth and onboarding, I prompt ChatGPT to generate hypotheses for user activation, in-app guides, and tooltip design that match personas and JTBDs. It drafts variations I can quickly test through Pendo or similar tools, supports product-led growth motions, and helps craft contextual copy that aligns with our value proposition without adding cognitive load.

Stakeholder communications get sharper and faster. I’ll ask for concise executive summaries, a version tailored for engineering leaders, and another for customer-facing teams. It’s especially effective for QBRs vs OKRs updates, where I need crisp narratives tied to outcomes, plus a plain-English articulation of risks and trade-offs for empowered product teams.

The guardrails matter. I set clear AI risk management boundaries, prevent any sensitive data from entering prompts, and align usage with data governance and regulatory compliance requirements. I also version and review prompts just like product artifacts, so the best ones evolve into a durable AI product toolbox the whole team can use.

If you’re getting started, pick one high-friction workflow—say, interview synthesis or PRD drafting—and timebox a week to build a repeatable prompt set and review rubric. Measure cycle-time savings and quality deltas, then expand to a second workflow. Within a month, you’ll have a lightweight operating model for AI Strategy that compounds across your roadmap.

Inspired by this post on Product School.

November 20, 2025
Vibe Check Part 2: Feel Something, Measure Everything with High-Impact Amplitude Analytics

I build products with equal parts intuition and instrumentation. When a campaign’s purpose is to spark a feeling, I still demand proof that those moments translate into measurable outcomes. Learn how you can use Amplitude to better track your vibe marketing initiatives in part 2 of our 3-part series.

Vibe marketing works when emotion and evidence move in lockstep. In my practice, I rely on Amplitude analytics as a unified analytics platform to connect the emotional resonance of a message to product-led growth—tracking how a compelling story influences user activation, retention, and revenue. The goal is simple: feel something, measure everything.

I start by instrumenting the journey around the vibe itself. That means a clean event taxonomy and consistent properties that capture the creative theme, channel, audience segment, and context (for example: campaign_id, creative_theme, entry_channel, audience_mood, landing_variant). Good data governance is non-negotiable—if the data isn’t trustworthy, neither are the insights. With this foundation, I can attribute emotional narratives to downstream behaviors with confidence.

From there, I map the funnel and define activation with intent. I track how vibe-forward touchpoints influence key milestones—first value moments, time-to-activation, and early feature engagement—then ladder those signals into retention analysis. Cohorting users by creative theme or channel helps me see which vibes convert initial curiosity into durable product habits, and which only produce short-lived spikes.

Experimentation is where the rigor shows. I use A/B testing to isolate the impact of a specific narrative, headline, or creative treatment, and I size tests based on minimum detectable effect (MDE) to avoid underpowered decisions. Guardrail metrics (activation, retention, and NPS) protect the experience while I iterate. When the numbers are tight, I supplement with directional reads—session quality, content depth, and return visits—while staying honest about causality.

Operationally, my team lives in shared Amplitude dashboards and notebooks. We annotate launches, align on outcomes vs output OKRs, and review weekly trendlines with our GTM partners. This cadence keeps empowered product teams focused on what matters: which vibes accelerate onboarding, deepen engagement, and ultimately improve unit economics. When a story resonates, the data should echo it across the funnel.

The biggest pitfalls I see are vanity metrics and disconnected systems. To avoid them, I link campaign data to product behavior, unify identifiers across tools, and ensure CRM integration so we can follow the customer journey end-to-end. The payoff is clarity: I can tell a creative team exactly which narrative unlocked user activation and which one stalled—then iterate with speed and precision.

Vibe marketing isn’t soft; it’s strategic. When we respect the craft of emotion and the discipline of measurement, we build experiences that people love and businesses depend on. If you’re ready to upgrade how you track the intangibles, Amplitude gives you the instrumentation to turn feelings into forward motion.

Inspired by this post on Amplitude – Best Practices.

November 13, 2025
Inside-Out vs Outside-In: How I Balance Both to Build Products Users Love—and CFOs Trust

Inside-out or outside-in thinking? I choose both. The strongest product strategies fuse a bold internal vision with relentless customer evidence, creating a flywheel that lifts adoption, engagement, and revenue while reducing risk.

When I lead with inside-out thinking, I articulate a clear product thesis, technical roadmap, and platform leverage. This is where we define points of parity and differentiation, sharpen our value proposition, and ensure our architecture scales. It’s disciplined, outcomes-first, and anchored in product positioning—not output checklists.

Outside-in thinking ensures that vision stays honest. I listen to customers, analyze friction in onboarding, instrument user activation, and study retention analysis to validate whether our promises translate into real user value. This is where product discovery, A/B testing, and in-app signals tell me what’s working, what needs refinement, and what we should stop doing.

In practice, I operationalize this balance through Software Experience Management. “Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.” That promise captures the core of how I align strategy with reality inside the product, not just around it.

Concretely, I combine product analytics with in-app guides and product tours to accelerate onboarding and improve user activation. I run targeted experiments to de-risk decisions, and I iterate quickly based on what users actually do—not just what they say. The result is a product-led growth engine that compounds over time.

This approach also builds trust with finance and go-to-market partners. Inside-out clarity gives us confident, sequenced bets; outside-in data provides proof that those bets pay off. When engagement expands and adoption climbs, the business case writes itself.

If you’re deciding where to start, begin with three moves: define activation events aligned to your value proposition, instrument the experience end-to-end, and ship one high-impact in-app guide to remove a known onboarding blocker. Then measure, learn, and iterate—quickly.

The truth is, great products emerge when conviction meets evidence. Inside-out sets the vision. Outside-in earns the right to scale it.

Inspired by this post on Pendo – Perspectives.

November 12, 2025
5 Costly UX Research Pitfalls I See Often—and How AI + Qual Insights Prevent Them

In product reviews and roadmap debates at HighLevel, I come back to a simple truth: great products start with great user research—but even seasoned teams fall into the same traps. After leading product discovery across empowered product teams and product trios, I’ve learned that a few avoidable mistakes consistently derail speed, quality, and outcomes.

Learn how to avoid the top five UX research pitfalls. Discover how AI and qualitative insights can help teams uncover the why behind user behavior.

The “why” behind user behavior is where durable growth lives. When we pair qualitative insights with analytics and a clear AI Strategy, we don’t just validate a solution—we de-risk the roadmap, improve user activation, and increase retention. Here are the five pitfalls I watch for and how I coach teams to avoid them.

Pitfall 1: Treating opinions as insights. Early in my career, I mistook strong stakeholder opinions for customer truth. Now I insist on a clear research question, a decision we will make with the evidence, and a hypothesis we’re trying to falsify. A/B testing is great for measuring impact when you’ve defined minimum detectable effect (MDE), but discovery research demands explicit learning goals and unbiased inputs.

How to avoid it: Write the decision statement first (“We will proceed with X if we learn Y”), then design the research. Keep a visible decision log so insights connect directly to product roadmapping and sprint planning, not to the loudest opinion in the room.

Pitfall 2: Leading questions and flawed methods. I still see interview guides that telegraph the desired answer. This corrupts the signal. Instead, I push teams to pilot guides with a product trio, remove solution language, and focus on behaviors. We complement interviews with in-app guides, targeted surveys, and session reviews using tools like Pendo and Intercom to capture moments of friction in-context.

How to avoid it: Ask neutral, behavior-first questions (“Tell me about the last time you…”) and validate with artifacts (screenshots, workflows). Pilot every guide with a colleague, then refine for clarity and neutrality.

Pitfall 3: Over-indexing on quantitative data and ignoring the why. Amplitude analytics and retention analysis tell me what happened; they rarely tell me why it happened. When teams chase dashboards without pairing them with qualitative interviews, we optimize for surface-level metrics and miss underlying jobs, anxieties, and unmet needs.

How to avoid it: Pair funnels and cohorts with a short round of qualitative interviews. Use Generative AI to summarize transcripts, cluster themes, and highlight contradictions, then validate themes against Amplitude analytics and CRM integration data. The synthesis is where insight emerges.

Pitfall 4: Recruiting bias—talking only to superfans or the most vocal detractors. If we only hear from power users, we build for edge cases; if we only hear complaints, we over-index on blockers. The result is a lopsided roadmap that misses mainstream value.

How to avoid it: Recruit across segments—new users, churned users, evaluators who never converted, and adjacent personas. Balance the sample and document who you didn’t talk to. For sensitive segments, lean on privacy-by-design practices and data governance so participants feel safe sharing candid feedback.

Pitfall 5: Weak synthesis and no path to action. Research often ends with a beautiful report that gathers dust. Insights must translate into choices: what we will do, what we will not do, and what we must learn next. Without this, research slows delivery without improving outcomes.

How to avoid it: Convert findings into atomic insights with evidence, confidence, and impact. Tie each insight to outcomes vs output OKRs, then schedule a decision review with the product trio. If you can’t articulate the decision, you haven’t finished the research.

How I use AI without losing the plot: I rely on LLMs for product managers to speed the busywork, not to replace judgment. Gen AI helps me transcribe, tag, and cluster themes; extract Jobs to Be Done; detect hesitation and sentiment; and draft UX writing variants for follow-up surveys. With a ChatGPT connector or similar tools, I can map qualitative themes to Amplitude analytics events and Pendo paths, revealing the narrative behind the numbers.

Guardrails matter: I apply AI risk management and privacy-by-design principles—no sensitive data in prompts, clear consent, and human-in-the-loop validation. AI is a force multiplier when the prompts are grounded in a solid research plan and the outputs feed a real decision.

A quick checklist I share with teams: define the decision and hypothesis; recruit a balanced sample; use neutral, behavior-first questions; triangulate quant with qual; synthesize into atomic insights; and link every insight to a concrete action or OKR. Do this, and you compress time-to-learning without sacrificing rigor.

When we respect the craft of research and thoughtfully apply AI, we consistently uncover the why behind user behavior—and build products that users adopt, love, and keep. That’s the fastest path to product-led growth and durable differentiation.

Inspired by this post on Amplitude – Perspectives.

November 11, 2025
From Sketch to Clickable Demo: My AI Prototyping Playbook to Build Apps in Hours

I’ve spent much of my career compressing the distance between a napkin sketch and something real customers can touch. At HighLevel, my product teams use generative AI to validate ideas faster, reduce risk earlier, and win stakeholder trust with evidence instead of slides. The goal isn’t to be flashy—it’s to be precise, testable, and repeatable.

Today, you can build it before you pitch it. AI prototyping can turn ideas into clickable demos in hours. Here are some tools to try and steps to follow.

I start every AI prototyping sprint by sharpening the problem statement and the outcome we care about. That means being explicit about the target user, jobs-to-be-done, and the riskiest assumptions. I define a minimum detectable effect (MDE) and tie it to outcomes vs output OKRs so everyone aligns on what “good” looks like before we touch a tool.

From there, I move from sketch to interface. I capture a rough flow (whiteboard, tablet, or even paper) and generate UI variations with my AI product toolbox—tools that translate structure into components and screens. I’ll iterate on information hierarchy and copy until the narrative supports the core job, borrowing techniques from UX writing. For product managers leaning into LLMs for product managers, this phase is about speed to feedback, not perfection.

Next, I wire data and logic. I connect a lightweight backend or spreadsheet, stitch in a CRM integration if needed, and add LLM calls through a ChatGPT connector or Claude Code. If the concept benefits from multi-step autonomy, I introduce agentic AI to orchestrate tasks across APIs. CustomGPT workflows help me encapsulate business rules so the demo behaves consistently in user paths we care about.

Governance is not optional at this stage. I apply privacy-by-design defaults, document data governance decisions, and run a quick AI risk management pass: input validation, prompt safety, rate limits, and fallback responses. This keeps the prototype credible and prevents false positives from polluting stakeholder perception.

With a click-through in hand, I instrument the experience so learning compounds. I drop in Amplitude analytics to track activation, task completion, and drop-off, and set up simple A/B testing when there’s a meaningful design or copy choice. This makes the prototype a learning vehicle, not just a demo.

Then I get it in front of users—fast. Five targeted conversations will beat fifty internal opinions. I run structured product discovery interviews, observe time-to-value, and capture objections. This is where empowered product teams shine: we make changes in real time, re-run the flow, and document what moves the needle for product-led growth.

When speed matters, I use a four-hour cadence: Hour 1 for problem framing and MDE; Hour 2 for sketch-to-UI generation; Hour 3 for data wiring and AI logic; Hour 4 for instrumentation and user walkthroughs. By the end, we have a clickable demo, preliminary analytics, and a clear decision on whether to advance, pivot, or park.

Finally, I translate insights into a concise artifact: the hypothesis we tested, the signal we observed, the trade-offs we made, and the next sprint plan for product roadmapping and sprint planning. The point is not to be right on the first try; it’s to learn precisely, cheaply, and quickly enough to invest with conviction.

If you adopt this approach, you’ll find that stakeholder management becomes easier, team energy rises, and your roadmap earns credibility. Build it before you pitch it, and let real interactions—not wishful thinking—do the heavy lifting.

Inspired by this post on Product School.

November 10, 2025
My Product Positioning Playbook: Craft Unforgettable Messaging That Wins Markets and Endures

Every market-winning product I’ve helped build started with a positioning statement that was clear, defensible, and memorable. When I lead new initiatives at HighLevel, Inc., I treat positioning as a product decision—because it sets the guardrails for what we prioritize, how we execute, and how we tell the story across the entire go-to-market engine.

Your product positioning statement decides if you stand the test of time. Learn how other expert products do it and how to write one that sticks.

At its core, a positioning statement is the sharpest articulation of who we serve, the problem we solve, the category we compete in, the value proposition we deliver, and why we win. It is not a tagline or a pitch deck sentence; it’s the decision calculus that aligns product, marketing, sales, and customer success so we can move fast in one direction.

Here’s the simple template I use and coach teams on: For [target customer/segment] who [urgent need or job-to-be-done], [product name] is a [category or frame of reference] that [core value proposition]. Unlike [primary alternative or status quo], it [competitive differentiation and reasons to believe]. When this fits, everything from roadmaps to demos becomes easier—and conversions tend to follow.

Start with the target segment. Be precise about who you are for. I triangulate with retention analysis and behavioral data (e.g., Amplitude analytics) to find the cohorts that activate quickly, retain well, and expand. If you cannot name the segment in one line, you’ll struggle to land positioning anywhere else.

Next, define the customer outcome. Tie the promise to measurable “outcomes vs output OKRs.” Customers buy progress, not features. State the job-to-be-done in their language and anchor it to a business result they already track.

Choose your category and points of parity. Category is a cognitive shortcut; it tells buyers where you sit on their mental map. Points of parity are table stakes you must match to be considered. If you skip parity, you look incomplete; if you skip category, you look confusing.

Then sharpen your competitive differentiation and value proposition. What do you do uniquely well that competitors can’t easily copy? Back it up with reasons to believe—proof points like speed-to-value, measurable ROI, data governance, or privacy-by-design and cybersecurity commitments. Credibility turns claims into confidence.

Validate the statement through rigorous A/B testing. I pressure-test the language across landing pages, onboarding flows, in-app guides, sales call talk tracks, and nurture sequences. Tools like Pendo, Intercom, and HubSpot make it easy to instrument message experiments and see what actually moves activation, conversion, and expansion.

Operationalize the winning statement across go-to-market strategy and product-led growth motions. Bake it into onboarding, product tours, pricing pages, and demo narratives. A strong positioning statement should shape prioritization in the roadmap as much as it shapes the headline on your website.

Beware common pitfalls. Don’t confuse vibe marketing for positioning. Avoid vague superlatives that any competitor could claim. Don’t aim for universal appeal; specificity sells. And never let the statement drift—revisit it after major launches, new segments, or shifts in competitive dynamics.

Here’s an example using the template: For revenue teams at mid-market SaaS companies who need faster, more predictable pipeline creation, SignalFlow is a unified analytics platform that turns product usage signals into qualified opportunities. Unlike generic CRMs and static lead scoring, it surfaces intent in real time and automates outreach, improving conversion by 22% within 30 days.

If your team debates features more than outcomes, it’s time to revisit your positioning. In my experience, one crisp sentence can unlock alignment, accelerate execution, and make your message stick. Write it, test it, and make it the north star for every decision you ship.

Inspired by this post on Product School.

November 5, 2025
The Product Playbook: Measuring Agent Performance with Pendo and Agent Analytics to Drive ROI

I treat agent performance analytics as a strategic product lever, not a back-office metric. When I combine Pendo’s product signals with Agent Analytics from our support systems, I get a unified view of where users struggle, how agents intervene, and which in-app experiences accelerate resolution. That visibility lets my team drive product-led growth and improve customer experience while lowering support costs.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practice, I build a clear scorecard that blends both product and support KPIs: first response time, resolution rate, first contact resolution, CSAT, containment/deflection rate, average handle time, ticket volume per active account, onboarding completion, user activation, and time-to-value. This balanced view ensures we reward not just speed, but durable outcomes that reduce repeat contacts and improve retention.

To make the data actionable, we connect our CRM integration, ticketing events, and Pendo product analytics in a unified analytics platform. That gives me cohort-level clarity—who needed help, what they were doing before opening a ticket, how agents responded, and whether users stayed engaged afterward. With clean instrumentation and consistent taxonomies, Agent Analytics becomes a reliable operating system for both product and support leadership.

I then use in-app guides, tooltips, and product tours to proactively address the top friction points that drive ticket volume. Through A/B testing, we compare cohorts exposed to guided workflows versus control groups, measuring deflection, faster task completion, and downstream conversion. When a guide meaningfully reduces tickets for a given workflow, we promote it from experiment to standard onboarding, and we feed those learnings back into our roadmap.

The real unlock comes from tying outcomes to business impact. I track how improvements in resolution quality and self-serve adoption influence expansion revenue, support cost per account, and risk signals like churn propensity. Retention analysis helps us validate whether reduced friction and better agent coaching translate into sustained engagement and healthier accounts.

Operationally, Agent Analytics helps me coach teams with precision. I spotlight high-performing behaviors, identify knowledge gaps, and standardize winning playbooks directly in the product via in-app guidance. This approach empowers agents, shortens onboarding for new hires, and keeps our best practices current as the product evolves.

None of this works without trust. We apply privacy-by-design principles and strong data governance, ensuring that analytics, coaching, and automation respect user consent and data minimization standards. With that foundation, we can scale confidently—experiment faster, learn from every interaction, and continuously improve the software experience.

If you’re getting started, begin by baselining your agent and product KPIs, ship one high-impact guide to deflect a top ticket driver, and review results weekly. Within a quarter, you’ll have a repeatable loop: diagnose friction, test an in-app solution, measure deflection and satisfaction, and reinvest the gains into the next set of improvements.

Inspired by this post on Pendo – Best Practices.

November 4, 2025
Build a Product Messaging Framework That Converts: Clarity, Consistency, Customer Connection

I’ve learned the hard way that features don’t win on their own—clear, consistent messaging does. When our teams at HighLevel rally around a single product messaging framework, we move faster, tell one story, and connect with customers in a way that actually converts. The right framework doesn’t just make marketing sharper; it aligns product, sales, and customer success on what we promise, why it matters, and how we prove it.

When I say “product messaging framework,” I mean a structured system that defines who we serve, the problems we solve, the outcomes we enable, and the value proposition that sets us apart. It includes points of parity that establish table stakes, differentiation that creates competitive separation, and proof points that make our claims credible. It maps features to benefits, organizes a messaging hierarchy from company to product to feature, and guides voice, tone, and lexicon so UX writing and go-to-market strategy stay consistent across channels.

Why does this matter? Because clarity reduces friction for buyers, consistency builds trust, and customer connection drives conversion and retention. A strong framework accelerates product discovery, strengthens product positioning, and improves onboarding and user activation. It also makes product-led growth repeatable by ensuring every touchpoint—from website to in-app guides—reinforces the same value proposition.

Here’s how I build a framework that stands up in the real world. I start with customer research and win/loss analysis to anchor on the ideal customer profile and jobs-to-be-done. I craft a positioning statement that articulates the target, problem, category, differentiation, and payoff. Then I define value pillars, each with concrete reasons to believe—customer quotes, data, and feature proof. I document points of parity and differentiation, map features to benefits and outcomes, and codify voice and terminology to keep UX writing tight. Finally, I build a messaging hierarchy (company, product, feature, segment) and an objection-handling guide so sales and support are equipped to respond consistently.

A simple litmus test keeps me honest: can a salesperson deliver a crisp elevator pitch, can a PM write a release note, and can a designer craft an in-app tooltip—all from the same source of truth? If yes, the framework is doing its job. If not, I iterate until the story is simple, believable, and memorable.

Operationalizing the framework is where impact compounds. I enable product trios and go-to-market teams with talk tracks, one-pagers, narrative decks, and a living glossary. I translate the framework into site copy, product tours, onboarding flows, and help content so customers experience the same story everywhere. I also thread it into product roadmapping and sprint planning to keep prioritization aligned with the core value proposition.

I measure what matters and refine relentlessly. I use A/B testing to validate headlines and calls to action, monitor activation and conversion across segments, and review retention analysis to see which value pillars correlate with long-term use. Feedback loops from sales calls, support tickets, and customer interviews feed back into the framework so it evolves with the market.

There are predictable pitfalls I try to avoid. Going feature-first instead of outcome-first makes messaging forgettable. Overselling differentiation without points of parity undermines credibility. Spreading across too many personas dilutes signal. And inconsistent tone across channels confuses buyers. A disciplined framework helps prevent all of these.

Treat your product messaging framework as a living system, not a slide. Revisit it when the market shifts, when your roadmap unlocks new value, or when your go-to-market strategy evolves. The payoff is real: tighter alignment, sharper positioning, faster execution, and a customer story that consistently earns attention—and conversion.

Inspired by this post on Product School.

November 3, 2025
Impact Analysis Mastery: Proven Steps to Predict, Measure, and Maximize Product Outcomes

When I think about the difference between a roadmap that moves the business and one that simply ships output, impact analysis is the habit that changes everything. It gives me and my product trios a disciplined way to forecast value, align stakeholders, and de-risk bets before a single sprint starts. Over the years, I’ve seen great ideas fail not because they were bad, but because we couldn’t articulate, test, and track their true impact. That’s the problem impact analysis solves.

Impact analysis, in practice, is a structured method for predicting how a proposed change will influence user behavior and business outcomes—and then validating those predictions with data. Uncover what impact analysis is, why it matters, and how to do it with proven methods and clear steps for product teams. When done well, it translates strategy into evidence-backed choices that strengthen our value proposition and accelerate product-led growth.

I use impact analysis at three key moments: during product discovery to vet opportunities, in product roadmapping and sprint planning to prioritize, and post-launch to confirm that outcomes beat expectations. It is equally useful for net-new features, UX improvements, pricing changes, and even enablement like in-app guides or product tours.

Step 1: Define the outcome with precision. I anchor every proposal to outcomes vs output OKRs, choose one primary success metric, and record the current baseline. If we plan to experiment, I estimate the minimum detectable effect (MDE) to ensure our A/B testing can actually validate the expected lift. This protects us from investing in ideas that are too small to measure or too broad to manage.

Step 2: Map the causal chain. I translate the idea into a simple impact map: feature change → user behavior (activation, frequency, conversion, retention) → business outcome (revenue, cost, risk, satisfaction). This clarifies what must change in user behavior and why users would care—forcing us to revisit our value proposition if the link feels thin.

Step 3: Size the upside and reach. I estimate who will be exposed (reach), how often (frequency), and the expected behavior change (conversion delta). I complement this with RICE (reach, impact, confidence, effort) or cost of delay to compare options. The goal isn’t perfect math; it’s consistent, transparent assumptions that we can pressure test with data.

Step 4: Evaluate risk, complexity, and dependencies. I assess technical effort, privacy-by-design considerations, data governance needs, and cross-team sequencing. This is where stakeholder management becomes essential—aligning Engineering, Design, GTM, and Security early so we don’t discover hidden blockers mid-sprint.

Step 5: Design the evidence plan. For changes where causality matters, I prefer A/B testing with the right MDE and guardrail metrics. I instrument events and set up dashboards in a unified analytics platform (Amplitude analytics, Pendo, or a homegrown stack) so we can monitor leading indicators quickly. If experiments aren’t feasible, I use sequential rollouts, synthetic controls, or pre-post analyses with clear caveats.

Step 6: Communicate the decision. I share a one-page impact brief that summarizes objectives, hypotheses, metric choices, expected lifts, risks, and the test plan. This reduces debate time, improves stakeholder trust, and enables empowered product teams to move faster with clarity.

Step 7: Ship, monitor, and learn. After launch, I track leading indicators within days and validate lagging outcomes over weeks. I run retention analysis and cohort reviews to confirm that behavior change sticks, and I write a short learning memo—especially when we miss—so future bets get sharper.

On a recent initiative, our team debated whether to build a new onboarding flow or invest in targeted in-app guides. The impact analysis showed the guide approach would reach 3x more users in the next quarter, require half the effort, and be easier to A/B test end-to-end. We shipped the guides, saw a measurable lift in activation, and then recycled those insights to inform the broader onboarding redesign. The analysis didn’t just pick a winner—it created a faster path to compounding outcomes.

Common pitfalls I watch for: chasing vanity metrics, assuming linear impact at scale, ignoring confidence and variance, and skipping instrumentation. Another trap is treating impact analysis as a heavyweight doc—keep it lightweight, comparable across initiatives, and tightly tied to decision-making.

My lightweight template: one sentence on the desired outcome and OKR; a causal chain with the key behavior change; a simple sizing with reach, impact, and confidence; risk and dependency notes; the experimentation plan; and the decision. If we can’t write that in one page, we probably don’t understand the bet well enough to pursue it yet.

The next time you review your roadmap, pick your top three bets and run this playbook. You’ll sharpen your prioritization, increase stakeholder confidence, and give your team a clear line of sight from product discovery to measurable outcomes. That’s how we build momentum, quarter after quarter.

Inspired by this post on Product School.

November 3, 2025
Mastering AI Evals: The Essential Product Manager Skill to Ship Safer, Smarter AI

In every AI-powered product I ship, evaluation is the difference between a compelling demo and a dependable customer experience. AI evaluation isn’t a nice-to-have; it’s a core product management competency that shapes quality, safety, and business outcomes from the first prototype to scale.

When I talk about AI evaluation, I mean a disciplined, repeatable way to measure model behavior across quality, safety, reliability, latency, and cost. Gen AI has changed the cadence of product decisions—models evolve weekly, prompts drift under real-world load, and edge cases multiply. Without rigorous evals, we risk shipping unpredictability.

My goal in this piece is simple: “Dive deep into AI evals, why they matter for PMs today, and how to master them with clear steps, examples, and best practices.” If you’re leading product strategy for LLMs, agentic AI, or applied AI features, this is the playbook I rely on.

Why this matters now: customers don’t judge AI by benchmarks, they judge by trust—did it help me, was it safe, was it fast? Strong AI evals let me set outcomes vs output OKRs, quantify risk, and make transparent trade-offs between accuracy, latency, and cost. They also give engineering and design clear guardrails to move fast without breaking user trust.

Step 1: Define the product problem and success metrics. I start by tying AI metrics to business outcomes—resolution rate, deflection rate, revenue lift, time-to-value—and include model-centric measures like hallucination rate, harmful content rate, latency, and token cost. This keeps experiments anchored to impact, not just model scores.

Step 2: Build a high-signal golden dataset. I curate real, anonymized user prompts from discovery and support channels, then add adversarial and long-tail cases. For generative tasks, I create rubric-based criteria for correctness, helpfulness, tone, and safety. This dataset becomes my regression suite as prompts, RAG pipelines, or models change.

Step 3: Choose the right evaluation methods. I combine deterministic unit tests for rules with LLM-as-judge scoring, pairwise preference tests for prompt variants, human review for critical flows, and red teaming for safety. I also apply privacy-by-design and strong data governance to ensure eval data handling meets compliance and customer expectations.

Step 4: Operationalize with CI/CD. Evals run automatically on every prompt, retrieval, or model update, with pass/fail gates and alerting. I track results in a unified analytics platform so product, engineering, and go-to-market teams see the same truth. If a change regresses key thresholds, we pause rollout or roll back.

Step 5: Optimize the cost–quality–latency triangle. Real products live within constraints. I analyze token budgets, caching strategies, model selection (e.g., small for classification, larger for complex generation), prompt structure, retrieval quality, and function-calling patterns. For agentic AI, I evaluate tool-use correctness and task completion reliability, not just text quality.

Step 6: Close the loop with experimentation. Offline evals get me confidence; online A/B testing validates business impact. I design tests with a clear minimum detectable effect (MDE), guard for novelty bias, and instrument activation, retention, and satisfaction in Amplitude or Pendo. Agent analytics help me pinpoint where users succeed or get stuck.

Step 7: Govern responsibly. I maintain model cards, decision logs, and incident playbooks. For customer-facing assistants, I gate risky actions, log explanations, and add human-in-the-loop escalation. AI risk management isn’t bureaucracy—it’s how we earn trust at scale.

A concrete example: building a customer support assistant. My success metrics include deflection rate, first-contact resolution, median response latency, and safe action rate. The golden dataset blends common queries, billing edge cases, account-specific retrieval checks, and adversarial prompts. Evals measure factuality against a knowledge base, tone alignment with brand guidelines, and safe tool use for CRM integration. Only after passing offline gates do we A/B test deflection and CSAT in production.

Common pitfalls I watch for: overfitting prompts to a tiny test set, relying solely on LLM-as-judge without human calibration, skipping safety tests when latency rises, and treating evaluations as a one-time launch task. The antidote is simple—regularly refresh datasets, diversify eval methods, and wire evals into the same release discipline as any core feature.

The payoff is compounding. With strong AI evals, we ship confidently, reduce incident rates, accelerate iteration, and communicate trade-offs clearly to stakeholders. More importantly, we build products customers trust—because quality isn’t a promise, it’s a practice we can measure every day.

Inspired by this post on Product School.

November 3, 2025
15 Must-Track Customer Retention Metrics to Crush Churn and Accelerate Sustainable Growth

I obsess over retention because it tells me the truth about product-market fit, value delivery, and revenue durability. In my role leading product strategy at HighLevel, I’ve learned that sustainable growth comes less from adding users and more from keeping the right ones engaged, successful, and expanding. The fastest way to get there is through a disciplined view of the right customer retention metrics.

Struggling to keep users? These customer retention metrics reveal what’s working, what’s not, and where to focus to reduce churn.

When I assess a product’s health, I look for a clean story across acquisition, activation, engagement, and expansion—then I validate that story against revenue outcomes. If those lines don’t reconcile, churn is coming. That’s why I track a core set of signals that expose value gaps early, guide product-led growth, and align go-to-market with actual customer outcomes.

Here are the 15 signals I rely on to diagnose retention risk and prioritize roadmaps: logo churn rate, gross revenue retention (GRR), net revenue retention (NRR), cohort retention by signup month, activation rate, time-to-value (TTV), feature adoption rate, DAU/WAU/MAU and stickiness (DAU/MAU), session frequency and duration, expansion revenue rate, contraction/downgrade rate, customer lifetime value (CLV), onboarding completion rate, customer health score, and support tickets per account with time to resolution. Together, these metrics show whether customers realize value quickly, keep finding more value over time, and are willing to grow with the product.

Here’s how I use them in practice. If activation rate or time-to-value slips, I invest in onboarding clarity, in-app guides, and product tours to remove friction and accelerate first success. If GRR weakens, I re-examine renewal messaging, pricing fairness, and critical feature gaps. If NRR stalls, I revisit packaging, discovery-driven upsell paths, and the expansion moments that naturally occur after users unlock initial value.

A unified analytics platform connecting product usage, lifecycle events, and CRM integration is essential. I pair cohort analysis in Amplitude analytics with qualitative insights from Intercom, then use Pendo to instrument in-app nudges and measure feature adoption lift. A/B testing helps me validate which interventions move the metrics that matter, not just vanity engagement.

Cadence matters. I review leading indicators weekly (activation, TTV, feature adoption), lagging indicators monthly (GRR, NRR, CLV), and cohort retention every quarter to ensure improvements compound. This rhythm keeps teams aligned on outcomes vs output and focuses energy where it reduces churn fastest.

If you adopt only one habit, make it this: tie every roadmap bet to a specific movement in these retention metrics, then measure relentlessly. When we do this well, our product doesn’t just acquire users; it earns loyal advocates—and that’s the most efficient growth engine there is.

Inspired by this post on Product School.

November 3, 2025
Build Customer Feedback Loops That Actually Drive Growth and Get Your Product Unstuck

What if your customer feedback loop is the reason you're stuck? Learn how to build one that fuels real growth and changes your product for the better.

I’ve seen talented teams spin for months because their customer feedback loop was noisy, slow, or misaligned with outcomes. The result is predictable: roadmaps packed with output, not impact. When we design feedback loops that are intentional, instrumented, and closed with customers, the product starts compounding value—and the business moves from reactive to product-led growth.

My definition of a strong customer feedback loop is simple: capture the right signals, synthesize them quickly, prioritize against outcomes, experiment to de-risk, and close the loop visibly with customers. If any link is weak, the whole system underperforms. More feedback isn’t better—better feedback is better.

Start with who you listen to. Segment feedback by persona, account tier, lifecycle stage, and “jobs to be done.” A founder’s feature request, a new user’s onboarding friction, and a power user’s edge case should not be weighted the same. This is the foundation of credible product discovery.

Instrument your product so qualitative and quantitative signals reinforce each other. I rely on funnel and cohort views in Amplitude analytics to see where activation or retention breaks, then layer in interviews, support tickets, and community threads for context. When telemetry and narrative align, the signal gets unmistakable.

Capture feedback where the user is. In-app guides and lightweight surveys via Pendo or Intercom surface timely prompts during key journeys (onboarding, activation, adoption, renewal). Pair those with structured inputs from sales notes and customer success reviews so you don’t bias toward only the most vocal users.

Standardize how you synthesize. Tag every item by problem statement, persona, job, and affected step in the journey. Roll these up into weekly themes your product trios can act on. The discipline here turns anecdotes into addressable opportunities.

Prioritize against outcomes, not volume. If your OKRs are outcomes vs output OKRs, tie each opportunity to a measurable product outcome like user activation rate, adoption depth, conversion, or retention. A thousand upvotes mean less than a clear path to move a core metric.

De-risk with evidence, not opinion. Frame hypotheses, define success metrics, and run A/B testing with a clear minimum detectable effect. Guardrail metrics matter—never trade a short-term click lift for a long-term retention drop. Experiments should accelerate learning, not justify pet projects.

Fold learning into product roadmapping and sprint planning. I expect prioritized feedback themes to map to roadmap bets with clear owners, milestones, and expected impact. This is how product management leadership signals what we will do—and what we will not do—based on evidence.

Close the loop, every time. Tell customers what changed because of their input—release notes, in-app messages, CSM follow-ups, or community updates. When people see their voice shaping the product, engagement and loyalty rise. This is also how you earn higher-quality feedback next time.

Set a cadence and governance that sticks. A weekly Voice of Customer review for the product trio, a monthly synthesis for cross-functional stakeholders, and a quarterly lookback tying changes to retention analysis creates organizational memory. Feedback isn’t a meeting; it’s a muscle.

Beware common failure modes. Don’t overweight loud accounts, confuse feature requests with problems, or ship one-off fixes that fragment your value proposition. Avoid vanity dashboards that show activity without decision-making power. If your loop doesn’t routinely change priorities, it isn’t a loop—it’s a suggestion box.

If you’re starting from scratch, keep it simple: define your core outcomes, instrument the top journeys, establish two capture channels (in-app and human-led), create a shared taxonomy, and commit to a weekly synthesis ritual. In a few cycles, you’ll see cleaner insights, tighter bets, and faster learning.

Done right, customer feedback loops are a competitive advantage. They sharpen product discovery, accelerate user activation, and compound retention—exactly what a modern, product-led organization needs to grow with confidence.

Inspired by this post on Product School.

November 3, 2025