Month: November 2025

The Product Playbook: Measuring Agent Performance with Pendo and Agent Analytics to Drive ROI

I treat agent performance analytics as a strategic product lever, not a back-office metric. When I combine Pendo’s product signals with Agent Analytics from our support systems, I get a unified view of where users struggle, how agents intervene, and which in-app experiences accelerate resolution. That visibility lets my team drive product-led growth and improve customer experience while lowering support costs.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practice, I build a clear scorecard that blends both product and support KPIs: first response time, resolution rate, first contact resolution, CSAT, containment/deflection rate, average handle time, ticket volume per active account, onboarding completion, user activation, and time-to-value. This balanced view ensures we reward not just speed, but durable outcomes that reduce repeat contacts and improve retention.

To make the data actionable, we connect our CRM integration, ticketing events, and Pendo product analytics in a unified analytics platform. That gives me cohort-level clarity—who needed help, what they were doing before opening a ticket, how agents responded, and whether users stayed engaged afterward. With clean instrumentation and consistent taxonomies, Agent Analytics becomes a reliable operating system for both product and support leadership.

I then use in-app guides, tooltips, and product tours to proactively address the top friction points that drive ticket volume. Through A/B testing, we compare cohorts exposed to guided workflows versus control groups, measuring deflection, faster task completion, and downstream conversion. When a guide meaningfully reduces tickets for a given workflow, we promote it from experiment to standard onboarding, and we feed those learnings back into our roadmap.

The real unlock comes from tying outcomes to business impact. I track how improvements in resolution quality and self-serve adoption influence expansion revenue, support cost per account, and risk signals like churn propensity. Retention analysis helps us validate whether reduced friction and better agent coaching translate into sustained engagement and healthier accounts.

Operationally, Agent Analytics helps me coach teams with precision. I spotlight high-performing behaviors, identify knowledge gaps, and standardize winning playbooks directly in the product via in-app guidance. This approach empowers agents, shortens onboarding for new hires, and keeps our best practices current as the product evolves.

None of this works without trust. We apply privacy-by-design principles and strong data governance, ensuring that analytics, coaching, and automation respect user consent and data minimization standards. With that foundation, we can scale confidently—experiment faster, learn from every interaction, and continuously improve the software experience.

If you’re getting started, begin by baselining your agent and product KPIs, ship one high-impact guide to deflect a top ticket driver, and review results weekly. Within a quarter, you’ll have a repeatable loop: diagnose friction, test an in-app solution, measure deflection and satisfaction, and reinvest the gains into the next set of improvements.

Inspired by this post on Pendo – Best Practices.

November 4, 2025
AI at Home, Impact at Work: Experiments That Supercharged My Product Leadership

I recently tuned into an insightful All Things Product episode featuring Teresa Torres and Petra Wille on how experimenting with AI in everyday life sharpens how we build AI-powered products at work. The core premise resonated deeply with my AI Strategy: low-stakes, personal experiments accelerate confidence, clarify limitations, and build an AI product toolbox we can bring into the office with rigor.

If you want to dive in, you can listen on Spotify or Apple Podcasts. I found the conversation especially relevant for product trios and anyone shaping LLMs for product managers in high-stakes environments.

The idea is simple but powerful: when I prototype with AI at home—where the stakes are low—I learn faster, make safer mistakes, and internalize critical product patterns. Over time, those patterns transfer directly to work: tighter context management, sharper bias awareness, clearer human-in-the-loop guardrails, and a more nuanced view of when to use AI as a thought partner versus when to consider agentic AI.

In my own practice, I’ve mirrored many of the scenarios discussed: using ChatGPT by OpenAI to plan meals, analyze public data sets like school budgets, and even sanity-check real estate evaluations. These seemingly mundane tasks are fertile ground for learning about context window limits, hallucination (artificial intelligence), AI bias, and privacy-by-design trade-offs. Each experiment helps me craft better prompts, structure data for clarity, and decide when a human review step is non-negotiable—core habits for AI risk management.

At work, I treat AI as a thought partner for writing, research synthesis, and contract review. I also explore when and how to responsibly evolve toward agentic AI for repeatable workflows. The distinction matters: a thought partner augments judgment; an agent automates execution. Building the right scaffolding—data governance, auditability, constraints, and escalation paths—ensures we unlock speed without compromising safety.

Three lines from the episode stayed with me: “I’m trying to write things that only I can write — that’s my guiding writing light right now.” — Teresa. “The more we use AI, the more we learn what it’s good at, what it’s not good at, and where context becomes a limitation.” — Teresa. “It’s a safer playground — we can build our toolbox at home before bringing those lessons to work.” — Petra. These are practical north stars for product management leadership in the GenAI era.

For anyone getting started, here’s what worked for me: begin with “low-stakes” personal experiments, write down your prompts and outcomes, and reflect on failure modes. Treat each activity as product discovery: What problem am I solving? What outcome matters? What data and context does the model need? Which decisions must stay human-in-the-loop? This discipline builds an AI product toolbox you can confidently apply to real customer problems.

I also keep a running toolkit of references and tools that inform my practice: Context window as a concept helps me size and sequence information. Visual and video tools like Midjourney and Sora expand how I think about multimodal experiences. I rotate between Claude by Anthropic and ChatGPT by OpenAI depending on task fit, and I’ve used Claude Code when I need structured assistance with code review. For knowledge capture and workflow, Readwise and Ghost help me structure insights and ship content.

If you want more structured learning paths, I found Josh Seiden’s Learn AI With Me, A 30-Day Sprint to be a practical primer, and the broader community conversation at Product at Heart Conference is invaluable. For a deeper grounding in risk, I recommend reviewing topics like Hallucination (artificial intelligence), AI bias, and Agentic AI—and revisiting the complementary episode, Context is King.

I’d love to hear how you’re experimenting: Where have you seen AI meaningfully reduce toil? Where does it still struggle? How are you balancing creativity, data safety, and compliance as you scale? Drop a comment below and let’s compare notes—especially on patterns that help product trios move faster without sacrificing trust.

Bottom line: start small at home, carry lessons into the office, and build with curiosity and intentionality. That’s how we level up our product discovery, sharpen our value proposition, and lead teams confidently through the GenAI transition.

Inspired by this post on Product Talk.

November 4, 2025
From Code to Roadmaps: My Proven Playbook for Engineers Becoming Product Managers

"From code commits to boardrooms. Here are real stories of software engineers who swapped bugs for roadmaps on the road to product manager." I’ve made that leap myself and helped many engineers do the same. In this piece, I share the playbook I use to guide high-potential ICs into impactful product management roles—without losing the engineering rigor that makes them special.

Engineers make exceptional product managers because we’re trained to decompose complex systems, debug ambiguity, and reason from first principles. The transition isn’t about abandoning code; it’s about expanding your scope from implementation details to customer outcomes, market context, and business impact.

The first shift is mental: move from shipping outputs to driving outcomes. Features are a means; value is the end. I anchor this change with outcomes vs output OKRs, ensuring every roadmap item ties to a measurable user or business result rather than a checklist of tickets.

Next, upskill deliberately in three areas: product discovery, product positioning, and stakeholder management. Learn to design unbiased customer interviews, synthesize patterns from qualitative and quantitative signals, and craft crisp value propositions that resonate with real segments. Then practice executive-ready communication—clear decisions, concise narratives, and no jargon crutches.

Here’s the practical, low-risk way to get PM experience without changing your title: form a product trios working group (design, engineering, product) around a real problem. Lead discovery with a weekly cadence, run lightweight experiments, and translate insights into a draft product roadmapping and sprint planning artifact. Ship small, learn fast, and narrate the learning.

Build a simple portfolio that proves product judgment. Include one-page problem briefs, discovery notes, customer quotes, prioritized opportunity trees, and a before/after roadmap snapshot. For each artifact, quantify the impact: activation lift, support ticket reduction, conversion improvement—whatever outcome your work influenced.

If you want to pivot internally, propose a 90-day experiment. Volunteer to own a well-bounded problem, commit to an outcomes dashboard, and set a weekly stakeholder update. Keep a minimal engineering contribution during the trial to de-risk the transition for your team while you demonstrate PM leverage across the squad.

If you’re interviewing externally, prepare two deep case studies: one discovery-led (how you reduced uncertainty) and one delivery-led (how you aligned stakeholders and shipped). Be explicit about trade-offs, risks you retired, metrics you moved, and lessons learned. The best signals of product sense are clarity under constraints and an ability to say “no” for good reasons.

Once you land the role, use a 30-60-90 plan. In the first 30 days, map users, workflows, metrics, and decision rhythms; in 60, run a focused discovery sprint and align on your hypothesis-led roadmap; by 90, deliver a thin slice that proves value and establishes credibility with empowered product teams. Keep your communication tight, your dashboards honest, and your customers close.

Common pitfalls: translating directly from solution space to roadmap without validating problems; equating stakeholder satisfaction with customer value; and mistaking velocity for progress. Avoid them by running small tests early, revisiting segment-specific value propositions, and anchoring trade-offs to product-market fit lessons.

If you’re standing at the edge of this transition, start where you are: choose one user pain, one measurable outcome, and one small bet. Treat it like a product: define success, experiment thoughtfully, and learn in public. The road from engineer to product manager isn’t a title change—it’s a shift in how you create value.

Inspired by this post on Product School.

November 3, 2025
Build a Product Messaging Framework That Converts: Clarity, Consistency, Customer Connection

I’ve learned the hard way that features don’t win on their own—clear, consistent messaging does. When our teams at HighLevel rally around a single product messaging framework, we move faster, tell one story, and connect with customers in a way that actually converts. The right framework doesn’t just make marketing sharper; it aligns product, sales, and customer success on what we promise, why it matters, and how we prove it.

When I say “product messaging framework,” I mean a structured system that defines who we serve, the problems we solve, the outcomes we enable, and the value proposition that sets us apart. It includes points of parity that establish table stakes, differentiation that creates competitive separation, and proof points that make our claims credible. It maps features to benefits, organizes a messaging hierarchy from company to product to feature, and guides voice, tone, and lexicon so UX writing and go-to-market strategy stay consistent across channels.

Why does this matter? Because clarity reduces friction for buyers, consistency builds trust, and customer connection drives conversion and retention. A strong framework accelerates product discovery, strengthens product positioning, and improves onboarding and user activation. It also makes product-led growth repeatable by ensuring every touchpoint—from website to in-app guides—reinforces the same value proposition.

Here’s how I build a framework that stands up in the real world. I start with customer research and win/loss analysis to anchor on the ideal customer profile and jobs-to-be-done. I craft a positioning statement that articulates the target, problem, category, differentiation, and payoff. Then I define value pillars, each with concrete reasons to believe—customer quotes, data, and feature proof. I document points of parity and differentiation, map features to benefits and outcomes, and codify voice and terminology to keep UX writing tight. Finally, I build a messaging hierarchy (company, product, feature, segment) and an objection-handling guide so sales and support are equipped to respond consistently.

A simple litmus test keeps me honest: can a salesperson deliver a crisp elevator pitch, can a PM write a release note, and can a designer craft an in-app tooltip—all from the same source of truth? If yes, the framework is doing its job. If not, I iterate until the story is simple, believable, and memorable.

Operationalizing the framework is where impact compounds. I enable product trios and go-to-market teams with talk tracks, one-pagers, narrative decks, and a living glossary. I translate the framework into site copy, product tours, onboarding flows, and help content so customers experience the same story everywhere. I also thread it into product roadmapping and sprint planning to keep prioritization aligned with the core value proposition.

I measure what matters and refine relentlessly. I use A/B testing to validate headlines and calls to action, monitor activation and conversion across segments, and review retention analysis to see which value pillars correlate with long-term use. Feedback loops from sales calls, support tickets, and customer interviews feed back into the framework so it evolves with the market.

There are predictable pitfalls I try to avoid. Going feature-first instead of outcome-first makes messaging forgettable. Overselling differentiation without points of parity undermines credibility. Spreading across too many personas dilutes signal. And inconsistent tone across channels confuses buyers. A disciplined framework helps prevent all of these.

Treat your product messaging framework as a living system, not a slide. Revisit it when the market shifts, when your roadmap unlocks new value, or when your go-to-market strategy evolves. The payoff is real: tighter alignment, sharper positioning, faster execution, and a customer story that consistently earns attention—and conversion.

Inspired by this post on Product School.

November 3, 2025
Impact Analysis Mastery: Proven Steps to Predict, Measure, and Maximize Product Outcomes

When I think about the difference between a roadmap that moves the business and one that simply ships output, impact analysis is the habit that changes everything. It gives me and my product trios a disciplined way to forecast value, align stakeholders, and de-risk bets before a single sprint starts. Over the years, I’ve seen great ideas fail not because they were bad, but because we couldn’t articulate, test, and track their true impact. That’s the problem impact analysis solves.

Impact analysis, in practice, is a structured method for predicting how a proposed change will influence user behavior and business outcomes—and then validating those predictions with data. Uncover what impact analysis is, why it matters, and how to do it with proven methods and clear steps for product teams. When done well, it translates strategy into evidence-backed choices that strengthen our value proposition and accelerate product-led growth.

I use impact analysis at three key moments: during product discovery to vet opportunities, in product roadmapping and sprint planning to prioritize, and post-launch to confirm that outcomes beat expectations. It is equally useful for net-new features, UX improvements, pricing changes, and even enablement like in-app guides or product tours.

Step 1: Define the outcome with precision. I anchor every proposal to outcomes vs output OKRs, choose one primary success metric, and record the current baseline. If we plan to experiment, I estimate the minimum detectable effect (MDE) to ensure our A/B testing can actually validate the expected lift. This protects us from investing in ideas that are too small to measure or too broad to manage.

Step 2: Map the causal chain. I translate the idea into a simple impact map: feature change → user behavior (activation, frequency, conversion, retention) → business outcome (revenue, cost, risk, satisfaction). This clarifies what must change in user behavior and why users would care—forcing us to revisit our value proposition if the link feels thin.

Step 3: Size the upside and reach. I estimate who will be exposed (reach), how often (frequency), and the expected behavior change (conversion delta). I complement this with RICE (reach, impact, confidence, effort) or cost of delay to compare options. The goal isn’t perfect math; it’s consistent, transparent assumptions that we can pressure test with data.

Step 4: Evaluate risk, complexity, and dependencies. I assess technical effort, privacy-by-design considerations, data governance needs, and cross-team sequencing. This is where stakeholder management becomes essential—aligning Engineering, Design, GTM, and Security early so we don’t discover hidden blockers mid-sprint.

Step 5: Design the evidence plan. For changes where causality matters, I prefer A/B testing with the right MDE and guardrail metrics. I instrument events and set up dashboards in a unified analytics platform (Amplitude analytics, Pendo, or a homegrown stack) so we can monitor leading indicators quickly. If experiments aren’t feasible, I use sequential rollouts, synthetic controls, or pre-post analyses with clear caveats.

Step 6: Communicate the decision. I share a one-page impact brief that summarizes objectives, hypotheses, metric choices, expected lifts, risks, and the test plan. This reduces debate time, improves stakeholder trust, and enables empowered product teams to move faster with clarity.

Step 7: Ship, monitor, and learn. After launch, I track leading indicators within days and validate lagging outcomes over weeks. I run retention analysis and cohort reviews to confirm that behavior change sticks, and I write a short learning memo—especially when we miss—so future bets get sharper.

On a recent initiative, our team debated whether to build a new onboarding flow or invest in targeted in-app guides. The impact analysis showed the guide approach would reach 3x more users in the next quarter, require half the effort, and be easier to A/B test end-to-end. We shipped the guides, saw a measurable lift in activation, and then recycled those insights to inform the broader onboarding redesign. The analysis didn’t just pick a winner—it created a faster path to compounding outcomes.

Common pitfalls I watch for: chasing vanity metrics, assuming linear impact at scale, ignoring confidence and variance, and skipping instrumentation. Another trap is treating impact analysis as a heavyweight doc—keep it lightweight, comparable across initiatives, and tightly tied to decision-making.

My lightweight template: one sentence on the desired outcome and OKR; a causal chain with the key behavior change; a simple sizing with reach, impact, and confidence; risk and dependency notes; the experimentation plan; and the decision. If we can’t write that in one page, we probably don’t understand the bet well enough to pursue it yet.

The next time you review your roadmap, pick your top three bets and run this playbook. You’ll sharpen your prioritization, increase stakeholder confidence, and give your team a clear line of sight from product discovery to measurable outcomes. That’s how we build momentum, quarter after quarter.

Inspired by this post on Product School.

November 3, 2025
Product Tree 101: The Visual Prioritization Framework I Rely on to Align Teams Fast

When my team is drowning in requests, the Product Tree is the visual tool that brings clarity and momentum. "Learn what a product tree is, how to use the product tree framework, and why it’s a powerful tool for smarter product prioritization." That’s exactly what I aim to share here—how I use it to align stakeholders, sharpen product strategy, and translate ideas into outcomes.

A product tree is a simple yet powerful metaphor for your product. The trunk represents the core value, the roots are the technical foundations and platform capabilities, the branches are product areas or themes, and the leaves are features, experiments, or opportunities. By placing ideas as leaves on the right branches—and making sure roots can actually sustain that growth—we turn a messy backlog into a coherent product roadmap.

Why do product managers swear by it? Because it forces outcomes over outputs, exposes trade-offs visually, and reveals where strategy is thin or overgrown. In one view, you see customer value, technical debt, and strategic focus—crucial for empowered product teams, product discovery, and stakeholder management. It’s also an excellent way to connect outcomes vs output OKRs to tangible delivery paths.

Here’s how I set it up. First, I define the trunk with a crisp product value proposition and the minimum set of experiences that make the product viable. This anchors everything else so we don’t mistake a shiny leaf for the core of the tree.

Next, I map branches to clearly named themes that mirror how customers perceive value—onboarding, activation, collaboration, analytics, or reliability. I keep branches aligned to outcomes to avoid feature-first thinking; this pays dividends during product roadmapping and sprint planning.

Then I add leaves: research insights, customer requests, experiments, and enabling features. I note intent (e.g., drive activation, reduce churn), expected impact, and a rough effort signal. This quickly surfaces which leaves grow the product and which are just twigs.

Finally, I draw roots—the enabling platform work and technical investments that make the branches sustainable. Performance, data governance, privacy-by-design, and scalability belong here. If the roots can’t support the canopy, the tree is at risk, and that becomes a visible, prioritizable problem rather than an invisible liability.

Once the tree is sketched, I facilitate a collaborative session with product trios and cross-functional partners. We prune low-impact leaves, cluster work by outcomes, and explicitly link branches to OKRs. In QBRs vs OKRs reviews, the tree becomes our single source of truth for trade-offs, helping stakeholders see why some requests move up and others wait.

In practice, I use the Product Tree to shape a near-term delivery plan and a longer-horizon narrative. Near term, it informs sprint planning and sequencing by ensuring the right roots land before the heavier branches. Longer term, it clarifies the growth story for product-led growth—what we’ll grow next and why it matters for customers.

A few tips from the trenches: anchor branches to customer outcomes, not internal org charts; spotlight enabling work so platform investments aren’t deprioritized; and revisit the tree after each discovery cycle to keep it fresh. The moment the tree feels lopsided, that’s your signal to rebalance bets or revisit assumptions in product discovery.

If you’re preparing for your next planning cycle, try a 60-minute Product Tree workshop. You’ll come away with a shared mental model, sharper prioritization, and a roadmap that is easy to communicate and defend—because everyone can see the product’s future taking shape right in front of them.

Inspired by this post on Product School.

November 3, 2025
Stop Shipping for the Sake of It: Master Outputs vs. Outcomes to Build Products That Win

Too many teams still celebrate what they ship rather than what they change. I’ve learned—sometimes the hard way—that the most expensive mistake in product management is confusing outputs with outcomes. Understand the key differences between output vs. outcome in product management — and how to keep your team focused on what really drives results.

Here’s how I draw the line: outputs are the features, tickets, and releases we produce; outcomes are the measurable changes in user behavior and business performance we create—activation rates, retention, expansion, and time-to-value. If an initiative doesn’t move a metric that matters, it’s output without impact. That’s how feature factories are born.

The confusion is costly because it distorts incentives. Teams optimize for velocity, story points, or deployment frequency and mistake motion for progress. Engineering excellence and DORA metrics matter, but they’re not substitutes for product outcomes. When OKRs drift into task lists, we ship more and learn less. I’ve seen ambitious roadmaps hit every delivery date and still miss the market because we didn’t change customer behavior.

To break that cycle, I anchor planning and reviews to outcome-based OKRs. A good objective might be: increase new-account user activation from 28% to 45% this quarter. The anti-pattern is: ship onboarding redesign v2. The former sets a clear behavioral target; the latter constrains creativity and locks us into a solution before discovery. This is the practical heart of outcomes vs output OKRs.

From there, I define leading indicators that predict the desired outcome—time-to-first-value, completion of core actions, day-7 retention—and instrument them early. Tools like Amplitude analytics help us see whether an experiment is unlocking behavior change or just producing activity. I also set guardrail metrics (support volume, performance, and NPS) so we don’t “succeed” by creating a new failure mode.

The delivery model matters, too. Empowered product teams—built as product trios of product, design, and engineering—own the problem and the outcome. We invest in product discovery to validate assumptions, size opportunities, and find the minimum viable change that moves the metric. A/B testing with a clear minimum detectable effect (MDE) makes our experiments faster, cheaper, and more conclusive.

Roadmaps then become strategic bets rather than feature lists. Each bet articulates the opportunity, the hypothesized solution, the expected outcome, and the evidence that would change our mind. In sprint planning, we slice increments to learn sooner, not just to deliver sooner. CI/CD accelerates shipping; outcome instrumentation accelerates learning.

Stakeholder conversations shift as well. Instead of debating which features to build, we align on the customer problem, the value proposition, and the measures of success. QBRs showcase what changed—activation, adoption, retention—not just what shipped. This is how we move from feature requests to outcome commitments and sustain product-led growth.

I’ve found that outcomes-first execution energizes teams. Clarity of purpose invites creativity, and the autonomy to experiment fuels ownership. When we celebrate behavior change over backlog burn-down, we stop playing to the roadmap and start playing to win the market.

If your team is stuck in output mode, start small: rewrite one key objective as an outcome, instrument a leading indicator, and run a scoped experiment. When the metric moves, let that win reset the culture. Momentum follows outcomes.

Inspired by this post on Product School.

November 3, 2025
Decode Why Users Do What They Do: A Proven Playbook for Customer Sentiment Analysis

I obsess over why users do what they do. When I connect the dots between behavior and emotion, product decisions get clearer, roadmaps get sharper, and outcomes improve fast. Customer sentiment analysis is the discipline that helps me bridge that gap between numbers and nuance—turning scattered feedback into a focused narrative that drives product-led growth and retention.

Want to understand the thoughts and feelings that drive user actions? This guide to customer sentiment analysis shows you how to listen and respond.

At its core, customer sentiment analysis blends quantitative signals (usage telemetry, conversion, churn) with qualitative insight (support conversations, reviews, in-app feedback) to reveal why users behave the way they do. I use it to pinpoint friction in onboarding, accelerate user activation, and reinforce the value proposition across the journey. The result is a product experience that not only performs but also resonates.

Here’s how I listen at scale. I aggregate inputs from support tickets and call transcripts, in-app feedback widgets, community posts, and social listening; I supplement them with product analytics from Amplitude analytics, guidance and event data from Pendo, and conversation and engagement patterns from Intercom. With strong CRM integration to HubSpot and a unified analytics platform, I can tie sentiment to accounts, lifecycle stages, and revenue impact—so every signal is actionable, not anecdotal.

On the analysis side, I segment feedback by journey stage (onboarding, activation, adoption, expansion, churn risk) and classify it by theme (usability, reliability, pricing, time-to-value). Gen ai and LLMs for product managers help me summarize large volumes of text, cluster topics, and score sentiment with speed, while I maintain guardrails through data governance, privacy-by-design, and clear AI risk management policies. The aim isn’t just a score—it’s a storyline I can act on.

Closing the loop is where sentiment turns into outcomes. If I see negative sentiment around first-run complexity, I streamline onboarding, add contextual product tours and in-app guides, and refine tooltip design and UX writing. I then validate improvements with A/B testing, watch minimum detectable effect (MDE) thresholds, and track movement on activation, NPS/CSAT, and early retention. This rhythm creates a durable feedback-to-feature pipeline that compounds over time.

Operationally, I run a recurring sentiment review with product trios and cross-functional leaders. We connect insights to outcomes vs output OKRs, pressure-test bets through product discovery, and prioritize work that measurably reduces friction. When sentiment and behavior point to the same problem, it moves to the top of the roadmap. When they diverge, we dig deeper before we build.

If you’re getting started, begin with the highest-value surfaces: onboarding and activation. Instrument the journey, centralize feedback, and label themes consistently. Use small, targeted experiments to address the loudest pain points, then scale what works. Over a few cycles, you’ll see clearer insights, faster decisions, and a product experience that feels intuitively “right” to your users—because it’s grounded in their words and their behavior.

Inspired by this post on Product School.

November 3, 2025
Mastering AI Evals: The Essential Product Manager Skill to Ship Safer, Smarter AI

In every AI-powered product I ship, evaluation is the difference between a compelling demo and a dependable customer experience. AI evaluation isn’t a nice-to-have; it’s a core product management competency that shapes quality, safety, and business outcomes from the first prototype to scale.

When I talk about AI evaluation, I mean a disciplined, repeatable way to measure model behavior across quality, safety, reliability, latency, and cost. Gen AI has changed the cadence of product decisions—models evolve weekly, prompts drift under real-world load, and edge cases multiply. Without rigorous evals, we risk shipping unpredictability.

My goal in this piece is simple: “Dive deep into AI evals, why they matter for PMs today, and how to master them with clear steps, examples, and best practices.” If you’re leading product strategy for LLMs, agentic AI, or applied AI features, this is the playbook I rely on.

Why this matters now: customers don’t judge AI by benchmarks, they judge by trust—did it help me, was it safe, was it fast? Strong AI evals let me set outcomes vs output OKRs, quantify risk, and make transparent trade-offs between accuracy, latency, and cost. They also give engineering and design clear guardrails to move fast without breaking user trust.

Step 1: Define the product problem and success metrics. I start by tying AI metrics to business outcomes—resolution rate, deflection rate, revenue lift, time-to-value—and include model-centric measures like hallucination rate, harmful content rate, latency, and token cost. This keeps experiments anchored to impact, not just model scores.

Step 2: Build a high-signal golden dataset. I curate real, anonymized user prompts from discovery and support channels, then add adversarial and long-tail cases. For generative tasks, I create rubric-based criteria for correctness, helpfulness, tone, and safety. This dataset becomes my regression suite as prompts, RAG pipelines, or models change.

Step 3: Choose the right evaluation methods. I combine deterministic unit tests for rules with LLM-as-judge scoring, pairwise preference tests for prompt variants, human review for critical flows, and red teaming for safety. I also apply privacy-by-design and strong data governance to ensure eval data handling meets compliance and customer expectations.

Step 4: Operationalize with CI/CD. Evals run automatically on every prompt, retrieval, or model update, with pass/fail gates and alerting. I track results in a unified analytics platform so product, engineering, and go-to-market teams see the same truth. If a change regresses key thresholds, we pause rollout or roll back.

Step 5: Optimize the cost–quality–latency triangle. Real products live within constraints. I analyze token budgets, caching strategies, model selection (e.g., small for classification, larger for complex generation), prompt structure, retrieval quality, and function-calling patterns. For agentic AI, I evaluate tool-use correctness and task completion reliability, not just text quality.

Step 6: Close the loop with experimentation. Offline evals get me confidence; online A/B testing validates business impact. I design tests with a clear minimum detectable effect (MDE), guard for novelty bias, and instrument activation, retention, and satisfaction in Amplitude or Pendo. Agent analytics help me pinpoint where users succeed or get stuck.

Step 7: Govern responsibly. I maintain model cards, decision logs, and incident playbooks. For customer-facing assistants, I gate risky actions, log explanations, and add human-in-the-loop escalation. AI risk management isn’t bureaucracy—it’s how we earn trust at scale.

A concrete example: building a customer support assistant. My success metrics include deflection rate, first-contact resolution, median response latency, and safe action rate. The golden dataset blends common queries, billing edge cases, account-specific retrieval checks, and adversarial prompts. Evals measure factuality against a knowledge base, tone alignment with brand guidelines, and safe tool use for CRM integration. Only after passing offline gates do we A/B test deflection and CSAT in production.

Common pitfalls I watch for: overfitting prompts to a tiny test set, relying solely on LLM-as-judge without human calibration, skipping safety tests when latency rises, and treating evaluations as a one-time launch task. The antidote is simple—regularly refresh datasets, diversify eval methods, and wire evals into the same release discipline as any core feature.

The payoff is compounding. With strong AI evals, we ship confidently, reduce incident rates, accelerate iteration, and communicate trade-offs clearly to stakeholders. More importantly, we build products customers trust—because quality isn’t a promise, it’s a practice we can measure every day.

Inspired by this post on Product School.

November 3, 2025
Your Ultimate ProductCon San Francisco 2025 Guide: Best Hotels, Eats & Drinks

Heading to ProductCon San Francisco 2025? I approach conference travel the same way I approach product strategy: optimize for outcomes, reduce friction, and invest in high-signal experiences. Here’s the playbook I use to choose the right hotel, find memorable meals, and make the most of every hour in the city.

For lodging, I prioritize walkability, safety, and quiet rooms so I can focus during sessions and recover at night. If you want to be steps from most venues and meetups, SoMa and the Yerba Buena corridor are ideal. InterContinental San Francisco, W San Francisco, and The Clancy (Autograph Collection) are reliable, business-friendly picks with strong Wi‑Fi and ample lobby space for impromptu one‑on‑ones. If you prefer classic energy and transit access, Union Square hotels like Hotel Nikko and The Westin St. Francis work well. For waterfront views and a calmer vibe, Hyatt Regency Embarcadero puts you by the Ferry Building with easy BART and Muni access.

My booking checklist is simple: reserve early, target a high floor away from elevators, and request early check‑in or late checkout around your session schedule. Loyalty programs often unlock better rates and quiet‑room preferences. If you need heads‑down time between talks, ask about day‑use meeting rooms or find a corner of the lobby with stable bandwidth. I also pack a compact power strip and a long USB‑C cable—two small upgrades that routinely save a day.

Coffee is the fuel of great product conversations. Near SoMa, I rotate between Blue Bottle (Mint Plaza), Sightglass (7th Street), and Philz (Front Street) for pre‑session caffeine and quick stand‑ups. If I’m on the Embarcadero side, the Ferry Building’s roasters are perfect for early starts, and morning lines move faster than you’d expect if you arrive just after opening.

For efficient lunches, I favor fast‑casual spots that can handle volume without sacrificing quality. Mixt, Souvla, Sweetgreen, Super Duper Burgers, and The Grove are dependable within a short walk of most downtown venues. When I need a higher‑signal lunch with a partner or prospect, I book a table slightly off the main corridor to avoid the rush—think Mourad for elevated Moroccan in SoMa or Boulevard along the Embarcadero for a polished, quiet conversation.

Dinner is where the best networking often happens, so I plan for atmosphere, acoustics, and a menu that works for mixed dietary needs. Kokkari Estiatorio (FiDi) excels for executive dinners. Liholiho Yacht Club is a creative, memorable choice for cross‑functional teams. Waterbar or Angler near the waterfront pair great food with views that impress visiting colleagues. For something more casual but still conversation‑friendly, Nopa or Sorella deliver consistently.

When it’s time for drinks, I think in terms of groups and goals. For panoramic views and small group catch‑ups, The View Lounge (Marriott Marquis) is a classic. For wine‑forward conversations with a quiet ambiance, Press Club near Yerba Buena works well. If you’re hosting a more energetic crew, Charmaine’s (SF Proper Hotel), Dirty Habit (Hotel Zelos), or 25 Lusk offer space, good music, and reliable service. For craft cocktails, Pacific Cocktail Haven and ABV are standouts if you don’t mind a short ride.

Transit and timing matter. From SFO or OAK, BART is often the fastest, most predictable route downtown; rideshare is convenient late at night. I walk whenever possible, but I time routes along well‑lit, busier streets and avoid sprinting between neighborhoods tight on time. Microclimates are real—bring layers, comfortable shoes, and a compact umbrella. I schedule 15‑minute buffers around key sessions to handle inevitable friend‑of‑a‑friend introductions.

If you need a professional setting for a quick working session, many hotels will extend lobby seating to guests and their visitors. For dedicated space, day passes at coworking operators like Industrious, CANOPY, or Regus are worth it when you’ve got a client briefing or board prep. For a more casual backdrop, Sightglass and Blue Bottle locations typically have reliable Wi‑Fi and just enough outlets if you arrive off‑peak.

Finally, a word on intent: I set a simple goal for each day—one meaningful connection, one surprising insight, and one concrete action to bring back to my team. ProductCon San Francisco 2025 is a catalyst if you design your experience with the same rigor you apply to your roadmap. If you spot me in a session or at a nearby cafe, say hello—I’m always up for trading notes on product strategy, pricing experiments, and what’s working in the field right now.

Quick note: restaurants and hours can change quickly—make reservations where possible and double‑check opening times the week of the event.

Inspired by this post on Product School.

November 3, 2025
Organizational Development Demystified: The Engine Behind Smarter Teams, Culture, and Growth

When people ask me how product organizations actually scale what works, I point them to a simple truth: organizational development is the operating system that makes strategy executable, teams empowered, and outcomes repeatable.

It turns out that organizational development isn’t just HR lingo. It’s the engine behind smarter teams, better culture, and long-term growth.

In practice, I think of organizational development as the discipline that aligns structure, incentives, rituals, and learning loops so empowered product teams can do their best work. It connects product management leadership with execution through clear decision rights, transparent roadmapping, and ways of working that reduce friction across product, design, and engineering.

On the ground, this looks like moving from activity measures to outcomes vs output OKRs, forming durable product trios to own customer problems end to end, and tightening stakeholder management so priorities don’t whipsaw week to week. It also means investing in onboarding that accelerates time-to-impact, creating feedback rituals that surface risks early, and using retention analysis to make smarter bets about where to double down.

The payoff is tangible: faster decision-making, fewer handoffs, and clearer accountability. Teams ship with confidence, leaders get leading indicators instead of lagging surprises, and employee retention at startups improves because people see how their work connects to a meaningful value proposition and product-led growth.

In my own practice, shifting to outcomes-first planning, establishing product trios, and clarifying interfaces across functions reduced decision latency, improved deployment frequency, and made ownership unmistakable. The organization became more resilient because the culture, processes, and metrics reinforced one another instead of competing for attention.

If you’re starting from scratch, begin by aligning on a small set of outcomes that matter, then redesign ceremonies and artifacts to serve those outcomes. Next, empower teams with clear autonomy and constraints—enough freedom to discover, enough guardrails to focus. Finally, make learning visible: use lightweight postmortems, discovery reviews, and customer signal dashboards so your operating system continuously improves.

Organizational development isn’t a one-time reorg; it’s a habit. When we treat it as a product—iterating on roles, rituals, and metrics just like we iterate on features—performance compounds, culture strengthens, and growth becomes sustainable.

Inspired by this post on Product School.

November 3, 2025
15 Must-Track Customer Retention Metrics to Crush Churn and Accelerate Sustainable Growth

I obsess over retention because it tells me the truth about product-market fit, value delivery, and revenue durability. In my role leading product strategy at HighLevel, I’ve learned that sustainable growth comes less from adding users and more from keeping the right ones engaged, successful, and expanding. The fastest way to get there is through a disciplined view of the right customer retention metrics.

Struggling to keep users? These customer retention metrics reveal what’s working, what’s not, and where to focus to reduce churn.

When I assess a product’s health, I look for a clean story across acquisition, activation, engagement, and expansion—then I validate that story against revenue outcomes. If those lines don’t reconcile, churn is coming. That’s why I track a core set of signals that expose value gaps early, guide product-led growth, and align go-to-market with actual customer outcomes.

Here are the 15 signals I rely on to diagnose retention risk and prioritize roadmaps: logo churn rate, gross revenue retention (GRR), net revenue retention (NRR), cohort retention by signup month, activation rate, time-to-value (TTV), feature adoption rate, DAU/WAU/MAU and stickiness (DAU/MAU), session frequency and duration, expansion revenue rate, contraction/downgrade rate, customer lifetime value (CLV), onboarding completion rate, customer health score, and support tickets per account with time to resolution. Together, these metrics show whether customers realize value quickly, keep finding more value over time, and are willing to grow with the product.

Here’s how I use them in practice. If activation rate or time-to-value slips, I invest in onboarding clarity, in-app guides, and product tours to remove friction and accelerate first success. If GRR weakens, I re-examine renewal messaging, pricing fairness, and critical feature gaps. If NRR stalls, I revisit packaging, discovery-driven upsell paths, and the expansion moments that naturally occur after users unlock initial value.

A unified analytics platform connecting product usage, lifecycle events, and CRM integration is essential. I pair cohort analysis in Amplitude analytics with qualitative insights from Intercom, then use Pendo to instrument in-app nudges and measure feature adoption lift. A/B testing helps me validate which interventions move the metrics that matter, not just vanity engagement.

Cadence matters. I review leading indicators weekly (activation, TTV, feature adoption), lagging indicators monthly (GRR, NRR, CLV), and cohort retention every quarter to ensure improvements compound. This rhythm keeps teams aligned on outcomes vs output and focuses energy where it reduces churn fastest.

If you adopt only one habit, make it this: tie every roadmap bet to a specific movement in these retention metrics, then measure relentlessly. When we do this well, our product doesn’t just acquire users; it earns loyal advocates—and that’s the most efficient growth engine there is.

Inspired by this post on Product School.

November 3, 2025