Customer feedback is the most reliable compass I have for product strategy and execution. Over the years leading product at HighLevel, I’ve built and refined a system that turns raw signals from users into clear, prioritized decisions our teams can confidently ship.
A practical guide to collecting and using product feedback in product management (from AI tools to early-stage tactics) for better product decisions.
My playbook starts with continuous discovery. I keep a steady flow of insights from sales calls, customer support threads, community forums, and in-product behavior so I can triangulate patterns rather than chase loud anecdotes. This mix of quantitative and qualitative data helps me separate urgent noise from strategically meaningful trends.
On the quantitative side, I rely on product analytics to ground the conversation. Amplitude analytics gives me activation, retention cohorts, and feature engagement, while controlled experiments and A/B testing validate whether an idea actually moves a target metric. Tying these signals to specific customer segments helps me see where product-led growth is working—and where it’s stalling.
For qualitative insight, I combine in-app guides and lightweight surveys (via tools like Pendo) with structured interviews and support escalations (often surfaced through platforms like Intercom). I map problems using the Kano Model to understand which requests are basic expectations, which are performance drivers, and which are potential delights. This keeps our roadmap focused on outcomes, not just outputs.
AI now accelerates the synthesis step. With LLMs for product managers in my AI product toolbox, I summarize interview transcripts, cluster themes across thousands of notes, and quantify sentiment without losing nuance. I still review raw artifacts to avoid hallucinations and preserve context, but AI reduces the time from signal to insight dramatically—freeing me to spend more energy on judgment and storytelling.
In early-stage contexts, I bias toward speed and proximity to users. I schedule founder- or PM-led discovery calls weekly, instrument product tours early, and launch scrappy in-product prompts to validate demand before over-investing. When data is sparse, I focus on high-signal channels (power users, churned customers with qualified use cases) and document crisp problem statements that connect directly to activation, retention analysis, and revenue outcomes.
Prioritization ties everything together. I translate insights into hypotheses aligned to outcomes vs output OKRs, then pressure-test them with feasibility and strategic fit. We run small, measurable experiments, track deltas in activation and retention, and adjust the product roadmapping and sprint planning cadence based on what the data and customers teach us.
This approach builds trust with stakeholders and creates empowered product teams. By grounding decisions in a transparent trail of feedback, analytics, and experiments, we reduce thrash, move faster, and—most importantly—ship product moments that customers value.
If you’re refining your own feedback engine, start by instrumenting the basics, set a weekly discovery rhythm, and let AI handle the heavy lifting on aggregation and synthesis. The compounding effect is real: better insights lead to better bets, which lead to better outcomes for your users and your business.
Trust is the currency of any high-stakes AI product, and nowhere is that more true than in healthcare. I recently dug into how Healio built an AI assistant for physicians—an audience that can’t afford to be wrong—and it’s a masterclass in balancing accuracy, transparency, and speed without compromising credibility.
Healio, a 125-year-old medical publishing company, set out to create Healio AI to help clinicians prepare for patient care. From the outset, their guiding principle was simple: physicians won’t trust you until you prove it. That lens shaped every decision—from discovery and prototyping to architecture, evaluation, and ongoing validation.
Discovery started with a survey of 300 healthcare professionals to understand real-world needs at the point of care. The headline insight: physicians primarily want AI for preparation, not bedside use. Even more surprising, the top ask wasn’t purely diagnostic support; it was help with patient communication and empathy—translating complex information into clear, accessible conversation.
Momentum mattered. After beginning with Figma mockups to validate workflows, the team built a working prototype in a single weekend using Cursor. That velocity wasn’t about cutting corners; it was about proving value quickly, reducing ambiguity, and iterating with concrete feedback from physicians.
Under the hood, the system employs RAG and hybrid search—combining lexical search, vector search, and semantic search across multiple trusted sources like PubMed. As any PM who has integrated biomedical literature knows, "just use PubMed" isn’t simple—there are five different ways to access the same data, each with trade-offs. The team made pragmatic choices to balance freshness, coverage, latency, and cost while preserving trust in source quality.
Designing for trust extended all the way to the citation UX. The team leaned into citations that physicians actually trust: subscripts, hover states, and progressive disclosure. This gave clinicians verifiable threads back to source material without overwhelming the core interaction, aligning with how experts want to audit evidence under time pressure.
Evaluation wasn’t left to chance. They stood up eight LLM judges for evals: safety, medical accuracy, faithfulness, relevancy, completeness, reasoning, clarity, and overall quality. Just as importantly, they treated those signals as directional, not definitive. In a high-stakes domain, physician feedback trumps LLM-as-judge feedback—so they complemented automated evals with direct reviews from practicing clinicians to calibrate quality and reduce hallucinations.
On the safety front, the team implemented HIPAA compliance and input guardrails for masking personal health information. That choice reflects strong data governance and privacy-by-design thinking: protect PHI by default, constrain prompts to safe boundaries, and make compliance a first-class citizen in the product architecture.
They also addressed monetization without compromising experience. Serving contextual ads while the LLM processes queries is a practical approach that preserves physician workflow efficiency and creates a clear, non-intrusive revenue model.
Critically, the work didn’t stop at launch. The Healio Innovation Partners provide ongoing discovery and validation, ensuring the system evolves with physician needs and the medical evidence base. This is the operating cadence you want for any AI product that sits at the intersection of safety, accuracy, and fast-changing knowledge.
My takeaways for building AI in high-stakes domains: prioritize retrieval-first pipelines over model cleverness; couple RAG with hybrid search across vetted sources; design citations that earn trust at a glance; use eval-driven development, but let domain-expert feedback be the ultimate judge; and embed regulatory compliance into your product strategy from day one. If trust is your North Star, this is a playbook worth emulating.
Across my teams and portfolio, I’m watching AI fundamentally reshape product-led growth—from static funnels and one-off playbooks to adaptive, compounding growth loops that learn in real time. The shift isn’t just technological; it’s an operating model change that rewards continuous discovery, rigorous instrumentation, and outcome-driven product strategy.
"Learn how AI is transforming PLG with a new generation of growth loops that can turn your product into a self-optimizing platform." That line captures what I’ve been building toward: systems that sense user intent, decide the next best action, act contextually, and learn to improve the loop with every interaction.
Here’s the core pattern I rely on. First, sense: unify product analytics and behavioral signals (think Amplitude analytics, Pendo events, Intercom conversations) into a single, queryable, privacy-safe layer. Second, decide: apply AI Strategy—LLMs for product managers, rules, and retrieval—to segment users by intent and probability of success. Third, act: deliver in-app guides, product tours, tooltips, or personalized nudges that accelerate user activation and time-to-value. Finally, learn: run A/B testing with a clear minimum detectable effect (MDE), then feed outcomes back into the model for continuous optimization.
Activation is where the gains start compounding. With gen ai, I can auto-generate tailored onboarding checklists, dynamic walkthroughs, and contextual help that adapts to the user’s role, data maturity, and current friction points. We’ve moved from generic product tours to precision guidance that updates based on real-time behavior—often lifting first-week activation and shortening time-to-first-value without adding support load.
Experimentation is the governor that keeps speed and quality in balance. I instrument every growth loop end to end and pair eval-driven development with A/B testing to confirm incremental impact. Amplitude analytics gives me cohort views and path analysis; Pendo or Intercom can deliver in-app variants; a unified analytics platform closes the loop on retention analysis so I’m not optimizing for click-through at the expense of long-term value.
Retention and expansion are where AI shines as a compounding engine. Retrieval-first pipeline patterns allow instant, contextual support that deflects tickets and boosts perceived product competence. Agentic AI can orchestrate next-best actions—prompting power users toward advanced features, surfacing value moments, or timing expansion prompts when success signals appear. The result is a virtuous cycle: better guidance drives deeper adoption, which improves model accuracy, which unlocks more relevant guidance.
None of this works without guardrails. I bake in AI risk management from the start: strict data governance, privacy-by-design, human-in-the-loop review for high-impact actions, transparent user consent, and continuous drift monitoring. The goal is reliable automation that users trust—augmented by clear fail-safes when confidence drops.
Operationally, I anchor the work in empowered product teams and product trios, focus on outcomes vs output OKRs, and practice continuous discovery to validate problems and solutions before scaling. The baseline metrics I watch: activation rate, time-to-value, week-four retention, PQL/PQA conversion, expansion revenue, and support deflection—each tied to a specific growth loop hypothesis.
If you’re starting fresh, begin with the highest-leverage loop: user activation. Instrument your onboarding journey, define the critical path to value, ship two to three personalized interventions, and measure impact with a precommitted MDE. Scale what wins, drop what doesn’t, and iterate weekly. Once activation is compounding, extend the same approach to adoption depth, collaboration features, and expansion triggers.
In practical terms, AI-powered PLG is less about flashy features and more about disciplined feedback loops. Build the sensing fabric, keep the decision layer auditable, ship small actions quickly, and treat learning as the product. Do that, and your product doesn’t just grow—it becomes a self-optimizing platform.
I’ve led product teams through countless discovery cycles, and nothing has accelerated our learning loops like AI. By weaving AI into our continuous discovery practice at HighLevel, I cut time-to-insight, reduce risk earlier, and keep our product strategy relentlessly focused on customer outcomes.
AI streamlines product discovery by accelerating research, prototyping, and validation, enabling teams to make faster, smarter, and user-driven decisions.
In the research phase, I use gen ai and LLMs for product managers to synthesize interviews, cluster themes, and surface unmet needs in minutes instead of days. Pairing those qualitative insights with behavioral signals in Amplitude analytics helps me spot high-intent cohorts and friction points at scale, so our problem framing is both human-centered and data-backed.
From there, I translate insights into crisp hypotheses and prioritize with the Kano Model and outcomes vs output OKRs. To keep experiments honest, I define a minimum detectable effect (MDE) up front and design A/B testing plans that reflect realistic traffic and seasonality, ensuring our decisions are statistically grounded rather than anecdotal.
Prototyping is where gen ai for product prototyping really shines. I spin up multiple UX flows, UI copy variants, and edge-case scenarios using prompt engineering, then iterate with rapid feedback from product trios. When needed, I mock in-app guides and product tours to validate onboarding concepts before we commit to code, preserving velocity without sacrificing quality.
For validation, I lean on a mix of lightweight experiments—fake-door tests, concierge pilots, and targeted A/B testing—augmented by in-product surveys via Pendo or Intercom. For AI-powered features, I apply eval-driven development to measure relevance, latency, and safety, so we can ship responsibly while maintaining the pace of learning.
This approach only works when the team is structured to move fast. Empowered product teams and product trios own discovery end-to-end, with clear guardrails around data governance, privacy-by-design, and AI risk management. That alignment lets us shift from opinions to evidence, and from output to outcomes, without friction.
If you’re getting started, pick one discovery loop to transform: automate research synthesis, prototype two to three variants with AI, and validate with a tightly scoped experiment. Instrument your analytics, track time-to-insight and time-to-prototype, and iterate your product roadmapping and sprint planning with what you learn. The payoff is immediate: faster cycles, stronger conviction, and a more user-driven path to product-led growth.
Happy New Year! I’m kicking off 2026 with a behind-the-scenes look at what’s changing in my product practice, the experiments I’m running with my teams at HighLevel, and the trends I’m most energized by—especially around continuous discovery, AI workflows, and building stronger coaching cultures.
If you want to listen to the conversation that sparked many of these reflections, you can find it here: Spotify | Apple Podcasts.
Why Teresa sunset the live deep-dive cohorts—and how on-demand and the new Discovery Habits Toolbox better support real behavior change. This pivot resonated with my own experience: some skills, especially discovery habits, only stick when they’re reinforced in the flow of real product work, not just in a time-boxed cohort. In my org, we’re leaning into on-demand learning paired with manager coaching to drive durable behavior change.
What leaders actually need to coach interviewing, assumption testing, and core discovery habits inside their orgs. I’ve found that empowered product teams thrive when leaders have lightweight coaching tools, practical prompts, and clear expectations for product trios. This is less about one-off training and more about building communities of practice where deliberate practice and feedback loops become routine.
Why training is shifting toward ongoing, leader-supported learning (and how AI will accelerate the shift). AI Strategy isn’t just about tools—it’s about learning systems. For LLMs for product managers to create leverage, we need eval-driven development, privacy-by-design, and clear guardrails. I’m building AI workflows that enable managers to review interviews, spot anti-patterns, and nudge teams toward better decisions—without replacing critical thinking.
Teresa’s move into paid subscriptions and why AI content doesn’t fit the classic “design once, run for years” course model. I see the same reality in my content roadmap: the half-life of AI guidance is short. That pushes us toward subscription models, tighter feedback loops, and a more adaptive go-to-market strategy for education products.
A sneak peek into the AI tools Teresa is building for discovery work—from interview coaching to near-ready interview snapshot generation. I’m particularly excited by tooling that scaffolds better interviews, sharpens assumption testing, and speeds up synthesis without skipping the human judgment step. These capabilities map directly to where I want my teams investing time: spending less energy on admin and more on learning from customers.
Petra’s plans for the year: community building with Product at Heart, a new product leadership email course, her Product Leadership Wheel, and workshops launching in Cairo. As someone who believes in conferences as high-quality “energy wells,” I’m inspired by how these programs create momentum for leaders who are upgrading their coaching muscles.
The role of conferences and retreats in staying grounded, inspired, and connected. I treat these gatherings as strategic resets—spaces to test ideas, confront blind spots, and deepen my network for future collaboration. The best outcomes often come from serendipitous hallway conversations and hands-on sessions where you can pressure test frameworks with peers.
How Teresa is staying on top of academic research (and why “synthetic users” aren’t ready for prime time). I agree: while synthetic data can be useful for scaffolding, it’s not a substitute for direct customer contact. Combine academic rigor with real-world interviewing and strong data governance—especially when operating under General Data Protection Regulation (GDPR).
The shared challenge of evaluating vendors and conference speakers making questionable AI claims. My heuristic: ask for clear problem statements, reproducible evaluations, grounded benchmarks, and a path to safe deployment. If a pitch can’t show measurable uplift or ignores compliance, it’s not ready for empowered product teams.
Key takeaways I’m carrying into 2026: delivery models matter; leaders need coaching tools, not just training; AI is reshaping how we teach and learn; experimentation is the theme of 2026; and community still energizes. That’s the blueprint I’m using to strengthen continuous discovery, refine our AI workflows, and sustain high standards in product management leadership.
What about you? How are you integrating AI workflows into your discovery practice, and what coaching tools are helping your managers reinforce the right habits? Share your approach—I’d love to learn what’s working in your context.
Resources & Links:
Follow Teresa Torres: https://ProductTalk.org
Follow Petra Wille: https://Petra-Wille.com
Teresa’s website: Product Talk
General Data Protection Regulation (GDPR)
Product Talk Academy
Deliberate Practice – ATP episode where Teresa talked about the ending live cohorts for Deep Dive classes
2026 is closer than it feels, and the signals are already clear. I’ve been synthesizing what I’m seeing across empowered product teams, boards, and cross-functional partners into a practical view of what matters next. A sharp look at product management trends for 2026. Not guesses, but signals from top product leaders shaping how PMs will actually work next.
In this analysis, I distill eleven shifts that are changing the craft—from outcomes vs output OKRs and continuous discovery to stronger product strategy and tighter product roadmapping and sprint planning. The throughline is simple: prioritize customer value, ship with focus, and measure what moves the business. These aren’t headline trends; they’re working patterns I’m seeing across high-performing organizations.
AI is no longer a side project—it’s part of the product manager’s core toolkit. Agentic AI, LLMs for product managers, and trustworthy AI workflows are accelerating discovery, sharpening problem framing, and enabling faster iteration. The best teams pair this with disciplined evaluation and experimentation, so insight compounds without sacrificing safety, privacy, or product quality.
Execution is getting crisper through product trios and stronger stakeholder management. When design, product, and engineering co-own discovery and delivery, teams reduce handoffs and increase clarity. That alignment translates into better prioritization, fewer context-switches, and a roadmap that reflects real trade-offs—not wish lists.
On growth, product-led growth remains a durable engine when it’s anchored in a compelling value proposition and instrumented end-to-end. Clear activation moments, in-app guides, and thoughtful product tours outperform brute-force acquisition. When we connect these motions back to product strategy and the roadmap, we create a repeatable loop that compounds adoption and retention.
Governance and trust are now table stakes. Privacy-by-design, data governance, and a pragmatic approach to regulatory compliance protect both users and velocity. Teams that build these practices into their operating model move faster because they avoid late-stage rework and maintain stakeholder confidence.
If you’re leading a product org—or aspiring to—this is your field guide to 2026. I’ll unpack where these shifts are strongest, how to apply them in your context, and the pitfalls to avoid. The aim is to give you clear language, concrete practices, and a sharper edge as you shape what your team builds next.
How do you help disadvantaged students take action on opportunities they don't even know exist? That question has been top of mind for me as I’ve explored how AI can augment—not replace—human mentorship. Recently, I dug into the work behind Zero Gravity, a UK-based platform using mentoring, community, and learning pathways to unlock elite career opportunities for state school students. Their approach reframed a core problem I care deeply about: the "knowing-doing gap."
I sat down with Elliot Little (Product Manager) and Dan St. Paul (Software Engineer) from Zero Gravity to unpack how they’re tackling this gap with an AI career co‑pilot. They’ve intentionally positioned the system as an orchestrator, not an automation tool—bridging the space between knowing what to do and actually doing it. As a product leader, I see this as a powerful pattern for Generative AI: use AI to coordinate steps, personalize guidance, and empower action in moments where confidence and clarity are fragile.
What resonated most was the humility of their build journey. They started with grand visions of AI mentors and synthetic avatars, then scaled back to something simpler and more effective. The first prototype—a job suitability summary—didn’t deliver the "wow moment" they expected. And they discovered that hiding the "LLM magic" backfired—students needed to feel the personalization. That insight aligns with my own experience: users must perceive the value for trust and motivation to compound.
From a UX standpoint, the team chose text chat over voice input and leaned into guided prompts rather than empty text boxes. That decision lowered cognitive load and increased completion rates—classic product management tradeoffs that privilege momentum over novelty. In my view, this is what good AI product strategy looks like: invite action with structure, then expand autonomy as confidence grows.
The technical backbone is equally thoughtful. Multi‑month journeys require rigorous context window management to avoid exploding token counts and degrading quality. I appreciated their pragmatic toolkit: context management techniques like removing stale tool calls, summarizing history, exposing tools conditionally. They also used application logic rather than complex RAG architectures to manage tool availability and context freshness. This is the kind of disciplined engineering that keeps systems reliable at scale without overcomplicating the stack.
Model selection was fit‑for‑purpose, not one‑size‑fits‑all. They’re using different models for different tasks, including "GPT-5 Nano for structured outputs, lighter models for quick replies." That modularity enables speed and cost control while preserving high‑fidelity moments where structure matters most.
Safeguarding was treated as a first‑class concern—non‑negotiable when you’re building AI for 16‑year‑olds. Their safeguarding architecture pairs moderation endpoints with external verification via Unitary. They also invested in building a failure taxonomy through internal red team/green team exercises. This is AI risk management done right: define failure modes early, test ruthlessly, and wire safety into the product surface area—not just the model layer.
Evaluation was grounded in outcomes, not demos. The team focused on whether students progressed from insight to action: applying, interviewing, and engaging with mentors. That aligns with how I run eval‑driven development—ship narrowly, measure real behavior, and iterate toward a repeatable "wow moment" that students can actually feel.
Looking ahead, I’m excited by what’s next: long‑term memory management for multi‑year student journeys. It’s a hard problem—balancing privacy, provenance, and portability—but it’s precisely where an AI career co‑pilot can compound value over time. The vision is compelling: a resilient companion that remembers goals, adapts to context, and orchestrates the right next step.
If you want to dive deeper, you can listen to the full conversation on Spotify and Apple Podcasts:
Listen to this episode on: Spotify | Apple Podcasts
Blue Dot Impact AI Safety Course – free AI safety course Elliot recommended: https://bluedot.org/
My key takeaways: build AI that augments human relationships, not replaces them; don’t hide the personalization—let learners feel it; privilege application logic over unnecessary architectural complexity; and treat safety, context, and evaluation as product features, not afterthoughts. That’s how we bridge the "knowing-doing gap" with integrity and scale.
I’ve sat in countless AI measurement debates and noticed a recurring gap. One major voice has been noticeably underrepresented in the AI measurement conversation: the product manager (PM) that’s leading development. From experience, PMs and developers do need different measurement tools—and making those differences explicit is exactly what speeds up decisions and improves outcomes.
Developers optimize the model and system layer. Their toolkit centers on eval-driven development: offline evals, regression suites, red-teaming, latency and throughput monitoring, token cost tracking, and hallucination rate reduction. On the delivery side, engineering teams watch DORA metrics alongside CI/CD performance to keep iteration fast and safe. When building LLM-backed experiences, they also care deeply about retrieval-first pipeline quality and context window management because those mechanics determine grounding, relevance, and consistency.
PMs, by contrast, own outcomes. We instrument user journeys end to end and define a clear north-star tied to value: activation, time-to-value, task success rate, retention analysis, support deflection, and revenue contribution. We rely on A/B testing frameworks and minimum detectable effect (MDE) planning to separate real impact from noise, and we consolidate behavioral signals in a unified analytics platform like Amplitude analytics and Pendo to understand adoption, friction, and cohort differences. This is the heart of product-led growth and continuous discovery: evidence, not anecdotes.
The fact that these toolboxes differ is a strength, not a weakness. Specialized metrics keep responsibilities crisp: developers guarantee model quality and reliability; PMs guarantee that quality translates into customer and business outcomes. What we need is an explicit metrics ladder that connects layers—model-level quality floors and SLOs, feature-level KPIs, and company-level results—so trade-offs are transparent and prioritization is principled.
In practice, I create a shared measurement contract for every AI initiative. It links eval sets to user-facing success criteria, defines acceptance thresholds, and spells out observability across the stack. We include governance from day one—AI risk management, privacy-by-design, and data governance—so we can scale responsibly without slowing teams down.
Here’s the AI product toolbox I give my teams: start with a concise value hypothesis; define a success rubric the customer would recognize; instrument the happy path and the failure path; plan experiments with MDE up front; segment results by persona and job-to-be-done; and close the loop with qualitative feedback inside the product via in-app guides, product tours, and lightweight surveys. For AI features specifically, add Agent Analytics for agentic AI, capture grounding sources for explainability, and log model/context inputs to make debugging and iteration repeatable. That way, LLMs for product managers stop being magic and start being manageable.
When we roll out a new assistant—whether a retrieval-augmented copilot or a voice AI agent—we set two dashboards: one for developers (eval pass rates, latency, context integrity, error budgets) and one for PMs (activation, task completion, deflection, satisfaction). The dashboards read differently by design, yet they are joined at the hip by shared definitions and experiment IDs. This lets us move quickly with confidence: engineering can tighten quality loops while product steers toward the outcome that matters most.
If you’re feeling the tension between model metrics and product metrics, don’t collapse them—connect them. Start with a thin slice, agree on 3–5 measurable outcomes, and let your evals and A/B tests work together. With a clear metrics ladder and a unified analytics platform, PMs and developers can each excel at their craft and still ship AI that customers love.
You ask an AI model for a feature brief. It returns polished prose, sensible recommendations, and a tidy set of success criteria. Then the review starts: the target segment is wrong, the customer evidence is anecdotal, a strategic constraint is missing, and nobody can tell where the claims came from.
The goal is not to give the model everything your company knows. The goal is to provide the smallest sufficient body of evidence for the decision in front of you, while preserving enough lineage for a reviewer to inspect the result.
Key takeaways
Start with a decision contract that defines the decision, audience, constraints, evidence standard, and required output.
Build a compact context pack from canonical strategy, relevant behavioral data, direct customer evidence, operating constraints, and decision history.
Retrieve before you generate. Use metadata, recency, authority, and relevance to select evidence instead of dumping entire repositories into the context window.
Preserve traceability. Every important claim should point to an evidence identifier, and the output should separate observations, inferences, and recommendations.
Version the prompt and context together, then evaluate the complete system through rework, review time, first-pass alignment, and evidence fidelity.
Start with the decision, not the document
Product teams often describe the artifact they want rather than the decision it must support. Draft a PRD, summarize these interviews, or write a roadmap rationale sounds concrete, but each request leaves the model to infer what matters.
That ambiguity changes retrieval. A positioning decision needs competitive and customer-language context. A prioritization decision needs strategy, affected users, behavioral evidence, constraints, and opportunity cost. Release notes need verified product behavior, the intended audience, and approved terminology. The same generic prompt cannot reliably determine those boundaries.
Before gathering evidence, write a decision contract with these fields:
Decision: What choice, judgment, or next action will this output support?
Audience: Who will review or use it, and what do they already know?
Deliverable: What sections, level of detail, and format are required?
Boundaries: What is explicitly out of scope, already decided, or prohibited?
Evidence standard: Which claims require direct evidence, and how should citations appear?
Uncertainty: What should the model do when evidence is missing, stale, or contradictory?
A weak request is: Summarize onboarding research. A decision-ready request is: Help the product trio decide whether the onboarding problem should enter discovery. Identify the affected cohort, observed friction, strength of evidence, unresolved questions, and the next research step. Do not recommend a roadmap commitment.
The second request gives retrieval a job. It tells the system which evidence to find and gives reviewers a basis for rejecting unsupported output.
Give conflicting evidence an explicit hierarchy
Most internal knowledge bases contain competing versions of reality. A planning deck may conflict with an approved strategy. A recent support conversation may contradict an older research summary. A customer request may not match observed behavior. Without an authority rule, the model may blend these artifacts into a confident compromise that nobody actually endorsed.
A practical default hierarchy is:
Current, approved strategy and explicit leadership decisions establish the frame.
Behavioral evidence establishes what users did within the measured population and period.
Verbatim customer evidence establishes what particular customers said and how they described the problem.
Support and operational signals reveal recurring friction that may need further validation.
Team hypotheses remain hypotheses until stronger evidence supports them.
This is a starting rule, not a universal ranking. Your hierarchy should match the decision. The important move is to state it. Freshness alone does not make an artifact authoritative, and authority alone does not make old evidence current. When two credible artifacts disagree, instruct the model to expose the conflict rather than reconcile it silently.
Build a minimum viable context pack
A context pack is the evidence package for one task. It is deliberately narrower than a company knowledge base. Each item earns its place by answering a question the requested output must address.
Context layer
Question it answers
Useful artifact
Strategic frame
Why does this problem matter now?
Approved strategy statement, objective, or decision principle
Affected user
Who experiences the problem?
Cohort definition, segment criteria, or relevant account profile
Behavior
What happened in the product?
Usage pattern, funnel analysis, retention signal, or journey evidence
Customer need
How do users describe the problem?
Verbatim interview excerpts, support conversations, or research synthesis
Constraints
What limits the solution space?
Technical, operating, commercial, or policy constraint
Decision history
What has already been decided or rejected?
Decision record with rationale and status
Do not fill every row by default. For a narrow writing task, two layers may be enough. For a prioritization decision, several may be essential. Start with the requested output and ask which evidence would allow a skeptical reviewer to verify each section.
The example works because each artifact has a different job. Five documents making the same strategic argument would create repetition, not coverage. Context quality comes from complementary evidence, not document count.
Turn each artifact into an evidence unit
Raw files are difficult to retrieve and easy to misread. Wrap each relevant slice in a small evidence unit:
Identifier: a stable label such as E1 or E2 that the output can cite.
Origin: the system, analysis, interview, or decision record from which it came.
Status: approved, draft, superseded, disputed, or observational.
Scope: the segment, cohort, workflow, product area, and period to which it applies.
Relevant finding: a concise summary written for the current decision.
Raw evidence: the excerpt, data slice, or linked artifact needed to inspect the summary.
Caveat: a known limitation, missing comparison, or unresolved contradiction.
This two-layer structure solves a common compression problem. The short summary conserves context-window space, while the raw excerpt preserves wording and qualifiers when nuance matters. Do not repeatedly summarize prior summaries. Each compression step can remove scope, uncertainty, and disagreement. Keep a path back to the underlying evidence.
You have enough context when every required part of the deliverable has relevant evidence, major conflicts are represented, and additional artifacts merely repeat what is already present. If an output section has no supporting evidence, either retrieve more or label the section as an open question. Do not ask fluent prose to hide the gap.
Retrieve, compress, and assemble in that order
Large context windows make it tempting to attach whole repositories. That usually transfers the curation problem to the model. Relevant evidence must now compete with stale plans, duplicate findings, unrelated segments, and abandoned decisions.
Translate the decision contract into evidence questions. Ask what strategic frame, customer signal, behavior, constraint, and decision history are required.
Filter by hard boundaries first. Exclude the wrong product area, segment, status, or period before semantic ranking.
Retrieve relevant slices rather than complete files. A paragraph, chart interpretation, interview excerpt, or decision entry is often the useful unit.
Check authority and freshness. Mark superseded items and retain an older artifact only when its historical context matters.
Check coverage and contradiction. Confirm that the pack represents the affected population and does not hide credible opposing evidence.
Compress each selected item into an evidence unit, retaining a link or raw excerpt for verification.
Assemble the context in a fixed interface so the model can distinguish instructions, evidence, and the requested output.
Retrieval should also preserve access boundaries. An AI layer should not expose an artifact to someone who could not access it in its system of record. Treat customer material and internal strategy as governed inputs, not convenient prompt text.
Use a stable context interface
I treat the prompt as an interface to the context system, not as the system itself. A useful interface contains these blocks in a consistent order:
Role and objective: the perspective the model should take and the decision it must support.
Audience: the people who will use the deliverable and the assumptions they already share.
Constraints: scope boundaries, settled decisions, prohibited claims, and required terminology.
Evidence: labeled units such as E1, E2, and E3, each with status, scope, summary, raw support, and caveats.
Explicit ask: the analysis or artifact required, expressed as concrete questions.
Output contract: required sections, length, ordering, and citation format.
Evidence rules: cite material claims, distinguish observation from inference, expose conflicts, and avoid unsupported facts.
Self-check: identify missing evidence, unverified assumptions, constraint violations, and statements that lack citations.
Do not rely on instructions such as be accurate or think carefully. They do not define what accuracy means for this task. A stronger rule is: Cite an evidence identifier after every material claim. If the pack does not support a claim, label it as an inference or omit it. List unresolved questions separately.
Diagnose output failures as context defects
Output symptom
Likely context defect
Corrective move
Generic recommendations
The pack lacks customer, behavior, or constraint evidence
Add decision-specific evidence instead of more role-playing instructions
Confident but outdated claims
Retrieval ignored status, authority, or recency
Filter superseded artifacts and define which record is canonical
Important nuance disappears
Compression removed qualifiers or disagreement
Restore raw excerpts and carry caveats into the evidence units
Long output that does not support a decision
The ask names an artifact but not the decision
Rewrite the decision contract and remove irrelevant context
Stakeholders distrust the result
Claims have no visible lineage
Require evidence identifiers and preserve links to underlying artifacts
Repeated runs produce different conclusions
The prompt or context changed without version control
Snapshot both inputs and compare one controlled change at a time
This diagnostic matters because prompt edits can disguise the real failure. If the wrong cohort entered the pack, a more detailed output format will only produce a better-organized mistake.
Manage context quality as a product system
A single well-curated prompt can produce a good result. A product team needs a system that can produce a good result again, show why it was good, and reveal what changed when quality declines.
Make the output auditable
Ask the model to separate three kinds of statements:
Observation: directly supported by an evidence unit.
Inference: a reasoned interpretation that connects observations.
Recommendation: a proposed action that depends on evidence, assumptions, and product judgment.
This distinction prevents a plausible interpretation from being presented as a measured fact. Behavioral analytics can show a pattern within its defined cohort and period; it does not, by itself, establish why the behavior occurred. A customer quote can establish that a person expressed a need; it does not, by itself, establish prevalence. The final recommendation still needs human judgment about strategy, tradeoffs, and risk.
For consequential work, request a smaller cited output first. Review its evidence mapping, then expand it into a PRD, roadmap narrative, or executive brief. This makes unsupported reasoning easier to catch than reviewing a long deliverable after the model has built several sections on the same weak assumption.
Version the whole generation package
Store these elements together for each run:
Workflow and template version
Decision contract
Context snapshot and evidence identifiers
Retrieval and filtering rules
Prompt version
Model output
Human review result and requested changes
Prompt versioning without context versioning is incomplete. Two runs using identical instructions can diverge because an approved strategy changed, a stale analysis entered retrieval, or a different set of interviews was selected. The context snapshot lets you explain that difference.
Evaluate the workflow, not the elegance of one answer
Create a small evaluation set from real, recurring product tasks. Keep the decision and expected evidence stable while testing changes to retrieval, compression, context ordering, or instructions. Change one major variable at a time; otherwise you will not know what improved the result.
Review each run against a consistent rubric:
Evidence fidelity: Do claims accurately represent the cited material and its scope?
Coverage: Does the output address every required part of the decision?
Constraint adherence: Does it respect settled decisions, exclusions, and required terminology?
Traceability: Can a reviewer follow important claims back to evidence?
Uncertainty handling: Are missing, stale, or contradictory inputs visible?
Decision usefulness: Can the intended audience act, decide, or request the right next evidence?
When an evaluation fails, route the defect to the right layer. Evidence fidelity usually points to retrieval, source selection, or compression. Constraint failures point to the context interface. A technically correct but unusable deliverable points back to the decision contract. This turns AI quality from a subjective debate into a product improvement loop.
Template workflows only after you understand their evidence needs
Discovery synthesis, roadmap rationale, feature briefs, and release notes are good candidates because they recur and have recognizable inputs. Give each workflow its own decision contract, required context layers, retrieval filters, output contract, and evaluation rubric. Do not force them into one universal mega-prompt.
Start with one workflow your team already performs frequently. Take a real task, define the decision, assemble a compact evidence pack, assign identifiers, and review the result against the rubric above. Save the complete generation package. On the next run, change one weak layer and compare the review burden.
Once that loop is repeatable, AI stops being a blank page with a clever prompt. It becomes a governed product workflow whose inputs, reasoning boundaries, and quality can be inspected and improved.
INDUSTRY 2025: The Product Conference is circled on my calendar for good reason. In my role leading product management at HighLevel, I look for events that sharpen strategy, accelerate learning, and connect me with operators who ship. This one consistently delivers on all three, and 2025 promises to raise the bar for product management leadership.
Join Pendo at INDUSTRY in Cleveland, Ohio.
First, I expect deeply actionable product strategy insights—beyond platitudes. I’m prioritizing conversations on outcomes vs output OKRs, product roadmapping and sprint planning, and how great teams articulate a crisp value proposition while maintaining points of parity that matter. I’m going in with specific questions on product-market fit lessons and how to systematize strategic bets without stifling discovery.
Second, the surge of AI in product work is too important to observe from the sidelines. I’m comparing approaches across AI Strategy, LLMs for product managers, prompt engineering, and eval-driven development—especially in retrieval-first pipeline patterns. My focus: where AI genuinely improves product discovery, in-app guides, and customer support ai strategy, and where it risks adding complexity without outcomes.
Third, the community is unmatched for conference networking and pragmatic learning. I’m intentional about meeting product trios who run continuous discovery at scale, as well as leaders who’ve cracked stakeholder management under pressure. These are the moments where competitive differentiation is born—through candid stories of what didn’t work and why.
Fourth, I’m eager to stress-test data practices that power product-led growth. I’ll be exchanging notes on retention analysis, unified analytics platform decisions, user activation, and how teams integrate qualitative feedback with event data to inform roadmaps. I’m also interested in how practitioners leverage platforms like Pendo, Amplitude analytics, Intercom, and HubSpot to reduce time-to-insight and craft effective product tours and in-app guides.
Fifth, I treat INDUSTRY as a checkpoint for leadership growth. I’m looking for fresh takes on empowering product teams, first principles decision making, organizational development, and the IC to manager transition. The best sessions don’t just inspire; they give me two moves I can apply with my team on Monday.
To make the most of the week, I’m applying a continuous discovery mindset: arrive with clear learning goals, capture portable frameworks, and translate at least two insights into experiments before wheels-up. If you’re focused on product strategy, product discovery, and product-led growth, we’ll have plenty to compare and build on together.
I’ll be in Cleveland ready to learn, share, and connect with peers who care about craft and outcomes. If you’re attending, let’s compare notes on what’s working, what’s stalled, and how we can raise the bar for product management leadership in 2025 and beyond.
I build AI products with a simple conviction: disciplined experimentation beats intuition. Over the years, I’ve refined a practical playbook that helps my teams learn faster, reduce risk, and turn every release into a smarter next step.
Product experimentation isn’t luck; it’s a method. Learn how top AI product managers test, measure, and grow smarter with every release.
I begin every effort with a crisp hypothesis, an expected user or business outcome, and unambiguous success criteria tied to outcomes vs output OKRs. Before writing a line of code, I define primary metrics and guardrails so we know what “good” looks like—and what to stop.
When the change affects UX, pricing, or activation flows, I favor A/B testing with the statistical rigor to back decisions. We calculate the minimum detectable effect (MDE), choose appropriate randomization units, and pre-register the analysis plan to avoid p-hacking. This gives the team the confidence to scale wins and sunset underperformers quickly.
AI features demand a tailored approach, so I run eval-driven development before any user sees a variant. We curate golden datasets, score candidate prompts and models, and stress-test failure modes. This is where LLMs for product managers matters: prompt templates, context window management, and a retrieval-first pipeline are all evaluated for quality, latency, and cost-to-serve. I treat “hallucination rate,” safety violations, and bias as first-class metrics under AI risk management.
To de-risk launches, we ship behind feature flags with CI/CD, monitor DORA metrics, and roll out in stages. Product trios own problem framing to solution delivery, which shortens feedback loops and preserves accountability. If early signals drift from our hypotheses, we pause, adjust, and re-run—no sunk-cost thinking.
Measurement is non-negotiable. I instrument user journeys end-to-end with Amplitude analytics, track activation and retention analysis, and map behavior to learning objectives. We consolidate logs and events into a unified analytics platform so qualitative insights from customer research pair cleanly with quantitative trends.
Continuous discovery keeps the engine running. Weekly customer conversations, in-product feedback, and lightweight prototypes ensure we validate needs, not just solutions. The output flows into product discovery, product roadmapping and sprint planning, and a reusable AI product toolbox that scales across teams.
Finally, I protect the culture that makes experimentation work: we celebrate invalidated hypotheses, document decisions, and optimize for outcomes over output. That’s how empowered product teams sustain product-led growth—even as complexity grows.
If you’re building AI features today, adopt this playbook to maximize learning velocity, minimize risk, and compound advantage. The method is straightforward: form strong hypotheses, test with rigor, measure what matters, and let evidence—not HiPPOs—guide the roadmap.
The most valuable lesson I’ve learned leading product organizations is that portfolio choices make or break outcomes. In an era of infinite requests and finite teams, the question isn’t what we could build—it’s what we must build next. That’s why I’m codifying a pragmatic, AI-driven playbook to optimize the product portfolio while staying true to outcomes, not output.
AI-powered product portfolio optimization is here. Explore strategies and tools helping product leaders manage complexity and boost ROI.
My starting point is a data backbone that connects strategy to reality. I aggregate product usage, revenue by segment, cost-to-serve, retention cohorts, and support signals into a unified analytics platform, then layer a retrieval-first pipeline so LLMs can reason over clean context. Instrumentation matters: Amplitude analytics, Pendo, and in-app guides provide the behavioral and activation signals that make prioritization measurable.
From there, I translate strategy into an objective decision system. I express outcomes vs output OKRs, align initiatives to value proposition and competitive differentiation, and classify opportunities with the Kano Model. LLMs for product managers help cluster voice-of-customer at scale; with thoughtful prompt engineering and AI workflows, I can map themes to jobs-to-be-done, quantify demand, and de-duplicate asks across stakeholders.
Execution hinges on evidence. I run A/B testing with a clear minimum detectable effect (MDE), pair it with eval-driven development for AI features, and ship through CI/CD while tracking DORA metrics. This closes the loop between product roadmapping and sprint planning and real-world performance—activation, retention analysis, and Web Vitals inform the next set of portfolio bets.
Trust is a feature, so governance is built-in. Privacy-by-design, data governance, and AI risk management guide how we store, prompt, and evaluate models. I apply guardrails to sensitive workflows and define success metrics that balance short-term ROI with long-term resilience and regulatory compliance.
The operating model matters as much as the models themselves. Product trios and empowered product teams run continuous discovery, pressure-test assumptions in QBRs vs OKRs, and make trade-offs visible. Stakeholder management becomes easier when the portfolio narrative is anchored in transparent scenarios and shared metrics.
If you’re getting started, here’s my flow: unify data, define outcomes, segment opportunities, simulate scenarios, and test fast. Use LLMs to synthesize signals you’d never humanly read, then make one focused bet per team that moves a measurable KPI. Rinse, learn, and reallocate—portfolio optimization is a living system, not an annual meeting.
Ultimately, the promise of this new playbook is simple: less noise, sharper focus, and compounding ROI. By pairing AI Strategy with disciplined product management leadership, we can manage complexity with clarity—and consistently build what matters most.
