Tag: eval-driven development

AI Ethics That Win Trust: The Product Manager’s Playbook for Safe, Scalable Innovation

I’ve learned that the fastest way to lose customers with AI is to ship something powerful but unpredictable. The fastest way to earn their loyalty is to ship something powerful and trustworthy. That’s the job.

AI ethics in product management isn’t about theory anymore. It’s the line between trusted products and unpredictable ones. Here’s what PMs need to know.

When I frame AI ethics for my team, I translate principles into practices that protect customers and accelerate velocity. We bake trust into product strategy, delivery, and operations—so ethics is not a separate checklist, but a core capability that compounds over time.

First, I anchor the roadmap on explicit outcomes and guardrails. We set success metrics alongside ethical constraints, tying them to outcomes vs output OKRs, so teams know not only what to achieve but what to avoid. If a feature can’t meet our trust thresholds, it doesn’t ship—no matter how impressive the demo.

Data is where trust starts. We enforce data governance from day one: clear data lineage, collection minimization, role-based access, and privacy-by-design defaults. We document lawful bases for processing, consent flows, and retention policies, then automate checks so they run with every change—not just at launch.

On the model side, we use eval-driven development to turn subjective “looks good” into measurable quality. We design evaluations for safety, bias, robustness, and performance; we red-team prompts; and we test failure modes in realistic conditions. For LLMs, we lean on a retrieval-first pipeline to ground responses in authoritative data, and we apply context window management and prompt engineering patterns to reduce hallucinations.

In the product experience, we make ethical choices visible. That means clear disclosures when AI is in the loop, user controls to review and correct outputs, and transparent UX writing that avoids overclaiming. In-app guides and thoughtful tooltip design help users understand capabilities and limits without friction.

Shipping safely requires operational discipline. We build kill switches, human-in-the-loop overrides for high-risk actions, and incident playbooks that pair incident management with threat detection and response. SRE partnerships ensure observability covers both model behavior and customer impact, with rollback paths ready when drift or regressions appear.

Governance is a team sport. I maintain an AI risk register, review it with security, legal, and product trios, and brief leadership on residual risks and mitigations. Regulatory compliance isn’t a final hurdle; it’s a design input that shapes technical choices long before code reaches production.

Build vs buy decisions carry ethical implications too. Vendor due diligence covers model provenance, data handling, eval results, and incident history—not just feature checklists. Contracts codify SLAs, audit rights, and deletion commitments so our obligations to customers flow down the stack.

Finally, we earn trust in public. We publish model facts, change logs, and limitations in a customer-facing trust center, and we invite feedback loops that turn real-world usage into better safeguards. Stakeholder management matters here: being candid about trade-offs often increases confidence more than chasing perfection.

This is how I keep teams fast without being reckless: ethics as a product capability, not a poster. Build with intention, measure what matters, and make it easy for customers to understand, control, and benefit from your AI. That’s how we ship innovation that stays trusted—at scale.

Inspired by this post on Product School.

January 15, 2026
How We Built an AI Career Co‑pilot that Turns Knowing into Doing for Disadvantaged Students

How do you help disadvantaged students take action on opportunities they don't even know exist? That question has been top of mind for me as I’ve explored how AI can augment—not replace—human mentorship. Recently, I dug into the work behind Zero Gravity, a UK-based platform using mentoring, community, and learning pathways to unlock elite career opportunities for state school students. Their approach reframed a core problem I care deeply about: the "knowing-doing gap."

I sat down with Elliot Little (Product Manager) and Dan St. Paul (Software Engineer) from Zero Gravity to unpack how they’re tackling this gap with an AI career co‑pilot. They’ve intentionally positioned the system as an orchestrator, not an automation tool—bridging the space between knowing what to do and actually doing it. As a product leader, I see this as a powerful pattern for Generative AI: use AI to coordinate steps, personalize guidance, and empower action in moments where confidence and clarity are fragile.

What resonated most was the humility of their build journey. They started with grand visions of AI mentors and synthetic avatars, then scaled back to something simpler and more effective. The first prototype—a job suitability summary—didn’t deliver the "wow moment" they expected. And they discovered that hiding the "LLM magic" backfired—students needed to feel the personalization. That insight aligns with my own experience: users must perceive the value for trust and motivation to compound.

From a UX standpoint, the team chose text chat over voice input and leaned into guided prompts rather than empty text boxes. That decision lowered cognitive load and increased completion rates—classic product management tradeoffs that privilege momentum over novelty. In my view, this is what good AI product strategy looks like: invite action with structure, then expand autonomy as confidence grows.

The technical backbone is equally thoughtful. Multi‑month journeys require rigorous context window management to avoid exploding token counts and degrading quality. I appreciated their pragmatic toolkit: context management techniques like removing stale tool calls, summarizing history, exposing tools conditionally. They also used application logic rather than complex RAG architectures to manage tool availability and context freshness. This is the kind of disciplined engineering that keeps systems reliable at scale without overcomplicating the stack.

Model selection was fit‑for‑purpose, not one‑size‑fits‑all. They’re using different models for different tasks, including "GPT-5 Nano for structured outputs, lighter models for quick replies." That modularity enables speed and cost control while preserving high‑fidelity moments where structure matters most.

Safeguarding was treated as a first‑class concern—non‑negotiable when you’re building AI for 16‑year‑olds. Their safeguarding architecture pairs moderation endpoints with external verification via Unitary. They also invested in building a failure taxonomy through internal red team/green team exercises. This is AI risk management done right: define failure modes early, test ruthlessly, and wire safety into the product surface area—not just the model layer.

Evaluation was grounded in outcomes, not demos. The team focused on whether students progressed from insight to action: applying, interviewing, and engaging with mentors. That aligns with how I run eval‑driven development—ship narrowly, measure real behavior, and iterate toward a repeatable "wow moment" that students can actually feel.

Looking ahead, I’m excited by what’s next: long‑term memory management for multi‑year student journeys. It’s a hard problem—balancing privacy, provenance, and portability—but it’s precisely where an AI career co‑pilot can compound value over time. The vision is compelling: a resilient companion that remembers goals, adapts to context, and orchestrates the right next step.

If you want to dive deeper, you can listen to the full conversation on Spotify and Apple Podcasts:

Listen to this episode on: Spotify | Apple Podcasts

Resources mentioned:

Zero Gravity: https://zerogravity.co.uk/

Unitary – AI-powered content moderation: https://www.unitary.ai/

Blue Dot Impact AI Safety Course – free AI safety course Elliot recommended: https://bluedot.org/

My key takeaways: build AI that augments human relationships, not replaces them; don’t hide the personalization—let learners feel it; privilege application logic over unnecessary architectural complexity; and treat safety, context, and evaluation as product features, not afterthoughts. That’s how we bridge the "knowing-doing gap" with integrity and scale.

Inspired by this post on Product Talk.

January 8, 2026
PMs and Developers Need Different AI Metrics—Here’s How That Builds Faster, Better Products

I’ve sat in countless AI measurement debates and noticed a recurring gap. One major voice has been noticeably underrepresented in the AI measurement conversation: the product manager (PM) that’s leading development. From experience, PMs and developers do need different measurement tools—and making those differences explicit is exactly what speeds up decisions and improves outcomes.

Developers optimize the model and system layer. Their toolkit centers on eval-driven development: offline evals, regression suites, red-teaming, latency and throughput monitoring, token cost tracking, and hallucination rate reduction. On the delivery side, engineering teams watch DORA metrics alongside CI/CD performance to keep iteration fast and safe. When building LLM-backed experiences, they also care deeply about retrieval-first pipeline quality and context window management because those mechanics determine grounding, relevance, and consistency.

PMs, by contrast, own outcomes. We instrument user journeys end to end and define a clear north-star tied to value: activation, time-to-value, task success rate, retention analysis, support deflection, and revenue contribution. We rely on A/B testing frameworks and minimum detectable effect (MDE) planning to separate real impact from noise, and we consolidate behavioral signals in a unified analytics platform like Amplitude analytics and Pendo to understand adoption, friction, and cohort differences. This is the heart of product-led growth and continuous discovery: evidence, not anecdotes.

The fact that these toolboxes differ is a strength, not a weakness. Specialized metrics keep responsibilities crisp: developers guarantee model quality and reliability; PMs guarantee that quality translates into customer and business outcomes. What we need is an explicit metrics ladder that connects layers—model-level quality floors and SLOs, feature-level KPIs, and company-level results—so trade-offs are transparent and prioritization is principled.

In practice, I create a shared measurement contract for every AI initiative. It links eval sets to user-facing success criteria, defines acceptance thresholds, and spells out observability across the stack. We include governance from day one—AI risk management, privacy-by-design, and data governance—so we can scale responsibly without slowing teams down.

Here’s the AI product toolbox I give my teams: start with a concise value hypothesis; define a success rubric the customer would recognize; instrument the happy path and the failure path; plan experiments with MDE up front; segment results by persona and job-to-be-done; and close the loop with qualitative feedback inside the product via in-app guides, product tours, and lightweight surveys. For AI features specifically, add Agent Analytics for agentic AI, capture grounding sources for explainability, and log model/context inputs to make debugging and iteration repeatable. That way, LLMs for product managers stop being magic and start being manageable.

When we roll out a new assistant—whether a retrieval-augmented copilot or a voice AI agent—we set two dashboards: one for developers (eval pass rates, latency, context integrity, error budgets) and one for PMs (activation, task completion, deflection, satisfaction). The dashboards read differently by design, yet they are joined at the hip by shared definitions and experiment IDs. This lets us move quickly with confidence: engineering can tighten quality loops while product steers toward the outcome that matters most.

If you’re feeling the tension between model metrics and product metrics, don’t collapse them—connect them. Start with a thin slice, agree on 3–5 measurable outcomes, and let your evals and A/B tests work together. With a clear metrics ladder and a unified analytics platform, PMs and developers can each excel at their craft and still ship AI that customers love.

Inspired by this post on Pendo – Perspectives.

January 7, 2026
4 Costly Misconceptions About Building AI Agents—and How I Turn Them Into Wins

I’ve lost count of how many times I’ve been asked for a “quick AI agent” that can autonomously fix customer problems, write code, or run sales ops. The promise is intoxicating—and I get why. But in practice, sustainable impact comes from disciplined product thinking, not wishful automation. Drawing on my experience leading product for complex, agentic AI initiatives, I want to debunk four misconceptions I see repeatedly and share what actually works.

Misconception 1: AI agents are plug-and-play. The reality is that effective agentic AI behaves more like a new product line than a feature toggle. It needs clear job stories, domain grounding, tool access, and guardrails. I start by narrowing scope to one painful job to be done, then design AI workflows that reflect real constraints (SLAs, compliance, edge cases). From day one, I instrument with Agent Analytics and set up eval-driven development so we can see failure modes early and iterate with intent.

What consistently moves the needle is treating the agent like a teammate you onboard: define responsibilities, provide the right tools, and measure outcomes. I pair scripted validations with live evals, track containment rates and handoff quality, and balance precision/recall depending on the risk profile. This is slow to fast, not fast to broken.

Misconception 2: Bigger models make better agents. In my experience, architecture outperforms horsepower. A retrieval-first pipeline, tight context window management, and practical prompt engineering often beat an oversized model that hallucinates. Tool use matters more than model size: give the agent reliable APIs, clear schemas, and deterministic fallbacks. For LLMs for product managers, the play is to right-size the foundation model and invest in data quality, prompts, and evaluators that reflect your true acceptance criteria.

When I see erratic behavior, I don’t immediately swap models; I improve retrieval, prune irrelevant context, and clarify the agent’s planning loop. Most performance gains come from better state management and grounding rather than a pricier token budget.

Misconception 3: Agents replace teams. High-performing organizations design human-in-the-loop systems. I implement human review on high-risk actions, explicit escalation paths, and simple override mechanisms. That’s not just safety theater—it’s good product design. AI risk management and data governance are part of the product backlog, not an afterthought. In customer support ai strategy, for example, the agent drafts, a specialist approves, and the system learns from deltas to tighten future responses.

The social system matters as much as the technical one: clear role boundaries, audit trails, and feedback loops turn the agent into a force multiplier. Teams gain leverage without surrendering accountability.

Misconception 4: Shipping the agent equals success. Adoption is earned, not announced. I treat agent launches like any product-led growth motion: define activation events, remove friction with in-app guides and product tours, and A/B test prompts, tool choices, and UI affordances. We track time-to-value, task completion rate, and user trust signals (edits, undo patterns, and escalation requests). When we get those leading indicators right, retention follows.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

My playbook is simple and repeatable: frame the problem narrowly, ground the agent with the right tools and data, measure with eval-driven development and Agent Analytics, then grow adoption with a disciplined go-to-market inside the product. The agents that win don’t feel like magic—they feel dependable. That’s what customers trust, and that’s what scales.

Inspired by this post on Pendo – Best Practices.

January 4, 2026
3 Powerful Ways AI Is Rewriting Cybersecurity: Smarter Defense, Faster Response, Fewer Breaches

Every week, I watch the cybersecurity landscape shift under our feet. As a VP of Product Management, I’m responsible for building secure, resilient products—and that means understanding how artificial intelligence is transforming the way IT teams defend, respond, and even anticipate attacks.

Learn the ways in which AI is transforming both cybersecurity offense and defense for IT teams.

First, AI supercharges threat detection and prevention. Pattern-recognition models now sift through endpoint telemetry, identity signals, and network flows to surface anomalies in near real time. In practice, that means fewer false positives, faster prioritization, and earlier containment. We’re pairing behavioral analytics with enrichment from our SIEM/EDR stack so analysts get a ranked, explainable view of risk instead of a noisy alert queue—directly improving mean time to detect and laying the groundwork for scalable threat detection and response.

Second, AI accelerates incident response. We’ve embedded LLM-powered copilots into our SOC workflows to summarize alerts, propose next-best actions, and auto-generate draft remediation steps from playbooks. Orchestration then executes routine tasks—isolating endpoints, rotating credentials, updating tickets—while keeping a human-in-the-loop for approvals. To keep this safe, we use privacy-by-design principles, a retrieval-first pipeline for authoritative playbook content, and eval-driven development to measure precision/recall on suggested actions. The result is meaningful reduction in mean time to recover and more consistent incident management.

Third, the offense is getting smarter—and we need to be honest about it. Adversaries use gen AI to craft targeted spear-phishing, deepfake executive voice notes, and polymorphic malware that evades signature-based tools. We counter by red-teaming with AI, deploying deception tech to waste attacker cycles, and hardening identity as the new perimeter (MFA, conditional access, continuous risk scoring). Education matters, too: when employees see how convincing AI-generated lures have become, phishing reports spike and successful compromise rates drop.

None of this works without strong governance. We treat AI like any high-impact capability: rigorous data governance, model access controls, and AI risk management across the lifecycle. We log model prompts and outputs, restrict sensitive data via contextual policies, and continuously test for drift and bias. This is as much an IT leadership challenge as it is a technical one—clear ownership, well-defined runbooks, and regular tabletop exercises make the difference between resilience and chaos.

If you’re getting started, I recommend a focused 90-day plan: identify one high-signal detection use case, one response playbook ripe for automation, and one employee risk area (usually phishing) for immediate uplift. Instrument everything—latency, precision/recall, MTTR—and iterate with a cross-functional group spanning security engineering, SRE, and product management leadership. With disciplined AI strategy and guardrails in place, you can move faster, reduce noise, and stay ahead of adversaries without compromising data or trust.

Inspired by this post on Pendo – Perspectives.

January 4, 2026
AI Context Pulling Playbook: How I Get LLMs and Teams to Collaborate for Better Product Outcomes

In my role leading product, I’ve learned that the fastest path to higher-quality deliverables from large language models (LLMs) is not a clever prompt—it’s rigorous context. I call the practice AI context pulling: a repeatable way to assemble, compress, and structure the most relevant knowledge before the model ever starts generating. Done well, it turns generative AI into a dependable partner for discovery, prioritization, and execution.
AI context pulling means I proactively gather the right artifacts (customer insights, analytics, strategy, constraints), manage context windows intentionally, and shape the model’s task with clear objectives and guardrails. This reduces hallucinations, improves alignment, and creates traceability back to sources—critical for product management leadership and stakeholder trust.
Learn a new way in which product professionals can collaborate with AI to get even better results on their projects.
Here’s the simple flow I use: first, I define the intent (e.g., “synthesize discovery interviews for a positioning brief”). Next, I inventory relevant context: top customer pains from product discovery, usage patterns from Amplitude analytics, recent support trends from Intercom, and any constraints from our product strategy. Then I run a retrieval-first pipeline to select only the most pertinent slices—favoring recency, representativeness, and canonical sources.
Because context window management matters, I compress long documents into short, source-cited summaries and keep raw excerpts handy when nuance is important. My prompts follow a consistent structure: role and objective, constraints and audience, curated context, the explicit ask, preferred output format, and a brief self-check (e.g., “cite sources and flag uncertainty”). This is prompt engineering for reliability, not theatrics.
A quick example: when drafting a one-page feature brief, I attach three items—the product strategy paragraph that sets the frame, a usage cohort analysis that highlights who’s affected, and five verbatim customer quotes. I ask the LLM to propose a problem statement, success criteria, and a shortlist of solution hypotheses, each tied to a cited piece of evidence. The result is a grounded, decision-ready artifact I can share with product trios and stakeholders.
Tooling-wise, I keep it pragmatic. A lightweight retrieval-first pipeline (embeddings, metadata filters, and recency rules) ensures the LLM pulls what matters. I version prompts and contexts together so I can run quick A/B testing on output quality. And I log decisions and sources to support eval-driven development and continuous discovery.
Common pitfalls are avoidable. Too little context yields generic answers; too much overwhelms the model. Stale docs can mislead; curate aggressively. Vague asks invite fluffy prose; specify outcomes, audiences, and formats. If the task is high risk, I bias toward smaller, well-cited outputs and expand iteratively with human review in the loop.
To measure impact, I track rework rate, review time, and stakeholder alignment on first pass. Over time, teams adopting AI context pulling report clearer artifacts, faster synthesis cycles, and more confident decisions—because every recommendation traces back to evidence. That’s how humans and LLMs truly collaborate better: we provide the right context, and the model amplifies our judgment.
If you’re ready to operationalize this, start by templatizing your most common product workflows—discovery synthesis, roadmap rationale, and release notes—and attach small, high-signal context packs. With a retrieval-first mindset and disciplined prompting, AI becomes an extension of your product craft, not a gamble.

Inspired by this post on Pendo – Perspectives.

January 4, 2026
From Idea to Impact: How AI Supercharges Product Design, Testing, and Time-to-Value

AI is changing how I build products, not by replacing designers or researchers, but by amplifying the quality and speed of what our product trios can deliver. The real breakthrough isn’t a single tool; it’s the way genAI and traditional methods combine into a tighter discovery–design–delivery loop that shortens time-to-value without sacrificing rigor.

Learn how Pendo’s product design team is using genAI and traditional tools to speed up design and development.

In practice, that’s exactly the pattern I see working across my teams: we treat genAI as part of the AI product toolbox—great for rapid exploration, structured synthesis, and test preparation—while we rely on our proven techniques to validate outcomes. For early-stage concepting, I use prompt engineering to generate multiple storyboard options and interaction flows in minutes, then refine those outputs with our design system for alignment and accessibility. It’s a pragmatic “gen ai for product prototyping” approach that lets us compare more alternatives, faster, with better signal.

On the testing front, AI accelerates everything around A/B testing without diluting statistical discipline. We draft hypotheses, define success metrics, and estimate minimum detectable effect (MDE) with guardrails, then deploy variants via feature flags in CI/CD. That pairing—LLMs for product managers plus eval-driven development—keeps experiments reproducible while boosting deployment frequency. The outcome is fewer opinions, more evidence, and a tighter feedback loop from build to learn.

Research goes from weeks to days when we combine a retrieval-first pipeline for qualitative data with strong data governance. I’ll ingest interview notes, support tickets, and session transcripts to cluster themes, then pressure-test the clusters with live customer calls. Privacy-by-design and AI risk management remain non-negotiable: we redact sensitive fields, constrain context windows, and keep a human-in-the-loop for decisions that affect user experience or compliance.

Where analytics meets adoption, tools like in-app guides and product tours help us translate insights into behavior change. I’ll prototype a flow, auto-generate guidance variants, and run controlled rollouts to target segments, measuring activation and retention analysis in parallel. This is product-led growth in action: discover the friction, design the intervention, instrument the journey, and validate outcomes with unified analytics.

Organizationally, empowered product teams and continuous discovery make the difference. Our product trios work from outcomes vs output OKRs, pairing competitive differentiation with product strategy to keep bets focused. We meet weekly to review experiment readouts, model trade-offs with the Kano Model, and update product roadmapping and sprint planning based on verified learning—never vibes alone.

If you’re getting started, begin with one workflow—say, prototype generation plus structured experiment design—and measure impact across cycle time, experiment throughput, and decision quality. Layer in communities of practice to share prompt patterns, establish eval baselines, and codify what “good” looks like. The companies winning with AI aren’t chasing shiny objects; they’re building a repeatable system that turns curiosity into customer value.

Inspired by this post on Pendo – Best Practices.

January 4, 2026
Beyond Digital: How I Drive AI Transformation to Build Adaptive, Intelligent Organizations

Digital transformation set the foundation, but it’s no longer sufficient. In my work leading product teams, I’ve learned that real competitive advantage now comes from building systems that perceive, learn, and adapt—end to end, across the product lifecycle and the business operating model.

AI transformation goes beyond automation to create adaptive, intelligent organizations. Discover why it’s the next imperative and how to measure success.

Why is this the next imperative? Customers expect intelligent experiences, not just digitized workflows. Markets are shifting faster than roadmaps, and teams need systems that learn in production. For me, AI Strategy starts with a clear value thesis: where can intelligence amplify customer outcomes and compound business impact—whether in onboarding, customer support, or core product differentiation.

Practically, I frame AI transformation as a capability stack: data governance and privacy-by-design at the foundation; a retrieval-first pipeline to ground models in trusted context; agentic AI and AI workflows to orchestrate actions; and eval-driven development to continuously measure quality, safety, and relevance. Layered on top are operating rhythms—outcomes vs output OKRs, rapid experimentation, and incident management—that keep shipping disciplined and responsible.

I start with product discovery. Together with product trios, we target moments where intelligence removes friction or unlocks new value. We translate those opportunities into crisp outcomes (activation, time-to-first-value, resolution rate) and instrument them from day one. In customer support, for example, a customer support ai strategy might blend LLMs for product managers with retrieval-first grounding to deliver accurate, brand-safe answers and escalate seamlessly when needed.

On architecture, I prioritize context window management and robust integrations. CRM integration and event streams from tools like Intercom, HubSpot, Pendo, and a unified analytics platform provide the signals AI needs to adapt in real time. Prompt engineering patterns, guardrails, and privacy-by-design controls ensure responses remain trustworthy and compliant. When applicable, I explore agentic AI to orchestrate multi-step tasks with clear constraints and auditability.

Delivery is where transformation becomes measurable. I combine CI/CD practices with DORA metrics (deployment frequency, lead time, change failure rate, MTTR) to keep iteration fast and safe. On the product side, A/B testing with a minimum detectable effect (MDE) protects rigor, while eval-driven development tracks model accuracy, hallucination rates, and policy adherence before and after release. I tie these to business metrics like user activation, retention analysis, and support resolution time to ensure we’re shipping outcomes, not just output.

Governance is non-negotiable. AI risk management, regulatory compliance, and data governance anchor every phase—from dataset curation to prompt libraries and model routing. Threat detection and response and incident management processes are integrated so we can respond quickly when behavior drifts or new risks emerge.

Transformation also means evolving how teams work. I invest in empowered product teams, continuous discovery, and developer evangelism to spread best practices across domains. We share playbooks, reusable CustomGPT workflows, and an AI product toolbox to scale patterns like retrieval-first pipelines and safe prompt engineering across the portfolio.

The outcome is not just smarter features; it’s a more adaptive business. With clear OKRs, reliable telemetry, and responsible guardrails, AI becomes a force multiplier for product strategy and execution. If you’re moving beyond digital toward intelligence, start small, measure relentlessly, and let outcomes guide the journey.

Inspired by this post on Pendo – Perspectives.

January 4, 2026
4 Critical AI Risks Every CIO Must Tackle Now—and a Practical Playbook to Mitigate Them

I spend a lot of time with CIOs and IT leaders who are moving fast on generative AI. The momentum is real, but so are the risks. When AI touches core workflows, data, and customer experiences, we need a clear, pragmatic plan that blends AI Strategy with disciplined product management leadership and IT governance.

Learn about the risks that AI poses to IT teams, and how they can mitigate them.

Here are the four risks I see most often—and the playbook I use to de-risk delivery while preserving speed and innovation.

Risk #1: Shadow AI and data leakage. Teams experiment with unapproved tools, and sensitive data ends up in prompts, logs, or third-party services. Without strong data governance and privacy-by-design, even a small proof of concept can create outsized exposure.

How I mitigate it: start with an AI acceptable-use policy, data classification, and clear guardrails on what can be prompted. Deploy a redaction layer and secrets management before any model call. Favor a retrieval-first pipeline so models reason over vetted internal knowledge rather than raw or personal data. Conduct vendor due diligence and DPAs up front, and centralize audit logs to support regulatory compliance and incident response.

Risk #2: Hallucinations and unreliable outputs. LLMs are probabilistic; they can fabricate citations, numbers, or steps. In customer support and internal operations, this erodes trust and creates rework—especially when teams assume model answers are authoritative.

How I mitigate it: adopt eval-driven development with task-specific test sets, reference answers, and pass/fail thresholds that gate CI/CD. Ground models with retrieval, constrain outputs with schemas, and keep a human-in-the-loop for high-risk actions. A/B testing, error taxonomies, and continuous monitoring turn model behavior into measurable, improvable Web Vitals for AI reliability.

Risk #3: Expanded attack surface. Prompt injection, data exfiltration, supply chain risks in model providers, and insecure connectors can undermine existing cybersecurity controls. Traditional threat models often miss these new interaction patterns.

How I mitigate it: treat AI as a first-class asset in threat detection and response. Implement input/output filtering, allow/deny lists, content moderation, and strict isolation of tools and connectors. Red team prompts and tools regularly, rotate credentials, and codify runbooks with SRE and incident management for fast containment. Apply least privilege to agents, APIs, and vector stores, and monitor for anomalous tool-use.

Risk #4: Compliance, bias, and auditability gaps. As AI scales, questions about explainability, fairness, data residency, and retention move from theoretical to board-level. Without traceability, it’s hard to satisfy audits or respond to regulators.

How I mitigate it: embed privacy-by-design from the first sprint—data minimization, consent, purpose limitation, and retention controls. Maintain model cards, versioning, and lineage for prompts, datasets, and parameters. Centralize audit logs, set policies for high-risk use cases, and run periodic compliance reviews with security and legal. Cross-functional communities of practice keep changes aligned across product, engineering, and IT Leadership.

Operationally, I anchor AI initiatives to outcomes vs output OKRs, use empowered product teams and product trios to balance feasibility, value, and risk, and integrate model changes into CI/CD with quality gates. This creates a repeatable mechanism to ship safely, learn quickly, and scale what works.

If you’re standing up new AI workflows or hardening what you already have in production, this playbook gives you a practical path: drive adoption confidently, protect your data, and stay compliant while maintaining competitive velocity.

The bottom line: AI risk management isn’t a brake on innovation—it’s how we earn the right to go faster.

Inspired by this post on Pendo – Perspectives.

January 3, 2026
My Proven Experimentation Playbook for AI PMs: Faster Learning, Safer Launches, Bigger Wins

I build AI products with a simple conviction: disciplined experimentation beats intuition. Over the years, I’ve refined a practical playbook that helps my teams learn faster, reduce risk, and turn every release into a smarter next step.

Product experimentation isn’t luck; it’s a method. Learn how top AI product managers test, measure, and grow smarter with every release.

I begin every effort with a crisp hypothesis, an expected user or business outcome, and unambiguous success criteria tied to outcomes vs output OKRs. Before writing a line of code, I define primary metrics and guardrails so we know what “good” looks like—and what to stop.

When the change affects UX, pricing, or activation flows, I favor A/B testing with the statistical rigor to back decisions. We calculate the minimum detectable effect (MDE), choose appropriate randomization units, and pre-register the analysis plan to avoid p-hacking. This gives the team the confidence to scale wins and sunset underperformers quickly.

AI features demand a tailored approach, so I run eval-driven development before any user sees a variant. We curate golden datasets, score candidate prompts and models, and stress-test failure modes. This is where LLMs for product managers matters: prompt templates, context window management, and a retrieval-first pipeline are all evaluated for quality, latency, and cost-to-serve. I treat “hallucination rate,” safety violations, and bias as first-class metrics under AI risk management.

To de-risk launches, we ship behind feature flags with CI/CD, monitor DORA metrics, and roll out in stages. Product trios own problem framing to solution delivery, which shortens feedback loops and preserves accountability. If early signals drift from our hypotheses, we pause, adjust, and re-run—no sunk-cost thinking.

Measurement is non-negotiable. I instrument user journeys end-to-end with Amplitude analytics, track activation and retention analysis, and map behavior to learning objectives. We consolidate logs and events into a unified analytics platform so qualitative insights from customer research pair cleanly with quantitative trends.

Continuous discovery keeps the engine running. Weekly customer conversations, in-product feedback, and lightweight prototypes ensure we validate needs, not just solutions. The output flows into product discovery, product roadmapping and sprint planning, and a reusable AI product toolbox that scales across teams.

Finally, I protect the culture that makes experimentation work: we celebrate invalidated hypotheses, document decisions, and optimize for outcomes over output. That’s how empowered product teams sustain product-led growth—even as complexity grows.

If you’re building AI features today, adopt this playbook to maximize learning velocity, minimize risk, and compound advantage. The method is straightforward: form strong hypotheses, test with rigor, measure what matters, and let evidence—not HiPPOs—guide the roadmap.

Inspired by this post on Product School.

December 31, 2025
The New AI Playbook for Product Portfolio Optimization: Slash Complexity, Boost ROI

The most valuable lesson I’ve learned leading product organizations is that portfolio choices make or break outcomes. In an era of infinite requests and finite teams, the question isn’t what we could build—it’s what we must build next. That’s why I’m codifying a pragmatic, AI-driven playbook to optimize the product portfolio while staying true to outcomes, not output.

AI-powered product portfolio optimization is here. Explore strategies and tools helping product leaders manage complexity and boost ROI.

My starting point is a data backbone that connects strategy to reality. I aggregate product usage, revenue by segment, cost-to-serve, retention cohorts, and support signals into a unified analytics platform, then layer a retrieval-first pipeline so LLMs can reason over clean context. Instrumentation matters: Amplitude analytics, Pendo, and in-app guides provide the behavioral and activation signals that make prioritization measurable.

From there, I translate strategy into an objective decision system. I express outcomes vs output OKRs, align initiatives to value proposition and competitive differentiation, and classify opportunities with the Kano Model. LLMs for product managers help cluster voice-of-customer at scale; with thoughtful prompt engineering and AI workflows, I can map themes to jobs-to-be-done, quantify demand, and de-duplicate asks across stakeholders.

Execution hinges on evidence. I run A/B testing with a clear minimum detectable effect (MDE), pair it with eval-driven development for AI features, and ship through CI/CD while tracking DORA metrics. This closes the loop between product roadmapping and sprint planning and real-world performance—activation, retention analysis, and Web Vitals inform the next set of portfolio bets.

Trust is a feature, so governance is built-in. Privacy-by-design, data governance, and AI risk management guide how we store, prompt, and evaluate models. I apply guardrails to sensitive workflows and define success metrics that balance short-term ROI with long-term resilience and regulatory compliance.

The operating model matters as much as the models themselves. Product trios and empowered product teams run continuous discovery, pressure-test assumptions in QBRs vs OKRs, and make trade-offs visible. Stakeholder management becomes easier when the portfolio narrative is anchored in transparent scenarios and shared metrics.

If you’re getting started, here’s my flow: unify data, define outcomes, segment opportunities, simulate scenarios, and test fast. Use LLMs to synthesize signals you’d never humanly read, then make one focused bet per team that moves a measurable KPI. Rinse, learn, and reallocate—portfolio optimization is a living system, not an annual meeting.

Ultimately, the promise of this new playbook is simple: less noise, sharper focus, and compounding ROI. By pairing AI Strategy with disciplined product management leadership, we can manage complexity with clarity—and consistently build what matters most.

Inspired by this post on Product School.

December 29, 2025
Monetizing AI with Confidence: Proven Models, Smart Pricing, and ROI You Can Defend

I’ve learned the hard way that shipping an impressive AI demo is not the same as creating a durable revenue engine. In my role leading product strategy, I focus on one goal: connect AI capabilities to measurable customer outcomes, then price and package them so both value and margins are visible and defensible.

Monetizing AI features into profit isn’t trivial. Here are some clear strategies for capturing and pricing AI products and how to monetize with returns.

First, I clarify the business model. Add-on AI packs work when the value is concentrated in a specific workflow (for example, automated summarization or AI copilot assistance). Tiered packaging helps when AI elevates the overall experience across many features. Usage-based or consumption SaaS pricing is ideal when value scales with volume—tokens, documents processed, calls handled, or agents invoked—because it aligns price to realized outcomes.

Next, I align pricing mechanics with the customer’s value story. I anchor price against the baseline they know: hours saved, conversions gained, cases deflected, or risk reduced. Then I set floors based on unit economics—model inference, vector storage, and orchestration costs—so gross margins remain healthy as usage grows. Clear guardrails (quotas, rate limits, and context window management) prevent surprise bills and keep cost-to-serve predictable.

Packaging is where monetization becomes intuitive. I gate high-cadence, high-compute features behind premium tiers, and I expose quick wins (like smart suggestions) in core tiers to accelerate activation. For enterprise, I bundle governance, audit logs, data controls, and “privacy-by-design” features to justify step-up pricing and reduce procurement friction.

To sustain ROI, I run an eval-driven development loop. I define quality metrics (accuracy, helpfulness, latency, safety) and instrument the retrieval-first pipeline so I can isolate where value is created or lost. This lets me right-size models, tune prompts, and swap components without compromising outcomes or margins—critical for LLMs for product managers who must balance experience and cost.

Measurement is non-negotiable. I track activation, time-to-first-value, weekly engaged AI users, and feature-level retention. For revenue impact, I attribute uplift through A/B testing and minimum detectable effect thresholds, measuring conversion lift, ticket deflection, and cycle-time reductions. When customers see these numbers in their own dashboards, procurement turns into partnership.

Risk and compliance are part of the product, not an afterthought. I build in AI risk management, data governance, and red-teaming from day one. Clear data boundaries, human-in-the-loop controls, and transparent disclosures protect end users and make enterprise legal teams our allies rather than blockers.

Go-to-market matters as much as the model. I use product-led growth tactics—free AI credits, transparent meters, and in-app guides—to let users feel the value before the paywall. Sales enablement centers on the value proposition: faster outcomes, higher quality, and lower total cost of ownership, not just “gen ai” for its own sake. Pricing pages should showcase tiers, usage bands, and outcomes, eliminating guesswork.

Here’s the simple playbook I follow: validate the problem with continuous discovery, instrument the workflow, pilot with generous caps, and collect willingness-to-pay signals early. Then iterate the price meter, refine units of value (documents, messages, or actions), and align SKUs to buyer personas. Over time, I introduce agentic AI capabilities as premium modules when they demonstrably reduce steps or automate entire objectives.

When AI monetization works, it feels effortless to customers because the price mirrors the outcome. When it doesn’t, it’s usually because packaging hides value, pricing ignores unit economics, or ROI isn’t visible. By grounding strategy in value metrics, consumption-aware pricing, and rigorous evaluation, I’ve found we can scale AI revenue with confidence—and keep both customers and margins happy.

Inspired by this post on Product School.

December 22, 2025