2026 is closer than it feels, and the signals are already clear. I’ve been synthesizing what I’m seeing across empowered product teams, boards, and cross-functional partners into a practical view of what matters next. A sharp look at product management trends for 2026. Not guesses, but signals from top product leaders shaping how PMs will actually work next.
In this analysis, I distill eleven shifts that are changing the craft—from outcomes vs output OKRs and continuous discovery to stronger product strategy and tighter product roadmapping and sprint planning. The throughline is simple: prioritize customer value, ship with focus, and measure what moves the business. These aren’t headline trends; they’re working patterns I’m seeing across high-performing organizations.
AI is no longer a side project—it’s part of the product manager’s core toolkit. Agentic AI, LLMs for product managers, and trustworthy AI workflows are accelerating discovery, sharpening problem framing, and enabling faster iteration. The best teams pair this with disciplined evaluation and experimentation, so insight compounds without sacrificing safety, privacy, or product quality.
Execution is getting crisper through product trios and stronger stakeholder management. When design, product, and engineering co-own discovery and delivery, teams reduce handoffs and increase clarity. That alignment translates into better prioritization, fewer context-switches, and a roadmap that reflects real trade-offs—not wish lists.
On growth, product-led growth remains a durable engine when it’s anchored in a compelling value proposition and instrumented end-to-end. Clear activation moments, in-app guides, and thoughtful product tours outperform brute-force acquisition. When we connect these motions back to product strategy and the roadmap, we create a repeatable loop that compounds adoption and retention.
Governance and trust are now table stakes. Privacy-by-design, data governance, and a pragmatic approach to regulatory compliance protect both users and velocity. Teams that build these practices into their operating model move faster because they avoid late-stage rework and maintain stakeholder confidence.
If you’re leading a product org—or aspiring to—this is your field guide to 2026. I’ll unpack where these shifts are strongest, how to apply them in your context, and the pitfalls to avoid. The aim is to give you clear language, concrete practices, and a sharper edge as you shape what your team builds next.
I’ve spent the last few years turning AI from an intriguing demo into an operational advantage, and the clearest wins come when we treat agents as productized workflows—not toys. In practice, that means aligning agentic AI to a sharp product strategy, instrumenting everything, and scaling what works across the organization.
Learn how companies like Replit are consolidating workflows, creating one-person departments, and building systems for scale with Amplitude
When I talk about agentic AI, I’m focused on outcomes: fewer handoffs, faster cycle times, and measurable uplift in activation, retention, and NPS. The most successful rollouts start with a specific job-to-be-done, translate it into clear AI workflows, and then iterate with a tight feedback loop between data, design, and engineering.
My implementation playbook is simple and disciplined. First, choose a high-friction workflow and define success upfront. Second, make the build vs buy call on the foundation model, orchestration layer, and connectors. Third, establish AI risk management and safeguards early—before scale amplifies errors. Finally, run small, eval-driven releases and promote what performs.
Instrumentation is where the leverage compounds. With Amplitude analytics as a unified analytics platform, I design purposeful events (agent intent, tool calls, resolution state, human handoff), map funnels from user input to agent outcome, and cohort users by context to pinpoint lift. This gives me an honest read on where agents help, where they hinder, and what to tune next.
The “one-person departments” concept isn’t about doing more with less at all costs; it’s about assembling a tight loop of product management leadership, data, and automation so one operator can own a business outcome end-to-end. An agent handles the repeatable work, while the human focuses on judgment, edge cases, and continuous improvement that compounds.
As we scale, I look for platform scalability patterns: shared tools and policies, reusable prompt libraries, standardized evaluation suites, and consistent governance. That structure keeps agent performance predictable while preserving speed, and it aligns beautifully with product-led growth when agents are embedded directly in the product experience.
If you’re starting now, begin with a single, valuable workflow. Instrument it thoroughly with Amplitude analytics, make decisions from the data you see—not the demos you remember—and expand only after you’ve proven uplift. Iteration beats ambition here: agentic AI rewards teams who measure relentlessly and scale only what truly works.
Inspired by this post on Amplitude – Perspectives.
Support tickets are the rawest signal of product truth. Leading product teams at HighLevel, I’ve learned that the fastest way to build what customers value is to transform frontline conversations into a repeatable, data-driven system for discovery, prioritization, and execution.
What if your support and product teams could unlock CX insights to turn every ticket into strategic product intelligence? Explore how.
Here’s the operating system I rely on. First, I connect our support stack (think Intercom and our CRM integration) into a unified analytics platform so every conversation, tag, and resolution is queryable. I don’t just count tickets—I segment them by product area, customer segment, lifecycle stage, and revenue impact to reveal patterns that roadmaps can act on.
Next, we standardize a shared taxonomy. Agents apply concise, high-signal labels (problem type, severity, intent), and we augment that with AI-driven auto-tagging to reduce noise and improve recall. The result is trustworthy “voice of the customer” data that product managers and support leaders can both stand behind.
Prioritization then becomes rigorous and fair. I weight themes by severity, frequency, ARR exposure, and time-to-value, and tie them directly to outcomes vs output OKRs. Amplitude analytics helps me quantify impact—what’s breaking activation, what’s dragging conversion, what drives retention analysis—so the backlog reflects business outcomes, not opinions.
Discovery is continuous by design. Product trios (PM, design, engineering) run weekly reviews of the highest-signal themes, recruit users straight from recent tickets, and prototype solutions quickly. We validate ideas with A/B testing when appropriate and ship targeted in-app guides to reduce confusion before it becomes a ticket.
Crucially, we close the loop. When we release a fix or improvement, we notify affected customers and the agents who flagged the issue. We track downstream effects—ticket deflection, CSAT, feature adoption, and time-to-resolution—so everyone sees how customer support ai strategy accelerates product-led growth.
This approach also builds culture. Empowered product teams treat support as a strategic partner, not a cost center. Agents become co-creators of the roadmap, and PMs gain a steady stream of product discovery opportunities grounded in real user outcomes.
If you’re getting started, a simple 30-60-90 can help: in 30 days, unify the data and agree on taxonomy; in 60, instrument dashboards and adopt a weekly insights ritual; in 90, align priorities to OKRs, launch targeted fixes, and measure business impact. That’s how tickets turn into product truth—and how CX insights drive compounding wins.
Inspired by this post on Amplitude – Perspectives.
How do you help disadvantaged students take action on opportunities they don't even know exist? That question has been top of mind for me as I’ve explored how AI can augment—not replace—human mentorship. Recently, I dug into the work behind Zero Gravity, a UK-based platform using mentoring, community, and learning pathways to unlock elite career opportunities for state school students. Their approach reframed a core problem I care deeply about: the "knowing-doing gap."
I sat down with Elliot Little (Product Manager) and Dan St. Paul (Software Engineer) from Zero Gravity to unpack how they’re tackling this gap with an AI career co‑pilot. They’ve intentionally positioned the system as an orchestrator, not an automation tool—bridging the space between knowing what to do and actually doing it. As a product leader, I see this as a powerful pattern for Generative AI: use AI to coordinate steps, personalize guidance, and empower action in moments where confidence and clarity are fragile.
What resonated most was the humility of their build journey. They started with grand visions of AI mentors and synthetic avatars, then scaled back to something simpler and more effective. The first prototype—a job suitability summary—didn’t deliver the "wow moment" they expected. And they discovered that hiding the "LLM magic" backfired—students needed to feel the personalization. That insight aligns with my own experience: users must perceive the value for trust and motivation to compound.
From a UX standpoint, the team chose text chat over voice input and leaned into guided prompts rather than empty text boxes. That decision lowered cognitive load and increased completion rates—classic product management tradeoffs that privilege momentum over novelty. In my view, this is what good AI product strategy looks like: invite action with structure, then expand autonomy as confidence grows.
The technical backbone is equally thoughtful. Multi‑month journeys require rigorous context window management to avoid exploding token counts and degrading quality. I appreciated their pragmatic toolkit: context management techniques like removing stale tool calls, summarizing history, exposing tools conditionally. They also used application logic rather than complex RAG architectures to manage tool availability and context freshness. This is the kind of disciplined engineering that keeps systems reliable at scale without overcomplicating the stack.
Model selection was fit‑for‑purpose, not one‑size‑fits‑all. They’re using different models for different tasks, including "GPT-5 Nano for structured outputs, lighter models for quick replies." That modularity enables speed and cost control while preserving high‑fidelity moments where structure matters most.
Safeguarding was treated as a first‑class concern—non‑negotiable when you’re building AI for 16‑year‑olds. Their safeguarding architecture pairs moderation endpoints with external verification via Unitary. They also invested in building a failure taxonomy through internal red team/green team exercises. This is AI risk management done right: define failure modes early, test ruthlessly, and wire safety into the product surface area—not just the model layer.
Evaluation was grounded in outcomes, not demos. The team focused on whether students progressed from insight to action: applying, interviewing, and engaging with mentors. That aligns with how I run eval‑driven development—ship narrowly, measure real behavior, and iterate toward a repeatable "wow moment" that students can actually feel.
Looking ahead, I’m excited by what’s next: long‑term memory management for multi‑year student journeys. It’s a hard problem—balancing privacy, provenance, and portability—but it’s precisely where an AI career co‑pilot can compound value over time. The vision is compelling: a resilient companion that remembers goals, adapts to context, and orchestrates the right next step.
If you want to dive deeper, you can listen to the full conversation on Spotify and Apple Podcasts:
Listen to this episode on: Spotify | Apple Podcasts
Blue Dot Impact AI Safety Course – free AI safety course Elliot recommended: https://bluedot.org/
My key takeaways: build AI that augments human relationships, not replaces them; don’t hide the personalization—let learners feel it; privilege application logic over unnecessary architectural complexity; and treat safety, context, and evaluation as product features, not afterthoughts. That’s how we bridge the "knowing-doing gap" with integrity and scale.
I’ve sat in countless AI measurement debates and noticed a recurring gap. One major voice has been noticeably underrepresented in the AI measurement conversation: the product manager (PM) that’s leading development. From experience, PMs and developers do need different measurement tools—and making those differences explicit is exactly what speeds up decisions and improves outcomes.
Developers optimize the model and system layer. Their toolkit centers on eval-driven development: offline evals, regression suites, red-teaming, latency and throughput monitoring, token cost tracking, and hallucination rate reduction. On the delivery side, engineering teams watch DORA metrics alongside CI/CD performance to keep iteration fast and safe. When building LLM-backed experiences, they also care deeply about retrieval-first pipeline quality and context window management because those mechanics determine grounding, relevance, and consistency.
PMs, by contrast, own outcomes. We instrument user journeys end to end and define a clear north-star tied to value: activation, time-to-value, task success rate, retention analysis, support deflection, and revenue contribution. We rely on A/B testing frameworks and minimum detectable effect (MDE) planning to separate real impact from noise, and we consolidate behavioral signals in a unified analytics platform like Amplitude analytics and Pendo to understand adoption, friction, and cohort differences. This is the heart of product-led growth and continuous discovery: evidence, not anecdotes.
The fact that these toolboxes differ is a strength, not a weakness. Specialized metrics keep responsibilities crisp: developers guarantee model quality and reliability; PMs guarantee that quality translates into customer and business outcomes. What we need is an explicit metrics ladder that connects layers—model-level quality floors and SLOs, feature-level KPIs, and company-level results—so trade-offs are transparent and prioritization is principled.
In practice, I create a shared measurement contract for every AI initiative. It links eval sets to user-facing success criteria, defines acceptance thresholds, and spells out observability across the stack. We include governance from day one—AI risk management, privacy-by-design, and data governance—so we can scale responsibly without slowing teams down.
Here’s the AI product toolbox I give my teams: start with a concise value hypothesis; define a success rubric the customer would recognize; instrument the happy path and the failure path; plan experiments with MDE up front; segment results by persona and job-to-be-done; and close the loop with qualitative feedback inside the product via in-app guides, product tours, and lightweight surveys. For AI features specifically, add Agent Analytics for agentic AI, capture grounding sources for explainability, and log model/context inputs to make debugging and iteration repeatable. That way, LLMs for product managers stop being magic and start being manageable.
When we roll out a new assistant—whether a retrieval-augmented copilot or a voice AI agent—we set two dashboards: one for developers (eval pass rates, latency, context integrity, error budgets) and one for PMs (activation, task completion, deflection, satisfaction). The dashboards read differently by design, yet they are joined at the hip by shared definitions and experiment IDs. This lets us move quickly with confidence: engineering can tighten quality loops while product steers toward the outcome that matters most.
If you’re feeling the tension between model metrics and product metrics, don’t collapse them—connect them. Start with a thin slice, agree on 3–5 measurable outcomes, and let your evals and A/B tests work together. With a clear metrics ladder and a unified analytics platform, PMs and developers can each excel at their craft and still ship AI that customers love.
Buy-in isn’t a single meeting; it’s a designed journey. Over the years leading product strategy at HighLevel, I’ve learned that the fastest way to earn durable support is to reduce uncertainty, align on outcomes, and create visible momentum. Explore how to get buy-in from stakeholders with practical strategies, clear communication tips, and proven methods used by the best. Here’s the 7-step playbook my teams and I rely on to move from idea to aligned action.
Step 1 — Anchor on outcomes, not outputs. I start by writing a crisp problem statement, the target customer, and the measurable outcome tied to our North Star metric. I translate this into outcomes vs output OKRs so every stakeholder can see the difference between what we’ll ship and what we intend to change. This framing keeps discussions grounded in impact, not features.
Step 2 — Map stakeholders and incentives. Effective stakeholder management begins with a living map: economic buyers, executive sponsors, influencers, and operators. I capture each person’s goals, risks, and decision cadence. When I speak to Finance, I foreground cost and runway; with Sales, I emphasize pipeline and win rate; for Customer Success, I speak to retention and NPS. Meeting stakeholders where they are builds trust quickly.
Step 3 — Co-create early with the product trio. I pull the product trios (PM, Design, Engineering) into continuous discovery with GTM partners to validate assumptions and de-risk the solution. This is where empowered product teams shine—rapid discovery sprints, early prototypes, and clear learning objectives. Co-creating exposes blind spots early and transforms critics into champions.
Step 4 — Socialize a narrative, not a deck. Before any formal review, I circulate a short narrative memo that ties our product strategy to a clear value proposition, competitive differentiation, and go-to-market strategy. I include options and trade-offs so stakeholders feel invited to shape the path, not just stamp approval. Pre-wiring conversations ensure that the “meeting” is simply the last 10% of the decision.
Step 5 — Back the story with data and a viable plan. I combine retention analysis, funnel metrics, and customer evidence to demonstrate opportunity size and risk reduction. Then I outline a phased approach with product roadmapping and sprint planning, milestones, and success metrics. I highlight the smallest viable bet that proves value fast, along with contingency paths if we learn something unexpected.
Step 6 — Design the decision. I define the decision we need, by whom, and by when. The decision doc includes the problem, options, risks, mitigations, and the explicit ask. I schedule 1:1s to address concerns, then run a focused review with clear roles and time-boxed discussion. Clarity about the decision—and the criteria—prevents drift and protects timelines.
Step 7 — Sustain momentum post-approval. After the green light, I convert the plan into execution cadences: weekly demos, transparent dashboards, and QBRs vs OKRs check-ins to reinforce outcomes. We celebrate learning milestones, not just launches, and keep stakeholders informed with concise updates that tie progress to the original outcomes and value proposition. Momentum is the best antidote to second-guessing.
Clear communication and a repeatable process turn buy-in from a hurdle into a habit. When stakeholders see a compelling narrative, credible evidence, and a path to value, they don’t just approve—they advocate. Follow these seven steps and you’ll build alignment faster, ship smarter, and strengthen trust across the organization.
Keeping employees informed and engaged isn’t just a communications challenge—it’s a product challenge. When we treat internal tools like products with clear activation moments, measurable outcomes, and continuous discovery, adoption moves from hope to habit. Over the years, I’ve seen small changes in how we onboard, communicate, and measure compound into dramatically higher engagement, better compliance, and faster time-to-value.
“How to improve onboarding, compliance, and internal communications within your employee tools.” That question guides my approach end to end—from the moment someone logs in for the first time to the day they become an expert, championing best practices across their team.
First, I personalize onboarding to accelerate user activation. I map the critical first actions and design a lightweight sequence of product tours and in-app guides that surfaces only what matters right now. Progressive disclosure, clear UX writing, and thoughtful tooltip design reduce cognitive load. I measure time-to-first-value, A/B test checklist microcopy to remove friction, and use Intercom or Pendo to deliver contextual walkthroughs by role, location, and permission level. Amplitude analytics helps me validate that the guided path leads to the intended activation event and sustained usage.
Second, I make compliance effortless and measurable. Instead of long trainings, I embed micro-learnings and policy nudges directly in the flow of work, with just-in-time prompts and short, scenario-based confirmations. I segment by role to avoid alert fatigue and localize where regulations require nuance. Completion rates, quiz accuracy, and time-to-complete are tracked alongside qualitative feedback. When compliance messaging underperforms, I run A/B testing on tone, timing, and format, then iterate until adherence is both higher and faster.
Third, I orchestrate internal communications as lifecycle messaging—not announcements. Employees get targeted release notes, role-specific tips, and in-app reminders aligned to their stage: new, adopting, proficient, or champion. I avoid channel sprawl by making the primary source of truth available in the product, then reinforcing it via email or chat only when necessary. CRM integration and audience rules ensure relevance, while a champions network and office hours create human touchpoints that deepen trust and accelerate adoption.
Fourth, I close the loop with analytics and continuous discovery. I instrument key events and run retention analysis to understand which behaviors predict long-term engagement. I look at cohorts before and after a new guide or product tour, and I compare lift in user activation and feature adoption over 14-, 28-, and 90-day windows. Amplitude analytics provides the behavioral picture; surveys, interviews, and passive feedback widgets explain the why. Together, these inputs power a product-led growth approach for internal tools—observable, repeatable, and improvable.
When teams ask where to start, I pilot one persona, one workflow, and one high-value outcome. I define the activation event, instrument it, launch a single targeted in-app guide through Pendo or Intercom, and A/B test the onboarding microcopy. Two weeks later, I review retention cohorts and completion data, talk to users, and either scale the pattern or iterate. That cadence builds credibility quickly because it ties every communication to a measurable result.
The payoff is tangible: faster onboarding, higher compliance, clearer internal communications, and employees who feel supported rather than overwhelmed. With disciplined messaging, smart instrumentation, and ongoing discovery, we can turn internal tools into catalysts for performance—and transform engagement from a campaign into a culture.
I lead product teams building travel and hospitality experiences, and one lesson keeps repeating: companies that measure what matters move faster. Benchmarks turn gut feel into grounded product strategy, making it clear where activation, conversion, and retention are underperforming—and where we can unlock outsized growth.
Discover exclusive data and strategies from our Product Benchmark Report. Compare the travel and hospitality industry’s performance across key product metrics.
When I evaluate a product line, I start with a simple model: attract, convert, delight, and retain. For travel and hospitality specifically, I focus on search-to-book conversion, onboarding completion, first-booking activation rate, time-to-book, average booking value, cancellation rate, support contact rate, DAU/MAU stickiness, repeat booking rate, and long-term retention. These key product metrics reveal friction in discovery and checkout flows, surface pricing and inventory gaps, and quantify loyalty.
From there, I assemble a test-and-learn plan. Using Amplitude analytics to instrument the funnel and Pendo for in-app guides and product tours, my teams design A/B testing with a clear minimum detectable effect (MDE), prioritize hypotheses, and execute rapid, weekly iterations. This is classic product-led growth: reduce cognitive load in onboarding, streamline search and filter UX, clarify policies before payment, and personalize reactivation nudges to improve user activation and retention analysis.
Benchmarks are only as trustworthy as the underlying data. I insist on strong data governance, privacy-by-design practices, and clear event taxonomies so that insights remain reliable across quarters and across markets. That foundation keeps our decisions defensible with stakeholders and regulators while accelerating delivery.
Finally, we translate insights into action with crisp product roadmapping and sprint planning. Cross-functional product trios align OKRs to the biggest benchmark gaps, and we review progress in weekly performance rituals so every experiment ladders up to strategy. This cadence helps teams stay empowered and keeps leadership focused on outcomes, not output.
If you’re building in travel and hospitality, use these benchmarks as your starting line and your ongoing scorecard. Calibrate targets against peers, double down on what moves the needle, and let the data guide bold, customer-centered bets. When teams rally around meaningful metrics, momentum compounds.
Inspired by this post on Amplitude – Perspectives.
Your team probably doesn’t lack discovery techniques. It loses discovery when delivery becomes urgent. Customer contact clusters around planning, the team commits to a solution, the calendar fills, and assumptions quietly harden into backlog items. Everyone stays busy, but no one can point to the customer evidence behind the next decision.
Durable continuous discovery is an operating rhythm, not a research phase. The goal isn’t to conduct more interviews. It is to shorten the distance between customer reality and the decisions shaping your product. A weekly rhythm owned by a product trio can reduce rework, sharpen strategy, and keep discovery alive while delivery continues.
See your real discovery system before changing it
Adding a recurring customer interview to the calendar won’t fix a decision process built around handoffs. If ideas arrive from executives, become requirements in product, move to design, and reach engineering as implementation work, the interview is an extra activity attached to the side of the system. It isn’t part of how the team decides.
Start by making the existing system visible. Map what actually happened, including the awkward shortcuts and informal approvals. Do not draw the process described in a playbook.
Spend 60 minutes drawing how your team decides what to build. Show where ideas enter, who shapes them, who approves them, where customers appear, and how the team decides whether the result worked.
Compare your drawing with the drawings made by product, design, and engineering. Differences are evidence that the team does not share the same decision model.
Audit every product decision from last week in a 30-minute session. Include small decisions, not just roadmap commitments.
For each decision, record who made it, what information informed it, and whether the team had direct customer input or received a secondhand interpretation.
Mark the places where discovery and delivery reconnect. A production problem, adoption signal, support request, or implementation constraint can create a new discovery question; it should not disappear into a separate queue.
This process map and decision audit gives you a baseline without turning discovery into a maturity score. Look for the mechanism behind the misses. Perhaps customer input arrives after commitment. Perhaps the product manager is the only person who interprets it. Perhaps the team can describe the solution but not the opportunity it addresses.
Track a compact baseline: how recently the team had direct customer contact, which current decisions include direct input, where cross-functional decisions become handoffs, and which active solutions lack a named customer opportunity. Do not set targets yet. First identify where evidence stops influencing action.
If the process looks reasonable but the habit still collapses, inspect the six prerequisite mindsets: outcome-oriented, customer-centric, collaborative, visual, experimental, and continuous. Turn them into diagnostic questions:
Outcome-oriented: Can the team name the customer or business change it is trying to create, or only the feature it plans to ship?
Customer-centric: Does the team hear directly from customers, or mainly through sales, support, analytics, and stakeholder summaries?
Collaborative: Do product, design, and engineering make decisions together, or meet mainly to exchange work?
Visual: Is there one shared representation of the outcome, opportunities, solutions, and assumptions?
Experimental: Can the team name what could make the current idea fail?
Continuous: Does each learning activity lead to the next question, or does discovery end with a presentation?
Choose the weakest link as your first intervention. A team with output-based goals does not need a better interview script first; it needs an outcome that gives the interview a purpose. A team dominated by handoffs needs shared sensemaking, not another repository.
Install a weekly loop small enough to protect
A habit survives because its trigger, action, and output are clear. Put a recurring discovery block at a stable point in the team’s operating rhythm. Tie it to a current outcome and a live decision, not to a general ambition to understand users better.
Trigger: A protected calendar block recurs every week, including during active delivery.
Focus: The trio brings one current outcome, the decision in front of it, and the uncertainty preventing a confident choice.
Customer contact: The team has a direct customer touchpoint every week. That might be a customer interview, observation of a workflow, or a usability session connected to the current question.
Sensemaking: The trio separates what it observed from what it inferred.
Update: New evidence changes the opportunity solution tree or confirms why no change is warranted.
Commitment: The team names the next uncertainty and starts arranging the next customer contact.
A customer touchpoint is not any meeting attended by a customer. A sales demo, account review, or advisory session dominated by presentation may be valuable, but it does not automatically answer a discovery question. The useful test is whether the customer can reveal a real behavior, need, constraint, or reaction and whether the team can ask follow-up questions.
Prepare each touchpoint by completing this sentence: After this contact, the trio might decide whether… If you cannot finish it, the question is probably too broad. Starting with a decision also reduces the temptation to collect interesting comments that never affect the product.
During the interaction, capture concrete observations before interpretations. Afterward, answer five questions while the context is fresh:
What did the customer do, describe, or struggle to explain?
What interpretation is the team placing on that observation?
Which opportunity or assumption does it affect?
What decision changes, if any?
What remains uncertain enough to examine next?
The distinction between observation and interpretation matters. A customer abandoning a task is an observation. Assuming that price caused the abandonment is an interpretation. If the team records only the interpretation, an early guess can become institutional memory.
Recruiting is part of the habit, not administrative work that begins after an interview is requested. Give coordination to a named owner, maintain a rolling pool of relevant customers, and create simple paths for customer success and support to nominate people who recently experienced the problem. Start the next invitation before the current discovery cycle feels complete. Otherwise, every customer cancellation becomes a reason to skip the week.
When delivery pressure rises, protect the trigger and narrow the activity. Ask a smaller question, review a focused prototype, or examine one step in a workflow. Do not silently replace direct contact with an internal meeting and call the habit complete. If a customer cancels, use the protected time to recruit, refine the decision question, and reschedule. Preserve the rhythm without pretending the missing evidence exists.
Give the product trio ownership of decisions, not ceremonies
A product trio is not three people attending the same interview. It is product, design, and engineering sharing responsibility for understanding the opportunity and choosing how to address it. Attendance can rotate. Interpretation and decision-making cannot be delegated to one function and handed back as a deck.
Make the trio’s decision rights explicit at the start of an outcome. Record the outcome it owns, the decisions it can make autonomously, the constraints it must respect, what requires escalation, and where its evidence will remain visible. Without that contract, discovery may reveal a better direction while the roadmap continues unchanged because nobody knows who can act.
The responsibilities below are a practical starting point, not rigid job boundaries:
Product keeps the outcome, strategic context, customer segment, and pending decision visible.
Design helps the trio expose customer behavior, frame opportunities, and choose an appropriate way to learn.
Engineering surfaces feasibility, system behavior, data, and implementation assumptions before the solution becomes expensive to change.
The trio decides what the evidence means, which option remains viable, and what uncertainty deserves attention next.
Use a short shared debrief after customer contact. The format can remain simple:
Observation: What happened without interpretation?
Meaning: What plausible explanations fit the observation?
Decision: What will the trio change or preserve?
Unknown: What still blocks commitment?
This prevents the loudest interpretation from becoming the team’s conclusion. It also gives engineering a role before implementation and gives design a role beyond producing artifacts.
Leadership should ask for evidence of changed decisions, not proof that ceremonies occurred. Instead of asking how many interviews the team completed, ask which opportunity became clearer, which assumption weakened, what decision changed, and how the change connects to the outcome. Interview volume is easy to report and easy to game. Decision quality is harder to display, but it is the reason the habit exists.
Connect discovery evidence to strategy and delivery
A weekly customer conversation can still become theater if its evidence floats separately from strategy, roadmaps, and sprint planning. The opportunity solution tree provides a shared spine: the desired outcome sits at the top, customer opportunities sit beneath it, and candidate solutions connect to the opportunities they could address. That outcome-opportunity-solution structure keeps the team connected to why it is considering a particular feature.
Use the tree as a decision interface, not a workshop artifact:
Product strategy: Put the intended outcome at the top so the team can test whether its discovery work supports the strategic direction.
Roadmapping: Attach candidate solutions to named opportunities. Keep alternatives visible until evidence or a real constraint justifies commitment.
Sprint planning: Require each significant item to trace back to an opportunity and outcome. If it cannot, surface the mismatch before implementation.
Customer contact: Update the affected opportunity, solution, or assumption during the debrief. Do not wait for a separate documentation session.
Stakeholder communication: Show what changed in the tree, why it changed, and which decision follows. This is more useful than presenting a collection of customer quotations.
Keep a record of rejected options and the evidence or constraint behind each rejection. Otherwise, an old idea can return with a new label and consume another round of debate. The record should remain revisable: new customer behavior, technical capability, or strategic constraints can justify reopening a branch.
Measure whether evidence enters decisions
The safest discovery metric is not an isolated activity count. Measure the health of the loop:
Cadence: Did direct customer contact happen during the weekly rhythm?
Decision integration: Which current decision did that contact inform?
Shared ownership: Did the trio participate in sensemaking, even if every member did not attend the session?
Strategic traceability: Can a delivery item be traced to an opportunity and outcome?
Learning movement: Which belief, option, or assumption changed?
A team can conduct many interviews and learn very little if every conversation validates a solution already selected. Conversely, one focused interaction can be valuable when it exposes a faulty workflow assumption and changes a pending decision. Track cadence to protect the habit, but judge value by movement in the decision model.
Separate customer, model, and operational uncertainty in AI products
AI product teams face a specific discovery trap: an impressive model demonstration can make technical possibility look like customer demand. Keep different uncertainties separate so one kind of evidence does not answer a different question.
Customer uncertainty: What job is the person trying to complete? Where does the current workflow break? Under what conditions will the person trust, verify, correct, or reject an AI-assisted result?
Model uncertainty: Does the system produce acceptable behavior for the intended context? Which failures matter to the user, and how will the team evaluate them?
Operational uncertainty: Can the product obtain the required data and permissions? Where is human review needed? How will failures be detected, explained, and supported?
Customer contact can reveal workflow, language, trust conditions, and failure consequences. It cannot prove that the model behaves reliably. Model evaluations can reveal performance and failure patterns. They cannot prove that the workflow is valuable. Operational checks can establish feasibility and controls. They cannot prove adoption. Keep all of these linked to the same outcome while using the right evidence for each uncertainty.
On the opportunity solution tree, write opportunities in customer terms. “Use generative AI” is a solution direction, not an opportunity. “Reduce the effort required to turn a customer conversation into an accurate follow-up” describes a customer problem that could have AI and non-AI solutions. That distinction helps the trio discover value without becoming attached to a technology.
Fix the mechanism when the habit breaks
What you notice
Likely mechanism
What to change
The team talks to customers, but the roadmap never changes.
Sessions are disconnected from a live decision.
Write the decision before recruiting and record what changed immediately after the interaction.
Engineering joins only after discovery is complete.
The trio label is masking a handoff.
Include engineering in opportunity framing, assumption identification, and shared sensemaking. Session attendance can rotate.
Customer sessions repeatedly fall through.
Recruitment starts only after a question becomes urgent.
Maintain a rolling pool of relevant customers and assign coordination to a named owner.
The opportunity solution tree is stale.
The tree is treated as presentation material.
Update it during the debrief and remove or annotate branches that no longer have support.
Discovery pauses whenever delivery accelerates.
Discovery is scoped as a project rather than a continuous rhythm.
Protect the weekly trigger and narrow the question or method when capacity is tight.
Leadership keeps asking the team for certainty.
The team reports activities without showing their decision impact.
Show the outcome, changed opportunity or assumption, resulting decision, and remaining uncertainty.
Do not respond to a broken habit by adding more process everywhere. Match the intervention to the failure. A recruiting problem needs a pipeline. A decision-rights problem needs leadership alignment. A stale artifact needs an update trigger. A handoff problem needs shared sensemaking.
Key takeaways
Map the current decision system and audit last week’s decisions before adding a new discovery ceremony.
Anchor a direct customer touchpoint every week to a current outcome, decision, and uncertainty.
Let attendance vary when necessary, but keep interpretation and decisions jointly owned by the product trio.
Use the opportunity solution tree as the live connection between strategy, customer evidence, roadmap choices, and sprint work.
When delivery pressure rises, protect the trigger and shrink the activity instead of suspending the cadence.
For AI products, do not use customer enthusiasm as proof of model reliability or an evaluation result as proof of customer value.
Put the recurring customer touchpoint on the calendar, choose the outcome and decision it must inform, and name the product trio responsible for acting on what it learns. At the end of the next weekly cycle, do not ask whether the team “did discovery.” Ask what changed in the decision and what the team needs to learn next.
You have an AI capability on the roadmap. A vendor can demonstrate something credible almost immediately, while engineering believes an internal version would fit the product better. Both claims may be true, and neither one answers the decision in front of you.
The useful question is not simply whether to build or buy. You need to decide which parts of the capability create strategic advantage, what you must learn before committing further, which obligations you are prepared to own, and how you will leave if the economics or technology changes.
Draw the capability boundary before comparing options
Most weak build-versus-buy debates begin with a label that is too broad. AI assistant, support automation, recommendation engine, and enterprise search each describe an experience, not a single technical capability. Comparing a vendor’s finished product with an imagined internal system at that level guarantees an uneven evaluation.
Break the experience into layers before discussing ownership. An AI product might contain data connectors, ingestion, domain retrieval, ranking, generation, orchestration, evaluation, observability, policy guardrails, workflow logic, a user interface, and a human handoff. You can make a different decision for each layer.
Classify every layer by its strategic role:
Differentiation: The layer materially affects why customers choose, retain, or expand with your product. It may encode a proprietary workflow, use unique data, or create a feedback loop competitors cannot easily reproduce.
Parity: Customers expect the capability, but it is not a meaningful reason to choose you. Reliable billing infrastructure, standard integrations, and generic analytics plumbing often belong here.
Control: The layer may not be visible to customers, but it determines whether you can satisfy security, regulatory, reliability, cost, or product-policy obligations. Control can justify ownership even when the layer itself is not differentiating.
If this layer became substantially better, would it change the product’s value proposition or merely close a feature gap?
Does operating it create proprietary data, evaluation evidence, workflow knowledge, or customer insight that compounds over time?
Would dependence on a vendor’s roadmap prevent you from making an important product promise?
Could a close competitor buy the same capability and achieve roughly the same result?
Do privacy, residency, auditability, reliability, or recovery requirements force you to retain direct control?
Can your team support the layer after launch, including incidents, upgrades, security work, and user adoption?
A retrieval-augmented generation system shows why this decomposition matters. The right answer may be to build the parts that encode domain knowledge while buying fast-moving infrastructure around them.
Layer
Strategic question
Plausible initial posture
Domain retrieval and ranking
Does relevance depend on proprietary content, metadata, permissions, or customer context?
Build when this is central to answer quality and differentiation.
Orchestration and observability
Would owning the runtime create customer value, or only infrastructure work?
Buy when a platform provides adequate reliability, APIs, and portability.
Prompts, policies, guardrails, and evaluation cases
Do these artifacts encode product behavior, risk tolerance, and domain expertise?
Own the specifications and evidence even if a vendor executes them.
User workflow and human handoff
Is the workflow part of the product’s distinctive experience?
Build the differentiated interaction; integrate commodity components behind it.
The point is not that every retrieval system should use this split. The point is to stop forcing one ownership decision across layers with different strategic value. A composed architecture can give you speed at the edges and control at the center.
Compare time to value and total ownership cost separately
Buying and building usually produce different cost curves. Buying can reduce the initial implementation burden and provide proven operations. Building concentrates cost and complexity near the beginning but may create a better fit and more favorable economics at scale. Neither profile is automatically cheaper.
Evaluate the decision across two horizons. The first is time to activated value: how long it takes before the intended users complete the intended workflow successfully. The second is total cost of ownership over the period in which the capability must operate, evolve, and eventually migrate.
Do not treat a signed contract, completed deployment, or merged pull request as time to value. Procurement, security review, data preparation, integration, enablement, in-product guidance, and user activation sit between acquisition and an actual outcome. A fast purchase with weak adoption is not a fast result.
A useful cost model is:
Total ownership cost = acquisition or development + integration + operations + change + risk exposure + exit.
Apply the same formula to both choices. Teams often present the vendor’s full commercial cost against only the internal development estimate, or compare a subscription price with an imagined build that excludes maintenance. Both comparisons are misleading.
Cost area
Evidence needed for a buy option
Evidence needed for a build option
Acquisition or development
Subscription, per-seat or consumption charges, implementation fees, support tier, and expected price changes with growth.
Product, design, engineering, data, security, and platform capacity required to reach usable scope.
Integration
Connector work, identity and permission mapping, data transformation, API constraints, testing, and CI/CD maintenance.
Interfaces with existing systems, migration of current workflows, data contracts, and platform dependencies.
Operations
Internal administration, vendor management, incident coordination, usage monitoring, and workarounds for roadmap gaps.
On-call ownership, observability, model and dependency updates, incident response, capacity management, and reliability work.
Change
Configuration limits, professional services, retraining, contract changes, and waiting for vendor roadmap delivery.
Continuing product development, evaluation maintenance, documentation, enablement, and the opportunity cost of displaced roadmap work.
Risk exposure
Vendor outages, security posture, data handling, roadmap dependence, quota changes, and concentration risk.
Internal security gaps, insufficient operational maturity, key-person dependency, and failure to meet compliance obligations.
Exit
Data export, contract termination, migration assistance, replacement integration, and reconstruction of non-portable artifacts.
Decommissioning, data migration, user transition, and replacement of internally coupled components.
Run an expected case and a stress case for both options. For a vendor, stress usage, API consumption, support requirements, and the cost of additional environments or features. For an internal system, stress incident load, model or infrastructure changes, evaluation maintenance, and continued product demands. The purpose is not to produce a perfectly precise forecast. It is to expose which assumptions can overturn the decision.
Record those assumptions in the decision memo. If vendor consumption cost must stay within an agreed envelope, state that envelope internally and assign someone to monitor it. If the build case depends on reuse across several product surfaces, name those surfaces and verify that their teams actually intend to adopt the component. An unowned assumption is not a forecast; it is hidden risk.
Turn the debate into an evidence-based decision
A scorecard is useful only when it forces explicit trade-offs. It should not turn judgment into decorative arithmetic. Establish hard gates first, agree on the relative importance of the remaining criteria before vendor demonstrations or internal prototypes create attachment, and then evaluate both options against the same outcome.
How directly does the capability support the value proposition or defensibility?
Product strategy, roadmap commitments, customer workflow evidence, proprietary data advantages, and the importance of controlling behavior.
Build becomes more attractive as the capability determines why customers choose or stay.
Urgency and time to value
What is the cost of waiting, and when can users reach a meaningful outcome?
Procurement and security timelines, integration dependencies, build scope, launch readiness, enablement needs, and adoption path.
Buy becomes more attractive when delay is costly and the purchased path can reach activated value materially sooner.
Security and regulatory risk
Can either option verifiably meet non-negotiable obligations within the launch window?
Data-flow diagrams, privacy controls, residency, retention, audit logs, access controls, certifications, threat response, model lineage, and red-team practices.
An option that fails a mandatory obligation should be removed, regardless of its aggregate score.
Integration complexity
How much continuing work is hidden behind the initial connection?
Sandbox tests, API behavior, quotas, identity mapping, data contracts, failure modes, deployment workflow, and ownership of connectors.
Build gains ground when vendor constraints create persistent product or operational work; buy gains ground when internal integration and support exceed the apparent build scope.
AI leverage and portability
Which prompts, data, evaluations, embeddings, policies, and feedback become valuable, and can they move?
Export tests, API abstraction, model-routing options, ownership terms, deletion process, evaluation access, and migration design.
Build or a hybrid architecture gains ground when the vendor captures an asset central to future differentiation.
Security, regulatory compliance, and minimum reliability are gates, not preferences. A high score elsewhere cannot compensate for an option that cannot lawfully handle the data, meet a required recovery posture, or provide necessary audit evidence. The same logic applies to internal capacity: if no team can own production incidents, an attractive prototype is not a viable build option.
Use a product trio of product, design, and engineering to set the scorecard’s priorities. Bring security, data, finance, procurement, and operations into the criteria they own. This prevents a late-stage veto from appearing as a surprise when it was actually a missing requirement.
Then run comparable discovery work. Give the vendor a production-like workflow in a sandbox. Give the internal option a thin vertical slice that touches the real data and integration boundary. Test the same cases for outcome quality, failure handling, permissions, auditability, operator effort, integration behavior, and unit economics. A polished vendor demonstration and a rough internal prototype reveal different things; common acceptance cases make the evidence comparable.
Keep confidence separate from the decision direction. A criterion can favor building while resting on weak evidence. Mark it as an assumption and define the cheapest test that would resolve it. This is more useful than adding precision to a score whose inputs remain speculative.
The final memo should fit the decision, not the politics around it. Include the capability boundary, strategic classification of each layer, intended user outcome, hard gates, scorecard, cost assumptions, evidence quality, operational owner, exit path, and re-evaluation triggers. Anyone reading it later should be able to tell why the decision was reasonable at the time and which changed condition would justify revisiting it.
Run an AI-specific risk and portability pass
AI changes more than development speed. It introduces movable models, probabilistic behavior, data-dependent quality, metered usage, and artifacts that can become strategically valuable. A normal software procurement checklist will miss several of these dependencies.
Data route: Document what enters the system, which service receives it, where it is stored, how long it is retained, whether it can be used for training, how deletion works, and whether residency requirements apply. Include prompts, retrieved context, generated output, user feedback, and operational logs.
Model and quality governance: Require a way to identify the model, configuration, prompt, retrieval state, and policy version associated with important behavior. Decide who maintains evaluation cases, reviews regressions, investigates failures, and approves consequential changes.
Security and privacy: Verify role-based access, audit logs, PII handling, privacy-by-design controls, threat detection and response, and the vendor’s red-team and incident practices. For an internal build, require equally concrete evidence rather than assuming control equals safety.
Portability: Establish ownership and export mechanisms for source data, metadata, prompts, policies, evaluation sets, feedback, transcripts, and relevant logs. Treat a contractual right to export and a technically usable export as separate requirements.
Unit economics: Map every metered event in the actual workflow. Per-seat pricing, consumption charges, model usage, and orchestration can behave differently as adoption and workflow complexity grow. Test the economic model against expected and stressed usage.
Operational responsibility: Specify who diagnoses a failure that crosses your application, the vendor platform, a model provider, and a data source. Shared architecture does not remove accountability; it makes the handoffs more important.
Portability deserves an actual exit test. Ask the vendor to produce a representative export before the contract is final. Confirm its format, completeness, permission model, and usefulness in another environment. An export button is not evidence that you can reconstruct the product behavior that matters.
Prompts require the same caution. Access to prompt text is necessary, but equivalent behavior may still depend on a model, tool interface, retrieval implementation, or vendor-specific orchestration. Preserve the intent, policies, evaluation cases, and expected outcomes around a prompt, not just the string itself.
Embeddings can also create false confidence about portability. Preserve the original content, chunking inputs, metadata, permission relationships, and evaluation set so embeddings can be regenerated if the model or retrieval system changes. The derived vectors alone are not a complete migration asset.
For vendors, negotiate transparent API quotas, usable sandbox environments, data-export terms, growth price protections, and clear ownership of AI artifacts. Pressure-test the roadmap against your deployment cadence and ask how incidents, breaking changes, and model transitions are communicated. For an internal build, apply the same rigor to service levels, incident response, observability, model lineage, retention, and ongoing staffing.
Buying does not outsource your responsibility for the product’s behavior. Building does not prove that the behavior is controlled. Choose the implementation that can produce the evidence your risk level demands within the launch window.
Make a staged commitment with explicit re-evaluation triggers
A build-versus-buy decision does not need to be permanent to be disciplined. When uncertainty is high and speed matters, a bounded purchase can be a learning instrument. When differentiation or control is already clear, a minimum lovable internal slice can establish the core while purchased components accelerate everything around it.
For a buy-to-learn path, use this sequence:
Name the uncertainty. Decide whether you are testing demand, workflow fit, quality, integration feasibility, adoption, operational burden, or economics. Do not call a general implementation a pilot.
Bound the commitment. Limit initial scope, data exposure, coupling, and custom vendor work to what the learning objective requires. Preserve an adapter or interface where replacement would otherwise become expensive.
Instrument the outcome. Track whether intended users activate, return, complete the workflow, accept the output, escalate to a human, and create operational work. Monitor consumption and connector reliability alongside product use.
Review against prewritten triggers. Deepen the vendor integration if adoption is durable, economics remain acceptable, and integration pain is manageable. Move toward building if unique requirements emerge, strategic artifacts accumulate, vendor constraints block the roadmap, or costs reach the agreed inflection point. Stop if the user outcome does not materialize.
This approach works because a purchased solution can validate value before a deeper build commitment. The learning is reusable only if you retain the data model, evaluation evidence, workflow understanding, and user-behavior insight rather than burying them inside vendor-specific configuration.
For a build-to-differentiate path, keep the first scope narrow. Build the smallest end-to-end experience that proves the differentiating hypothesis. Buy mature infrastructure around it where doing so does not surrender the key data, policy, or product behavior. Isolate components behind explicit interfaces so a model, orchestration service, retrieval system, or observability layer can change without rewriting the entire experience.
Set re-evaluation triggers before launch, while nobody is defending a sunk decision:
Product trigger: Usage fails to become durable, or customers reveal a need that the current option cannot support.
Financial trigger: Consumption pricing, operating cost, or internal staffing moves outside the approved economic envelope.
Technical trigger: Integration maintenance, API limits, reliability, or roadmap mismatch begins delaying important releases.
Risk trigger: Data handling, retention, auditability, model governance, or regulatory obligations can no longer be met.
Strategic trigger: A previously generic layer begins creating proprietary data, workflow advantage, or meaningful differentiation.
Capacity trigger: The internal team can no longer sustain the operational burden, or gains the maturity needed to own a capability previously bought.
Assign an owner and a review event to each trigger. Without ownership, continuous re-evaluation becomes a good intention that loses to roadmap pressure. The decision memo should remain a living control surface for product, engineering, finance, security, and procurement, not an artifact filed after approval.
Do not neglect activation. Whether you build or buy, budget for workflow changes, onboarding, in-app guidance, support preparation, and measurement. Deployment creates availability. Repeated successful use creates value.
Key takeaways
Decompose an AI experience into layers before deciding who should own it.
Build differentiated or control-critical layers; buy parity where a vendor can accelerate activated value.
Compare both choices across time to value and total ownership cost using the same scope and service expectations.
Apply non-negotiable gates before a weighted scorecard, then test both options against common acceptance cases.
Own the data, policies, evaluation evidence, and migration path that protect your future leverage.
Use staged commitments and prewritten triggers so changing the decision becomes responsible management, not an admission of failure.
The next time this question reaches your roadmap review, do not ask for a permanent verdict on build or buy. Ask for a capability map, comparable evidence, an operational owner, a tested exit path, and the conditions that would change the answer. That gives you a decision you can defend now without mortgaging your ability to adapt later.
Your AI support pilot can look successful while the service operation gets worse. The agent closes more conversations, but customers repeat themselves after escalation, risky cases receive plausible but incomplete answers, and human agents inherit a queue made almost entirely of exceptions.
If you own this transformation, your job is not to install an AI agent. It is to redesign how customer demand moves through knowledge, automation, human judgment, and product feedback. You also need to prove that a conversation marked resolved was actually resolved. That requires an operating model, not just a deployment plan.
Start with an operating thesis, not a deflection target
That gap matters. If you treat the program as a software rollout, the technology can reach production while ownership, escalation rules, quality controls, and performance expectations remain designed for a human-only queue. The result is automation sitting on top of an unchanged operation.
The interviews were drawn from Intercom customers or prospects and centered on Fin or similar products. They are useful directional evidence from teams close to this transition, but they are not a vendor-neutral census of every customer service organization. Your own demand, risk profile, knowledge quality, and channel mix should determine the design.
I would begin with a one-page transformation brief. Force the leadership team to complete these fields before discussing a broad rollout:
Customer promise: Which customer outcome will become faster, easier, or more reliable?
Eligible demand: Which intents, channels, languages, customer states, and account types may enter the AI workflow?
Decision boundary: What may the AI explain, recommend, decide, or execute? These are different levels of authority.
Human boundary: Which ambiguity, consequence, customer request, or system condition requires a human?
Business hypothesis: Which cost, capacity, service-level, or growth constraint should improve if the workflow succeeds?
Quality gates: Which measures must improve, and which failure measures must not regress?
Learning owner: Who converts failures into knowledge fixes, workflow changes, model evaluations, or product improvements?
Do not make deflection the customer promise. Deflection records the absence of a human interaction; it does not establish that the customer’s problem was solved. A better promise names the intended outcome, such as completing a defined action correctly or answering an eligible question from an approved source without avoidable repetition.
Scope automation using two dimensions: how repeatable the work is and what happens when the answer is wrong. A simple decision matrix prevents the team from treating every incoming conversation as equally automatable.
Work pattern
AI role
Human role
Release condition
Repeatable and low consequence
Resolve from approved knowledge or execute a reversible workflow
Review samples and handle defined exceptions
Correct resolution and reliable rollback are demonstrated
Repeatable and higher consequence
Retrieve, summarize, validate inputs, or draft
Approve the final answer or action
Authoritative sources, approval capture, and auditability are in place
Ambiguous and low consequence
Ask clarifying questions, categorize, and route
Resolve cases that remain ambiguous
The escalation reason and collected context are visible to the human
Ambiguous and higher consequence
Collect only the minimum safe context, then stop
Own judgment, communication, and action
Hard escalation rules have been tested and cannot be bypassed conversationally
Risk is contextual. The same intent may be routine for one account state and consequential for another. Eligibility therefore belongs in the workflow itself, using customer state, requested action, permissions, available knowledge, and tool health. It should not live only in a prompt that asks the model to be careful.
Redesign the full conversation, especially the human handoff
The unit of design should be one end-to-end customer intent. Do not stop at the AI response. Trace what happens from the first message through resolution, escalation, downstream action, and learning:
Define the intent and entry conditions. State what the customer is trying to accomplish and which signals make the conversation eligible.
Name the authoritative knowledge. Identify the policy, product data, account data, or workflow state required to answer correctly.
Specify permitted actions. Separate explaining a process, recommending an action, preparing an action, and executing it.
Design the handoff packet. Give the human the context needed to continue without interrogating the customer again.
Capture a failure reason. Every failed or escalated attempt should produce a category that can be assigned to an owner.
Close the learning loop. Route the failure to knowledge, conversation design, support operations, product, engineering, or governance.
The handoff is where many apparently successful deployments reveal their real cost. If the human receives only a transcript, the AI has transferred a conversation but not the work. The agent must reconstruct the goal, identify what the system already attempted, verify customer-provided facts, and decide whether any prior answer can be trusted.
A useful handoff contract should include:
The customer’s detected goal and the intent assigned to it.
The material facts the customer supplied, with no invented completion of missing fields.
The approved sources used to form the answer.
Any tools called, actions attempted, results returned, and side effects created.
The point of uncertainty or the exact escalation rule triggered.
The unresolved question or recommended next action for the human.
The relevant transcript, available for verification rather than presented as the only summary.
Test the handoff as a product experience. Give a human agent only the packet and the underlying conversation, then observe whether the case can continue without the customer repeating information. Track missing fields and unnecessary rework as workflow defects. Do not hide that effort inside average handle time.
Knowledge needs the same discipline. For each automated intent, name one canonical source, one owner, a review trigger, and a withdrawal path. If two approved pages disagree, the correct AI behavior is not to blend them into a smooth answer. It is to stop, disclose the limitation appropriately, and route the conflict to an owner.
The AI agent does not create knowledge debt, but it can expose and distribute that debt at much greater speed. A missing article, stale policy, ambiguous field, or inaccessible account state can produce thousands of superficially different conversations with the same root cause. Aggregate failures by root cause instead of editing individual answers forever.
Use a failure taxonomy that separates at least these problems: missing knowledge, stale knowledge, conflicting knowledge, retrieval failure, unsupported reasoning, policy-boundary failure, tool or integration failure, incorrect eligibility, poor conversation design, routing failure, and incomplete handoff. Each category should map to a named owner and a defined corrective action. Otherwise, quality review becomes a list of examples rather than an operating system for improvement.
Redesign jobs before you promise headcount savings
Workforce impact is real, but it is not uniform. Headcount or hiring changed in 27.71% of the 166 interviews, often through slower Tier 1 hiring, freezes, natural attrition, or reallocation. That is materially less common than workflow and responsibility changes. The safest conclusion is not that AI automatically removes a fixed percentage of support cost. It is that repetitive demand can shrink while new oversight, exception, knowledge, and optimization work grows.
Calculate net capacity rather than gross deflection. The practical equation is:
Net capacity released = human work correctly avoided – new review, exception, maintenance, and recovery work.
Count the whole system. Include time spent reviewing samples, investigating severe failures, maintaining knowledge, configuring workflows, testing releases, repairing integrations, managing escalations, and helping customers recover from wrong actions. Also separate capacity released from cash savings. A team may use capacity to absorb growth, improve response time, eliminate backlog, or take on higher-complexity work without reducing current payroll.
Role design should follow the new work, not the fashionable job titles. You may create an AI specialist, automation manager, or AI-agent owner, but the essential question is who owns each recurring decision:
Frontline specialists resolve nuanced cases, identify failure patterns, validate knowledge gaps, and contribute difficult conversations to evaluation sets.
Support managers manage the changing workload mix, coach exception handling, monitor capacity, and decide where human judgment adds value.
AI or automation owners configure behavior, maintain evaluations, control releases, monitor production, and coordinate rollback.
Quality owners define error severity, audit both automated and human resolutions, and make recurring failure visible.
Knowledge owners approve canonical content, resolve conflicts, and remove information that should no longer be used.
Product and engineering owners fix product defects, data gaps, and tool failures that support conversations repeatedly expose.
These are responsibilities, not necessarily separate positions. A smaller organization may combine them, but it should not leave them implicit. One person can hold several responsibilities; one critical responsibility cannot be owned by nobody.
Write decision rights alongside role descriptions. Specify who may expand eligible intents, approve a high-consequence workflow, publish knowledge, change a prompt or model, accept a known quality limitation, pause automation, and communicate a customer-impacting failure. An AI owner who is accountable for outcomes but cannot stop a release is not an owner.
Do not wait until automation is broadly deployed to explain this shift. Before changing staffing plans, show people the future queue, the new performance expectations, the skills they can build, and the paths available for redeployment. Vague assurances create uncertainty, while premature savings commitments force managers to defend a number before the operation has demonstrated sustainable quality.
Measure correct outcomes, not apparent automation
A conversation can be closed, contained, or deflected without being correct. That is why an automation dashboard cannot double as a transformation scorecard. I would make cost per correct resolution the economic anchor, then constrain it with customer-experience and severity guardrails.
Define correct resolution for every intent before launch. At minimum, it should mean that the customer received an accurate and complete answer or action, the applicable policy was followed, the workflow created no unintended side effect, and no avoidable human rescue or repeat contact occurred during an intent-appropriate observation period. The period may differ by intent; a question answered immediately and a downstream account action do not reveal failure on the same schedule.
Measure
Question it answers
Common trap
Eligible demand coverage
How much inbound demand falls inside a clearly approved scope?
Expanding eligibility merely to make automation look larger
AI attempt rate
How often did the AI engage eligible demand?
Counting an attempt as a successful outcome
Audited correct autonomous resolution
How often did sampled AI completions fully meet the intent definition without rescue?
Relying only on closure status or customer silence
Repeat or reopened contact
Did the customer return because the original issue remained unresolved?
Missing a repeat that arrives through another channel or wording
Handoff recovery
Can a human continue efficiently with accurate context?
Measuring routing speed while ignoring repeated questions and reconstruction work
Cost per correct resolution
What does a genuinely completed outcome cost across the whole system?
Excluding review, knowledge, tooling, maintenance, and recovery effort
Severity-weighted failure
How much customer or business consequence did errors create?
Allowing a high average accuracy to hide rare but serious failures
New-work burden
How much human effort did automation introduce?
Treating oversight and maintenance as free capacity
Keep the denominators explicit. Eligible demand coverage is eligible conversations divided by total inbound conversations. AI attempt rate uses eligible conversations as its denominator. Audited correct autonomous resolution should use reviewed AI-completed conversations, not every inbound contact. Mixing those denominators lets a team report a large percentage without showing how much demand was actually solved.
Audit with two sampling paths. Use a representative sample to estimate ordinary performance across intents, channels, languages, and customer states. Add targeted samples for high-consequence actions, new releases, known weak spots, tool failures, unusual escalations, and complaints. A purely random sample can miss rare failures that matter more than common harmless mistakes.
Define error severity before reviewers see the results. A wording issue, an incomplete answer, a wrong policy explanation, an unauthorized disclosure, and an incorrect account action should not contribute equally to one accuracy average. Severity should change the required response: monitor, correct knowledge, roll back a workflow, disable an action, or initiate the relevant incident process.
Maintain separate executive and operating views. The executive view should show eligible volume, audited correct resolution, customer outcome measures, cost per correct resolution, severe-failure trend, capacity released, and where that capacity went. The operating view should break performance down by intent, channel, language, customer state, workflow version, knowledge version, tool, failure category, and escalation reason.
Versioning is essential for diagnosis. Record the model, instructions, knowledge snapshot, workflow configuration, tool version, and eligibility rules associated with each resolved conversation. When several components change together, you may know performance moved without knowing why. Controlled rollouts or eligible-traffic holdouts can provide stronger evidence than a simple before-and-after comparison, especially when demand mix or seasonality is changing.
Set release thresholds before looking at a candidate’s results. The exact threshold should reflect the consequence of the intent and your current human baseline; there is no responsible universal number. The release decision should require sufficient audited quality, acceptable handoff recovery, no prohibited failure, functioning rollback, and an owner for every material defect that remains open.
Scale through evidence-gated stages
Do not scale on a calendar promise. Move when the workflow has produced enough evidence for its next level of authority. A useful sequence separates learning about the problem from granting the system permission to act.
Baseline the demand and draw the boundary
Start with the highest-volume and highest-consequence intents, but do not assume they belong in the same release. Build an inventory containing volume, current human effort, customer outcome, approved knowledge, data requirements, available actions, reversibility, failure consequence, escalation destination, and owner.
Create an evaluation set from real, appropriately handled historical conversations. Remove or protect sensitive data according to your controls. Include ordinary examples, ambiguous requests, missing information, policy conflicts, tool failures, customer requests for a human, and known edge cases. The gate for leaving this stage is not model quality. It is a testable definition of correct behavior and a clear boundary around what the AI must not do.
Run in observation or approval mode
Let the AI classify, retrieve, summarize, or draft while a human retains final authority. Compare its proposed outcome with the completed human outcome. Instrument the failure taxonomy, inspect whether the correct knowledge was available, and test the handoff packet with frontline agents.
Use this stage to repair the system around the model. Many failures will belong to missing content, conflicting policy, broken integrations, weak eligibility, or unclear product behavior. Prompt editing cannot fix an absent source of truth or an action the underlying system cannot perform reliably.
Grant controlled autonomy to bounded work
Begin with stable, low-consequence demand supported by authoritative knowledge and reversible workflows. Enforce eligibility outside the conversational instructions where possible. Keep hard escalation rules for uncertainty, missing data, customer preference, unavailable tools, policy conflicts, and prohibited actions.
Review production samples and targeted risk cases. Watch repeat contacts, human recovery work, severe errors, and changes in the composition of the human queue. A falling queue is not automatically good if the cases that remain take much longer or arrive with damaged customer trust.
Expand one meaningful dimension at a time
Add an intent, channel, language, customer state, or action only after defining how that dimension changes knowledge, evaluation, escalation, and consequence. Reusing a workflow in a new language is not just translation if policies, terminology, tone, or available support paths differ. Adding tool execution is not just a better answer; it grants the system operational authority.
Version each expansion and preserve rollback. If you need causal clarity, avoid changing the model, knowledge, tools, instructions, and eligibility rules in the same release. When simultaneous changes are unavoidable, label the release as a system change and evaluate the combined behavior rather than attributing the result to one component.
Institutionalize the operating model
Only after correct resolution and total workload remain durable should you change long-term staffing assumptions, performance management, budgets, or reporting lines. Update role charters, decision rights, quality routines, release governance, incident ownership, knowledge operations, and planning models together.
Give recurring AI failures a path into the product roadmap. If customers repeatedly ask because the interface is unclear, a workflow fails, or account state is hard to understand, automating the explanation may reduce service effort while preserving the root cause. The better product decision may be to remove the need for the conversation.
Key takeaways
Treat AI customer service as an operating-model transformation, because workflows and responsibilities change before most reporting structures do.
Automate bounded intents, not an undifferentiated share of tickets. Repeatability and consequence should determine the AI’s authority.
Design the human handoff as a product. A transcript without facts, actions, sources, uncertainty, and next steps transfers the queue but not the work.
Use audited correct resolution and cost per correct resolution as anchors. Attempts, closures, containment, and deflection are supporting events, not proof of value.
Calculate net capacity after review, maintenance, exception, and recovery work. Keep that separate from any claimed payroll saving.
Scale only when quality, severity, handoff, ownership, and rollback gates have been met for the next expansion.
Your next move can be small and consequential. Choose one recurring intent, complete the transformation brief, name its canonical knowledge owner, write the handoff contract, and define how you will audit correct resolution. If you cannot assign the knowledge, failure, and release decisions, do not automate the intent yet. Resolving that ownership gap is the first real step in the transformation.
You have an alert queue full of low-context signals, analysts spending time assembling evidence, and pressure to show that AI can improve the operation. The tempting move is to add a copilot to the security console and call the problem solved.
The harder leadership decision is where AI may influence a security decision, where it may take action, and how you will know it is helping. The right goal is not an autonomous security operations center. It is a shorter, more reliable path from signal to containment, with explicit limits on what a model can do.
Design the decision loop before choosing the AI
AI-enabled cybersecurity operations are easier to manage when you separate three capabilities that vendors often bundle together:
Detection models identify patterns, anomalies, or risk signals in security telemetry.
Generative AI explains evidence, summarizes an incident, retrieves a relevant playbook, and proposes a next action.
Orchestration performs a deterministic operation such as collecting evidence, updating a ticket, isolating an endpoint, or rotating a credential.
These components should not share the same authority. An anomaly score is not proof of compromise. A fluent explanation is not an approved response. A tool call is not safe merely because the model produced valid syntax.
Map the operational loop before you evaluate a model:
Observe: collect the endpoint, identity, network, and application signals relevant to the use case.
Detect: rank suspicious activity without hiding the underlying evidence.
Enrich: add asset criticality, identity context, recent changes, and the applicable response procedure.
Decide: show the recommended action, its prerequisites, and the reason for escalation.
Act: send the approved instruction to deterministic automation with narrowly scoped permissions.
Learn: record the analyst’s disposition, edits, approval, execution result, and any reversal.
For each stage, name the owner, permitted inputs, expected output, failure mode, and fallback. If the AI service becomes unavailable, established detections and response paths should continue to work. If the model produces a poor recommendation, an analyst should be able to reject it without fighting the workflow.
This map is also the product specification. It gives security engineering, SRE, product management, and risk owners a shared object to review. It prevents the initiative from collapsing into a feature list such as summarization, chat, and automation without a defined operational result.
Start with one detection decision, not another alert stream
A strong first use case has frequent decisions, usable feedback, and enough context to evaluate the model. It should improve an existing analyst workflow instead of creating a separate queue that someone must remember to check.
Behavioral models can examine endpoint telemetry, identity signals, and network flows to find activity that fixed signatures may miss. The useful product is not the anomaly itself. It is a ranked case that tells the analyst what changed, which evidence drove the score, what asset or identity is exposed, and what decision is required.
Use these criteria to choose the first workflow:
The decision is specific. “Investigate unusual authentication behavior for a privileged identity” is testable. “Use AI to detect threats” is not.
The evidence is available at decision time. If analysts must leave the workflow and search several systems before judging the recommendation, the AI is working with incomplete context.
The disposition is captured. Confirmed threat, benign activity, insufficient evidence, and duplicate are more useful than a generic closed status.
The existing path remains visible. Analysts should be able to compare the AI-ranked case with the evidence they already trust.
A wrong answer is recoverable. Begin with prioritization and investigation support, not an irreversible action.
Do not treat a smaller alert queue as proof of better detection. A model can reduce noise by suppressing useful signals. Measure precision and recall together: precision asks how much surfaced work was relevant, while recall asks how much relevant activity the workflow found. Because missed incidents may become visible only later, define how labels will be corrected when an investigation changes the original disposition.
Mean time to detect also needs a precise starting point. Decide whether the clock begins when the event occurs, when telemetry reaches the platform, or when an existing control first observes it. Otherwise, a faster model can appear to improve detection while ingestion or analyst queue time remains untouched.
The launch question is therefore not “Did the model find anomalies?” Ask whether it moved the right cases forward sooner, preserved the evidence needed for judgment, and avoided pushing material risk below the analyst’s line of sight.
Give the response copilot context, not unchecked authority
Incident response is a natural place for generative AI because analysts repeatedly assemble timelines, summarize evidence, search runbooks, draft ticket updates, and prepare remediation steps. Those tasks are language-heavy, but the actions they inform can disrupt production or destroy evidence.
Use a retrieval-first flow for response recommendations:
Retrieve the approved playbook and the version that applies to the incident type.
Assemble the facts the model is permitted to see, including the alert evidence and relevant asset context.
Generate a recommendation tied to a named playbook step rather than relying on the model’s general memory.
Check prerequisites, identity permissions, environment, and action scope through policy code outside the model.
Present the evidence, proposed action, expected impact, and rollback path to the designated approver.
Execute the approved operation through a deterministic orchestration layer.
Log the retrieved material, prompt, output, approval, tool arguments, result, and subsequent reversal or escalation.
This architecture makes an important distinction: the model can propose an action, but policy and people grant authority. The model should never be able to expand its own permissions or substitute a different tool when the approved operation fails.
An authority ladder gives that distinction operational force. Use the following as a starting policy and adapt it to the blast radius of your environment:
Action class
Examples
AI role
Required control
Read-only support
Summarize evidence, retrieve a runbook, collect approved diagnostics
Generate or execute within a fixed scope
Least-privilege access, complete logging, and no mutation permissions
Reversible operational change
Update a ticket, isolate an endpoint, rotate a credential
Recommend and prepare the action
Named human approval, validated target, impact warning, and tested rollback
High-blast-radius or irreversible change
Block a production network segment, alter broad access policy, delete data or evidence
Explain and escalate only
Incident command process and approval from the responsible system owner
Endpoint isolation can interrupt legitimate work. Credential rotation can break services when dependencies are unknown. Deleting data can permanently remove forensic evidence. Put those consequences beside the approval button, and provide a safe alternative such as collecting more evidence or opening an incident bridge.
Test the copilot as a security product, not as a conversational demo. Your evaluation set should cover correct recommendations, missing prerequisites, conflicting evidence, obsolete playbooks, requests outside the user’s permission, sensitive data, malformed tool arguments, and situations that require refusal or escalation. Measure whether the recommendation is grounded in the approved playbook, whether the action is appropriate, and whether the system preserved the required approval boundary.
Begin in shadow mode, where recommendations are evaluated but cannot change systems. Move next to draft-only assistance. Permit bounded execution only after the team has defined promotion criteria, rollback behavior, and an owner who can stop the workflow.
Prompt and output logs deserve the same access discipline as other sensitive security records. They may contain identities, indicators, configuration details, or incident evidence. Apply contextual data policies before information reaches the model, restrict access to the logs, and make retention a deliberate governance decision rather than a vendor default.
Counter AI-enabled attacks by changing the process
Change the process that turns a convincing message into access, money movement, or sensitive disclosure:
Require an out-of-band verification step for unusual executive requests, especially when the request changes credentials, access, payment details, or normal procedure.
Do not let familiarity with a voice, writing style, profile image, or caller ID serve as identity proof.
Harden identity controls with multifactor authentication, conditional access, and continuous risk scoring.
Give help-desk and operations teams a defined escalation path when a requester applies urgency or asks them to bypass verification.
Train employees with realistic AI-generated lure patterns, then measure reporting behavior and successful compromise rather than course completion alone.
Use AI-assisted red-team exercises to test the process, and use deception controls where they can divert attacker effort without putting production data at risk.
This reframes awareness training. Employees are not expected to become media-forensics experts. They need to notice when a request crosses a risk boundary and know the exact verification step to take. Product leaders can help by removing friction from the safe path: make reporting easy, make escalation visible, and avoid punishing someone who pauses a suspicious request.
The same principle applies to detection. Do not build the defense around whether content “looks AI-generated.” Build it around identity, behavior, privilege, asset sensitivity, and the actions an attacker is attempting.
Use a 90-day plan with measurable promotion gates
A focused 90-day plan is enough to establish an operating model if you keep the scope narrow: one high-signal detection decision, one mature response playbook, and one employee risk path such as phishing. The purpose is not to automate the security operation in a quarter. It is to prove that the decision loop can become faster without weakening control.
Days 1-30: define the workflow and baseline
Map the current signal-to-action path and identify where time, context, or consistency is lost.
Name a product owner, security owner, model-risk owner, and operational approver for the workflow.
Select the detection decision, response playbook, and employee risk process in scope.
Record baseline mean time to detect, mean time to recover, queue time, disposition quality, and the existing failure modes.
Define the data the model may access, the data it must not access, and the identity under which each tool operation runs.
Write the authority ladder, fallback behavior, stop condition, and rollback procedure before connecting production tools.
Days 31-60: evaluate in shadow mode
Run the detection model beside the existing workflow and compare ranked cases with analyst dispositions.
Test response recommendations against approved playbooks, including ambiguous and adversarial cases.
Review false positives and false negatives with analysts instead of reducing model quality to one aggregate score.
Confirm that sensitive-data policies, model access controls, prompt and output logging, and audit access work as designed.
Run a tabletop exercise covering model failure, unavailable retrieval, unsafe recommendations, excessive permissions, and orchestration failure.
Set promotion criteria for model quality, operational benefit, privacy, access control, and reversibility. Use thresholds appropriate to the risk of the chosen workflow rather than copying a generic benchmark.
Days 61-90: release bounded capability
Release the detection workflow to a defined analyst group while preserving the established fallback.
Enable draft-only response assistance before allowing any system mutation.
Permit only the actions covered by the approved authority policy; keep high-blast-radius changes outside model execution.
Review analyst edits, rejections, approvals, reversals, and escalations to find where the workflow lacks context.
Compare mean time to detect and recover with the baseline, while checking that precision, recall, privacy, and control failures have not regressed.
Make the next release decision explicitly: expand, hold, narrow the scope, or stop. A pilot that exposes an unsafe assumption has still produced a useful result.
The dashboard should separate outcomes from guardrails. Detection and recovery time tell you whether the operation improved. Precision, recall, recommendation correctness, and playbook grounding tell you how the model behaved. Rejections, manual edits, reversals, unauthorized-action attempts, and sensitive-data policy violations tell you whether the workflow is safe enough to scale.
Acceptance rate alone is not a quality metric. Analysts may accept a recommendation because it is correct, because the interface makes editing difficult, or because workload encourages quick approval. Review the resulting action and later incident outcome, not only the click.
Governance must continue after launch. Assign an owner to every model-enabled workflow, control access by role and context, version the model and retrieved playbooks, retain an auditable decision record, test for drift and bias, and repeat tabletop exercises when permissions or orchestration change. A model update is a security-product release, even when it arrives through a managed vendor.
Key takeaways
Optimize the full signal-to-action loop; do not add a disconnected AI queue.
Let models detect, summarize, and recommend, while policy and named people control authority.
Ground response guidance in approved, versioned playbooks before generating remediation steps.
Use shadow mode, draft-only assistance, and bounded execution as separate promotion stages.
Defend against convincing AI-generated lures by hardening identity and verification processes, not by expecting perfect human detection.
Your next operating review should end with three named decisions: the detection workflow you will improve, the response action the AI may only recommend, and the metric that would stop the release. Once those are explicit, AI becomes a governable capability instead of an open-ended security experiment.
