Tag: AI workflows

  • How Product Leaders Turn AI Strategy Into an Operating System

    How Product Leaders Turn AI Strategy Into an Operating System

    Your AI roadmap probably isn’t short of ideas. The hard decision is which ideas deserve production responsibility: a user promise, a quality bar, a failure path, an owner, and a reason to keep funding them after launch.

    You operationalize AI by turning those decisions into a repeatable management system. The broader shift from experiments to execution makes that system more important than any individual model choice. It lets your teams discover useful applications, ship them responsibly, teach customers how to use them, and decide from evidence whether to scale, change, or stop.

    Turn AI ambition into a portfolio of bounded bets

    An AI strategy is not a list of places where a model could be added. It is a set of choices about which customer or business problems deserve investment, how much authority AI should receive, and what evidence will justify the next commitment.

    Start every candidate with a one-page opportunity contract. If the team can describe the model but cannot complete the contract, the idea is not ready for prioritization.

    • User and moment: Name the person, the task they are trying to complete, and the point in the workflow where the difficulty occurs.
    • Current behavior: Record how the task works without the proposed feature. Use an observable baseline such as completion, elapsed time, handoffs, abandonment, rework, or cost per completed task.
    • AI contribution: State whether AI will classify, retrieve, recommend, generate, summarize, or take an action. Avoid vague phrases such as “AI-powered experience.”
    • Expected change: Identify the user behavior that should change first and the customer or business outcome that should follow.
    • Boundaries: List what the system must not decide, which data it must not use, and which users or scenarios are outside the initial release.
    • Consequence and reversibility: Describe what happens when the system is wrong and whether the user can inspect, correct, undo, or escalate the result.
    • Next evidence: Define the smallest test that could reduce the most important uncertainty. That might be a workflow prototype, customer discovery, a retrieval test, or an evaluation against representative cases.

    This contract forces an important distinction between assistance and authority. Drafting a reply for a person to review is not the same product as sending that reply automatically. Recommending an account action is not the same as applying it. The second version has a larger blast radius, a different trust requirement, and a stricter need for auditability and recovery.

    Begin with the minimum authority required to create value. Increase autonomy only when the evidence supports it. This is not timidity. It is a sequencing decision that lets you learn about quality and user behavior before accepting a larger operational risk.

    Prioritize the resulting bets across six lenses: customer value, workflow frequency, data readiness, evaluability, blast radius, and operating cost. Do not collapse them into a decorative score that hides disagreement. Use them to expose the trade-off. A frequent, valuable task may still be a poor first bet if critical failures cannot be detected. A low-risk task may be easy to ship but too marginal to earn repeat use.

    Write a stop condition at the same time as the investment case. For example: stop if the team cannot construct a credible evaluation set, if the workflow requires data the product cannot responsibly access, or if users do not reach the intended outcome after the experience and onboarding have both been tested. A portfolio becomes manageable when stopping is a designed decision rather than an admission of defeat.

    Define production readiness before the team starts building

    A prototype proves that a system can produce a compelling result once. A product must produce an acceptable result across the situations that matter, make its limitations understandable, and recover when the result is not acceptable.

    Give each AI bet a production contract before it enters committed delivery. The contract should contain:

    • The user promise: Describe what the product will help the user accomplish. Do not promise intelligence in the abstract.
    • The context boundary: Specify which product data, retrieved knowledge, instructions, tools, and prior interactions the system may use.
    • The quality dimensions: Choose criteria that fit the task, such as correctness, completeness, groundedness, policy compliance, tool execution, tone, or structured-output validity.
    • Scenario-specific thresholds: Set release criteria for meaningful segments and failure types instead of relying on one average score. The acceptable standard for brainstorming copy is not the acceptable standard for changing an account or communicating a binding decision.
    • The fallback: Define what the user sees and can do when confidence is inadequate, a tool fails, retrieval returns weak context, or the output violates a rule.
    • The operating envelope: Set the latency, reliability, and cost constraints needed for the workflow to remain viable.
    • The data rules: Record what may be retained, what must be removed, who can inspect traces, and how sensitive information is handled.
    • The instrumentation plan: Name the events, evaluation results, feedback, escalations, and outcome measures required to make the next decision.

    There is no universal quality threshold for an AI feature. The right threshold depends on the consequence of an error, the user’s ability to detect it, and the availability of a safe recovery path. Set the bar by scenario and harm, then make the release decision against that bar. An aggregate average can conceal a severe failure in a smaller but important segment.

    Build the evaluation set before tuning the experience

    Create a versioned evaluation set from the workflow you intend to support. Include ordinary cases, meaningful variations, known edge cases, and inputs that should trigger a refusal, clarification, or handoff. Label the expected outcome and the unacceptable failure. Do not require exact wording unless exact wording is part of the product requirement.

    Run that set against the initial baseline and after changes to prompts, models, retrieval, tools, policies, or orchestration. Preserve results by scenario so the team can see both improvements and regressions. A single overall score is useful for orientation; it is not enough for a launch decision.

    Automated checks work well for properties that can be specified clearly, such as output structure, required fields, tool completion, forbidden content, or citation presence. Use structured human review where quality depends on judgement. Keep the rubric stable enough to compare versions, and change it deliberately when the product promise changes.

    Design the failure experience as part of the feature

    Users do not experience your evaluation score. They experience a suggestion they cannot verify, a slow response, an action they did not intend, or a dead end after the system fails. Design those moments before launch.

    • Show the context or inputs that materially shaped the result when doing so helps the user judge it.
    • Make generated content editable before it becomes externally visible.
    • Require explicit confirmation before consequential or difficult-to-reverse actions.
    • Preserve the original state and provide rollback where the underlying workflow permits it.
    • Offer a clear manual path when the system cannot complete the task.
    • Capture corrections and escalations as learning signals without treating every user edit as proof that the system was wrong.

    Do not place sensitive production data into an unapproved model, connector, or testing tool. The downside can include unauthorized disclosure, retention outside your controls, and regulatory or contractual exposure. Use an approved environment and appropriately protected or de-identified test material while privacy and security owners validate the production path.

    Run one decision loop from discovery through scale

    AI initiatives become expensive when discovery, delivery, launch, and governance operate as separate queues. The useful unit of management is one decision loop with shared artifacts, named owners, and explicit gates.

    1. Discover the workflow: Observe the current task, its failure points, the information available at the decision moment, and the user’s existing workarounds. Validate that the problem matters before testing how impressive a model can appear.
    2. Shape a complete slice: Select the smallest workflow that can deliver an outcome, including its context, interface, recovery path, and instrumentation. A prompt without those elements is a component, not a product increment.
    3. Pass the build gate: Approve committed delivery only when the opportunity contract, production contract, evaluation set, data path, and accountable owners are credible.
    4. Deliver through normal product planning: Put evaluation cases, telemetry, fallback behavior, privacy work, and operational readiness into the roadmap and sprint scope. Do not leave them in a separate “hardening” phase after the visible feature is complete.
    5. Launch a new behavior: Use onboarding, in-app guidance, examples, and product tours to show when the capability is useful, what input it needs, and how the user should review the result. The activation event should represent completed value, not a button click.
    6. Review and decide: Compare outcomes with the baseline, inspect evaluation performance by scenario, locate adoption drop-offs, and review cost, reliability, incidents, and new risks. End with a decision to scale, revise, constrain, or stop.

    A practical ownership split keeps this loop moving. Product owns the customer outcome, scope, adoption, and portfolio decision. Engineering owns the production system, reliability, observability, and cost controls. Design owns comprehension, user control, and recovery in the experience. The evaluation owner maintains cases, rubrics, baselines, and regression visibility. Privacy, security, legal, or compliance owners define required controls according to the risk. The business or operational owner defines any human review policy and accepts changes to the real-world process.

    One directly responsible leader should assemble the evidence and drive the launch recommendation, but that role does not erase specialist approval where it is required. Record the decision, conditions, and unresolved risks. Otherwise the same debate returns at every review and nobody can tell why the system was allowed to progress.

    Use risk-tiered oversight. A reversible drafting aid with no sensitive data does not need the same review path as an agent that changes customer records, sends external communications, or initiates a financial action. Increase review, auditability, confirmation, and monitoring as authority and consequence increase. This keeps governance proportional and makes the path to approval understandable before work begins.

    At each portfolio review, use the same compact decision packet: baseline and current outcome, scenario-level evaluation movement, activation funnel, operating performance, incidents or policy exceptions, learning completed, and the next requested commitment. A polished demonstration can support the discussion, but it cannot substitute for this evidence.

    Measure value, quality, adoption, and risk separately

    AI dashboards become misleading when usage, answer quality, customer value, and system health are blended into one success number. They answer different questions and lead to different decisions. Keep the layers separate, then connect them with a driver tree.

    LayerQuestionUseful measuresDecision it informs
    Customer or business outcomeDid the workflow become meaningfully better?Task completion, resolution, conversion, elapsed time, rework, or cost per successful outcomeWhether the use case deserves continued investment
    User behaviorAre eligible users reaching and repeating the value?Eligibility, exposure, first attempt, successful completion, repeat use, abandonment, fallback, and escalationWhether to change positioning, onboarding, interaction design, or workflow placement
    System qualityIs the result fit for the intended task?Scenario pass rate, human rubric results, groundedness where required, tool success, structured-output validity, and critical-failure countWhether to change context, retrieval, prompts, models, tools, or scope
    OperationsCan the product deliver the experience sustainably?Latency, reliability, retries, failure rate, incidents, and cost per successful taskWhether architecture and unit economics support scale
    Risk and controlAre safeguards working at the level of authority granted?Policy exceptions, unauthorized actions, sensitive-data events, confirmations, rollbacks, and human escalationsWhether to add controls, reduce authority, constrain availability, or pause

    Build the adoption funnel around the real workflow: eligible user, meaningful exposure, first attempt, successful outcome, and repeat use when the need occurs again. Define the repeat window from the natural frequency of the task. A daily workflow and a quarterly workflow cannot share a useful retention window.

    Do not mistake interaction volume for value. More messages can mean the user is retrying after poor results. A low cost per response can hide an expensive task that requires several responses and a manual correction. Favor successful outcomes per eligible user and cost per successful outcome, then use interaction-level metrics to diagnose what happened inside the journey.

    The metric layers also tell you where to intervene:

    • If evaluation quality is acceptable but activation is weak, inspect discoverability, positioning, onboarding, and whether the feature appears at the right workflow moment.
    • If first use is strong but successful completion is weak, inspect inputs, context retrieval, interaction design, tool execution, and recovery.
    • If completion is strong but repeat use is weak, verify that the use case is naturally repeatable and that the experience created enough value to displace the old behavior.
    • If adoption is strong but critical failures or operating costs are outside the contract, constrain the release while you fix the production system. Popularity does not neutralize risk or poor economics.
    • If the outcome improves, scenario evaluations remain acceptable, users return when the need recurs, and operating constraints hold, you have evidence to expand availability or authority.

    This is how measurement becomes a funding mechanism rather than a reporting ritual. Each signal points to a different action, and each review produces a clear next commitment.

    Key takeaways for your next AI portfolio review

    • Treat every AI idea as a bounded product bet with a named user, baseline workflow, expected outcome, authority level, and stop condition.
    • Require a production contract covering quality, evaluation, fallback, data, economics, instrumentation, and failure recovery before committed delivery begins.
    • Build privacy, evaluation, telemetry, onboarding, and operational readiness into the roadmap and sprint scope instead of postponing them until launch.
    • Grant the minimum authority needed to create value, then expand autonomy only when quality, adoption, control, and operational evidence support it.
    • Measure customer outcomes, user behavior, system quality, operations, and risk as connected but distinct layers.
    • End every review with an explicit decision to scale, revise, constrain, or stop, plus the evidence required for the next decision.

    At your next portfolio review, choose one leading AI candidate and refuse to discuss the model first. Write the opportunity contract, define its production bar, assign the owners, and identify the first complete workflow you can measure. If those decisions are clear, the technology has a path to become a product. If they are not, another prototype will only postpone the real work.

    References

  • The New AI Playbook for Product Portfolio Optimization: Slash Complexity, Boost ROI

    The New AI Playbook for Product Portfolio Optimization: Slash Complexity, Boost ROI

    The most valuable lesson I’ve learned leading product organizations is that portfolio choices make or break outcomes. In an era of infinite requests and finite teams, the question isn’t what we could build—it’s what we must build next. That’s why I’m codifying a pragmatic, AI-driven playbook to optimize the product portfolio while staying true to outcomes, not output.

    AI-powered product portfolio optimization is here. Explore strategies and tools helping product leaders manage complexity and boost ROI.

    My starting point is a data backbone that connects strategy to reality. I aggregate product usage, revenue by segment, cost-to-serve, retention cohorts, and support signals into a unified analytics platform, then layer a retrieval-first pipeline so LLMs can reason over clean context. Instrumentation matters: Amplitude analytics, Pendo, and in-app guides provide the behavioral and activation signals that make prioritization measurable.

    From there, I translate strategy into an objective decision system. I express outcomes vs output OKRs, align initiatives to value proposition and competitive differentiation, and classify opportunities with the Kano Model. LLMs for product managers help cluster voice-of-customer at scale; with thoughtful prompt engineering and AI workflows, I can map themes to jobs-to-be-done, quantify demand, and de-duplicate asks across stakeholders.

    Execution hinges on evidence. I run A/B testing with a clear minimum detectable effect (MDE), pair it with eval-driven development for AI features, and ship through CI/CD while tracking DORA metrics. This closes the loop between product roadmapping and sprint planning and real-world performance—activation, retention analysis, and Web Vitals inform the next set of portfolio bets.

    Trust is a feature, so governance is built-in. Privacy-by-design, data governance, and AI risk management guide how we store, prompt, and evaluate models. I apply guardrails to sensitive workflows and define success metrics that balance short-term ROI with long-term resilience and regulatory compliance.

    The operating model matters as much as the models themselves. Product trios and empowered product teams run continuous discovery, pressure-test assumptions in QBRs vs OKRs, and make trade-offs visible. Stakeholder management becomes easier when the portfolio narrative is anchored in transparent scenarios and shared metrics.

    If you’re getting started, here’s my flow: unify data, define outcomes, segment opportunities, simulate scenarios, and test fast. Use LLMs to synthesize signals you’d never humanly read, then make one focused bet per team that moves a measurable KPI. Rinse, learn, and reallocate—portfolio optimization is a living system, not an annual meeting.

    Ultimately, the promise of this new playbook is simple: less noise, sharper focus, and compounding ROI. By pairing AI Strategy with disciplined product management leadership, we can manage complexity with clarity—and consistently build what matters most.

    Inspired by this post on Product School.

    Book a consult png image
  • 10 AI Business Models You Need Now: Proven Playbooks Turning Algorithms into Revenue

    10 AI Business Models You Need Now: Proven Playbooks Turning Algorithms into Revenue

    I’ve spent the past few product cycles re-architecting roadmaps around one simple reality: AI is no longer just a feature—it’s a business model. The companies winning market share are those that treat models, data, and workflows as monetizable assets with defensible moats, not science projects.

    AI business models are rewriting value creation. Learn how smart teams turn algorithms into profit engines, reshaping entire industries.

    From my seat in product leadership, I evaluate AI bets through three lenses: durable value (moat and differentiation), measurable outcomes (clear ROI), and unit economics (gross margins under real-world load). With that frame, here are ten AI business models I see performing now—and how I decide when to invest.

    1) API-first Model-as-a-Service. I monetize foundation or specialized models via an API, priced by tokens, requests, or time-in-context. Success hinges on latency, accuracy, and “context window management” that balances quality with cost. This is where “consumption SaaS pricing” shines and where disciplined rate-limiting, observability, and SLAs build trust.

    2) Vertical AI copilots. I package domain-specific expertise (legal, healthcare, finance, field service) into workflow-native assistants that surface next-best actions. Because these copilots live where work happens, I price on outcomes—time saved, revenue recovered, or risk reduced—aligning value with customer metrics and accelerating product adoption.

    3) Agentic AI automation. When autonomous agents handle multi-step tasks across tools, I lean toward per-outcome or per-job pricing. Reliability is the moat, so I invest early in eval-driven development, robust guardrails, and human-in-the-loop QA. This model compounds fast once agents can execute end-to-end workflows with transparent audit trails.

    4) Copilot add-ons inside existing SaaS. I’ve seen “AI Assist” tiers deliver immediate ARPU lift and retention gains. The playbook: start with high-frequency, high-friction jobs (drafts, summaries, enrichment), then expand to proactive suggestions. This aligns tightly with product strategy and lets me stage value without overhauling the core experience.

    5) Insights-as-a-Service via data network effects. I transform exhaust data into benchmarking, predictions, and prescriptive recommendations—while honoring privacy-by-design and data governance. The more customers I onboard, the stronger the patterns, and the higher the switching costs. Pricing ties to seats plus an outcomes or value metric.

    6) Retrieval-first pipeline for enterprise knowledge. I land with high-accuracy answers over customer data (search, summarize, cite), then expand into workflow automations. This “retrieval-first pipeline” reduces hallucinations, boosts trust, and creates defensibility through connectors, semantic indexing, and continuous relevance tuning—an ideal fit for LLMs for product managers prioritizing reliability.

    7) Open source monetization. When I bet on openness, I monetize hosting, support, enterprise controls, and compliance features. The advantage is developer love and rapid iteration; the moat is operational excellence at scale, plus integrations customers rely on. This model converts community momentum into predictable revenue.

    8) Marketplaces for prompts, skills, and agents. I create a platform for third-party extensions and charge a take rate on usage. The flywheel spins when developers see distribution, customers see breadth, and I enforce strong quality bars. The roadmap focuses on governance, discovery, and safe execution policies.

    9) Solutions with forward deployed engineers. For complex rollouts, I pair product with specialized implementation to guarantee outcomes. Revenue blends software plus services, accelerating time-to-value and informing the roadmap with real-world constraints. Over time, learnings fold back into scalable, self-serve capabilities.

    10) AI risk, security, and compliance tooling. As AI scales, so does the need for policy enforcement, monitoring, and auditability. I monetize via platform subscriptions that address model provenance, data leakage prevention, red teaming, and reporting. Strong “AI risk management” is now a purchasing requirement, not a nice-to-have.

    How do I choose among these models? I start with the customer’s biggest workflow pain, map it to the fastest path to measurable outcomes, and align pricing with value creation. Then I build defensibility through data advantage, distribution, and governance. If a model deepens trust, improves margins, and compounds learning, it earns a place on the roadmap.

    Inspired by this post on Product School.

    Book a consult png image
  • Monetizing AI with Confidence: Proven Models, Smart Pricing, and ROI You Can Defend

    Monetizing AI with Confidence: Proven Models, Smart Pricing, and ROI You Can Defend

    I’ve learned the hard way that shipping an impressive AI demo is not the same as creating a durable revenue engine. In my role leading product strategy, I focus on one goal: connect AI capabilities to measurable customer outcomes, then price and package them so both value and margins are visible and defensible.

    Monetizing AI features into profit isn’t trivial. Here are some clear strategies for capturing and pricing AI products and how to monetize with returns.

    First, I clarify the business model. Add-on AI packs work when the value is concentrated in a specific workflow (for example, automated summarization or AI copilot assistance). Tiered packaging helps when AI elevates the overall experience across many features. Usage-based or consumption SaaS pricing is ideal when value scales with volume—tokens, documents processed, calls handled, or agents invoked—because it aligns price to realized outcomes.

    Next, I align pricing mechanics with the customer’s value story. I anchor price against the baseline they know: hours saved, conversions gained, cases deflected, or risk reduced. Then I set floors based on unit economics—model inference, vector storage, and orchestration costs—so gross margins remain healthy as usage grows. Clear guardrails (quotas, rate limits, and context window management) prevent surprise bills and keep cost-to-serve predictable.

    Packaging is where monetization becomes intuitive. I gate high-cadence, high-compute features behind premium tiers, and I expose quick wins (like smart suggestions) in core tiers to accelerate activation. For enterprise, I bundle governance, audit logs, data controls, and “privacy-by-design” features to justify step-up pricing and reduce procurement friction.

    To sustain ROI, I run an eval-driven development loop. I define quality metrics (accuracy, helpfulness, latency, safety) and instrument the retrieval-first pipeline so I can isolate where value is created or lost. This lets me right-size models, tune prompts, and swap components without compromising outcomes or margins—critical for LLMs for product managers who must balance experience and cost.

    Measurement is non-negotiable. I track activation, time-to-first-value, weekly engaged AI users, and feature-level retention. For revenue impact, I attribute uplift through A/B testing and minimum detectable effect thresholds, measuring conversion lift, ticket deflection, and cycle-time reductions. When customers see these numbers in their own dashboards, procurement turns into partnership.

    Risk and compliance are part of the product, not an afterthought. I build in AI risk management, data governance, and red-teaming from day one. Clear data boundaries, human-in-the-loop controls, and transparent disclosures protect end users and make enterprise legal teams our allies rather than blockers.

    Go-to-market matters as much as the model. I use product-led growth tactics—free AI credits, transparent meters, and in-app guides—to let users feel the value before the paywall. Sales enablement centers on the value proposition: faster outcomes, higher quality, and lower total cost of ownership, not just “gen ai” for its own sake. Pricing pages should showcase tiers, usage bands, and outcomes, eliminating guesswork.

    Here’s the simple playbook I follow: validate the problem with continuous discovery, instrument the workflow, pilot with generous caps, and collect willingness-to-pay signals early. Then iterate the price meter, refine units of value (documents, messages, or actions), and align SKUs to buyer personas. Over time, I introduce agentic AI capabilities as premium modules when they demonstrably reduce steps or automate entire objectives.

    When AI monetization works, it feels effortless to customers because the price mirrors the outcome. When it doesn’t, it’s usually because packaging hides value, pricing ignores unit economics, or ROI isn’t visible. By grounding strategy in value metrics, consumption-aware pricing, and rigorous evaluation, I’ve found we can scale AI revenue with confidence—and keep both customers and margins happy.

    Inspired by this post on Product School.

    Book a consult png image
  • A Practical AI Workflow for Product Manager Cover Letters

    A Practical AI Workflow for Product Manager Cover Letters

    You have found a product role that fits, but the blank page is slowing you down. AI can produce a polished draft in seconds. That is not the hard part. The hard part is choosing the evidence that will make a hiring manager believe you can solve this company’s product problems.

    Your cover letter should make one decision easier: whether to interview you. The workflow below helps you turn a job description and your verified career evidence into a short, role-specific argument without surrendering your judgment or voice to an AI tool.

    Design the letter for the hiring manager’s first scan

    Plan for a first scan of under 30 seconds and a final length of 200-300 words. That constraint is useful. It forces you to decide which parts of your experience matter for this role instead of compressing your entire resume into prose.

    A strong PM cover letter gives the reader evidence for a few practical questions:

    • Do you understand the customer and product problem behind the role?
    • Have you made consequential product decisions, or have you only participated in product processes?
    • Can you connect your work to activation, adoption, retention, revenue, efficiency, or another relevant outcome?
    • Can you work with engineering and other functions to turn an ambiguous problem into a shipped, measured result?
    • Why is this experience useful to this company now?

    You do not need to answer every question with a separate story. Choose the few competencies the role emphasizes and make every paragraph carry evidence for at least one of them. If a sentence does not improve the case for interviewing you, it is consuming scarce attention.

    Key takeaways

    • Write one argument for one role, not a general biography that could accompany every application.
    • Build a verified evidence bank before asking AI to draft anything.
    • Use AI to extract requirements, map evidence, produce alternatives, and critique the result. Do not use it to invent facts.
    • Show decisions and outcomes rather than restating responsibilities from your resume.
    • Keep the final letter to 200-300 words and make sure it still sounds like something you would say.

    Build a truth set before you open the drafting prompt

    Generic AI writing usually begins with incomplete inputs. If you provide only the job description and your resume, the model has to guess which experiences matter, how they connect, and what tone represents you. Its guesses may sound plausible while being strategically weak or factually unsafe.

    Give the model two structured inputs instead: a role brief and an evidence bank. The role brief describes what the employer appears to need. The evidence bank contains only claims you can defend in an interview.

    Create the role brief

    Read the job description once as a candidate and again as a product manager diagnosing a problem. Separate broad language such as ownership or collaboration from concrete expectations such as improving onboarding, scaling a platform, conducting discovery, positioning a product, supporting go-to-market execution, or aligning stakeholders.

    Then use this prompt:

    Prompt: Extract the core competencies and product problems from this job description. For each one, include the exact phrase that supports your interpretation, the likely work involved, and the evidence a hiring manager would need to see. Group duplicate or overlapping requirements. Do not write a cover letter and do not infer company facts that are not stated.

    Review the output yourself. A repeated phrase can be a signal, but frequency alone does not establish priority. Pay particular attention to responsibilities described as immediate, core, accountable, or tied to a named business or customer problem.

    Create the evidence bank

    For each relevant experience, record the elements that make it usable:

    • Context: the product, customer, market, or operational setting.
    • Signal: what you learned from customers, data, the market, or internal constraints.
    • Decision: what you chose, changed, prioritized, delayed, or rejected.
    • Trade-off: what competing concern made the decision difficult.
    • Collaboration: how engineering, design, go-to-market, operations, or executives participated.
    • Outcome: what changed and how you measured it.
    • Business meaning: why that change mattered beyond the product metric.

    Give every evidence record a simple label such as E1 or E2. Preserve the exact metric, timeframe, scope, and level of ownership you can support. If you influenced a decision, do not let the draft say you owned it. If you know the direction of an outcome but not a defensible number, do not add a precise percentage.

    Now ask AI to map evidence rather than manufacture a narrative:

    Prompt: Map the evidence records to the role brief. Use only the supplied facts. For every proposed claim, cite its evidence label. Mark a requirement as unsupported when there is no credible match. Recommend the strongest role-specific examples, but do not draft the letter yet.

    This mapping exposes a weak application early. If the central requirement has no supporting evidence, another round of prompting will not solve the problem. You may need a more honest adjacent example, a narrower claim, or a decision not to invest further in that application.

    Use AI as an analyst, variant generator, and critic

    The useful AI workflow is not a single command to write a great cover letter. It is a sequence that separates analysis from evidence selection and writing. That separation makes errors easier to notice and revisions easier to control.

    1. Extract the role’s competencies and product problems.
    2. Map your verified evidence to those requirements.
    3. Build an outline in which every paragraph has a defined job.
    4. Generate alternative versions from the approved outline.
    5. Audit the strongest version for unsupported claims, weak reasoning, generic language, and voice.

    This follows a practical pattern: extract the competencies, draft an outline, compare alternatives, and then refine tone and clarity. You retain the decisions that matter: which evidence is fair, which trade-off is important, and which version represents you.

    Generate alternatives without losing factual control

    Light A/B testing in this context means comparing two drafts against the same rubric. It does not mean sending different claims to the same employer. Hold the evidence constant and vary the framing.

    Prompt: Write two cover-letter drafts of 200-300 words from the approved outline. Use only facts tied to evidence labels. Draft A should lead with the customer and product problem. Draft B should lead with the most relevant product outcome. Preserve any unresolved fact as a visible placeholder. Do not add company praise, metrics, technologies, or scope that I did not provide.

    Do not ask the model which version is best without defining best. Have it compare the drafts on role relevance, evidence integrity, decision clarity, outcome clarity, company specificity, and consistency with your normal voice. The winning draft is not necessarily the most fluent one. It is the one that makes the strongest truthful case with the least reader effort.

    Run a claim-level audit

    Before polishing, force the model to show its work:

    Prompt: Audit every sentence in this draft. For each sentence, identify the role requirement it serves, the evidence label that supports it, and any wording that overstates ownership, causality, scope, or certainty. Flag generic sentences that could be sent unchanged to another company. Do not rewrite until the audit is complete.

    Review every flag manually. AI can detect a mismatch between the draft and the material you supplied, but it cannot determine whether your underlying memory is accurate. That remains your responsibility.

    Draft the cover letter as a four-part product argument

    A compact PM cover letter works when each part performs a different function. You need a value proposition, evidence of judgment, evidence of collaboration, and a specific connection to the company’s current need.

    Open with relevance, not ceremony

    Your first sentence should connect the product problem you solve, the customer you understand, and the outcome you tend to drive. Enthusiasm can appear later, but it cannot substitute for relevance.

    Use this pattern: I build [product or capability] for [customer], turning [important problem] into [verified outcome]. The need for [role-specific competency] is where my experience with [relevant context] is most applicable.

    Replace every bracket with evidence. If the sentence becomes crowded, remove a concept rather than stacking more clauses. The opening is a positioning statement, not an executive summary of your career.

    Prove product judgment with a decision

    The central paragraph should show how you converted an ambiguous signal into a product decision. Duties describe the process around you. Decisions reveal your judgment within it.

    Use this pattern: When [customer or product signal] revealed [problem], I chose [decision] over [alternative] because [trade-off]. Working with [relevant partners], I [execution mechanism], which changed [verified outcome] and mattered because [business value].

    Quantify impact when you have a defensible measure. Activation, retention, and adoption can be stronger evidence than vanity metrics when they reflect the actual goal of the work. If a valid number is unavailable, name the observable outcome without inventing precision.

    Show how the work moved through the team

    Product leadership is not demonstrated by adding cross-functional to a list of adjectives. Show the mechanism. Did you create clarity from conflicting customer signals? Did you align engineering around a platform trade-off? Did discovery change the roadmap? Did positioning work alter the go-to-market plan?

    Your second role-specific example can be shorter than the first. Use it to prove that you can partner with an empowered product team and move from insight to delivery without claiming everybody else’s work as your own.

    Close on the problem ahead

    The closing should answer why this company and why now without turning into a paragraph of praise. Connect a need visible in the role description to the experience you have already proven. If you refer to the company’s product, roadmap, market, or customers, use only information you have verified.

    Use this pattern: The opportunity to [role-specific problem or responsibility] is a direct match for my experience in [evidence-backed capability]. I would welcome a conversation about how that experience could help [company’s stated objective].

    That is enough. A confident close asks for the next conversation. It does not need to repeat the opening, summarize every paragraph, or declare that you are the perfect candidate.

    Edit until every sentence earns its space

    The final editing pass is where a serviceable AI draft becomes your cover letter. Check the logic before polishing the language.

    • Role mapping: Does every paragraph connect to a core requirement, or is it merely impressive in isolation?
    • Decision clarity: Can the reader identify what you decided and why?
    • Outcome clarity: Does the letter describe a change in customer or business results rather than a list of shipped outputs?
    • Ownership accuracy: Are you distinguishing between led, owned, influenced, partnered, and supported?
    • Company specificity: Could any sentence be sent unchanged to several unrelated employers?
    • Evidence integrity: Can you defend every metric, scope claim, and causal statement in an interview?
    • Voice: Would you naturally use these words when speaking with a hiring manager?
    • Compression: Can you remove a clause without losing evidence or meaning?

    Repair the common AI failure patterns

    • Job-description echo: If the draft says you are skilled in discovery, strategy, and stakeholder management, replace the list with one decision that demonstrates the relevant capability.
    • Resume narration: If a paragraph walks through successive roles, cut the chronology and keep the experience that maps directly to this job.
    • Adjective stacks: Replace strategic, innovative, data-driven, and customer-centric with a concrete signal, choice, or measurement.
    • Unsupported certainty: Change claims about the company’s strategy or roadmap unless you verified them. The job description can support a connection, but it does not give you inside knowledge.
    • Manufactured causality: Do not say your action caused an outcome when the available evidence supports only contribution or association.
    • Borrowed voice: Remove phrases you would not say aloud, even if they sound polished. Fluency is not authenticity.

    Keep a reusable evidence bank and a core structural template, but create a fresh evidence map for each serious application. Slot in two role-specific examples, run the claim audit, and read the final version aloud. If a sentence is difficult to say naturally, it will probably be difficult to defend naturally in an interview.

    For your next application, do not begin by asking AI to write. Begin by deciding what the employer needs to believe and which verified experience gives them a reason to believe it. Once those decisions are sound, AI can help you express them faster. Send the letter when it is concise, specific, and unmistakably yours.

    References

  • Context-Driven AI Product Engineering That Survives Production

    Context-Driven AI Product Engineering That Survives Production

    Your AI feature can look excellent in a demo and still fail in production. The prompt has not changed, but the user, account, permissions, available data, and business decision have. A fluent answer built on the wrong context is still the wrong answer.

    If your team keeps rewriting instructions to fix inconsistent results, inspect what the model can see, why it can see it, and what it is expected to do with that information. Context-driven AI product engineering turns those decisions into a versioned, measurable product system rather than hiding them inside one large prompt.

    Determine whether context is actually the bottleneck

    Runtime context is the complete package available to the model for a specific task. It includes instructions, retrieved evidence, permissions, conversation state, memory, tool definitions, metric definitions, output requirements, and stop conditions. Prompt text is only one part of that package.

    This distinction matters because different failure classes require different fixes. A prompt change cannot retrieve a missing CRM record. A larger model cannot make a stale policy current. Better prose cannot repair an authorization error. Start by assigning every bad result to the layer that produced it.

    • Evidence is missing: the necessary record, document, event, or metric never reached the system.
    • Evidence was available but not selected: retrieval, filtering, metadata, or ranking favored the wrong material.
    • Evidence is stale or contradictory: the system lacks a freshness rule or conflict-resolution policy.
    • The procedure is incomplete: the model has facts but not the sequence, metric definition, or decision rule needed to use them.
    • The scope is unsafe: the context contains data the current user, role, tenant, or workflow should not access.
    • The answer contract is unclear: the model does not know when to cite evidence, expose uncertainty, request missing input, call a tool, or abstain.
    • The answer is technically correct but operationally unhelpful: it does not fit the user’s role, decision, timing, or next action.

    For one failed session, reconstruct the full path instead of reading only the final answer:

    1. Capture the user’s request, detected intent, role, tenant, and relevant permissions.
    2. Record the retrieval queries, filters, candidate results, metadata, and ranking scores.
    3. Show which candidates entered the context, which were excluded, and why.
    4. Inspect the assembled instructions, evidence, memory, tool contracts, and output schema.
    5. Record every tool call, returned result, retry, timeout, and policy decision.
    6. Compare the answer with the evidence that was actually available at generation time.

    The resulting trace gives you a practical decision tree. If the correct evidence was absent from the candidate set, fix ingestion or retrieval. If it was retrieved but excluded, fix ranking or context packing. If it entered the prompt but the answer contradicted it, test instruction hierarchy, conflict handling, or model behavior. If the evidence and answer were both correct but the user still could not act, fix the product experience.

    This is why a retrieval-first, context-aware design usually creates more leverage than another round of isolated prompt editing: it makes the evidence path visible and gives each failure an identifiable owner.

    Write a context contract before choosing the architecture

    A context contract defines what the AI needs for one product task, where that context may come from, how it must be constrained, and what the system should do when the contract cannot be satisfied. It is the interface between product intent and runtime engineering.

    Consider an account-risk assistant used by a customer success manager. Its contract could look like this:

    Contract fieldDecision to makeExample implementation
    Task boundaryWhat may the AI decide or produce?Summarize risk signals and propose a next step; do not change the account record.
    Authorized evidenceWhich information is both relevant and permitted?CRM fields, recent support history, approved playbooks, and defined product-usage metrics visible to the current user.
    Identity and scopeWhich user, tenant, account, and role govern access?Resolve all four before retrieval and preserve them through every tool call.
    FreshnessHow current must each evidence type be?Carry the captured-at timestamp and qualify the answer when a required record exceeds the product’s approved freshness window.
    Conflict ruleWhat happens when trusted inputs disagree?Expose the conflict and its timestamps instead of silently choosing one value.
    ProcedureWhich reasoning process should the workflow execute?Identify the account, retrieve authorized signals, apply metric definitions, compare evidence, state caveats, and propose an action.
    Output contractWhat structure must the response follow?Answer, supporting evidence, caveats, recommended action, and provenance.
    Abstention ruleWhen should the system decline to conclude?Report missing evidence when a required record, metric definition, or permission check is unavailable.
    Audit payloadWhat must be reproducible later?Context-contract version, evidence identifiers, timestamps, policy version, tool results, and model configuration.

    The contract should keep five kinds of context distinct. Task context says what the user is trying to accomplish. Evidence context contains facts relevant to that task. Policy context defines permissions, governance, and prohibited behavior. Interaction context carries the useful parts of the current conversation and approved long-term memory. Execution context defines tools, schemas, retries, and stop conditions.

    Keeping those layers separate prevents a common production mistake: treating all text as equally authoritative. A user’s request should not override a permission rule. A retrieved comment should not outrank an approved policy. An old conversation should not silently redefine a current metric. Your assembly logic needs an explicit precedence order for these collisions.

    Personalization belongs in the contract too. Intent and role should narrow context, not merely add more of it. A finance user may need policy-safe excerpts and transaction evidence. A customer success user may need current account activity and support history. A product manager may need metric definitions, cohorts, experiment state, and caveats. Role-aware assembly and scoped memory make the same underlying capability useful without exposing every available field to every request.

    You know the contract is testable when each field can become a pass-or-fail assertion. Did the workflow apply the current permission scope? Did it include the required metric definition? Did it expose a conflict? Did it abstain when decisive evidence was unavailable? If a requirement cannot be tested or observed, it is still an aspiration rather than an engineering contract.

    Build context assembly as a controlled pipeline

    The production unit is not a prompt template. It is the pipeline that converts a user request into a bounded evidence packet and an executable task. That pipeline should have explicit stages:

    1. Authorize the request. Resolve identity, role, tenant, account scope, and permitted operations before searching for evidence. Apply access controls again before generation as a second check.
    2. Normalize the inputs. Give each record or chunk a stable identifier plus source type, owner, tenant, timestamp, policy classification, schema version, and other metadata needed for filtering.
    3. Generate retrieval candidates. Combine semantic retrieval for conceptually related language with keyword retrieval for exact identifiers, product names, codes, and policy terms.
    4. Filter and rank for the task. Use intent, role, account, freshness, authority, and source-level confidence in addition to semantic similarity.
    5. Resolve stale and conflicting evidence. Apply the contract’s freshness and precedence rules before the model sees the packet. Preserve unresolved conflicts as explicit context.
    6. Pack the context window. Allocate space by priority, remove duplicates, keep decisive passages intact, and exclude material that does not change the task.
    7. Execute through a defined interface. Supply tool schemas, metric definitions, procedure steps, output fields, citation requirements, and abstention conditions.
    8. Attach provenance and emit a trace. Store identifiers and versions needed to reproduce the decision without indiscriminately copying sensitive raw content into logs.

    Hybrid retrieval is useful because semantic and lexical search solve different problems. Semantic search can find a relevant concept expressed in different words. Keyword search protects exact matches such as an account identifier, event name, plan code, or policy term. Metadata then makes the results usable: a highly similar passage from the wrong tenant or an obsolete policy is not a valid result.

    Authorization must shape retrieval itself. Do not search a global corpus, rank everything, and rely on a final prompt instruction to hide unauthorized results. That approach can expose sensitive material to intermediate services, caches, traces, or debugging tools even if it never appears in the final answer. Filter at the retrieval boundary, preserve tenant and role scope through tool calls, and validate the assembled packet before generation.

    Context-window management is also a relevance problem, not just a token-count problem. Reserve capacity in a deliberate order: non-negotiable policy and permissions, the current task, decisive evidence, required procedure and definitions, recent interaction state, then supplemental material. When the packet is too large, compress or drop lower-priority evidence rather than truncating whichever section happens to come last.

    Memory needs its own product rules. Short-term conversation state should retain unresolved references, user corrections, and active task decisions. Long-term memory should be scoped to durable facts that the product is allowed to retain. Define how memory is written, validated, refreshed, read, and deleted. Dumping a full transcript into every turn increases noise and can revive facts or instructions that no longer apply.

    For analytical products, context must include a procedure as well as data. A reliable workflow starts with the decision to be made, anchors it to metric definitions and guardrails, retrieves trusted data, generates testable hypotheses, segments the evidence, and returns options with trade-offs and caveats. That structured analyst loop is far easier to evaluate than a broad instruction to analyze the data.

    The same restraint applies to agents. Use multiple steps or tools when decomposition makes the task clearer, safer, or more verifiable. Each step needs an input schema, permitted tools, completion condition, failure path, and evidence handoff. Agentic patterns are most useful when task decomposition reduces real complexity; extra autonomy without a clearer control boundary simply creates more places for context to drift.

    Ship with layered evaluations, observability, and ownership

    Evaluate the evidence path before scoring the prose

    A single answer-quality score hides the layer that failed. Build an evaluation stack that follows the same stages as the runtime pipeline:

    • Retrieval evaluation: Was the required evidence present in the candidate set, and where did it rank?
    • Assembly evaluation: Did the final packet include required facts and policies, exclude unauthorized or irrelevant material, preserve provenance, and respect freshness rules?
    • Behavior evaluation: Did the model follow the procedure, use the supplied evidence, handle conflicts, cite support, and abstain when required?
    • Answer evaluation: Was the result correct, grounded, complete enough for the task, and structured as promised?
    • Product evaluation: Did the user complete the task, reach an answer faster, correct the output, return to the capability, or escalate to a human?
    • Operational evaluation: Did latency, context size, cost, tool failures, permission denials, and fallback behavior stay within the product’s approved limits?

    Your offline evaluation set should represent the failure surface, not just normal requests. Include different roles and intents, sparse accounts, stale records, contradictory inputs, missing definitions, empty retrieval, tool failures, unauthorized requests, and cases where abstention is the correct result. Label the evidence that should be retrieved as well as the answer that should be produced. Otherwise, a system can pass by reaching the right conclusion through the wrong material.

    Version the evaluation cases, context contract, retrieval configuration, policy set, prompt, tools, and model independently. Change one major layer at a time when possible. If a model upgrade, ranking change, and prompt rewrite ship together, an improved aggregate score will not tell you what worked or which change caused a regression in a sensitive slice.

    After offline acceptance, use staged online experiments with a predeclared outcome, guardrails, acceptance threshold, and minimum detectable effect. Task success, groundedness, time to first answer, adoption, and deflection can all be useful, but only when they match the workflow. A support assistant should not optimize deflection by confidently blocking necessary escalation. An analytical assistant should not optimize speed by dropping caveats required for a sound decision.

    Instrument enough to reproduce failure without creating a new data risk

    For each request, emit a structured event envelope containing the workflow and context-contract versions, detected intent, authorized scope, retrieval-query identifier, evidence identifiers, ranking metadata, freshness state, tool outcomes, policy decisions, answer status, latency, and user feedback. This gives product and engineering a common record for diagnosing failure.

    Do not default to logging every raw prompt, retrieved document, or tool response. Production context can contain customer data, confidential policy, or personal information. Prefer stable identifiers, approved redaction, access-controlled traces, and retention rules. Keep the minimum raw material needed for authorized debugging and evaluation, and make data ownership explicit.

    Roll out in stages: run the new pipeline against offline cases, observe it without user impact where possible, expose it to a constrained cohort, compare it with the existing experience, and expand only after both quality and operational guardrails hold. Preserve a feature flag, a known-safe fallback, and a rollback path for context changes as well as model changes.

    Give every context surface an owner

    Context crosses organizational boundaries, so shared responsibility without named ownership turns into drift. Assign decisions explicitly:

    • Product owns the task boundary, target user, intended decision, outcome metric, failure taxonomy, and acceptance trade-offs.
    • Design owns how evidence, uncertainty, correction, abstention, and human handoff appear in the experience.
    • AI and platform engineering own retrieval, ranking, assembly, tool interfaces, reproducibility, evaluation infrastructure, and fallbacks.
    • Data owners own schemas, metric definitions, lineage, freshness, and the authoritative status of each collection.
    • Security, privacy, and governance owners define permitted use, redaction, retention, and audit requirements.
    • SRE owns service-level monitoring, failure alerts, capacity behavior, deployment safety, and rollback readiness.

    A Staff AI Engineer can connect these concerns by turning research choices into repeatable workflows and shared evaluation infrastructure, but that role should not become the sole owner of product judgment, source governance, or production reliability. Cross-functional execution works when each decision has one accountable owner and the whole group uses the same context trace and evaluation results.

    Treat context changes like code changes. A release should identify the changed source, schema, ranking rule, contract, or policy; show the affected evaluation slices; state the expected product outcome; and preserve a rollback path. CI/CD guardrails, drift monitoring, and human review turn context from an informal prompt dependency into an operable platform capability.

    Key takeaways

    • Diagnose the failed layer before editing the prompt. Missing evidence, bad ranking, stale data, unsafe scope, incomplete procedure, and weak UX are different problems.
    • Define a context contract for each workflow: task boundary, authorized evidence, freshness, precedence, procedure, output, abstention, and audit payload.
    • Authorize before retrieval, rank with task and metadata signals, and validate the assembled packet before generation.
    • Manage the context window by authority and decision value, not by filling every available token.
    • Evaluate retrieval, assembly, model behavior, answer quality, user outcomes, and operational performance separately.
    • Version context components independently, release them through staged controls, and assign an accountable owner to every surface.

    At your next AI product review, do not approve the experience from the final answer alone. Ask to see the evidence packet, permission scope, context-contract version, failed evaluation slices, runtime trace, and rollback path. Those artifacts reveal whether the feature is dependable or merely persuasive.

    Start with one production workflow whose failures matter to users. Trace its most common failure, write the contract, repair the responsible layer, and require the change to pass both offline evaluation and a guarded rollout. Once that loop works, you have the foundation for a reusable context platform rather than another prompt that only works in the demo.

    References

  • 2026 Support Capacity Playbook: Bold AI Automation, Smarter Staffing, Zero‑Surprise SLAs

    Capacity planning has always been a high-stakes exercise in customer service, and when you miss, the signal shows up fast in backlogs and SLAs. I’ve lived that pressure across multiple cycles, and 2026 will reward teams that plan differently. AI fundamentally changes capacity planning because it changes the work. It resolves the bulk of your volume, speeds up execution, and elevates the complexity and value of what humans handle. The consequence is simple: planning models must evolve. This is the final installment in my 2026 customer service planning series, and I’m focusing on the tension every leader feels right now—be ambitious about automation, but avoid the trap of understaffing if your assumptions don’t hold. My goal is to share how AI changes the logic of capacity planning, what I’ve learned implementing these practices with my team and with customers, and the common traps to avoid. Traditional planning rests on relatively stable assumptions: volume grows predictably, work types stay consistent, handle times don’t swing dramatically, and productivity improves slowly with better tools and training. In an AI-first model, none of that is guaranteed, and the fundamentals flip. The mix of work changes as AI absorbs a growing share of simpler conversations, leaving humans with deeper, more time-consuming issues that demand human-to-human connection. Demand can actually increase when you remove friction, so AI can both resolve more and attract more volume. Human time splits differently as teammates solve customer problems and also review AI behavior, give feedback, improve content, and support system-level work. Performance becomes dynamic, not fixed—automation rate isn’t a one-time number; it can rise with care and fall with neglect. If you plan for 2026 using a pre-AI model—assuming similar productivity, similar work mix, and a linear relationship between volume and headcount—you will underestimate what it now takes to run a high-performing support organization. There are many metrics you can track, but the one to put at the center is automation rate (AI Agent involvement rate × AI Agent resolution rate). This single construct tells me what share of total volume AI actually resolves, how much work remains for humans, how much additional demand humans can absorb, and how ambitious I can be with headcount. Early in the journey, I prioritize raising involvement—getting the AI involved in more conversations. Once involvement is high, I shift to resolution on the hardest remaining work, where each additional 1% of automation can represent several people’s worth of capacity. In my 2026 plans, automation rate sits alongside projected inbound volume, average “output” per person for the more complex work that remains, and occupancy—how much time is allocated to customer-facing interactions versus operational and strategic work. Together, those inputs give a realistic picture of how many people you need and where they should spend their time. First, plan boldly on automation, but match it with investment. I do not cap automation assumptions at 40–50% “because AI is new.” Many teams are already modeling 60%, 70%, even 80%+ for 2026—when they invest in AI ownership and content. The investment is non-negotiable: named ownership for AI performance (AI ops, knowledge management, conversation design), clear automation targets by work type (e.g., informational vs. personalized vs. actions vs. deep troubleshooting), realistic expectations for what’s easy to automate and what’s not, and a concrete plan to raise automation over time in monthly or quarterly steps rather than a single jump. To decide where to invest first, I dig into the data. I start with the biggest volume drivers, separate content-led issues from those dependent on data or complex procedures, assume higher resolution potential for content-led topics once the knowledge base is in shape, and set more modest initial resolution expectations for system-dependent flows. Then I stair-step improvements as the systems, data contracts, and workflows mature. In short, bold automation goals only work when paired with the team structure, content, and systems required to reach them—and the discipline to iterate. Second, expect human “output” per person to go down. That’s a mindset shift. Historically, we assumed individual productivity would stay flat or tick up as tools improved. In an AI-first model, humans handle fewer conversations but more complex, cross-functional issues—and create more value despite lower case counts. I model a lower “cases closed per person” than prior-year baselines, explicitly assume the remaining work is more complex and time-consuming, and redefine productivity to include system-level work like AI Agent improvements, content updates, and policy or workflow change management. I also report “capacity created” from automation alongside human outputs, so leadership sees the full picture. Third, rethink occupancy: more time off the queues, on higher-value work. Traditional occupancy splits time between inbox and training, meetings, and breaks. Now there’s an expanding “out-of-inbox” portfolio that directly affects AI performance and overall capacity: reviewing AI-handled conversations, improving AI Agent triaging and handovers, contributing to content and procedures, feeding insights to product and engineering, and supporting system changes that reduce future volume. I set lower inbox occupancy targets than before and make the rationale explicit. People aren’t working less—they’re working differently. In planning, I assume more time spent on improvement and system work, make it visible (for example, X% in inbox and Y% on AI and system improvement), and treat this as critical, not a “nice to have.” If you don’t proactively allocate it, it won’t happen—and your automation and performance targets will suffer. Fourth, work with the finance team early, and treat your plan as a set of assumptions. Capacity planning with AI is a set of bets across automation rate, human output, demand growth, occupancy, and where surplus capacity (if any) goes. I bring finance in early, show that the plan is dynamic and directly tied to AI performance, and label every lever as an assumption with ranges. I commit to a quarterly review cadence with finance to compare assumptions versus reality and adjust headcount, targets, and investment as needed. The risks are real: if automation grows slower than expected and you stop backfilling too early, you’ll be understaffed for months. Hiring and onboarding take time, so course-correcting late creates strain. If you do produce surplus capacity, have a clear strategy to reallocate those teammates to higher-value work—improving systems, feeding insights back to product, supporting new channels, and driving proactive CX—rather than defaulting to reductions. I also set explicit guardrails—if automation rate misses by five points for two consecutive months, we pause planned reductions and revisit hiring gates. If it over-performs, we shift people into backlog eradication, content upgrades, or proactive outreach, so we bank compounding value. To set your team up for success in 2026, anchor your plan on automation rate, be honest that humans will handle fewer but harder conversations, and protect time for system improvements. Partner early and often with finance, avoid shrinking too fast, and design a plan for surplus capacity so you’re never caught flat-footed. If AI is going to handle the majority of your customer conversations, your plan has to be designed to help it do that well and to keep your team set up for meaningful, sustainable work. A 2026 plan built on adaptable assumptions—not fixed predictions—will hold up as your work, your systems, and your customers’ expectations continue to change. If you’d like future editions like this, subscribe and stay close—I’ll keep sharing what’s working, what isn’t, and how to tune your customer support AI strategy in real time.

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • AI in Product Design: My Proven Playbook, Real Use Cases, and the Tools That Win Faster

    AI in Product Design: My Proven Playbook, Real Use Cases, and the Tools That Win Faster

    In product design, AI has shifted from novelty to non-negotiable. I’ve watched teams accelerate discovery, compress prototyping cycles, and turn ambiguous ideas into validated experiences faster than ever—without sacrificing quality or customer trust.

    AI in product design has quickly moved from new to necessary. Here are the AI product design tools and approaches you need to stay relevant in this decade.

    From my vantage point leading product teams, “necessary” means AI is woven throughout the product lifecycle—discovery, prioritization, prototyping, validation, and iteration—not bolted on. The goal isn’t to chase hype; it’s to build durable advantage with clear AI Strategy, disciplined execution, and measurable outcomes.

    First, anchor the work in strategy. Tie every AI initiative to a specific customer problem and value proposition, then express that linkage with outcomes vs output OKRs. This keeps teams focused on real impact and avoids feature-chasing. It also sharpens product positioning and clarifies where AI can deliver competitive differentiation versus simple points of parity.

    Second, upgrade discovery. I rely on AI workflows to synthesize interviews, cluster themes, and surface insights at scale. A retrieval-first pipeline—grounding models in our own data—improves factuality and reduces hallucinations. Combine this with strong data governance and privacy-by-design so insights are trustworthy and compliant from day one.

    Third, make quality measurable. Adopt eval-driven development: define evaluation sets and acceptance thresholds that reflect real user tasks before you ship. Pair that with A/B testing and minimum detectable effect (MDE) discipline, so you learn quickly and confidently. Add safety guardrails (red-teaming prompts, content filters, and bias checks) to manage AI risk without slowing the pace.

    Fourth, enable empowered product teams. Product trios (PM, design, engineering) should co-create prompts, prototypes, and evaluation criteria. Give designers and PMs practical tools—LLMs for product managers, structured prompt templates, and reusable components—so AI-augmented work becomes the default, not a special project.

    Where does AI shine in product design today? Concept exploration and market scans, turning fuzzy opportunity spaces into crisp problem statements. Rapid wireframes and interaction ideas, using gen ai for product prototyping to explore multiple design directions in minutes. UX writing that adapts tone and reduces friction across onboarding, tooltip design, and microcopy.

    It also excels at guided experiences. I’ve seen strong lifts in user activation when we pair in-app guides and product tours with context-aware suggestions. For support and education use cases, a retrieval-grounded assistant can deflect tickets, shorten time-to-value, and reinforce the product’s value proposition at the exact moment a user needs help.

    Voice is another frontier. A well-scoped voice AI agent can accelerate complex workflows (think data entry or multi-step configurations) when hands-free is faster or more intuitive. Just be intentional about when agentic AI adds net value versus when a simple UI tweak would do.

    On the tooling side, my AI product toolbox is pragmatic and modular. For analytics and learning loops, Amplitude analytics and Pendo help quantify behavior changes and retention analysis. For in-product engagement and feedback routing, Intercom and HubSpot integrate cleanly with LLM-driven tagging and summarization. For ideation and automation, I use a ChatGPT connector and Claude Code for quick scripts, data wrangling, and prompt experiments. The constant: a retrieval-first pipeline that grounds models in approved knowledge and maintains context window management at scale.

    Risk management is built in, not bolted on. Set clear AI risk management policies, catalog model and data dependencies, and document decisions. Align with regulatory compliance requirements early, and keep an audit trail of prompts, datasets, and eval results. That’s how you move fast without breaking trust.

    If you’re getting started, begin small: pick one high-friction workflow, add a retrieval-grounded copilot, and measure the lift. Use the results to inform product roadmapping and sprint planning, then scale to adjacent use cases. With disciplined discovery, sharp evaluation, and the right tooling, AI becomes a force multiplier for product teams and a clear win for customers.

    Inspired by this post on Product School.

    Book a consult png image
  • From Concierge to AI Marketing Engine: Inside Mowie’s Document Hierarchy Playbook

    From Concierge to AI Marketing Engine: Inside Mowie’s Document Hierarchy Playbook

    I’m constantly asked by SMB owners: What if your small business could have a full marketing team—automated content calendars, customer segmentation, and channel-specific posts—without the headcount? That question is no longer hypothetical; it’s precisely the promise behind Mowie, and the way they got there is a masterclass in practical AI product development.

    I recently listened to Chris O'Connor (CEO) and Jessica Valenzuela (Co-Founder) of Mowie, an AI marketing platform built for small and medium-sized businesses in restaurants, retail, and e-commerce. Their story starts with a concierge marketing service—doing the work by hand for overwhelmed owners—and evolves into a fully automated AI product.

    They walk through their "document hierarchy" approach: how Mowie crawls the web to build a "dossier" about each business, infers customer segments and marketing pillars, and generates quarterly content calendars with channel-specific posts. As a product leader, this is the kind of retrieval-first pipeline that consistently outperforms naive prompt chaining because it builds durable context before generation.

    They also unpack the technical challenges of structuring unstructured data and the evolution from rigid schemas to loosely structured markdown. In my experience with LLMs for product managers, markdown becomes a flexible intermediate representation that’s easy to diff, trace, and feed back into models without brittle parsing.

    Equally important, they use customer feedback—from calendar approvals to regeneration requests—as their primary evaluation signal. That’s eval-driven development in practice: close the loop with lightweight evals that reflect genuine user intent, not proxy metrics.

    The planning model is elegant: the three mini-calendars—public events, business-specific events, and recommended campaigns—roll up into a coherent plan that eliminates the blank-page problem and enables steady, predictable execution.

    Crucially, they’re building traceability so customers can see which context documents influenced their content. This kind of transparency increases trust, accelerates edits, and supports governance in regulated categories where auditability matters.

    Onboarding and data collection stay pragmatic: let the system crawl first, ask humans only for deltas, and progressively profile over time. It’s a pattern I advocate in continuous discovery and AI workflows—keep humans in the loop without overwhelming them, and make the right action the easy action.

    Early on, they used Simon Sinek's Golden Circle framework to validate demand and sharpen messaging. Framing the "why" before the "what" helps teams maintain a crisp value proposition and tighten their go-to-market strategy.

    Performance measurement goes beyond vanity metrics by connecting marketing performance back to point-of-sale data for attribution. The ability to tie campaigns to revenue events is the bridge from clever content to accountable outcomes.

    What’s next is equally compelling: deeper attribution, omnichannel expansion, and digital out-of-home displays. For SMBs, that points to a unified analytics platform spanning email, social, and in-store touchpoints—exactly where modern marketing is headed.

    My takeaways for builders: invest in a retrieval-first pipeline with a resilient document hierarchy; prefer loosely structured markdown over rigid JSON when dealing with messy inputs; design human-in-the-loop controls that double as evals; and always connect activity to business outcomes. That’s how you turn an idea into a repeatable system that scales.

    If you want to explore further, start here: Mowie AI — AI marketing platform for SMBs. For early validation and storytelling, revisit Simon Sinek's Golden Circle.

    Inspired by this post on Product Talk.

    Book a consult png image
  • Automated Insights for Product Teams: Uncover Causal ‘Aha’ Moments in Minutes, Not Weeks

    Automated Insights for Product Teams: Uncover Causal ‘Aha’ Moments in Minutes, Not Weeks

    I’ve spent countless cycles guiding teams through the maze of dashboards, SQL pulls, and ad‑hoc analyses—only to watch truly meaningful patterns emerge far too late. Automated insights are the next frontier in product analytics: a shift from manual exploration to AI that proactively surfaces what matters most. When we let the system do the heavy lifting, we accelerate discovery, reduce bias, and give product trios the clarity to act.

    Finding causal connections in product data involves exhaustive searches and tests. We trained our AI to find “aha” moments in minutes instead of weeks.

    Here’s what that means in practice for product management: the platform continuously scans events, cohorts, and segments; prioritizes signals linked to activation, conversion, and retention; and highlights likely causes behind meaningful movements in your core KPIs. Instead of sifting through endless funnels and cohorts, I get ranked hypotheses I can validate with targeted A/B testing and minimum detectable effect (MDE) guardrails.

    This approach turns analytics into action. Automated insights reduce time-to-learning, tighten our discovery loops, and make continuous discovery tangible—especially when we’re aligning roadmaps, designing experiments, and refining onboarding. Whether you’re using tools like Amplitude analytics or instrumenting a unified analytics platform, the value is the same: faster, clearer paths to customer impact.

    I’ve seen teams unlock retention analysis breakthroughs by spotting counterintuitive patterns—like a specific feature combination or an overlooked step in onboarding—well before they would have surfaced through manual analysis. With AI workflows scanning the noise and elevating the signal, we can focus on decisions: ship or iterate, scale or sunset, double down or pivot. That’s empowered product teams in action.

    If you’re building for product-led growth, this is the leverage you’ve been waiting for. Automated insights transform how we prioritize, test, and communicate strategy—bringing us from gut feel and lagging indicators to explainable, causal narratives we can stand behind. The outcome is simple: more confident bets, less waste, and a faster path to durable product-market fit.

    Inspired by this post on Amplitude – Best Practices.

    Book a consult png image
  • Operationalizing AI: A Practical System for Scalable Growth

    Operationalizing AI: A Practical System for Scalable Growth

    Your AI pilot works in the demo. Then it reaches a live workflow and slows down: the data is incomplete, nobody owns the exceptions, reviewers apply different standards, and the team cannot prove whether the result improved revenue, cost, speed, or retention.

    The gap is not model quality alone. Scalable growth requires an operating system around the model: a constrained business outcome, a mapped workflow, approved data, explicit decision rights, measurable quality, controlled releases, and a path for handling failure. Build those pieces around one valuable use case, and AI can become a repeatable business capability instead of a collection of pilots.

    Choose the growth constraint before the AI use case

    Do not begin with a broad instruction to “find an AI use case.” That framing encourages teams to start with a model capability and search for somewhere to place it. Start with a constrained business problem instead.

    The unit of investment should be a decision or task inside a customer or employee journey. “Build a churn copilot” is too broad. “Before a renewal review, summarize approved usage and CRM signals, identify the evidence of risk, and propose an action for the customer success manager to review” is narrow enough to test.

    Most growth-oriented opportunities fit into four useful lanes:

    • Revenue: improve qualification, conversion, expansion, cross-sell, or win-back decisions. Measure the commercial event, not the number of AI recommendations generated.
    • Efficiency: reduce the cost, handling time, rework, or backlog associated with a repetitive process. Good candidates have high task volume and outputs that can be checked without recreating the work.
    • Speed: shorten a discovery, delivery, or release cycle. If the workflow serves software delivery, deployment frequency can be relevant, but it is not evidence of customer or commercial value by itself.
    • Activation and retention: make onboarding, guidance, or support more contextual. Measure whether customers reach the intended product behavior and continue receiving value, not whether they clicked an AI-generated tooltip.

    A disciplined portfolio can pair one revenue use case with one efficiency use case, define success before development, and release each through a narrow MVP. That balance matters. An efficiency-only roadmap can shrink costs without creating differentiation, while an unconstrained revenue bet can consume attention without proving economic value.

    Screen each candidate with the same questions:

    • What business metric should move, and what is its current baseline?
    • Which person, decision, and moment in the workflow create that movement?
    • Does the task occur often enough to justify a reusable solution?
    • Are the required inputs available, current, and approved for this purpose?
    • Can a reviewer distinguish an acceptable result from an unacceptable one?
    • What happens when the system is wrong, and can the action be reversed?
    • Who owns the outcome after the launch team moves on?

    My test is blunt: if you cannot name the workflow event, the owner, the baseline, and the failure consequence, you do not yet have an implementation candidate. You have a discovery question. Fund the learning needed to answer it before funding scale.

    Convert the use case into a controlled workflow

    An AI feature becomes operational when its behavior is defined inside the surrounding work. That means understanding what happens before the model is called, what the model may do, how its output is checked, and what happens next.

    Begin by mapping the task as it is performed, choosing one step to augment, selecting the right automation method, and iterating against an explicit quality bar. Do the task manually while mapping it if the real process is unclear. Policy documents often describe the intended path; observation reveals the exceptions that determine whether automation will survive production.

    1. Name the trigger. Specify the event that starts the workflow, such as a support request, renewal review, onboarding milestone, invoice submission, or product release.
    2. Identify the inputs. Record each system, document, field, permission, and freshness requirement. Separate required evidence from optional context.
    3. Expose the decisions. Write down the classifications, judgments, calculations, and approvals a person currently makes. Hidden judgment is where apparently simple automations tend to break.
    4. Specify the output. Define its schema, audience, channel, timing, and acceptable evidence. “Produce a helpful answer” is not a specification.
    5. Map exceptions. Include missing records, contradictory inputs, unsupported requests, low-confidence cases, policy conflicts, and unavailable downstream systems.
    6. Assign each step to code, retrieval, an LLM, or a person. The workflow should use the simplest reliable mechanism for each job.
    7. Define the handoff. State who reviews the result, what they can change, when the workflow must stop, and where failures are recorded.

    Use each form of automation for the work it can control

    Use deterministic code for exact calculations, validation rules, permissions, routing, and other behavior that should produce the same answer from the same inputs. Use an LLM where language is ambiguous, inputs are unstructured, or the task requires drafting, summarizing, extracting, or classifying meaning.

    When the answer must reflect company facts, policy, or customer history, retrieve the approved information at runtime instead of expecting the model to remember it. A retrieval-first design can connect behavioral and CRM context to account signals and recommended actions, while preserving a visible trail back to the evidence used.

    Keep a person in the path when the consequence is material, the action is difficult to reverse, or the definition of a correct result remains contested. Human review is not a permanent excuse for weak quality, however. The reviewer needs defined criteria, enough context to make a decision, and an easy way to correct and categorize the failure.

    Write an execution contract, not just a prompt

    A production instruction set should define more than tone and role. Treat it as an execution contract containing:

    • the objective and the business context;
    • the permitted inputs and authoritative evidence;
    • the decision criteria the system must apply;
    • the required output structure;
    • the actions it may and may not take;
    • the conditions that require refusal or escalation;
    • the way uncertainty should be represented;
    • examples of acceptable, unacceptable, and edge-case behavior.

    For an agentic workflow, increase authority in deliberate stages: observe, draft, recommend, act after approval, and only then act within defined limits. Do not jump from a convincing chat demonstration to autonomous execution. Agentic AI needs explicit guardrails and verifiable quality before it can safely take work out of a human queue.

    Measure business value, workflow performance, and AI quality separately

    A dashboard that reports requests, tokens, or generated answers tells you that the feature was used. It does not tell you whether the business improved. You need separate measures because an AI system can look healthy at one layer while failing at another.

    Measurement layerWhat to trackWhat it reveals
    Business outcomeConversion, expansion, cost per completed outcome, cycle time, activation, or retentionWhether the investment affects the growth constraint it was chosen to address
    Workflow performanceCompletion, rework, exception, escalation, abandonment, and end-to-end latencyWhether the surrounding process can absorb and use the AI output
    AI qualityCorrectness, evidence support, instruction adherence, output validity, and appropriate refusalWhether the system behaves acceptably across expected and difficult cases
    Risk and operationsUnauthorized data exposure, prohibited actions, overrides, incidents, rollback events, and unresolved failuresWhether growth is being purchased with unacceptable operational or trust costs

    Build the measurement path before the rollout:

    1. Capture the baseline. Measure the existing workflow using the same outcome definition you will use after launch. Otherwise, a faster AI step can hide slower review, higher rework, or shifted labor elsewhere.
    2. Create a representative evaluation set. Use permitted examples from normal, difficult, and failure-prone cases. Define the expected result and the critical errors for each case.
    3. Weight failures by consequence. Formatting errors, unsupported factual claims, privacy failures, and unauthorized actions should not disappear into one average score.
    4. Run offline evaluations before exposure. Test the complete combination of instructions, model, retrieval, tools, and output validation. A model score alone does not represent the production system.
    5. Release behind a feature flag. Start with a controlled cohort, preserve the ability to roll back, and compare outcomes. Use A/B testing when assignment and outcome measurement are credible; use a phased rollout when they are not.
    6. Record versions. Log the model, instructions, retrieval configuration, tools, and policy version associated with each result so a regression can be traced.
    7. Turn failures into future tests. Categorize meaningful production failures and add them to the evaluation set before the next release.

    This is the practical meaning of eval-driven development: instrument the system, watch for drift, and tighten the delivery loop while changes remain controlled by feature flags. It turns evaluation from a launch checkpoint into part of product development.

    Use a scale gate that includes economics

    Do not scale because the demo is impressive or employees like the interface. Require four decisions:

    • The business outcome is moving in the intended direction, or there is credible evidence that the workflow is producing the leading behavior tied to it.
    • Quality remains acceptable across normal cases, edge cases, and high-consequence failures.
    • Total cost per successful outcome is viable after model usage, retrieval, storage, human review, escalation, rework, and operations are included.
    • The operating owner can detect, contain, and learn from failures without depending on the original project team.

    If a pilot fails one of these gates, the decision is not automatically to cancel it. Narrow the scope, change the workflow, improve the evidence, or stop. What matters is that expansion is earned by measured behavior rather than assumed from adoption.

    Scale through guardrails, reusable components, and clear ownership

    Governance should make routine decisions faster. When every team has to rediscover which data is permitted, which evaluation is sufficient, and who can approve a release, governance becomes a sequence of meetings. When those expectations are encoded in a standard launch record, teams know the path before they build.

    Create a minimum launch record for every workflow

    • the business outcome, baseline, and accountable owner;
    • the workflow boundary, users, and authorized actions;
    • the approved data sources, access controls, retention rules, and prohibited data;
    • the evaluation set, acceptance criteria, and critical failure classes;
    • the human review and escalation conditions;
    • the logging, monitoring, feature flag, and rollback plan;
    • the model, retrieval, tool, and vendor dependencies;
    • the incident owner and the method for notifying affected internal teams or customers when appropriate.

    Privacy-by-design, data governance, red-teaming, and defined review gates are growth infrastructure. They reduce repeated risk debates and make the safe path reusable across launches.

    If a workflow touches personal data, confidential customer content, employment decisions, payments, security actions, or contractual commitments, involve the appropriate privacy, security, legal, financial, or people owner before live use. The downside is not limited to a poor answer. The workflow can expose restricted data or take an action the business cannot easily reverse.

    Assign ownership beyond launch

    Four responsibilities must be explicit, even when one person holds more than one:

    • Business outcome ownership: decides whether the workflow is worth continuing based on the target metric and economics.
    • Workflow ownership: manages exceptions, reviewer behavior, process changes, and user feedback.
    • Technical ownership: controls releases, versions, integrations, reliability, monitoring, and rollback.
    • Risk ownership: defines the policy boundary and approves material changes to data, authority, or exposure.

    This prevents a common operating failure: the product team treats launch as completion, while the operations team inherits a changing probabilistic system without the tools or authority to manage it.

    Standardize the recurring parts, not every local process

    Once working use cases expose recurring needs, turn those needs into shared capabilities. Useful candidates include identity and permissions, governed retrieval connectors, evaluation tooling, instruction and model versioning, observability, feature flags, rollback controls, and cost attribution.

    Keep the final workflow close to the business team that understands the customer, exceptions, and outcome. Centralize the controls and infrastructure that should be consistent. This creates leverage without forcing every function into the same process.

    Review the portfolio as a set of products, not permanent projects. The decision for each workflow should be to expand it, fix a known constraint, narrow its authority, or retire it. Continuous discovery with product trios can refine the prompts, data sources, and experience while evidence determines what scales and what stops.

    Operationalizing AI: three questions leaders ask

    Should you build a central AI platform first?

    Usually, no. Start with the minimum secure infrastructure required for a valuable workflow. Standardize a component when several use cases need the same capability or when inconsistency creates material risk. Data access, identity, logging, and release controls may need early consistency; a broad internal platform without proven workflows can become an expensive set of assumptions.

    How do you know a pilot is ready to scale?

    A pilot is ready when it improves the intended business or workflow outcome, stays within quality and risk boundaries, has viable cost per successful outcome, and can be operated without daily intervention from its builders. Usage and positive comments are supporting signals, not a scale decision.

    Where should a human remain in the loop?

    Keep human approval where consequences are high, actions are difficult to reverse, evidence is incomplete, or acceptable judgment cannot yet be specified. Remove or reduce review only when evaluations and production monitoring show that the remaining risk is understood and controlled. A reviewer who merely clicks approve without adding judgment is not a guardrail; it is latency disguised as governance.

    For your next AI proposal, require a one-page charter containing the outcome, workflow boundary, owner, baseline, approved data, evaluation set, failure policy, release plan, and full cost model. If a line is blank, fund discovery to resolve it. If the charter is complete, release the smallest useful workflow behind a control, learn from real failures, and widen its authority only when the evidence earns it.

    References

  • How to Build a Self-Improving AI Support Operation

    How to Build a Self-Improving AI Support Operation

    Your AI support agent handled the easy questions, produced an encouraging early lift, and then stopped getting better. The same topics still reach human agents. Content fixes happen when someone remembers. The aggregate resolution rate moves, but nobody can explain why.

    If that describes your operating review, a newer model is unlikely to be the first thing you need. You need a closed operating loop: every weak conversation becomes evidence, every useful insight gets an owner, and every change is tested against the next conversation it is meant to improve.

    Measure the improvement loop, not just resolution rate

    A self-improving support operation is not an agent that quietly rewrites or retrains itself. It is a managed system in which live conversations expose failure modes, people convert those failures into controlled changes, and later conversations show whether the changes worked.

    Resolution rate is an outcome of that system, not a diagnosis. An aggregate rate cannot tell you which intent deteriorated, why the agent handed a customer to a human, or whether a change repaired one topic while damaging another. It can also be misleading when eligibility changes. Expanding automation into harder intents may lower the rate while increasing the number of conversations resolved. Excluding difficult intents can produce the opposite effect.

    Start by documenting exactly what your denominator includes and what counts as a resolution. Keep that definition stable enough to compare periods, and report resolved volume alongside the rate. Then add the views that turn a dashboard into a work queue:

    • Coverage: Which inbound conversations are eligible for AI handling, and which are excluded?
    • Outcome by intent: Where does the agent resolve, hand off, or fail to answer?
    • Failure reason: Was the problem missing knowledge, weak retrieval, incorrect behavior, poor routing, or an issue the product itself must solve?
    • Quality: Did an audit, repeated contact, reopened conversation, or another trusted signal indicate that the apparent resolution was weak?
    • Change throughput: How many identified failures are waiting for diagnosis, testing, approval, or release?

    The intent-level view matters because it gives the owner somewhere to act. A falling aggregate rate is merely a warning. A cluster of unresolved questions about one feature, tied to one failure reason, is a tractable product and operations problem.

    Classify the failure before choosing the fix

    Teams waste cycles when every poor answer is treated as a documentation problem. Use a small failure taxonomy to route each issue to the layer that can actually repair it.

    Failure classWhat you observeLikely action
    Knowledge gapNo current, approved answer existsCreate or repair the canonical content
    Retrieval gapThe answer exists, but the agent does not receive or select itImprove structure, segmentation, metadata, or retrieval configuration
    Behavior gapThe right information is available, but the response is incomplete or misappliedAdjust instructions, examples, or agent configuration
    Routing gapThe agent should escalate but does not, or the handoff loses essential contextChange escalation conditions and the handoff payload
    Product gapNo support answer can resolve the underlying problemSend the evidence to product or engineering instead of disguising it as a content task

    This distinction prevents two common errors: endlessly rewriting accurate content when retrieval is broken, and asking the support agent to explain around a product defect that requires an actual fix.

    Give one owner the authority and the improvement queue

    Shared participation is useful. Shared accountability is not. One person should own the performance of the AI support operation, even though support, product, content, engineering, and security may contribute to individual changes.

    The title can be AI operations lead, support operations specialist, or something else. The mandate is what matters: identify underperforming intents, maintain the improvement backlog, coordinate changes across functions, enforce the evaluation process, and report what improved or regressed.

    Ownership becomes especially important after the launch surge fades. At Dotdigital, performance held at about 2,800 resolved conversations per month for three consecutive months. The response was to create a dedicated support operations specialist role focused on snippets, content, and the agent’s resolution capability. The lesson is not that every company needs the same job title. It is that a plateau without an empowered owner tends to remain a plateau.

    Do not bury improvement work in the general support queue. A customer ticket can close while the underlying failure remains. Create a separate, persistent record for the system-level issue, with fields that make it possible to trace evidence through to an outcome:

    • Representative conversation links and the affected intent
    • The observed failure and its customer consequence
    • The failure class and the evidence supporting that diagnosis
    • The knowledge, retrieval, behavior, routing, or product artifact to change
    • The accountable owner and required reviewer
    • The evaluation cases that must pass
    • The release status, version, and deployment date
    • The live signal that will be checked after release

    Define done as more than content published or configuration changed. An improvement is complete only when the change is linked to its originating evidence, reviewed at the appropriate risk level, tested, released, and checked in live operation.

    For prioritization, assess recurrence, consequence, confidence in the diagnosis, and effort separately. Do not let raw volume make the decision by itself. A rare failure involving access, privacy, or an irreversible customer action can deserve attention before a frequent wording problem. Conversely, a recurring low-risk knowledge gap may be the best candidate for a fast content repair.

    Turn live failures into governed, testable changes

    Feedback does not improve an agent merely because it was collected. A thumbs-down, a handoff, or an unresolved conversation is a signal, not a root cause. The operating loop has to convert that signal into a specific hypothesis and then close the loop.

    1. Collect: Group common handoffs and unresolved conversations by intent instead of reading them as isolated tickets.
    2. Diagnose: Assign a failure class and confirm that the proposed layer is actually responsible.
    3. Prioritize: Select the issue using recurrence, consequence, confidence, and effort.
    4. Change: Modify the smallest responsible artifact rather than making broad agent changes by default.
    5. Evaluate: Test the originating failures, realistic variations, and already-passing cases that could regress.
    6. Release and observe: Record what shipped, monitor the affected live intent, and feed any new failure back into the queue.

    Write the hypothesis before making the change: for this intent, changing this artifact should reduce this failure reason without degrading these existing behaviors. That sentence forces clarity about what success means and which regression cases belong in the evaluation set.

    When a live failure reveals a missing case, promote it into the regression set after the fix. Over time, the evaluation suite becomes a practical memory of mistakes the operation should not repeat. That is where compounding comes from: the team is not merely correcting answers; it is preserving each correction as a reusable control.

    Match governance to the blast radius

    Fast iteration and responsible review are compatible when the rules are explicit. A useful governance model distinguishes changes by consequence:

    • Low blast radius: A correction to an approved fact, an obsolete product step, or a missing limitation can follow a lightweight peer review and the relevant evaluation cases.
    • Moderate blast radius: Retrieval, behavior, and routing changes that can affect several intents should receive cross-functional review and a controlled release.
    • High blast radius: Actions involving permissions, account access, customer data, money, or security need stronger approval, a safe test environment, a rollback path, and an obvious route to a human.

    A wrong explanation can create confusion. A wrong action can change an account or expose data. Treating those changes as equivalent either slows harmless content repairs or makes consequential automation unsafe.

    Use focused sprints without making improvement episodic

    A concentrated sprint is useful when the backlog has accumulated or a set of topics is visibly underperforming. In one focused Anthropic effort, the team audited unresolved queries, repaired weak content, converted recurring macros into AI-usable snippets, and monitored live performance. That is a practical pattern for clearing known gaps quickly.

    The sprint should strengthen the standing loop, not replace it. Keep the same taxonomy, backlog, review rules, and evaluation artifacts after the concentrated work ends. Otherwise, the operation improves during special events and drifts between them.

    Make the improvement work visible in each operating review. Show the failure observed, the artifact changed, the evaluation result, and the live outcome or next check. Name the person who drove the repair. This rewards the behavior that creates durable gains instead of celebrating only a headline rate that few people can explain.

    Make AI-ready knowledge part of product launch readiness

    Company-specific support knowledge does not appear because the underlying model is capable. The agent needs current, approved information in a form it can retrieve and apply. Missing or contradictory knowledge is an operating failure, not a model mystery.

    Treat knowledge as production infrastructure. Every topic needs an owner. Important changes need versions and effective dates. Retired instructions need to be removed or clearly superseded. The agent’s ingestion and retrieval path needs verification, just as the customer-facing help experience does.

    A canonical source of truth does not have to be one enormous help article. It means there is one approved origin for the product facts from which help-center content, agent snippets, human macros, and other downstream formats are derived. When those formats are authored independently, contradictions are almost inevitable.

    Add an AI support gate to the new product introduction process. Before a feature is considered ready, confirm that:

    • A named owner is accountable for keeping the feature’s knowledge current.
    • The canonical material explains what changed, who can use it, how it works, and where its boundaries are.
    • Known limitations and escalation conditions are explicit rather than left for the agent to infer.
    • The effective version or release state is clear, so old and new instructions cannot be confused.
    • The content has been ingested or indexed and retrieval has been tested.
    • Expected support intents and representative evaluation cases are ready before inbound volume arrives.
    • Support has a defined path for returning launch-day failures to product, engineering, or the knowledge owner.

    This is not only administrative hygiene. In my organization, embedding a canonical source of truth into launch readiness has consistently supported resolution rates above 50% for new features from day one. That result is evidence for the operating model, not a universal benchmark; intent mix, product complexity, and the definition of resolution still matter.

    Do not automatically turn every human answer into permanent knowledge. First decide whether the resolution is generalizable. If it is, update the canonical material. If it is a legitimate exception, encode the escalation path. If the underlying issue is a product defect, preserve the conversation as product evidence and route it accordingly. The objective is a cleaner system, not simply more content.

    Key takeaways for your next operating review

    • Define self-improvement as a managed loop from conversation evidence to a verified change, not autonomous model learning.
    • Keep resolution rate, resolved volume, coverage, failure reasons, and change throughput visible together.
    • Assign one accountable owner with authority to coordinate support, content, product, and engineering.
    • Classify each failure before fixing it so knowledge, retrieval, behavior, routing, and product problems reach the right layer.
    • Turn repaired failures into regression cases, and apply stronger review as the blast radius increases.
    • Make canonical, AI-ready knowledge a launch requirement instead of a cleanup task for support.

    At your next review, take one recurring unresolved intent and trace it all the way through: evidence, diagnosis, owner, change, evaluation, release, and live result. If any link is missing, that is the first operating gap to repair. Once the path works for one intent, make it the default path for every failure worth learning from.

    References