I’ve been pushing hard to operationalize AI for real product work, and this episode zeroes in on the moment Claude Code stops feeling like a demo and starts behaving like a dependable teammate. If you’ve ever wondered how to go from clever prompts in the browser to durable, repeatable workflows on your machine, this walkthrough is for you.
Listen on: Spotify | Apple Podcasts.
My first honest reaction to installing and configuring the desktop agent was the all-too-relatable “this tool thinks everything is a code repo” reality. That framing helped me reset expectations fast: instead of treating it like a magical universal assistant, I began designing guardrails, context, and repeatable routines—exactly how I’d onboard a new team member.
The shift from Claude-in-the-browser to Claude Code on my machine was the unlock. Locally, it can finally work with my files, folders, and workflows. That meant I could ground it in real artifacts—project docs, meeting notes, product specs, and historical decisions—so responses weren’t just plausible; they were contextual and verifiable.
On setup, I now treat /init and Claude MD files as my product requirements. I define roles, boundaries, and canonical sources up front, then run in a deliberate “walled garden.” The “treat it like an intern” model works beautifully: scope access intentionally, expand privileges as trust grows, and keep a tight audit trail of what it can touch and why.
Surprisingly, task management became my ideal on-ramp. It’s easy to validate, the feedback loops are tight, and the ROI is immediate. I export calendar windows rather than granting full calendar access, then let the agent map priorities into Trello, reconcile time blocks, and surface trade-offs. Fast wins build confidence—mine and the agent’s.
Model switching matters more than I expected. When speed is king and “good enough” will do, Haiku keeps the loop snappy. When stakes are higher—complex synthesis, nuanced product strategy, or gnarly ambiguity—I step up to Claude Opus 4.5. Being intentional about when to optimize for latency versus depth is a quiet superpower.
Web tasks can still spiral. When that happens, I pause its autonomy, toggle to fewer steps, and ask, “What are you doing?” Paired with Claude’s Web fetch tool, this makes the agent explain its chain-of-thought planning without exposing hidden reasoning, so I can spot brittle assumptions, prune distractions, and re-ground the task.
Content retrieval has become a killer workflow. I point the agent at my archives—blog posts, book drafts, transcripts, notes—and ask, “Where have I talked about this before?” It assembles a map of prior art, connects themes I’d forgotten, and prevents me from reinventing work. Over time, this evolves into a Zettelkasten-style research system that upgrades rigor and accelerates synthesis.
I’ve also turned Claude Code into a publishing engine. From a single transcript, it drafts titles, descriptions, show notes, and chapters, then routes artifacts to Ghost for formatting. Before anything ships, I run fact-checking workflows that validate claims against transcripts and research sources. The output improves, but more importantly, the scaffolding makes quality repeatable.
Reusable workflows compound. I rely on slash commands to trigger common jobs, break down larger efforts with sub-agents, and wire in hooks and plugins where external systems are needed. This is agentic AI at its most practical: fewer hero prompts, more reliable processes.
Audience analytics and content prioritization are helpful with caveats. I let the agent cluster themes and flag gaps, then I pressure-test its suggestions against first-party data and strategic goals. As with any model-driven insight, triangulation beats blind faith.
Two metaphors guide my day-to-day. First, Claude Code is like a dog—sometimes it returns with the stick, sometimes it gets lost in the woods. Second, the “intern” framing keeps me honest: don’t hand it the whole company on day one. With that mindset, my output jumped—more volume without sacrificing quality—because the workflow scaffolding got better.
In this episode, I cover what Claude Code is and why it’s useful even if you’re not an engineer, the real difference between the browser experience and running locally, how to shape behavior with /init and Claude MD files, why task management is the perfect proving ground, when to export calendar windows versus connecting directly, and when model-switching makes sense—Haiku for speed, Opus for depth.
I also dig into debugging web tasks by asking “What are you doing?”, content retrieval workflows across personal archives, building reusable slash-command systems with sub-agents, hooks, and plugins, practical publishing stacks from transcripts, fact-checking against transcripts and research sources, and using analytics to prioritize content—with a healthy respect for uncertainty.
If you’ve been trying to make Claude Code feel less like “throwing a stick into the woods,” this is the candid, tactical tour I wish I’d had on day one. Drop your questions and experiments below—I’m eager to compare notes and refine the playbook together.
I’ve been exploring what I call the next level of vibe coding: orchestrating agentic AI to build complex product artifacts in minutes, not days. The breakthrough comes from ditching linear handoffs and embracing true parallelism—letting specialized agents tackle the work simultaneously while I steer the orchestration. In product management contexts where speed and clarity matter, this shift changes everything.
Building a KPI Driver Tree in two hours becomes possible when you stop building sequentially and start building with parallel agents.
For product leaders, a KPI Driver Tree is the fastest way to make strategy legible. It ties high-level outcomes to the levers we can actually pull—features, channels, pricing, onboarding, activation, and retention mechanics—so we can prioritize with confidence. Done well, it connects outcomes vs output OKRs, clarifies measurement, and aligns the team around a shared, testable model of growth.
Here’s how I operationalize it with agentic AI and AI workflows. I spin up a small team of specialized parallel agents: a Metrics Librarian (taxonomy and definitions), a Data Modeler (event and table design), a Research Synthesizer (voice of customer and causal hypotheses), a UX Prototyper (visualizing the tree and flows), and a QA/Evaluator (logic and consistency checks). An Orchestrator coordinates these agents, resolves conflicts, and composes outputs into a single, production-ready artifact—while I set constraints, review deltas, and decide.
In a typical two-hour sprint, all agents run at once. While the Metrics Librarian finalizes the KPI ontology, the Data Modeler validates instrumentable events and joins, and the UX Prototyper renders an interactive driver tree for a unified analytics platform. Meanwhile, the Synthesizer maps qualitative insights to quantitative levers, and the Evaluator stress-tests assumptions. Because we’re not waiting for sequential handoffs, we converge on a coherent driver tree and its initial measurement plan in one pass.
The payoff isn’t just speed—it’s higher-quality decisions. Parallel agents reduce context loss, expose trade-offs earlier, and allow me to compare multiple viable paths side-by-side. This accelerates continuous discovery, aligns with product strategy, and gives product managers and LLMs for product managers a clear, living map of how inputs roll up to outcomes. It’s the closest I’ve found to running a product trio at machine speed.
Guardrails matter. I pair this approach with strong data governance, privacy-by-design, and eval-driven development so every agent’s output is testable and auditable. Clear prompts, scoped corpora, and consistent acceptance criteria keep the Orchestrator honest, while lightweight Agent Analytics helps me see where reasoning falters and where to improve the system.
If your team is still tackling analytics artifacts sequentially—requirements, then instrumentation, then visualization—consider switching mental models. Treat the driver tree as the backbone, empower parallel agents to co-create around it, and reserve human judgment for the critical calls. This is vibe coding for product management: creative, fast, and grounded in measurable outcomes.
If your team has plenty of dashboards but still spends too much time turning a product question into a cohort, an explanation, and a decision, the bottleneck is no longer data collection. It is the work between asking the question and acting on the answer.
Treat the upgrade as a decision system, not an AI shortcut
A weak rollout starts by giving everyone access and encouraging them to try prompts. That produces activity, but it does not establish whether the technology is improving product work.
Define the unit of value as a completed decision. Each use of AI Visibility should move through a traceable sequence:
Start with a specific product question that could change an action.
Translate the question into an explicit cohort and metric definition.
Examine the relevant behavioral evidence.
Draft a narrative that separates observations from interpretations.
Record the decision, owner, and next action.
The enhancements reduce different kinds of friction inside that sequence. AI chat can reduce the interface work involved in expressing a segment. Content generation can reduce the effort required to turn analysis into a readable brief. A clearer interface can make the workflow easier for cross-functional partners to follow. Reliability improvements can support confidence in the system. None of those changes removes the need to define the question or approve the conclusion.
I would begin with two or three recurring, high-value use cases, not every analytics task. A good pilot question appears often, has a trusted baseline for comparison, and ends in a recognizable decision. Activation analysis, churn exploration, and experiment reporting meet those conditions for many product teams.
Match each enhancement to a concrete product job
Do not ask a team to use AI for analytics in the abstract. Give each workflow an input contract: the decision being considered, the population, the behavior, the observation period, the metric, and the exclusions. This prevents a fluent prompt from hiding an underspecified question.
Find an activation bottleneck without redefining activation
An activation question usually sounds simple: which new users reach value, and where do the others stop? The difficult part is deciding what counts as a new user, what behavior represents value, how long the observation period lasts, and which internal or test activity should be excluded.
Set those definitions before opening AI chat. Then describe the desired cohort in behavioral language and use chat-driven segmentation to iterate on it. Before analyzing the result, compare the AI-created segment with a known cohort, a manually configured version, or an established dashboard. If the populations differ, investigate the definition rather than explaining the chart.
Once the segment is accepted, use content generation to draft a brief that identifies the observed drop-off, the affected population, the relevant comparison, and the question that deserves further discovery. Keep causal language out unless the evidence supports it. A funnel can show where behavior changes; it does not, by itself, explain why.
Explore churn precursors without turning correlation into cause
Churn analysis becomes unreliable when a cohort mixes users who never activated, customers who became inactive, and accounts that formally cancelled. Those are different states with different product implications.
Write a plain-language definition of the state you care about before generating the segment. A useful prompt pattern is: create a cohort of the specified customer population that completed the core behavior during the reference period but did not complete it during the comparison period; exclude internal and test activity; then separate the result by the business attribute relevant to the decision.
Use AI chat to test legitimate variations in that definition, not to invent the definition for you. When a behavioral difference appears, label it as a precursor or association until customer evidence or an experiment supports a causal explanation. The next action may be another analysis, a customer interview, or a retention experiment. It should not automatically be a roadmap commitment.
Draft experiment reports without delegating the decision
AI-generated experiment summaries are useful because the structure is repetitive even when the decision is not. Give the system the approved hypothesis, eligible population, exposure definition, primary outcome, guardrail measures, and underlying analysis. Ask for a draft that covers what changed, what remained uncertain, which segments require caution, and what decision the evidence supports.
The generated narrative should never become the statistical authority. The experiment analysis remains the record for effect estimates, uncertainty, and data-quality caveats. The brief exists to make that evidence understandable and actionable. If the prose and the analysis disagree, correct the prose before it travels to stakeholders.
Put human review around definitions and conclusions
AI can make a loosely defined request look finished. That is the central operating risk. The safest control is to review the workflow where meaning enters and where meaning leaves: validate the segment before interpreting the result, then validate the narrative before sharing it.
Validate the segment before reading the result
Confirm the identity unit. A user, device, workspace, and customer account are not interchangeable.
Check that event names and properties map to the team’s current tracking taxonomy.
Make inclusion rules, exclusions, sequence requirements, and observation periods explicit.
Compare membership or aggregate trends with a trusted manual definition when one exists.
Inspect surprising differences before using them as evidence. A mismatch may come from the cohort definition rather than user behavior.
Store a plain-language definition with the accepted cohort so another person can reproduce the analysis.
Validate the narrative before distributing it
Require each material claim to point back to a chart, table, or approved metric.
Separate observed behavior from a proposed explanation.
Verify that the population, date range, and comparison in the prose match the analysis.
Remove unsupported causal language and any detail the audience is not permitted to access.
State the decision, the remaining uncertainty, and the person responsible for the next action.
Content generation reduces drafting work; it does not transfer review responsibility to the model. This distinction is especially important for executive briefs, where polished language can make a weak inference appear more certain than it is.
Govern prompts, access, and workflow changes
Basic prompt templates, access policies, review steps, and data-governance controls turn experimentation into a repeatable capability. A prompt template should specify the business question, required definitions, exclusions, expected output, evidence standard, and reviewer. Access should follow the same least-privilege principles applied to the underlying analytics data.
Reliability also needs operational visibility. Keep a lightweight record of the original question, accepted cohort definition, supporting analysis, generated brief, reviewer, and resulting decision. When an answer changes unexpectedly, that record helps you distinguish a tracking problem from a cohort change, a prompt change, or an interpretation error.
Measure whether the rollout changes product decisions
Prompt volume and generated summaries are adoption signals, not proof of value. Establish a baseline before the pilot, run the selected use cases through the new workflow, and compare the result using measures tied to decisions.
Signal
How to observe it
What a weak result means
Time-to-insight
Track elapsed time from an accepted question to a reviewed analysis brief.
If the time does not fall, find the handoff or review step that still creates delay.
Stakeholder adoption
Track whether product, design, engineering, growth, and leadership use the workflow in recurring decisions.
If only analysts use it, the interface or output may not fit cross-functional work.
Decision velocity
Track elapsed time from requesting evidence to recording an explicit decision or next action.
If output increases but decisions do not move sooner, the workflow is producing content rather than clarity.
Review quality
Count material corrections to cohort definitions, metrics, and conclusions before and after sharing.
If rework rises, improve the event taxonomy, prompt contract, validation process, or reviewer guidance before expanding access.
Trust exceptions
Record cases in which an AI-assisted result conflicts with validated analytics or cannot be reproduced.
If exceptions persist, pause expansion and resolve the data, definition, or workflow problem.
Judge the pilot as a system. Faster segmentation with heavy correction is not a win. Faster drafting with unchanged decision velocity is not a win either. The useful outcome is a shorter path from question to reviewed decision, with stable or improving quality.
Expand only after the pilot workflow is reproducible. At that point, turn the accepted prompt patterns, cohort definitions, review criteria, and measurement approach into a shared operating playbook. The cleaner interface can help more partners participate, but the playbook is what keeps participation consistent.
Key takeaways
Use Amplitude AI Visibility to shorten a decision workflow, not merely to increase the volume of segments and summaries.
Begin with two or three recurring use cases that have trusted baselines and recognizable decisions.
Define the population, behavior, period, metric, and exclusions before asking AI to create a segment.
Validate cohort meaning before interpreting behavior, then validate the generated narrative before sharing it.
Scale the workflow only when faster output is accompanied by reproducibility and sound review.
Choose the next recurring product decision that still involves too much manual translation. Write its input contract, capture its current path to a reviewed decision, and use that single workflow to determine whether AI Visibility is removing the right friction.
Your AI agent may look convincing in a demonstration and still disappear from daily work. If people try it once but return to spreadsheets, dashboards, tickets, and manual handoffs, you do not have an awareness problem. You have a workflow design problem.
Real adoption begins when a specific user can delegate a meaningful part of a recurring job, understand the agent’s limits, and see that the resulting decision or action is better. Product leaders create those conditions by narrowing the workflow, defining the agent’s authority, measuring the complete decision loop, and expanding autonomy only after the evidence supports it.
Choose a workflow, not a place to add AI
Starting with “Where can I deploy an agent?” pushes the team toward a feature. Start with “Which recurring decision or action is unnecessarily difficult?” That question keeps the work tied to customer or business value.
A good first workflow is frequent enough to generate feedback, narrow enough to evaluate, and bounded enough that a mistake can be caught before it causes material harm. It also has an identifiable beginning and end. “Help people be more productive” is not a workflow. “Use approved customer evidence to prepare the next-best-action options for a campaign review” is much closer.
Evaluate candidate workflows against six practical criteria:
Trigger: The user can recognize the moment when the agent should enter the workflow.
Frequency: The job repeats often enough for the user to form a habit and for the team to learn from actual use.
Grounding: The agent can retrieve the approved data, policies, history, or customer evidence required to do the job.
Completion: The team can observe whether the task reached a useful end state, rather than merely whether the model returned text.
Decision boundary: Everyone can state what the agent may decide, what requires approval, and what it must never do.
Recoverability: An incorrect recommendation or action can be rejected, corrected, or reversed without disproportionate damage.
Mark each candidate high, medium, or low on those criteria. Do not hide a weak decision boundary behind an attractive use case. A repetitive workflow with clear evidence and a review point is usually a better adoption bet than an ambitious end-to-end process with unclear ownership.
This is also why natural-language access alone is not an agent strategy. It can lower the barrier between a user’s question and an analytical answer, which may improve activation. Adoption becomes more valuable when the answer connects to a defined next action and the eventual impact of that action can be observed.
Write the selected workflow in one sentence before approving a roadmap:
When [user] encounters [trigger], the agent uses [approved context] to [recommend, prepare, or execute an action]; [person or policy] controls [decision boundary], and success is measured by [workflow or customer outcome].
Agent workflow template
If the team cannot complete that sentence without vague language, discovery is not finished.
Write an adoption contract before writing the roadmap
An agent changes who performs work, which information informs it, and where accountability sits. That is an operating-model decision disguised as a product feature. A one-page adoption contract makes the change explicit before implementation creates momentum around the wrong behavior.
The contract should answer seven questions:
Who is the intended user? Name the role and the situation, not a broad department.
What job is being delegated? Separate information retrieval, analysis, recommendation, preparation, and execution. They carry different risks.
What outcome should improve? Connect the workflow to an existing customer or business outcome, not to the amount of AI content produced.
Which information is authorized? Identify the systems of record, retrieval scope, freshness requirements, and data that must remain unavailable.
Where does human judgment remain mandatory? Put approval at the consequential decision, not at an arbitrary screen in the interface.
How should uncertainty and failure appear? Define when the agent should cite evidence, ask for missing context, abstain, escalate, or report that a tool failed.
What earns expansion? Specify the quality, adoption, outcome, and risk signals required before the agent receives more users, tools, or autonomy.
This contract prevents a common measurement error: treating interaction volume as value. Conversations, generated documents, and tool calls are outputs. They can help diagnose behavior, but they do not show that the workflow improved. Activation, successful completion, repeat use at the next relevant trigger, and retention are stronger adoption signals. They still need to connect to a journey outcome such as a better decision, a completed customer task, or a validated change.
Use outcomes versus output OKRs to keep the distinction visible. An output key result might promise to launch an agent or add integrations. An outcome key result should describe the behavior or customer result that the workflow is intended to change. The delivery milestone belongs in the plan; it should not masquerade as proof of adoption.
The contract also makes prioritization easier. A request for another model, data connector, or agent tool must improve a named part of the workflow. If it cannot be tied to grounding quality, task completion, user control, or the target outcome, it is probably infrastructure enthusiasm rather than a product requirement.
Earn autonomy through observable stages
Do not jump from a chat interface to autonomous execution because the happy-path demo worked. Autonomy should advance in stages, with a different role for the user and a different standard of evidence at each stage.
Capability stage
What the agent does
Human responsibility
Evidence needed to advance
Explain
Retrieves and synthesizes approved information
Checks the evidence and interprets it
Grounding, completeness, and answer-quality evals
Recommend
Produces alternatives or ranks possible next actions
Makes the decision and records important overrides
Relevance, reasoning, boundary, and decision-support evals
Prepare
Creates a draft action, configuration, or artifact without committing it
Edits and approves before execution
Task-specific correctness, policy, format, and exception evals
Act
Executes a bounded action through approved tools
Supervises exceptions and reviews consequential cases
Reliable task completion, tool behavior, auditability, and recovery controls
The stages are not a maturity contest. Some workflows should remain in recommendation or preparation mode because the consequences of an incorrect action outweigh the benefit of removing approval. Human-in-the-loop design is useful when the person has evidence, authority, and enough context to intervene. A mandatory click from someone who cannot evaluate the result adds friction without adding control.
Before releasing each stage, create an evaluation set that represents the actual workflow. Include normal cases, ambiguous requests, missing or stale context, policy boundaries, conflicting evidence, and tool failures. For every case, record the expected behavior, unacceptable behavior, scoring rubric, and evidence the evaluator should inspect.
Do not collapse evaluation into a single pass rate. An answer can be fluent and wrong, properly grounded but irrelevant, or correct while attempting an unauthorized action. Score the dimensions that matter independently: retrieval and grounding, task correctness, tool selection, instruction adherence, policy compliance, escalation behavior, and completion quality.
Treat prompts and evaluation datasets as versioned product assets. When the model, prompt, retrieval logic, tool definition, or policy changes, rerun the relevant evaluation set and preserve the result with the release. Otherwise, a team can improve one visible behavior while silently degrading another.
A retrieval-first design is especially important when the workflow depends on institutional knowledge. The agent should use authorized context before relying on general model knowledge, expose enough evidence for the user to inspect, and ask for clarification or abstain when required context is unavailable. That behavior may look less magical in a demonstration, but it is much easier to trust in repeated work.
Measure the entire agent loop, not the chat surface
A traditional feature funnel can tell you who opened an agent and who returned. It cannot explain whether the agent retrieved the right context, selected the right tool, required extensive correction, or produced an action that affected the intended outcome. Agent Analytics must reconstruct the path from intent to result.
Instrument the workflow as a connected event chain:
Intent and eligibility: Which workflow was triggered, and was the user and situation within scope?
Context: Which approved knowledge or data was retrieved, and was essential context unavailable?
Reasoning path: Which plan or action sequence did the system select?
Tool behavior: Which tools were called, which arguments were passed, and where did errors or retries occur?
Human intervention: Did the user accept, edit, reject, override, or abandon the result?
Completion: Did the workflow reach its defined end state?
Outcome: Did the customer or business indicator named in the adoption contract move in the intended direction?
Apply privacy-by-design to that event model. Logging every raw prompt, retrieved record, or tool payload by default can create unnecessary exposure. Decide which fields are required for product learning, who may access them, how sensitive data is handled, and how long the information is retained. Data governance belongs in the instrumentation design, not in a review after launch.
Review four layers together:
Quality: Evaluation results by task and failure dimension.
Behavior: Activation, successful completion, repeat use, abandonment, edits, and overrides.
Outcome: The customer or business result attached to the workflow.
Risk and reliability: Boundary violations, unsupported claims, tool failures, escalations, and consequential incidents.
Each layer corrects a possible misreading. High usage with weak quality can mean users are compensating for the system. Strong offline quality with little repeat use can mean the workflow is not important or the interaction arrives at the wrong moment. Completion without an outcome can mean the agent is accelerating work that should not have been done. Outcome movement without traceability makes it difficult to know whether the agent deserves credit or whether the result will persist.
Use qualitative evidence to explain those patterns. Review corrections and overrides, collect feedback at the point of use, and connect support signals to roadmap decisions. A generic satisfaction question is less useful than asking what evidence was missing, which step the user repeated manually, or why the recommendation could not be acted on.
When comparing user-facing variants, define the primary outcome and minimum detectable effect before running an A/B test. This prevents the team from declaring success based on an incidental movement in a convenient metric. A/B testing is appropriate only where traffic, exposure, and risk make controlled experimentation meaningful; rare or consequential actions need direct evaluation, review, and guardrails instead.
Make agent adoption an operating change
A launch campaign can create trials. It cannot resolve unclear ownership, weak evaluation, missing context, or a workflow that asks users to supervise the agent without giving them useful control. Sustainable adoption requires a product operating model around the capability.
Give a product trio responsibility for the complete workflow and pair it with the people who can close the distance between a prototype and production use:
Product management owns the user problem, target outcome, decision boundary, adoption contract, and expansion decision.
Design owns how intent, evidence, uncertainty, approval, correction, and escalation appear in the experience.
Engineering owns retrieval, tool permissions, system behavior, observability, release controls, and recovery paths.
A forward deployed engineer or equivalent customer-facing technical partner helps expose the real context, integrations, and exceptions hidden by a clean prototype.
Data and risk owners define acceptable model behavior, privacy constraints, access rules, and the evidence required for governance.
The leadership cadence should follow the learning loop. Discovery identifies a high-value workflow and pressure-tests it with user evidence. Pre-release review examines evaluations and failure modes. A narrow rollout tests the workflow with explicit human checkpoints. Operating reviews examine quality, behavior, outcomes, and incidents together. Expansion adds a capability, population, tool, or level of autonomy only when the prior boundary is performing as intended.
This model should influence AI hiring as well. A strong AI product candidate should be able to turn a broad ambition into a bounded workflow, define an evaluation rubric, separate model quality from product outcomes, place human judgment at the right decision, and explain what evidence would justify more autonomy. Prompt fluency without those skills is not product leadership.
Key takeaways
Start with one recurring, bounded workflow whose completion and outcome can be observed.
Write an adoption contract covering the user, trigger, delegated job, approved context, decision boundary, failure behavior, and expansion criteria.
Progress from explanation to recommendation, preparation, and bounded action only as evaluation and production evidence improve.
Version prompts, retrieval logic, tool definitions, and evaluation datasets with releases.
Instrument intent, context, tool calls, human intervention, completion, and downstream outcomes as one decision loop.
Scale when quality, repeat use, workflow outcomes, and risk controls agree – not when a demonstration attracts attention.
Your next move does not need to be a company-wide agent mandate. Put three candidate workflows through the six selection criteria. Choose the one with the clearest trigger, evidence, completion point, and decision boundary. Then write its adoption contract and evaluation set before funding a broad build. If the narrow workflow earns repeat use and improves its named outcome, you will have evidence for the next capability – and a repeatable method for every agent that follows.
This is the year to build your personal operating system. For me, that line isn’t a slogan; it’s a commitment to eliminate context switching, compress decision cycles, and turn fragmented information into a reliable source of truth. As a product leader, I needed a system that blends judgment, data, and automation—so I built mine around Claude Code.
When I say “personal operating system,” I mean an integrated set of AI workflows, rituals, and tools that capture knowledge, structure decisions, and automate execution. It’s where product discovery meets delivery: a place to synthesize signals, prioritize with clarity, and move from insight to action without friction. The outcome is fewer ad hoc decisions, more deliberate strategy, and a calmer, more focused day.
Claude Code sits at the center because it helps me translate intent into working software and repeatable processes. I use it to scaffold small utilities, write adapters for APIs, and evolve prompts into robust patterns. It accelerates everything from research synthesis and PRD drafting to backlog grooming and stakeholder updates—while keeping me in the loop for final judgment.
Under the hood, I run a retrieval-first pipeline that connects notes, docs, tickets, research transcripts, and roadmaps into a searchable, living memory. With careful context window management, I feed only the most relevant snippets into Claude Code, preserving accuracy and speed. The result: richer answers, fewer hallucinations, and an assistant that “remembers” what matters without drowning in noise.
My daily loop is simple: capture, synthesize, decide, and act. I capture customer signals and meeting notes into a personal knowledge management vault; synthesize patterns with prompt engineering that emphasizes evidence; decide using outcomes vs output OKRs; and act by generating drafts, creating tasks, and updating artifacts. Claude Code helps me wire this end-to-end, so the system works even on my busiest days.
If you’re implementing this from scratch, start small. Pick one high-friction workflow—say, product feedback triage—and build a narrow agentic AI flow to classify, summarize, and route items. Use eval-driven development to test prompts against known edge cases. Add guardrails and privacy-by-design practices from day one, then expand to neighboring workflows once the first loop is reliable.
Governance matters. I treat AI risk management, data governance, and security as first-class citizens: limited data scopes, clear audit trails, human-in-the-loop approvals, and rollback plans. Feature flags control changes; observability tracks drift and quality; and a simple playbook documents how we deploy, monitor, and improve the system.
Measure what this personal operating system earns you. Track decision latency, cycle time from signal to action, meeting-to-output ratios, and the signal-to-noise ratio of inputs. When the system is working, you’ll feel it: fewer meetings, more momentum, and sharper product strategy supported by trustworthy AI workflows.
The goal isn’t to automate judgment—it’s to protect it. By letting Claude Code handle the glue work and information wrangling, I preserve energy for high-leverage thinking: positioning, sequencing, and trade-offs. Build your personal operating system now, and make this the year your product practice runs with clarity and composure.
You have AI pilots that demo well, enthusiastic teams asking for broader rollout, and executives expecting the investment to show up in operating results. Yet the closer you get to production, the longer the list of unresolved questions becomes: Who owns the workflow? How will quality be measured? What happens when the model is wrong? Can the economics survive real usage?
The next move is not to launch more pilots. It is to install a system that can repeatedly turn a validated use case into a governed, measurable, and improving production workflow. That system is what separates AI experimentation from mature deployment.
A successful pilot is not evidence of production readiness
AI adoption is already common enough that adoption itself tells you very little. Among more than 2,400 global customer service professionals, 82% of senior leaders invested in AI in 2025, 87% planned to invest in 2026, and only 10% described their deployment as mature. The sample is specific to customer service, so those figures are better used as a directional benchmark than as a universal maturity rate. The underlying execution problem applies much more broadly: buying or piloting AI is easier than making it dependable inside a core workflow.
A pilot is designed to answer a narrow learning question. Can the model classify this request, draft this response, summarize this record, or choose the next action under controlled conditions? Production has to answer a harder question: can the entire workflow create enough value, across ordinary and difficult cases, while remaining safe, observable, supportable, and economically sensible?
I use a simple test. If the team can describe the model but cannot describe the operating workflow around it, the work is still a prototype. A production case should make each of these elements explicit:
Outcome: The customer or business result that should improve, plus the current baseline.
Workflow boundary: Where AI enters, which decisions it may make, which systems it may use, and where its authority ends.
Quality standard: The evaluation cases, acceptance criteria, and failure categories that determine whether a release is good enough.
Safe failure path: What the system does when information is missing, a tool fails, a policy is triggered, or the requested action exceeds its authority.
Accountability: A named product owner for the outcome and a named operational owner for production performance.
Economics: The value created and the full cost of inference, retrieval, tools, review, support, and incident handling.
Learning mechanism: How production failures and user corrections return to the evaluation set and release process.
These are not finishing tasks to schedule after the model works. They are part of the product. Deferring them creates a predictable trap: the pilot looks increasingly impressive while the distance to a responsible launch quietly grows.
Do not confuse automation coverage with maturity, either. A system can handle many requests and still be immature if nobody can explain why it made a decision, detect a quality regression, contain a failure, or calculate the result. Conversely, a narrowly scoped workflow can be mature when its boundaries, controls, outcomes, and ownership are clear.
Promote each workflow through explicit maturity gates
Maturity should be earned workflow by workflow. An organization does not become mature because it has a central AI team, an approved model vendor, or a large portfolio. It becomes mature when important workflows can move through a repeatable sequence of decisions without relying on heroics.
Stage
Decision to make
Evidence required to advance
Reason to hold
Discover
Is this a valuable and appropriate problem for AI?
A defined user problem, current baseline, workflow map, risk classification, and initial build-versus-buy view
The use case is driven by model novelty, has no meaningful outcome, or depends on inaccessible data
Prove
Can the proposed workflow improve on the current process?
Representative evaluation cases, a working prototype, documented failure modes, and a controlled comparison with the baseline
Success appears only in curated demos, or the team cannot reproduce the result across realistic cases
Operate
Can the workflow run safely and reliably in production?
Monitoring, escalation, access controls, auditability, incident procedures, release controls, rollback, and an accountable operator
Failures cannot be detected or contained, or production responsibility is still ambiguous
Scale
Should usage, autonomy, channels, or organizational reach expand?
Sustained outcome improvement, acceptable quality and risk, validated economics, user adoption, and reusable operating components
Volume is growing faster than quality, cost, support capacity, or governance can be understood
The purpose of a gate is not to create a committee. It is to prevent enthusiasm, executive attention, or sunk cost from substituting for evidence. The domain team should be able to prepare the evidence as part of normal product development. Specialist review should become more demanding only as the possible consequence of failure increases.
Give every workflow a short deployment contract. Keep it in the same system where the team manages releases and evaluations, not in a presentation that disappears after approval. The contract should include:
The intended user, job to be done, business outcome, and current baseline.
The inputs the workflow accepts and the outputs or actions it may produce.
The actions that are prohibited, require confirmation, or must be routed to a person.
The data sources, retrieval rules, system permissions, retention rules, and privacy constraints.
The evaluation set, quality dimensions, acceptance criteria, and known limitations.
The failure taxonomy, escalation path, incident owner, and customer recovery procedure.
The prompt, model, retrieval, tool, and policy versions included in the release.
The production metrics, cost measures, rollout control, and rollback conditions.
The product owner, operational owner, and risk approvers.
The acceptance criteria will differ by workflow. A drafting assistant, an internal search experience, and an agent authorized to modify a customer account should not face the same bar. Base the bar on consequence, reversibility, detectability, and recovery. If an error can create an irreversible change, expose sensitive data, make a material commitment, or deny someone an important service, require an appropriate human authorization step rather than relying on average model performance.
The deployment contract also makes scope changes visible. Adding a new tool, data source, channel, language, model, or autonomous action is not merely more traffic. It changes the system’s failure surface. Update the contract, extend the evaluation set, and pass the relevant gate again.
Build three feedback loops before increasing autonomy
A mature deployment learns at three levels: whether the workflow creates value, whether its decisions meet the required standard, and whether the production system remains reliable. If any loop is missing, the team can collect impressive activity metrics while the actual product deteriorates.
Connect model behavior to a business outcome
Start with the baseline process, not an AI metric. If the workflow is intended to resolve a support request, qualify an opportunity, complete an onboarding step, or assist an employee, measure how that outcome happens without the new system. Otherwise, you will know that the AI generated output but not whether it improved anything.
Use a metric stack that separates outcomes from diagnostics:
Business outcome: The customer, revenue, cost, risk, or productivity result the investment is meant to change.
Workflow outcome: Completion, resolution, successful handoff, correction, rework, abandonment, or another measure of whether the task reached its intended end.
Quality and safety: Correctness, grounding, policy compliance, appropriate escalation, harmful failure, and user correction.
Economics: Cost per successful outcome, including model usage, infrastructure, external tools, human review, support, and remediation.
The layers diagnose different problems. A prompt change may improve an offline score without changing task completion. More automation may reduce handling work while increasing corrections. A cheaper model may lower inference cost but create enough rework to raise the cost per successful outcome. Do not compress those effects into one AI score.
When traffic and product conditions support an experiment, compare the AI workflow with the current experience. Define the decision metric and minimum detectable effect before running an A/B test. For lower-volume or higher-risk workflows, use controlled rollout evidence, expert review, and structured case analysis rather than pretending a small sample provides statistical certainty.
Turn evaluations into release criteria
An evaluation set is not a collection of attractive examples. It should represent ordinary work, difficult edge cases, policy boundaries, known failures, and the situations in which the system should refuse or escalate. Build it before optimizing the prompt so the team cannot unconsciously redefine success around whatever the prototype already does well.
For each case, record the expected behavior and why it is expected. Some outputs can be checked against a deterministic answer. Others need a rubric that distinguishes task completion, factual support, instruction following, tone, policy compliance, and escalation quality. Where reviewers can reasonably disagree, capture that disagreement instead of forcing false precision into a single label.
Use offline and online evaluation for different jobs. Offline evaluation protects releases by testing candidate changes against a stable set. Online evaluation reveals distribution shifts, new user behavior, integration failures, and outcomes that cannot be recreated fully before launch. Neither is sufficient on its own.
Version the entire behavior-producing system: model, prompt, retrieval configuration, knowledge snapshot, tools, policies, and routing logic. A model comparison is not meaningful if the surrounding system changed silently. For every proposed release, make the decision policy explicit: ship, hold, narrow the scope, expand gradually, or roll back. This is the practical core of eval-driven development with target metrics and a decision policy defined before launch.
Operate the workflow as a production service
AI introduces variable outputs, but it still depends on familiar production systems: identity, permissions, data pipelines, APIs, queues, search, external tools, and user interfaces. A model can appear to be wrong when retrieval returned stale information or a downstream tool rejected an action. Monitoring only the final text hides the failure that engineers need to fix.
Trace the workflow end to end. Subject to your privacy and retention rules, capture the release version, retrieval and tool events, policy decisions, response, escalation, user correction, and eventual workflow outcome. Monitor distributions and failure categories, not just averages. An acceptable overall score can conceal a serious regression for a particular intent, customer segment, channel, or action.
When the workflow depends on changing or private knowledge, connect it to governed retrieval instead of expecting the base model to contain the right answer. Use safe integration points for tools, least-privilege access, and explicit authorization for consequential actions. CI/CD, feature flags, canary releases, observability, audit trails, privacy controls, red teaming, and human review form a practical control plane for releasing changes without exposing the entire population at once.
Every material production failure should produce more than an incident ticket. Classify the failure, add or update the corresponding evaluation case, correct the prompt, retrieval, policy, tool, or interface responsible, and retest the workflow before restoring scope. That turns operational pain into a permanent improvement in the release system.
Use 30-60-90 days to build the scaling system
A useful 30-60-90-day sequence starts with two lighthouse use cases. The goal is not to force every use case into production within a quarter. It is to prove that your organization can move valuable workflows through the same gates, shared controls, and learning loops.
Days 0-30: narrow the portfolio and establish accountability
Inventory active pilots and classify each as discovery, proof, operation, or scale. Do not let a polished demo assign its own stage.
Select two lighthouse workflows using customer impact, feasibility, strategic relevance, and risk. Choose workflows meaningful enough to matter but bounded enough to operate responsibly.
Record the current process and baseline before the AI changes user or employee behavior.
Name the product owner, operational owner, and required risk decision-makers for each workflow.
Complete the first version of each deployment contract, including the autonomy boundary and safe failure path.
Make the build-versus-buy decision at the workflow level. Include data access, integration, auditability, evaluation portability, operating cost, and switching constraints.
Pause pilots that have no accountable owner, no measurable outcome, or no plausible route through the operating gate.
This first phase is where leadership earns focus. A broad AI mandate often creates a queue of unrelated prototypes, each with its own vendor, data assumptions, and definition of success. Choosing lighthouse workflows gives the platform and governance work a real customer instead of turning them into abstract architecture programs.
Days 31-60: install evaluation, controls, and workflow operations
Build the offline evaluation set from representative work, edge cases, policy boundaries, and failures already found during discovery.
Define acceptance criteria and the release decision policy before further prompt or model optimization.
Integrate the necessary retrieval and tools through governed access points. Keep permissions narrower than the user’s full access where the workflow does not need it.
Add observability across retrieval, reasoning inputs, tool execution, output, escalation, and business outcome.
Prepare feature flags, a controlled rollout, rollback, incident procedures, and a customer recovery path.
Run the workflow with appropriate human oversight. Record corrections and escalations as structured evidence, not informal feedback in chat.
Train the people who will supervise, support, and improve the workflow. Update operating procedures before transferring real responsibility to AI.
Training cannot be limited to prompt tips. Operators need to know what the system may do, how its failure modes appear, when to intervene, how to report a new failure, and who can change production behavior. Product and engineering teams need the same vocabulary for evaluation, incidents, and risk.
Days 61-90: expand evidence, not enthusiasm
Increase scope only for workflows that meet their operating gate. Expansion may mean more traffic, another intent, a new channel, or greater autonomy; evaluate each change explicitly.
Compare the production outcome and cost with the original baseline. Include corrections, review, support, and remediation in the economics.
Turn repeated needs into shared components such as model access, retrieval, identity, evaluation infrastructure, observability, policy enforcement, and audit logging.
Move validated production failures into the evaluation suite and confirm that the release process catches them.
Review job responsibilities, incentives, staffing assumptions, and training needs created by the redesigned workflow.
Hold a portfolio decision for every remaining pilot: advance, narrow, combine, pause, buy, or stop.
Organizational change is part of this phase. As AI altered customer service work, 45% of teams updated job descriptions and 40% increased AI training. That is a useful warning against treating adoption as an in-app onboarding problem. If AI takes responsibility for part of a workflow, someone must take responsibility for supervising it, handling exceptions, and improving the system.
Assign decision rights clearly. The domain product team should own the user problem, outcome, workflow design, evaluation cases, and adoption. A platform function should own shared access, retrieval, observability, release infrastructure, and policy enforcement. Risk specialists should define control requirements and review higher-consequence uses. The operational owner should manage quality, escalations, and incidents after launch. Executive leadership should decide portfolio priority, capacity, and which bets no longer deserve investment.
This structure avoids two common extremes. A fully centralized AI team becomes a delivery bottleneck and loses domain context. Fully independent teams duplicate infrastructure and apply inconsistent controls. Centralize reusable capabilities and non-negotiable policies; keep workflow outcomes and day-to-day learning with empowered domain teams.
Expect pressure to spread successful patterns. In customer service organizations, 52% planned to scale AI into areas such as customer success, marketing, and sales. Reuse the platform, governance, evaluation methods, and operating vocabulary. Do not copy a support workflow into another function and assume its value, risks, permissions, or quality bar remain valid.
FAQ: decisions that determine whether AI scales
Should AI be owned centrally or by product teams?
Use a federated model. Centralize capabilities that become safer, cheaper, or more consistent when shared: approved model access, identity, data controls, retrieval services, evaluation tooling, observability, auditability, incident standards, and risk policies. Embed workflow ownership in the domain team that understands the user, process, and business outcome. A central group can set the paved road, but it should not become the permanent product team for every AI use case.
When is an AI workflow ready for more autonomy?
Increase autonomy when the workflow has demonstrated acceptable behavior for the exact action and population being added, failures are detectable, consequences are containable, rollback works, and an operational owner can handle exceptions. Do not remove human review merely because the average quality score improved. Judge autonomy by the worst credible consequence, the reversibility of the action, and the system’s ability to recognize when it should stop.
Autonomy is not binary. The system can retrieve information, recommend an action, draft the result, ask for confirmation, execute within a limited permission, or execute and trigger retrospective review. Choose the narrowest level that captures the value. Expand only when evidence supports the next level.
When should a pilot be stopped rather than scaled?
Stop or reframe a pilot when it has no accountable workflow owner, cannot beat a meaningful baseline, works only on curated inputs, requires unacceptable access, has no safe failure path, or creates more review and remediation than the outcome justifies. Also stop when the supposed AI problem is actually a broken policy, missing data, or poorly designed process that should be fixed directly.
A failed autonomy concept can still reveal a useful assistive product. If execution is too risky, narrow the workflow to retrieval, recommendation, drafting, or exception detection. That is a product decision, not a face-saving exercise. The right scope is the one that creates measurable value under an operating model you can defend.
At your next AI portfolio review, ask each owner to bring a baseline, deployment contract, evaluation evidence, and a clear gate decision. Fund shared infrastructure where the lighthouse workflows expose a recurring need. Expand only after the operating evidence catches up with the demo. That is how you turn a collection of pilots into an AI capability that can carry real responsibility.
I just tuned into the latest conversation on the upcoming Product at Heart 2026, and it hit on the exact challenges product leaders are navigating right now: curating meaningful content in a world where AI moves faster than our agendas, designing formats that create real connection, and ensuring every minute earns its place. Listening to Petra Wille and Teresa Torres map out the speaker lineup, workshops, and structural shifts, I found myself nodding along—this is the kind of thoughtful curation we need if we want product teams and product leaders to walk away with practical value, not just inspiration.
Listen to this episode on: Spotify | Apple Podcasts
What stood out immediately is the bold move to a single-track conference for 2026. In an era of gen ai hype and endless breakouts, this choice signals clear intent: tighter curation, a shared experience, and less FOMO. The team isn’t carving out a separate AI track—and I love that decision. Their stance is simple and sensible: No AI track—AI will show up everywhere, but not as a siloed topic. The team sees it as part of the everyday toolkit. That mirrors how high-performing, empowered product teams actually work today—AI Strategy and AI workflows are part of the operating system, not a side show.
The keynote lineup is already compelling. Christian Idiodi (SVPG) brings storytelling that turns product principles into habits you can actually use on Monday. Elaine Kasket, cyber-psychologist, exploring digital afterlife and AI replicas, will push us to think more deeply about the human side of our systems. And Teresa Torres will be sharing what she’s learning about AI—exactly the kind of continuous discovery mindset we need as we integrate LLMs into product discovery and delivery.
I’m also thrilled to see roundtables become what they’re calling an “alternative track.” That’s a smart way to deepen learning without fragmenting attention. The best conference ROI I’ve had often comes from targeted small-group conversations—where product trios compare approaches, swap metrics frameworks, or challenge each other’s product strategy assumptions. It’s a design choice that rewards curiosity and builds communities of practice.
We also get a behind-the-scenes look at Teresa’s Maker Studio workshop, where participants will build personal AI workflows. That’s exactly the hands-on, practitioner-first approach teams need right now—less demo theater, more systems that stick. If your roadmap includes integrating LLMs into continuous discovery or augmenting your team’s decision velocity, this kind of guided practice is gold.
The broader workshop slate looks deep and balanced. Expect returning favorites and practical frameworks: Rich Mironov on the realities of product leadership in complex orgs; Büşra’s metrics workshop translating outcomes into action; and an overview of additional workshops from Rich Mironov, Büşra Coşkuner, Marcus Castenfors, and Özlem Yüce. From success metrics to toolkits for product managers, the content spans IC to product management leadership—ideal if you’re stepping into new roles or scaling empowered product teams.
One of the most exciting evolutions is the Product Leadership Event, now a 1.5-day retreat. The format blends talk sessions, mini-workshops, dinners, and small-group excursions (boat rides, improv, etc.), giving leaders time and space to exchange playbooks, stress-test decisions, and build real relationships. It’s capped at 60 attendees (all in product leadership roles) to keep it intimate and useful. As someone who believes in outcomes vs output OKRs and first principles decision making, I appreciate how this structure encourages depth over breadth—and real accountability among peers.
Here are the core takeaways I’m carrying into my own planning: single-track means tighter curation, so every talk has to earn its place. Roundtables are growing into an “alternative track,” offering more ways to engage beyond stage talks. Workshops go deep and meet you where you are—IC, manager, or executive. And the leadership retreat expands to maximize learning from peers, not just from the stage. If you care about product discovery, product strategy, and conference networking that leads to actual business impact, this program looks thoughtfully engineered.
If you’re planning your 2026 calendar—or just curious how conferences evolve alongside the craft—this is a thoughtful walkthrough of what to expect. Come say hi to Teresa and Petra—on stage, at a roundtable, or somewhere in the hallway conversations that make these events memorable.
For more context and resources mentioned, explore: Product at Heart, Arne Kittler, Mind the Product, Christian Idiodi of Silicon Valley Product Group, Elaine Kasket, House of Beautiful Business, The 7 Habits of Highly Effective People by Stephen Covey, Rich Mironov, Marty Cagan, Claude Code, Codex by OpenAI, Marcus Castenfors, Büşra Coşkuner and her Success Metrics: A Playbook for Product Managers, Özlem Yüce’s Essential Toolkit for Product Managers, Petra’s Product Leadership Wheel (PLwheel), and Netlight.
Follow Teresa Torres: https://ProductTalk.org
Follow Petra Wille: https://Petra-Wille.com
Full transcripts are only available for paid subscribers.
Your tender inbox contains a deadline, a stack of attachments, and a chain of decisions that still lives in people’s heads. The tempting response is to buy a model that can read PDFs. That solves only the most visible part of the problem.
A useful tendering product must determine which documents matter, extract requirements with evidence, match those requirements to a catalog, retrieve approved pricing, draft an offer, identify uncertainty, and route exceptions before anything reaches the customer. If you lead AI or product strategy for a manufacturer or supplier, your first goal should not be an autonomous bidder. It should be the smallest tender category in which every decision can be observed, evaluated, and improved.
Pick a bounded quote, not the entire tendering department
Construction tendering is too broad for a credible first release. Product categories have different terminology, selection rules, catalogs, pricing structures, and exception patterns. A system that works for one bounded category has not automatically learned how to quote every building product.
One effective wedge started with radiator requests for a single design partner before expanding to other building products. That constraint made the catalog, expected outputs, and expert reviewers knowable. It also created a place to learn the real workflow rather than designing from an idealized process diagram.
Choose your wedge using operational criteria, not enthusiasm for the model:
The request appears often enough for reviewers to recognize recurring patterns.
The relevant catalog is bounded and maintained by a clear owner.
A domain expert can explain why a product is suitable, unsuitable, or uncertain.
The correct price can be traced to an approved system or document.
Historical tenders and reviewer corrections are available for evaluation.
An error can be caught during review before it becomes a customer-facing commitment.
A design partner is especially valuable because the work is not fully documented. In one implementation, the product team spent a week observing the process on-site. That kind of observation exposes the browser tabs, informal checks, catalog shortcuts, and exception handling that an interview alone can miss.
Follow one tender from the incoming email to the final offer. At every handoff, record five things: the input, the decision, the evidence used, the person or system responsible, and the condition that triggers an exception. If you cannot state those five things, you do not yet have a well-defined agent task.
Keep three layers of information separate from the beginning:
Stated requirements: what the tender explicitly asks for, with the originating file, page, section, or table cell.
Interpretations: conclusions the system or reviewer draws when terminology is ambiguous, incomplete, or inconsistent.
Commercial decisions: the selected product, approved price, assumptions, exclusions, and offer language.
This separation matters because a polished offer can hide a weak inference. A reviewer needs to see where the tender ends and the system’s judgment begins.
Define the first outcome as a review-ready tender case: organized source documents, structured requirements, proposed product matches, price provenance, unresolved issues, and a draft offer. That is a more useful product boundary than “understands construction PDFs.” It gives the reviewer something concrete to accept, correct, or reject.
Turn the workflow into a decision graph with specialist agents
A chatbot is the wrong mental model. Tendering is a decision graph in which an early classification or extraction error can contaminate every downstream step. Real packages can range from a short request to more than 1,800 pages describing an entire building. The system therefore needs to plan work, retain state, reconcile evidence, and know when coverage is incomplete.
Use agents only where a task requires interpretation, planning, or exception handling. Keep exact operations – file handling, arithmetic, catalog queries, identifiers, template validation, and access control – in deterministic code or approved systems.
Stage
Preferred control
Required output
Intake
Rules plus a classifier
A tender case containing the email, attachments, document types, and routing status
Requirement extraction
Parser tools plus a specialist agent
Structured requirements with source locations, missing fields, and ambiguities
Product matching
Catalog retrieval plus a reasoning agent
Candidate products, requirement coverage, incompatibilities, and rationale
Pricing
Approved database, CPQ, or pricing service
Exact product-price records with source and validity information
Offer generation
Controlled template plus a drafting agent
A draft that distinguishes confirmed facts, assumptions, and exclusions
Quality review
Rules plus a separate review agent
A pass or block decision with issue codes and supporting evidence
Human approval
Domain and commercial policy
An approval, correction, rejection, or escalation that becomes evaluation data
Each agent should have an explicit contract. Specify its permitted inputs, tools, output schema, evidence requirements, completion test, and escalation behavior. “Find the right product” is not a contract. “Return catalog candidates that meet the extracted requirements, identify uncovered requirements, cite the catalog evidence, and abstain when no candidate qualifies” is much closer to one.
For large document sets, require the workflow to maintain a task plan. It should inventory files, identify relevant sections, process bounded units of work, track completed and pending units, reconcile repeated or conflicting requirements, and run a final coverage check. A generated answer is not proof that the package was fully processed.
The review agent deserves its own role. Asking the drafting agent to “check your work” keeps creation and approval inside the same reasoning path. A separate reviewer can inspect the draft against the extracted requirements, catalog evidence, price records, and policy rules. It should return defects and a gate decision rather than silently rewriting the offer. Silent rewriting makes it harder to identify which upstream component failed.
This pattern has practical value because a dedicated review agent can catch errors before human review, much like a separate code review step. Independence comes from the reviewer’s task and evidence contract; adding more agent personas without distinct responsibilities only creates orchestration overhead.
The interface is part of the architecture. During discovery, a dedicated web workbench can be more useful than hiding the workflow behind a legacy integration. Put the source document, extracted requirement, proposed match, price evidence, and review issue within the same review path. That gives the product team control over feedback capture and makes the reason for each correction visible. One tendering product used its own web application to iterate toward greater automation rather than beginning as a backend-only integration.
You can still read from and write to existing systems at defined boundaries. The distinction is between integration and dependence: integrate with systems of record for catalogs, prices, customers, and approved quotes, but do not let an inflexible legacy screen determine how reviewers inspect an emerging AI workflow.
Evaluate every decision before judging the complete quote
An end-to-end result tells you whether a tender case failed. It rarely tells you why. If the final product is wrong, the defect may have come from document routing, requirement extraction, catalog retrieval, product reasoning, pricing, drafting, or review. A single overall accuracy score collapses those failure modes into an unactionable number.
Pricing: exact agreement with the approved source, correct product-price association, formula validation, and rejection of unavailable or invalid records.
Offer generation: required-field completeness, consistency with selected products and prices, correct treatment of assumptions, and unsupported statements.
Review: detection of known defects, false blocks on valid cases, issue classification, and evidence quality.
Slice failures by characteristics that change the work: document length, file type, layout, product category, presence of tables, and conflicting or revised requirements. An aggregate score can improve while performance deteriorates on the long or unusual tenders that consume most reviewer attention.
Use end-to-end measures for the product outcome: review time, correction volume, correction severity, exception rate, percentage of cases that reach the defined completion state, and whether the workflow finishes before its operational deadline. Keep commercial outcomes separate from model correctness. Quote acceptance or win rate can be affected by price, availability, competition, customer relationships, and sales execution; it should not be treated as a clean extraction or matching metric.
Observability must connect those layers. For each tender case, retain the task plan, agent inputs and outputs, tool calls, retrieved catalog or price records, prompt and model versions, gate decisions, latency, failures, and human corrections. Complex agent chains can exceed what generic monitoring exposes, which is why custom tracing and Agent Analytics became necessary in a production tendering workflow.
Capture reviewer feedback as structured data, not only as an edited final document. Store the original output, corrected value, responsible stage, reason code, evidence used, and final disposition. Useful reason codes include missing requirement, incorrect extraction, unsupported product match, pricing issue, unresolved conflict, invalid assumption, and drafting defect.
Do not feed every edit directly back into the system and call it self-learning. A reviewer may change wording for preference, apply customer-specific knowledge, or correct an upstream error in the final draft. Validate the correction, assign it to the right component, and add it to the corresponding evaluation set. That turns human review into controlled learning rather than an untraceable feedback loop.
Release changes through two gates. First, the modified agent must pass its own evaluation set. Second, the complete workflow must pass the end-to-end set because an improvement in one component can change the assumptions of another. The trace should show exactly which version produced every customer-facing artifact.
Earn autonomy one commercial boundary at a time
“Straight-through processing” is incomplete unless you define where the straight-through path ends. Automatically extracting requirements is not the same risk as automatically selecting a product, committing a price, writing to the CPQ, or sending an offer to a customer.
Use an autonomy ladder with an explicit boundary at each stage:
Shadow: the system processes live-shaped cases, but its outputs do not affect the operational tender.
Assist: it organizes documents and extracts requirements while a person performs matching, pricing, and drafting.
Draft: it proposes products and produces an offer, but a human must review and approve every case.
Gated processing: it completes predefined internal actions for in-scope cases and sends all exceptions to a reviewer.
External dispatch: it sends an offer without case-by-case approval only when commercial policy explicitly permits that action and every required gate passes.
Eligibility for a higher-autonomy path should be machine-checkable. At minimum, confirm that the product category is in scope, every required document was processed, required fields are present, source evidence is attached, conflicts and revisions are resolved, the product match satisfies its evidence rules, the price comes from an approved valid source, the review agent reports no blocking defect, and the full trace is retained.
A wrong product or price can create margin, delivery, contractual, and customer-trust exposure. If an offer may create a binding commitment, keep human approval until the appropriate commercial and legal owners have defined the policy for automatic dispatch. The safe alternative is to automate preparation while preserving approval at the commitment boundary.
Operational controls matter after launch. Give reviewers a visible exception queue, make the reason for every block legible, preserve manual processing when the AI path is unavailable, and provide a way to suspend autonomous actions without disabling access to already processed cases. Assign owners for catalog quality, pricing validity, tender policy, model behavior, and production reliability; otherwise each exception will bounce between teams.
Expansion should follow evidence and customer pull. A request to replace an existing CPQ system is a meaningful product-market signal, but it is also a change in product scope. CPQ replacement introduces responsibilities for quote versions, approval policy, catalog administration, pricing governance, integrations, and records. Treat that request as a roadmap decision, not as proof that the original agent workflow already covers those capabilities.
Key takeaways
Start with one product category, one known workflow, and reviewers who can explain the correct decision.
Optimize first for a review-ready tender case, not an impressive answer from a general chatbot.
Use deterministic systems for exact operations and specialist agents for interpretation, planning, and exceptions.
Require structured outputs, source evidence, explicit completion tests, and an abstain-or-escalate path from every agent.
Evaluate each stage separately, then use end-to-end metrics to measure the operational outcome.
Increase autonomy only when observable eligibility gates protect the next commercial boundary.
At your next roadmap review, put one representative tender on the screen and draw the decision path from email to offer. Name the owner, evidence, pass condition, and exception path for every node. Wherever those are missing, the next task is workflow discovery, not another agent. Once the graph is explicit, you can automate one bounded decision, measure it, and earn the right to automate the next.
Happy New Year! I’m kicking off 2026 with a behind-the-scenes look at what’s changing in my product practice, the experiments I’m running with my teams at HighLevel, and the trends I’m most energized by—especially around continuous discovery, AI workflows, and building stronger coaching cultures.
If you want to listen to the conversation that sparked many of these reflections, you can find it here: Spotify | Apple Podcasts.
Why Teresa sunset the live deep-dive cohorts—and how on-demand and the new Discovery Habits Toolbox better support real behavior change. This pivot resonated with my own experience: some skills, especially discovery habits, only stick when they’re reinforced in the flow of real product work, not just in a time-boxed cohort. In my org, we’re leaning into on-demand learning paired with manager coaching to drive durable behavior change.
What leaders actually need to coach interviewing, assumption testing, and core discovery habits inside their orgs. I’ve found that empowered product teams thrive when leaders have lightweight coaching tools, practical prompts, and clear expectations for product trios. This is less about one-off training and more about building communities of practice where deliberate practice and feedback loops become routine.
Why training is shifting toward ongoing, leader-supported learning (and how AI will accelerate the shift). AI Strategy isn’t just about tools—it’s about learning systems. For LLMs for product managers to create leverage, we need eval-driven development, privacy-by-design, and clear guardrails. I’m building AI workflows that enable managers to review interviews, spot anti-patterns, and nudge teams toward better decisions—without replacing critical thinking.
Teresa’s move into paid subscriptions and why AI content doesn’t fit the classic “design once, run for years” course model. I see the same reality in my content roadmap: the half-life of AI guidance is short. That pushes us toward subscription models, tighter feedback loops, and a more adaptive go-to-market strategy for education products.
A sneak peek into the AI tools Teresa is building for discovery work—from interview coaching to near-ready interview snapshot generation. I’m particularly excited by tooling that scaffolds better interviews, sharpens assumption testing, and speeds up synthesis without skipping the human judgment step. These capabilities map directly to where I want my teams investing time: spending less energy on admin and more on learning from customers.
Petra’s plans for the year: community building with Product at Heart, a new product leadership email course, her Product Leadership Wheel, and workshops launching in Cairo. As someone who believes in conferences as high-quality “energy wells,” I’m inspired by how these programs create momentum for leaders who are upgrading their coaching muscles.
The role of conferences and retreats in staying grounded, inspired, and connected. I treat these gatherings as strategic resets—spaces to test ideas, confront blind spots, and deepen my network for future collaboration. The best outcomes often come from serendipitous hallway conversations and hands-on sessions where you can pressure test frameworks with peers.
How Teresa is staying on top of academic research (and why “synthetic users” aren’t ready for prime time). I agree: while synthetic data can be useful for scaffolding, it’s not a substitute for direct customer contact. Combine academic rigor with real-world interviewing and strong data governance—especially when operating under General Data Protection Regulation (GDPR).
The shared challenge of evaluating vendors and conference speakers making questionable AI claims. My heuristic: ask for clear problem statements, reproducible evaluations, grounded benchmarks, and a path to safe deployment. If a pitch can’t show measurable uplift or ignores compliance, it’s not ready for empowered product teams.
Key takeaways I’m carrying into 2026: delivery models matter; leaders need coaching tools, not just training; AI is reshaping how we teach and learn; experimentation is the theme of 2026; and community still energizes. That’s the blueprint I’m using to strengthen continuous discovery, refine our AI workflows, and sustain high standards in product management leadership.
What about you? How are you integrating AI workflows into your discovery practice, and what coaching tools are helping your managers reinforce the right habits? Share your approach—I’d love to learn what’s working in your context.
Resources & Links:
Follow Teresa Torres: https://ProductTalk.org
Follow Petra Wille: https://Petra-Wille.com
Teresa’s website: Product Talk
General Data Protection Regulation (GDPR)
Product Talk Academy
Deliberate Practice – ATP episode where Teresa talked about the ending live cohorts for Deep Dive classes
I’ve spent the last few years turning AI from an intriguing demo into an operational advantage, and the clearest wins come when we treat agents as productized workflows—not toys. In practice, that means aligning agentic AI to a sharp product strategy, instrumenting everything, and scaling what works across the organization.
Learn how companies like Replit are consolidating workflows, creating one-person departments, and building systems for scale with Amplitude
When I talk about agentic AI, I’m focused on outcomes: fewer handoffs, faster cycle times, and measurable uplift in activation, retention, and NPS. The most successful rollouts start with a specific job-to-be-done, translate it into clear AI workflows, and then iterate with a tight feedback loop between data, design, and engineering.
My implementation playbook is simple and disciplined. First, choose a high-friction workflow and define success upfront. Second, make the build vs buy call on the foundation model, orchestration layer, and connectors. Third, establish AI risk management and safeguards early—before scale amplifies errors. Finally, run small, eval-driven releases and promote what performs.
Instrumentation is where the leverage compounds. With Amplitude analytics as a unified analytics platform, I design purposeful events (agent intent, tool calls, resolution state, human handoff), map funnels from user input to agent outcome, and cohort users by context to pinpoint lift. This gives me an honest read on where agents help, where they hinder, and what to tune next.
The “one-person departments” concept isn’t about doing more with less at all costs; it’s about assembling a tight loop of product management leadership, data, and automation so one operator can own a business outcome end-to-end. An agent handles the repeatable work, while the human focuses on judgment, edge cases, and continuous improvement that compounds.
As we scale, I look for platform scalability patterns: shared tools and policies, reusable prompt libraries, standardized evaluation suites, and consistent governance. That structure keeps agent performance predictable while preserving speed, and it aligns beautifully with product-led growth when agents are embedded directly in the product experience.
If you’re starting now, begin with a single, valuable workflow. Instrument it thoroughly with Amplitude analytics, make decisions from the data you see—not the demos you remember—and expand only after you’ve proven uplift. Iteration beats ambition here: agentic AI rewards teams who measure relentlessly and scale only what truly works.
Inspired by this post on Amplitude – Perspectives.
Your AI support pilot can look successful while the service operation gets worse. The agent closes more conversations, but customers repeat themselves after escalation, risky cases receive plausible but incomplete answers, and human agents inherit a queue made almost entirely of exceptions.
If you own this transformation, your job is not to install an AI agent. It is to redesign how customer demand moves through knowledge, automation, human judgment, and product feedback. You also need to prove that a conversation marked resolved was actually resolved. That requires an operating model, not just a deployment plan.
Start with an operating thesis, not a deflection target
That gap matters. If you treat the program as a software rollout, the technology can reach production while ownership, escalation rules, quality controls, and performance expectations remain designed for a human-only queue. The result is automation sitting on top of an unchanged operation.
The interviews were drawn from Intercom customers or prospects and centered on Fin or similar products. They are useful directional evidence from teams close to this transition, but they are not a vendor-neutral census of every customer service organization. Your own demand, risk profile, knowledge quality, and channel mix should determine the design.
I would begin with a one-page transformation brief. Force the leadership team to complete these fields before discussing a broad rollout:
Customer promise: Which customer outcome will become faster, easier, or more reliable?
Eligible demand: Which intents, channels, languages, customer states, and account types may enter the AI workflow?
Decision boundary: What may the AI explain, recommend, decide, or execute? These are different levels of authority.
Human boundary: Which ambiguity, consequence, customer request, or system condition requires a human?
Business hypothesis: Which cost, capacity, service-level, or growth constraint should improve if the workflow succeeds?
Quality gates: Which measures must improve, and which failure measures must not regress?
Learning owner: Who converts failures into knowledge fixes, workflow changes, model evaluations, or product improvements?
Do not make deflection the customer promise. Deflection records the absence of a human interaction; it does not establish that the customer’s problem was solved. A better promise names the intended outcome, such as completing a defined action correctly or answering an eligible question from an approved source without avoidable repetition.
Scope automation using two dimensions: how repeatable the work is and what happens when the answer is wrong. A simple decision matrix prevents the team from treating every incoming conversation as equally automatable.
Work pattern
AI role
Human role
Release condition
Repeatable and low consequence
Resolve from approved knowledge or execute a reversible workflow
Review samples and handle defined exceptions
Correct resolution and reliable rollback are demonstrated
Repeatable and higher consequence
Retrieve, summarize, validate inputs, or draft
Approve the final answer or action
Authoritative sources, approval capture, and auditability are in place
Ambiguous and low consequence
Ask clarifying questions, categorize, and route
Resolve cases that remain ambiguous
The escalation reason and collected context are visible to the human
Ambiguous and higher consequence
Collect only the minimum safe context, then stop
Own judgment, communication, and action
Hard escalation rules have been tested and cannot be bypassed conversationally
Risk is contextual. The same intent may be routine for one account state and consequential for another. Eligibility therefore belongs in the workflow itself, using customer state, requested action, permissions, available knowledge, and tool health. It should not live only in a prompt that asks the model to be careful.
Redesign the full conversation, especially the human handoff
The unit of design should be one end-to-end customer intent. Do not stop at the AI response. Trace what happens from the first message through resolution, escalation, downstream action, and learning:
Define the intent and entry conditions. State what the customer is trying to accomplish and which signals make the conversation eligible.
Name the authoritative knowledge. Identify the policy, product data, account data, or workflow state required to answer correctly.
Specify permitted actions. Separate explaining a process, recommending an action, preparing an action, and executing it.
Design the handoff packet. Give the human the context needed to continue without interrogating the customer again.
Capture a failure reason. Every failed or escalated attempt should produce a category that can be assigned to an owner.
Close the learning loop. Route the failure to knowledge, conversation design, support operations, product, engineering, or governance.
The handoff is where many apparently successful deployments reveal their real cost. If the human receives only a transcript, the AI has transferred a conversation but not the work. The agent must reconstruct the goal, identify what the system already attempted, verify customer-provided facts, and decide whether any prior answer can be trusted.
A useful handoff contract should include:
The customer’s detected goal and the intent assigned to it.
The material facts the customer supplied, with no invented completion of missing fields.
The approved sources used to form the answer.
Any tools called, actions attempted, results returned, and side effects created.
The point of uncertainty or the exact escalation rule triggered.
The unresolved question or recommended next action for the human.
The relevant transcript, available for verification rather than presented as the only summary.
Test the handoff as a product experience. Give a human agent only the packet and the underlying conversation, then observe whether the case can continue without the customer repeating information. Track missing fields and unnecessary rework as workflow defects. Do not hide that effort inside average handle time.
Knowledge needs the same discipline. For each automated intent, name one canonical source, one owner, a review trigger, and a withdrawal path. If two approved pages disagree, the correct AI behavior is not to blend them into a smooth answer. It is to stop, disclose the limitation appropriately, and route the conflict to an owner.
The AI agent does not create knowledge debt, but it can expose and distribute that debt at much greater speed. A missing article, stale policy, ambiguous field, or inaccessible account state can produce thousands of superficially different conversations with the same root cause. Aggregate failures by root cause instead of editing individual answers forever.
Use a failure taxonomy that separates at least these problems: missing knowledge, stale knowledge, conflicting knowledge, retrieval failure, unsupported reasoning, policy-boundary failure, tool or integration failure, incorrect eligibility, poor conversation design, routing failure, and incomplete handoff. Each category should map to a named owner and a defined corrective action. Otherwise, quality review becomes a list of examples rather than an operating system for improvement.
Redesign jobs before you promise headcount savings
Workforce impact is real, but it is not uniform. Headcount or hiring changed in 27.71% of the 166 interviews, often through slower Tier 1 hiring, freezes, natural attrition, or reallocation. That is materially less common than workflow and responsibility changes. The safest conclusion is not that AI automatically removes a fixed percentage of support cost. It is that repetitive demand can shrink while new oversight, exception, knowledge, and optimization work grows.
Calculate net capacity rather than gross deflection. The practical equation is:
Net capacity released = human work correctly avoided – new review, exception, maintenance, and recovery work.
Count the whole system. Include time spent reviewing samples, investigating severe failures, maintaining knowledge, configuring workflows, testing releases, repairing integrations, managing escalations, and helping customers recover from wrong actions. Also separate capacity released from cash savings. A team may use capacity to absorb growth, improve response time, eliminate backlog, or take on higher-complexity work without reducing current payroll.
Role design should follow the new work, not the fashionable job titles. You may create an AI specialist, automation manager, or AI-agent owner, but the essential question is who owns each recurring decision:
Frontline specialists resolve nuanced cases, identify failure patterns, validate knowledge gaps, and contribute difficult conversations to evaluation sets.
Support managers manage the changing workload mix, coach exception handling, monitor capacity, and decide where human judgment adds value.
AI or automation owners configure behavior, maintain evaluations, control releases, monitor production, and coordinate rollback.
Quality owners define error severity, audit both automated and human resolutions, and make recurring failure visible.
Knowledge owners approve canonical content, resolve conflicts, and remove information that should no longer be used.
Product and engineering owners fix product defects, data gaps, and tool failures that support conversations repeatedly expose.
These are responsibilities, not necessarily separate positions. A smaller organization may combine them, but it should not leave them implicit. One person can hold several responsibilities; one critical responsibility cannot be owned by nobody.
Write decision rights alongside role descriptions. Specify who may expand eligible intents, approve a high-consequence workflow, publish knowledge, change a prompt or model, accept a known quality limitation, pause automation, and communicate a customer-impacting failure. An AI owner who is accountable for outcomes but cannot stop a release is not an owner.
Do not wait until automation is broadly deployed to explain this shift. Before changing staffing plans, show people the future queue, the new performance expectations, the skills they can build, and the paths available for redeployment. Vague assurances create uncertainty, while premature savings commitments force managers to defend a number before the operation has demonstrated sustainable quality.
Measure correct outcomes, not apparent automation
A conversation can be closed, contained, or deflected without being correct. That is why an automation dashboard cannot double as a transformation scorecard. I would make cost per correct resolution the economic anchor, then constrain it with customer-experience and severity guardrails.
Define correct resolution for every intent before launch. At minimum, it should mean that the customer received an accurate and complete answer or action, the applicable policy was followed, the workflow created no unintended side effect, and no avoidable human rescue or repeat contact occurred during an intent-appropriate observation period. The period may differ by intent; a question answered immediately and a downstream account action do not reveal failure on the same schedule.
Measure
Question it answers
Common trap
Eligible demand coverage
How much inbound demand falls inside a clearly approved scope?
Expanding eligibility merely to make automation look larger
AI attempt rate
How often did the AI engage eligible demand?
Counting an attempt as a successful outcome
Audited correct autonomous resolution
How often did sampled AI completions fully meet the intent definition without rescue?
Relying only on closure status or customer silence
Repeat or reopened contact
Did the customer return because the original issue remained unresolved?
Missing a repeat that arrives through another channel or wording
Handoff recovery
Can a human continue efficiently with accurate context?
Measuring routing speed while ignoring repeated questions and reconstruction work
Cost per correct resolution
What does a genuinely completed outcome cost across the whole system?
Excluding review, knowledge, tooling, maintenance, and recovery effort
Severity-weighted failure
How much customer or business consequence did errors create?
Allowing a high average accuracy to hide rare but serious failures
New-work burden
How much human effort did automation introduce?
Treating oversight and maintenance as free capacity
Keep the denominators explicit. Eligible demand coverage is eligible conversations divided by total inbound conversations. AI attempt rate uses eligible conversations as its denominator. Audited correct autonomous resolution should use reviewed AI-completed conversations, not every inbound contact. Mixing those denominators lets a team report a large percentage without showing how much demand was actually solved.
Audit with two sampling paths. Use a representative sample to estimate ordinary performance across intents, channels, languages, and customer states. Add targeted samples for high-consequence actions, new releases, known weak spots, tool failures, unusual escalations, and complaints. A purely random sample can miss rare failures that matter more than common harmless mistakes.
Define error severity before reviewers see the results. A wording issue, an incomplete answer, a wrong policy explanation, an unauthorized disclosure, and an incorrect account action should not contribute equally to one accuracy average. Severity should change the required response: monitor, correct knowledge, roll back a workflow, disable an action, or initiate the relevant incident process.
Maintain separate executive and operating views. The executive view should show eligible volume, audited correct resolution, customer outcome measures, cost per correct resolution, severe-failure trend, capacity released, and where that capacity went. The operating view should break performance down by intent, channel, language, customer state, workflow version, knowledge version, tool, failure category, and escalation reason.
Versioning is essential for diagnosis. Record the model, instructions, knowledge snapshot, workflow configuration, tool version, and eligibility rules associated with each resolved conversation. When several components change together, you may know performance moved without knowing why. Controlled rollouts or eligible-traffic holdouts can provide stronger evidence than a simple before-and-after comparison, especially when demand mix or seasonality is changing.
Set release thresholds before looking at a candidate’s results. The exact threshold should reflect the consequence of the intent and your current human baseline; there is no responsible universal number. The release decision should require sufficient audited quality, acceptable handoff recovery, no prohibited failure, functioning rollback, and an owner for every material defect that remains open.
Scale through evidence-gated stages
Do not scale on a calendar promise. Move when the workflow has produced enough evidence for its next level of authority. A useful sequence separates learning about the problem from granting the system permission to act.
Baseline the demand and draw the boundary
Start with the highest-volume and highest-consequence intents, but do not assume they belong in the same release. Build an inventory containing volume, current human effort, customer outcome, approved knowledge, data requirements, available actions, reversibility, failure consequence, escalation destination, and owner.
Create an evaluation set from real, appropriately handled historical conversations. Remove or protect sensitive data according to your controls. Include ordinary examples, ambiguous requests, missing information, policy conflicts, tool failures, customer requests for a human, and known edge cases. The gate for leaving this stage is not model quality. It is a testable definition of correct behavior and a clear boundary around what the AI must not do.
Run in observation or approval mode
Let the AI classify, retrieve, summarize, or draft while a human retains final authority. Compare its proposed outcome with the completed human outcome. Instrument the failure taxonomy, inspect whether the correct knowledge was available, and test the handoff packet with frontline agents.
Use this stage to repair the system around the model. Many failures will belong to missing content, conflicting policy, broken integrations, weak eligibility, or unclear product behavior. Prompt editing cannot fix an absent source of truth or an action the underlying system cannot perform reliably.
Grant controlled autonomy to bounded work
Begin with stable, low-consequence demand supported by authoritative knowledge and reversible workflows. Enforce eligibility outside the conversational instructions where possible. Keep hard escalation rules for uncertainty, missing data, customer preference, unavailable tools, policy conflicts, and prohibited actions.
Review production samples and targeted risk cases. Watch repeat contacts, human recovery work, severe errors, and changes in the composition of the human queue. A falling queue is not automatically good if the cases that remain take much longer or arrive with damaged customer trust.
Expand one meaningful dimension at a time
Add an intent, channel, language, customer state, or action only after defining how that dimension changes knowledge, evaluation, escalation, and consequence. Reusing a workflow in a new language is not just translation if policies, terminology, tone, or available support paths differ. Adding tool execution is not just a better answer; it grants the system operational authority.
Version each expansion and preserve rollback. If you need causal clarity, avoid changing the model, knowledge, tools, instructions, and eligibility rules in the same release. When simultaneous changes are unavoidable, label the release as a system change and evaluate the combined behavior rather than attributing the result to one component.
Institutionalize the operating model
Only after correct resolution and total workload remain durable should you change long-term staffing assumptions, performance management, budgets, or reporting lines. Update role charters, decision rights, quality routines, release governance, incident ownership, knowledge operations, and planning models together.
Give recurring AI failures a path into the product roadmap. If customers repeatedly ask because the interface is unclear, a workflow fails, or account state is hard to understand, automating the explanation may reduce service effort while preserving the root cause. The better product decision may be to remove the need for the conversation.
Key takeaways
Treat AI customer service as an operating-model transformation, because workflows and responsibilities change before most reporting structures do.
Automate bounded intents, not an undifferentiated share of tickets. Repeatability and consequence should determine the AI’s authority.
Design the human handoff as a product. A transcript without facts, actions, sources, uncertainty, and next steps transfers the queue but not the work.
Use audited correct resolution and cost per correct resolution as anchors. Attempts, closures, containment, and deflection are supporting events, not proof of value.
Calculate net capacity after review, maintenance, exception, and recovery work. Keep that separate from any claimed payroll saving.
Scale only when quality, severity, handoff, ownership, and rollback gates have been met for the next expansion.
Your next move can be small and consequential. Choose one recurring intent, complete the transformation brief, name its canonical knowledge owner, write the handoff contract, and define how you will audit correct resolution. If you cannot assign the knowledge, failure, and release decisions, do not automate the intent yet. Resolving that ownership gap is the first real step in the transformation.
Your AI agent demo works. Now the harder questions arrive: Which actions can it take, how will anyone know it helped, and who owns a bad decision? If those answers are deferred until launch, you do not yet have a product ready to scale. You have a capability looking for permission.
Your job as a product leader is to turn uncertain model behavior into a dependable operating system for one valuable task. That means designing the job, the workflow, the controls, the measurement, and the adoption path together. Model quality matters, but it cannot compensate for an undefined outcome, excessive access, weak tools, or a launch that asks users to trust what they cannot inspect or reverse.
Start with an operating contract, not an agent persona
Names such as sales agent, support copilot, or operations assistant are too broad to guide product decisions. They hide disagreements about what the system can see, what it can change, when it should stop, and what success means. Treating an agent as a product line with a narrow job, grounded data, tool access, and guardrails forces those disagreements into the open while they are still inexpensive to resolve.
Write an operating contract before debating models or interfaces. It should answer the following questions in language that product, engineering, operations, security, and the domain owner can all review:
Who is the user? Name the role performing the job, not a market segment. An account administrator and a support specialist may need different evidence, permissions, and explanations even when they use the same underlying model.
What event starts the job? Specify the observable trigger: a customer request arrives, a record enters an exception state, or a user asks for a particular action. A generic invitation to chat is not a job boundary.
What outcome counts as done? Define a state outside the conversation. The answer might be an approved response, a correctly updated record, a validated recommendation, or a complete handoff. A fluent message is output, not necessarily an outcome.
What evidence may the agent use? List permitted systems, required records, freshness requirements, and data the agent must not retrieve. If the task requires an authoritative record, make its absence a stop condition rather than an invitation to infer.
Which tools may it call? Separate read, draft, and write permissions. An agent that can inspect a record does not automatically need permission to change it, and permission to draft an action does not imply permission to execute it.
What constraints must always hold? Capture business rules, policy boundaries, approval requirements, and prohibited actions. Enforce these constraints in tool and application layers, not only in natural-language instructions.
When must it stop or escalate? Missing required evidence, conflicting records, unsupported requests, tool failures, and policy exceptions should lead to a defined fallback. The agent should not improvise its way around a boundary.
Who remains accountable? Name the owner who approves the contract, reviews failures, and decides whether autonomy can expand. Accountability cannot be assigned to the agent itself.
A compact job statement makes the contract easier to test:
When [trigger] occurs, help [user] achieve [observable outcome] using [approved evidence and tools]. If [stop condition] occurs, hand off to [role] with [required context].
For example, a support agent might retrieve an approved knowledge record and relevant account facts, prepare a response, and stop when identity, policy, or account data is unresolved. Its handoff would include the customer’s request, the evidence retrieved, the steps attempted, and the exact question requiring a specialist. That is a testable product definition. Build a support agent is not.
Add a negative scope as well. State what the agent will not do in the current release, even if the model appears capable of doing it. This keeps a successful pilot from quietly becoming authorization for unrelated work.
The final test is simple: can two reviewers inspect the same run and agree whether the job was completed within the contract? If they need to debate whether the answer merely sounded reasonable, the definition of done is still too vague.
Design the successful path and the failure path as an explicit sequence:
Retrieve authorized evidence. Fetch only the records relevant to the job. Preserve record identifiers, versions, and freshness so the result can be inspected later.
Construct minimal task state. Carry the user’s identity, requested outcome, validated facts, previous tool results, pending approvals, and unresolved questions. Do not treat an ever-growing chat transcript as the system of record.
Choose from allowed actions. Give the model a constrained set of tools and make unavailable actions genuinely unavailable. A prompt that says do not call a privileged endpoint is not access control.
Validate tool inputs. Use typed schemas, required fields, enumerated values where appropriate, and server-side authorization. Reject malformed or unauthorized calls before they reach the underlying system.
Validate the resulting state. Check deterministic business rules after execution. A successful API response only proves that the call ran; it does not prove that the user’s job was completed correctly.
Finish, recover, or hand off. Return an accepted outcome, retry only when retrying is safe, or create the handoff package specified in the operating contract.
Tool quality deserves product attention. Each consequential tool should expose the smallest permission needed, return machine-readable errors, support a preview when possible, and make repeated requests safe where the underlying operation permits it. Reversible operations need a tested undo path. Irreversible operations need tighter authorization and should not be made safe merely by adding another sentence to the prompt.
Context also needs a budget based on relevance, not on the maximum number of tokens the model accepts. Rank evidence by authority and usefulness. Remove unrelated history. Distinguish verified records from user claims and model-generated summaries. When two authoritative records conflict, preserve the conflict and route it through the stop condition instead of blending them into a plausible answer.
Build the evaluation set before the launch plan
Your evaluation set is the executable version of the operating contract. It should represent the situations that matter to the job, including conditions in which the correct behavior is to refuse, ask for information, or escalate.
Scenario class
What the evaluation should verify
Normal path
The agent retrieves the required evidence, selects the correct tool, satisfies the acceptance criteria, and records a complete result.
Ambiguous request
The agent asks for the missing fact or offers bounded choices instead of assuming the user’s intent.
Missing or stale evidence
The workflow stops, refreshes through an approved path, or escalates according to the contract.
Tool failure
The agent does not claim success, duplicate a consequential action, or lose the task state needed for recovery.
Policy boundary
The prohibited call is blocked by the system, the response explains the available path, and the event is auditable.
Human handoff
The receiving person gets the request, relevant evidence, attempted actions, unresolved issue, and recommended next step.
Score the dimensions separately. A single average can hide the failure that matters most.
Outcome correctness: Did the external result meet the job’s acceptance criteria?
Grounding: Did the response use the required evidence without inventing unsupported facts?
Tool behavior: Were the correct tool, arguments, order, and authorization used?
Policy compliance: Did every prohibited or approval-gated action remain inside its boundary?
Recovery: Did the workflow handle missing data, timeouts, and partial failures without misrepresenting the result?
Handoff quality: Could the receiving person continue without reconstructing the entire run?
Use deterministic assertions wherever the expected state can be checked directly. Use domain review for judgment that depends on policy or professional context. Model-based evaluators can help classify or prioritize a larger sample, but they should not become the only judge of a high-consequence action.
Run scripted evaluations whenever the model, prompt, retrieval logic, tool schema, policy, or orchestration changes. Sample live runs after release to find failure patterns the fixed set does not yet represent, subject to your data-access and retention rules. Add confirmed failures back into the regression set. That is how eval-driven development turns observed behavior into a tighter product.
Select the model after this evaluation loop exists. Compare candidates on the acceptance criteria, latency, operating cost, and operational constraints of the job. The right model is the least complex option that clears the required bar with the complete workflow around it. A model swap should be one testable hypothesis among retrieval, context, tool, state, and prompt changes, not the automatic response to erratic behavior.
Govern autonomy at the action boundary
Governance becomes practical when you classify what the agent may do, not how intelligent it appears. The important distinction is the consequence of the next action: whether it changes state, whether the change can be reversed, and who bears the cost of an error.
Action class
Typical behavior
Default product control
Advise
Summarizes evidence or recommends a next step without changing system state.
Show the supporting evidence and let the user ignore, revise, or escalate the recommendation.
Draft
Creates an editable response, plan, or proposed update that has not been sent or committed.
Require review before external effect. Capture material edits and rejection reasons as feedback.
Execute a reversible action
Changes a record or starts a bounded workflow with a reliable recovery path.
Begin with a preview and explicit approval. Enforce scope in the API, record the action, and make undo visible.
Execute a consequential action
Creates an irreversible, financial, regulatory, security, or substantial customer impact.
Keep a qualified human decision-maker in the path unless the organization has explicitly approved a narrower control model. The agent can assemble evidence and prepare the action without owning the decision.
Do not borrow one accuracy threshold for all four classes. A summarization defect and an unauthorized payment are not interchangeable errors. Set release criteria by action class, and report prohibited-action failures separately rather than averaging them together with low-consequence quality issues.
Human review only reduces risk when the reviewer can make an informed decision. A confirmation button attached to a vague summary creates approval theater. The review interface should show:
The exact action that will occur and the system it will affect.
The evidence used, including record identifiers or other traceable references.
Any missing, stale, or conflicting information.
The expected side effects and whether the action can be reversed.
Clear options to approve, edit, reject, or escalate.
For a handoff, replace approve with a receiving workflow. The person taking over needs a concise task summary, the user’s original intent, the evidence already checked, tool results, the reason automation stopped, and the next decision. Measuring whether that package is usable is more valuable than celebrating a low handoff rate.
Enforcement belongs at the tool boundary. Authenticate the user and agent, authorize each operation, validate inputs, limit accessible records, and block disallowed transitions on the server. Natural-language instructions can guide behavior, but they are not a substitute for permissions, policy checks, or transaction controls.
Keep an audit record proportionate to the risk. For a consequential run, that commonly includes the requesting identity, agent and configuration version, evidence identifiers, tool calls and results, approval decision, final state, and any reversal or escalation. Do not log raw prompts, private records, or retrieved content by default merely because they may be useful later. Decide what is necessary, who can access it, and how long it should be retained as part of AI risk management and data governance.
Assign human ownership across the operating system. Product owns the target outcome and adoption decision. A domain owner approves acceptance criteria and policy interpretation. Engineering owns tool reliability and recovery. Security and privacy owners approve data and access controls. Operations owns monitoring, handoffs, and incident response. One person may cover more than one role, but no responsibility should disappear into the phrase the agent decided.
Governance review should be triggered by meaningful change, not only by a launch meeting. Revisit the contract when you change the model, retrieval source, tool schema, permission, policy, action class, or target user. Review it again when live behavior reveals a new failure mode. That keeps governance attached to the product lifecycle instead of turning it into a document that goes stale after approval.
Measure the entire path from opportunity to outcome
Start the funnel before the conversation. If you count only people who already opened the agent, you cannot distinguish poor discovery from poor execution. Define an eligible opportunity for the specific job, then instrument the path through completion.
agent_opportunity_detected: The product can identify that the target job is present for an eligible user.
agent_offer_exposed: The relevant entry point or contextual suggestion is shown.
agent_invoked: The user starts the workflow or an authorized trigger starts it on the user’s behalf.
agent_action_proposed: The workflow produces a recommendation, draft, or preview inside the operating contract.
agent_approval_resolved: The proposed action is approved, edited, rejected, or escalated where review applies.
agent_task_completed: The external acceptance criteria are satisfied and the final state is recorded.
agent_outcome_reversed: The result is undone, reopened, corrected, or otherwise found not to be durable.
The names are less important than consistent semantics. Record the job type, user role, action class, model and workflow version, tool result, and final disposition. Use identifiers and controlled classifications where possible instead of copying sensitive prompt or retrieved content into analytics.
Metric
Useful definition
Common misreading
Activation
Eligible users who complete their first accepted valuable outcome divided by eligible users exposed, for a named cohort and measurement window.
Counting a first prompt or first response as activation even when no job was completed.
Task completion
Eligible initiated tasks that meet the external acceptance criteria divided by eligible initiated tasks.
Using a model’s claim of completion or a successful API call as proof of success.
Containment
Eligible tasks completed without human takeover divided by eligible tasks started, paired with quality and later correction signals.
Rewarding fewer handoffs even when the agent should have escalated.
Time to value
Elapsed time from the eligible trigger to an accepted outcome, including waiting for review when review is part of the workflow.
Measuring response latency while ignoring the rest of the job.
Acceptance and editing
Results accepted as presented, accepted after a material edit, rejected, or escalated. Define material for the job.
Treating any click on approve as equal, regardless of the correction required before approval.
Handoff quality
Handoffs containing the required context and accepted as usable by the receiving role divided by all handoffs.
Viewing every handoff as failure instead of distinguishing correct escalation from avoidable escalation.
Cost per successful outcome
Variable model, tool, infrastructure, and human-review costs divided by accepted completed outcomes.
Optimizing token cost while ignoring rework, review time, or failed attempts.
Risk signals
Blocked prohibited calls, unauthorized attempts, reversals, policy escalations, and incidents, reported as counts and against the relevant opportunity denominator.
Combining materially different events into one average quality score.
Segment these metrics by job, user role, action class, workflow version, tool, and risk class. An overall completion rate can improve while a high-consequence segment gets worse. Version-level segmentation also tells you whether a prompt, retrieval, model, or interface change actually altered behavior.
Pair leading signals with durable outcomes. Edits, rejection, undo, escalation, and approval time can expose friction quickly. Repeated successful use, lower rework, and movement in the target business outcome tell you whether the product is creating lasting value. An increase in escalation is not automatically bad: it may mean the control became easier to use. Inspect whether the escalation was correct and whether the receiving person could act on it.
Let evidence earn each expansion of autonomy
Adoption is a behavior-change problem. Users need to notice the agent at the moment the job occurs, understand its boundary, inspect its work, and recover when it is wrong. A generic product tour may create awareness, but it does not establish trust in a consequential workflow.
Move through deployment modes according to evidence rather than a predetermined calendar:
Shadow mode: Run the workflow without exposing a result or changing state. Compare its proposed outcome with the accepted human outcome and use disagreements to improve the contract and evaluations.
Assisted mode: Let the user request a recommendation or editable draft. Make the evidence and limitations visible, and collect structured edit and rejection reasons.
Approved execution: Show the exact proposed change and require explicit confirmation before the tool commits it. Test authorization, audit, recovery, and handoff paths under live operating conditions.
Bounded autonomy: Allow execution only for the job, users, data, conditions, and limits approved in the operating contract. Continue monitoring outcomes and preserve a kill switch, rollback path, and accountable operator.
Advancement should depend on the evaluation suite, live outcome quality, tool reliability, policy compliance, recovery readiness, and the receiving team’s ability to handle escalations. If the evidence is mixed, narrow the action class or eligible population. Do not compensate for unresolved risk by making the prompt longer.
The interface should answer the user’s practical questions before asking for trust:
Why is the agent appearing at this moment?
What task can it complete, and what remains the user’s responsibility?
Which records or evidence will it use?
What will change if the user approves?
Can the result be edited or undone?
Where does the task go if the agent cannot complete it?
Surface the agent inside the existing workflow when the eligible job appears. State the action in task language, such as prepare this response or verify and update this record, rather than ask AI anything. Keep preview, edit, reject, undo, and escalation controls visible at the decision point. Contextual guidance is most useful when it removes a known piece of friction, not when it explains AI in general.
Use experiments for choices that are safe to vary: entry-point placement, explanation copy, prompt starters, preview layout, or the order of optional steps. Do not A/B test away required approvals, access controls, or safety boundaries. Time-to-value, task completion, edits, undo patterns, and escalation requests provide a more useful adoption picture than raw message volume.
Define activation as the first accepted outcome, not the first interaction. For a drafting workflow, that may be the first reviewed artifact that is actually used. For an operations workflow, it may be the first verified state change. The exact event should match the operating contract, and retention should measure return when the same job recurs rather than habitual chatting that produces no business result.
Key takeaways: use this launch gate
Before exposing an agent to production data or expanding its autonomy, require a clear yes to each question:
Can the job be stated with one user, one trigger, one observable outcome, and explicit stop conditions?
Are read, draft, and write permissions separated and enforced outside the prompt?
Does the evaluation set cover ambiguity, missing evidence, tool failure, policy boundaries, and handoff behavior?
Can every consequential tool validate authorization, return a clear result, and recover safely where recovery is possible?
Is the action classified by consequence and reversibility, with an appropriate approval path?
Can a reviewer see the evidence, proposed effect, missing information, and recovery option before approving?
Is there a named owner for outcomes, policy interpretation, monitoring, escalation, and incident response?
Can analytics connect an eligible opportunity to an accepted outcome, later correction, cost, and risk?
Can the product be narrowed, paused, or rolled back without waiting for a new model release?
A no does not have to stop all learning. It should stop the unsafe action. Move the pilot to shadow, advisory, or draft mode while the missing control is built.
For your next roadmap review, bring four artifacts instead of another open-ended demo: the operating contract, the evaluation matrix, the action classification, and the instrumented outcome funnel. Ship the smallest permissioned workflow that can prove value. Let observed outcomes, not confidence in the demo, earn the next level of autonomy.
