To truly transform with AI, I’ve learned it’s never just about the technology—it’s about redesigning how we work. The teams that win don’t bolt AI on; they re-architect around it. That means rethinking roles, workflows, and governance to build a system that sustains and improves AI performance over time.
In The 2026 Customer Service Transformation Report, teams at every stage of maturity describe human agents taking on more proactive work—training AI systems, handling the hardest queries, and owning tasks that demand judgment. Job descriptions are shifting, too, with many organizations explicitly adding AI-related responsibilities.
I’m also seeing a clear rise in dedicated AI specialists. Conversation analysts, knowledge managers, and AI operations leads are fast becoming standard. For support professionals, this opens new, higher-leverage career paths—and creates a talent pipeline that blends service excellence, data fluency, and product thinking.
Support once centered on queue-level activity—ticket triage, routing, translations, and answering FAQs. Now, as AI handles more frontline interactions, our human roles are moving up the stack toward optimization, oversight, and continuous improvement.
According to the latest research, 45% of teams report updating job descriptions to include AI-related responsibilities, with 40% saying their human agents are now more focused on training AI systems. Another 27% report that human agents primarily handle the most complex escalations and edge cases, while a quarter say agents are doing more consultative and strategic work.
Even at the initial deployment stage, 16% of teams report spending less time handling support volume since implementing AI – and among teams who’ve reached maturity, that figure rises to 28%.
When Intercom’s Research, Analytics & Data Science (RAD) team interviewed 166 of our customers, similar themes emerged. Nearly all participants (≈95%) reported meaningful workflow changes, with manual processes being handled by AI, and humans focusing more on monitoring or fine-tuning AI outputs. Eighty-three percent of participants also reported seeing their team’s roles and responsibilities change to become more strategic and supervisory in nature.
AI is reshaping support teams: organizations are adding conversation analysts (32%), knowledge managers (30%), AI operations leads (28%), and support automation specialists (24%). Just 8% report no new AI roles.
It’s not just the work that’s evolving; organizational structures are, too. Some teams are reallocating existing talent into AI-focused roles; others are hiring entirely new skill sets. Many of the most common job titles in this space didn’t exist two years ago.
Consider a Senior AI Knowledge Manager, Beth-Ann Sher, who transitioned from a help center manager role. Like many careers transformed by AI, her work evolved from administrative to strategic. Instead of focusing solely on customer-facing, self-serve content, her mandate expanded to designing and optimizing knowledge inputs that directly improve AI Agent Fin’s performance—work that materially lifts resolution rates.
Or look at a Senior Conversation Designer, Fred Walton, hired specifically for an AI-first function. He focuses on frictionless customer journeys with Fin, smoothing handoffs between automation and human support while keeping customer satisfaction front and center—hallmarks of mature AI workflows and conversation design.
In high-performing organizations, roles like these typically sit within a dedicated AI support team under senior CS leadership. Clear ownership and accountability for AI performance is critical; without it, optimization stalls and trust erodes.
These shifts aren’t isolated. Take Robb Clarke from RB2B. He went from Head of Technical Operations to Head of AI. With Fin, his focus moved from repetitive support questions to managing knowledge and improving the system behind it—freeing him to be proactive about product improvements and fix issues before they hit customers.
Or consider Eric Broulette from Bloomerang, a support leader who leaned into AI and became the VP of Support and Education. By deploying Fin, his team found breathing room to invest in what’s next. Agents stepped into new roles, contributed to meaningful projects, and built skills that had previously felt out of reach. As Eric puts it: “Do not wait to embrace AI. It will unlock more career growth for your teams than you can imagine.”
Leaders are racing ahead with real AI in support. Explore the 2026 Customer Service Transformation Report to see where deployment is stalling, benchmark your team, and get practical steps to scale automation that delights.
Bringing AI into support will eventually change every agent’s day-to-day work. For leaders at the start of the journey, that can feel daunting. My perspective: the most successful teams treat this as an operating model shift, not a tooling rollout—anchored in AI Strategy, governance, and continuous improvement.
Be transparent about what’s changing, why it matters, and how success will be measured. Define how AI performance will be evaluated (resolution rate, containment, CSAT impact), empower agents to train and improve the system, and communicate how responsibilities will evolve. When teams help build the AI, they’re invested in making it great.
Here’s the playbook I rely on with support leaders: First, reset expectations about time allocation—less time in the queue, more time improving the AI system that serves the queue. Second, elevate knowledge management as a core capability. Prioritize content quality and coverage for your AI Agent, and carve out dedicated “out of the inbox” time so every agent contributes. Third, keep outcome metrics—especially resolution rate—front and center. It gives the team a north star for experimentation and iteration.
Scaling AI is as much a people challenge as it is a technology challenge. As automation takes on more work, support roles become more proactive, strategic, and cross-functional—even early in the journey. Responsibilities expand, new roles emerge, and team structures adapt to concentrate on and amplify AI performance. In the process, support careers are transformed.
If you’re leading this shift, now’s the moment to reimagine your operating model: clarify ownership, invest in knowledge and conversation design, adopt eval-driven development, and build the muscle for continuous improvement. That’s how you move from tickets to strategy—and unlock compounding value for your customers, your business, and your teams.
I’ve led product organizations through multiple growth chapters, and the pattern is always the same: the tighter the alignment between product, sales, and marketing, the faster you scale. Reflecting on the journey of Chris Degnan — the first sales hire at Snowflake who spent 11 years helping scale the company from zero to $3.5 billion in revenue as its CRO while partnering with four different CEOs — I’m struck by how consistently the fundamentals win. The playbook isn’t mysterious; it’s disciplined execution, ruthless clarity, and a go-to-market strategy that matures with each revenue stage.
At $10M ARR, the CRO role is hands-on and founder-adjacent. You’re close to the product, running point on key deals, pressure-testing messaging, and building credibility with early customers. By $1B+, the job is organization design: segmentation, international expansion, forecast accuracy, enablement, recruiting, and cross-functional orchestration. The shift is from deal quarterback to system architect — standing up repeatable, auditable processes that produce reliable outcomes across regions, segments, and industries.
Sales leaders who can’t sell the product themselves don’t last. Whether you sit in product management leadership or run the field, you need to master discovery, speak the customer’s language, and translate use cases into value. That also means getting fluent in solutions engineering — understanding integrations, data paths, security, and the operational realities buyers live with. I’ve found this hands-on competence to be the fastest way to earn trust internally and externally, and to keep product strategy grounded in market truth.
The MEDDIC methodology is the foundation for every durable sales org — and, frankly, a founder’s best insurance policy. MEDDIC forces alignment on qualification criteria, from Metrics to Economic Buyer to Decision Process and Identifying Pain. When product and sales both operate to this standard, roadmap bets improve, marketing targets sharpen, and win rates climb. It’s not paperwork; it’s pattern recognition at scale.
High-output CROs obsess over the right numbers. Pipeline coverage by segment and stage; conversion rates through each gate; sales cycle length by use case; average selling price and discount discipline; consumption predictability when you have consumption SaaS pricing; and post-sale expansion velocity. The art is deciding which two or three metrics are the organization’s true north at a given stage — then designing enablement, compensation, and operating cadence around them.
On operating cadence, the week in the life at scale is predictable for a reason. Forecast reviews that surface risk early. Deal reviews that coach to MEDDIC depth, not activity theater. Enablement blocks to uplevel managers and ICs. Recruiting time — always. Customer roadshows to refine value proposition and product positioning. And standing meetings with product, marketing, and finance to keep the GTM motion, roadmap, and unit economics in sync.
Compensation is a force multiplier or a silent saboteur. Keep it simple, consistent, and aligned to the current motion. Early on, weight new logo acquisition and land quality; as you mature, balance new business with expansion, multi-product adoption, and healthy consumption. Guardrails matter — cap over-discounting, reward multi-threading, and avoid plans that create end-of-quarter cliff behavior. The best plans reinforce the behaviors you want your culture to scale.
Technical CEOs often underestimate how much narrative, segmentation, and process discipline great GTM requires. The handoff from founder-led GTM to sales-led growth is where many teams stall. My rule: prove one repeatable motion in one segment before you add complexity. Codify the buyer’s journey, instrument the funnel, and make sure product strategy and enablement move in lockstep.
Culture sets the ceiling. You have to find the fakers, manage-uppers, and passengers quickly — people who look busy but don’t move pipeline, who talk big but avoid accountability, or who ride the momentum of others. The mantra that has saved me endless time: “When there’s doubt, there’s no doubt”. Move fast, but with humanity; be clear on expectations, coach hard, and when it’s not a fit, make the change before the team does it for you.
Feedback is the operating system of a high-performing org. Leaders at every level need to be coachable — on message discipline, on forecast rigor, on how they develop people. I’ve benefited from straight talkers who hold a high bar, and I try to pay that forward. The fastest way to raise organizational IQ is to institutionalize feedback loops across sales, product, and marketing — from post-mortems to win-loss analysis to field-sourced roadmap reviews.
What separates exceptional ICs from the rest? Hunger, intellectual honesty, and a builder’s mindset. They qualify hard, align to customer metrics early, multi-thread to power and value, and partner tightly with solutions engineering. They don’t hide from gaps; they surface them, and they know exactly what they need from product, marketing, and leadership to win.
Executive teams that scale share a few traits: crisp segmentation decisions, single-threaded ownership for outcomes, and healthy conflict that resolves into commitment. Dysfunction, by contrast, looks like metrics roulette, opaque decision-making, and a tolerance for exceptions that become precedent. Make the rules explicit and the exceptions rare.
Leaders like Frank Slootman have popularized intensity, speed, and focus — and there’s real power there when paired with clarity and data. The lesson I carry forward: move fast on people decisions, keep the message simple, and measure what matters. Equally important is knowing where that approach can backfire — when speed outruns learning, or when pressure erodes cross-functional trust. The best operators balance urgency with systems thinking.
Most AI companies will face a go-to-market reckoning. Model quality won’t save a weak motion. The winners will articulate a hard-nosed ROI, solve specific workflow pain, address data governance and security head-on, and show measurable lift — not demo dazzle. In other words, the same fundamentals apply; the stakes and scrutiny are just higher.
If you’re building or rebuilding your revenue engine, start here: define your ideal customer profile and segmentation with ruthless clarity; adopt MEDDIC and teach it across product and sales; align compensation to today’s motion; instrument the funnel and inspect it weekly; and cultivate a culture where feedback is fuel. Do that, and the path from $0 to $3.5B stops feeling like mythology — and starts looking like math.
Today, I’m excited to share 12 major updates to Fin’s Procedures and Simulations—the foundation that lets Fin handle complex work while keeping teams fully in control of the customer experience.
In my work building AI workflows with product and support leaders, I’ve seen how the right blend of natural language instructions, deterministic controls, and fully agentic behavior turns Fin into a reliable problem solver. Procedures make this blend possible by enabling Fin to act like a human—yet with the repeatability and governance of software. Simulations then let us test those complex Procedures at scale before they reach customers, so we can deploy with confidence.
Together, these capabilities make Fin self-manageable, transparent, and ready for genuinely complex work.
Here’s what’s new at a glance: we’ve made Procedures easier to build and maintain; enhanced deterministic controls for precision and policy compliance; expanded agentic behavior so Fin can adapt in real time; and delivered more powerful Simulations to validate end-to-end workflows before go-live.
Why did we build this? Many teams see early AI gains in speed, coverage, and cost to serve—but then hit a ceiling. They keep AI confined to simple automation and information retrieval, rather than setting it up to handle the nuanced, multi-step workflows they still trust to humans. We designed Procedures and Simulations to remove that ceiling, so teams can confidently set up, govern, and iterate on complex AI workflows without bottlenecks.
Follow the AI lifecycle as it cycles from Analyze to Train to Test to Deploy. This streamlined loop spotlights the TRAIN phase, underscoring faster iteration and feedback that power more capable procedures and realistic simulations.
We also heard that teams needed an easy way to connect data so Fin could reliably check customer status or eligibility and then take action. And they didn’t want to route through engineering every time they needed to create or amend logic for mid-conversation decisions. Procedures combines natural language instructions and intuitive data connector setups. You tell Fin in your own words how you want it to behave, and you’ll be guided through creating conditional steps so Fin will react consistently, with the option to add in any code snippets for circumstances where absolute precision is required. Once you build one Procedure, we believe you’ll want to build several, so Fin will constantly read the conversation it’s in to ensure it’s following the most relevant Procedure, and jump to a more relevant one if the user intent changes.
I know that taking something like this live the first time can feel like a leap of faith. That’s exactly why we built Simulations—to test Procedures comprehensively, uncover edge cases, and launch with confidence.
Reaching mature deployment takes a deliberate, ongoing commitment to training workflows, validating them before deployment, measuring performance in production, and refining them over time. At Intercom, we call this the Fin Flywheel: train, test, deploy, analyze. Procedures form the foundation of the train stage, and Simulations make the test stage reliable at scale. Together, they enable Fin to handle complex work, and teams to stay in control of it.
Procedures: Define exactly how Fin handles complex work. With Procedures, I can set Fin up to resolve complex, time-consuming queries that require multiple steps or business logic. Fin follows standard operating procedures and applies sound judgment—just like a seasoned teammate—so even complicated queries are resolved in controllable, predictable ways.
A snapshot of the Procedures builder in action, mapping a clear path for handling damaged food orders while letting teams train Fin on examples, target channels, quickly test updates, and publish with Set live.
Procedures combine three powerful elements. First, natural language instructions. You write a Procedure in plain language, just like documenting a process for a new teammate. You can paste in your existing SOPs, write from scratch, or let AI draft them for you, then iterate yourself.
What’s new: Draft Procedures with AI. Share an outline of your process and Fin drafts a complete Procedure using your conversation history, knowledge hub content, and relevant data. If additional context is needed, it prompts you with clarifying questions to make sure the Procedure is thorough and tailored to your use case, significantly reducing setup time. For example: if you’re creating a refund workflow, the system can draft conditional paths for eligibility, approval thresholds, and verification steps based on your historical cases and policies.
What’s new: Break complex workflows into Sub-procedures. Write a process once and reference it across multiple Procedures by breaking it down into reusable steps, called Sub-procedures. This makes workflows easier to read, faster to build, and simpler to maintain as things change.
Second, deterministic controls. Natural language is flexible, but some steps need to be exact. You can layer in deterministic controls where precision matters, starting with a fully natural language Procedure and introducing structure gradually where it adds value: conditional steps (branching logic) to handle decision points so Fin’s behavior is consistent and predictable; data connectors so Fin can pull information from your tools or take actions automatically; code snippets for when absolute accuracy is essential; and checkpoints to pause for approval or hand off to a teammate.
Fin demonstrates structured troubleshooting: a transaction dispute flow with eligibility checks, clear IF/ELSE steps, and quick Data Connector actions like freezing a card or pulling invoices, streamlining complex support tasks.
What’s new: Instruct Fin to read specific content from your knowledge hub. You can set clear rules for Fin to reference a specific policy or article from your knowledge hub in defined situations so Fin always surfaces the right context in a conversation.
What’s new: Explicit Procedure switching under defined conditions. You can set rules that deterministically trigger a switch to a different Procedure, for example, escalating to a complaints Procedure if specific risk signals are detected mid-conversation.
What’s new: Internal notes for human handoffs. When Fin hands off to a teammate, it can now include internal notes with relevant context so the person picking up the conversation knows exactly what happened and what needs to happen next.
Third, fully agentic behavior. Because real conversations rarely follow the happy path, Procedures let Fin reason through what’s happening and adapt—jumping to the right step or switching Procedures entirely if a customer changes their mind or the issue shifts.
Procedures and Simulations in action: Fin rehearses a food order damage scenario, confirming details and progressing through each trigger. Teams validate complex flows end to end as steps turn green and outcomes are tracked.
What’s new: Automatic Procedure switching. If a customer starts in a billing workflow but then asks about cancelling their subscription, Fin transitions to the relevant Procedure without forcing the customer to restart.
What’s new: Structured data extraction from uploaded files. Fin can now extract structured data directly from PDFs and images uploaded by customers—like invoices, forms, or receipts—and use that data within the conversation. Customers don’t have to copy and paste or repeat themselves.
As MONY Group put it:
“ If a customer starts down one path but their issue turns out to be something else entirely, Fin adapts seamlessly – no more getting stuck in loops or forcing customers into the wrong workflow. ”
Simulations help teams rehearse procedures and verify outcomes before going live. Run all tests or launch a new one to ensure Fin handles tricky customer scenarios—from damage confirmation to refunds and missing subscriptions.
The result is a conversation that feels fluid, but always follows your intended rules.
Making complexity easier to manage is just as important as unlocking new capabilities. Beyond the core updates, we’ve focused on creation, governance, and scale—while keeping ownership with your team.
What’s new: Improved instruction authoring. We’ve made it easier to write, edit, and structure Procedures, so building and updating them takes less time and requires less effort.
What’s new: Reporting on when Procedures trigger, resolve, or hand off. You can now track how Procedures are performing directly within the Procedures UI, seeing exactly when they trigger, when they resolve, and when they hand off to a teammate. This visibility helps you spot issues early and improve over time.
Customer stories from Raylo and Mony Group show how Fin now resolves payment issues and complex claims in-chat, checks account data via APIs, and lifts CSAT to about 94%, highlighting the impact of Procedures and Simulations.
Simulations: Test complex workflows at scale before they reach customers. Simulations let you validate how Procedures will perform before anything goes live, and continuously revalidate as things change. Deploying complex AI can feel uncertain; Simulations remove that uncertainty so you can launch with confidence and iterate safely.
You can simulate full conversations. For any Procedure, choose a user or customer segment and run a complete, multi-turn simulated conversation. You see every step Fin takes, how it applies your rules, reasons through decisions, and where it passes or fails—giving you the observability to debug and fix issues before they ever reach customers.
What’s new: Upload images for richer testing. Simulations now support image uploads, so you can test workflows that involve receipts, invoices, or forms—the same inputs your customers actually send.
What’s new: Clearer visibility into Fin’s reasoning. You can now see exactly how Fin is thinking through each step of a Simulation, making it easier to understand behavior, catch unexpected decisions, and refine Procedures with confidence.
You can also use AI to create, store, and rerun tests. Writing test coverage manually doesn’t scale. Fin’s AI Assistant generates Simulations directly from your Procedures, suggesting realistic edge cases like partial refund disputes, missing invoice uploads, or no subscription found, so you can expand coverage without expanding overhead. All the Simulations you create are stored in a central library. When a product changes, a policy updates, or a Procedure is edited, hit “run all” to instantly check whether anything has regressed. This applies the same rigor to AI automation that engineering teams bring to software testing.
What’s new: AI-suggested Simulations. You can now use AI to generate a full set of Simulations from any Procedure. The AI Assistant suggests realistic variations based on your workflow, so you can build comprehensive test coverage fast.
Customers are already seeing this in production. “Fin can now handle payment-related queries that were never possible before… The impact on CSAT and overall CX has been pretty shocking – the Payment Information procedure CSAT is sitting at ~94%, and CX score is significantly higher than our average.” – Raylo
“Procedures have fundamentally changed what we can achieve with Fin. Previously, complex processes like cashback claim investigations could only be handled through a static form on our website… Now, Fin can handle these sophisticated scenarios in real-time within the conversation itself. It checks account information via API calls, makes complex decisions, and guides customers through the entire claims process dynamically.” – MONY Group
Procedures and Simulations are available now. I’m eager to see how teams use these updates to scale agentic AI, deliver faster resolutions, and raise the bar for customer experience—without sacrificing control, compliance, or quality.
Where is the true boundary between product and engineering—and what happens when it gets blurry? I’ve led and coached teams through this question many times, and I’ve learned that clarity here isn’t just a nice-to-have; it’s foundational to quality, velocity, and team health.
I’ve seen well-intentioned product managers step in to “help” by taking ownership of bug triage, tech debt prioritization, or even system architecture. At first, it feels productive. Over time, it creates role confusion, slows decision-making, and burns out PMs—while paradoxically lowering engineering quality. The “CEO of the product” myth and legacy IT, project-based mindsets are usually at the root. Treating engineers as “order takers” breaks down in evergreen product environments.
The healthiest collaboration model is simple and disciplined: The product trio owns the “what”; engineering owns the “how”. Product managers are not people managers for engineers—and shouldn’t be accountable for engineering quality. Our job is to frame the problem, align on outcomes, and continuously discover value with customers—not to supervise technical execution.
If quality is a problem, the solution is escalating and fixing the system, not managing individual bugs. In practice, that means surfacing patterns and elevating them to engineering leadership, who can address root causes—staffing, skills, code health, CI/CD gaps, observability, or process design—rather than asking PMs to paper over issues with status updates. This keeps accountability where it belongs and reinforces outcomes vs output OKRs.
One high-leverage move is to remove unnecessary intermediaries. Removing the PM as a middleman creates better flow and clearer ownership. Create direct paths for stakeholders to get bug status without routing everything through product. Use dashboards, shared tools, or Slack channels instead of one-off updates. In my teams, shared Jira views, Slack incident channels, and status pages eliminated handoffs, improved stakeholder management, and gave engineers the space to solve problems end-to-end.
Strong engineering leadership is non-negotiable. What strong engineering leadership should own (and why that matters) is the technical system, quality guardrails, sustainable pace, and the practices that uphold them—incident management, code review rigor, test coverage, and SLOs with SRE. Skilled engineering teams naturally push back when boundaries are crossed—and that’s a good thing. It signals ownership, craft pride, and a pathway to durable execution.
When do I step in as product? Primarily to clarify desired outcomes, sequencing, and trade-offs—bringing customer and business context to the table. I structure product roadmapping and sprint planning around value slices and risks, not task lists. I align on decision rights early: architecture and tech debt strategies live with engineering; product strategy, positioning, and success metrics live with product; discovery and prioritization live with the product trio.
Here are the system-level moves I’ve found most effective: Escalate systemic quality issues to engineering leadership, not individual contributors. Advocate for real engineering leadership if your org expects product teams—not IT teams. Then reinforce a culture of continuous discovery so product, design, and engineering make better upstream decisions together. This is how empowered product teams ship higher-quality outcomes—without burning anyone out.
If you’ve ever found yourself acting as the middleman for bug status or being asked to “own” engineering decisions outside your expertise, you’re not alone. Reset the boundaries, make work visible, and double down on shared outcomes. In my experience, the moment we clarify roles and remove status theater, quality rises, cycle time improves, and everyone does the job they were hired to do—better.
Human-in-the-loop oversight is the fastest and most reliable way I know to elevate AI quality, build user trust, and reduce risk. At HighLevel, my teams treat oversight as a product feature—not an afterthought—because dependable AI experiences come from deliberate design choices across data, models, and people.
When I say “human-in-the-loop,” I mean a system that blends automation with targeted human judgment at key moments: during data curation, prompt engineering, evaluation, deployment, and post-launch learning. This approach turns “AI workflows” into measurable, repeatable processes and keeps me honest about what’s working, what’s drifting, and where a human safety net must step in.
Architecturally, I start with a retrieval-first pipeline to ground outputs in trusted knowledge, then wrap it in guardrails. Deterministic preprocessing, careful prompt engineering, and post-processing validators catch obvious failure modes. Confidence thresholds and policy checks route ambiguous or sensitive cases to a human reviewer, while clear, auditable traces show why the system chose automation versus escalation. This balance supports reliability at scale while preserving agility for “agentic AI” patterns when they add value.
Quality is only real if I can measure it, so I build with eval-driven development from day one. I maintain golden datasets, rubric-based scoring guidelines, and an automated evaluation harness that runs on every change to prompts, models, or data. Pre-production gates protect against regressions, while production telemetry surfaces drift by segment and use case. When it’s time to run experiments, I use A/B tests sized with a minimum detectable effect (MDE) to avoid overfitting to noise.
Operationally, I optimize for outcomes, not output. I track task success rate, time-to-resolution, safety violation rate, hallucination rate, and cost-to-serve, then connect these to outcomes vs output OKRs. The signal I want is simple: are we reliably solving the user’s job-to-be-done with lower effort and higher confidence? If not, I tighten prompts, refine retrieval, or expand human review where it pays off most.
Risk governance is non-negotiable. I design with privacy-by-design and data governance from the start—role-based access, audit trails, PII redaction, and red-team tests for safety. Clear reviewer playbooks and calibration sessions reduce bias and ensure consistent decisions. These practices aren’t bureaucracy; they’re how I operationalize AI risk management while maintaining velocity.
Teams make or break this model. I empower product trios to own the full lifecycle—discovery, build, and learning—so feedback loops close quickly. In-product feedback widgets, reviewer queues, and incident management playbooks help us respond in hours, not weeks. Over time, human review becomes a targeted scalpel rather than a blanket requirement as the system learns and improves.
Economics guide the level of oversight. I treat each workflow like a portfolio: where the value of accuracy is high and ambiguity is common, I route more to humans; where tasks are simple, frequent, and well-bounded, I automate aggressively. The goal isn’t zero humans—it’s optimal humans, deployed precisely where their judgment compounds ROI.
If you’re getting started, begin with one high-impact workflow, establish your golden set and evaluation rubric, and wire in a simple review queue. Prove the lift, then scale. In the short video above, I walk through the patterns I use to design these loops, measure quality with rigor, and ship AI that teams—and customers—can trust.
I’ve learned that the most effective partner product marketing is less about decks and more about decisions. When I collaborate with partner product marketing managers, we translate complex capabilities from a unified analytics platform into crisp, outcome-led narratives that customers can act on. This is where product positioning and go-to-market strategy intersect to create momentum for product-led growth.
In my experience, the strongest partner product marketing managers operate like solution orchestrators. They align value propositions across partners, clarify the problem-solution fit, and articulate competitive differentiation without drowning teams in feature lists. By anchoring messaging in clear customer pains and measurable gains, they help everyone—from solutions engineering to sales—tell the same story with confidence.
My playbook starts with outcomes. We define the “why” in terms customers care about, then quantify it with retention analysis, user activation, and time-to-value. That evidence shapes positioning, enables tighter points of parity and differentiation, and ensures our value proposition resonates in market. The result is faster alignment and fewer cycles spent debating messaging without data.
Cross-functional execution makes or breaks the strategy. I partner closely with solutions engineering to validate solution patterns, and with sales to balance sales-led motions alongside product-led growth. Strong stakeholder management keeps discovery loops tight: we capture objections early, refine narratives quickly, and reduce friction across the funnel.
On the tactics side, I rely on A/B testing to de-risk bold messaging changes and to optimize in-app guides and product tours. We set a minimum detectable effect upfront, instrument journeys with Amplitude analytics, and iterate quickly. This gives the team statistical confidence while keeping speed high—especially when refining narratives for complex partner solutions.
Ultimately, great partner product marketing illuminates the shortest path from capability to customer value. When we pair disciplined positioning with data-driven learning, we strengthen our go-to-market strategy and build durable competitive advantage. That’s how we turn strong solutions into market-leading stories that win—and keep—customers.
Inspired by this post on Amplitude – Best Practices.
You are about to approve an AI feature. The demo works, the team has an adoption dashboard, and every response can collect a thumbs-up or thumbs-down. Yet nobody can answer the questions that will matter after launch: Did the feature help customers finish the job? Was the improvement caused by the AI? Did quality hold across important customer segments? Was the gain worth the latency, cost, and risk?
Do not solve that problem by adding more charts. Build an evidence chain from eligibility and exposure through model behavior and human action to a completed customer outcome. An AI-ready measurement system makes model telemetry and product behavior part of the same decision. That is what lets you improve prompts, retrieval, models, and product design without confusing technical progress with customer value.
Key takeaways
Define the product decision, eligible population, primary outcome, guardrails, and minimum detectable effect before choosing events or building dashboards.
Instrument a traceable sequence from eligibility to exposure, request, response, user action, task completion, and repeat value. Shared identifiers matter more than a large event catalog.
Keep model quality, product behavior, reliability, cost, risk, and business outcomes as separate measurement layers, but make them queryable through the same identities and version fields.
Move through offline evaluation, production shadowing, and a controlled rollout. Each stage answers a different question and needs its own exit criteria.
End every experiment with an explicit decision: ship, iterate, restrict, or stop. A result that produces another indefinite request to collect data is not a decision system.
Start with an evidence contract, not an event list
An instrumentation plan often begins too late in the reasoning process. Someone opens a spreadsheet and lists clicks, generations, feedback actions, and errors. The events may all be valid, but they do not guarantee that the resulting data can answer a product question.
Start with a one-page evidence contract. It should force the product, engineering, data, and AI owners to agree on the decision they are trying to make. Complete these fields before implementation:
Decision: State what will change if the evidence is positive, negative, or inconclusive. For example, the decision might be whether to expand a drafting assistant from one workflow to every workflow.
User problem: Name the job the customer is trying to complete. Avoid substituting the proposed AI capability for the problem.
Eligible population: Define who could reasonably benefit, including account type, workflow state, permission, and any relevant exclusions.
Intervention: Specify what is different from the current experience. Include the product surface and the model, prompt, retrieval, and guardrail configuration that define the treatment.
Primary outcome: Choose one customer behavior that represents successful completion of the job. Give it an exact numerator, denominator, and observation window.
Diagnostics: Identify the signals that will explain why the outcome moved, such as output acceptance, editing, retries, fallbacks, and time to completion.
Guardrails: Define the reliability, safety, customer-experience, and cost conditions that the treatment cannot violate.
Decision rule: Predefine the minimum effect worth detecting, how uncertainty will be handled, which segments will be inspected, and what would cause an early rollback.
A useful hypothesis has a visible causal claim: For an eligible cohort, a defined AI experience will improve a named task outcome over a stated observation window, while specific guardrails remain acceptable. Consider a support workflow. “Customers will like AI drafts” is not testable enough. “Giving eligible support agents an AI-generated draft will improve successful ticket completion without degrading customer satisfaction, safety, latency, or cost per successful resolution” tells you what to instrument and what could veto a rollout.
Separate the six measurement layers
One composite AI score is tempting and usually unhelpful. A single number hides trade-offs and makes failures difficult to diagnose. Keep the layers distinct:
Measurement layer
Question it answers
Useful measures
Decision it informs
Eligibility and adoption
Did the intended customer have a real opportunity to use the feature?
Eligible users or accounts, exposures, first use, repeat use
Reach, discoverability, onboarding, and denominator quality
Task outcome
Did the customer complete the job better?
Task success, time to value, completion without rework, durable repeat behavior
Whether the feature creates customer value
Model quality
Was the output usable for this use case?
Rubric score, groundedness where relevant, acceptance, edits, rejection, regeneration
Prompt, retrieval, data, and model improvements
Reliability and efficiency
Can the experience operate consistently?
Latency, error rate, fallback rate, availability, cost per successful outcome
Architecture, model routing, and operational readiness
Risk and trust
Did the system cross a boundary that should block scale?
Safety violations, moderation triggers, unsupported responses, user overrides
Guardrails, restrictions, and rollback
Business outcome
Does the customer value become durable business value?
Activation, retention, support deflection, account expansion, or attributable revenue
Investment level and product strategy
Choose one primary outcome for the experiment. The other layers are not decorative. Product and model diagnostics explain the result, while guardrails can veto it. A faster workflow that creates unacceptable safety failures is not a win. A highly rated output that does not improve task completion is not yet a product outcome.
Instrument one traceable chain, not a bag of events
The core unit of AI analytics is a traceable attempt to complete a job. You need to follow that attempt across the product interface, AI runtime, and downstream outcome. If each system produces isolated records, the dashboard may show healthy model performance and healthy adoption without revealing whether the same customers received both.
A practical event sequence looks like this:
ai_feature_eligible: The user or account entered a state in which the feature could provide value. This creates the denominator for reach and experiment eligibility.
ai_feature_exposed: The experience was actually rendered or otherwise made available. Keep assignment separate from display so delivery failures remain visible.
ai_request_submitted: The customer initiated an AI-assisted action. Capture the intended use case, not the full sensitive input by default.
ai_response_generated: The AI system produced a response. Record the configuration, latency, error state, fallback behavior, and attributable cost.
ai_response_presented: The output reached the customer. A generated response that never rendered should not count as a usable response.
ai_output_action_taken: The customer accepted, copied, edited, regenerated, rejected, or undid the output. Preserve the difference between no action and an explicit rejection.
ai_task_outcome_recorded: The workflow reached its product-level success or failure state. Link this outcome to the request even if it occurs later in another system.
ai_repeat_value_observed: The user or account returned to the workflow and obtained value again. This distinguishes novelty from an emerging habit.
Those names are examples, not a mandatory standard. Your taxonomy should match the language of your product. The important distinction is semantic: eligibility is not exposure, exposure is not use, generation is not delivery, delivery is not acceptance, and acceptance is not task success.
Give every layer the same join keys
The event chain works only when the records can be joined without relying on an email address, timestamp guess, or mutable account field. At minimum, decide how you will represent:
Identity: Stable user and account identifiers, plus an explicit anonymous-to-authenticated identity rule where needed.
Workflow: A workflow or task identifier that survives navigation, retries, and asynchronous processing.
AI execution: Request and response identifiers that distinguish one customer request from multiple internal model or retrieval calls.
Experiment state: Experiment identifier, assigned variant, assignment timestamp, and the reason a user or account was eligible.
Configuration: Model, prompt template, retrieval index, tool, policy, and guardrail versions. A treatment is not stable if these change invisibly during the test.
Product context: Use case, surface, lifecycle stage, account segment, permission state, and other dimensions selected in the evidence contract.
Operational result: Latency, error class, fallback reason, moderation result, and cost fields defined consistently across providers.
Governance: Schema version, data classification, consent or policy state where applicable, and retention treatment.
Capture context at the time of the event. If an account changes plan or segment later, a query should not silently rewrite the conditions under which the experiment ran. Preserve both the stable identity and the relevant historical snapshot.
Apply privacy-by-design to inputs, outputs, and feedback. Raw prompts and generated text can contain customer data that does not belong in a broadly accessible analytics platform. Prefer structured categories, redacted attributes, content-type labels, and references to a separately governed evaluation store. Store the minimum information needed for the decision, not every token merely because it is available.
Catch instrumentation defects before launch
AI workflows create several failure modes that ordinary click tracking can miss. Add these checks to the release path:
Count one logical customer request separately from provider retries, tool calls, retrieval queries, and fallback calls. Otherwise usage and cost denominators will disagree.
Use idempotency or deduplication rules for events emitted by asynchronous jobs. A replayed queue message should not create a second successful task.
Version an event when its meaning changes. Adding an optional property may be compatible; changing what counts as task success is a new semantic contract.
Test identity resolution across the full journey, including anonymous use, authentication, account switching, shared workspaces, and delayed downstream outcomes.
Reconcile generated, presented, and acted-on counts. A large unexplained gap often reveals a delivery, client, or instrumentation failure before it becomes a misleading product conclusion.
Turn model quality into a product scorecard
An offline model score and an online product metric answer different questions. The offline evaluation asks whether a configuration can produce an acceptable result on a defined set of cases. The online measurement asks whether the experience changes behavior and outcomes for real customers. You need both, and you should not let either impersonate the other.
Use denominators that expose failure
Every rate should state what had the opportunity to enter its numerator. These definitions are more useful than labels such as quality score or engagement:
Task success rate = successful target tasks divided by eligible tasks that reached the defined opportunity.
Delivered response rate = responses presented to the customer divided by valid submitted requests.
Helpful output rate = reviewed outputs that satisfy the use-case rubric divided by outputs with a completed review.
Fallback rate = requests that used the defined fallback path divided by eligible AI requests.
Safety intervention rate = requests that triggered a defined safety intervention divided by requests evaluated by that policy.
Cost per successful outcome = attributable AI runtime cost divided by successful target tasks. Use a consistent cost boundary so model, retrieval, and fallback costs are not included selectively.
Repeat value rate = users or accounts that complete the target task again within the chosen window divided by those that first completed it.
Display the numerator, denominator, missing-outcome count, and metric definition beside the rate. A percentage can look healthy because delivery failures disappeared from its denominator or because only enthusiastic users submitted feedback.
Human signals such as thumbs, edits, acceptance, deflection, and customer satisfaction are valuable diagnostics, but each has an interpretation problem. Thumbs reflect the minority who choose to respond. Acceptance can reward a convenient draft that still needs correction later. A large edit may mean the output was poor, or that it provided a useful starting structure. Regeneration can indicate failure, exploration, or a request for variety. Pair these signals with task completion, time to value, downstream correction, and representative human review.
Build the offline evaluation around the product decision
A representative evaluation set is a product artifact, not merely a model-engineering artifact. Construct it deliberately:
Define the unit being judged. It may be an answer, classification, draft, action plan, tool decision, or completed multi-step workflow.
Write a rubric that separates must-pass requirements from preferences. Include factual or grounded behavior, task completion, policy compliance, and format only where they matter to the user job.
Sample the cases the target population actually produces. Preserve important slices such as use case, complexity, language, account type, or risk level when those dimensions affect the decision.
Define how ambiguous cases, missing context, and evaluator disagreement will be handled. Do not force false certainty into a label simply to complete a dataset.
Record the exact model, prompt, retrieval, tool, and guardrail configuration for every run. A score without a reproducible configuration cannot guide a rollout.
Keep a stable benchmark for comparison while adding a governed set of newly discovered failure cases. If every prompt change also changes the test, improvement becomes impossible to interpret.
Offline success is an entry condition for production learning, not evidence of customer impact. It can eliminate weak configurations cheaply and expose slice-level failures before customers encounter them. It cannot tell you whether people discover the feature, trust it, change their behavior, or retain because of it.
Run experiments as a sequence of risk-reducing gates
Do not ask one A/B test to discover whether the model works, whether the infrastructure survives production, whether the interface is understandable, and whether the business case holds. Move through offline evaluation, production shadowing, and controlled rollout. Each gate removes a different uncertainty.
Offline evaluation: Compare the candidate configuration with the current baseline on the representative evaluation set. Review overall quality, must-pass requirements, important slices, safety behavior, and cost. Exit only when the candidate is good enough to justify production exposure.
Shadow mode: Run the candidate against production traffic without showing its output to customers or changing the workflow. Use this stage to verify input distribution, integration behavior, latency, failures, fallbacks, policy coverage, and attributable cost. Shadow mode cannot demonstrate customer lift because the customer never experiences the treatment.
Controlled rollout: Deliver the experience through a feature flag to a randomized treatment group while preserving a valid control. Measure the primary outcome and guardrails using the assignment unit specified in the evidence contract.
Scaled release: Expand only after the decision rule is met. Continue monitoring for distribution shifts, configuration changes, operational regressions, cost drift, and safety failures that a time-bounded experiment may not capture.
Feature flags are more than a release convenience. They preserve a control, enable a rapid rollback, restrict exposure when a feature is safe only for a defined cohort, and separate model deployment from product exposure. Name an owner for the flag, the rollout decision, and the rollback action before traffic begins.
The product decision and the hypothesis being tested.
The eligible population and every exclusion that will be applied.
The baseline experience and the complete treatment configuration.
The randomization unit, assignment method, and exposure definition.
The primary metric, including numerator, denominator, and observation window.
The minimum detectable effect: the smallest improvement that would be material enough to justify the cost or complexity of rollout.
Guardrail definitions, acceptable boundaries, and rollback conditions.
The diagnostic metrics that may explain the result but will not be promoted to primary after the test begins.
The segments that will be examined and why they matter to the product decision.
The analysis method, expected decision point, and owner of the final call.
The minimum detectable effect is a product choice before it is a statistical input. If a smaller gain would not change the roadmap, do not design the experiment around detecting it. Traffic, baseline behavior, outcome variability, assignment unit, observation window, and the selected effect all shape whether the experiment can be conclusive. When traffic is insufficient, the honest choices are to run longer, test a larger change, use a nearer but defensible outcome, combine learning with other evidence, or decline to run an underpowered experiment. Lowering the standard after seeing the result does not create evidence.
Avoid the analysis traps specific to AI products
Do not treat every generation as an independent experimental subject. A single user or account may generate repeatedly, and those observations share the same behavior and assignment.
Randomize at the account level when treatment can spill across a shared workspace, team process, or common customer record. User-level randomization in that setting can contaminate the control.
Do not analyze only people who clicked the AI control. Treatment may change whether they click, so filtering on that action can remove part of the treatment effect. Start from the assigned eligible population and use triggered views as diagnostics.
Do not change the model, prompt, retrieval source, or guardrail silently inside a treatment. If an urgent fix is necessary, record the version boundary and decide whether the test remains interpretable.
Do not optimize an intermediate signal in isolation. More generations can mean adoption or repeated failure; more acceptance can coexist with lower downstream accuracy; faster responses can be worse responses.
Do not repeatedly inspect the result, stop when it looks favorable, and then present that stopping point as planned. Follow the pre-registered analysis or use a statistical design that explicitly supports sequential decisions.
Do not search every segment for a winner after an inconclusive overall result. Treat an unexpected segment pattern as a hypothesis for validation, not automatic authorization to scale.
Create an operating loop that can say stop
A technically correct dashboard does not create accountability. The system becomes useful when the team knows who reviews each signal, what action follows, and which metric has authority when measures disagree.
Leadership view: Primary customer outcome, durable business outcome, cost per successful outcome, major guardrails, rollout status, and decision owner.
Product view: Eligibility-to-outcome funnel, activation, repeat use, retention by cohort, time to value, and the diagnostics behind the current experiment.
AI quality view: Offline rubric results, online review results, feedback behavior, fallbacks, and performance by use case, model version, and important slice.
Operations and trust view: Latency, errors, availability, cost, moderation triggers, safety interventions, and rollback state.
Every view should resolve to the same metric registry. The registry needs a definition, owner, source events, inclusion and exclusion rules, observation window, grain, version, and change history. If task success means one thing in the product review and another in the model review, a common dashboard tool will not create a common truth.
Put measurement into the delivery workflow
During discovery, write the evidence contract alongside the problem statement. The primary outcome should be agreed before the implementation solution hardens.
During implementation, review event semantics, identity, privacy, configuration versioning, and metric formulas. Run automated schema checks with the same seriousness as other release validations.
Before rollout, verify the offline gate, shadow-mode results, experiment assignment, dashboards, alerts, flag owner, and rollback path.
During the experiment, review data quality and guardrails on the agreed cadence. Distinguish operational monitoring from an unplanned search for a favorable outcome.
At the decision point, record the result, uncertainty, segment findings, guardrail status, configuration, and action. Make the record reusable by the next prompt, retrieval, model, or experience iteration.
After the decision, remove abandoned dashboards and events, close obsolete flags, and update the evaluation set with newly validated failure modes. Measurement debt compounds when every experiment leaves permanent debris.
The decision itself should fall into one of four states:
Ship: The primary outcome meets the decision rule, the evidence is interpretable, and guardrails and economics remain acceptable.
Iterate: The result is not ready to scale, but diagnostics identify a plausible and testable failure in quality, retrieval, interaction design, reliability, or targeting.
Restrict: The value is credible only for a defined cohort or use case, and that boundary can be enforced and validated without creating unacceptable risk.
Stop: The effect is below what would justify the investment, a critical guardrail fails, the economics do not work, or the experiment cannot be made interpretable without redesign.
Cost, safety, privacy, and customer trust are not secondary metrics that a conversion lift can overrule. If one is a hard boundary, say so in the evidence contract and give it the power to stop the rollout.
If your current analytics cannot support this full system, start with one high-value AI workflow. Write its evidence contract, implement the traceable event chain, assemble a representative offline evaluation set, and place the experience behind a controlled flag. Your first useful deliverable is not a larger dashboard. It is a product decision that can be made without debating what the data was supposed to mean.
For years, I chased the elusive goal of delivering a perfect customer experience. Today, with AI embedded in our support operations, that standard is finally within reach—and it’s reshaping how we prioritize, design, and scale service.
In “The 2026 Customer Service Transformation Report,” teams report early, tangible wins from AI: faster responses, higher efficiency, and consistent coverage across languages and time zones. Those gains create the capacity we’ve always needed. The more we push the technology, the more quality improvements we unlock.
This marks a fundamental shift. As AI takes on more, our focus can finally move from firefighting to crafting the customer experience. When the AI is working, the measure of success becomes how well it’s working—across accuracy, tone, resolution, and end-to-end journey quality.
I’ve seen this transformation firsthand. Mature AI deployment gives my team “breathing room,” so we can design for consistently excellent outcomes rather than obsess over deflection. That means widening access to support, removing friction on the path to resolution, and anticipating customer needs before they escalate.
In our own support organization, we opened support to trial customers, accelerated first response times, and added consultative sessions during onboarding. We absorbed a 300% increase in total demand without adding headcount—made possible by deep integration of an AI Agent and a disciplined AI strategy.
Teams with mature customer service deployments are nearly three times likelier to say they always meet increasing expectations—27% vs 9% at initial rollout—highlighted by bold orange and gray comparison bubbles.
Across the industry, the pattern is similar. When teams initially deploy AI, only 9% say they can always meet customer expectations. That number triples as teams reach a mature level of deployment. Even as expectations rise, the organizations that deeply integrate AI—complete with clear ownership, robust instrumentation, and continuous improvement loops—are the ones most likely to meet (and exceed) the bar.
Looking ahead to 2026, I expect omnichannel consistency to become a key differentiator. The data shows planned investment is distributed nearly equally across chat, email, and social messaging (36% each), closely followed by phone/voice (31%). The question is no longer “Which channel should we optimize?” but “How do we deliver a consistent, AI-powered experience everywhere our customers are?”
Teams that solve for omnichannel consistency will bridge the long-standing gap between what customers expect and what support can deliver. Every touchpoint becomes an opportunity to exceed expectations and build durable trust.
Consider Clay, a team that scaled support without sacrificing quality. Support is one of their main growth drivers, and as their customer base expanded, ticket volume surged. Early on, they concentrated much of their effort in Slack, cultivating close, transparent community relationships. But relying on a single channel created friction as they grew; customers wanted the flexibility of email and in-app chat, and Clay needed to deliver the same high standard everywhere.
Where AI investment is headed for customer service in 2026: chat, social, and email lead at 36%, with phone/voice close behind at 31%. A bold visual snapshot of shifting channel priorities in CX.
By unifying their support experience with an AI Agent, Clay brought consistency across channels. Today, AI is involved in 90% of all queries and handles half of Clay’s total volume, upwards of 7,000 queries a month. First response rates improved significantly, freeing the team to focus on proactive, high-impact work.
That work includes identifying content gaps for education and content marketing, reaching customers before they need to ask for help, and surfacing feature requests and recurring challenges to product teams. Clay proves that when support is truly great, it becomes a competitive edge.
So how do you build a superior customer experience with an AI Agent? Here are five principles I use when scaling toward mature deployment.
1) Treat customer experience like a product. Treating support as a product means designing, building, and managing the support experience with the same rigor as your core product. You define goals (faster onboarding, higher CSAT or CX Score, lower churn). You map flows (AI starts the conversation, human handovers, proactive nudges). You instrument the journey (track handoffs, drop-offs, success states). You run tests and ship improvements (tone tweaks, fallback paths, training updates). You own the outcomes (gather feedback, measure performance, use insights to continuously improve the system).
Leaders are racing ahead with real AI in support. Explore the 2026 Customer Service Transformation Report to see where deployment is stalling, benchmark your team, and get practical steps to scale automation that delights.
2) Lead with AI, back with humans. AI isn’t replacing the human touch. It’s redefining when, where, and how it’s most valuable. In a scaled model, AI is the first responder and the end point for most conversations. Humans step in where they add the most value—particularly during high-stakes issues—and those handoffs should feel seamless. Meanwhile, your team focuses on improving AI performance and optimizing the end-to-end journey.
3) Be proactive. Use AI to anticipate needs, guide customers before problems arise, and nudge them toward successful outcomes. This is where customer support AI strategy shines—moving from reactive triage to journey orchestration that protects momentum and builds trust.
4) Build for trust. Many customers still carry the legacy of clunky chatbots that delivered vague answers and dead ends. You earn trust by showing that your system works. Don’t hide your AI Agent behind layers of “choose an option.” Get customers to the AI quickly, demonstrate real problem-solving, and ensure that when a human is needed, they join with full context to resolve complex issues efficiently.
5) Make it feel personal. Your AI Agent represents your brand. The way it speaks, follows policies, and responds matters. Use tone control, fallback logic, and language preferences to align the experience to your standards. Consistency builds trust; personality builds connection and loyalty.
Perfect really is possible. With deep AI implementation, you can scale comprehensive, fast, and personal support across channels—so customers feel supported not just when they reach out, but throughout their journey. That’s the promise of modern AI workflows in support, and it’s what will separate leaders from laggards in the years ahead.
Your team can turn a behavioral anomaly into a polished prototype within hours rather than over weeks, yet still stall when it is time to choose a problem, approve a test, or act on the result. That is the central trap in AI-accelerated product development: producing artifacts faster does not automatically produce better decisions.
The useful unit of acceleration is the complete learning loop: detect a meaningful signal, frame the opportunity, explore distinct hypotheses, validate the riskiest assumptions, ship with controlled exposure, and use production evidence to decide what happens next. You need one operating model across that loop, not a collection of disconnected AI shortcuts.
Key takeaways
Optimize for time from signal to a decision backed by evidence, not the number of analyses, prototypes, or tickets generated.
Give every investigation an outcome contract: the customer behavior, target cohort, primary metric, guardrails, and decision that the work is intended to inform.
Use AI to create alternatives that represent different value hypotheses. More cosmetic variants usually create more review work without expanding what you can learn.
Carry the same cohort, metric definitions, hypothesis, and constraints from discovery into the production experiment. This prevents the handoff from silently changing the question.
Let agents act only where their permissions, thresholds, audit trail, and rollback path are explicit. Autonomy should expand with evidence, reversibility, and trust.
Design one loop from a product signal to a decision
Most teams first apply AI to individual tasks. An agent summarizes a dashboard. A model drafts a product requirements document. A design tool generates a flow. A coding assistant implements it. Each task becomes faster, but the work still waits between tasks because nobody has defined what evidence is sufficient, who can make the next decision, or what outcome the change should affect.
An agent that discovers more anomalies while the product trio reviews opportunities through the same overloaded process has created a longer inbox. The bottleneck has moved; it has not disappeared. The remedy is to treat a decision-ready hypothesis, rather than an AI-generated artifact, as the unit of product work.
A practical discovery loop has the following sequence:
Write the outcome contract. Name the customer behavior you want to change, the cohort in which it matters, the primary outcome metric, the metrics that must not deteriorate, and the decision this evidence will support.
Map the driver tree. Break the outcome into observable behavioral drivers. This gives the agent a bounded search space and prevents a broad metric movement from producing an equally broad list of possible features.
Issue an investigation brief. Tell the agent which definitions, segments, releases, and time comparisons it may use; which data it may access; what it should monitor; and whether it may only recommend or may also initiate an approved workflow.
Require an evidence packet. An anomaly should arrive with the affected cohort, direction and materiality of the movement, relevant timing, instrumentation checks, plausible alternative explanations, and the next question worth answering.
Record the decision. The product trio should accept, reject, defer, or refine the hypothesis and state why. That decision becomes context for the next investigation instead of disappearing into a meeting.
For an onboarding problem, the outcome contract might identify accounts attempting their first meaningful setup, define the activation behavior precisely, name downstream retention and support demand as guardrails, and authorize the agent to investigate friction without changing the customer experience. That is much more useful than asking AI to find onboarding insights. The broad request has no stopping condition and no decision attached to it.
The driver tree then narrows the investigation. Activation might depend on starting setup, completing required configuration, reaching an initial value-bearing action, and returning to use that value. The point is not to make the tree exhaustive. It is to show which behaviors could plausibly explain the outcome and which are observable in your product data.
This is where continuous agents can provide real leverage. They can monitor established metrics, inspect funnel and cohort movements, and surface material changes such as an activation decline in a valuable cohort or a retention change following a release. They can also compare segments and assemble supporting context without waiting for a fresh manual analysis request.
But the alert is not yet an opportunity, and correlation is not a causal explanation. A broken event, a changed identity rule, a traffic-mix shift, or a simultaneous release can resemble a change in customer behavior. Make instrumentation confidence and alternative explanations mandatory fields in the evidence packet. If either is weak, the next action is to improve the evidence, not to generate a feature.
The product trio still owns the consequential judgment: whether the problem is worth solving, what tradeoff is acceptable, which customer evidence is missing, and whether the likely value justifies the delivery cost. AI should remove investigative toil and expose overlooked evidence. It should not hide a strategic choice inside an automated recommendation.
Use AI to expand hypotheses without expanding waste
Generative design changes the economics of exploration. Once a measurable opportunity is clear, high-fidelity flows can be produced in hours instead of stretching across weeks. That makes it practical to inspect several possible mechanisms before production code is written.
Cheap variation also creates a new failure mode. If every stakeholder can request another screen, the team spends its saved production time reviewing undifferentiated options. The prompt should therefore ask for distinct value hypotheses, not a gallery of cosmetic alternatives.
Build the prototype brief from the evidence packet. It should contain:
Target user and context: the affected cohort, the job it is trying to complete, and the point at which friction appears.
Observed evidence: the behavioral signal, qualitative context if available, instrumentation caveats, and alternative explanations that remain open.
Value hypothesis: why a proposed mechanism should change the target behavior, stated in a form that can be rejected.
Meaningfully different mechanisms: alternatives that change how value is delivered, explained, sequenced, or experienced rather than merely changing visual treatment.
Outcome and guardrails: the primary behavior to influence and the accessibility, privacy, brand, reliability, and business constraints that every variation must respect.
Instrumentation needs: the events and properties required to tell whether people encounter, understand, use, and benefit from the proposed experience.
A useful review question is: If these alternatives perform differently, will you learn something about customer value? If the answer is no, the variations probably differ in presentation but not in hypothesis. Asking AI for more of them will not improve the decision.
Match the validation method to the uncertainty:
Concept validation addresses whether the intended user understands the proposition and considers it relevant.
Usability validation addresses whether the user can recognize the next step, complete the flow, and recover from confusion.
Production experimentation addresses whether exposure changes actual behavior under real product conditions.
Cohort-level follow-through addresses whether an immediate movement is accompanied by the activation, retention, or expansion outcome the team ultimately cares about.
Do not ask a prototype to answer a production question. A polished interaction can expose comprehension and usability problems, but it cannot establish that the experience will improve retention. Conversely, do not consume production capacity to answer a basic usability question that a prototype could resolve before engineering begins.
Define the decision rule before each validation step. State what evidence would cause the trio to advance the hypothesis, revise it, or stop. This prevents a compelling AI-generated design from becoming the default simply because it exists. High fidelity is a communication advantage, not proof of value.
Carry the discovery contract into production
The discovery-to-delivery handoff often introduces more error than the tools remove. A metric is renamed, a cohort becomes broader, a design constraint disappears from the ticket, or an experiment is configured to answer a slightly different question. The team ships quickly and then debates what the result means.
Before implementation begins, the trio should be able to point to a compact delivery contract containing:
The customer problem, target cohort, and value hypothesis.
The primary outcome metric and the metrics that protect against unacceptable side effects.
The exact event, property, identity, and segment definitions needed for evaluation.
The minimum detectable effect, meaning the smallest change that would be consequential enough to alter the product decision.
The planned exposure controls, eligibility rules, rollback conditions, and owner.
The accessibility, privacy, data-governance, reliability, and brand constraints inherited from discovery.
The result that would lead to shipping, iteration, further investigation, or rollback.
Set the minimum detectable effect before examining experiment results. The question is not merely whether a statistical difference can be found. It is whether the experiment can detect an effect large enough to matter to the decision. If realistic exposure cannot provide decision-worthy power, acknowledge that limitation. Consider a more substantial intervention, a longer evidence path, or a different validation method instead of asking an underpowered test for certainty it cannot provide.
Risky changes should be gated behind feature flags and delivered through a controlled CI/CD path. A flag limits exposure and creates a rollback mechanism; it does not, by itself, make a release an experiment. You still need stable assignment, defined eligibility, trustworthy instrumentation, and a predeclared interpretation plan.
Not every change is suitable for an A/B test. Some changes are required, too interconnected for clean isolation, or exposed to too little eligible traffic for a decision-worthy test. The discipline still applies: state the expected behavioral change, release progressively when possible, validate the instrumentation, inspect guardrails, and choose the review point before launch.
When production data arrives, evaluate more than the aggregate primary metric. Confirm that exposure and events behaved as intended. Inspect the cohorts named in the original opportunity. Check whether the result varies across important segments. Then follow the downstream activation or retention signal that justified the work. Production conditions include latency, reliability, real data, competing tasks, and repeated use; prototype enthusiasm does not remove any of them.
Finally, record the product decision and feed it back into the system. The agent should know which hypothesis was accepted, what actually shipped, what the experiment showed, and why the team chose to scale, revise, or stop. Without that context, the next automated investigation starts from activity rather than accumulated learning.
Give agents decision rights, guardrails, and a balanced scorecard
Agentic workflows become risky when a team discusses autonomy as a general capability. Decision rights need to be assigned to a defined action in a defined context. The same agent may safely monitor an established metric, recommend an investigation, prepare a prototype, and still require explicit approval before changing a customer experience.
Use the following as a starting policy, then tighten it to your data sensitivity, product risk, and operational controls:
Work
Agent role
Human decision gate
Required control
Monitor an established metric
Run continuously within approved read access
Metric definitions and alert conditions approved in advance
Access boundaries, instrumentation-health checks, and an audit log
Investigate an anomaly
Assemble evidence and recommend hypotheses
Product trio decides whether the signal represents a meaningful opportunity
Cohort context, alternative explanations, confidence, and traceable queries
Generate a prototype or implementation draft
Prepare alternatives and supporting artifacts
Design and engineering approve customer experience and technical choices
Accessibility, privacy, brand, architecture, and data-use constraints
Launch a customer-facing experiment
Prepare configuration; execute only when policy explicitly permits it
Named owner approves exposure, success criteria, and rollback path
Feature flag, eligibility rules, MDE, guardrails, monitoring, and rollback
Trigger a CRM or in-app workflow
Act only inside preapproved conditions
Owner approves audience, message, frequency, and stop rules
Consent-aligned data, bounded actions, suppression logic, and reviewable history
The key distinction is not human versus autonomous work. It is whether the action is bounded, observable, reversible, and aligned to an approved outcome. An agent can be highly autonomous inside a narrow monitoring job and strictly advisory when a decision affects customers, commitments, or sensitive data.
Three governance questions should appear in every agent brief: What may the agent observe? What may it decide? What may it change? Add the owner who reviews its reasoning, the evidence it must preserve, and the mechanism that stops or reverses an action. This turns broad principles such as decision rights, reasoning transparency, and outcome alignment into enforceable operating rules.
Measure flow, quality, outcomes, and risk together
A scorecard focused only on speed will reward premature action. A scorecard focused only on business outcomes will hide whether the operating system is actually improving. Track four dimensions:
Flow: time to insight, time to action, manual analysis effort, and waiting time between investigation, decision, validation, and release.
Decision quality: whether investigations include instrumentation checks and alternative explanations, and whether experiments have a hypothesis, MDE, guardrails, and interpretation rule before launch.
Customer and business outcomes: the relevant movement in activation, retention, expansion, or another outcome named in the contract, including differences across the target cohorts.
Risk: actions outside approved permissions, privacy or access violations, misleading analyses caused by instrumentation problems, customer-impacting errors, and rollbacks.
The relationships between these measures are diagnostic. Shorter time to insight with unchanged time to action means the decision queue is now the bottleneck. More agent-initiated initiatives with flat activation or retention means the organization has increased automation, not product value. Lower manual analysis effort paired with weaker evidence packets means the work became cheaper by discarding necessary scrutiny.
The percentage of initiatives initiated by agents can be useful as an adoption indicator, but it is a poor destination metric. The meaningful result is a shorter, more reliable path to customer and business impact. Keep outcome measures beside time-to-insight, time-to-action, agent-initiated work, and manual analysis effort so local efficiency cannot masquerade as progress.
Start with one bounded learning loop
Do not begin by making every product workflow agentic. Choose one recurring, measurable problem in a trusted part of the data, such as onboarding friction, activation, or retention for a defined cohort. Then roll out the operating model in sequence:
Timestamp the current stages from signal detection through decision, validation, release, and post-launch review. This establishes where work actually waits.
Stabilize the outcome, cohort, event, and segment definitions. If the instrumentation is not trustworthy, repair it before automating interpretation.
Run the agent in read-only, recommendation mode. Require the standard evidence packet and audit whether its conclusions can be reproduced.
Connect approved investigations to the prototype brief. Ask the product trio to select among distinct hypotheses and document why.
Carry the selected hypothesis into the delivery contract, feature flag, instrumentation plan, and evaluation rule.
Permit automated actions only after the team has defined bounded permissions, monitoring, stop conditions, ownership, and rollback.
Review whether the loop became faster without weakening decision quality, customer outcomes, or governance. Expand the model only where that balance holds.
If an agent cannot show how it reached a conclusion, keep it in an investigative support role. If the team cannot state what result would change its decision, pause the experiment design. If cycle time falls but no relevant outcome improves, revisit the opportunity selection and hypothesis quality rather than adding more automation.
For your next active product problem, write the outcome contract before requesting an AI analysis or prototype. Give the agent a bounded investigation brief, require the trio to compare meaningfully different hypotheses, and move the chosen hypothesis into production without changing its metric or cohort. That single end-to-end loop will tell you more about your AI readiness than a long inventory of tools.
The test is straightforward: if AI helps you reach a consequential, auditable product decision sooner and learn from the result, it has accelerated product development. If it merely creates more things to review, it has accelerated output.
Your team may already use AI to draft emails, summarize calls, research accounts, and answer website questions. Yet the lead still waits in a queue, the seller still reconstructs context, and the customer still repeats the same information after every handoff.
If you are deciding how to make go-to-market genuinely AI-native, do not start with another list of tools. Decide which part of the revenue journey can run as a complete, observable workflow: AI handles defined decisions and actions, humans take over at explicit boundaries, and both work from the same customer state.
Redraw the revenue workflow before automating its tasks
Adding an assistant to every department can make individual tasks faster without making the revenue system faster. Marketing produces more content, sales receives more research, and customer success gets more summaries, but the queues and handoffs between those functions remain intact. More output can even make those bottlenecks worse.
An AI-native workflow has a different unit of design. It owns a bounded outcome over time. It can observe an event, retrieve approved context, choose among permitted actions, update the CRM, evaluate what happened, and either continue or escalate. The distinction matters: generating a follow-up email is a task; noticing that a qualified buyer has gone quiet, selecting the appropriate follow-up, sending it under policy, recording the attempt, and changing course based on the response is a workflow.
Map one live customer journey before discussing models or vendors. For every transition, write down six things:
Trigger: What observable event starts the work? Examples include a demo request, an unanswered question, a completed trial action, or a missed follow-up.
State: What must be known before anyone acts? Include the account, buyer, stage, previous interactions, consent, product usage, and open commitments that matter.
Decision: What choice is being made? Qualification, routing, next-best action, escalation, or disqualification should not be hidden inside a vague prompt.
Action: What is the agent actually allowed to do? Drafting, sending, booking, calling, demonstrating, updating a field, or creating a human task are different permission levels.
Evidence: What will prove that the action was appropriate and completed? Preserve retrieved passages, tool results, timestamps, policy checks, and the resulting CRM change.
Exception owner: Who takes responsibility when confidence is low, the buyer objects, the data conflicts, or the request falls outside policy?
This map exposes where AI can remove elapsed time rather than merely reduce typing. Baseline the current path using measures you already trust: time between stages, abandonment points, manual touches, repeated discovery, and incomplete CRM records. Then choose one customer outcome, such as completing a qualified next step, instead of treating generated messages as success.
My test is simple: if the proposed system cannot identify its current state, show why it acted, and recover from a failed action, it is still a feature. It is not yet part of the revenue operating system.
Start with a bounded inbound motion, then earn more autonomy
I would usually start where the buyer has already expressed intent. An inbound visitor asking a product question or requesting a demonstration gives you a clear trigger, an identifiable job, and a natural human fallback. You can observe whether the interaction advances the buyer without asking an agent to manufacture demand across an ambiguous market.
A strong first workflow has five properties:
The entry event and desired next state are unambiguous.
The agent can answer from an approved and maintainable body of knowledge.
Most actions are reversible, or a human can approve them before execution.
Failure and frustration can be detected quickly.
The business outcome appears in a system of record rather than a separate AI dashboard.
Inbound qualification, guided product education, demo support, meeting preparation, and structured follow-up often fit these conditions. A practical early implementation can be deliberately modest: a voice interaction, reusable product demonstrations, and a retrieval-first knowledge layer. Retrieval gives the agent current, company-approved material without forcing every sales fact into a prompt, and it gives evaluators evidence against which to judge an answer.
Treat the interaction surface as part of the workflow, not as decoration. In one documented implementation, adding a realistic avatar changed how prospects behaved: they interrupted, probed, and requested demonstrations in ways associated with a live sales conversation. That is evidence that an interface changes the behavior it invites, not proof that every buyer or sales motion needs an avatar. Test chat, voice, and video against the buyer’s actual job. Do not choose the most human-looking interface by default.
Expand autonomy in gates rather than with one large launch:
Shadow: The agent recommends an answer, decision, or next action while a human remains responsible for execution. Use disagreements to build the first evaluation set.
Constrained execution: The agent handles approved questions and actions, writes every result to the CRM, and routes exceptions to a named person.
Bounded workflow ownership: The agent can continue across interactions and days, but only inside an explicit state machine, policy envelope, and escalation contract.
Adjacent expansion: Reuse proven capabilities in another stage, such as onboarding or customer success, only after the first workflow is stable and measurable.
Gate movement on evidence, not enthusiasm or a calendar date. The agent should not receive a new action merely because it can generate plausible language. It should receive that action when you can detect a bad decision, contain its consequence, and restore the customer journey.
Give agents distinct jobs and humans a real handoff contract
A single prompt that qualifies, pitches, retrieves facts, controls tools, remembers history, evaluates itself, and decides when to escalate becomes difficult to test. It also mixes goals that can conflict. A persuasive sales response and a conservative policy check should not compete for attention in the same undifferentiated instruction block.
A useful architecture separates five responsibilities:
Knowledge or creator agent: Turns approved documentation, training material, and transcript patterns into versioned playbooks and retrievable knowledge.
Conversation agent: Handles the live interaction, asks the next appropriate question, and stays within the current objective.
Workflow orchestrator: Maintains state across channels and time, selects the next permitted step, invokes tools, and pauses when an exception occurs.
Evaluator: Scores the interaction for grounding, policy compliance, conversation quality, sentiment, and task completion.
Human owner: Resolves ambiguity, negotiates unusual terms, restores trust, and changes the playbook when the system exposes a recurring gap.
Specialized conversation roles can also help manage latency, context limits, and model weaknesses. Greeting, discovery, qualification, and pitching do not necessarily need identical instructions or context. The important design move is not the number of agents; it is the ability to isolate a responsibility and test it. Deterministic paths can handle predictable stages while orchestration manages contextual departures.
Do not split a workflow into a swarm merely because multi-agent architecture sounds advanced. Start with the fewest independently testable components. Split one when its context becomes noisy, its latency threatens the experience, its permissions need separate control, or its failures require a different evaluation method.
The orchestrator should persist a compact state record that another worker can understand. At minimum, capture the current objective, stage, known facts, supporting evidence, confidence, unresolved questions, commitments already made, next permitted action, and current owner. Keep the transcript available, but do not force every downstream decision-maker to reconstruct state from raw conversation history.
A human handoff is a product contract, not an emergency notification. Define its triggers before launch. Useful triggers include:
The agent cannot find approved evidence for a material answer.
Confidence falls below the level you have defined for that action.
The buyer repeats a correction, expresses frustration, or explicitly requests a person.
The request involves commercial, security, legal, or contractual judgment outside the approved playbook.
A tool fails, CRM state conflicts with the conversation, or the proposed action would duplicate an existing commitment.
The receiving person needs more than a transcript link. Send the buyer’s goal, the current stage, facts already established, the reason for escalation, supporting evidence, actions attempted, promises made, urgency, and the decision the human must make. Pause further automated outreach until ownership is acknowledged. Otherwise, the agent can send a cheerful follow-up while a seller is handling a sensitive objection.
After the human resolves the case, specify whether control returns to the workflow and what state must change first. That return path is where many pilots quietly become permanent manual operations.
The CRM must carry this shared state in both directions. Agents should read the latest account context and write back decisions, actions, outcomes, and evidence. A system that conducts a convincing conversation but leaves the record incomplete creates invisible work for the next person. Tight CRM integration and persistent workflow orchestration are what turn an interface into an operating capability.
Use evaluations and revenue governance as the control plane
A polished demonstration proves that a workflow can succeed once. Revenue leaders need evidence that it succeeds repeatedly, fails visibly, and improves without silently regressing. That requires an evaluation system tied to release decisions.
Build the control loop in this order:
Define the failure taxonomy. Separate unsupported facts, policy violations, missed discovery, poor routing, broken tools, excessive latency, incorrect CRM updates, weak handoffs, and incomplete outcomes. A single quality score hides the repair you need.
Create a representative evaluation set. Include common interactions, important segments, known edge cases, adversarial requests, tool failures, and examples that should trigger escalation. Label the expected action and unacceptable actions, not only an ideal sentence.
Review production conversations aggressively at the start. One practical deployment pattern reviews every interaction during early operation and tapers toward a sample of about 5% as confidence grows. The reduction is earned through observed quality; it is not a default schedule. Customer review, evaluator scoring, and sampling can operate as one quality loop.
Turn failures into regression tests. When a human corrects an answer, routing decision, or handoff, add a durable test before changing the prompt or playbook. Otherwise, fixing one conversation can break another without detection.
Release progressively. Use proof-of-concept validation, controlled exposure, A/B rollout where appropriate, CRM logs, and dashboards. Preserve a rollback path for prompts, models, playbooks, tools, and policies.
Expand authority only after two kinds of evidence agree. Agent quality must remain acceptable, and the intended business outcome must improve without shifting damage into complaints, bad-fit pipeline, downstream rework, or customer churn.
Your dashboard should distinguish system quality from business performance. Both are necessary, and neither can substitute for the other.
Qualified progression, completed bookings, stage movement, activation, retention, and downstream rejection of poor-fit opportunities
Operating health
Can the capability run and improve reliably?
CRM completeness, failed actions, recovery, human review load, overrides, version history, and cost per completed outcome
Do not reward the agent for conversion alone. A system can raise a local conversion metric by overpromising, qualifying weak opportunities, or making escalation harder. Review performance by segment and release version, and keep quality and downstream outcome guardrails beside the target metric.
Governance also needs named owners. The CRO should own the end-to-end revenue outcome and the interlocks across marketing, sales, solutions engineering, onboarding, and customer success. Product and AI leaders should own agent behavior, experience, evaluation infrastructure, and release gates. Revenue operations should own CRM state, definitions, attribution, and operational dashboards. Functional leaders should own their playbooks and exception policies. Humans in the workflow should own judgment where trust, negotiation, and ambiguity matter.
Centralize the parts that must remain consistent: data definitions, core tooling, pricing guardrails, evaluation standards, and foundational enablement. Let segment plays, partner motions, and contextual field execution stay closer to the teams that understand them. This avoids two common extremes: every function buying its own disconnected agent, or a central AI group becoming the queue for every revenue experiment.
Pair gated releases with a 24-26 month design horizon. The longer view is not a promise that you can forecast models or markets precisely. It forces you to ask what breaks at higher volume, which capabilities must remain centralized, how roles will change, and what data or evaluation debt would block the next level of autonomy. The release in front of you can remain small while the operating architecture anticipates scale.
The first leadership review should produce five concrete artifacts: a workflow map, an agent charter, a human handoff contract, an evaluation set, and a dashboard with a named decision-maker for every release gate. If the meeting ends with only a vendor shortlist, the transformation has not yet been designed.
Key takeaways
If an agent cannot read and update shared customer state, it is an interface attached to the revenue system, not a worker inside it.
If the handoff criteria and return path are unclear, the agent’s autonomy is already too broad.
If production failures do not enlarge the evaluation suite, the organization is collecting incidents rather than compounding learning.
If a buyer must repeat discovery after escalation, the context architecture has failed even when the AI behaved politely.
If the roadmap is organized around tools instead of revenue-state transitions, no executive truly owns the transformation.
At your next go-to-market review, choose one live inbound path and walk a real lead through it from trigger to next customer outcome. Assign state, evidence, permissions, and an exception owner at every transition. Automate only the steps whose failure you can detect and recover from. That is how you earn the right to give AI more of the revenue journey.
Over the past few years, I’ve led cross-functional teams to deploy agentic AI in production, and I’ve learned that success rarely hinges on the model alone. It comes from methodically designing the right workflows, instrumenting every step, and building a feedback loop that compounds. Learn how companies like Replit are consolidating workflows, creating one-person departments, and building systems for scale with Amplitude.
When I talk about AI agents, I’m describing software that behaves like a focused teammate—owning a clear job to be done end-to-end. In practice, that means consolidating fragmented tasks into a single accountable “one-person department,” then giving it the context, tools, and analytics to perform reliably. This is how agentic AI moves beyond demos into durable business impact.
I start with outcomes, not algorithms. I map a driver tree from business goals (e.g., lower response time, higher activation, better retention) to the specific moments an agent can influence. This outcome-first alignment keeps scope tight, informs guardrails, and grounds the value proposition in measurable change instead of vanity metrics.
Next, I define the workflow the agent will fully own. I look for high-volume, rules-adjacent processes—think lead qualification, support triage, or billing inquiries—where clear decision criteria already exist but human time is the bottleneck. I document triggers, inputs, decision points, and handoffs, then design the ideal-state flow the agent will run autonomously, with transparent escalation paths to humans.
On architecture, I favor a retrieval-first pipeline to keep responses accurate and current. I scope the knowledge base, implement context window management, and standardize tools the agent can call (search, CRM actions, ticket updates). For teams new to this, I coach “LLMs for product managers” fundamentals so we make sensible trade-offs between speed and reliability rather than chasing model-of-the-week headlines.
Instrumentation is where the system becomes self-improving. I use Amplitude analytics and an Agent Analytics schema to track intent detection, tool usage, resolution rate, time-to-resolution, deflection, and escalation causes. A unified analytics platform lets me connect agent outcomes to core product metrics—activation, retention, and conversion—so we can see the real revenue and experience impact, not just local efficiency gains.
To validate impact, I run A/B testing when traffic allows, setting a minimum detectable effect (MDE) upfront to avoid inconclusive reads. In lower-volume scenarios, I lean on eval-driven development: curated test sets for edge cases, scenario-based regression suites, and error taxonomies that accelerate iteration. Feature flags let us stage capabilities safely (shadow mode, assistive, autonomous) while we monitor deltas before full rollout.
Reliability and trust are designed in from the start. I apply AI risk management practices—privacy-by-design, data governance, and policy-aligned prompt templates—paired with observability to trace decisions. Clear escalation policies, incident management runbooks, and human-in-the-loop checkpoints ensure the agent fails safe, not silently.
Shipping cadence matters. I use CI/CD to increase deployment frequency, keep prompts and tools versioned, and gate risky changes with targeted rollouts. As patterns stabilize, we scale horizontally to new use cases, sharing core capabilities (retrieval, analytics, guardrails) as a platform. This is how “one-person departments” multiply without multiplying overhead.
Change management closes the loop. I partner with product trios and frontline teams to co-design prompts, set acceptance criteria, and define what “good” looks like in plain language. In-app guides and product tours introduce the agent’s role and limits, and structured feedback channels feed directly into our discovery and iteration rhythm.
The throughline of this playbook is simple: treat agents like real teammates with a job description, operating procedures, and performance reviews. With disciplined workflow design, a retrieval-first pipeline, and outcome-level instrumentation in Amplitude, agentic AI stops being a science project and starts compounding into durable product-led growth.
Inspired by this post on Amplitude – Perspectives.
You have customer interviews, an AI prototype, and a launch request. What you may not have is a defensible chain connecting them. The prototype can look convincing while the team still disagrees about the customer problem, the acceptable failure rate, the limits of automation, and what should happen when the model or a connected tool fails.
A durable AI product workflow makes those decisions explicit. It connects customer evidence to a bounded opportunity, the opportunity to an interaction model, that model to an evaluation contract, and the contract to a guarded production release. You should be able to trace every automated action backward to a customer need and forward to a metric, an owner, and a recovery path.
Turn interviews into an opportunity map, not a feature request
AI products often go wrong before anyone writes a prompt. A customer describes a slow or frustrating task, someone proposes an assistant, and the proposed interface quietly becomes the problem definition. The team then tests whether it can build the assistant instead of whether solving that part of the workflow changes the customer’s outcome.
Start by defining the discovery boundary. Name the user, the workflow, the outcome the user is trying to reach, and the part of that outcome your product could reasonably influence. Keep interviews in the same outcome or product space when you synthesize them. A small batch of three interviews can be enough to produce a useful first draft, but it is not a universal saturation threshold or proof that you understand the market.
The sequence of synthesis matters. Analyze each interview on its own before looking for patterns across interviews. That preserves the situation, sequence, and meaning around each customer’s comments. If you combine transcripts immediately, repeated vocabulary can appear more important than the underlying context, and an unusual but consequential problem can disappear into the average.
Write the outcome anchor. State whose behavior or result should change. Avoid a feature-shaped outcome such as “increase use of the AI assistant.” A better outcome describes progress in the customer’s work.
Create one snapshot per interview. Capture the customer’s goal, the relevant sequence of events, key moments, obstacles, current workaround, and evidence supporting each inferred opportunity.
Separate observation from interpretation. Preserve what happened and what the customer said separately from the team’s explanation of why it happened. Label uncertainty instead of filling gaps with generated prose.
Synthesize across snapshots. Look for shared opportunities, meaningful differences, dependencies, and contradictions. Similar wording does not automatically mean the same need.
Organize opportunities before proposing solutions. Build an opportunity solution tree or equivalent map that connects the product outcome to customer opportunities. Keep solution ideas outside the opportunity labels.
Review the generated structure as a team. Ask what was merged incorrectly, what was missed, what lacks evidence, and which branch reflects a solution disguised as a need.
AI is useful here as a first-pass analyst, not as an authority. It can extract moments, propose opportunity statements, and suggest a hierarchy. Human reviewers contribute product context, recognize important exceptions, and challenge confident-looking inferences. The strongest practical model is an AI-generated draft that the team refines.
Your exit gate for discovery is not a polished tree. It is agreement on a selected opportunity, the evidence behind it, the customer outcome it should influence, and the opportunities deliberately excluded from the current scope. If the team cannot explain those choices without mentioning a model or interface, it is not ready to prototype.
Choose assistance or autonomy before choosing the architecture
The next decision is not which model to use. It is what responsibility the product will accept. An LLM can generate or classify content. An agent wraps model behavior in a workflow that plans, uses tools, retains relevant state, and attempts to complete an outcome. That difference changes the customer promise, the evaluation plan, the permission model, and the consequences of failure.
Decision
Copilot
Agent
Best task shape
High-context work that benefits from judgment, nuance, or brand voice
Bounded, tool-heavy work with a verifiable completion state
Customer promise
Drafts, explains, recommends, or accelerates
Completes an agreed task within a defined scope
Human role
Reviews and commits the result
Sets policy, handles exceptions, and approves sensitive actions
Default permissions
Read, retrieve, and propose
Narrowly scoped tool access, including only the writes required for the task
Primary proof
Useful, grounded output that improves the user’s work
End-to-end task success without unacceptable actions or loops
Failure consequence
A poor suggestion reaches the reviewer
A poor decision can propagate into another system
When the task still depends on tacit knowledge or subjective review, start with a copilot. When it is bounded, tool-heavy, and objectively checkable, consider an agent. The safer product progression is to start assistive and grant autonomy only after success is measurable. Autonomy should be earned capability by capability, not declared at the product level.
You can make that progression concrete without redesigning the entire experience. Let the product draft first. Then let it recommend a plan and show the evidence behind the recommendation. Next, allow reversible actions through a narrow tool whitelist. Keep approval immediately before actions that affect customers, money, permissions, or durable data. Expand the scope only when production evidence supports the previous boundary.
Once the responsibility is clear, define the architecture around it:
Authoritative context: retrieve relevant product, account, policy, or workflow information before asking the model to decide. A retrieval-first pipeline reduces dependence on whatever happens to be encoded in model weights.
Explicit scope: state the role, allowed objectives, prohibited actions, and conditions that require escalation.
Controlled tools: expose only the operations needed for the selected job. Apply unit limits and validate tool inputs outside the model.
Deliberate memory: separate temporary working state, durable customer facts, and governing policy. Do not treat the entire conversation history as an undifferentiated memory store.
Visible checkpoints: show the user what will happen, what data will be used, and which action requires approval.
Traceable execution: record retrieval results, model and prompt versions, tool calls, approvals, guardrail events, and final task status.
This architecture is more durable than a large prompt because each component has a distinct failure mode and owner. Retrieval can be evaluated for evidence quality. Tools can be tested deterministically. Policy can be reviewed independently. The model remains important, but it no longer carries responsibilities that ordinary software can enforce more reliably.
The exit gate is a written responsibility boundary. The team should be able to say what the product may read, what it may write, what it must never do, when a person intervenes, and how successful completion is verified. If any answer is “the model will decide,” the boundary is still incomplete.
Write the evaluation contract before optimizing the prompt
A compelling demo proves that a path can work. It does not establish how often it works, which inputs break it, whether its evidence is trustworthy, or whether it completes the customer’s job at an acceptable cost. Prompt iteration without an evaluation contract tends to optimize whatever the last reviewer noticed.
Write the contract in product language. For each target task, define the eligible input, the expected outcome, the evidence the product may use, allowed actions, prohibited outcomes, completion criteria, escalation conditions, and fallback. Add latency and cost limits chosen for your product economics. There is no universal threshold that makes an AI workflow production-ready; the important discipline is setting the threshold before seeing launch results.
Build the evaluation set from discovery evidence. Include representative customer inputs, important workflow variations, ambiguous cases, missing context, conflicting instructions, tool failures, and requests the product must refuse or escalate. Remove or protect sensitive data according to your governance rules. Every case should identify the acceptable outcome, not merely an ideal sentence, because multiple responses may solve the same job.
For copilots, measure the quality of assistance
Time to first token: how long the user waits before the response begins.
Response latency: how long the useful result takes to complete.
Groundedness: whether material claims are supported by the authoritative context supplied to the model.
User satisfaction: whether the assistance was useful in the actual workflow, not merely fluent.
Task impact: whether the user completes the selected job faster, with less effort, or with fewer corrections, using the outcome defined during discovery.
For agents, measure the whole execution
Task success rate: successfully completed eligible tasks divided by all eligible attempts. Define completion in the customer’s system of record where possible.
Steps per task: the number of model and tool steps required to finish. A rising count can expose inefficient planning or repeated work.
Tool error rate: failed, rejected, or malformed tool calls relative to attempted calls.
Loop detection: executions stopped because the agent repeated actions or failed to make progress.
Guardrail triggers: attempts blocked or redirected by policy. A trigger is diagnostic evidence, not automatically a success or a failure.
Human escalation: tasks handed to a person because the agent lacked permission, confidence, context, or a valid recovery path.
Cost per successful task: total execution cost divided by successful completions. Cost per request can hide expensive retries and failed runs.
Containment rate: eligible tasks completed within the automated workflow without human handling. Publish the eligibility and escalation rules with the metric so teams do not improve containment by narrowing the denominator invisibly.
These agent analytics complement rather than replace end-to-end task success. A fast response can still be wrong. A low tool error rate can coexist with a bad plan. High containment can be harmful if the agent completes the wrong task. Choose one outcome metric, pair it with quality and safety constraints, and retain the diagnostic metrics needed to find the cause of failure.
Route failures to the component that can fix them. Unsupported claims point first to retrieval and grounding. Correct plans with failed actions point to tool integration. Repeated steps point to orchestration or stopping logic. Frequent, legitimate escalations may mean the autonomy boundary is too broad. High model scores with low customer satisfaction should send the team back to the opportunity definition or user experience.
The exit gate is a versioned evaluation suite with release criteria, prohibited outcomes, an approved cost ceiling, and named escalation rules. Run it against every material change to the model, prompt, retrieval configuration, tool contract, or policy. Treat prompts and evaluation cases as product assets under version control, not as text pasted into a dashboard.
Release through gates and design the failure path
Deployment is where an AI capability becomes a product promise. The team now has to manage model variability, external tool behavior, changing knowledge, permissions, cost, and customer expectations at the same time. A launch plan that covers only the happy path is unfinished.
Put the capability behind a feature flag. Separate deployment from exposure so the team can stop new executions without waiting for a code release.
Open a gated beta around one bounded job. Limit the eligible users, tool permissions, data scope, and advertised promise. Make it clear whether the product recommends an action or performs it.
Use a canary for broader production traffic. Expand exposure gradually while comparing task success, guardrail events, tool errors, latency, escalation, and cost per successful task with the release criteria.
Change one material layer at a time when practical. Simultaneous changes to the model, prompt, retrieval index, tools, and policy make regressions difficult to attribute.
Expand only after the previous boundary is stable. More users, more tools, and more autonomy are separate risk decisions. Do not bundle them into one rollout.
Keep rollback and fallback distinct. Rollback restores a known model, prompt, policy, or tool version. Fallback gives the customer a safe alternative when the AI path is unavailable.
Model unavailable: preserve the user’s work, explain that automation is unavailable, and offer the established manual path.
Retrieval unavailable or evidence missing: do not silently generate an ungrounded answer. Ask for the missing context, provide a limited response, or escalate.
Write tool fails: stop, report the actual system state, and reconcile before retrying. Blind retries can duplicate durable actions.
Execution stops making progress: terminate the loop at the configured limit and hand over the trace rather than consuming resources indefinitely.
Policy or permission check fails: block the action, preserve the audit record, and route the user to an authorized path.
Tool behavior changes: disable the affected capability until its contract and evaluation cases pass again.
Privacy and auditability belong in the release gate, not in a later compliance review. Document what customer data enters prompts, retrieval, memory, and logs; who can access it; how long each class is retained; and how deletion propagates. For actions affecting customers, money, permissions, or durable data, preserve enough detail to reconstruct the input, retrieved evidence, model and prompt version, tool parameters, approval, guardrail result, and final system state.
The operating stack also needs an ownership decision. Build the workflow logic, data model, and user experience that encode your differentiated value. Consider buying undifferentiated capabilities such as observability, prompt versioning, red-team infrastructure, and policy enforcement when an external component meets your control and governance needs. This build-versus-buy boundary keeps product attention on the parts customers actually choose you for without treating commodity infrastructure as strategically unique.
The production exit gate should require a visible scope statement, passing evaluations, a feature flag, a rollback target, a customer-safe fallback, usable audit traces, an incident owner, and a tested escalation route. If the team cannot explain what the customer sees during failure, it has not finished designing the feature.
Keep discovery, evaluation, and production in one learning loop
Once the product is live, production behavior becomes new discovery input. That does not mean replacing customer conversations with dashboards. Metrics show where the workflow breaks; customer evidence explains what the break means and whether fixing it matters.
Review failures against the original opportunity map. Concentrated escalation around one scenario may reveal an opportunity that was hidden during initial synthesis. High groundedness with low satisfaction may indicate that the product answered accurately but tackled the wrong job. A growing step count may expose orchestration waste, while a rising tool error rate points to integration reliability. If cost per successful task increases, inspect failure and retry paths before making the model cheaper; optimizing unit cost cannot rescue an unsuccessful workflow.
Every meaningful production failure should produce at least one durable change: a corrected opportunity assumption, a new evaluation case, a narrower permission, a tool-contract test, a policy update, a clearer interaction, or a revised fallback. That is how customer discovery and operational learning remain connected instead of becoming separate product and engineering rituals.
Key takeaways
Synthesize each customer interview separately before looking across interviews, then review the AI-generated opportunity structure with human judgment.
Select a customer opportunity before selecting the AI interface. A fluent prototype is not evidence that the underlying job matters.
Use a copilot for judgment-heavy work and consider an agent only for bounded, tool-heavy tasks with verifiable completion.
Define task success, prohibited outcomes, escalation, cost, and fallback before optimizing prompts or choosing a model.
Measure copilots as assistance and agents as end-to-end execution. Do not mistake latency, containment, or tool-call success for customer success.
Release behind flags, expand through gated exposure, and rehearse rollback, fallback, and incident paths before granting more autonomy.
At your next AI product review, ask to see the outcome and opportunity map, the responsibility boundary, the evaluation contract, and the rollout and recovery plan. If one is missing, pause the launch decision at that handoff. Closing that gap is usually more valuable than adding another prompt, tool, or autonomous step.
