If nobody on our team trains the Agent on how to communicate, it will sound like an LLM when it speaks to customers—because it is one. I never want a customer to feel like they’re talking to a machine that doesn’t get them. That’s why I treat conversation design as a core product capability, not an afterthought.
Conversation design is an emerging discipline in AI-first support teams built to solve this exact problem. In practice, I make someone explicitly own how the Agent communicates—tone, structure, level of detail, customer experience, and the handoff and escalation process—because that’s where trust is won or lost.
When there’s no clear owner and no explicit guidance, the Agent starts making its own choices. I’ve seen it over-explain when a short answer would do, reply in a flat tone when a customer is frustrated, or trigger a handoff too late. None of those are model problems; they’re design problems.
The cost is measurable. Customers who get awkwardly structured responses won’t trust the answer—even when it’s accurate—so they escalate to a human to hear the same thing phrased differently. Others will skip the Agent entirely. And when the Agent does hand off, a poor transition means the support rep inherits a frustrated customer. Every one of these outcomes is avoidable; conversation design exists to prevent them.
I’ve seen A/B tests where a warmer, more conversational opening message meaningfully lifted customer satisfaction—CSAT moved from 72.8% to 78.4%. A single design change, applied to the very first message, drove a measurable difference. That’s the kind of leverage I look for as a product leader.
Here’s the scope I use when I talk about conversation design—five areas that shape the customer experience end to end:
1) Tone and personality: Define the Agent’s voice, level of detail, and how formal or casual it should sound—and specify where that register adapts to the situation (for example, urgent access issues versus exploratory product questions).
Design how your AI agent talks. Set tone, style, and product naming rules, then preview replies instantly. Clear callouts showcase brand voice consistency and flexible formatting so your bot communicates like your team.
2) Response structure: Ensure the Agent matches the level of detail to the customer’s request, keeping answers tight when the ask is simple and expanding only when complexity demands it.
3) Handoff logic: Decide when to escalate, how to communicate the transition, and what context to carry over so the human teammate can help immediately without rework.
4) Interaction flow: Map how a conversation progresses—clarifying questions, answers, resolution, or handoff—and design for smooth pivots when customers change direction.
5) Response quality: Go beyond technical correctness to ensure answers feel clear, helpful, and on-brand. Accuracy without clarity erodes trust.
To put this into practice, I start with the feel of the conversation. Before tuning individual responses, I write down one tight paragraph describing the Agent’s voice. I don’t need a full brand bible—just a north star I can use to make consistent decisions about tone. The voice stays consistent, while the register adapts to the context: a locked-out customer needs directness and speed; a feature explorer might value more context and examples.
I design the handoff with extreme care because it’s one of the highest-friction moments. Customers shouldn’t have to re-explain anything. The support rep should receive the full conversation history, the underlying context, what the Agent already tried, and why the escalation happened. Even the phrasing matters—“Let me connect you with a teammate who can help with this” feels very different from a silent handover.
The new CX Score adds context to every conversation: a donut chart surfaces drivers like policy feedback and effort, while a side panel explains why this interaction earned a 3 based on signals from an AI agent chat.
I also build a failsafe. If the Agent can’t resolve the issue cleanly, a graceful fallback still gives the customer a smooth experience. A customer might be frustrated with AI at that point, but a well-handled transition can turn that around.
Follow-ups deserve the same rigor as handoffs. If someone drops mid-conversation—with the Agent or a human—how do we reach back out to confirm they got what they needed? Most teams miss this moment; customers don’t.
Another common pitfall is over-explaining. The Agent has access to a lot of information, and left unguided, it will overshare. The fix is simple: match the answer’s depth to the question. A password reset shouldn’t take three paragraphs; a complex integration might. When there’s more to offer, the Agent should ask before expanding.
I also design for the conversation the customer is actually having—not the script I wish they’d follow. Customers change direction, stack questions, or bring up unrelated follow-ups. The Agent should pivot with them, not force them back into a rigid flow. I also consider whether flows vary by channel and whether different segments merit distinct experiences.
On the instruction side, I keep guidance short. Teams often react to edge cases by adding more rules until the LLM is parsing paragraphs before it can reply. I’ve seen it everywhere. My rule: if it’s about content or information, it belongs in the knowledge base. If it’s about tone or handling specific situations, it belongs in the Agent’s instructions. “Be direct about pricing” does more than a paragraph explaining the philosophy behind your pricing communication strategy.
If you’re using Fin, much of this work happens in Guidance. It’s where conversation design takes shape, helping you define how the Agent should sound, how much it should say, and how it should respond in different situations.
On a crisp grid, 'Blueprint' appears as editable vector paths, underscoring a methodical plan. The image promotes the AI Agent Blueprint—a framework to launch and scale customer service automation with confidence.
Most teams won’t hire a dedicated conversation designer on day one—that’s fine. But someone still needs to own the Agent’s communication, even if it’s part of an existing role. I’ve often seen this start within support operations or knowledge management. As the Agent scales to more conversations, the responsibility becomes formal—and eventually becomes a dedicated role.
Here’s how I’d start, step by step:
1) Name an owner. Make accountability explicit; it doesn’t have to be a new hire.
2) Pick one conversation type that isn’t landing well. Look for cases where the Agent answered correctly but the customer still escalated or left negative feedback. If you’re using Fin, CX Score can help you surface these; it shows which topics and conversation types are scoring poorly and why, so you can see whether the issue is answer quality, customer effort, or something else.
3) Audit the Agent’s instructions. If they’ve grown beyond a few focused rules, trim them. Move content into the knowledge base and keep instructions focused on behavior.
4) Fix your worst handoff. Review a handful of conversations that escalated. Did the customer have to repeat themselves? Did the rep have enough context? Redesign that single transition first.
The impact of these small improvements compounds. A warmer opening can lift CSAT, trimming instructions makes responses sharper, and a better handoff prevents reps from inheriting frustrated customers. None of this requires new knowledge—just someone paying close attention to the conversation itself and designing it with intention.
When teams evaluate AI Agent options for customer service, I often see the rigor aimed at the wrong subset of criteria. After leading and observing dozens of proof of concept (POC) efforts with our customers and prospects, I understand why performance—accuracy scores, resolution rates, and benchmark tests on curated datasets—soaks up most of the attention. But those indicators alone won’t guarantee success once you leave the sandbox and face real customers.
If your POC only proves that the AI “works,” you’re missing the bigger picture. Here’s what else I look for to make the best long-term decision.
How does it handle your real-world setup?
Performance is table stakes, but it has to reflect the messiness of an actual support environment. The best-performing Agents don’t just get answers right—they exhibit resilient, human-like behavior under pressure. I watch how the Agent behaves when it doesn’t know an answer: does it recover or spiral? Does it stay on track through multi-step requests, and how gracefully does it hand off to human agents? If your knowledge base depends on a retrieval-first pipeline, test cross-source retrieval and grounding—not just single-document lookups.
When I build evaluation scenarios, I put the Agent through its paces with a broad, realistic mix:
Multi-turn queries that require the Agent to carry context across a conversation, not just answer isolated questions.
Vague or fragmented inputs, like typos, grammatical errors, and incomplete questions, because that’s how customers actually write.
Edge cases and sensitive scenarios, like billing disputes, frustrated customers, and questions that sit at the boundary of what the Agent is trained on.
Different phrasings of the same question. An Agent that handles one version well but fails on a rephrasing has a knowledge problem, not a performance problem.
Queries that require pulling from multiple knowledge sources. Real issues are rarely answered by a single help article, and an Agent that can only handle single-source questions will hit a ceiling fast.
Multilingual conversations, if your customer base requires it. Performance can vary significantly across languages and it’s better to discover that in testing than in production.
This preparation is worth the effort. Any Agent can look impressive in a demo; what matters is how it holds up as part of your team, serving your customers in production.
What does it feel like to interact with the Agent?
Two AI Agents can post the same quantitative scores—resolution rates, containment rate, and more—and still deliver very different customer experiences. Resolution rate tells me whether the Agent finishes conversations; it says nothing about how customers felt during them. I deliberately assess the experience, not just the outcome, because conversation design shapes trust and brand perception.
Here’s what I look for to ensure the AI Agent is enjoyable to interact with:
Is the tone natural and on-brand, or does it feel robotic and generic?
Does it build trust early in the conversation, or does it create friction that makes customers want to immediately request a human?
When it doesn’t know the answer, does it handle that gracefully?
When it hands off to a human, is that transition seamless, or does the customer feel abandoned?
As George Dilthey at Clay put it when evaluating their AI setup: “Keep what’s important to your business up front and center. For us, that was transparency and control over the customer experience.”
That framing is exactly right. The Agent represents your brand in every conversation. Customers don’t experience “accuracy,” they experience conversations. An Agent that’s technically accurate but tonally off-brand will erode customer trust over time.
I make the experience dimension explicit in my POCs. I have people on my team—and when possible, a small cohort of real customers—interact with the Agent under realistic conditions. Then I ask how it felt, not just whether it worked.
Can you keep improving it after launch?
This is the dimension most teams don’t evaluate at all, and it’s possibly the most important one. Choosing an Agent that works today and ensures you can continuously improve the customer experience over time requires more than a functional demo. You’re buying a system that must get better every week, not just during the first sprint.
The feedback loop
Can your team easily review conversations and identify where the Agent is underperforming? Can you pinpoint specific gaps (missing knowledge, incorrect tone, poor handoff decisions) and act on them quickly? The faster the loop between “something isn’t working” and “we’ve fixed it,” the more value compounds over time. In practice, that means instrumenting conversations, leveraging Agent Analytics, tagging misroutes and tone slips, and running targeted evals on known failure modes.
The speed of iteration
When you identify a gap, how quickly can you address it? This is partly a question of tooling (how easy is it to update knowledge, refine guidance, adjust behavior?) and partly a question of team capability. The teams getting the most out of AI are the ones that have changed how they operate and made continuous improvement a part of their everyday work. They’ve committed to going all-in for the long term, not just the first few weeks when launching their AI Agent. We treat this as eval-driven development: automate evaluations that mirror real tickets, tighten prompt engineering and retrieval settings, and ship small fixes daily.
The vendor partnership
The vendor behind the Agent matters just as much as the solution itself. You’re choosing a partner for transformation that will help you evolve how your business delivers customer experience. Ask:
How does customer feedback influence the product roadmap, and can they show you examples?
If you have feedback on limitations or weaknesses, do they engage transparently or get defensive?
What kind of support will you get post-launch?
Are they shaping where AI customer experience is going, or reacting to what others are building?
How a vendor responds to those questions tells you more about the long-term relationship than any benchmark result.
What a good POC proves
If your POC only proves “the AI works,” you haven’t done enough. A strong proof of concept tests performance in realistic conditions, evaluates the experience from the customer’s perspective, and validates the system that will support continuous improvement after launch. Done well, it sets you up for long-term operational success and builds organizational AI readiness—not just a flashy demo.
Most teams ship AI agent personalities by accident—emergent quirks, brittle prompts, and uneven behavior. We refused to let that happen. From day one, we treated personality as a first-class product surface, one that should be designed, instrumented, and iterated with the same rigor as any core capability.
Learn how we designed Global Agent’s personality and fine-tuned its inquisitiveness and helpfulness using Agent Analytics.
In my role leading product at HighLevel, Inc., I framed our approach around agentic AI and conversation design: personality is not “flavor text”; it is the control system for how an agent interprets context, asks questions, and decides when to act. Our product strategy prioritized clarity, empathy, and consistency—so the agent would be curious enough to resolve ambiguity without becoming interrogatory, and helpful enough to move work forward without overstepping.
We made that intent measurable. Using behavioral analytics, we defined operational signals such as clarification-question rate, resolution-path efficiency, and escalation quality. We combined eval-driven development with targeted A/B testing to compare prompt patterns and tool strategies, ensuring each change had a clear hypothesis and measurable outcome.
To calibrate inquisitiveness, we mapped decision points where the agent should ask follow-ups versus proceed autonomously. Prompt engineering codified those thresholds, while a retrieval-first pipeline reduced unnecessary questions by improving context completeness up front. When the agent did ask, we constrained tone and cadence to keep queries concise, respectful, and progress-oriented.
To enhance helpfulness, we prioritized precise action-taking and unambiguous guidance. Context window management preserved relevant facts without diluting intent, and guardrails aligned with AI risk management principles ensured the agent stayed within policy, privacy, and compliance boundaries. The result was an assistant that resolved more tasks end-to-end, with fewer stalls and clearer handoffs when human help was warranted.
Agent Analytics became our nervous system. We instrumented every dialog turn to attribute outcomes to design choices, then used driver trees to connect micro-behaviors to macro results like time-to-resolution and customer satisfaction. This closed-loop view let us ship confidently, knowing which levers improved helpfulness, which sharpened curiosity, and which merely added noise.
Process mattered as much as tooling. Product trios ran continuous discovery with customers to surface edge cases—ambiguous intents, multi-intent turns, and sensitive scenarios—while our engineering partners operationalized experiments with clean rollback paths. We favored small, testable changes over sweeping rewrites, building momentum and trust with each iteration.
The payoff is a personality that feels consistent across use cases: curious when clarity is missing, decisive when action is obvious, and transparent when limits are reached. Users experience fewer dead ends, faster resolutions, and a brand voice that shows up the same way every time—because it was defined, measured, and improved on purpose.
If you’re building agentic AI, don’t leave personality to chance. Treat it like a product: set clear outcomes, instrument deeply with Agent Analytics, and iterate with eval-driven development and A/B testing. That’s how curiosity becomes a feature, helpfulness becomes a habit, and your agent becomes reliably, intentionally excellent.
Inspired by this post on Amplitude – Best Practices.
Your AI sales agent answers the pricing question, then recommends the wrong plan. It identifies a promising buyer, then sends the conversation to the wrong queue. If the underlying facts, decision rules, or routing policy are missing, another prompt adjustment cannot fix the problem.
You need a knowledge operating system, not a larger folder of sales collateral. The goal is to give the agent the smallest reliable path from a buyer’s question to an accurate answer, an appropriate recommendation, useful qualification, and the correct next step.
Key takeaways
Separate product facts, decision guidance, and execution policy. Each solves a different part of the sales conversation.
Turn long documents into focused, approved knowledge records with an owner, scope, effective date, and explicit boundaries.
Launch a complete sales motion for a narrow set of buyer intents before trying to document everything.
Test recommendations, qualification, routing, and escalation behavior, not just whether the agent can repeat a correct sentence.
Convert unanswered, incorrect, and disengaged conversations into a managed improvement queue.
Design for answering, recommending, qualifying, and routing
A conventional knowledge base helps someone find information. An AI sales agent has a harder job: it must interpret a buyer’s situation and decide what to do with the information it retrieves.
A language model does not inherently know your current plans, qualification criteria, commercial boundaries, or customer-specific use cases. That context is unique to your business and must be made explicit. Fluency cannot compensate for a missing policy.
I find it useful to divide sales knowledge into three layers:
Knowledge layer
What it contains
What the agent should do with it
Typical failure when it is missing
Product facts
Pricing, plan structure, capabilities, limitations, availability, and supported use cases
Give a direct, accurate answer
The agent guesses, gives a vague response, or repeats obsolete information
Decision guidance
Plan-fit logic, relevant constraints, case studies, approved comparisons, and the context behind product facts
Explain which option fits and why
The answer is technically correct but does not help the buyer decide
Execution policy
Qualification questions, required fields, routing conditions, escalation rules, and actions the agent may take
Advance the conversation within defined authority
The agent collects irrelevant details, makes an unsupported commitment, or routes the buyer incorrectly
Facts answer, “What does the product do?” Decision guidance answers, “Is this appropriate for my situation?” Execution policy answers, “What should happen next?” Audit your knowledge against all three questions.
Set a hard boundary around commercial exceptions. The agent should not infer an unlisted discount, invent a contractual commitment, or turn an internal hypothesis into a buyer-facing claim. It should state the approved terms, gather the information required by policy, and route the exception to an authorized person. A plausible but unauthorized promise can create financial and legal exposure.
Turn scattered documents into governed knowledge
Build sales-ready knowledge records
A long document can be correct and still be poor input for an agent. Pricing may be buried below an obsolete introduction. A feature table may omit the condition that changes plan fit. A battlecard may combine approved facts with a rep’s unverified notes.
Convert those documents into focused records. Each record should cover one buyer intent or one tightly related decision. Use a consistent template:
Buyer intent: The question or decision this record addresses, including common alternative phrasings.
Approved answer: The direct response the agent may give without qualification.
Decision context: Why the fact matters and when it changes the recommendation.
Constraints and exceptions: What the answer does not cover, including conditions that require clarification.
Next question or action: The appropriate follow-up, qualification step, route, or escalation.
Scope: The plans, markets, customer types, channels, or agents allowed to use the record.
Evidence location: The canonical product, pricing, or policy record from which the answer was derived.
Owner and approver: The people accountable for accuracy and authorization.
Lifecycle metadata: Effective date, review status, and whether the record replaces an earlier version.
For example, a record about plan fit should not stop after naming a plan. It should state the relevant requirement, identify the condition that changes the answer, give the agent an approved follow-up question, and define where to route a buyer whose situation falls outside the standard policy. The recommendation then becomes reproducible rather than improvised.
Keep buyer language in the record. Prospects rarely use your internal taxonomy, and the same intent may appear as a product question, an outcome question, or a comparison. Alternative phrasing helps the retrieval layer recognize that these expressions belong to the same approved answer.
Create an authority hierarchy
A centralized repository is valuable only if the agent can distinguish current authority from historical residue. Define the hierarchy before connecting more content:
Designate one canonical record for each product fact, commercial rule, or routing policy.
Make approved sales explanations point back to that record rather than becoming independent versions of the truth.
Treat scripts and examples as phrasing aids unless they are explicitly approved to carry facts.
Keep drafts, call notes, chat fragments, and retired material outside the agent’s usable knowledge until they are reviewed.
Do not let the agent reconcile conflicting records by choosing the newest upload or blending the language. When two approved items disagree, the safe behavior is to withhold the disputed claim, follow the defined escalation path, and send the conflict to its owner.
Ownership should also be specific. A knowledge owner maintains the record. A domain approver authorizes sensitive claims. An operations owner monitors how the agent uses the record in conversations. One person may hold more than one role, but every role needs a name rather than a department-shaped placeholder.
Target knowledge by audience and action
Internal knowledge is not automatically buyer-facing knowledge. A qualification score may guide routing without being disclosed. A battlecard may help frame an approved comparison without exposing internal commentary. A security question may require an authorized answer rather than the agent’s summary of a sales note.
Mark each record as buyer-answerable, decision-only, action-only, or restricted. Then expose only the appropriate material to each agent, channel, market, and sales motion. This kind of centralization and content targeting reduces duplication while keeping internal policy separate from the words a prospect sees.
Launch the smallest complete sales motion
Trying to clean every sales document before launch creates a long project with no conversational evidence. Launching with disconnected FAQs creates a different failure: the agent answers isolated questions but cannot move the buyer forward.
The better unit of scope is a complete motion for a bounded set of intents. For each selected intent, the agent needs an answer, the relevant fit logic, the next qualification question, a route or resolution, and an escalation path.
Prioritize by demand and consequence
Start with questions that appear repeatedly, delay buyers, consume sales time, signal meaningful intent, or cause material damage when answered incorrectly. Pricing, plan differences, core capabilities, common use cases, qualification, and routing are natural candidates when they dominate your actual inbound conversations.
Two simple calculations help quantify repetitive work: team time reclaimed = average response composition time x question frequency, while buyer wait avoided = number of prospects asking x average response time. These calculations are useful for prioritizing knowledge work, but neither should be presented as revenue without downstream evidence.
Add consequence to the ranking. A frequent low-risk question may save time, but an infrequent error involving price, eligibility, security, or a contractual promise may deserve earlier treatment. Frequency tells you where the volume is. Consequence tells you where control matters.
Your initial release is ready when the selected motion has:
Approved answers for the recurring and commercially important questions in scope.
Plan-fit or use-case guidance where the buyer needs a recommendation rather than a fact.
Explicit qualification fields and follow-up questions.
Routing rules for the standard paths.
A visible no-answer and human-escalation path.
Owners and lifecycle metadata for every active record.
A representative evaluation set based on real buyer language.
Test the conversation, not the sentence
A retrieval test can show that the right paragraph was found. It cannot show that the agent asked the necessary follow-up, respected a restriction, or routed the lead correctly. Evaluate the complete interaction.
Your test set should include direct questions, paraphrases, multi-part questions, ambiguous requests, outdated assumptions, missing qualification details, requests for exceptions, and scenarios that should be escalated. For each case, check:
Is every factual claim aligned with approved knowledge?
Does the response answer the buyer’s actual question before adding detail?
Does the recommendation apply the right conditions rather than matching a keyword?
Does the agent ask only for information required by the qualification policy?
Does it avoid unsupported commitments and internal-only language?
Does the final route or escalation match the execution rule?
Record pass or fail at the behavior level and attach a reason to every failure. Do not allow a strong average score to conceal a severe pricing or policy error. High-consequence failures should block that behavior from release until the knowledge or policy is corrected.
Use a controlled launch with an obvious human path. The point is to begin collecting real conversational evidence early, not to claim autonomy before the boundaries are reliable. Fast deployment and continuous iteration work when the feedback loop is designed before traffic arrives.
Run the knowledge flywheel from real conversations
Once the agent is live, conversation failures become your most useful knowledge backlog. Do not place every poor result under a generic label such as bad answer. Classify the mechanism so the right owner can fix it.
Coverage gap: No approved record addresses the buyer’s intent.
Retrieval failure: The right knowledge exists, but the wrong record was selected or the right one was missed.
Freshness failure: The agent used information that should have been replaced or retired.
Guidance failure: The fact was correct, but the recommendation ignored relevant context.
Qualification failure: The agent skipped a required question, collected unnecessary information, or misread the answer.
Routing failure: The collected information was correct, but the next action did not follow policy.
Boundary failure: The agent disclosed restricted material or made an unauthorized claim.
Conversation failure: The content was accurate, but the response was unclear, repetitive, or poorly sequenced.
Turn each confirmed failure into a work item containing the conversation, intent, root cause, affected knowledge record, accountable owner, proposed change, and regression test. A change is not complete when the wording is edited. It is complete when the test passes, the approved version is published, and conflicting text is retired.
Track a small set of operational signals that lead to decisions:
Signal
What it tells you
What to do with it
Coverage
Which eligible buyer intents have an approved answer and action path
Add knowledge where demand and consequence justify it
Reviewed correctness
Whether sampled claims match the approved record
Repair facts, retrieval, or response generation
Knowledge conflicts
Where active records disagree or overlap ambiguously
Resolve authority and retire obsolete material
Qualification completion
Whether required information was collected for eligible conversations
Improve questions, field definitions, or sequencing
Routing compliance
Whether the next action matched the approved rule
Correct policy logic or integrations
Buyer progression
Whether the conversation reached its intended next step
Inspect guidance and friction, then validate changes against downstream outcomes
Content health
Which active records lack an owner, approval, scope, or lifecycle status
Repair governance before stale content becomes a live failure
Review these signals by intent and sales path. A global average can look acceptable while one plan, market, or routing branch fails repeatedly. Conversion can be a useful downstream outcome, but it is not proof that a knowledge change caused the result. Traffic mix, offer changes, seasonality, and human follow-up can also move it. Use controlled comparisons where practical and pair outcome data with conversation-level review.
Reserve a recurring weekly block for unanswered questions, disengaged prospects, high-consequence errors, and unresolved conflicts. Process pricing, product, and policy changes as immediate knowledge events rather than waiting for the review block. This is where knowledge management becomes an operating responsibility instead of a cleanup project.
Start with your recent inbound conversations. Choose the most repeated unanswered question, the most consequential incorrect answer, and one routing failure. Convert each into an owned knowledge record, add a regression test, and release only the behavior that passes. That small loop is the foundation of a sales agent you can trust with progressively more of the funnel.
You’ve probably been asked a deceptively simple question: should your Shopify store add an AI shopping agent? The hard part isn’t installing another chat widget. It is deciding whether the agent can help an uncertain shopper choose correctly without recommending unavailable products, misreading policy, or making a costly order change.
Treat this as a commerce-system decision, not a chatbot decision. A useful agent must connect conversation, live product data, cart behavior, checkout, and post-purchase service. The evaluation framework below will help you separate a persuasive demo from a system you can trust with customers and revenue.
The native test: can the agent read, reason, and act?
“Shopify-native” should describe an architecture, not a distribution channel. Being listed in an app marketplace or embedded in a storefront does not make an agent native. The meaningful test is whether Shopify remains the operational source of truth while the agent uses its data and APIs in the customer’s current context.
A concrete implementation shows how high that bar can be: a Shopify connection can expose products, variants, content, live inventory, order data, policies, and transactional APIs to the same customer-facing agent. That combination matters because a correct product description is still a bad answer if the relevant size is unavailable, and an accurate return policy is insufficient if the customer must start over somewhere else to use it.
I would evaluate an agent at four capability levels. A product that stops at the first level may still be useful, but it should not be presented internally as an autonomous commerce agent.
Capability
What the agent needs
What you should ask it to prove
Answer
Product content, store policies, and current catalog facts
Answer a precise product or policy question and identify the relevant item, variant, or rule
Recommend
Catalog relationships, inventory, conversation context, and the shopper’s constraints
Turn an ambiguous request into a short, reasoned shortlist instead of returning generic search results
Transact
Cart or order APIs, authentication, permissions, and confirmation controls
Update a test cart or prepare an order change while showing exactly what will happen before execution
Recover
Shared state across shopping, service, and human escalation
Resolve a support interruption and resume the customer’s original shopping task without asking for the same context again
Freshness is part of correctness. During a test, change the availability of a variant in Shopify and repeat the same shopping request. The agent should stop recommending that variant once the change is reflected in the connected system. Run a similar test with a policy update. A polished answer based on yesterday’s state is not a small quality defect; it can create a promise your operations team must later unwind.
Actions deserve an even stricter test. Ask the vendor to demonstrate the complete chain: customer identification, authorization, interpretation of the request, action preview, explicit confirmation, API execution, and a visible result. If any step is simulated, ask which one. A fast setup can reduce implementation effort, but it does not prove that the agent is accurate, observable, or safe.
Design the shopping dialogue around decisions, not keywords
Traditional ecommerce search works best when the customer already knows the product vocabulary. A shopping agent earns its place when the request is incomplete: a gift for a partner, a mattress for a particular sleep preference, or shoes that must work across road and trail conditions. The agent’s first job is not to produce an answer. It is to discover which answer would be useful.
A strong product-discovery dialogue follows a repeatable decision sequence:
Restate the customer’s job in plain language so a misunderstanding becomes visible early.
Identify hard constraints first, such as an in-stock variant, required use case, compatibility, budget boundary, or delivery requirement.
Ask only for information that could change the recommendation. A question that does not alter ranking or eligibility is conversational overhead.
Present a small shortlist and tie each option to the constraints the customer supplied.
Explain the meaningful tradeoff between the options instead of declaring a universal winner.
Offer the next useful action: compare details, select a variant, update the cart, or continue narrowing the choice.
This sequence turns conversation design into a product requirement. For every recommended item, your evaluation record should capture the customer’s stated need, the product facts used, the reason the item fits, the tradeoff disclosed, current variant availability, and the next question or action. That record gives your team something concrete to inspect when a recommendation is challenged.
Test whether the reasoning is responsive rather than decorative. Change one important answer while holding the rest of the conversation constant. If the customer switches from occasional use to daily use, removes a budget constraint, or requires an available color, the shortlist should change when that information is material. If the products remain identical regardless of the customer’s answers, the experience is probably search with conversational packaging.
The agent should also know when not to narrow further. Once the customer has enough information to choose, another question adds friction. Conversely, confidence should not be manufactured when catalog data cannot resolve the request. A safe response identifies the missing fact, asks for clarification, or hands the conversation to a person with the constraints already summarized.
Product cards can accelerate the final step, but the interface should preserve the reasoning that produced them. An image, name, and price answer “what is this?” The conversation must also answer “why does this fit me?” That is the difference between displaying inventory and assisting a decision.
Make shopping and support one customer state machine
A shopper does not experience your sales and support departments as separate funnels. The same person can compare products, ask about shipping, check an existing order, correct a variant, and return to buying in one session. Routing each intent to a separate tool forces the customer to reconstruct context at every boundary.
Model the journey as one state machine: discover, decide, transact, service, and resume. The agent can move between those states, but it should retain the customer’s goal, constraints, products considered, cart state, relevant order, completed actions, and unresolved question. That shared state is more important than whether the organization labels the current message “sales” or “support.”
Use different controls for different action classes:
Read-only actions, such as showing order status, should still require appropriate customer identification but do not change commercial state.
Reversible shopping actions, such as adding or removing a cart item, should be immediately visible and easy for the customer to undo.
Financially consequential actions, including refunds, paid order changes, and subscription updates, should require authentication, an exact action summary, explicit confirmation, and a durable result or receipt.
Ambiguous or unsupported actions should stop safely and transfer to a person. The agent must not treat conversational enthusiasm, silence, or an inferred preference as consent.
That last distinction protects both the customer and the business. A mistaken recommendation can usually be reconsidered. An executed refund or subscription change creates financial and operational consequences. If the system cannot preview and verify the exact action, keep that workflow read-only and let a trained person execute it.
The transition back to shopping also needs deliberate design. After resolving a delivery problem or order correction, the agent should restore the prior context and offer a relevant path forward. It should not force an upsell into every service interaction. The next best action after a serious order problem is often confirmation that the problem is resolved. Commercial momentum comes from reducing friction, not from ignoring the customer’s immediate priority.
When escalation is necessary, pass a structured handoff rather than a transcript dump. Include the detected intent, verified identity state, constraints already collected, products or orders involved, actions attempted, results returned by Shopify, and the unresolved decision. A human agent should be able to continue with the next question, not repeat the first one.
Measure incremental commerce value and operational risk
Chat conversion is an attractive metric and an easy one to misread. People who open a shopping conversation may already have higher intent than people who do not. Comparing those groups directly can credit the agent for demand it did not create.
Where traffic permits, randomize eligible storefront sessions between an agent experience and the existing experience. Measure the result across all eligible sessions, not only the visitors who choose to chat. That intention-to-treat view reduces self-selection bias and answers the executive question: what changed when the store made the agent available?
Use a balanced scorecard rather than a single conversion target:
Business outcomes: completed orders per eligible session, revenue per eligible session, average order value, checkout completion, and assisted revenue.
Decision outcomes: recommendation engagement, product-detail visits after a recommendation, add-to-cart actions, and successful comparison flows.
Downstream quality: cancellations, returns, exchanges, and contacts caused by a poor recommendation or incorrect expectation.
Service outcomes: successful action completion, repeat contact for the same problem, human escalation, and time to a confirmed resolution.
Agent quality: use of current catalog facts, in-stock recommendation rate, policy accuracy, clarification behavior, safe refusal, and correct tool execution.
Risk outcomes: unauthorized or incorrect actions, failed confirmations, customer complaints, policy exceptions, and cases requiring operational repair.
Conversion and average order value belong beside returns and cancellations. An agent can raise the initial basket by recommending a more expensive option while reducing customer fit. Without the downstream view, the dashboard rewards the sale and hides the repair.
Your event model should make the journey reconstructable. Useful events include agent opened, intent classified, clarification answered, recommendation shown, product selected, cart changed, checkout started, order completed, support action requested, confirmation received, action completed, and human handoff. Join these events through an appropriately governed session or order identifier so the team can inspect both funnel movement and individual failure paths.
Build the evaluation before the vendor demo
Create scenarios from your real catalog shape, policies, and failure modes. For each scenario, write the expected outcome and the failures that would make deployment unacceptable. Include:
An ambiguous shopping request that requires clarification.
Two products that appear similar but differ on a constraint customers care about.
An unavailable variant that would otherwise be the best match.
A question whose answer depends on store policy rather than product copy.
A conversation that moves from shopping to order support and back again.
A request that sounds actionable but lacks required authentication or confirmation.
An unsupported request that should trigger a safe handoff.
A catalog or policy change made after the agent’s initial synchronization.
Run the same scenarios repeatedly and record the underlying catalog state each time. You are testing consistency, grounding, and recovery, not literary elegance. A shorter answer that uses the correct variant and policy is more valuable than a fluent answer that improvises.
Expand autonomy in the order of consequence
A staged rollout lets evidence determine how much authority the agent receives:
Evaluate offline with approved scenarios and representative catalog data.
Launch read-only product discovery and policy answers with an obvious human fallback.
Add visible, reversible cart actions after recommendation quality is stable.
Introduce authenticated order-support workflows with previews and confirmations.
Enable financially consequential actions only after tool execution, auditability, exception handling, and operational ownership have been tested end to end.
Define the owner of each failure before launch. Product should own the intended customer behavior and success measures. Commerce or operations should own policy and workflow correctness. Engineering should own integration reliability and observability. Customer support should own escalation quality and emerging failure patterns. The exact reporting lines can vary; an unowned failure queue cannot.
Review both aggregates and conversation-level traces after release. Aggregate metrics tell you whether the experience is moving the business. Traces tell you why. A small cluster of incorrect variant recommendations or failed order actions can disappear inside a healthy conversion average while creating disproportionate customer harm.
Key takeaways
A Shopify-native agent should use live commerce data and governed APIs; storefront placement alone is not enough.
The agent’s product-discovery job is to uncover decision criteria, apply hard constraints, explain tradeoffs, and lead to the next useful action.
Shopping and support should share customer context, but the agent’s permissions must become stricter as actions become harder to reverse.
Conversation conversion is a diagnostic metric, not causal proof. Measure incremental results across eligible traffic whenever possible.
Pair conversion and average order value with returns, cancellations, incorrect actions, and operational repair costs.
Begin with read-only assistance and expand autonomy only after each workflow proves accurate, observable, recoverable, and properly owned.
Before you approve a purchase, bring a vague shopping request, a policy edge case, an unavailable variant, a mixed sales-and-support conversation, and a consequential order action into a live test store. If the agent cannot show where its answer came from, what it will change, and how it fails safely, you have found the next product requirement, not a detail to defer until launch.
You are not really deciding whether an LLM can chat about a menu. You are deciding whether it can turn a messy WhatsApp exchange into a correct, payable order without making the customer or venue staff repair its work.
That distinction changes the product. The hard parts are structured order state, deterministic commerce operations, response time, failure recovery, and venue-specific evaluation. Get those right and WhatsApp can become a genuine ordering channel. Get them wrong and you have a fluent chatbot sitting in front of an unreliable transaction.
Key takeaways
Define success as a confirmed, recoverable order in the system of record, not a conversation that sounded helpful.
Let the model interpret customer language, but keep menu data, prices, modifiers, delivery eligibility, payment state, and order commits behind deterministic tools.
Store the current order as structured state outside the transcript. A conversation is evidence of intent, not an order ledger.
Measure useful response time across the complete WhatsApp-to-POS path, then remove tool round trips and parallelize safe read operations.
Make item identification accuracy the primary trust metric, supported by guardrails for modifiers, payments, duplicate submissions, handoffs, and latency.
Evaluate every venue against its real menu and rules, then turn recurring configuration, tests, and operating procedures into reusable templates.
Define the product around a completed order
WhatsApp is the interface, not the product boundary. The product boundary should run from the customer’s first request to an order state that the venue can fulfill and the customer can verify.
Write an order contract before choosing models or orchestration frameworks. The contract is the minimum structured state required to fulfill, charge for, recover, and audit an order. It will usually include:
The venue and the applicable menu version.
Canonical item identifiers, quantities, and customer-facing item names.
Required and optional modifier selections, represented by identifiers rather than prose alone.
Fulfillment method, such as pickup or delivery.
The validated delivery result when delivery is requested.
A system-generated quote, including the values the customer must approve before payment or submission.
Payment-link and payment states, without treating a generated link as proof of payment.
Customer confirmation state, POS submission state, and the resulting order identifier.
The current owner of the interaction: agent, venue staff, or a defined recovery process.
The contract gives product, engineering, operations, and venue teams the same definition of done. It also exposes where autonomy is not yet safe. If the integration cannot validate a delivery zone, for example, the agent should collect the address and hand the order to a person. It should not infer eligibility from a conversational guess.
Order stage
The agent’s job
Condition before proceeding
Discover
Map natural language to menu candidates and explain relevant options.
One supported item is identified, or the agent asks a specific clarifying question.
Configure
Capture quantity, required modifiers, exclusions, and additions.
Every required choice is present and valid for that item.
Fulfillment
Resolve pickup or delivery and call the applicable eligibility checks.
The requested fulfillment method is supported for this order.
Quote and payment
Retrieve the authoritative quote and create the approved payment flow.
Prices and payment state come from the commerce system, not generated text.
Commit
Present the structured summary and submit the confirmed order once.
The customer has confirmed the current version and the POS returns a result.
Status and recovery
Report system-backed status or transfer the interaction with its context intact.
The response is tied to an order identifier or an explicit handoff owner.
Pay particular attention to the acceptance boundary. A friendly message such as “your order is being prepared” is an operational commitment. It must only appear after the system of record has accepted the order. If submission times out or returns an ambiguous result, the safe response is that confirmation is still pending, followed by a status check or human recovery. Guessing success can create duplicate orders, missed orders, and payment disputes.
You can still launch with partial automation, but name it accurately. Menu search, order drafting, and staff-assisted submission can deliver value while the integrations mature. The mistake is allowing the customer to believe the order was accepted when the product has only generated a plausible summary.
Keep the order deterministic even when the conversation is not
Customers do not speak in schemas. They change quantities, refer to items by incomplete names, add a second request before answering the first question, and revise earlier choices. Your architecture has to translate that non-deterministic conversation into structured, POS-compatible data without losing which version the customer actually approved.
My rule is simple: the model may interpret intent and propose an order-state change, but deterministic services must validate and commit it. The transcript should never be the only place where the current order exists.
A reliable turn can follow this sequence:
Load the current structured order, venue configuration, and relevant menu context.
Interpret the latest message as a proposed change: add, remove, replace, modify, confirm, cancel, pay, or request status.
Resolve referenced items and modifiers to canonical identifiers.
Call read-only tools for availability, configuration, fulfillment rules, or quotes as needed.
Validate the proposed change against required modifiers and venue rules.
Write a new order-state version and generate the next response from that validated state.
Use a separate, idempotent write operation when the customer confirms submission.
This design makes corrections much safer. If the customer says, “Make the second one large and remove the fries,” the agent should apply a state delta to the identified lines, validate the revised configuration, and show the updated summary. It should not regenerate the entire order from memory and hope that unrelated details remain intact.
Tool contracts should be narrow and explicit. Menu search should return canonical candidates and the information needed to distinguish them. Item detail should return valid modifier groups. A quote tool should return authoritative values. A payment tool should return a system-created link or a structured error. An order-submission tool should return an accepted identifier, a definite rejection, or an unresolved state that triggers recovery.
Do not let the model invent a price, payment URL, availability claim, delivery decision, or order status. These are business facts with financial and operational consequences. The response composer can explain them in natural language, but the underlying values must come from an approved system.
Separate reads from writes in the architecture. Independent menu and item lookups can often run in parallel. Writes should be serialized against a known order-state version. Every commit operation should accept an idempotency key so a retry cannot create a second order. If the state changed after the customer saw the summary, require confirmation of the new version rather than silently committing it.
The same discipline applies to human handoff. Transfer the structured cart, unresolved question, relevant tool results, and submission state along with the transcript. A handoff that forces staff to reread the entire conversation and reconstruct the order is not graceful degradation; it is deferred manual work.
Choose the orchestration pattern from the service objective, not from architectural fashion. Under tight response constraints, AITropos chose direct tool calls instead of MCP or a multi-stage pipeline to reduce orchestration overhead. That is not a universal argument against MCP. It is a reason to benchmark the actual path. Compare end-to-end latency, traceability, schema governance, failure isolation, and engineering cost using representative ordering turns. If an abstraction adds useful control, keep it. If it only adds another round trip, remove it.
Manage latency as part of the customer experience
The model’s inference time is only one part of latency. From the customer’s perspective, the clock starts when the message is sent and stops when a useful next action arrives. Context retrieval, menu search, validation, payment calls, POS submission, message delivery, retries, and overloaded queues all sit inside that interval.
Instrument the complete path before optimizing it. Capture timestamps for message receipt, context assembly, model execution, every tool call, state validation, response creation, and outbound delivery. Report median and tail latency by turn type. A single average can hide a checkout path that is consistently slower than menu questions.
At minimum, separate these turn classes:
Menu discovery and recommendation.
Item identification and configuration.
Cart edits and corrections.
Delivery or fulfillment validation.
Quote and payment-link creation.
Order confirmation and POS submission.
Order-status retrieval.
Human escalation and recovery.
Set a service objective for each class from observed channel behavior and the operational risk of delay. There is no useful universal number. A status lookup and a multi-item order edit do different work. What matters is that the team can see which component consumes the budget and what happens when that component times out.
Optimize in the order that removes uncertainty as well as delay:
Remove unnecessary model and tool round trips. Load the active order and venue configuration before asking the model what to do.
Parallelize independent read operations, such as resolving multiple products mentioned in one message.
Prefetch likely item context so the agent does not discover basic menu facts one call at a time.
Inject only the context needed for the current turn. An oversized prompt moves latency rather than eliminating it.
Keep deterministic validation outside the model when a rule or schema check can answer immediately.
Give every external dependency a timeout, an observable error state, and a safe recovery path.
Use concise responses that advance the order. Extra prose increases reading time and can obscure the decision you need from the customer.
Watch the failure mode on the other side of aggressive optimization. Cached menu metadata can reduce retrieval work, but stale availability or price data can create a wrong commitment. Define which fields are stable enough to cache, how they are invalidated, and which values must be retrieved at quote or submission time. Speed is valuable only when the answer remains authoritative.
When a slow operation cannot be avoided, use an honest progress message and preserve the pending state. Do not fill the wait with repeated acknowledgements that imply completion. If the customer sends another message while the tool is running, the state machine should know whether to queue the change, cancel the pending operation, or ask the customer to wait for its result.
Evaluate each venue, then template what repeats
Make item accuracy precise enough to govern decisions
Item identification accuracy deserves to be the primary trust metric. If the agent resolves the wrong item, every later component can behave perfectly and still produce the wrong order. AITropos treats order item identification accuracy as its most important KPI, giving model, prompt, retrieval, and fallback decisions a common objective.
Define the metric before building a dashboard. I would count an attempted line item as correct only when the canonical item, quantity, and required modifier interpretation match the customer’s resolved intent. A necessary clarification is not automatically an error; it should count against a separate clarification-burden metric. Otherwise, the team may improve apparent accuracy by asking the customer to confirm every obvious detail.
Do not let the primary KPI hide transaction failures. Pair it with guardrails for:
Unsupported substitutions or invented items.
Missing and invalid required modifiers.
Customer corrections after the agent presents a summary.
Quote, payment-link, and POS tool failures.
False confirmations, unresolved submissions, and duplicate commits.
Order completion and abandonment by journey stage.
Human handoff rate, reason, and time to recovery.
End-to-end latency by turn class and venue.
Link corrections back to the original decision. If the customer changes an item because the agent misunderstood it, label the item-resolution turn rather than treating the correction as an unrelated edit. That is how production behavior becomes useful evaluation data instead of a collection of support anecdotes.
Simulate failures before customers encounter them
A venue-specific evaluation suite should use that venue’s menu identifiers, modifiers, availability behavior, delivery rules, payment flow, and POS adapter. A generic restaurant benchmark can test language understanding, but it cannot tell you whether the agent knows that a particular size requires a particular modifier or that two similar menu names map to different SKUs.
Build test families for:
Incomplete names, colloquial references, and ambiguous matches.
Several products requested in one message.
Required modifiers, exclusions, additions, and invalid combinations.
Quantity changes, replacements, removals, and cancellation.
Unavailable items and acceptable alternatives.
Pickup, delivery, and addresses that cannot be validated.
Quote changes before confirmation.
Payment failure, delayed payment state, and an abandoned payment flow.
Tool timeouts, malformed tool results, retries, and uncertain POS submission.
Interrupted conversations that resume with an existing cart.
Requests that require staff judgment rather than autonomous execution.
Generate the expected structured order independently from the agent being tested. Otherwise, the same model can reproduce its own misunderstanding in both the answer and the grade. Keep a small, human-reviewed set of critical conversations alongside the larger generated suite, and add every material production failure to the permanent regression set.
Scale matters when menus contain many combinations. Before each new venue goes live, AITropos runs thousands of simulated customer conversations overnight. The number alone is not the release gate. Coverage, a trustworthy expected answer, and clear failure categories are what make simulation useful.
Simulation also cannot reproduce every production condition. Follow it with a staff sandbox and a controlled production phase. Use only redacted, properly authorized customer conversations in evaluation systems, and retain no more personal data than the test requires.
I would treat any path that invents a price or payment state, falsely confirms an order, or can duplicate a commit as release-blocking. Other thresholds should reflect the venue’s menu complexity, existing human baseline, handoff capacity, and the cost of a wrong order. Record those thresholds before the final test run so launch pressure cannot redefine success afterward.
Roll out autonomy in observable stages
Start with a venue that is operationally manageable but representative enough to expose real modifiers, fulfillment rules, and integration behavior. An unusually simple pilot may produce a clean demo while postponing the problems that determine whether the product can scale.
Configuration: ingest and normalize the menu, map canonical identifiers, mark required modifiers, connect fulfillment and payment rules, and produce a completeness report. No customer-facing ordering is enabled.
Sandbox: let venue staff run realistic conversations while write tools remain disabled or point to a test environment.
Approval mode: allow the agent to prepare a structured order, but require a person to approve the commit. Measure how often the person changes it and why.
Constrained production: enable autonomous submission for the supported venue, fulfillment modes, and order types, with a staffed handoff path and rapid rollback.
Expansion: widen scope only after production traces confirm the accuracy, latency, recovery, and operational workload expected by the release criteria.
For every stage, decide who can pause the agent, how staff take over an active conversation, how the customer learns that a person has taken over, and how an uncertain submission is reconciled before another order is created. These are product requirements, not post-launch operating notes.
Once one venue works, resist copying its prompt and integrations into a new branch. Make venue differences configuration wherever possible: normalized menu schemas, modifier patterns, fulfillment policies, tool mappings, escalation contacts, evaluation packs, and dashboard dimensions. Keep truly distinct behavior explicit rather than burying it in prompt prose.
The scalability payoff can be substantial. AITropos reduced new-venue onboarding from three months to a few weeks, while domain templates are being used to shorten it further. Track your own onboarding work by category: configuration, data cleanup, integration, prompt or policy changes, evaluation, venue training, and launch support. If every venue still requires bespoke code and a rewritten conversation flow, the product has not yet separated its platform from its implementations.
Your next step should be concrete. Choose one representative venue and create three artifacts: the canonical order contract, a failure-and-recovery matrix for every tool, and a venue-specific evaluation set built from redacted, authorized scenarios. If those artifacts cannot show what happens when item resolution, a modifier, delivery validation, payment, or POS submission fails, the agent is not ready to accept orders. Once those states are explicit, model and architecture choices become testable decisions rather than matters of confidence.
You don’t have a chatbot problem. You have an operating-model decision: which customer outcomes may an AI agent own, when must a person take over, and who is accountable when the system gets it wrong?
Get those decisions right and one frontline system can qualify buyers, resolve routine requests, and give specialists better conversations. Get them wrong and you will automate confusion: weak meetings enter the pipeline, unresolved tickets look like successful deflection, and customers repeat themselves after every handoff.
Give the agent a job with an observable finish line
The phrase ‘handle customer conversations’ is not a usable product requirement. It describes a channel, not a job. An agent needs a bounded responsibility, the information and actions required to perform it, and an event that tells you whether the work was completed correctly.
Write a job card before designing prompts or choosing a model. It should specify:
Customer job: the need the agent is expected to address, such as qualifying an inbound buyer or resolving a known setup question.
Eligible intents: the requests it may own and the requests it must immediately transfer.
Allowed actions: retrieve an approved answer, update a permitted field, schedule a meeting, initiate a workflow, or route to a named queue.
Completion event: a correctly qualified meeting, a documented disqualification, a verified resolution, or an accepted handoff.
Failure event: an unsupported answer, an incorrect action, a dropped conversation, a lost handoff, or an outcome that violates policy.
Accountable owner: one person who owns performance across the model, knowledge, workflow, integrations, and operating policy.
The finish line matters because apparent activity is easy to mistake for value. A calendar booking is not a sales success if the buyer does not meet the qualification rules. A conversation that ends without a human transfer is not a support resolution if the customer simply gives up.
Correct disqualification and justified escalation should count as valid outcomes. The objective is not to force every conversation into automation. It is to move every eligible conversation to the right outcome with the least avoidable effort.
Start by running the agent beside an existing human path. Parallel operation gives you a visible fallback, preserves service while the system is learning, and makes outcome quality easier to compare. Broaden ownership only after the agent performs reliably on the job it already has.
Route by customer intent, not your organization chart
Customers do not arrive thinking in sales and support queues. A question about a feature may come from an anonymous buyer, a trial user, an existing customer considering an upgrade, or a customer blocked from completing a task. The words can be identical while the correct response, permitted data, and next action are completely different.
This is why CRM integration and conversation context are core parts of the product rather than optional enrichment. The agent needs enough verified context to determine which job it is performing. It should not expose account-specific information, alter a record, or initiate a commercial workflow until identity and permissions are clear.
A practical conversation policy follows this sequence:
Establish the relationship. Determine whether the person is an anonymous visitor, prospect, trial user, customer, or authorized account contact.
Classify the job. Identify the outcome the customer wants, not merely the keywords in the message.
Retrieve permitted context. Load only the account, conversation, product, and lifecycle information needed for that job.
Ask for missing facts. Collect the minimum qualification or troubleshooting details required to make the next decision.
Complete or transfer. Take an approved action when confidence, policy, and permissions allow it. Otherwise, move the conversation to the correct person.
Record the disposition. Store the recognized intent, facts collected, actions attempted, outcome, and reason for any handoff.
The handoff is part of the agent experience. It should contain the person’s identity and account state, the stated goal, relevant facts, knowledge consulted, actions already attempted, results, and the recommended next step. A transcript dump is not enough. It makes the human reconstruct the problem and usually makes the customer repeat it.
Define transfer triggers before launch. Useful triggers include missing or contradictory approved knowledge, insufficient identity, an action outside the agent’s permissions, repeated failed attempts, an explicit request for a person, a commercial exception, or a conversation where relationship judgment matters more than speed.
Keep the commercial objective visible without letting it corrupt support. Resolve the customer’s blocking issue before introducing an upgrade unless the customer explicitly asks about buying. Likewise, a low-intent visitor does not need to be forced into a meeting. The agent can direct that visitor to useful self-service material and preserve context for a later conversation.
Measure sales creation and support resolution separately
A single automation rate hides the decisions you need to make. Sales and support share an interface, but they create different outcomes. Give each motion its own scorecard and connect the two through shared measures for handoff quality, trust, and customer effort.
Motion
Primary outcome
Diagnostic signals
Downstream proof
Sales
A correctly qualified meeting, documented disqualification, or appropriate nurture path
Qualified, disqualified, dropped, routed, and handoff-accepted conversations
Opportunity creation, attributable pipeline, and revenue
Support
A correct routine resolution or a context-rich transfer
Intent, topic, repeated attempt, escalation reason, time to resolution, and where customers abandon the flow
Successful resolution, repeat contact, sentiment, and CSAT
Shared experience
A trustworthy completion with no unnecessary restart
Unsupported answers, incorrect actions, lost context, policy violations, and customer-requested transfers
Outcome quality by intent, channel, customer type, and agent version
Give agent-originated sales conversations a distinct origin field in the CRM. Retain the conversation identifier, final disposition, and qualification facts, then follow each cohort through opportunity and close. If agent results disappear inside total inbound performance, you cannot tell whether the system created incremental pipeline, shifted work from another channel, or merely booked more low-quality meetings. Meetings, pipeline, and revenue need explicit attribution.
Support needs the same discipline. Do not treat a lack of escalation as proof of resolution. Examine whether the requested task was completed, whether the answer came from approved knowledge, whether the customer returned with the same issue, and whether the handoff arrived in a usable state. Topic and intent analytics should reveal where demand is rising, where customers get stuck, and which workflows actually shorten resolution.
Use a high-performing human on the same channel as the operational benchmark. That comparison is more useful than a generic automation target because it preserves the standards customers already experience. It is a target for your system, not a claim that every agent meets it. Compare like with like: the same eligible intents, customer mix, qualification policy, and access to knowledge.
Before expanding eligibility, use eval-driven development and controlled experiments. Keep the eligibility rules stable during a comparison, segment results by intent, and change one major layer at a time. If the prompt, knowledge base, routing policy, and action permissions all change together, a better aggregate score will not tell you what improved or which new failure mode you introduced.
Put one owner over knowledge, guardrails, and iteration
A customer-facing agent is a production system, not a launch asset. Product knowledge changes. Qualification rules change. Integrations fail. Customers find language the original tests did not cover. Performance will drift unless someone owns the whole loop.
That owner needs program-level responsibility. In sales, the role may be an AI SDR program lead. In support, it may sit with an AI operations or product leader. The title matters less than the decision rights: the owner must be able to change eligibility, knowledge, prompts, workflows, routing, evaluation criteria, and rollout scope.
The operating loop should be explicit:
Review outcomes by intent. Inspect successful completions as well as failures; a passing aggregate can conceal one dangerous category.
Classify the failure. Separate knowledge gaps, intent errors, policy mistakes, tool failures, permission problems, poor handoffs, and correct answers delivered in an unhelpful way.
Fix the smallest upstream cause. Update the audited knowledge when the fact is missing, the workflow when the action is wrong, the policy when the boundary is unclear, or the conversation design when the interaction creates friction.
Replay representative evaluations. Test the changed component against known successful cases, known failures, ambiguous requests, and transfer scenarios.
Release to limited eligibility. Preserve the human fallback and monitor the affected intent before increasing traffic or adding actions.
Record the change. Version the knowledge, prompt, policy, workflow, and evaluation set so a metric movement can be traced to a real product change.
Ground answers in a retrieval-first pipeline backed by audited knowledge. The generative layer should explain and adapt approved information; it should not invent product behavior, policy, eligibility, or commercial commitments. When the agent can take action, give each action its own identity checks, required fields, permission boundary, confirmation behavior, and failure path.
CRM context improves relevance, but it also increases the cost of a permission mistake. Apply privacy-by-design at the workflow level: retrieve only what the current job needs, verify identity before exposing account details, restrict actions by role, and preserve an audit trail of what the agent saw and did. A fluent response does not compensate for unauthorized access.
The rollout is incomplete until human work changes. Salespeople should gain time for higher-conversion conversations, multi-stakeholder account development, guided trials, and situations where judgment affects the buying process. Support specialists should receive the nuanced, emotionally sensitive, or genuinely novel problems with the context already assembled.
Is the job bounded? The eligible intents, required context, allowed actions, and prohibited actions are written down.
Is success observable? Sales quality reaches pipeline and revenue; support quality reaches real resolution rather than mere non-escalation.
Is the transfer designed? Triggers are explicit, the receiving queue is known, and the human receives a structured handoff instead of a raw transcript.
Is attribution separate? Agent-originated conversations, dispositions, downstream outcomes, and versions can be analyzed without disappearing into channel totals.
Is trust engineered? Approved knowledge, evaluations, identity controls, action permissions, privacy rules, and audit records exist before broad access does.
Has human capacity been reassigned? Sales and support specialists have named higher-value work to absorb the time the agent releases.
If any answer is no, do not widen the agent’s scope yet. Tighten the job, instrumentation, or boundary that is missing. More traffic will amplify an unclear operating model faster than it will improve one.
Your next move is small but concrete: choose one frequent intent with audited knowledge and an unambiguous finish line. Write its job card, run it beside the existing human path, assign one accountable owner, and track the outcome through the system that ultimately matters. Expand only when the agent is reliably completing that job and the human team is using the released capacity deliberately.
Inbound leads shouldn’t wait for a rep’s calendar. When we first launched The Service Agent Blueprint, support leaders finally had a clear AI path. Go-to-market and revenue teams are now facing similar uncertainty, so I’m introducing The Sales Agent Blueprint—a practical map for launching and scaling AI for sales with confidence.
For most sales teams, inbound motions require a lot of manual work. I’ve watched leads pile up in queues, waiting for availability rather than being prioritized by buyer intent. That delay costs meetings, pipeline, and momentum—and it’s exactly where a modern AI Strategy can transform your go-to-market strategy.
Agents can run sales conversations end to end – engaging buyers, qualifying leads, and routing high-intent opportunities to the right team to move prospective buyers forward quickly. Humans will still be involved, but will move their focus to the consultative conversations and higher-value work they did not have time to focus on before. In practice, this shift enables cleaner AI workflows, better conversation design, and a healthier balance between sales-led growth and product-led growth.
The questions many go-to-market and revenue leaders are facing now are where do you start? What should success look like? How do you actually test and deploy these solutions? These are the right questions—and the ones I hear most often when teams weigh build vs buy decisions, evaluation frameworks, and CRM integration nuances.
The Sales Agent Blueprint answers those questions. It’s designed to be a strategic guide for sales, revenue, and AI transformation leaders who want to deploy AI for inbound sales fast, prove value, and build momentum. If you’re aiming for eval-driven development, this will help you define success up front and operationalize it.
What’s inside is simple by design yet deep enough to take you from zero to value. The Sales Agent Blueprint is structured around two tracks that reflect how high-performing teams adopt agentic AI: first, launch for quick wins; next, scale for durable growth.
Coming soon: Sales Agent Blueprint. A sleek, blueprint-inspired teaser with the call to 'Scale it' signals tools, playbooks, and workflows to grow revenue, streamline operations, and scale teams with confidence.
Today, I’m releasing the first part of the Blueprint: “Launch it.” It’s a practical guide for getting your Agent live and seeing real results. You’ll learn how to deploy a Sales Agent that runs inbound sales conversations end to end, engaging buyers, qualifying leads, and routing high-intent opportunities to the right outcome in real time—without disrupting your current CRM integration or pipeline processes.
By the end of the “Launch it” track, you’ll be ready to execute with clarity. Here’s how I frame the essential steps, based on what consistently works in the field.
Understand what a Sales Agent is: Discover why they’re different from chatbots and how they work. Build a business case: Prove the basic economics of AI, decide whether to buy or build, and get the buy-in and budget you need to move forward.
Evaluate an Agent: Learn how to define success, choose the right evaluation criteria, and run a focused, high-impact assessment with our five-step framework.
Deploy with confidence: Build a deployment plan that gets your Agent live quickly to engage buyers at peak intent. Learn what to expect at each stage.
Introducing the Sales Agent Blueprint. This crisp, grid-based graphic spotlights step 1—Launch it—signaling day-one activation for an AI sales agent. Explore the framework and get started at fin.ai/blueprint/sales.
Continuously improve performance: After launch, your Agent becomes a system to manage. We’ll show you how to implement a repeatable process to train, test, deploy, and optimize.
The second track, “Scale it” (coming soon), focuses on the organizational and systems design work that unlocks compounding gains. Launching AI is only the beginning. To unlock its full potential, you need to rewire your inbound sales motion—redesigning the buyer journey, building AI-first systems and ownership models, and rethinking how pipeline is generated and scaled. This is where governance, measurement, and team roles evolve to support sustainable growth.
I’ll be building this Blueprint in public as I navigate the same challenges—sharing what works, what to avoid, and how to accelerate time-to-value without sacrificing quality or trust. If you’re ready to turn intent into revenue with agentic AI, this is your head start.
The Sales Agent Blueprint is live now. Explore the full guide at fin.ai/blueprint/sales and start your “Launch it” sprint today.
A prospect reaches your pricing page with a real buying question. The form promises a reply, but the reply arrives after the prospect has moved on, chosen a competitor, or forgotten why the question mattered.
An AI customer agent can remove that delay, but speed is only the entry requirement. The harder product problem is deciding whom to qualify, what evidence to collect, which next step to offer, and how to preserve enough context that a salesperson can continue the conversation without starting over.
Start with routing decisions, not chatbot dialogue
The purpose of a sales qualification agent is not to produce a pleasant conversation or a high lead score. Its job is to make a defensible next-step decision while the buyer’s intent is still active.
That distinction matters because conversational fluency can hide weak commercial logic. An agent may sound helpful while booking low-fit meetings, sending strong prospects down a generic self-serve path, or marking inferred information as confirmed. Those failures make the pipeline look larger before they make it less trustworthy.
Define the available outcomes before you write prompts. Most inbound motions need some version of these routes:
Route
Minimum evidence
Agent action
Sales-ready
The problem fits the product, the buyer needs sales involvement, and the timing or buying process satisfies your acceptance rule.
Offer an appropriate meeting, create or update the CRM record, and send the qualification evidence.
Self-serve
The use case is viable, but the buyer can select a plan, begin a trial, or complete signup without a salesperson.
Recommend the relevant path, help the buyer take the next action, and preserve the conversation for later use.
Promising but not ready
There is plausible fit, but intent, timing, authority, or requirements remain unresolved.
Provide the useful resource or follow-up path defined by your policy without manufacturing urgency.
Not a fit
A hard requirement conflicts with the product’s supported scope or the request belongs elsewhere.
State the limitation clearly and redirect the person without placing an unqualified meeting on a seller’s calendar.
Human exception
The request involves an existing account, a sensitive claim, a complex commercial exception, or information the agent cannot verify.
Escalate with the context already collected and identify the unresolved question.
Keep fit and readiness as separate dimensions. A large, recognizable account can be a strong fit and still be months away from a decision. A highly motivated buyer can be ready to act and still need a capability you do not provide. Combining both dimensions into one opaque score conceals the reason behind the route and makes mistakes difficult to diagnose.
Separate hard constraints from soft signals as well. A required capability that does not exist may be a hard stop. A vague timeline is usually uncertainty to resolve, not automatic disqualification. Firmographic enrichment can help prioritize an account, but it cannot confirm what a buyer has not actually said.
For every consequential route, require three outputs: a reason code, the evidence behind it, and the next action. If the agent cannot produce all three, it has not completed qualification. It has merely assigned a label.
Turn your qualification policy into an executable conversation
A natural-language playbook makes sales policy easier to express, and current customer agents can be instructed to follow qualification rules, address approved objections, and move buyers toward defined outcomes. Natural language does not remove ambiguity, however. If two experienced salespeople would interpret a rule differently, the agent will not reliably repair the policy for you.
Ask only what changes the route
Traditional lead forms collect fields because the CRM has columns. A conversation should be more selective. Every question should either help the buyer, determine fit, resolve readiness, or select the correct action.
Open from observable context, such as the plan, feature, or integration the person is exploring.
Answer the buyer’s current question before turning the exchange into discovery.
Ask the smallest useful question that could change the route.
Branch from the answer instead of walking every prospect through the same questionnaire.
Confirm the important facts before treating them as qualification evidence.
Explain the recommended next step and let the buyer act while still in the conversation.
If someone asks whether a specific integration is available, answer that question first. Then ask how the integration fits the intended workflow if the answer would affect plan selection or sales involvement. Leading with budget, company size, or phone number when none of those details helps answer the immediate question makes the agent feel like a form with typing animation.
A useful qualification schema usually covers the following areas, but the agent should collect only the fields relevant to the current branch:
The problem or use case the buyer is trying to address.
The capabilities, integrations, or constraints that determine product fit.
The consequence of leaving the problem unsolved, when that affects urgency or route.
The buyer’s role in evaluation and the remaining buying process.
The intended timing and any event driving it.
Commercial expectations or budget when those facts genuinely affect the path.
Identity and account context, with a clear distinction between what was stated, enriched, or inferred.
Do not ask about budget merely because a familiar qualification framework includes it. If pricing is public and the buyer can start without sales assistance, the better action may be to explain plan fit and help the buyer proceed. If commercial terms require human involvement, budget or purchasing process may become relevant later in the branch.
Preserve provenance instead of filling blanks with guesses
Store each material qualification field with its provenance. Buyer-stated, externally enriched, model-inferred, and unknown are different states. Treating them as interchangeable creates false confidence in the CRM.
An enriched company size may help prioritize a conversation, but it is not buyer-confirmed budget. A page visit may indicate interest in a feature, but it is not a confirmed requirement. An enthusiastic phrase may indicate intent, but it is not a purchasing timeline. Keep those distinctions visible to the routing logic and the salesperson receiving the lead.
I would not allow the agent to write a final qualification status unless every required field is either supported by evidence or explicitly marked unknown. Unknown information is operationally useful: it tells the seller what still needs to be resolved. Fabricated completeness does the opposite.
Evaluate decisions, not just responses
Build a scenario set from the situations that cause real routing disagreements. Include a high-fit account with low intent, a small account with urgent intent, an existing customer asking a sales-shaped question, a buyer requiring an unsupported capability, a returning visitor, a pricing objection, conflicting information, and a request the agent should escalate.
For each scenario, define the expected answer, acceptable route, required CRM writes, escalation condition, and forbidden behavior. Then test the complete journey. A correct answer followed by the wrong calendar, duplicated CRM record, or context-free handoff is still a failed qualification experience.
Assign a business owner to every consequential knowledge area. When pricing, packaging, an integration, or an eligibility rule changes, update the canonical material and rerun the scenarios affected by that change. A polished answer based on stale commercial information is more dangerous than an explicit handoff because the buyer has little reason to question it.
Define the agent’s boundaries in the same system. It should know when it may explain published pricing, when it must avoid inventing discounts, when roadmap questions need human confirmation, and when a security, legal, or contractual claim requires escalation. The safe fallback is not a vague non-answer. It is a clear statement of what remains unverified and a context-rich route to someone authorized to answer.
Use memory as buyer state, not as an unlimited transcript
Memory is valuable when a returning visitor does not have to repeat the use case, plan under consideration, or unresolved objection. A customer agent can recognize returning context and continue the buying journey, but old information should not silently override new facts.
Store a compact buyer state: confirmed needs, important constraints, questions already answered, current route, unresolved items, and the last agreed next step. Keep timestamps and provenance so the system can notice when a current statement conflicts with an earlier one. Ask for confirmation when a material fact may have changed.
Enrichment deserves the same discipline. Use it to improve context and routing, not to pretend the agent knows the buyer personally. Record where enriched data came from, apply your privacy and retention controls, and give buyer-stated information precedence when the two conflict.
Make the handoff a product deliverable
Booking a meeting is not the end of qualification. It is the beginning of a human handoff. Passing only a name, email address, and transcript forces the salesperson to reconstruct the conversation under time pressure.
The handoff package should contain:
Identity and account information, including the provenance of enriched fields.
The buyer’s problem and intended outcome in the buyer’s own terms.
Confirmed requirements, constraints, timing, and buying-process details.
Questions answered and the approved information used to answer them.
Objections, unresolved questions, and any claim requiring human confirmation.
The selected route, its reason code, and the evidence that supported it.
The next action already promised to the buyer.
A link to the full conversation for detail or audit.
Test the first minute of the seller’s follow-up. Can the seller see why the lead was routed, what the buyer already knows, and what must happen next? If the opening question repeats discovery the agent just completed, the handoff has broken the continuity you used AI to create.
Measure whether the agent creates incremental pipeline
Meeting count is an attractive but incomplete success metric. Bookings can rise because the agent reaches previously unattended demand, because it diverts buyers who would have booked with a human anyway, or because it lowers the qualification bar. Only the first outcome is unambiguously additive.
Instrument the full decision funnel rather than the chat interface alone:
Reach: eligible visitors, conversations initiated, response latency, and coverage by channel or time window.
Conversation quality: questions answered, unresolved-answer rate, corrections, escalations, and abandonment before a useful action.
Qualification quality: completion of required evidence, unknown-field rate, route distribution, seller acceptance, and rejection reasons.
Commercial outcome: accepted qualified opportunities, pipeline created, trial or self-serve conversion, win rate, contract value, sales-cycle progression, and cost per accepted opportunity.
Audit both error directions. False positives waste seller time and inflate forecasts. False negatives are quieter: a strong buyer is sent away, mislabeled as self-serve, or blocked by an unanswered question. Review unsuccessful routes as well as booked meetings, because the most expensive qualification error may never appear in the sales team’s queue.
Establish your own baseline separately for hours and pages with human coverage and those without it. If feasible, use a randomized holdout among otherwise eligible sessions. If randomization would create an unacceptable buyer experience, compare matched cohorts by page, channel, segment, visitor status, and time window. Do not compare an overnight agent cohort with daytime human coverage and call the difference an AI effect.
A controlled rollout can begin on a high-intent surface or during a coverage gap. First run routing in shadow mode and compare the proposed decisions with qualified human judgment. Then enable one consequential action at a time, such as self-serve guidance before autonomous meeting booking. Keep a human review path for exceptions and expand only when answer quality, routing precision, CRM completeness, and buyer outcomes remain acceptable together.
Your north-star measure should reflect accepted commercial value, such as incremental qualified opportunities or incremental pipeline per eligible visitor. Pair it with guardrails for incorrect claims, seller rejection, buyer complaints, CRM errors, and missed high-fit leads. An agent that creates more records while reducing trust has not improved the sales system.
Key takeaways
Treat the AI customer agent as a decision system, not a conversational layer placed in front of a lead form.
Define sales-ready, self-serve, not-ready, not-fit, and human-exception routes before writing dialogue.
Keep fit separate from readiness, and preserve whether each field was buyer-stated, enriched, inferred, or unknown.
Ask only questions that help the buyer or change the route; answer the buyer’s immediate question before running discovery.
Make structured qualification evidence, unresolved issues, and the promised next action part of every human handoff.
Measure incremental accepted opportunities and pipeline, not chat volume, MQL count, or booked meetings in isolation.
Start with one high-intent entry point. Write its route contract, connect only approved knowledge, test the difficult scenarios, and compare shadow decisions with the people who currently qualify those leads. Give the agent authority gradually. The goal is not to automate the most conversation; it is to make the right buying path available at the moment the buyer is ready to take it.
Your AI support agent is containing more conversations, but you still cannot answer the question that matters: did it solve the customer’s problem, or did it merely avoid a handoff?
You do not need another dashboard of conversation counts and token usage. You need a closed loop that connects customer outcomes to individual agent decisions, turns failures into test cases, and makes each release safer than the last. Here is the operating model I would put in place.
Define resolution before you optimize the agent
Continuous improvement starts with an outcome contract, not an analytics implementation. If product, support, and engineering use different definitions of success, every subsequent metric will create debate instead of direction.
Write the first outcome contract for one customer journey. Keep it to one page and specify:
Eligible cases: Which requests can the agent reasonably handle? Separate supported journeys from requests that must go directly to a person.
The unit of analysis: Decide whether success belongs to a conversation, a case, or a completed task. A customer who restarts the same issue in a second conversation has not necessarily created a new problem.
Completion evidence: Name the customer statement, downstream system event, human judgment, or evaluator result that confirms the task was completed.
Acceptable escalation: Define when handing the case to a person is the correct outcome. A policy-mandated handoff is not an agent failure.
Failure conditions: Include abandonment, repeated rephrasing, an incorrect action, an unsupported claim, a failed tool call, and a handoff without usable context where they apply.
The observation window: Choose how long you will look for a repeat contact or reversal before treating the resolution as durable. The right interval depends on the journey, so publish it with the metric.
Then separate the signals that teams commonly collapse into one number:
Signal
What it tells you
How to use it
Verified resolution
The eligible customer task was completed using the evidence in your outcome contract.
Use this as the primary outcome for a resolvable journey.
Containment
No person entered the interaction.
Treat it as a routing and labor signal, not proof of success.
Escalation
The agent transferred the case to a person.
Split appropriate escalation from avoidable escalation before drawing conclusions.
Reliability
Tools, retrieval, guardrails, and fallbacks behaved as intended.
Use step-level rates to locate the mechanism behind an outcome.
Performance
The customer waited for the first response and for the complete task.
Track both typical and tail latency so a healthy average does not hide slow cases.
Efficiency
The agent consumed tokens, model spend, tool spend, and human effort.
Measure cost per successful eligible task, not just cost per conversation.
Groundedness
Claims based on retrieved material are supported by the retrieved material.
Evaluate retrieval-dependent journeys separately from general response quality.
The distinction between resolution and containment is particularly important. An unresolved conversation that never reached a person improves containment while making the customer experience worse. Conversely, a fast and well-prepared escalation can be the right resolution path.
Use explicit formulas in the metric catalog. For example, resolution rate should be verified successful eligible cases divided by eligible cases. Cost per success should be the applicable model and tool cost divided by verified successful cases. If you change the eligibility rule or verification method, version the definition rather than silently changing the chart.
Do not let the agent’s own declaration of success become your only evidence. Prefer an observable business event or explicit customer confirmation. When neither exists, use a documented evaluation rubric and periodically compare automated judgments with human review. The goal is not to pretend ambiguity has disappeared. It is to make the ambiguity visible and consistent.
Instrument the trajectory, not just the conversation
Traditional product analytics can tell you that a conversation opened, escalated, and closed. It cannot explain why an agent chose a tool, which knowledge it retrieved, what a guardrail rejected, or where a multistep task went off course. Agent behavior includes nondeterministic trajectories, tool chains, prompt context, retrieval, policies, and fallbacks. Your telemetry has to preserve that sequence.
Build three connected records:
Case record: A stable identifier for the customer need, the normalized intent, channel, eligible cohort, final outcome, escalation reason, customer signal, and any repeat-contact relationship.
Run record: The model, prompt, policy, knowledge, tool, evaluator, and experiment versions used for one execution, along with total latency, token use, cost, guardrail events, fallback state, and final response.
Step record: Each retrieval, model decision, function call, tool result, validation, retry, and handoff. Capture inputs and outputs in an appropriately protected form, plus status, latency, error type, and the next step selected.
Every run should be reproducible at the level needed to investigate it. Version anything that can change behavior: system instructions, prompt templates, policies, model configuration, tool schemas, knowledge snapshots, routing rules, guardrails, and evaluators. Record experiment assignment on the run itself. Otherwise, a failed trace can become impossible to explain after the underlying assets change.
You need three views over these records. The aggregate scorecard shows whether customer and business outcomes are moving. The trace explorer shows the decision path behind a particular result. The evaluation system tests candidate behavior against expected results. None is a substitute for the others: aggregates reveal scale, traces reveal mechanisms, and evaluations reveal whether a proposed change is safe to release.
Privacy has to be part of the data model. Support conversations can contain personal or sensitive information, and copying every raw transcript into a broadly accessible analytics system creates unnecessary exposure. Separate human-identifiable data from model telemetry and mask sensitive content while retaining useful semantic context. Apply access controls and retention rules to raw content, and let most analysis operate on sanitized traces, structured labels, and protected identifiers.
Before declaring instrumentation complete, take one unresolved case and answer five questions from the data alone: What was the customer trying to do? What path did the agent take? Which versions governed that path? At which step did reality diverge from the expected behavior? What outcome evidence followed? Any missing answer is a concrete telemetry gap.
Turn production failures into a daily improvement queue
Analytics becomes continuous improvement only when an observation can enter a queue, acquire an owner, produce a test, and reach production. A weekly dashboard presentation without that path is reporting, not learning.
Create one improvement queue shared by product, support, engineering, and the people responsible for the agent. Feed it from failed evaluations, avoidable handoffs, repeat contacts, low-confidence outcomes, guardrail events, tool errors, and anomalies in journey-level metrics.
At each human handoff, prefill the case identifier and trace, then ask the support specialist for three short inputs:
What was the customer actually trying to accomplish?
What prevented the agent from completing it?
What reusable change could prevent the same failure?
Keep the capture lightweight. The frontline should add information the trace cannot supply, not reconstruct a conversation the system already recorded. This turns human support into a sensor for recurring defects and supports the principle that a solved customer issue should create a chance to prevent its recurrence.
Cluster related cases before prioritizing them. Fixing a single transcript can overfit the agent to one phrasing; fixing a recurring failure pattern improves a journey. Give every cluster a normalized intent and one primary root-cause label:
Knowledge: The required information was missing, stale, contradictory, or too difficult to retrieve.
Retrieval: Useful information existed, but the wrong material was selected or relevant material was omitted.
Instruction or policy: The agent had ambiguous, conflicting, or incomplete directions.
Tooling: A function was unavailable, called incorrectly, timed out, or returned unusable data.
Conversation design: The agent failed to collect required information, explain a limitation, confirm an action, or recover from confusion.
Routing and handoff: The case went to the wrong destination, escalated too late, or arrived without sufficient context.
Capability boundary: The task was eligible on paper but was not reliably achievable with the current model and workflow.
Product or process: The customer encountered a defect or an underlying workflow that support automation alone cannot repair.
Do not use the taxonomy to force certainty. Allow an unknown label, but review unknowns regularly; a growing unknown bucket usually means the taxonomy or telemetry has stopped describing production.
Each queue item should contain the affected journey and cohort, linked traces, observed and expected behavior, root-cause hypothesis, customer consequence, recurrence evidence, responsible owner, proposed smallest change, a new or updated evaluation case, rollout method, and rollback condition. That record makes the improvement auditable from failure to fix.
Prioritize with four questions instead of reacting to the loudest transcript: How often does the pattern appear? How consequential is it when it happens? How confident are you in the root cause? How reversible is the proposed change? A frequent knowledge miss with a narrow, testable correction is a good candidate for rapid iteration. An uncommon policy failure with serious consequences may deserve attention first even if its volume is low.
A prompt edit is a product change. So is a new knowledge document, model version, tool description, routing rule, or policy. Treating these assets as untracked configuration makes regressions difficult to detect and harder to reverse.
Use a four-part improvement loop: update the controlled asset, test complete simulated conversations, deploy through a controlled release, and analyze production outcomes. That train-test-deploy-analyze cycle keeps learning connected to customer behavior rather than ending at the prompt editor.
Leading the Support function for a company that builds a leading Agent and AI-forward customer service platform has been, for me, unique, exciting, and yes—daunting. It’s where product ambition meets operational reality, and where every decision I make is immediately tested by customers who expect excellence.
It’s unique because we use the same technology as our customers. We live in the product every day, which puts us in a privileged position to be the voice of the customer across the organization. That tight feedback loop has shaped how I prioritize, what I build next, and how I measure success.
It’s exciting because we get to try all of the new features and capabilities of Fin and the Intercom helpdesk. With a relentless focus on AI innovation, I’ve had access to remarkable tools that help us deliver an incredible customer experience—and I’ve seen firsthand how the right workflows and guardrails turn those tools into outcomes.
And it’s daunting because expectations for our own Customer Support (CS) team are sky high. If we can’t deliver incredible support using our own technology, we undermine its value proposition. That imperative has kept me honest, focused, and fast.
In our new research, “The 2026 Customer Service Transformation Report,” we’ve been sharing how forward-looking teams use AI to transform their support models. If you’d like to get straight to the report, download it here.
When Intercom changed its focus in late 2022 to prioritize the customer service use case, we undertook a critical review of the support experience we were delivering and committed to driving meaningful change under an AI-first framework. That was a turning point: I aligned product strategy and operations around a single north star—automate with quality, and elevate humans to higher-value work.
Three years on, Fin now resolves over 81% of all our customer support volume, delivering immediate and high-quality resolutions. We have absorbed a 300%+ increase in customer demand since 2022 without proportional headcount growth. Without Fin, we would have needed at least 100 additional CS team members to meet that demand and our improved service levels – a net saving to Intercom of between $7.5M–$9M annually.
Throughout this work, we drew on research from the 2026 Customer Service Transformation Report and applied the lessons directly to our own org design, knowledge management, and AI workflows. What follows is our story of transformation and how we achieved a mature deployment of Fin.
The problems we set out to solve
Back in 2022, our challenges looked familiar to any modern support organization, and I knew we needed a step-change—not incremental tweaks.
We faced increased support demand from new and existing customers: Intercom was launching major features and changes at speed, driving up overall customer conversation volume and requiring additional headcount for the CS team. I could see we were scaling people faster than processes—unsustainable without automation.
Our support policy (as defined by our service level objectives) was not based on a high bar: In most cases, we were only committed to “business hours” coverage for the majority of our customers, impacting first response times. Even with SLOs that were not considered best in class, we were struggling to meet our commitments. I wanted 24/7 coverage and faster first responses without sacrificing quality.
We wanted to do more: As we pivoted our strategy, we wanted to open new routes to our support team, such as providing support to website visitors with technical questions and to trial customers. That meant meeting customers earlier in their journey with accurate, on-brand responses—at scale.
What we did
We made a very conscious decision to become our own best reference customer. As Intercom embraced the opportunity that generative AI presented to transform customer service, we intentionally moved to an AI-first strategy for our Customer Support team. I set a simple operating principle: ship value quickly, measure relentlessly, and let evidence guide the next bet.
We started with the highest-volume, informational queries and saw our resolution rates climb quickly. With that foundation in place, we pushed Fin further, training it on deeper documentation and internal procedures, and eventually giving it the ability to take actions on behalf of customers. As Fin took on more complex work, our results started to compound—and trust in the system grew across the organization.
Early adoption and building trust. When “AI Assist” features came to the Intercom Inbox, the CS team got early exposure to AI and were empowered to provide feedback directly to our product teams. This built awareness and trust across the team about what we were trying to achieve with AI, and helped shape the product roadmap. We were also the first beta customer for Fin, rolling it out to a subset of customers to watch sentiment and outcomes closely. With no adverse reaction and an initial resolution rate of over 25%, we deployed Fin to most customer segments within weeks. I’ll never forget the first week we put Fin in front of real customers—the silence of issues that never reached humans was the loudest signal of success.
Knowledge management as a product. We recognized quickly that time spent tuning our help center and knowledge assets for Fin would pay dividends. We transitioned our Help Center Manager into a “Knowledge Manager,” with a dedicated remit to optimize content for Fin. We embedded knowledge creation into our “New Product Introduction” (NPI) process, targeting that Fin would resolve at least 50% of customer issues at every new product and feature launch. Over time, we added new sources, including “Developer Documents,” enabling Fin to handle increasingly complex issues. We built a culture of continuous improvement—allocating “out of the inbox” time so every teammate could close content gaps and raise the bar.
Conversation design end-to-end. To ensure a consistent, high-quality customer experience, we created a new “Conversation Designer” role that owns the journey across automation and human handoffs. Using Intercom’s Workflows, we introduced “skills-based routing” so that when a customer asks for a human, the conversation reaches someone with the right expertise quickly. This is now handled by Fin directly using a feature called “Attributes.” The result: a seamless, on-brand experience regardless of channel or escalation path.
Leaders are racing ahead with real AI in support. Explore the 2026 Customer Service Transformation Report to see where deployment is stalling, benchmark your team, and get practical steps to scale automation that delights.
Organization changes that unlocked leverage. As we scaled Fin, we stood up a dedicated AI Support team under a senior CS leader to continuously optimize automation and define our AI adoption strategy across the journey. We restructured human roles into “Technical Support Specialist” and “Technical Support Engineer” to better align with the complexity of incoming work. We also expanded Support Operations to focus on optimization—using AI to uplevel Enablement, Workforce Management, QA, Process Management, and Data Insights. Just as important, we reset expectations about the balance between time spent supporting customers directly versus improving AI. That mindset shift created compounding returns.
Pushing Fin further with new capabilities. As capabilities matured, we were early adopters and saw measurable wins:
Fin Guidance: Multiple Guidance rules provide additional controls and a more personalized, targeted experience for customers.
Fin Tasks and Procedures: Enables Fin to carry out activities such as updating customers on incident status and deep troubleshooting for technical issues.
Insights: AI-driven dashboards provide deep insight into Fin’s performance and surface recommendations for further optimization. Insights also provides a Customer Experience (CX) Score for every customer interaction, enabling more targeted improvement efforts and opening up new ways to close the loop with customers who have had a poor experience.
What we achieved
What started as a focused effort to improve our customer support experience became the strongest proof point for what’s possible when you fully embrace AI. Fin now resolves over 81% of all our customer support volume and has allowed us to absorb a 300%+ increase in demand without proportional headcount growth. Over 90% of our customers now benefit from improved first response performance, 24/7 coverage, and outbound phone support.
What the numbers don’t fully capture is the shift in how our team operates. With volume absorbed by Fin, our CS teammates now deliver consultative support—guiding next best actions, deepening product adoption, and contributing directly to retention and expansion. Customers that receive these engagements adopt Fin at a much deeper level and achieve greater support success. What was once a reactive, volume-driven team is now a function that generates significant revenue.
What’s next
Customer expectations are always rising, so we’re building on our progress by embracing the Fin Flywheel—an actionable framework for ongoing improvement and optimization. This keeps us honest about the discipline required to sustain AI performance at scale.
Train: Teach Fin to resolve even the most complex queries with Procedures, knowledge, and policies.
Test: Run fully simulated customer conversations from start to finish to see exactly how Fin will behave before going live.
Deploy: Set Fin live across every channel – voice, email, chat, and social – for consistent support wherever customers reach out.
Analyze: Use AI-powered Insights to analyze and improve Fin’s performance and deliver better customer experiences.
We are also investing in our support teammates so they can adjust to the new world of AI—taking on more complex work and being valued for the subject matter expertise, consultative engagement, and empathy they bring to the role. That human layer is where differentiation shines.
We will continue to develop and share best practices for deploying an Agent, based on our own experience with Fin and the lessons learned from our most forward-looking customers. These are captured and continually evolving in The Agent Blueprint.
Transformation takes commitment
The most successful teams aren’t bolting AI onto old processes; they’re rebuilding support around it—investing in knowledge and people alongside technology, and treating AI as a continuous discipline rather than a one-time deployment. That’s the real change required. For support teams willing to make it, there’s a rare opportunity to redefine what customer service can deliver—higher CSAT, faster resolution, and durable ROI.
Your dashboard says the ticket was resolved. The customer remembers repeating the problem, moving between an AI agent and a teammate, and discovering that company policy still blocked the outcome they wanted. Product, Support, and Operations can all look at the same conversation and reach different conclusions.
If you are considering conversation-based customer experience scoring, the hard part is not asking an AI model for a rating. It is designing a measurement system that distinguishes the experience from its causes, shows people why the score exists, and sends each cause to someone who can change it.
A useful score separates experience from ownership
A customer experience score should answer a narrow question: how well did this interaction work for the customer? It should not silently answer a different question, such as whether the support agent performed well or whether the product team made the right policy decision.
Those questions overlap, but they are not interchangeable. A teammate can give a clear and accurate explanation of an unpopular refund policy. The teammate’s answer quality may be strong while the overall experience remains poor. An AI agent can use a warm tone while giving an incorrect answer. The sentiment may look positive even though the handling failed. A product limitation can make resolution impossible despite excellent support work.
This is why a credible score needs several layers:
Outcome: Was the customer’s request resolved, partially resolved, redirected to a workable next step, or left unresolved?
Answer quality: Were the responses clear, accurate, relevant, and internally consistent? Evaluate AI and human responses separately when both participated.
Customer effort: Did the customer repeat information, survive avoidable handoffs, chase a promised follow-up, or clarify something the company should already have understood?
Emotional context: Did the customer express strong frustration, anger, relief, gratitude, or delight? Treat emotion as context rather than a verdict by itself.
Product or service feedback: Was the customer reacting to a bug, missing capability, reliability problem, delivery failure, confusing design, or service issue?
Policy feedback: Was the real source of dissatisfaction a refund rule, eligibility condition, account limit, return policy, or another business decision?
Score the experience first. Attribute the drivers second. Assign ownership third. Reversing that order creates predictable dysfunction: teams defend their own performance, difficult conversations get excluded, and the metric becomes a political argument instead of a customer signal.
Design the score as a diagnosis, not a black box
Leadership may want one number for a dashboard, but the useful product is the diagnostic record underneath it. If a support leader cannot open a low-scoring conversation and see why it received that result, the number is not ready for coaching, prioritization, or executive reporting.
The minimum record behind each score
For every eligible conversation, preserve these fields:
Overall experience band: A small set of anchored labels is easier to calibrate than a decimal-heavy score that implies unsupported precision.
Eligibility status: Record whether the interaction was scored, excluded under a defined rule, or genuinely lacked enough information.
Outcome status: Resolved, partially resolved, unresolved, or unclear.
Answer-quality results: Separate evaluations for AI and teammate contributions where applicable.
Driver codes: Effort, strong emotion, product or service feedback, policy feedback, and any operational reason codes you have explicitly defined.
Evidence: The specific message or interaction event that supports each driver. A generated explanation without transcript evidence is an assertion, not an explanation.
Plain-language summary: What the customer needed, what happened, and why the experience earned its band.
System metadata: The scoring model, rubric, and schema versions used to produce the record.
I would begin with anchored experience bands rather than pretending the system can distinguish tiny numerical differences. A practical rubric might distinguish a strong experience, an acceptable experience with minor friction, a weak experience with material friction or incomplete resolution, and a poor experience with an unresolved outcome, serious inaccuracy, contradiction, or excessive burden.
The labels matter less than the anchors. Reviewers need observable criteria for each band. Phrases such as good conversation or unhappy customer leave too much room for interpretation. Criteria such as customer repeated the account history after a handoff or answer contradicted an earlier commitment can be checked against the transcript.
Do not let emotion dominate the rubric. A customer may arrive angry because of a product outage and receive excellent assistance. Another may remain polite after receiving a materially wrong answer. Emotion can increase urgency and explain the experience, but it cannot substitute for outcome, accuracy, and effort.
Do not average away disagreement between dimensions either. An acceptable overall score can conceal an inaccurate AI answer that a teammate later repaired. Preserve that AI-quality failure as a driver so the AI product team can add it to an evaluation set even when the customer ultimately gets a resolution.
Make the metric reliable enough for decisions
A score can look stable while measuring a changing subset of conversations. If short threads, low-context requests, escalations, or mixed AI-human interactions are harder to score, improvements in the average may simply reflect which conversations entered the denominator.
Define eligibility before calibration. Spam, automated notifications, internal-only threads, and interactions with no customer request may reasonably sit outside the metric. A short conversation should not be excluded merely because it is short, and a difficult conversation should not be excluded merely because the model is uncertain. Track uncertainty explicitly rather than removing inconvenient cases from view.
Your recurring dashboard should show:
The share of eligible conversations that received a score.
The distribution across experience bands, not just an average.
The mix of positive and negative drivers.
Results split by AI-only, teammate-only, and mixed handling.
Relevant slices such as channel, language, issue type, conversation length, product area, and escalation path.
The active model, rubric, and schema versions.
Calibration should happen against human judgment before the score becomes a target. Use a representative set containing routine resolutions, short exchanges, long investigations, escalations, emotionally charged threads, AI-only conversations, human-only conversations, and AI-to-human handoffs. Have independent reviewers apply the same rubric, examine disagreements, and rewrite any criterion that depends on intuition rather than observable evidence.
Then test the slices separately. Aggregate agreement can hide systematic failure in one language, channel, issue class, or interaction type. The acceptable level of disagreement depends on the decision. A model used to discover recurring workflow friction can tolerate more uncertainty than one used in individual performance management.
Keep the adjudicated examples as a regression set. Re-run them whenever you change the prompt, model, rubric, knowledge architecture, conversation parser, or driver definitions. Review newly common failure patterns as well; a frozen evaluation set eventually stops representing the work.
Model changes require visible reporting boundaries. A more contextual scoring system may produce a one-time shift without a corresponding decline in support quality. Backfill historical conversations with the new version when that is practical. Otherwise, annotate the change on every trend view and establish a new baseline. Never splice two scoring regimes into one continuous line and ask leaders to interpret the movement as operational performance.
Turn low scores into routed work, not dashboard theatre
A low score is only a symptom. The driver determines who should investigate it and what kind of intervention is plausible. Sending every poor experience to the support manager guarantees that product defects, policy choices, and broken workflows will be misclassified as coaching problems.
Primary driver
What to inspect
Primary owner
Default next action
AI answer quality
Inaccuracy, contradiction, irrelevant guidance, or repeated clarification
AI product and knowledge owners
Correct the underlying knowledge or response path, then add the failure to the AI evaluation set
Teammate answer quality
Unclear explanation, incorrect guidance, missed question, or inconsistent commitment
Support lead or enablement owner
Review the conversation against the rubric, then improve coaching, documentation, or access to information
Map the failing transition and remove the avoidable step, ownership gap, or workflow rule
Product or service feedback
Bug, missing capability, confusing design, reliability issue, delivery failure, or service breakdown
Relevant Product, Engineering, or service owner
Cluster related conversations, connect them to the product area, and decide whether the response is a fix, discovery work, or an explicit trade-off
Policy feedback
Refund, return, eligibility, account, usage, or limit rule
Business or operations owner responsible for the policy
Separate unclear communication from disagreement with the policy, then revise the explanation, the policy, or neither – deliberately
Strong negative emotion
The event that triggered the emotion and whether the issue remains unresolved
Triage owner, followed by the owner of the actual cause
Prioritize review where appropriate, but do not treat emotion alone as proof of agent failure
Automation should route the evidence package, not just the score. Include the conversation link, customer request, outcome, overall band, driver codes, supporting messages, scoring version, and proposed owner. That context lets the receiving team judge the issue without rereading an entire thread or trusting an opaque summary.
Use separate operating lanes for individual cases and recurring patterns. A materially incorrect answer may need immediate review. Repeated handoff friction usually needs aggregation so Operations can see the broken transition. Product and policy feedback becomes useful when related conversations are clustered around a shared problem, while still retaining representative examples.
Count affected conversations consistently rather than allowing a verbose customer to create many separate votes within one thread. Preserve the denominator for every filter. A driver that appears frequently in one product area may look dominant in a filtered dashboard while remaining uncommon across the full support mix.
For recurring themes, maintain a problem record with the driver, affected journey, frequency, severity, controllable owner, proposed intervention, status, and comparable post-change cohort. This converts conversation scoring into a product and operations feedback loop. Without that record, the same issue can be rediscovered in every review without anyone becoming accountable for changing it.
After an intervention, compare like with like: the same scoring version, eligibility rules, issue type, and relevant handling path. If the score improves but coverage falls, or the issue mix changes, you do not yet know whether the intervention worked.
Earn the right to replace CSAT
Conversation scoring addresses a real blind spot: survey metrics describe the customers who choose to respond, while a conversation-based system can evaluate a much broader share of eligible support volume. That makes it attractive as a replacement for CSAT, but broader coverage does not automatically make the new metric valid.
Start in shadow mode. Continue the existing reporting while you calibrate the new score, inspect disagreements, and learn which drivers are actionable. Do not demand that the two measures match. They observe different things: one evaluates evidence in the interaction, while the other records a respondent’s self-reported reaction.
Move the conversation score into operational reviews once teams can inspect its reasoning and route its drivers. Move it into executive reporting only after coverage, version changes, and slice-level performance are visible. Consider reducing or retiring a survey only when all of the following are true:
Eligibility and coverage are stable enough that changes in the denominator cannot masquerade as experience improvements.
The rubric has been calibrated against human review, including difficult and ambiguous conversations.
Explanations consistently point to transcript evidence rather than merely producing plausible prose.
Important channels, languages, issue classes, and AI-human handling paths have been checked separately.
Model and rubric changes are versioned, regression-tested, and visibly marked in reporting.
Driver routing produces owned work, and teams can show what they changed because of the signal.
Material disagreements between the conversation score and survey feedback are investigated rather than averaged away.
Keep a higher standard for individual performance decisions. A conversation score can flag work for human QA, but it should not become an automatic employee rating merely because it covers more conversations. Product limitations, customer history, policy constraints, and model error can all affect the result. Use the driver record and human review to establish what the teammate actually controlled.
Key takeaways
Measure the customer’s experience separately from the performance of the AI agent, teammate, product, policy, or workflow that shaped it.
Keep an overall band for scanning, but preserve outcome, answer quality, effort, emotion, feedback drivers, evidence, and version metadata underneath it.
Report coverage and score distribution together; an unexplained denominator change can invalidate the trend.
Calibrate with representative human-reviewed conversations and retest meaningful slices after every scoring change.
Route each driver to the owner who can change it, then measure a comparable cohort after the intervention.
Replace CSAT only after the conversation score has earned trust as both a measurement system and an operating loop.
At your next customer experience review, bring one low-scoring conversation, its evidence-backed driver record, and the owner capable of changing that driver. If the meeting ends with only a debate about whether the number is fair, calibration is unfinished. If it ends with a named intervention and a valid way to examine comparable future conversations, the score is doing useful work.
