Tag: customer support ai strategy

How to Operate AI Customer Agents as a Reliable CX System
AI customer agents are expanding from answering routine questions toward handling complex workflows and potentially supporting more of the customer lifecycle. The operational challenge is no longer simply whether an agent can produce a plausible answer. It is whether the organization can keep that agent accurate, controlled, measurable, and ready whenever the business changes.

Taken together, the source reports point to a practical operating model: connect product releases to knowledge updates, test behavior before exposure, measure the full interaction rather than a narrow survey sample, and assign people to improve the system continuously. That turns an AI agent from a channel feature into managed CX infrastructure.

Key takeaways
- Agent reliability depends on a continuous train, test, deploy, and analyze cycle, not a one-time implementation.
- A product release is not operationally complete until the agent has current, unambiguous, and retrievable information about it.
- Pre-release evaluation should test realistic customer questions, policy conditions, system actions, and required human handoffs.
- Survey metrics remain useful, but conversation-level analysis provides broader visibility into answer quality, effort, sentiment, and recurring friction.
- Human roles increasingly shift toward knowledge stewardship, exception handling, policy design, evaluation, and cross-functional CX improvement.
Treat the agent as a product system, not a chatbot

The Pioneer 2025 report describes Fin 3 through four operating stages: training, testing, deployment, and analysis. It reports that Procedures combines natural-language instructions with deterministic controls for complex work, while Simulations is intended to test behavior before customers encounter it. The report also describes deployment across additional channels, including Slack and Discord, improvements to Voice, and analytics features such as CX Score Reasons and Topic Trends.

These are vendor-reported capabilities, but the underlying operating principle applies beyond one platform. An agent that can act in business systems needs more than fluent language generation. It needs explicit procedures, boundaries on what it may do, test cases that expose failure modes, controlled channel deployment, and evidence showing what happened after release.

The same report presents a longer-term Customer Agent vision built around roles, goals, persistent memory, business knowledge, and interoperability. That vision should be distinguished from currently reported product functionality. It nevertheless clarifies the governance challenge: as an agent gains continuity and operational reach, errors can travel across more stages of the customer journey. Ownership of objectives, data, permissions, escalation, and measurement therefore becomes part of CX design.

This also changes how success should be framed. Resolution volume is an operational output, but a dependable CX system must also answer whether the agent followed policy, used current knowledge, completed the intended action, recognized an exception, and left the customer with an acceptable amount of effort. Automation without those checks can move work while concealing deterioration in the experience.

Move agent readiness into the product release process

The NPI playbook focuses on a common source of agent failure: products change faster than their supporting knowledge. When a feature launches without usable documentation, the source reports that the agent may hand conversations to people just as launch-related volume rises. The resulting backlog is therefore not only a support problem; it is a release-readiness problem.

A stronger definition of done includes agent readiness. The NPI source recommends bringing support or knowledge specialists into product walkthroughs, product marketing kick-offs, and pre-release testing. It also calls for a named owner, whether an NPI manager, knowledge manager, support lead, or product operations owner. The title can vary, but accountability cannot be distributed so widely that nobody verifies readiness.

The required knowledge must be designed for retrieval as well as human reading. According to the source, documentation should include both internal feature names and the phrases customers actually use, expand acronyms, state plan and availability conditions explicitly, and reproduce the substance of screenshots or videos in text. This is important because information can be technically present yet remain difficult for an agent to retrieve or apply correctly.

Release work must also remove knowledge that a launch has invalidated. Searching related articles, macros, notes, and workflows can reveal stale or contradictory guidance. Duplicate content deserves particular attention: competing versions of an answer can create inconsistent agent behavior even when the newest article is accurate.

Testing then connects knowledge preparation to customer outcomes. The NPI playbook recommends assembling likely questions from launch content, beta feedback, and early support conversations; running them in the environment customers will use; rating the answers; correcting the underlying content or structure; and repeating the evaluation. Conditions such as phased rollout, plan eligibility, regional availability, and mandatory human escalation require explicit coverage rather than an assumption that the agent will infer the right behavior.

This creates a two-speed control model. Before launch, teams test expected questions and known edge cases. After launch, they watch real conversations for unexpected language, missing scenarios, or product behavior that the original documentation did not anticipate. The feedback should return to the release tracker, knowledge source, procedure, or product team according to the root cause.

Measure experience at conversation scale

Release evaluation shows whether an agent appears ready, but production measurement shows whether that readiness survives real customer behavior. The CX measurement source reports that CSAT captures less than 10% of conversations and that respondents tend to represent more extreme reactions. On that account, survey results leave a large unobserved middle and cannot by themselves explain whether dissatisfaction arose from service, product behavior, or policy.

The source describes an alternative in which AI evaluates every human and agent interaction across dimensions such as service quality, resolution, and customer effort. It reports that Intercom’s CX Score assigns interactions a score from 1 to 5, exposes reasons behind the score, and gives most teams roughly five times the coverage of CSAT alone. Those product-specific claims are reported by the source rather than independently verified here, but they illustrate the broader distinction between voluntary feedback and systematic conversation review.

Fuller coverage does not make direct customer feedback obsolete. CSAT can still capture what a customer chooses to say, while conversation analysis can detect repeated explanations, handoff friction, weak answer quality, unresolved intent, and neutral interactions that generate no survey response. The two signals answer different questions and should be interpreted together rather than forced into a single interchangeable benchmark.

New coverage also requires new baselines. The measurement source cautions against transferring an old CSAT target directly to a conversation-scoring system because the populations and methods differ. It recommends correlating the new score with operational measures such as first response time and time to close, then examining underlying attributes including answer quality, customer effort, and product feedback. Its illustrative targets of 80% for Fin support, 70% for human support, and 78% overall are examples derived from the scenario described in that article, not universal standards.

Segmentation is equally important. Complex, high-touch cases should not automatically be compared with transactional contacts, and aggregate results can hide a poorly performing topic or channel. Useful analysis separates agent and human conversations, examines topics and handoffs, and preserves context about case type. The most actionable output is not the score alone but a reason that can be routed to a responsible owner.

Build one improvement loop across CX, product, and knowledge

The sources approach AI customer agents from different angles: the Pioneer report emphasizes expanding capabilities and a broader customer-agent vision; the NPI playbook concentrates on release and knowledge readiness; and the measurement article addresses visibility after deployment. Their combined implication is that these activities cannot remain separate programs.

A low-quality interaction might originate in several places. The knowledge may be missing or contradictory, the procedure may express the wrong policy, the product may behave unexpectedly, the agent may fail to retrieve applicable information, or the case may require a human specialist. Conversation-level reasons help locate the problem, but the organization still needs a route from evidence to correction and then to re-evaluation.

That operating loop changes human work. Customer-facing specialists remain essential for sensitive, ambiguous, or exceptional cases, while also contributing customer language, testing scenarios, escalation criteria, and knowledge improvements. Product and engineering teams become accountable for the support consequences of releases. Knowledge teams manage information as production input, and CX leaders set objectives that balance resolution, effort, policy compliance, and service quality.

The most revealing opportunities may sit in interactions that are neither failures nor successes. Broader conversation analysis can surface answers that were technically acceptable but unnecessarily difficult, impersonal, or incomplete. Improving that middle ground requires more than tuning a model: it may require clearer documentation, a better workflow, a product fix, or a different escalation rule.

As agents acquire more roles, memory, knowledge, and access to business systems, CX operations will increasingly resemble product operations for a continuously changing service. Organizations that establish release gates, evaluation sets, conversation-level diagnostics, and unambiguous ownership will be better positioned to expand agent responsibility without allowing reliability to become an afterthought.

References
July 3, 2026
How I Make AI Agents Speak Like Our Team: A Conversation Design Playbook That Lifts CSAT

If nobody on our team trains the Agent on how to communicate, it will sound like an LLM when it speaks to customers—because it is one. I never want a customer to feel like they’re talking to a machine that doesn’t get them. That’s why I treat conversation design as a core product capability, not an afterthought.

Conversation design is an emerging discipline in AI-first support teams built to solve this exact problem. In practice, I make someone explicitly own how the Agent communicates—tone, structure, level of detail, customer experience, and the handoff and escalation process—because that’s where trust is won or lost.

When there’s no clear owner and no explicit guidance, the Agent starts making its own choices. I’ve seen it over-explain when a short answer would do, reply in a flat tone when a customer is frustrated, or trigger a handoff too late. None of those are model problems; they’re design problems.

The cost is measurable. Customers who get awkwardly structured responses won’t trust the answer—even when it’s accurate—so they escalate to a human to hear the same thing phrased differently. Others will skip the Agent entirely. And when the Agent does hand off, a poor transition means the support rep inherits a frustrated customer. Every one of these outcomes is avoidable; conversation design exists to prevent them.

I’ve seen A/B tests where a warmer, more conversational opening message meaningfully lifted customer satisfaction—CSAT moved from 72.8% to 78.4%. A single design change, applied to the very first message, drove a measurable difference. That’s the kind of leverage I look for as a product leader.

Here’s the scope I use when I talk about conversation design—five areas that shape the customer experience end to end:

1) Tone and personality: Define the Agent’s voice, level of detail, and how formal or casual it should sound—and specify where that register adapts to the situation (for example, urgent access issues versus exploratory product questions).

Design how your AI agent talks. Set tone, style, and product naming rules, then preview replies instantly. Clear callouts showcase brand voice consistency and flexible formatting so your bot communicates like your team.

2) Response structure: Ensure the Agent matches the level of detail to the customer’s request, keeping answers tight when the ask is simple and expanding only when complexity demands it.

3) Handoff logic: Decide when to escalate, how to communicate the transition, and what context to carry over so the human teammate can help immediately without rework.

4) Interaction flow: Map how a conversation progresses—clarifying questions, answers, resolution, or handoff—and design for smooth pivots when customers change direction.

5) Response quality: Go beyond technical correctness to ensure answers feel clear, helpful, and on-brand. Accuracy without clarity erodes trust.

To put this into practice, I start with the feel of the conversation. Before tuning individual responses, I write down one tight paragraph describing the Agent’s voice. I don’t need a full brand bible—just a north star I can use to make consistent decisions about tone. The voice stays consistent, while the register adapts to the context: a locked-out customer needs directness and speed; a feature explorer might value more context and examples.

I design the handoff with extreme care because it’s one of the highest-friction moments. Customers shouldn’t have to re-explain anything. The support rep should receive the full conversation history, the underlying context, what the Agent already tried, and why the escalation happened. Even the phrasing matters—“Let me connect you with a teammate who can help with this” feels very different from a silent handover.

The new CX Score adds context to every conversation: a donut chart surfaces drivers like policy feedback and effort, while a side panel explains why this interaction earned a 3 based on signals from an AI agent chat.

I also build a failsafe. If the Agent can’t resolve the issue cleanly, a graceful fallback still gives the customer a smooth experience. A customer might be frustrated with AI at that point, but a well-handled transition can turn that around.

Follow-ups deserve the same rigor as handoffs. If someone drops mid-conversation—with the Agent or a human—how do we reach back out to confirm they got what they needed? Most teams miss this moment; customers don’t.

Another common pitfall is over-explaining. The Agent has access to a lot of information, and left unguided, it will overshare. The fix is simple: match the answer’s depth to the question. A password reset shouldn’t take three paragraphs; a complex integration might. When there’s more to offer, the Agent should ask before expanding.

I also design for the conversation the customer is actually having—not the script I wish they’d follow. Customers change direction, stack questions, or bring up unrelated follow-ups. The Agent should pivot with them, not force them back into a rigid flow. I also consider whether flows vary by channel and whether different segments merit distinct experiences.

On the instruction side, I keep guidance short. Teams often react to edge cases by adding more rules until the LLM is parsing paragraphs before it can reply. I’ve seen it everywhere. My rule: if it’s about content or information, it belongs in the knowledge base. If it’s about tone or handling specific situations, it belongs in the Agent’s instructions. “Be direct about pricing” does more than a paragraph explaining the philosophy behind your pricing communication strategy.

If you’re using Fin, much of this work happens in Guidance. It’s where conversation design takes shape, helping you define how the Agent should sound, how much it should say, and how it should respond in different situations.

On a crisp grid, 'Blueprint' appears as editable vector paths, underscoring a methodical plan. The image promotes the AI Agent Blueprint—a framework to launch and scale customer service automation with confidence.

Most teams won’t hire a dedicated conversation designer on day one—that’s fine. But someone still needs to own the Agent’s communication, even if it’s part of an existing role. I’ve often seen this start within support operations or knowledge management. As the Agent scales to more conversations, the responsibility becomes formal—and eventually becomes a dedicated role.

Here’s how I’d start, step by step:

1) Name an owner. Make accountability explicit; it doesn’t have to be a new hire.

2) Pick one conversation type that isn’t landing well. Look for cases where the Agent answered correctly but the customer still escalated or left negative feedback. If you’re using Fin, CX Score can help you surface these; it shows which topics and conversation types are scoring poorly and why, so you can see whether the issue is answer quality, customer effort, or something else.

3) Audit the Agent’s instructions. If they’ve grown beyond a few focused rules, trim them. Move content into the knowledge base and keep instructions focused on behavior.

4) Fix your worst handoff. Review a handful of conversations that escalated. Did the customer have to repeat themselves? Did the rep have enough context? Redesign that single transition first.

The impact of these small improvements compounds. A warmer opening can lift CSAT, trimming instructions makes responses sharper, and a better handoff prevents reps from inheriting frustrated customers. None of this requires new knowledge—just someone paying close attention to the conversation itself and designing it with intention.

Inspired by this post on The Intercom Blog.

June 18, 2026
Salesforce to Acquire Fin for ~$3.6B: Powerful AI Synergy, Product Strategy Takeaways

I’m processing a milestone moment for SaaS, AI strategy, and product leadership. One statement captures the news with clarity: “We’re excited to share that we just signed an agreement for Salesforce to acquire Fin for ~$3.6B. The transaction is expected to close in the fourth quarter of Salesforce’s fiscal year 2027.” As a product leader, I see this as a high-conviction bet on agentic AI, Customer Agents, and CRM integration at massive scale.

The backstory matters, and it’s remarkable: “Fin started as Intercom 15 years ago. We changed our name to cap our transformation just weeks ago. We were a darling of the SaaS era and invented so many of the patterns you see in software today. Nearly four years ago, in need of a reboot, we jumped on weeks-old modern LLMs to create and define the category we know as Customer Agents today.” That arc—from SaaS pioneer to LLM-powered category creator—illustrates how bold pivots, shipped with urgency and clear product strategy, can reset the trajectory of a company and a market.

From a product management lens, this deal reinforces a few truths: category creation rewards those who move first with conviction; “reboots” succeed when they’re anchored in genuine customer value; and modern LLMs, applied through disciplined roadmapping and eval-driven development, can unlock step-change outcomes in customer support ai strategy and product-led growth. It also signals the rising centrality of agentic AI and operational AI workflows inside the CRM.

The leadership dimension is just as instructive. As the announcement framed it: “Salesforce invented modern software and SaaS. And Marc Benioff is like the final boss of tech founder CEOs. In seat for 27 years, he’s one of the last of his era. Still pushing, pivoting, placing big bets.” That ethos—placing big, principled bets while adapting the operating model—sets the tone for what sustained product management leadership looks like at scale.

Customer continuity and acceleration are clearly emphasized: “To our customers: Over the past few years we’ve been shipping intensely. Including recently our groundbreaking model, Apex, and our paradigm-defining internal agent, Operator. With the resources of Salesforce this will only accelerate. And yet little will practically change. I’ll still be CEO, Des will still be running R&D, we’ll both still be committed to continuing to lead this category. Thank you very sincerely and deeply for your belief in us.” For practitioners, the signal is strong: continued focus on shipping, sharper execution readiness, and tighter integration paths inside the Salesforce ecosystem.

Smiles, clinking glasses, and a roundtable toast in a cozy private room capture the energy of a big day—celebrating Salesforce's definitive agreement to acquire Fin and the teams joining forces for what's next.

There’s a human heartbeat here too: “While this is not the end, it is a major, pivotal, special, and emotional moment for us.” Moments like this remind me that building enduring products is equal parts craft and courage—powered by teams who commit to the long game, navigate uncertainty, and still ship relentlessly.

Strategically, I expect near-term priorities to center on secure data flow and governance, deep CRM integration, and unifying telemetry for Agent Analytics across channels. On the roadmap, I’d anticipate tighter alignment between LLM safety, retrieval-first pipelines, and enterprise-grade observability—plus thoughtful go-to-market strategy enabling sales-led growth to complement product-led growth. The real unlock comes when Customer Agents are natively orchestrated with Service, Sales, and Marketing workflows—measured with clear outcomes vs output OKRs and reinforced by robust knowledge management.

For fellow product leaders, the takeaways are actionable: define category boundaries with crisp value propositions; balance speed with governance; invest in eval-driven development and continuous discovery; and keep your product trios aligned around measurable customer outcomes. Above all, build the operating cadence—metrics, rituals, and talent—that lets you compound small wins into durable differentiation.

And I appreciate the spirit of this closing line: “And now, time to get back to work. See you at our next product launch in a few weeks. (:” That’s the mindset that turns a headline into execution: celebrate briefly, then ship the next proof point.

Inspired by this post on The Intercom Blog.

June 15, 2026

Secure System Access for AI Agents: A Phased Control Model

An AI agent becomes operationally valuable when it can move beyond explaining a process and complete the underlying work. That same transition gives the agent access to sensitive data and consequential actions, so integration must be designed as both a product capability and a security boundary.

The practical objective is not maximum access. It is the smallest dependable set of permissions that lets an agent resolve a well-defined workflow, supported by deterministic controls, observable outcomes, and a clear path to human intervention.

System access changes both the value and the risk

Without backend access, an agent can describe how to update an account, check a renewal, or report a damaged order. With access to a CRM, billing platform, or order-management system, it can potentially retrieve the relevant record and complete the request during the conversation. The Intercom article presents this shift from answering to acting as a central difference between basic AI adoption and mature deployment.

The article cites Intercom’s 2026 Customer Service Transformation Report, reporting improved metrics among 87% of teams with mature AI deployments, compared with 62% overall. It also reports that 82% of senior leaders said their teams had invested in AI during the preceding year, while only 10% said they had reached mature deployment. These source-reported figures suggest an integration gap, but they do not independently establish that system access caused the reported improvements or that an integration is secure.

Security therefore cannot be added after the workflow succeeds. A customer-facing interface may remove the need to visit a separate application, but it must not remove identity and authorization checks. The agent still needs a trustworthy way to associate the request with the correct customer, determine what that customer is permitted to do, and constrain the backend operation accordingly.

Choose workflows where access justifies its complexity

Not every automated conversation benefits equally from deeper integration. Intercom reports the results of rebuilding four fixed, scripted Tasks as Procedures with system access. Over the 12 months through May 2026, the reported resolution rate for its bounce-list workflow rose from 9.3% to 79.9%, while bug reporting increased from 9.2% to 66.5%. Email forwarding moved from 44.9% to 66.5%, but Messenger installation rose only from 67% to 69.2%.

The variation is more instructive than the headline gains. According to the article, the bounce-list process required multi-step reasoning, dynamic branches, and error recovery. Bug reporting still ended in a human handoff, but the procedure improved that handoff by pre-triaging the issue, surfacing possible GitHub matches, extracting relevant URLs, and requesting impersonation access. Messenger installation was already a comparatively linear process, leaving less room for improvement.

A suitable first integration is therefore not merely a popular support topic. It should be high-volume and repeatable, have an identifiable system owner, and depend on live data or actions that materially change the outcome. Existing APIs improve feasibility, but the security review should also consider data sensitivity, reversibility, authorization complexity, and the consequences of acting on an ambiguous request.

Use an access ladder instead of a single launch

The phased approach described by Intercom can also serve as a security model. Each stage expands capability only after the workflow and its controls have produced enough evidence to justify the next step.

Stage	Agent capability	Appropriate use	Control emphasis
No integration	Guide, troubleshoot, check policy, triage, and route	Discover where explanations repeatedly lead to manual work	Evaluate answer quality, routing accuracy, and escalation behavior
Read-only access	Retrieve approved fields such as order or subscription status	Resolve information requests without changing a record	Restrict endpoints, records, and fields; verify customer authorization
Write access	Update records or initiate actions such as cancellations or refunds	Complete bounded workflows after earlier stages are dependable	Validate inputs, limit action scope, record outcomes, and require approval where consequences warrant it

Mock responses can test branching logic before an API is ready, as the Intercom article notes. It also proposes a temporary human-in-the-loop step when an integration is still several engineering sprints away. These methods can validate the workflow and expose missing requirements, but simulated success should not be treated as proof that production identity, authorization, failure recovery, and audit controls are ready.

Put deterministic controls around probabilistic decisions

Plain-language workflow instructions can guide an agent, but security-critical constraints should not depend solely on the model interpreting those instructions correctly. A safer architecture places enforceable controls between the agent and each backend system.

Control	Practical design implication
Dedicated identity	Give the agent its own service identity rather than borrowing a staff account, so permissions and activity remain attributable.
Least privilege	Allow only the endpoints, operations, records, and fields required by the selected workflow.
Read and write separation	Keep retrieval permissions distinct from mutation permissions and grant write access only when the use case requires it.
Independent policy enforcement	Validate identity, authorization, limits, and required inputs outside the model before executing an operation.
Bounded actions	Prefer narrow, purpose-built operations over unrestricted database or administrative access.
Human approval and escalation	Route ambiguous, exceptional, sensitive, or difficult-to-reverse cases to an authorized person.
Auditability and monitoring	Record the request, decision, tool call, result, and escalation so failures and unusual patterns can be investigated.
Safe failure behavior	Prevent retries, timeouts, or partial completion from producing duplicated or inconsistent changes.

The integration request should document the workflow in plain language, identify every read and write point, name the system owner, and specify the minimum required fields. It should also define how success and harm will be measured: not only whether the agent completed the conversation, but whether it selected the correct record, performed the authorized action once, protected restricted data, and escalated when it lacked sufficient confidence or permission.

This framing also improves the business case. Engineering is being asked to expose a narrowly scoped capability with explicit boundaries, rather than to provide broad access to a general-purpose agent. Leadership can then compare measurable workflow value with implementation effort and residual risk.

Key takeaways

System access creates value when it lets an agent complete work, but it simultaneously expands the security boundary.
The best initial workflow is frequent, bounded, operationally meaningful, and owned by a team that can approve its data and actions.
Progress from no integration to read-only retrieval and then to narrowly scoped write operations; do not treat access as an all-or-nothing decision.
Enforce identity, authorization, field restrictions, action limits, and audit logging outside the model’s natural-language instructions.
Evaluate correctness, unauthorized-action risk, failure recovery, and handoff quality alongside resolution rate.

The strongest long-term pattern is a portfolio of small, governed capabilities rather than one broadly privileged agent. Each successful workflow can supply the evidence needed to extend access deliberately, while keeping the consequences of error visible and contained.

References

Intercom — Win Executive Buy-In for AI Agent System Access: Unlock Actions, Boost Resolution, Cut Costs

June 11, 2026

Why We Made Fin the Most Open Agent: Instant HubSpot & Freshdesk Support With 76% Resolutions

I’ve spent my career pairing product strategy with customer reality, and nothing is more clear right now than the demand for openness and speed. Today, we’re announcing that Fin can be used as a Service Agent on top of HubSpot and Freshworks, meaning you can use the world’s best Agent without migrating off your helpdesk.

Hubspot and Freshdesk customers can now:

Get Fin live, integrated, and working seamlessly in less than an hour.

Delivering a 76% average resolution rate.

Across all customer channels (voice, email, chat, social, and more).

Resolving complex queries that require reading and writing to third party systems.

With everything fully configurable to follow the unique policies of every individual business.

This launch is a very visible step in a journey we’ve been on from day one: building an open, customer-first platform that plays well with the rest of your stack. We’ve long known that businesses want flexibility in how they configure their customer-facing tech stack. Since the very beginning, we have built Fin as an open platform, with APIs, MCPs, CLI, and opening up access to Apex, our proprietary trained model that delivers best in class performance.

To make things easy for our customers, we have extensive public documentation of our product on our website, in our help center, and in our developer docs. We are the only Agent company in our space to do this, others hide most details behind sign-in screens, which we don’t believe is the right thing to do.

Open Agent platforms will win because customers refuse to be boxed into closed ecosystems. We now believe our category has reached a stage where customers demand open platforms, that those who open up are more likely to win, and those who remain closed and protectionist will accelerate their demise.

We are operating in a fast changing world, and customers do not want to be locked into a single vendor or closed ecosystem. They want the ability to experiment, to swap things in and out, and move everything with ease, technically and commercially.

In an open world, the best product will win. In a world where businesses can easily swap vendors, the best product will win. We are happy to compete on that front, confident that Fin delivers the best customer experience and the highest performance.

From a product management lens, this openness is powered by agentic AI patterns paired with robust CRM integration. Under the hood, we use Model Context Protocol (MCP), well-documented APIs, and orchestrated AI workflows to read from and write to third-party systems. That’s how Fin handles true multi-channel work—including voice AI agent scenarios—while giving teams the observability they need through Agent Analytics.

If you are a Hubspot or Freshdesk customer, you can now have Fin integrated and live within an hour, without needing any help from us. We’re here if you want us, but as part of our commitment to building an open platform, we’ve designed everything to be self-servable—start in minutes or watch a quick demo of how everything works.

Fin for Hubspot

Fin for Freshdesk

Inspired by this post on The Intercom Blog.

June 9, 2026
A Game-Changing Leap in Voice AI: Fin Voice 2, Apex Flash, and a Live Demo You Can Trust

In competitive markets, I see two options: try to win the game competitors set, or choose to play a different game. In the "Customer Agents" category, I’ve watched too many glossy, fabricated demos—especially around voice—mask the real challenges. Voice is just extremely hard. We all know the future of customer experiences will be Agent-driven voice, yet most of us haven’t actually spoken with a modern AI Agent when calling a business because the tech hasn’t been truly ready in the wild. Today, the bar moves.

What changed? There’s a live, public demo of cutting-edge voice tech you can stress test yourself—no smoke, no mirrors. I recommend taking it for a spin: https://fin.ai/voice. It’s fast, natural, and, yes, very, very good.

For context, yesterday brought Apex Flash, their newest and fastest model, built for the unique demands of low latency channels like voice. Today comes Fin Voice 2, a major upgrade to Fin Voice with over 20 new features, and the first product built on Apex Flash.

Here are the three things that stood out to me—and why they matter for customer support AI strategy and product strategy.

First — thanks to Apex Flash, Fin Voice 2 is now the fastest, most natural Agent for phone, with higher resolution rates and customer satisfaction scores than ever before. Apex Flash is trained on millions of customer experience interactions, fine tuned for customer service, and can be configured to understand all your knowledge and follow all your policies. The result is higher resolution at significantly lower latency—the best of both worlds for voice AI agent performance.

Speed and naturalness here aren’t accidental. Most voice AI products are slow because they convert speech to text, send it to a general model, get a text answer, and then convert it back to speech. Fin Voice 2 was designed to work differently, separating the real time layer that handles speech processing, and the layer that generates answers. That architecture is purpose-built for the demands of customer service on voice.

Powered by Apex Flash, Fin Voice 2 raises the bar on quality and speed—boosting resolution rates and guidance following while cutting time to first audio and semantic search latency, with a lift in CSAT too.

Second — Fin Voice 2 can handle complex queries end to end: taking actions in external systems, verifying callers’ identities, processing refunds, booking appointments, and more. Phone is a high-stakes channel, and Fin adapts to customers across emotional states, clarifies when needed, and confirms key details before taking action. Most of the time, Fin can resolve the query in full, and when it can’t, it seamlessly hands off to the human team, maintaining full customer context and history. You also get multiple improvements to call quality, plus proactive outbound calls to follow up on unresolved issues—all orchestrated by robust AI workflows.

Third — Fin Voice 2 gives you total control with industry-leading tools to configure and manage how Fin behaves. You get rich, detailed insights into call behavior and quality, the most common topics of calls, and one-click recommendations to improve. As with everything in Fin, you can fully self-serve and then manage it all with ease, without requiring professional services. Many vendors only let you set up their voice agent under supervision; with Fin, you get everything you need to iterate fast.

If you haven’t tried the demo yet, go check it out: https://fin.ai/voice. If you prefer to wait, don’t be surprised when you end up speaking with it at a favorite brand soon.

From a product management lens, this is what matters: latency is a feature customers feel; transparency builds trust in enterprise AI; and control is non-negotiable for CX leaders. The combination of a purpose-built, agentic AI architecture, measurable gains in resolution and CSAT, and true self-serve configuration signals that voice is moving from prototype theater to production reality. That’s the different game I want our industry to play.

Inspired by this post on The Intercom Blog.

June 4, 2026
Stop Support Tickets Before They Start: How AI Unsticks Users and Lifts Conversions

Every moment of friction in a product carries a hidden cost: attention drifts, motivation wanes, and the next click becomes a support ticket—or worse, silent churn. Over the years, I’ve learned to treat “stuck” as an urgent product signal, not just an operational nuisance. When we unstick users in the flow, we protect revenue, brand trust, and the momentum that powers product-led growth.

Learn how Amplitude’s Global Support team uses AI Assistant to reduce support tickets, prevent user churn, and increase conversions.

I reference that line often because it captures a proven pattern: meet users where confusion peaks and resolve it instantly. In my practice, the formula is straightforward—pair behavioral analytics and session replay with a just-in-time AI Assistant, routed by clear driver trees. This transforms support from reactive firefighting into a proactive, in-product experience that accelerates onboarding and boosts user activation.

Here’s how I operationalize it. First, I use Amplitude analytics and behavioral analytics to surface high-friction steps—pages with elevated drop-off, loops, or rage clicks. Session replay clarifies the “why” behind the numbers, while cohort and retention analysis reveal who’s most at risk. Then I deploy targeted in-app guides and tooltip design to preempt known pitfalls, while an AI Assistant handles real-time questions with context from our knowledge base and product docs.

The AI Assistant is more than a chatbot. With well-structured AI workflows, it detects intent, pulls precise snippets from docs-as-code, and handles routine issues instantly. When complexity spikes, it executes a graceful handoff to consultative support via Intercom or a Zendesk integration—preserving conversation history and sentiment cues—so humans spend time where judgment matters. This hybrid model keeps response times low without sacrificing quality.

To de-risk changes, I lean on A/B testing and feature flags. I measure time-to-value, activation rate, and funnel conversion as leading indicators, while tracking ticket deflection, CSAT, and NRR as trailing indicators. The goal isn’t just fewer tickets; it’s faster learning loops and a compounding improvement in user outcomes. When we see activation curves steepen and onboarding friction flatten, we know the system is working.

Practically, I start with the top three friction points in onboarding, implement narrow in-app guides, and deploy the AI Assistant with strict guardrails and clear escalation paths. Weekly reviews align product, customer success, and solutions engineering around shared telemetry—so we tune prompts, content, and UI patterns together. Over time, I’ve seen ticket volume decline meaningfully, while conversion and retention rise as users experience fewer dead ends.

If you’re evaluating where to begin, identify the moments where confusion compounds—pricing configuration, integrations, and data mapping are common culprits. Then introduce targeted, context-aware help right where users hesitate. You’ll not only prevent “every stuck user” from turning into a ticket—you’ll convert friction into confidence, and confidence into growth.

Inspired by this post on Amplitude – Best Practices.

June 1, 2026
AI-Ready Customer Support: An Operating Model That Works
You may already have an AI agent, a vendor shortlist, or pressure to automate more tickets. But if your policies conflict, ownership is unclear, and agents routinely rely on tribal knowledge, adding AI will expose those weaknesses at customer speed.

The practical goal is not to make every ticket autonomous. It is to build a support operation in which AI can resolve the right issues, recognize when it lacks authority or information, and help your team improve the system after every failure.

Key takeaways
- Start with a bounded customer problem, not a general mandate to automate support.
- Treat knowledge as a controlled production input with owners, audience rules, and review triggers.
- Define acceptable outcomes, prohibited actions, and escalation conditions before configuring the agent.
- Preserve a middle path where a human can unblock the AI without taking over the entire conversation.
- Expand automation only when evaluation results, live outcomes, and operational ownership support it.
Start with the queue, not the model

An AI-ready operation begins with a resolvable job. “Handle customer support” is too broad. “Help authenticated customers update their billing details under the current policy” is something you can document, test, monitor, and constrain.

Choose an initial queue where demand is meaningful, the desired outcome is clear, and the governing policy is reasonably stable. Avoid starting with cases that depend on negotiation, undocumented exceptions, or several teams making judgment calls behind the scenes. Those cases may become suitable later, but they are poor places to learn basic operational control.

Review a representative slice of conversations from that queue. For each one, record the customer’s intent, the information required, the systems touched, the policy applied, the final outcome, and any human judgment that changed the path. This turns a pile of tickets into a resolution map.

Pay special attention to cases that look identical at first but require different actions. A refund request may depend on plan type, purchase date, account state, or a regulatory restriction. These branches are where a fluent answer can still be operationally wrong.

You also need to decide where AI will sit. In most established operations, the safer path is to work through the support systems, queues, and reporting practices your team already uses. Replacing the help desk and automating the work at the same time creates two migrations and makes failures harder to diagnose.

Turn knowledge into a controlled production input

Your help center is only one part of the answer set. Reliable support may also depend on internal runbooks, policy clarifications, troubleshooting steps, approved reply snippets, product limitations, escalation instructions, and information held by product or customer success teams.

Bring those materials into a governed knowledge inventory. Every record should answer seven operational questions:
<!– wp:list {
May 28, 2026
Beyond Accuracy: How I Evaluate AI Customer Service Agents That Delight and Scale
When teams evaluate AI Agent options for customer service, I often see the rigor aimed at the wrong subset of criteria. After leading and observing dozens of proof of concept (POC) efforts with our customers and prospects, I understand why performance—accuracy scores, resolution rates, and benchmark tests on curated datasets—soaks up most of the attention. But those indicators alone won’t guarantee success once you leave the sandbox and face real customers.

If your POC only proves that the AI “works,” you’re missing the bigger picture. Here’s what else I look for to make the best long-term decision.

How does it handle your real-world setup?

Performance is table stakes, but it has to reflect the messiness of an actual support environment. The best-performing Agents don’t just get answers right—they exhibit resilient, human-like behavior under pressure. I watch how the Agent behaves when it doesn’t know an answer: does it recover or spiral? Does it stay on track through multi-step requests, and how gracefully does it hand off to human agents? If your knowledge base depends on a retrieval-first pipeline, test cross-source retrieval and grounding—not just single-document lookups.

When I build evaluation scenarios, I put the Agent through its paces with a broad, realistic mix:
- Multi-turn queries that require the Agent to carry context across a conversation, not just answer isolated questions.
- Vague or fragmented inputs, like typos, grammatical errors, and incomplete questions, because that’s how customers actually write.
- Edge cases and sensitive scenarios, like billing disputes, frustrated customers, and questions that sit at the boundary of what the Agent is trained on.
- Different phrasings of the same question. An Agent that handles one version well but fails on a rephrasing has a knowledge problem, not a performance problem.
- Queries that require pulling from multiple knowledge sources. Real issues are rarely answered by a single help article, and an Agent that can only handle single-source questions will hit a ceiling fast.
- Multilingual conversations, if your customer base requires it. Performance can vary significantly across languages and it’s better to discover that in testing than in production.
This preparation is worth the effort. Any Agent can look impressive in a demo; what matters is how it holds up as part of your team, serving your customers in production.

What does it feel like to interact with the Agent?

Two AI Agents can post the same quantitative scores—resolution rates, containment rate, and more—and still deliver very different customer experiences. Resolution rate tells me whether the Agent finishes conversations; it says nothing about how customers felt during them. I deliberately assess the experience, not just the outcome, because conversation design shapes trust and brand perception.

Here’s what I look for to ensure the AI Agent is enjoyable to interact with:
- Is the tone natural and on-brand, or does it feel robotic and generic?
- Does it build trust early in the conversation, or does it create friction that makes customers want to immediately request a human?
- When it doesn’t know the answer, does it handle that gracefully?
- When it hands off to a human, is that transition seamless, or does the customer feel abandoned?
As George Dilthey at Clay put it when evaluating their AI setup: “Keep what’s important to your business up front and center. For us, that was transparency and control over the customer experience.”

That framing is exactly right. The Agent represents your brand in every conversation. Customers don’t experience “accuracy,” they experience conversations. An Agent that’s technically accurate but tonally off-brand will erode customer trust over time.

I make the experience dimension explicit in my POCs. I have people on my team—and when possible, a small cohort of real customers—interact with the Agent under realistic conditions. Then I ask how it felt, not just whether it worked.

Can you keep improving it after launch?

This is the dimension most teams don’t evaluate at all, and it’s possibly the most important one. Choosing an Agent that works today and ensures you can continuously improve the customer experience over time requires more than a functional demo. You’re buying a system that must get better every week, not just during the first sprint.

The feedback loop

Can your team easily review conversations and identify where the Agent is underperforming? Can you pinpoint specific gaps (missing knowledge, incorrect tone, poor handoff decisions) and act on them quickly? The faster the loop between “something isn’t working” and “we’ve fixed it,” the more value compounds over time. In practice, that means instrumenting conversations, leveraging Agent Analytics, tagging misroutes and tone slips, and running targeted evals on known failure modes.

The speed of iteration

When you identify a gap, how quickly can you address it? This is partly a question of tooling (how easy is it to update knowledge, refine guidance, adjust behavior?) and partly a question of team capability. The teams getting the most out of AI are the ones that have changed how they operate and made continuous improvement a part of their everyday work. They’ve committed to going all-in for the long term, not just the first few weeks when launching their AI Agent. We treat this as eval-driven development: automate evaluations that mirror real tickets, tighten prompt engineering and retrieval settings, and ship small fixes daily.

The vendor partnership

The vendor behind the Agent matters just as much as the solution itself. You’re choosing a partner for transformation that will help you evolve how your business delivers customer experience. Ask:
- How does customer feedback influence the product roadmap, and can they show you examples?
- If you have feedback on limitations or weaknesses, do they engage transparently or get defensive?
- What kind of support will you get post-launch?
- Are they shaping where AI customer experience is going, or reacting to what others are building?
How a vendor responds to those questions tells you more about the long-term relationship than any benchmark result.

What a good POC proves

If your POC only proves “the AI works,” you haven’t done enough. A strong proof of concept tests performance in realistic conditions, evaluates the experience from the customer’s perspective, and validates the system that will support continuous improvement after launch. Done well, it sets you up for long-term operational success and builds organizational AI readiness—not just a flashy demo.

Inspired by this post on The Intercom Blog.
May 22, 2026

How to Deploy an Operator AI Agent in Customer Operations

Your support team probably does not need another chatbot that summarizes a ticket on command. It needs help with the operational work surrounding every ticket: finding why escalations changed, keeping knowledge accurate, correcting broken automations, coordinating incident communication, and showing human reps what deserves attention next.

An operator AI agent can take on that work, but only if you design it as an operating system for customer operations rather than a conversational layer over support APIs. The useful version closes the loop from signal to diagnosis to tested change. The dangerous version produces plausible commentary and receives permission to act before it has earned trust.

Define the job as a closed loop, not a chat box

A customer-facing AI agent handles an individual customer’s request. An operator agent works on the system around those requests: conversations, help content, automation configuration, performance data, incident workflows, and the human queue.

That distinction changes the product requirement. The agent is not complete when it answers a question such as why escalations increased. It is complete when it can investigate the increase, identify a supported cause, determine which operational object needs attention, prepare a change, test that change where possible, and route it to the right person for approval.

Observe: Detect a question, anomaly, scheduled task, failed conversation, release brief, or incident.
Diagnose: Select the relevant metrics and attributes, inspect representative conversations, and separate recurring patterns from isolated cases.
Locate the control point: Determine whether the problem sits in knowledge, guidance, a procedure, a data connector, an automation rule, or a human workflow.
Propose: Produce a concrete artifact such as an article diff, configuration change, procedure, incident audience, or prioritized queue.
Verify: Run a simulation or another appropriate check and expose failures, edge cases, and remaining uncertainty.
Act and learn: Apply an approved change, record what happened, and monitor the affected outcome for regression.

Consider the prompt, Why did escalations rise last week? A reporting copilot returns a chart. A useful operator identifies which escalation definition applies, segments the change, reads relevant conversations, finds the repeated cause, checks whether the corresponding help content or automation is deficient, and prepares the smallest defensible correction. That progression from an operational question to an actionable proposal is already possible across analysis, knowledge maintenance, automation building, and human support workflows.

Write the acceptance criteria around that complete handoff. Require the evidence used, the proposed artifact, the scope of impact, the verification result, the named reviewer, and any action the agent is forbidden to take. If the output still leaves an operations manager rebuilding the context manually, you have a chat assistant, not an operator.

Build reliability below the model and price that work honestly

A foundation model with API access can make a persuasive prototype. It can query ticket data, summarize conversations, and write a report that appears coherent. The hard part begins when different workspaces use different fields, configurations, workflows, permissions, and definitions of success.

The model should not have to rediscover your operating rules on every run. Encode those rules in purpose-built tools and reusable skills. A tool performs one bounded operation, such as retrieving a conversation, searching knowledge, or running a defined report. A skill coordinates several tools to complete a business job, such as debugging a failed resolution or rolling a policy change through the help center.

Operator’s production architecture is described as having more than 50 tools and 10 multi-step skills. Those counts are not targets to copy. They illustrate how quickly the hidden surface area grows once an agent must do dependable operational work instead of demonstrating a few API calls.

System layer	Job it must perform	Failure you should test for	Control to add
Semantic retrieval	Find content by meaning, not only exact words	Irrelevant or incomplete evidence produces a confident diagnosis	Evaluate retrieval against real support questions and known content gaps
Attribute awareness	Know which metrics, fields, and custom attributes are populated and meaningful	The agent invents a pattern from sparse or unused fields	Expose field definitions, coverage, allowed joins, and missing-data signals
Atomic tools	Perform narrow reads or writes predictably	A broad API wrapper allows an unintended query or change	Use typed inputs, constrained scopes, explicit permissions, and structured results
Domain skills	Chain tools according to a repeatable customer-operations method	The same request follows a different process on each run	Define required steps, exit conditions, evidence, and escalation paths
Review interface	Turn reasoning into charts, diffs, tests, and proposals	A reviewer approves a wall of prose without understanding the change	Render the decision in the format appropriate to the object being changed

Semantic retrieval and attribute awareness deserve particular attention. Retrieval grounds the agent in the content that can actually answer the question. Attribute awareness stops it from treating every available field as equally meaningful. A custom field that exists but is almost never populated should not become the foundation of an operational recommendation.

Give every tool a contract before the model can call it:

The business purpose and the questions it is allowed to answer.
The read and write permissions it requires.
The preconditions that must be true before it runs.
The evidence and identifiers it must return.
Its behavior when data is missing, ambiguous, stale, or inconsistent.
The audit event, approval requirement, and rollback path for a write.

Evaluate build versus buy beyond the demonstration

A proof of concept establishes that a model can produce a plausible answer with your data. It does not establish that the answer is grounded, that the proposed action is safe, or that the system will behave consistently as configurations change.

For a build decision, include retrieval tuning, permission design, tenant isolation, tool maintenance, skill development, evaluation data, observability, proposal interfaces, audit history, rollback behavior, and on-call ownership. Also ask who will update the agent when a support object, metric definition, product policy, or API changes. If these responsibilities do not have durable owners, the internal agent will age like any other unsupported operations system.

For a buy decision, ask the vendor to demonstrate your difficult cases rather than its preferred prompts. Use a conversation with conflicting evidence, an unused custom attribute, an outdated localized article, a misconfigured rule, and a proposed write with a wide blast radius. Inspect the evidence, tool trace, permissions, diff, test result, and audit record. The quality of the generated prose is one of the least informative parts of that evaluation.

Put a proposal boundary around every material action

Moving from analysis to live changes is a different class of production problem. A wrong summary wastes time. A wrong configuration can degrade customer outcomes across every conversation that matches it. An incorrect outbound message cannot be recalled after customers have read it.

I would give the agent autonomy according to consequence, not according to how confident its language sounds:

Read: Search content, inspect conversations, calculate approved metrics, and assemble evidence. Run these tasks autonomously within access controls and log every operation.
Recommend: Explain a root cause or rank an opportunity. Attach the underlying conversations, segments, fields, and assumptions so a person can challenge the conclusion.
Prepare: Draft an article, procedure, rule, connector configuration, customer response, or queue. Save it as a proposal with no production effect.
Change: Publish, configure, send, or otherwise alter the live operation only after the required reviewer sees the exact scope and explicitly approves it.

A proposal is a structured change object, not a paragraph asking for trust. Production-grade operator systems can present reviewable diffs before applying changes, allowing the reviewer to accept, reject, or refine the work. The same principle should govern any operator implementation.

Your review screen should answer six questions without forcing the approver into another tool:

What object will change?
What exact fields, passages, rules, or recipients are affected?
What evidence connects the observed problem to this change?
What test ran, and which cases failed or remained untested?
Who must approve, and which permission will execute the action?
How can the change be reversed, and what cannot be reversed?

Customer outreach needs the strictest treatment because sending is effectively irreversible. Do not approve a batch from a conversational summary that hides the audience. The safe alternative is a preview containing the resolved customer list, inclusion logic, exclusions, exact message variants, delivery channel, and approver. Start by allowing the agent to prepare that package while a person performs the send.

Simulation also needs a visible place in the proposal. If the agent modifies an automation procedure, show which representative conversations were tested, the expected outcome for each, the observed outcome, and why any mismatch occurred. An overall pass label is not enough to reveal an important edge case.

Human approval is not a permanent substitute for system quality. If reviewers routinely accept proposals without inspecting them, the control has become ceremonial. Track corrections, rejections, rollbacks, and the evidence reviewers open. Use those signals to improve the relevant retrieval rule, tool, skill, or interface.

Roll out workflows in increasing order of consequence

Choose the first workflow by its operating characteristics. A strong starting candidate recurs frequently, consumes expert attention, has accessible evidence, produces a clear artifact, and has a named reviewer. It should also allow the agent to be useful before it receives broad write permission.

A practical rollout sequence looks like this:

Recurring operations analyst. Give the agent one standing question, such as what changed in escalations or automation performance. Define the metric, comparison period, relevant segments, evidence requirements, and report destination. Require links to representative conversations and allow the conclusion that no action is warranted. Compare its reasoning with an experienced operator’s review until the failure modes are understood.
Knowledge steward. Feed it a release brief or policy change. Ask it to find affected help content, identify missing coverage, and prepare article diffs in the required voice and format. Include localized variants where they exist. The reviewer should validate product behavior, instructions, links, policy language, and whether the proposed set of pages is complete before publishing.
Automation maintainer. Start with known failed conversations. Ask the agent to distinguish a content gap from a rule, procedure, guidance, or connector problem; prepare the smallest correction; define triggers and edge cases; and simulate the result. Do not grant live configuration access until the tool trace and tests make the diagnosis reproducible.
Human-operations coordinator. Use the agent to assemble an incident audience, draft targeted responses, prepare coaching evidence, or prioritize a rep’s queue. These workflows can save substantial coordination time, but they touch customer communication and employee decisions. Begin in preparation mode, expose the selection logic, and expand autonomy only after identity, permission, review, and audit controls have been exercised.

This sequence is a risk ordering, not a universal maturity model. A read-only weekly analysis is easier to inspect and reverse than an outbound incident campaign. A knowledge proposal has a reviewable artifact. A live automation change affects future conversations, while customer communication may create an immediate and irreversible consequence. Move forward when the evidence and controls for the next class of action are ready, not merely because the previous feature launched.

Measure the completed loop, not chat activity

Prompt counts and conversation volume tell you that people opened the product. They do not tell you that customer operations improved. Build the scorecard around the operational loop:

Diagnostic quality: Whether the proposed root cause survives expert review, whether its evidence supports the conclusion, and how often factual correction is required.
Operational throughput: Time from a detected signal to a reviewed proposal and from an approved proposal to a verified change.
Artifact quality: Acceptance, revision, rejection, and rollback patterns for knowledge, automation, configuration, and communication proposals.
Customer outcome: Resolution, escalation, repeat contact, and sentiment for the affected topic after the change, interpreted alongside volume and case mix.
Safety: Permission denials, attempted out-of-scope actions, failed simulations, unauthorized writes, rollbacks, and missing audit events.
Human leverage: Expert time spent collecting evidence, recreating context, drafting the artifact, and reviewing the final proposal.

Do not make automation rate the only goal. A higher rate can coexist with poor resolutions or avoidable escalations. Treat it as one diagnostic measure and pair it with customer outcomes, correction rates, and topic-level regressions.

Create an evaluation set from real operating conditions: known content gaps, misconfigured rules, legitimate escalations, sparse attributes, conflicting evidence, localized content, and incidents with precise audience criteria. Give each case an expected outcome, required evidence, allowed tools, and forbidden action. Re-run the set when the model, retrieval system, tool, skill, permissions, or support configuration changes.

Scheduled work is where the leverage begins to compound. An operator can run recurring analysis and deliver the resulting report without waiting for a manager to remember the question. Keep an owner on every scheduled job, however. That owner should know where failures appear, when the task last completed, which data it used, and how to pause it.

Key takeaways

An operator agent improves the system around customer conversations; it is not simply another customer-facing bot.
The product boundary should cover observation, diagnosis, proposal, verification, approval, action, and monitoring.
Reliable behavior comes from grounded retrieval, attribute awareness, bounded tools, encoded domain skills, and structured review surfaces.
Grant autonomy by consequence: broad freedom to inspect approved data, tighter controls to prepare changes, and explicit approval for production writes.
Roll out recurring analysis before knowledge changes, automation configuration, and customer communication unless your own risk profile clearly supports another order.
Measure supported diagnoses, accepted artifacts, customer outcomes, human time, and safety events rather than prompt volume alone.

Your next step is to choose one recurring operational question and write down the evidence it requires, the artifact a good answer should produce, the person who will review it, and the actions the agent must not take. Once that loop works reliably, add one downstream proposal. That is a much stronger foundation for an operator agent than beginning with an open-ended prompt and a broad API key.

References

May 14, 2026

4 Costly Agent Analytics Myths—And the Data-Backed Metrics I Rely on Instead

In my work with product, operations, and support leaders, I’m often asked to help make sense of Agent Analytics—what to track, how to attribute outcomes, and where to invest. After reviewing countless dashboards and running experiments across human agents and AI agents, I’ve learned that some of the most common measurement beliefs are precisely the ones that lead teams astray.
What comes up in conversation with leaders about Agent Analytics, and why not everything is what it seems.
Below, I unpack four pervasive myths I encounter and share the data-centered practices I use to replace them. My goal is simple: help you upgrade the way you measure performance so you can improve customer outcomes, accelerate learning, and scale impact with confidence.
Myth 1: “Lower average handle time (AHT) means higher performance.” AHT is useful but incomplete. When teams optimize solely for speed, they often push complexity into repeat contacts, reopens, or escalations. In the data, that shows up as a weak or negative relationship between lower AHT and durable outcomes like first contact resolution (FCR), customer effort, or revenue per conversation.
Reality and what I measure instead: I right-size speed by pairing AHT with intent-level resolution and recontact rate. For simple intents (password reset, billing address update), shorter is usually better. For complex intents (tiered troubleshooting, multi-step verification), “right-speeding” wins—slightly longer interactions that prevent rework. Practically, that means segmenting by intent complexity using behavioral analytics, tracking weighted “intent resolution rate,” and monitoring repeat-contact windows (24–168 hours) to catch downstream pain.
Myth 2: “AI agent containment tells the whole story.” A high containment rate can mask failure modes such as unresolved intent, silent abandonment, or low-quality handoffs that frustrate customers and spike human workload later.
Reality and what I measure instead: I break containment into three parts for voice and chat flows: (1) intent resolution without escalation, (2) graceful handoff quality when escalation is necessary, and (3) post-handoff efficiency and satisfaction. For voice AI agent experiences, I also track escalation clarity (did the transcript summarize history and intent?), time-to-human, and customer satisfaction on the combined interaction. This provides a fuller view of customer support ai strategy effectiveness and avoids over-crediting automation for partial wins.
Myth 3: “Quality is subjective, so it can’t be measured at scale.” Teams often default to sporadic QA because they assume it can’t be standardized across channels or agent types. The result is noisy feedback loops and stalled coaching.
Reality and what I measure instead: Quality becomes measurable when it’s grounded in observable behaviors linked to outcomes. I use a rubric anchored in behavioral analytics (e.g., verified customer need, correct resolution path, policy compliance, empathy markers) and validate it via correlation with FCR, recontact, and retention analysis. To scale, I combine calibrated human reviews with AI-assisted scoring, check inter-rater reliability weekly, and use driver trees to connect quality levers to business results. This creates a consistent, coachable signal for both human agents and AI flows.
Myth 4: “If the dashboard is green after launch, we’ve won.” Early wins can reflect novelty effects, cherry-picked routing, or short-term incentives that don’t persist. Declaring victory too soon locks in fragile gains and hides regressions across cohorts.
Reality and what I measure instead: I treat go-live as the start of learning. I use A/B testing with a clear minimum detectable effect (MDE), stagger ramps, and hold out stable control cohorts for at least one full demand cycle. I track outcomes vs output OKRs—focusing on intent resolution, customer effort, and revenue/customer health over vanity metrics. I also monitor seasonality and channel mix shifts inside a unified analytics platform to ensure improvements generalize beyond the first week.
How I operationalize this day to day: (1) define intents and complexity upfront, (2) unify journey data across channels, (3) instrument resolution and recontact rigorously, (4) apply driver trees to isolate what actually moves outcomes, and (5) iterate via disciplined experiments rather than sweeping changes. This approach aligns product and operations, speeds up coaching, and ensures AI investments compound rather than decay.
If you’re rethinking your Agent Analytics stack, start by replacing each myth with a sharper metric: pair AHT with intent-level resolution, pair containment with handoff quality and satisfaction, pair QA with outcome-linked rubrics, and pair green dashboards with robust experiments. The payoff is a measurement system that earns trust, guides better decisions, and consistently improves customer and business results.

Inspired by this post on Pendo – Best Practices.

May 7, 2026

How to Evaluate a Shopify-Native AI Shopping Agent

You’ve probably been asked a deceptively simple question: should your Shopify store add an AI shopping agent? The hard part isn’t installing another chat widget. It is deciding whether the agent can help an uncertain shopper choose correctly without recommending unavailable products, misreading policy, or making a costly order change.

Treat this as a commerce-system decision, not a chatbot decision. A useful agent must connect conversation, live product data, cart behavior, checkout, and post-purchase service. The evaluation framework below will help you separate a persuasive demo from a system you can trust with customers and revenue.

The native test: can the agent read, reason, and act?

“Shopify-native” should describe an architecture, not a distribution channel. Being listed in an app marketplace or embedded in a storefront does not make an agent native. The meaningful test is whether Shopify remains the operational source of truth while the agent uses its data and APIs in the customer’s current context.

A concrete implementation shows how high that bar can be: a Shopify connection can expose products, variants, content, live inventory, order data, policies, and transactional APIs to the same customer-facing agent. That combination matters because a correct product description is still a bad answer if the relevant size is unavailable, and an accurate return policy is insufficient if the customer must start over somewhere else to use it.

I would evaluate an agent at four capability levels. A product that stops at the first level may still be useful, but it should not be presented internally as an autonomous commerce agent.

Capability	What the agent needs	What you should ask it to prove
Answer	Product content, store policies, and current catalog facts	Answer a precise product or policy question and identify the relevant item, variant, or rule
Recommend	Catalog relationships, inventory, conversation context, and the shopper’s constraints	Turn an ambiguous request into a short, reasoned shortlist instead of returning generic search results
Transact	Cart or order APIs, authentication, permissions, and confirmation controls	Update a test cart or prepare an order change while showing exactly what will happen before execution
Recover	Shared state across shopping, service, and human escalation	Resolve a support interruption and resume the customer’s original shopping task without asking for the same context again

Freshness is part of correctness. During a test, change the availability of a variant in Shopify and repeat the same shopping request. The agent should stop recommending that variant once the change is reflected in the connected system. Run a similar test with a policy update. A polished answer based on yesterday’s state is not a small quality defect; it can create a promise your operations team must later unwind.

Actions deserve an even stricter test. Ask the vendor to demonstrate the complete chain: customer identification, authorization, interpretation of the request, action preview, explicit confirmation, API execution, and a visible result. If any step is simulated, ask which one. A fast setup can reduce implementation effort, but it does not prove that the agent is accurate, observable, or safe.

Design the shopping dialogue around decisions, not keywords

Traditional ecommerce search works best when the customer already knows the product vocabulary. A shopping agent earns its place when the request is incomplete: a gift for a partner, a mattress for a particular sleep preference, or shoes that must work across road and trail conditions. The agent’s first job is not to produce an answer. It is to discover which answer would be useful.

A strong product-discovery dialogue follows a repeatable decision sequence:

Restate the customer’s job in plain language so a misunderstanding becomes visible early.
Identify hard constraints first, such as an in-stock variant, required use case, compatibility, budget boundary, or delivery requirement.
Ask only for information that could change the recommendation. A question that does not alter ranking or eligibility is conversational overhead.
Present a small shortlist and tie each option to the constraints the customer supplied.
Explain the meaningful tradeoff between the options instead of declaring a universal winner.
Offer the next useful action: compare details, select a variant, update the cart, or continue narrowing the choice.

This sequence turns conversation design into a product requirement. For every recommended item, your evaluation record should capture the customer’s stated need, the product facts used, the reason the item fits, the tradeoff disclosed, current variant availability, and the next question or action. That record gives your team something concrete to inspect when a recommendation is challenged.

Test whether the reasoning is responsive rather than decorative. Change one important answer while holding the rest of the conversation constant. If the customer switches from occasional use to daily use, removes a budget constraint, or requires an available color, the shortlist should change when that information is material. If the products remain identical regardless of the customer’s answers, the experience is probably search with conversational packaging.

The agent should also know when not to narrow further. Once the customer has enough information to choose, another question adds friction. Conversely, confidence should not be manufactured when catalog data cannot resolve the request. A safe response identifies the missing fact, asks for clarification, or hands the conversation to a person with the constraints already summarized.

Product cards can accelerate the final step, but the interface should preserve the reasoning that produced them. An image, name, and price answer “what is this?” The conversation must also answer “why does this fit me?” That is the difference between displaying inventory and assisting a decision.

Make shopping and support one customer state machine

A shopper does not experience your sales and support departments as separate funnels. The same person can compare products, ask about shipping, check an existing order, correct a variant, and return to buying in one session. Routing each intent to a separate tool forces the customer to reconstruct context at every boundary.

Model the journey as one state machine: discover, decide, transact, service, and resume. The agent can move between those states, but it should retain the customer’s goal, constraints, products considered, cart state, relevant order, completed actions, and unresolved question. That shared state is more important than whether the organization labels the current message “sales” or “support.”

This is where a connected agent can do more than answer FAQs. Current Shopify-oriented implementations can handle tracking, returns, exchanges, refunds, order changes, shipping questions, and subscription updates through connected procedures and APIs. Each additional action increases usefulness, but it also increases the consequence of a misunderstanding.

Use different controls for different action classes:

Read-only actions, such as showing order status, should still require appropriate customer identification but do not change commercial state.
Reversible shopping actions, such as adding or removing a cart item, should be immediately visible and easy for the customer to undo.
Financially consequential actions, including refunds, paid order changes, and subscription updates, should require authentication, an exact action summary, explicit confirmation, and a durable result or receipt.
Ambiguous or unsupported actions should stop safely and transfer to a person. The agent must not treat conversational enthusiasm, silence, or an inferred preference as consent.

That last distinction protects both the customer and the business. A mistaken recommendation can usually be reconsidered. An executed refund or subscription change creates financial and operational consequences. If the system cannot preview and verify the exact action, keep that workflow read-only and let a trained person execute it.

The transition back to shopping also needs deliberate design. After resolving a delivery problem or order correction, the agent should restore the prior context and offer a relevant path forward. It should not force an upsell into every service interaction. The next best action after a serious order problem is often confirmation that the problem is resolved. Commercial momentum comes from reducing friction, not from ignoring the customer’s immediate priority.

When escalation is necessary, pass a structured handoff rather than a transcript dump. Include the detected intent, verified identity state, constraints already collected, products or orders involved, actions attempted, results returned by Shopify, and the unresolved decision. A human agent should be able to continue with the next question, not repeat the first one.

Measure incremental commerce value and operational risk

Chat conversion is an attractive metric and an easy one to misread. People who open a shopping conversation may already have higher intent than people who do not. Comparing those groups directly can credit the agent for demand it did not create.

Ninja Transfers reported that 10% of its conversations converted to orders with values 20% above the store’s average order value. That is a useful customer result, but it is a vendor-supplied case from one merchant, not a universal benchmark or proof that the agent caused the full difference. Your business case should depend on your own incremental test.

Where traffic permits, randomize eligible storefront sessions between an agent experience and the existing experience. Measure the result across all eligible sessions, not only the visitors who choose to chat. That intention-to-treat view reduces self-selection bias and answers the executive question: what changed when the store made the agent available?

Use a balanced scorecard rather than a single conversion target:

Business outcomes: completed orders per eligible session, revenue per eligible session, average order value, checkout completion, and assisted revenue.
Decision outcomes: recommendation engagement, product-detail visits after a recommendation, add-to-cart actions, and successful comparison flows.
Downstream quality: cancellations, returns, exchanges, and contacts caused by a poor recommendation or incorrect expectation.
Service outcomes: successful action completion, repeat contact for the same problem, human escalation, and time to a confirmed resolution.
Agent quality: use of current catalog facts, in-stock recommendation rate, policy accuracy, clarification behavior, safe refusal, and correct tool execution.
Risk outcomes: unauthorized or incorrect actions, failed confirmations, customer complaints, policy exceptions, and cases requiring operational repair.

Conversion and average order value belong beside returns and cancellations. An agent can raise the initial basket by recommending a more expensive option while reducing customer fit. Without the downstream view, the dashboard rewards the sale and hides the repair.

Your event model should make the journey reconstructable. Useful events include agent opened, intent classified, clarification answered, recommendation shown, product selected, cart changed, checkout started, order completed, support action requested, confirmation received, action completed, and human handoff. Join these events through an appropriately governed session or order identifier so the team can inspect both funnel movement and individual failure paths.

Build the evaluation before the vendor demo

Create scenarios from your real catalog shape, policies, and failure modes. For each scenario, write the expected outcome and the failures that would make deployment unacceptable. Include:

An ambiguous shopping request that requires clarification.
Two products that appear similar but differ on a constraint customers care about.
An unavailable variant that would otherwise be the best match.
A question whose answer depends on store policy rather than product copy.
A conversation that moves from shopping to order support and back again.
A request that sounds actionable but lacks required authentication or confirmation.
An unsupported request that should trigger a safe handoff.
A catalog or policy change made after the agent’s initial synchronization.

Run the same scenarios repeatedly and record the underlying catalog state each time. You are testing consistency, grounding, and recovery, not literary elegance. A shorter answer that uses the correct variant and policy is more valuable than a fluent answer that improvises.

Expand autonomy in the order of consequence

A staged rollout lets evidence determine how much authority the agent receives:

Evaluate offline with approved scenarios and representative catalog data.
Launch read-only product discovery and policy answers with an obvious human fallback.
Add visible, reversible cart actions after recommendation quality is stable.
Introduce authenticated order-support workflows with previews and confirmations.
Enable financially consequential actions only after tool execution, auditability, exception handling, and operational ownership have been tested end to end.

Define the owner of each failure before launch. Product should own the intended customer behavior and success measures. Commerce or operations should own policy and workflow correctness. Engineering should own integration reliability and observability. Customer support should own escalation quality and emerging failure patterns. The exact reporting lines can vary; an unowned failure queue cannot.

Review both aggregates and conversation-level traces after release. Aggregate metrics tell you whether the experience is moving the business. Traces tell you why. A small cluster of incorrect variant recommendations or failed order actions can disappear inside a healthy conversion average while creating disproportionate customer harm.

Key takeaways

A Shopify-native agent should use live commerce data and governed APIs; storefront placement alone is not enough.
The agent’s product-discovery job is to uncover decision criteria, apply hard constraints, explain tradeoffs, and lead to the next useful action.
Shopping and support should share customer context, but the agent’s permissions must become stricter as actions become harder to reverse.
Conversation conversion is a diagnostic metric, not causal proof. Measure incremental results across eligible traffic whenever possible.
Pair conversion and average order value with returns, cancellations, incorrect actions, and operational repair costs.
Begin with read-only assistance and expand autonomy only after each workflow proves accurate, observable, recoverable, and properly owned.

Before you approve a purchase, bring a vague shopping request, a policy edge case, an unavailable variant, a mixed sales-and-support conversation, and a consequential order action into a live test store. If the agent cannot show where its answer came from, what it will change, and how it fails safely, you have found the next product requirement, not a detail to defer until launch.

References

Intercom – Fin for Ecommerce: The Shopify-native AI Agent transforming product discovery and sales

May 7, 2026

Tag: customer support ai strategy

Key takeaways

Treat the agent as a product system, not a chatbot

Move agent readiness into the product release process

Measure experience at conversation scale

Build one improvement loop across CX, product, and knowledge

References

System access changes both the value and the risk

Choose workflows where access justifies its complexity

Use an access ladder instead of a single launch

Put deterministic controls around probabilistic decisions

Key takeaways

References

Key takeaways

Start with the queue, not the model

Turn knowledge into a controlled production input

Define the job as a closed loop, not a chat box

Build reliability below the model and price that work honestly

Evaluate build versus buy beyond the demonstration

Put a proposal boundary around every material action

Roll out workflows in increasing order of consequence

Measure the completed loop, not chat activity

Key takeaways

References

The native test: can the agent read, reason, and act?

Design the shopping dialogue around decisions, not keywords

Make shopping and support one customer state machine

Measure incremental commerce value and operational risk

Build the evaluation before the vendor demo

Expand autonomy in the order of consequence

Key takeaways

References