How do you lead a support team in this new world with AI metrics? That question has been front and center for me as we integrate AI-first customer service tools into our daily operations.
The technology is amazing, but our assumptions and processes for understanding and leveraging AI metrics are very different from traditional support metrics. Our new CX Score is the perfect example.
Two months ago, we launched CX Score – a new way to analyze every single conversation and give you a complete view of your support experience. I was genuinely excited as someone who’s battled with CSAT survey mechanics, teammate exclusion processes for CSAT, and the nagging truth that this is only a small portion of our volume. As we’ve navigated CX Score, we’ve learned lessons that apply broadly to AI metrics. Two takeaways stand out for me.
First, lots of data calls for new processes. One of the first things we noticed was the sheer amount of data. For better or worse, CSAT was a small enough sample size to review every comment – particularly unhappy ones. Our QA team would read and categorize each response, and follow up with customers. Managers would read most comments for their team (~15 in total per manager), and discuss in 1:1s.
But what do you do with 1,600+ reviews across the org? This is the reality of AI metrics, and when you have more data than ever before, the old processes don’t scale. We briefly tried reviewing all unhappy CX ratings. We tried taking a sample, but this felt just as limited as CSAT. We exported the trends and conversation data back into an LLM for analysis, but without in-depth prompting the results were only okay.
What worked was reframing how we use the signal. Because CX Score is great for reviewing trends, we use it to measure week-over-week performance for both Fin and as a team wide KPI for human support. We also use CX Score to review specific targeted areas, like a new hire’s conversations on a certain product area. And we use it to review the customer experience for a group of customers, or to analyze a customer’s entire case history so we can lean in at the customer level.
Second, the complexities of AI mean we won’t always know the “why” – and that’s ok! People naturally want to know “the why” – especially support folks. When we started using CX Score, one of the biggest challenges was the team wanting to dig deeper into why a specific score was given. While the score provides a great overview, people wanted a detailed, step-by-step explanation. But LLMs are mostly a “black box” – especially to the everyday person. As AI becomes more and more ingrained in our work, we’ll need to accept not always knowing every detail.
This required a mindset shift for both the wider team and leadership as we moved into a world of AI metrics. We focused on the outcome vs. the process, celebrating positives and highlighting insights and actions previously impossible with only CSAT. We refused to compare to humans, reminding ourselves that many of the unknowns of AI are equally true with humans; even with a large survey, we never know for sure how customers feel. And we acknowledged emotions. Our Ops Manager William would poll the leadership team in our weekly ops meeting, asking, “In one word, how did the CX Score make you feel last week?” That simple ritual gave managers space to surface wins and challenges, and it kept us grounded.
What’s next is continuing to revamp how we work and adjusting our collective mindset for AI tools and metrics. The pros highly outweigh the cons, so I encourage you to jump in and start experimenting with AI metrics, especially where they can augment customer experience, operational cadence, and product feedback loops.
Lastly, this technology is improving very quickly. Just yesterday we added deeper AI explanations and additional attributes to explain the CX Score and aggregate summaries across topics. I’m excited to try it out!
Subscribe to The Ticket here – a bi-weekly LinkedIn newsletter delivering key insights for customer service professionals in this time of mind-blowing change.
Inspired by this post on The Intercom Blog.
Leave a Reply