Tag: Agent Analytics

Make Every Answer the Last: Building a Self-Improving AI Support Engine for 2026

Once I’ve defined the right roles on my team, the next move is to design an operating model that makes progress a habit. My goal is simple: every interaction should strengthen the system so the AI Agent keeps improving over time.

I anchor the team on a mantra that has never failed me: “The first time you answer a question should be the last.” That single statement reframes support as a compounding system rather than a one-off activity.

The ambition is to ensure every resolution makes the next one faster and more accurate, so fewer issues repeat, quality compounds, and support scales naturally. That doesn’t happen by accident—it requires intentional design.

In practice, this comes down to four essentials: clear ownership of performance, guardrails that make iteration fast and safe, feedback loops that turn learning into routine upgrades, and a culture that celebrates the work of improvement—not just the outcomes. Here’s how I put that into play.

First, I start with clear ownership. Ambiguity is one of the most common reasons AI performance plateaus. When no one truly owns how the AI Agent performs, feedback gets lost, issues linger, and improvements stall.

On high-performing teams, I assign a single owner—often an AI ops lead—responsible for making the AI Agent better. They review resolution trends to spot underperformance, make targeted updates to content, configuration, and behavior, coordinate with product and engineering on systemic blockers, and set improvement priorities, targets, and timelines. The title matters less than the mandate; what matters is clear authority to drive change across teams.

Real-world example: At Dotdigital, AI performance plateaued after a strong start—resolving around 2,800 conversations per month for three consecutive months. To drive resolution rates up, the team created a dedicated support operations specialist role, filled by an experienced agent with deep product knowledge. This person will focus on refining snippets, improving content, and enhancing the AI’s resolution capabilities.

Second, I make iteration fast and safe. As the AI Agent takes on more volume and complexity, change can start to feel risky—so teams hesitate, and performance stalls. Lightweight governance fixes that by making the path from insight to action predictable.

I keep the rules simple and explicit: which changes need review (and which don’t), who the decision-makers are, how we test updates before they go live, where feedback flows so it’s seen and acted on, and when progress gets reviewed on a steady cadence. Governance isn’t bureaucracy—it’s what keeps improvement routine and safe.

Real-world example: Anthropic ran a focused “Fin hackathon” sprint to improve their AI Agent’s resolution rate. The team audited unresolved queries, identified underperforming topics, and created or updated content to close gaps. They converted frequently used macros into AI-usable snippets, monitored Fin’s performance during live support, and continuously refined content based on real interactions. This structured approach enabled rapid improvement while maintaining quality standards.

Third, I build a system that learns by default. AI performance isn’t static, but many organizations treat it like a one-time implementation. The most successful teams operationalize learning: they analyze where the AI Agent struggles and feed those insights directly into structured improvements.

The signals are straightforward: review common handoffs to humans, track unresolved queries by topic or intent, measure resolution rate trends over time, and use those inputs to prioritize fixes and content upgrades. Whether you follow a formal loop like the Fin Flywheel framework or something lighter, the goal is the same—make improvement inevitable.

Fourth, I treat content as competitive infrastructure. Your AI Agent is only as good as what it knows. As George Dilthey, Head of Support at Clay, put it: “That’s when we realized: AI doesn’t just come up with information out of nowhere, you have to feed it. We were spending all our time evaluating tools when we should’ve been focused on content.”

I operationalize knowledge like infrastructure: every topic has a clear owner, content is structured, versioned, and ingestion-ready, new products ship with source-of-truth content by default, and changes ship on a schedule—not when someone finds time. This is the backbone that differentiates teams who scale confidently from those who stall out.

In my organization, we’ve evolved our New Product Introduction (NPI) process by aligning early with R&D on a single, canonical source of truth that becomes the foundation for all downstream content—including what the AI Agent uses to resolve queries. By embedding content creation into launch readiness, not as an afterthought, we’ve consistently hit 50%+ resolution rates on new features from day one.

Finally, I make belief visible. Even the best system will stagnate if people stop believing in it. Belief can fade quietly unless you reinforce it on purpose. I keep it strong by sharing specific wins regularly, highlighting improvements with metrics, and recognizing the people behind the gains—then giving them space to lead. This isn’t just about morale; it keeps everyone aligned on the bigger play.

When you put it all together—clear ownership, safe iteration, a learning system by default, and content as infrastructure—AI performance compounds. As the AI Agent gets better, the entire support model becomes faster, more reliable, and truly scalable. That’s the foundation of a modern, AI-first support organization.

Next, I’ll take this a level deeper and share how capacity planning changes when AI handles the majority of inbound volume and your team shifts into higher-value roles. If scaling with confidence is the goal, this is where the operating model pays off.

Inspired by this post on The Intercom Blog.

December 9, 2025
How I’m Rebuilding Customer Service for 2026: An AI‑First Playbook for Real Impact

Like many support leaders right now, I’m deep in 2026 planning. The more I map scenarios and stress-test assumptions, the clearer one thing becomes: the way work gets done has fundamentally changed, and that change must reshape our customer service organization.

In 2026, you won’t get the full value of AI by keeping your org chart, systems, and operating model the same. You need to think differently about how support is structured, how performance is owned, and how your systems evolve around an AI-first model. That’s the lens I’m using across my team and our cross-functional partners.

To help you do the same, I’m launching a 2026 customer service planning series. Over the next five weeks, I’ll share how I’m approaching roles, skills, organizational design, and an operating model that makes AI the backbone of support—not a bolt-on feature.

We’ll publish each edition here and on LinkedIn. If you’d rather get them by email as soon as they go live, drop your details and I’ll send each edition straight to your inbox.

But before you can make any of those decisions, you need the right mindset and the right internal conditions for change. That’s where I’m starting this week.

Week 1: Start with a mindset shift

If you were building support from scratch today, you’d design around AI from day one. That’s the mindset to carry into 2026—and it’s the mindset I’m using to guide investment and accountability.

Too many teams still treat AI like a feature instead of infrastructure. They tack it onto existing processes, limit scope to tier-one issues, and never evolve the organization or systems around it. I’ve seen that approach stall progress and fragment the customer experience.

Those teams are thinking too small. They chase incremental efficiency, underinvest in the system change required to make AI successful, and get stuck. The result: a reactive team, a choppy customer experience, and value left on the table.

AI Agents are fully capable, end-to-end resolution engines. They fundamentally change the architecture of support.

To plan effectively and get the most value out of the technology, you need to adjust your mental model. Here are the mindset changes I’m prioritizing.

1) Move from ‘AI as a tool’ to ‘AI as infrastructure’

For the past decade, support systems have been the intermediary between customers and human support agents. AI isn’t an intermediary, it’s the first touchpoint (and often the last), the primary resolver, it manages workflows, orchestrates handoffs, and takes real actions.

Planning with the “AI is a tool” mindset leads to small optimizations that don’t move the needle. Planning with the “AI is infrastructure” mindset lets you redesign around the real sources of value creation.

Here’s what I’m designing around in 2026:

• Clear ownership of Agent performance

• A feedback loop that never shuts off

• A shared understanding of when humans step in

• Systems that evolve as AI capabilities expand

This framing sets up every decision that comes later in your planning process.

2) Look at how the work is changing

You need to plan your 2026 support organization around what the distribution of work will be—not what it is today. AI has shifted where volume goes, what humans spend time on, where judgment is needed, how performance is measured, and how the customer experience is designed.

If your planning assumes the current distribution is stable, you’ll design the wrong structure. I’m modeling for the work that’s coming, not just the work on our queue today.

3) Think like a product leader

When customers primarily interact with your AI Agent, support becomes responsible for designing the customer experience—not just managing it.

“Support is becoming a product function, and you are becoming a product leader”

Design your 2026 support org for AI from day one. This Gamma testimonial shows how an AI agent (Fin) resolves 80%+ of inbound requests, letting a small team scale customer service efficiently without increasing headcount.

Support is now a product surface, and support teams act like AI product teams. They:

• Design the customer experience

• Create and curate the knowledge layer that drives AI quality

• Maintain continuous improvement loops and tune system behavior over time

This is a big shift. Your planning—hiring, skills, rituals, and metrics—needs to reflect that evolution.

4) Redefine performance

This is a big mental leap for support leaders. Traditional performance was measured on speed and satisfaction, but AI performance is measured on resolution, impact, and system reliability.

Planning for 2026 means assuming that:

• Humans will handle a smaller % of volume.

• Customer experience will be shaped by AI’s performance, not throughput

• “Support productivity” gets measured differently

When AI handles the bulk of your support volume, you need new metrics for how your team creates value. In practice, that means instrumenting AI and human-in-the-loop workflows with the same rigor you’d apply to a customer-facing product.

5) Understand that your value increases as AI takes on more work

You need to re-orient your team around AI’s performance to get the most value out of it. The more complex work you give it, the higher impact it will have.

Instead of routing complex, messy questions straight to your human team, shift their focus to improving the AI system so it can take on more over time.

Automating low-effort questions reduces noise, but automating complex workflows changes the economics of your entire team. It creates asymmetric returns that compound as AI absorbs the work that once demanded the most time and skill.

6) Plan for adaptability

A big difference between traditional planning and 2026 planning is simple: change will be constant.

“Change is hard, but the teams that adapt will be the ones who get the most out of this opportunity”

AI learns, evolves, and improves continuously. I’m asking, “How do we build an organization designed to adapt fast as the system evolves?” That question is informing everything from team topology to knowledge governance and experimentation cadence.

Food for thought

Heading into 2026, your org chart will look different—and that’s a good thing. Your people will play new, more meaningful roles as designers, curators, and stewards of an AI-first customer experience.

Once you accept that 2026 demands a different way of thinking, working, and planning, you can move to the next stage: designing the support organization that fits this future. I’ll share exactly what that looks like next week, including roles, skills, and ownership models that have worked well in my experience.

Want the full series delivered by email? Drop your details and I’ll send each edition to your inbox as soon as it’s published.

Inspired by this post on The Intercom Blog.

November 18, 2025
Win AI Search: Proven Playbook to Get Your Startup Recommended by ChatGPT & Perplexity

AI search is quickly becoming the new homepage for startups. When a buyer asks a model for the best tools, they often take the short list at face value. I treat this moment as a product surface I can influence with strategy, content, structure, and distribution—much like any other go-to-market channel.

Early on, I set a simple objective for my team and me: "Learn how LLMs like ChatGPT and Perplexity decide which startups to recommend and what signals help a brand get discovered in AI search." That sentence became our north star for experiments, instrumentation, and content architecture.

Here is the mental model that consistently holds up in practice. Large language models synthesize answers from a knowledge graph built from crawled content, citations, and high-signal sources. They weight consensus, clarity, recency, authority, and machine-readability. I don’t pretend to know the internals, but across hundreds of tests, the same patterns correlate with being surfaced and cited.

First, I make our entity unambiguous. I standardize the company name, product names, and leadership bios across the site and external profiles. I implement Organization and Product markup with schema.org and link out with sameAs to authoritative profiles like LinkedIn, Crunchbase, GitHub, and key directory listings. The goal is to collapse ambiguity so AI search knows exactly who we are and which claims are attributable to us.

Next, I publish definitive, answer-first pages. For every core query—what we do, who it’s for, outcomes, differentiators, pricing, comparisons, and integrations—I ship a page that leads with a crisp summary, then supports it with evidence, examples, and plain language. I include Q&A sections, realistic use cases, and named case studies so models can quote and ground responses in verifiable facts.

I then make the site maximally machine-readable. I add schema.org for SoftwareApplication, Product, FAQPage, and HowTo where relevant. I keep titles, H1/H2 structure, internal links, and metadata descriptive and consistent. I expose last-modified dates, maintain an XML sitemap, and keep a visible changelog and release notes. Freshness matters—Perplexity, in particular, tends to privilege recent, well-cited material when answering time-sensitive questions.

Citations are non-negotiable. I earn credible mentions on third-party properties, analyst lists, comparison pages, and customer reviews. I prioritize authoritative placements over volume, then make sure our site references those sources to reinforce the signal. When Perplexity cites our page alongside a respected third-party review, our inclusion rate in answers rises noticeably.

I also design for developers, buyers, and machines at once. That means clean docs, integration pages, and transparent security and trust content. Clear API references, integration guides, and reliability notes give models concrete artifacts to summarize. Pricing, privacy, and support policies reduce uncertainty and increase the likelihood that an answer will include us.

Measurement turns this from a hunch into a system. I run controlled content experiments, track minimum detectable effect on discovery and mentions, and instrument referral patterns from AI assistants when citations appear. I monitor which prompts surface our brand, which sources are cited, and which pages are repeatedly used as references. When we move a KPI, we codify the pattern into our playbook and scale it.

Trust is the compounding advantage. I maintain a transparent trust center, privacy-by-design posture, and clear data governance practices. I remove vague claims, back up benefits with evidence, and keep all performance or security statements auditable. Models tend to lift brands that feel low-risk, well-documented, and widely corroborated.

If you want a fast start, here’s the checklist I rely on. Standardize your entity and ship schema.org. Publish answer-first pages for core jobs-to-be-done, comparisons, and integrations. Earn authoritative third-party citations and reference them. Keep release notes, changelogs, and dates current. Instrument AI discovery and iterate based on what gets cited. Do this consistently, and your startup earns a fair shot at being recommended when buyers ask AI for the best options.

Inspired by this post on Amplitude – Best Practices.

November 7, 2025
The Product Playbook: Measuring Agent Performance with Pendo and Agent Analytics to Drive ROI

I treat agent performance analytics as a strategic product lever, not a back-office metric. When I combine Pendo’s product signals with Agent Analytics from our support systems, I get a unified view of where users struggle, how agents intervene, and which in-app experiences accelerate resolution. That visibility lets my team drive product-led growth and improve customer experience while lowering support costs.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practice, I build a clear scorecard that blends both product and support KPIs: first response time, resolution rate, first contact resolution, CSAT, containment/deflection rate, average handle time, ticket volume per active account, onboarding completion, user activation, and time-to-value. This balanced view ensures we reward not just speed, but durable outcomes that reduce repeat contacts and improve retention.

To make the data actionable, we connect our CRM integration, ticketing events, and Pendo product analytics in a unified analytics platform. That gives me cohort-level clarity—who needed help, what they were doing before opening a ticket, how agents responded, and whether users stayed engaged afterward. With clean instrumentation and consistent taxonomies, Agent Analytics becomes a reliable operating system for both product and support leadership.

I then use in-app guides, tooltips, and product tours to proactively address the top friction points that drive ticket volume. Through A/B testing, we compare cohorts exposed to guided workflows versus control groups, measuring deflection, faster task completion, and downstream conversion. When a guide meaningfully reduces tickets for a given workflow, we promote it from experiment to standard onboarding, and we feed those learnings back into our roadmap.

The real unlock comes from tying outcomes to business impact. I track how improvements in resolution quality and self-serve adoption influence expansion revenue, support cost per account, and risk signals like churn propensity. Retention analysis helps us validate whether reduced friction and better agent coaching translate into sustained engagement and healthier accounts.

Operationally, Agent Analytics helps me coach teams with precision. I spotlight high-performing behaviors, identify knowledge gaps, and standardize winning playbooks directly in the product via in-app guidance. This approach empowers agents, shortens onboarding for new hires, and keeps our best practices current as the product evolves.

None of this works without trust. We apply privacy-by-design principles and strong data governance, ensuring that analytics, coaching, and automation respect user consent and data minimization standards. With that foundation, we can scale confidently—experiment faster, learn from every interaction, and continuously improve the software experience.

If you’re getting started, begin by baselining your agent and product KPIs, ship one high-impact guide to deflect a top ticket driver, and review results weekly. Within a quarter, you’ll have a repeatable loop: diagnose friction, test an in-app solution, measure deflection and satisfaction, and reinvest the gains into the next set of improvements.

Inspired by this post on Pendo – Best Practices.

November 4, 2025
3 Hidden Hurdles Blocking Effective AI Agents—and How I Turn Them into Business Wins

AI agents promise leverage at scale, yet too many proofs of concept stall before they create measurable value. Over the past several launches, I’ve seen the same patterns repeat across IT and operations. The mandate is clear: “Discover three key challenges IT and ops teams face when building and managing AI agents that drive real business wins.” Here’s how I frame the work, where teams get stuck, and the playbook I use to move from demo to durable outcomes.

Hurdle 1: fragmented data and weak data governance. Agentic AI is only as strong as the data it can reliably access. In most organizations, knowledge is scattered across CRMs, ticketing tools, wikis, and data lakes—each with different schemas, permissions, and freshness guarantees. Without privacy-by-design and consistent access patterns, agents hallucinate, miss context, or violate policies. This isn’t a model problem—it’s an information architecture problem.

My approach starts with an integration-first mindset: anchor the agent to authoritative systems via CRM integration, unify retrieval across knowledge sources, and enforce role-based access at query time. I pair this with data contracts, lineage, and content freshness SLAs so the agent never acts on stale or restricted information. A unified analytics platform and strong data governance let me monitor coverage, drift, and security posture as the knowledge footprint grows.

Hurdle 2: reliability, observability, and AI risk management. Even well-fed agents can behave unpredictably without tight control loops. Teams often lack Agent Analytics, standardized evals, and guardrails to catch prompt injection, tool abuse, or subtle regressions. The result is fragile behavior that erodes trust with IT, security, and front-line operators.

I build a reliability stack that looks a lot like SRE for agentic AI: scenario-based evaluations before release, production tracing of every step and tool call, red-teaming for threat detection and response, and policy enforcement at runtime. Hallucination mitigation, input validation, and fallbacks (including human-in-the-loop) are non-negotiable. We track latency, cost, accuracy, and safety incidents in one Agent Analytics view so we can ship confidently and iterate quickly.

Hurdle 3: workflow integration and organizational adoption. The best agent can still fail if it can’t take action in real systems or if change management is an afterthought. Agents must fit the way people actually work—permission models, SLAs, audit trails, and existing approval paths—instead of creating shadow processes that confuse teams.

I integrate agents directly into systems of record and daily tools—ticketing, CRM, knowledge bases—so outcomes are auditable and reversible. I define clear RACI, rollout guardrails, and metrics in product roadmapping and sprint planning (e.g., first-contact resolution, time-to-resolution, deflection, cost per task). We ship narrowly scoped capabilities first, pair them with in-app guides and product tours, and expand privileges as confidence and KPIs improve. This is product management leadership, not just prompt engineering.

In practice, the pattern is consistent. For customer support, we anchored the agent to the CRM, knowledge base, and incident runbooks with strict access controls, then layered policy checks for regulated data. With unified analytics, we measured precision/recall of suggested actions, tracked cost and latency, and flagged risky prompts. The result: higher accuracy, cleaner handoffs, and faster time-to-value without sacrificing compliance.

If your agents aren’t delivering, start here: fix the data plane, instrument the control plane, and design for real workflows. Do this well and you’ll move beyond flashy demos to durable productivity gains and competitive differentiation—while keeping security, governance, and stakeholders on your side.

Inspired by this post on Pendo – Perspectives.

October 25, 2025
4 Costly Misconceptions About AI Agents—and What Product Leaders Must Do Instead

Building AI agents looks deceptively simple right now. After leading multiple agentic AI initiatives, I’ve learned that the difference between a demo and a dependable product comes down to disciplined product discovery, ruthless scoping, and a clear AI Strategy that aligns with business outcomes. Here are four common misconceptions I correct early with stakeholders—and the practices I use to avoid expensive detours.

Misconception 1: “An LLM plus a few prompts is a production-ready agent.” In reality, production-grade agents require orchestration and rigor: tool-use and retrieval, memory design, state management, deterministic fallbacks, and continuous evaluation. I instrument Agent Analytics from day one to trace tool calls, latency, error codes, and cost per task; then I use A/B testing with a clear minimum detectable effect (MDE) to validate improvements before broad rollout. This is where product roadmapping and sprint planning matter—sequencing capabilities so we avoid building speculative features that don’t move outcomes.

Misconception 2: “More autonomy is always better.” The right autonomy level is contextual and risk-adjusted. For high-stakes workflows, I design for human-in-the-loop and role-based guardrails, grounded in privacy-by-design and data governance. Policies like least-privilege access, audit logs, and reversible actions reduce operational risk while still delivering leverage. In practice, this hybrid approach also controls cost: narrower scopes, clearer prompts, and bounded tool access reduce hallucination surface area and improve reliability—key to AI risk management.

Misconception 3: “If we build it, users will adopt it.” Adoption is earned with thoughtful onboarding and in-app guidance, not promised by a feature launch. I pair agent launches with targeted product tours, contextual tooltips, and progressive disclosure to drive user activation and product-led growth. Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement. Whether you use Pendo or a comparable solution, the principle stands: instrument the experience, run experiments, and iterate quickly based on evidence, not intuition.

Misconception 4: “Security, compliance, and governance can wait.” Deferring controls is a false economy. I embed AI risk management from day zero: prompt injection defenses, PII redaction, DLP, grounding and citation strategies, and threat detection and response. Clear data retention policies, vendor diligence, and model evaluation standards keep leadership, security, and legal aligned. This is the crux of building trust—and it’s far easier to design up front than to retrofit under pressure.

How I execute in practice: start with a tightly framed use case tied to a measurable outcome; define outcomes vs output OKRs; build a slim vertical slice to validate feasibility; instrument Agent Analytics from the first commit; ship behind feature flags; and operationalize learning loops across support, success, and GTM. The result is a durable path to product-market fit for agentic AI—one that compounds learning while minimizing blast radius.

The leaders who win with AI agents won’t be the ones who move fastest in a demo. They’ll be the ones who manage risk transparently, learn in public with their users, and turn continuous insight into competitive differentiation. If you’re planning your next agent milestone, align the roadmap to outcomes, treat governance as a feature, and make adoption your North Star.

Inspired by this post on Pendo – Best Practices.

October 25, 2025
WTF is MCP? The powerful protocol giving enterprise AI agents real-world autonomy

I get asked this constantly by boards, CIOs, and product teams: WTF is MCP, and why does it matter for enterprise AI? Here’s my straightforward take from the trenches of rolling out agentic AI across complex, regulated environments—and why it changes how we design, govern, and scale autonomous capabilities.

“Model Control Protocol gives your AI agents arms and legs to go do stuff with your data.” That framing resonates because it’s both simple and accurate. MCP turns passive “chatbots” into active agents that can safely take action within defined guardrails.

In practice, MCP is the connective tissue between models and the tools, systems, and workflows we trust. It standardizes how agents request permissions, execute tasks, and report outcomes—so enterprises can move from demos to durable operations. The benefit isn’t just autonomy; it’s autonomy with accountability, aligned to our AI Strategy and data governance obligations.

When I pilot agentic AI in production, I start with a narrow scope: which systems the agent touches (for example, CRM integration via HubSpot), what actions it can take (read, write, or propose), and what evidence it must log (inputs, outputs, and approvals). That discipline keeps us compliant with privacy-by-design while unlocking real business impact.

Great MCP use cases emerge where read-write actions compress time-to-value. Think: pulling Amplitude analytics cohorts to personalize outreach, auto-generating Pendo in-app guides based on feature adoption, or triggering customer support workflows with predefined playbooks. Each action is observable, reversible, and measured—because in the enterprise, repeatability beats novelty.

From a product management leadership perspective, I treat MCP-enabled agents like any other product surface. We define clear outcomes, not outputs: success rate per task, mean time to resolution, quality score, and safety incidents. We validate uplift with A/B testing and a minimum detectable effect (MDE) before scaling. Then we feed results into an Agent Analytics dashboard, just as we would for product-led growth funnels.

Governance is where MCP earns trust. I enforce least privilege, time-boxed credentials, environment isolation, and tamper-evident audit logs. Every tool call is tied to a business purpose, owner, and SLA. We integrate with existing threat detection and response processes so cybersecurity teams see the same telemetry they’re used to—no shadow AI, no surprises.

There’s also an adoption playbook that works: start with a contained domain, ship a sandboxed agent, require human-in-the-loop approvals, then progressively relax controls as accuracy and alignment improve. Document the boundaries in plain language, and instrument everything from day one. This is how we de-risk AI risk management while accelerating impact.

The most exciting shift is cultural: teams move from asking “Can the model do this?” to “What outcomes should the agent own—and what guardrails make that safe?” That mindset unlocks empowered product teams, clearer ownership, and faster iteration. MCP is simply the operational backbone that lets those choices stick.

If you’re evaluating where to start, pick one workflow with high frequency, clear rules, and measurable outcomes. Wire it to MCP with tight scopes, ship it to a friendly cohort, and learn aggressively. Autonomy isn’t the end goal—reliable, governed value is. MCP just makes that scalable.

Inspired by this post on Pendo – Best Practices.

October 25, 2025
6 Hard Questions Your AI Agents Must Answer to Win: Performance, Risk, and Real ROI

“Do you know how your AI agents are performing?” I ask this question in every review because it exposes whether we’re managing by outcomes or by anecdotes. Too often, teams point to latency, token counts, or completion rates and call it a day—useful signals, but not the story.

In my role, shipping agentic AI into production means I need decision-quality evidence, not vibes. That starts with Agent Analytics built on a unified analytics platform and instrumentation that lets me trace behavior, quantify value, and manage risk. Below are the six questions I use to separate novelty from durable impact.

1) What outcome are we optimizing for—and how do we measure it? If we can’t map the agent’s work to outcomes vs output OKRs, we’re optimizing noise. I anchor on task success rate, time-to-resolution, containment rate (no human handoff), cost per successful outcome, and downstream business impact (retention, conversion, NPS/CSAT) to keep us honest.

2) Are the right guardrails in place for AI risk management and data governance? I expect documented policies for prompt injection defenses, PII redaction, access control, and auditability. Every tool call should be permissioned, every data boundary explicit, and every failure mode observable. If we can’t demonstrate compliance by design, we’re scaling risk instead of value.

3) Can I explain every decision the agent made? Agentic AI needs traceability: prompts, intermediate reasoning, tool calls, retrieved context, and final outputs. I route key events into Amplitude analytics so product, engineering, and risk can slice behavior end to end. If we can’t reconstruct the path to an answer, we can’t debug, improve, or trust it.

4) What is the true cost per successful outcome? Raw token spend is misleading. I model total cost of ownership across retries, tool usage, escalations, and human review time—then benchmark against a consumption SaaS pricing lens. If cost per resolution trends up as volume grows, we haven’t built a scalable system; we’ve built a demo.

5) How does the agent learn without breaking what already works? My bar is a disciplined experimentation loop: offline evals, online A/B testing with clear guardrails, and a rollback plan. We predefine a minimum threshold for improvement before rollout and track regressions by persona, task type, and channel so we can localize fixes quickly.

6) Where is this agent creating durable differentiation? I look for capabilities competitors can’t easily copy: unique data advantages, superior tool orchestration, or workflows that compound learning. If the edge is just a base model prompt, the moat will evaporate; if it’s embedded in product workflows and proprietary signals, we’re building advantage.

Answering these six questions turns agentic AI from a novelty into a managed system. With Agent Analytics feeding a unified analytics platform, we can tie behavior to business outcomes, enforce governance, and make portfolio trade-offs grounded in evidence. The result is a product management leadership motion that prioritizes real ROI over vanity metrics—and scales with confidence.

If you’re not satisfied with the answers today, start by instrumenting the journey end to end, aligning metrics to OKRs, and setting clear risk thresholds. The compounding effects show up quickly when every iteration is measurable, explainable, and accountable.

Inspired by this post on Pendo – Best Practices.

October 24, 2025
The Real Reason Pendo Built Agent Analytics—and How It Drives Adoption, Revenue, and Trust

I’ve learned the hard way that the toughest part of launching in-app agents and guided experiences isn’t the build—it’s proving, quickly and credibly, that they move the business. If I can’t quantify adoption, engagement, deflection, and time-to-value, stakeholder confidence erodes and iteration slows. That’s exactly why an Agent Analytics capability matters: it turns opaque interactions into measurable outcomes that product, customer success, and engineering can all act on.

When I evaluate a capability like Agent Analytics, I anchor on a few questions. Which segments adopt the agent, and where does engagement drop? What fraction of issues are successfully deflected versus escalated? Which prompts, product tours, and in-app guides drive conversion and retention—and which add friction? How does agent usage correlate with onboarding completion, core feature activation, and long-term retention analysis? If I can answer those with a unified analytics platform, I can prioritize confidently.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practice, I map an outcomes-first measurement plan: define a north-star (e.g., activated accounts), articulate contributing metrics (guide completion rate, agent task success, session depth), then run targeted A/B testing on copy, timing, and placements. With the right analytics, I can compare cohorts exposed to in-app guides and product tours against a control, validate impact, and double down on the patterns that consistently improve adoption and stickiness.

Cost and risk are just as important as growth. An effective Agent Analytics view helps me model support deflection, time-to-resolution, and escalation rates so I can quantify cost savings without sacrificing quality. On the risk side, I look for early-warning signals—low-confidence responses, repeated handoffs, or anomalous usage—so I can intervene before they turn into churn or brand concerns. The point isn’t vanity metrics; it’s operational clarity that enables responsible, scalable product-led growth.

This also changes team dynamics. Product trios get a shared source of truth for decisions, engineering gains sharper specs informed by real behavior, and customer-facing teams can see which experiences reliably unlock value for each segment. Instead of debating opinions, we iterate on evidence—tightening the loop between product roadmapping and sprint planning, UX writing, and go-to-market strategy.

My 90-day playbook looks like this: establish a baseline for adoption and engagement; instrument agent interactions end to end; ship two or three small, high-leverage experiments in onboarding and help experiences; and review results in weekly rituals. By day 90, I expect to see a clear line from agent engagement to activation and retention, along with a repeatable testing cadence that compounds learning.

I’ve seen the same pattern across products and markets: once teams illuminate the black box of in-app assistance with rigorous, actionable analytics, customer confidence rises, onboarding accelerates, and roadmaps get sharper. If you’re evaluating Pendo or already running it, put Agent Analytics at the center of your measurement strategy—and let your data, not assumptions, guide the next iteration.

Inspired by this post on Pendo – Perspectives.

October 24, 2025