Your AI support agent handled the easy questions, produced an encouraging early lift, and then stopped getting better. The same topics still reach human agents. Content fixes happen when someone remembers. The aggregate resolution rate moves, but nobody can explain why.
If that describes your operating review, a newer model is unlikely to be the first thing you need. You need a closed operating loop: every weak conversation becomes evidence, every useful insight gets an owner, and every change is tested against the next conversation it is meant to improve.
Measure the improvement loop, not just resolution rate
A self-improving support operation is not an agent that quietly rewrites or retrains itself. It is a managed system in which live conversations expose failure modes, people convert those failures into controlled changes, and later conversations show whether the changes worked.
Resolution rate is an outcome of that system, not a diagnosis. An aggregate rate cannot tell you which intent deteriorated, why the agent handed a customer to a human, or whether a change repaired one topic while damaging another. It can also be misleading when eligibility changes. Expanding automation into harder intents may lower the rate while increasing the number of conversations resolved. Excluding difficult intents can produce the opposite effect.
Start by documenting exactly what your denominator includes and what counts as a resolution. Keep that definition stable enough to compare periods, and report resolved volume alongside the rate. Then add the views that turn a dashboard into a work queue:
- Coverage: Which inbound conversations are eligible for AI handling, and which are excluded?
- Outcome by intent: Where does the agent resolve, hand off, or fail to answer?
- Failure reason: Was the problem missing knowledge, weak retrieval, incorrect behavior, poor routing, or an issue the product itself must solve?
- Quality: Did an audit, repeated contact, reopened conversation, or another trusted signal indicate that the apparent resolution was weak?
- Change throughput: How many identified failures are waiting for diagnosis, testing, approval, or release?
The intent-level view matters because it gives the owner somewhere to act. A falling aggregate rate is merely a warning. A cluster of unresolved questions about one feature, tied to one failure reason, is a tractable product and operations problem.
Classify the failure before choosing the fix
Teams waste cycles when every poor answer is treated as a documentation problem. Use a small failure taxonomy to route each issue to the layer that can actually repair it.
| Failure class | What you observe | Likely action |
|---|---|---|
| Knowledge gap | No current, approved answer exists | Create or repair the canonical content |
| Retrieval gap | The answer exists, but the agent does not receive or select it | Improve structure, segmentation, metadata, or retrieval configuration |
| Behavior gap | The right information is available, but the response is incomplete or misapplied | Adjust instructions, examples, or agent configuration |
| Routing gap | The agent should escalate but does not, or the handoff loses essential context | Change escalation conditions and the handoff payload |
| Product gap | No support answer can resolve the underlying problem | Send the evidence to product or engineering instead of disguising it as a content task |
This distinction prevents two common errors: endlessly rewriting accurate content when retrieval is broken, and asking the support agent to explain around a product defect that requires an actual fix.
Give one owner the authority and the improvement queue
Shared participation is useful. Shared accountability is not. One person should own the performance of the AI support operation, even though support, product, content, engineering, and security may contribute to individual changes.
The title can be AI operations lead, support operations specialist, or something else. The mandate is what matters: identify underperforming intents, maintain the improvement backlog, coordinate changes across functions, enforce the evaluation process, and report what improved or regressed.
Ownership becomes especially important after the launch surge fades. At Dotdigital, performance held at about 2,800 resolved conversations per month for three consecutive months. The response was to create a dedicated support operations specialist role focused on snippets, content, and the agent’s resolution capability. The lesson is not that every company needs the same job title. It is that a plateau without an empowered owner tends to remain a plateau.
Do not bury improvement work in the general support queue. A customer ticket can close while the underlying failure remains. Create a separate, persistent record for the system-level issue, with fields that make it possible to trace evidence through to an outcome:
- Representative conversation links and the affected intent
- The observed failure and its customer consequence
- The failure class and the evidence supporting that diagnosis
- The knowledge, retrieval, behavior, routing, or product artifact to change
- The accountable owner and required reviewer
- The evaluation cases that must pass
- The release status, version, and deployment date
- The live signal that will be checked after release
Define done as more than content published or configuration changed. An improvement is complete only when the change is linked to its originating evidence, reviewed at the appropriate risk level, tested, released, and checked in live operation.
For prioritization, assess recurrence, consequence, confidence in the diagnosis, and effort separately. Do not let raw volume make the decision by itself. A rare failure involving access, privacy, or an irreversible customer action can deserve attention before a frequent wording problem. Conversely, a recurring low-risk knowledge gap may be the best candidate for a fast content repair.
Turn live failures into governed, testable changes
Feedback does not improve an agent merely because it was collected. A thumbs-down, a handoff, or an unresolved conversation is a signal, not a root cause. The operating loop has to convert that signal into a specific hypothesis and then close the loop.
- Collect: Group common handoffs and unresolved conversations by intent instead of reading them as isolated tickets.
- Diagnose: Assign a failure class and confirm that the proposed layer is actually responsible.
- Prioritize: Select the issue using recurrence, consequence, confidence, and effort.
- Change: Modify the smallest responsible artifact rather than making broad agent changes by default.
- Evaluate: Test the originating failures, realistic variations, and already-passing cases that could regress.
- Release and observe: Record what shipped, monitor the affected live intent, and feed any new failure back into the queue.
Write the hypothesis before making the change: for this intent, changing this artifact should reduce this failure reason without degrading these existing behaviors. That sentence forces clarity about what success means and which regression cases belong in the evaluation set.
When a live failure reveals a missing case, promote it into the regression set after the fix. Over time, the evaluation suite becomes a practical memory of mistakes the operation should not repeat. That is where compounding comes from: the team is not merely correcting answers; it is preserving each correction as a reusable control.
Match governance to the blast radius
Fast iteration and responsible review are compatible when the rules are explicit. A useful governance model distinguishes changes by consequence:
- Low blast radius: A correction to an approved fact, an obsolete product step, or a missing limitation can follow a lightweight peer review and the relevant evaluation cases.
- Moderate blast radius: Retrieval, behavior, and routing changes that can affect several intents should receive cross-functional review and a controlled release.
- High blast radius: Actions involving permissions, account access, customer data, money, or security need stronger approval, a safe test environment, a rollback path, and an obvious route to a human.
A wrong explanation can create confusion. A wrong action can change an account or expose data. Treating those changes as equivalent either slows harmless content repairs or makes consequential automation unsafe.
Use focused sprints without making improvement episodic
A concentrated sprint is useful when the backlog has accumulated or a set of topics is visibly underperforming. In one focused Anthropic effort, the team audited unresolved queries, repaired weak content, converted recurring macros into AI-usable snippets, and monitored live performance. That is a practical pattern for clearing known gaps quickly.
The sprint should strengthen the standing loop, not replace it. Keep the same taxonomy, backlog, review rules, and evaluation artifacts after the concentrated work ends. Otherwise, the operation improves during special events and drifts between them.
Make the improvement work visible in each operating review. Show the failure observed, the artifact changed, the evaluation result, and the live outcome or next check. Name the person who drove the repair. This rewards the behavior that creates durable gains instead of celebrating only a headline rate that few people can explain.
Make AI-ready knowledge part of product launch readiness
Company-specific support knowledge does not appear because the underlying model is capable. The agent needs current, approved information in a form it can retrieve and apply. Missing or contradictory knowledge is an operating failure, not a model mystery.
Treat knowledge as production infrastructure. Every topic needs an owner. Important changes need versions and effective dates. Retired instructions need to be removed or clearly superseded. The agent’s ingestion and retrieval path needs verification, just as the customer-facing help experience does.
A canonical source of truth does not have to be one enormous help article. It means there is one approved origin for the product facts from which help-center content, agent snippets, human macros, and other downstream formats are derived. When those formats are authored independently, contradictions are almost inevitable.
Add an AI support gate to the new product introduction process. Before a feature is considered ready, confirm that:
- A named owner is accountable for keeping the feature’s knowledge current.
- The canonical material explains what changed, who can use it, how it works, and where its boundaries are.
- Known limitations and escalation conditions are explicit rather than left for the agent to infer.
- The effective version or release state is clear, so old and new instructions cannot be confused.
- The content has been ingested or indexed and retrieval has been tested.
- Expected support intents and representative evaluation cases are ready before inbound volume arrives.
- Support has a defined path for returning launch-day failures to product, engineering, or the knowledge owner.
This is not only administrative hygiene. In my organization, embedding a canonical source of truth into launch readiness has consistently supported resolution rates above 50% for new features from day one. That result is evidence for the operating model, not a universal benchmark; intent mix, product complexity, and the definition of resolution still matter.
Do not automatically turn every human answer into permanent knowledge. First decide whether the resolution is generalizable. If it is, update the canonical material. If it is a legitimate exception, encode the escalation path. If the underlying issue is a product defect, preserve the conversation as product evidence and route it accordingly. The objective is a cleaner system, not simply more content.
Key takeaways for your next operating review
- Define self-improvement as a managed loop from conversation evidence to a verified change, not autonomous model learning.
- Keep resolution rate, resolved volume, coverage, failure reasons, and change throughput visible together.
- Assign one accountable owner with authority to coordinate support, content, product, and engineering.
- Classify each failure before fixing it so knowledge, retrieval, behavior, routing, and product problems reach the right layer.
- Turn repaired failures into regression cases, and apply stronger review as the blast radius increases.
- Make canonical, AI-ready knowledge a launch requirement instead of a cleanup task for support.
At your next review, take one recurring unresolved intent and trace it all the way through: evidence, diagnosis, owner, change, evaluation, release, and live result. If any link is missing, that is the first operating gap to repair. Once the path works for one intent, make it the default path for every failure worth learning from.
Leave a Reply