Month: December 2025

From Activation to Retention: A Practical Experiment System

Your acquisition dashboard can look healthy while retained usage stays stubbornly flat. If onboarding completions rise but customers do not return, the team may have optimized a checkpoint rather than a value-producing behavior.

The fix is not simply to run more tests. You need a connected operating system: define activation as a testable hypothesis, verify that it predicts retention, instrument the journey, and use controlled experiments to remove the friction that matters. That turns three separate growth activities into one learning loop.

Treat activation as a retention hypothesis

Activation is not the moment a customer finishes your onboarding flow. It is the specific, observable behavior that you believe signals meaningful product value and predicts longer-term use.

That distinction matters because product teams can make almost any shallow milestone improve. A progress bar can increase profile completion. A product tour can increase feature exposure. A shorter form can increase setup completion. None of those changes proves that customers reached a reason to return.

A usable activation definition needs six parts:

Unit: Decide whether you are measuring a person, workspace, account, or organization. In a collaborative B2B product, one person completing setup may not mean the account is active.
Behavior: Name the customer action that represents value, such as connecting a live data source, inviting a teammate, sending a first campaign, or completing an initial automation.
Threshold: State whether one occurrence is sufficient or whether the behavior must reach a minimum frequency, depth, or breadth.
Window: Set the period in which the behavior must happen. For example, an activation definition might require the event to occur within seven days of signup.
Downstream test: Name the later retained behavior that activation is expected to predict. Without this, activation is just another funnel conversion.
Eligibility: Document who belongs in the denominator and which test accounts, internal users, unsupported plans, or incomplete signups are excluded.

Write the definition as one sentence that another analyst could implement without asking what you meant. An illustrative version is: An eligible new account activates when it connects a live data source and completes its first automation within seven days of signup.

Then challenge every word. Why is the account the unit? Does a connected source contain live data or merely credentials? Does an automation have to run successfully? Why is seven days the relevant window? What recurring behavior should appear later if this event genuinely represents value?

Do not force one global definition across unrelated jobs. A marketer building a campaign and an administrator configuring a workspace may follow different paths to value. Use persona- or use-case-specific definitions when the underlying value differs, then make any aggregate reporting transparent about how those segments are combined.

My rule is simple: activation earns attention as a growth outcome only after it shows a credible relationship with retained use. Until then, it remains a hypothesis.

Prove that activation separates retained customers

You need three measurements to understand activation properly. A single conversion percentage hides whether customers are moving faster and whether the milestone has any relationship with future behavior.

Metric	How to define it	Decision it supports
Activation rate	Eligible new units that meet the full activation definition divided by all eligible new units in the cohort	How many customers reach the proposed value threshold?
Time to activation	Elapsed time from the agreed starting event to completion of the activation threshold	Where can the team shorten the path to value?
Early retention	Share of a signup cohort that repeats a meaningful value behavior at the selected retention horizon	Does activation predict a reason to return?

Activation rate tells you reach. Time to activation tells you speed. Cohort-based retention analysis tells you whether the proposed activation event deserves to matter.

Start with customers from the same signup period and split them into activated and non-activated groups. Compare their subsequent retention using the same retained action and horizon. Then repeat the comparison for the properties most likely to change the journey: role, plan, acquisition channel, use case, and onboarding path.

Read the result as a diagnostic, not as automatic proof:

If activated customers remain more likely to perform the retained behavior, you may have a useful leading indicator.
If the groups separate briefly and then converge, the event may represent early momentum without durable value.
If the groups barely separate, revisit the activation behavior, threshold, window, retention horizon, and instrumentation.
If only one persona shows a meaningful separation, a global activation definition may be concealing distinct value paths.
If activation predicts generic logins but not repetition of the core value behavior, your retention metric is probably too shallow.

Choose the retention horizon from the product’s natural cadence. A retained action should represent value expected at that stage of the customer lifecycle, not whichever interval happens to be the dashboard default. Returning to a daily workflow, completing a recurring business process, and renewing a periodic task are different behaviors and should not be flattened into an unqualified return visit.

Keep one important limitation visible: customers with high intent may be more likely both to activate and to remain. That makes the relationship correlational. To build a stronger causal case, run a randomized intervention that helps eligible customers reach activation, then inspect downstream retention as well as the immediate funnel result. The broader measurement discipline is to use experiments, holdouts, and incrementality when a decision requires more than correlation.

Version the activation definition rather than editing it silently. A change to the behavior, threshold, window, unit, or eligibility rules breaks comparability with earlier cohorts. Record the effective date and preserve the old definition long enough to understand the discontinuity.

Instrument the journey before optimizing it

An activation debate often turns out to be an instrumentation debate. One dashboard counts people, another counts accounts, a third includes internal traffic, and lifecycle messaging uses a separate rule again. No experiment can settle a question when the underlying outcome changes between systems.

Map the journey into the smallest useful sequence of discrete events:

Eligibility begins, such as account creation or entry into a supported plan.
The customer starts the setup or value journey.
Required prerequisites are completed.
The first meaningful value action succeeds.
The full activation threshold is met.
The customer repeats the retained value behavior at the chosen horizon.

Do not add events merely because a screen exists. Each event should answer a decision question: where customers stop, how long a step takes, which path they choose, or whether the promised outcome occurred.

Attach properties that explain meaningful variation. Role, plan, channel, and use case are useful when they change eligibility, intent, product access, or the path to value. Onboarding path and experiment assignment are essential when you need to connect an intervention to its outcome.

Before trusting a funnel, validate the tracking end to end with a known test account. Check the following:

Does the event fire only after the action succeeds, or does a click count even when the operation fails?
Can retries, refreshes, or background jobs produce duplicates?
Are anonymous sessions joined to the correct identified user and account?
Does the event timestamp represent the customer action or delayed processing?
Are mutable properties, such as plan or role, interpreted at event time or at query time?
Are employees, automated tests, demonstrations, and deleted accounts handled consistently?
Does the analytics count reconcile with the product’s operational record for the same eligibility rules and period?

If your analytics platform supports computed cohorts or derived metrics, calculate activation from its component events instead of firing a separate activation event with independent logic. That keeps the definition inspectable. If a separate event is necessary for downstream messaging, test it against the computed definition and alert on divergence.

Create a short metric contract containing the metric owner, unit, eligibility rules, event sequence, threshold, window, identity logic, exclusions, retained action, and current definition version. Product, engineering, data, marketing, and customer success should use that same contract.

A shared measurement layer across product, marketing, CRM, and revenue systems can shorten decision cycles, but tool consolidation does not repair ambiguous definitions. Establish the contract first, then make the systems conform to it.

Apply privacy-by-design to the properties you collect. Every attribute should have a defined purpose, access boundary, and retention policy. Collecting more segmentation data than you can govern creates risk without making the experiment more valid.

Run experiments as decisions, not releases

Once the baseline is trustworthy, diagnose the bottleneck before choosing a treatment. A low activation rate is an outcome, not a diagnosis.

If eligible customers never start, inspect wayfinding, permissions, value proposition clarity, and whether the next action is visible.
If they start but do not complete setup, inspect unnecessary fields, unclear requirements, external dependencies, errors, and handoffs.
If they complete setup but do not perform the value action, setup may be disconnected from the job they came to do.
If they activate but do not retain, reducing onboarding friction alone is unlikely to solve the underlying value or product-quality problem.
If one segment succeeds while another stalls, target the treatment instead of averaging away the difference.

Turn that diagnosis into an experiment card before implementation. Include:

Observation: The precise funnel step, segment, and behavior that indicate a problem.
Hypothesis: The mechanism you believe prevents customers from progressing.
Audience and unit: Who is eligible and whether randomization occurs by user, account, or another unit.
Treatment: The smallest meaningful product or lifecycle change that tests the mechanism.
Primary outcome: Activation rate or time to activation, defined by the metric contract.
Retention validation: The later behavior and horizon that determine whether the gain is durable.
Guardrails: Product-specific measures for errors, quality, unwanted actions, support burden, or other important tradeoffs.
Analysis plan: Minimum detectable effect, sample assumptions, planned segments, stopping rule, and decision rule.

Set the minimum detectable effect to match your traffic reality. If the available population cannot distinguish the effect that would change your decision, do not hide that limitation behind a busy experiment calendar. Test a more consequential change, collect observations for longer under a valid plan, or use discovery methods to improve the hypothesis before spending engineering time.

Pre-register the outcome and decision rules. Under a fixed-horizon design, honor the planned analysis point. If the team needs continuous monitoring, use an appropriate sequential method rather than repeatedly checking an ordinary test and stopping when the result looks favorable. Mature experimentation standardizes minimum detectable effect, pre-registration, guardrails, and valid sequential testing instead of improvising them for each launch.

Good activation treatments usually test one of four mechanisms:

Remove work: Eliminate unnecessary fields or steps, detect configuration automatically, pre-populate safe defaults, or defer nonessential setup.
Clarify the next action: Use progressive disclosure, a checklist tied to the activation behavior, or contextual guidance at the point of uncertainty.
Make success observable: Confirm that the value action worked and show the customer what changed as a result.
Reinforce the same path: Align lifecycle email, in-product messaging, and customer-success outreach around the next value-producing action rather than sending competing prompts.

Do not call an experiment successful just because activation rises. Interpret the immediate and downstream outcomes together:

Activation improves and retention improves: The treatment is a candidate to ship, subject to uncertainty and guardrails.
Activation improves but retention is not mature: Treat the result as provisional until the planned retention window closes.
Activation improves but retention declines: Do not ship on the leading metric alone. The treatment may be pushing low-quality completion or weakening customer understanding.
Activation is unchanged but time to activation falls: Decide whether the speed improvement creates enough customer or operating value to justify the change.
Neither metric moves: Check exposure, instrumentation, statistical sensitivity, and the assumed mechanism before declaring the entire opportunity unimportant.

AI can help analysts and product managers identify anomalies, generate segment cuts, draft hypotheses, and prepare stakeholder updates. It should not silently redefine a cohort, choose a winner, or alter a stopping rule. Require every AI-assisted conclusion to expose its underlying query, cohort definition, experiment version, assumptions, and data lineage. That keeps faster analysis from becoming faster confusion.

Build an operating cadence around durable value

Activation work weakens when it belongs only to the onboarding team. Product and design shape the path. Engineering and data establish trustworthy signals. Marketing sets expectations before signup. Lifecycle messaging and customer success influence what happens after it. All of them can improve a local metric while pulling the customer in different directions.

Use one scorecard and a recurring review with a stable agenda:

Trust: Review tracking changes, identity problems, definition versions, and unusual movements before discussing performance.
Behavior: Examine activation rate, time to activation, and retention by signup cohort and priority segment.
Experiments: Review exposure, planned decision points, guardrails, and whether retention evidence has matured.
Discovery: Add customer feedback, support patterns, and observed journey friction that could explain the quantitative result.
Decisions: Record what will ship, stop, continue, or be investigated, along with the evidence and owner.

Keep the backlog organized by journey bottleneck and mechanism, not by a loose collection of interface ideas. A proposed tooltip, automated default, email, and setup redesign may all test the same uncertainty. Seeing that relationship helps you choose the least expensive intervention that can produce a decisive learning.

Frame the objective around customer behavior: help more eligible new accounts reach recurring value sooner. Activation rate and time to activation are leading outcomes; retained use is the validation. This is more useful than output commitments such as launching a tour, shipping a checklist, or running a fixed number of tests. The discipline is to align product work with outcomes rather than output.

Once the event stream and eligibility logic are reliable, you can close the loop in near real time. A stalled prerequisite can trigger contextual help. A successfully completed value action can prompt the next relevant behavior. A customer who already activated should exit introductory messaging. Measure each intervention as part of the same system, and preserve consent, frequency controls, and clear ownership before automating it.

Key takeaways

Define activation with an explicit unit, behavior, threshold, time window, eligibility rule, and downstream retention test.
Compare activated and non-activated customers from the same signup cohorts before treating activation as a reliable leading indicator.
Measure activation rate, time to activation, and early retention together; each answers a different product question.
Validate the full event journey and publish a versioned metric contract before using the data for experiments or automated messaging.
Set the minimum detectable effect, stopping rule, retention horizon, and guardrails before an A/B test begins.
Do not ship a short-term activation lift that weakens retained behavior, product quality, or another material guardrail.

Start this week with one persona and one signup cohort. Write the activation definition in a single implementable sentence, validate its component events with a known account, and compare later retained behavior for customers who did and did not activate. If the definition survives that test, queue one experiment against the largest observed bottleneck. That is enough to replace disconnected growth activity with a system that learns.

References

December 2, 2025

How to Build Self-Service Analytics Teams Actually Trust

If product managers still open analyst tickets for routine funnel, activation, and retention questions, your analytics transformation has not reached self-service. Buying licenses and publishing dashboards may increase access, but access is not the same as decision autonomy.

You are not trying to turn every product manager into an analyst. You are creating a governed path from question to evidence to decision, while preserving analyst time for problems that require deeper investigation. That takes a shared measurement layer, curated entry points, clear ownership, and operating rituals that make data part of product work.

Start with the bottleneck, not the analytics platform

A self-service analytics transformation should begin with the service your teams experience today. Pick a common product question, such as which users complete activation, where a critical journey loses customers, or which cohorts retain. Ask a product manager to answer it from a standing start, then observe every dependency between the question and a decision.

Look for five distinct sources of friction:

Discovery friction: the product manager cannot find the relevant event, metric, or approved dashboard.
Definition friction: two reports use the same metric name but calculate it differently.
Construction friction: the data exists, but answering the question requires an analyst to build or join the view.
Trust friction: the product manager can create a chart but still needs someone to confirm that it is correct.
Decision friction: the chart answers what happened but does not connect the behavior to a product choice.

This diagnostic separates a tooling problem from a measurement or operating-model problem. Consolidating scattered tools into a unified analytics platform can reduce search and construction friction. It will not resolve inconsistent definitions, unclear ownership, or a culture in which every decision still needs analyst approval.

Establish a baseline before changing the stack. Track the elapsed time from a clearly stated question to decision-ready evidence, the mix of routine and investigative analyst requests, the frequency of conflicting metric definitions, and the product decisions that cite behavioral evidence. Avoid choosing a universal benchmark that your context cannot support. Measure your current state, separate question types, and set improvement targets from that baseline.

The intended end state is narrow enough to test. A product manager should be able to examine activation, funnel drop-off, and cohort retention without joining a reporting queue. An engineer should be able to see the behavioral signal after a release. An analyst should spend less time reproducing standard views and more time on questions whose ambiguity or complexity merits specialist work. When evidence is visible to the people making the decision, discovery and delivery can share the same facts.

Build a governed measurement layer before expanding access

Giving more people permission to query inconsistent data produces faster inconsistency. Self-service becomes trustworthy only when teams share a stable vocabulary for customer behavior and business outcomes.

Treat that vocabulary as a product. Events describe observable behavior; metrics encode an interpretation of that behavior. The distinction matters. A signup event may have a precise trigger, but an activation metric still needs a qualifying action, an eligible population, a time window, and explicit exclusions. If those choices live only inside one dashboard, the dashboard is carrying business logic that other teams cannot reliably reuse.

Use a standard instrumentation workflow for every new event or property:

Start with the decision. Record the question the team needs to answer and the action that could change because of the result.
Define the behavior. Specify the event trigger, required properties, allowed values, exclusions, and the naming convention it follows.
Assign accountability. Name the owner, identify the affected product flow, and record privacy and access requirements.
Implement consistently. Use the same instrumentation pattern and carry the change through the normal CI/CD path instead of relying on an undocumented one-off.
Validate before release. Compare the emitted payload with the tracking contract and check that required properties arrive with expected values.
Publish for reuse. Add the definition to the catalog and expose it through the appropriate curated report, rather than leaving users to discover raw events by trial and error.

This is the practical value of treating data requests like product requests: a team can ask for an event or property with a defined purpose, owner, and privacy classification, while a repeatable path takes it from instrumentation through CI/CD, documentation, and curated analytics.

Your catalog entry should answer the questions a future user will otherwise send to an analyst:

What behavior does this event represent, and what does it not represent?
Exactly when does it fire?
Which properties are required, and what do their values mean?
Which users, accounts, environments, or internal activities are excluded?
Who owns the definition and approves a semantic change?
What privacy classification and role-based access rules apply?
Which approved metrics and dashboards depend on it?
Is it active, deprecated, or scheduled for replacement?

Choose a naming convention that reveals intent, such as an object plus a past-tense action, and apply it consistently. The exact grammar matters less than eliminating synonymous events and ambiguous labels. Do not silently rename or repurpose an event after teams have built reports on it. Deprecate it explicitly, identify the replacement, and update dependent views so a semantic change does not masquerade as a change in customer behavior.

Governance should make the safe path easier, not turn every question into an approval request. Standard definitions, privacy-by-design, role-based access, named owners, and clearly labeled dashboards are the guardrails. Product teams should remain free to explore within them. That balance preserves both speed and trust.

Give non-technical teams a curated front door

A blank analytics canvas is not a self-service experience. It transfers the construction work to users without giving them the context needed to interpret the result. Start with a small set of approved views that answer recurring product questions, then let experienced users branch into deeper exploration.

For one critical product flow, publish three discoverable dashboards:

An activation dashboard that shows the eligible population, qualifying behavior, and relevant segments.
A journey dashboard that exposes conversion between meaningful steps and makes drop-off visible.
A retention dashboard that uses a documented cohort definition and return behavior.

Three well-governed dashboards are more useful than a large library nobody can navigate. Each one should state the question it answers, intended audience, metric definitions, default filters, exclusions, owner, and review status. If a chart is exploratory rather than authoritative, label it accordingly. Users should not have to infer whether they are looking at a decision-grade view or a working draft.

Build enablement around real decisions. Generic feature training teaches people where buttons live; it does not teach them which metric to trust. Use a current product question to show how to select a cohort, inspect a funnel, compare segments, and move from an observed pattern to the next investigation. Targeted onboarding, in-app guidance, and product tours can reinforce that path when users return to the platform.

Then hold a weekly readout for the teams using the flow. Ask what they learned, which decision changed, where a definition was unclear, and which missing property blocked the analysis. Starting with one end-to-end flow, three core dashboards, and weekly decision readouts gives you a controlled environment in which to repair the system before scaling it.

Watch the first few self-service attempts closely. If users repeatedly choose the wrong event, improve the label and catalog entry. If they can build a chart but cannot explain the denominator, curate the metric rather than adding more training. If sensitive properties are broadly visible, fix access design before inviting more users. Friction observed during onboarding is product feedback about the analytics experience.

Change ownership, decision rituals, and success measures

Self-service changes the division of work; it does not eliminate analysts or central governance. Product trios should define measurement needs while they shape a solution, not after engineering has finished it. The data function should own reusable semantics and quality mechanisms. Leaders should make evidence part of routine decisions instead of treating analytics as a separate reporting activity.

A workable ownership split looks like this:

Role	Primary ownership	Boundary
Product trio	Decision question, hypothesis, measurement plan, interpretation, and product action	Does not redefine shared metrics inside a local dashboard
Data and analytics	Event taxonomy, reusable metric definitions, validation patterns, and deeper investigation	Does not become the required builder for every routine chart
Engineering	Accurate instrumentation, implementation consistency, and release validation	Does not decide business meaning without product and data input
Platform and governance owners	Access, privacy controls, catalog standards, dashboard hygiene, and lifecycle management	Does not approve every permitted exploration
Product leaders	Decision rituals, outcome accountability, and protection of specialist capacity	Does not reward unsupported numbers simply because they arrive quickly

Carry measurement through the entire product lifecycle. During discovery, write the expected behavior and the evidence that would change the team’s view. During delivery, make instrumentation part of acceptance criteria. After release, inspect the same agreed signals instead of inventing success measures retrospectively. Before an A/B test, state the hypothesis and justify the minimum detectable effect so the team understands what result the experiment is designed to detect.

The weekly decision ritual can remain simple:

What product question did you answer?
Which governed event, metric, cohort, or dashboard supported the answer?
What decision changed, or what new uncertainty must be resolved?
What defect in instrumentation, definition, access, or documentation slowed you down?

This keeps dashboards connected to action and creates a visible backlog for the analytics product itself. It also prevents login counts from becoming the transformation’s main success measure. A person can log in frequently without reaching a trustworthy conclusion.

Measure the operating change instead:

Time from question to decision-ready evidence, segmented by question type.
Routine questions resolved by the product team without an analyst handoff.
Analyst capacity spent on recurring report construction versus deeper investigation.
Critical events and metrics with complete definitions, owners, and privacy classifications.
Duplicate dashboards, conflicting definitions, validation failures, and data-quality incidents.
Discovery, experiment, and post-release decisions that reference governed behavioral evidence.

Read these signals together. If platform usage rises but routine tickets remain unchanged, access improved while autonomy did not. If self-service rises while definition disputes and quality incidents increase, governance is lagging adoption. If tickets fall but teams cannot name decisions informed by data, people may have stopped asking rather than become self-sufficient.

Do not remove analyst support merely to make the ticket count look better. The intended shift is from repetitive construction to higher-leverage work: investigating ambiguous patterns, improving measurement quality, supporting sound experiment design, and helping teams interpret questions that exceed the safe limits of a standard dashboard.

Key takeaways

Define self-service as the ability to reach trustworthy, decision-ready evidence without a routine analyst handoff, not as access to an analytics tool.
Standardize event definitions, properties, ownership, privacy requirements, validation, and documentation before expanding access.
Begin with one critical flow and three curated views: activation, journey conversion, and retention.
Teach analytics through live product questions and reinforce it with weekly decision readouts.
Measure time-to-insight, ticket mix, governed coverage, quality failures, and decisions changed; logins alone cannot show autonomy.
Scale only after teams can move faster without creating conflicting metrics, unsafe access, or hidden analyst dependencies.

Choose one critical flow in your next planning cycle. Baseline its current question-to-decision path, define the tracking contract, publish the three governed views, and schedule the first weekly readout. Let that flow prove the operating model before you broaden the rollout. Self-service scales when each new team inherits a trusted path, not another blank workspace.

References

December 2, 2025

Own Your AI: 4 Essential Roles to Supercharge Support and Prevent Performance Drift by 2026

AI doesn’t fail because the model is bad, it fails because ownership is missing.

When someone truly owns your AI, everything changes. Resolution and automation rates climb, the system self-improves, and the customer experience transforms in ways a dashboard alone will never show you.

This is part three of our five-part series on customer service planning for 2026. We’ll be sharing all five editions on our blog and on LinkedIn.

If you’d rather have them emailed to you directly as they’re published, drop your details here.

Last week, we introduced the four roles that make AI actually work in a support organization. These roles are already showing up inside the teams who are scaling AI the fastest, and this week, we get closer to the ground.

Here’s what these roles look like in practice — what they do, how they work, and why your AI performance will inevitably drift without them.

AI operations lead — owns AI performance, every day. I think of this person as the air-traffic controller for our AI Agent. I treat the AI as a living system that needs ongoing supervision, evaluation, and tuning. This role is accountable for what leaders care about most: quality, reliability, and continuous improvement.

The AI ops lead sees the whole picture: conversation quality, missing knowledge, flawed assumptions, unexpected failures, new opportunities for automation, and the subtle signals that the system is beginning to drift. In practice, that vigilance is the difference between steady gains and slow decline.

Day-to-day, here’s what I expect from this role.

1. Reviews AI conversations and surfaces performance patterns. The AI ops lead monitors the AI Agent’s behavior — the tone shift after a product launch, a sudden dip in resolution for a specific intent, or conversation clusters revealing new customer behavior. They scan for anomalies, trends, and early warnings, with an emphasis on what’s happening right now, not last week. Without this intentional ownership, I’ve watched a 2% dip turn into a 10% drop in days.

2. Prioritizes fixes and improvements. Once patterns emerge, they triage fixes like a product team handles bugs. Missing or incorrect content? They route it to the knowledge manager. Behavioral issues? They adjust guidance and guardrails. Action or system issues? They partner with the automation specialist. This connective tissue turns individual fixes into compounding improvements.

3. Defines and maintains AI guardrails. Leaders everywhere worry about AI doing things it shouldn’t. This role answers that fear by establishing clarification logic, escalation rules, “never answer” policies, and safety boundaries. The goal is predictable behavior that protects customer trust — an essential pillar of any AI Strategy and AI risk management practice.

4. Aligns reporting with leadership. The AI ops lead reports on resolution rate, CX Score, CSAT, automation coverage, and hours saved — making the economic impact visible. That visibility is a foundational step in any credible customer support ai strategy.

Why this role exists now. AI systems are dynamic and require constant tuning. A small dip in quality quickly becomes an operational issue, and no existing role naturally owns that. When someone does, teams feel the benefit almost immediately.

Knowledge manager — builds and maintains the structured knowledge AI depends on. I hear the same thing from leaders again and again: AI is only as good as the content you give it. This role is rapidly evolving from classic knowledge management into knowledge strategy — part content designer, part systems thinker, part information architect. Their job is to build the knowledge scaffolding that lets AI answer accurately, consistently, and safely.

Here’s how the knowledge manager creates leverage.

1. Writes, maintains, and improves support knowledge — continuously. After every product change, they update articles, remove duplication, resolve contradictions, and pay down “knowledge debt” that quietly erodes accuracy. The upkeep is shaped by AI performance; when patterns expose gaps, they fix the source.

2. Structures knowledge for AI, not for browsing. Traditional help centers are for humans skimming pages. AI needs clean intent signals, crisp formatting, and clearly structured language. The knowledge manager designs that structure as intentionally as the content itself.

3. Works hand-in-hand with AI ops. Many performance issues stem from missing or unclear knowledge. When the AI ops lead surfaces recurring misunderstandings or low-resolution categories, the knowledge manager resolves the root cause at the source.

4. Ensures accuracy and compliance at scale. As AI handles more sensitive situations, the knowledge manager safeguards correctness, currency, and compliance — critical for data governance and regulatory alignment.

5. Develops a cross-functional knowledge strategy. The role creates a canonical, cross-functional source of truth that product, engineering, product marketing, go-to-market, and support (AI and human) can all rely on.

Why this role exists now. This is one of the highest-leverage positions in an AI-first support org. Teams like Rocket Money and Anthropic are hiring knowledge managers because AI accuracy depends on the quality of knowledge feeding it. Without this role, resolution rate caps out early and never climbs.

Conversation designer — designs how the AI speaks, clarifies, and interacts. AI isn’t just a tool customers use; it’s a representative they interact with. Tone, clarity, pacing, and conversational structure matter, especially in voice. Every word affects perceived expertise, trustworthiness, and brand. The conversation designer ensures the AI feels human-friendly without pretending to be human — the sweet spot that builds trust without misleading customers.

In my experience, staffing conversation design early accelerates results. It changes not only how we tune AI, but how we understand the end-to-end customer experience.

Here’s what great conversation design looks like.

1. Shapes the AI’s tone, voice, and communication style. This role refines phrasing, tunes politeness, adjusts how confusion is handled, and shapes micro-interactions that determine whether customers feel cared for or dismissed. On voice channels, natural cadence is make-or-break.

2. Designs flows for high-value conversations. They design how the AI clarifies intent, branches, communicates uncertainty, verifies details, escalates, hands off, and returns to the main thread without feeling mechanical — treating customer experience as a product with language as the interface.

3. Translates procedures and complex workflows into natural language and logic. As AI runs structured procedures and actions, this role becomes a conversational system architect, translating SOPs into conditional logic with exceptions and fallbacks. For example, in Intercom, our conversation designer uses Simulations to run simulated conversations to see where the AI Agent gets confused, over-confident, or awkward, and refine flows until the interaction feels effortless end-to-end.

4. Ensures transitions to humans feel smooth and respectful. Handoffs should provide clear context to the human agent and maintain continuity so customers never feel dropped.

Why this role exists now. As AI becomes the primary interface, conversation design directly influences trust, brand perception, and operational outcomes. It’s a core competency for any Generative AI and LLMs for product managers program.

Support automation specialist — builds the backend actions that allow AI to do real work. If the conversation designer shapes expression, this role shapes capability. They transform AI from an answering machine into an outcome engine by bridging AI and the systems it must safely and deterministically act on.

Support teams increasingly expect AI to do what a human would do: refund a charge, adjust a subscription, verify an identity, update an account setting, or pull relevant data. That expectation creates a new technical role at the edge of support, ops, and engineering.

What I rely on this specialist to deliver.

1. Creates and maintains backend workflows the AI executes. This includes building and maintaining: Fin Tasks. Fin Procedures with embedded steps. Action flows that call internal and external APIs. Automations that span billing systems, user identity layers, CRM objects, subscription entitlements, refund tools, and more. They ensure the AI can act compliantly and predictably — the playbooks that turn intent into action.

2. Owns the integrations required for advanced automation. Many problems require data elsewhere — billing platforms, internal databases, systems of record. The specialist ensures the AI can retrieve, validate, and use that information safely, often partnering closely on CRM integration and internal services.

3. Partners closely with product and engineering. Some workflows require new endpoints, permission layers, safety gates, or deterministic fallbacks. This role drives those changes across the stack.

4. Ensures reliability and safety at every step. Guardrails, validation logic, exception handling, safe execution paths — all are essential. They confirm that the AI has access to the correct data, the action matches policy, edge cases are accounted for, risky flows have deterministic constraints, and every action is auditable and reversible.

Why this role exists now. Customers don’t want answers, they want outcomes. AI can now deliver those outcomes, but only with the right backend scaffolding. This role modernizes operational architecture and unlocks end-to-end automation.

How these roles work together — the new operating loop. These roles aren’t silos; they’re interdependent parts of one system. The AI ops lead identifies patterns and performance gaps. The knowledge manager resolves inaccuracies or missing content. The conversation designer improves clarity, tone, and flow. The automation specialist expands the system’s ability to take action. Each improvement compounds the next, moving you from early automation to transformational resolution rates through continuous refinement.

This loop is what separates teams that plateau early from teams that scale AI into a reliable, high-performing system — the essence of a durable AI Strategy.

How to get started (even if you can’t hire all four roles today). Most teams phase into this model: assign partial ownership, formalize responsibilities, then specialize as AI volume grows. Here’s the progression I recommend.

Phase 1: Assign ownership. Give each role’s core responsibilities to someone who can devote five to 10 hours weekly. Early on, support ops, enablement, senior ICs, and technically inclined teammates can anchor the work.

Phase 2: Formalize the responsibilities. As AI resolves more queries, optimization becomes core operational work. Formalizing ownership prevents performance drift and knowledge debt.

Phase 3: Specialize and hire. Once AI handles 50–70% of incoming volume, these responsibilities become full-time roles. Investing in specialization becomes essential infrastructure for the next scale stage.

The bottom line. AI changes the shape of your support team. These four roles — AI operations lead, knowledge manager, conversation designer, and support automation specialist — form the backbone of the AI-first support organization. They bring order to a constantly changing environment and enable AI to deliver the outcomes leaders and customers expect heading into 2026.

Next week, we’ll continue the 2026 planning series with a deep dive into org design models for AI-first support teams — how to structure people, workflows, and accountability in a world where AI resolves most conversations before a human ever sees them.

To follow along with the series and have each new edition emailed to you directly, drop your details here.

Inspired by this post on The Intercom Blog.

December 2, 2025
AI vs. Human Judgment in Customer Interviews: The Hard‑Won Lessons That Changed My Mind

I recently revisited a topic I once pushed back on: using AI to analyze (and maybe even synthesize) customer interviews. After six months of real-world experiments and countless conversations with seasoned product leaders, I’ve evolved my perspective. There is meaningful value here—but only when we’re clear about where AI helps and where it quietly erodes the hard-won customer understanding that powers great product decisions.

If you want to experience the conversation that sparked this reflection, you can listen to the episode on Spotify or Apple Podcast, and watch the discussion here: YouTube. It’s a candid, practical exploration of AI’s role in continuous discovery, and it mirrors what I’m seeing on the ground with product trios and empowered product teams.

Here’s the crux: AI raises the floor for beginners but accelerates experts even more. That matches my experience—early-career PMs get structure, momentum, and a confidence boost, while experienced interviewers can move faster without sacrificing nuance. But there’s a catch. If your interviewing skills aren’t solid yet, AI can create a veneer of insight that masks shallow understanding. In other words, it can help you go wrong more efficiently.

The conversation makes an important distinction between analysis and synthesis. Analysis is about extracting signals from the interview. Synthesis is about building meaning—connecting patterns, weighing contradictions, and deciding what to do next. AI can speed up the former with summaries and highlights. The latter—true synthesis—still demands expert judgment, context, and empathy.

One line from the episode stuck with me: your unpolished interview skills matter more than any shiny new AI workflow. I’ve felt that firsthand. When interview quality is uneven, dropping transcripts into an LLM won’t save you. You still need to synthesize every interview individually so the signals remain traceable and credible. That discipline keeps teams aligned, prevents overfitting to noise, and builds the organizational memory that fuels better bets.

We also explored the operational reality most teams face: interviews pile up. Backlogs grow. Leaders want speed. This is where “expert + AI” shines. With the right prompts, templates, and context, tools like ChatGPT and Claude can help transform raw transcripts into structured artifacts you can trust—provided a strong interviewer sets the frame and makes the calls. That balance preserves both velocity and quality.

What changed my mind most was the evidence from experiments—running sets of interviews through different LLMs and comparing outcomes. The patterns were consistent: beginner + AI is usually better than nothing, but the real performance gains come from expert + AI. When experts guide the process, AI becomes an accelerant rather than a crutch.

A favorite story in the episode takes a detour into building a gaming PC—an unexpected but perfect metaphor for AI’s limits. You can get great step-by-step guidance from a model, but when context shifts or edge cases appear, expertise is what keeps you from making expensive mistakes. Customer interviews are like that. Empathy comes from human interaction; AI can’t replace the experience of talking directly to your customers.

My practical guidance for teams integrating AI into continuous discovery: start with interviewing fundamentals, separate analysis from synthesis, and standardize how you capture single-interview learnings. If you need a tight template for this, refer to “The Interview Snapshot: How to Synthesize and Share What You Learned from a Single Customer Interview.” Use AI for summaries, clustering, and draft artifacts—but have an expert finalize the narratives, evaluate trade-offs, and document assumptions.

If you’re scaling this across an organization, invest in training first, then in workflows. Build a lightweight operating system for discovery: consistent interview guides, “story-based” techniques, and a shared library of prompts. Consider resources like “The Interview Coach,” as well as practical write-ups such as “Customer Interview Analysis: Where AI Helps and Hurts.” These help teams avoid common pitfalls and make better use of AI in high-judgment moments.

My bottom line: AI isn’t magic. It can help, but only if your interviews are strong and you provide the right context. Customer understanding is a competitive moat; outsourcing it entirely will cost you in the long run. Use AI to accelerate—not replace—the human judgment that makes product discovery work.

Resources and links worth exploring: ChatGPT, Claude, The Interview Snapshot: How to Synthesize and Share What You Learned from a Single Customer Interview, The Interview Coach, and Customer Interview Analysis: Where AI Helps and Hurts.

I’d love to hear how your team is using AI in discovery. What’s working, what’s risky, and where do you draw the line between automation and judgment? Share your experiences in the comments—our community learns faster when we compare notes.

Inspired by this post on Product Talk.

December 2, 2025
From Output to Outcomes: How I Align Stakeholders Around a True Product Operating Model

When I push our organization to adopt the product operating model, I’m emphasizing a foundational shift—from “shipping roadmaps of features (output)” to solving real customer and business problems, measured by “business results (outcomes)”. That’s the difference between activity and impact, and it’s the only way to build durable value at scale.
This change inevitably reaches beyond the product organization. It reshapes how company stakeholders in Sales, Marketing, Customer Success, Finance, Legal, Security, and Operations engage with product teams, and it reframes what they expect from us. Instead of asking, “When will feature X ship?” they learn to ask, “How will we move the outcome that matters?”
In practice, the product operating model is a contract: product teams commit to outcomes, and stakeholders commit to partnership. That partnership means we co-own the problem, align on evidence, and share accountability for results. The reward is clarity—everyone sees how their work ladders to strategy and why the sequence of work makes sense.
Here’s how I align stakeholders around this model. First, I ground everything in outcomes vs output OKRs. We replace feature roadmaps with a clear strategy, prioritized problems, and measurable objectives. Our product roadmapping and sprint planning then serve the objectives—not the other way around—so capacity is allocated to the highest-leverage bets.
Second, I build empowered product teams around product trios (product, design, engineering). We practice continuous discovery with stakeholders: we share opportunity trees, test riskiest assumptions early, and bring partners into research when it informs go-to-market strategy, pricing, or enablement. This keeps us honest and avoids late-stage surprises.
Third, I establish operating rhythms that make outcomes visible. Monthly stakeholder reviews focus on progress toward objectives and what we’re learning—not status theater. Quarterly, we connect OKRs to business performance so leaders can see the throughline from discovery and delivery to pipeline, retention, or margin. If priorities shift, we renegotiate objectives explicitly.
Fourth, I define metrics that stakeholders trust. We use a balanced set of leading indicators (activation, engagement, cycle time) and lagging indicators (revenue, retention, unit economics). We socialize definitions early so no one debates the scoreboard mid-game. The result: faster decisions and less “data whiplash.”
Fifth, I invest in change management. Moving from outputs to outcomes can feel threatening if your success has historically been measured by launch volume or roadmap commitments. I address this head-on with training, transparent comms, and clear decision rights. The message is simple: outcomes create more autonomy for empowered product teams and more predictability for stakeholders.
At HighLevel, this approach has been especially powerful when cross-functional dependencies are high. For example, when we set an objective to improve user activation for a new CRM integration, we didn’t promise a bundle of features. We committed to a measurable lift in activation and a shorter time-to-value, co-owned with Customer Success and Marketing. That alignment unlocked smarter experiments, tighter enablement, and a more credible launch narrative.
The anti-patterns are predictable: treating OKRs as a renaming of the roadmap, equating discovery with indecision, or isolating product decisions from go-to-market strategy. The cure is equally consistent: bring stakeholders into discovery, attach every bet to an objective, and show progress with evidence—not just demos.
Ultimately, the product operating model is a leadership choice. It asks us to trade certainty theater for learning velocity, and feature checklists for business impact. When stakeholders see that shift pay off—in faster cycles, clearer priorities, and results that matter—support for the model moves from compliance to conviction.

Inspired by this post on SVPG.

December 1, 2025
Unlock AI Product Roadmaps: Essential Tools Every PM Needs to Prioritize and Ship Faster

In my role leading product teams, the AI product roadmap isn’t just a plan—it’s the operating system for how we discover value, prioritize with rigor, and ship with confidence. The pace has changed, the stakes are higher, and the best product managers are now orchestrating AI capabilities, data, and customer insight in near-real time.

Master the evolving art of the AI product roadmap. Prioritize smarter, turn data into direction and insight into action, only much faster.

When I say “AI product roadmap,” I’m talking about a living system that blends strategy, discovery, and delivery. It’s less about dates and more about outcomes, risk reduction, and sequencing learning. In practice, that means combining AI Strategy with product roadmapping and sprint planning, then validating each bet with real customer signals.

For prioritization, I anchor on outcomes vs output OKRs and connect them to measurable signals across the funnel. Continuous discovery keeps insights flowing, while a unified approach to analytics and retention analysis tells me where the lift is. This lets me rank initiatives not just by impact and effort, but by how quickly we can learn, iterate, and compound value.

On discovery, product trios are non-negotiable. We prototype early with gen ai and LLMs for product managers to accelerate concept validation and reduce ambiguity. When customers can co-create through in-app guides or lightweight product tours, we turn vague needs into crisp problem statements and testable hypotheses far faster.

On delivery, I pair tight feedback loops with experimentation. A deliberate cadence of A/B testing and strong instrumentation ensures we’re learning every sprint, not just launching. The goal is to de-risk decisions quickly, keep momentum high, and translate signals into roadmap movement without thrash.

Under the hood, the AI stack matters. I rely on a retrieval-first pipeline to ground models in trusted data, and I’m intentional about privacy-by-design and data governance from day one. As agentic AI patterns emerge, I put evaluation workflows in place so we can ship confidently—and safely—without slowing down innovation.

Finally, alignment is the multiplier. Clear narrative roadmaps tied to customer outcomes help stakeholders see trade-offs, while crisp interfaces with go-to-market and CRM integration close the loop from roadmap to revenue. When everyone can trace a line from AI strategy to shipped value, prioritization becomes easier and trust grows.

If you’re feeling the acceleration, you’re not alone. With the right AI product toolbox—rooted in discovery, grounded in data, and delivered through tight feedback loops—you can move faster, learn smarter, and build products your customers can’t live without.

Inspired by this post on Product School.

December 1, 2025