Tag: observability

  • From Customer Signals to Reliable Product Operations

    From Customer Signals to Reliable Product Operations

    Customer signals become operationally useful only when a team knows what each signal can establish, how quickly it requires action, and who owns the next decision. A support complaint, a workflow metric, and a detailed customer story may describe the same experience, but they do not carry the same context or call for the same response.

    The two source articles illuminate opposite ends of this system. The incident-management article shows how customer impact should trigger rapid containment, while the product-discovery article explains why early evidence usually needs enrichment before it supports a durable product commitment. Together, they suggest a product operations model that separates detection, diagnosis, recovery, and learning without disconnecting them.

    Key takeaways

    • Signals should be classified by purpose: some reveal that customers are being harmed, while others help explain why.
    • The cost of waiting should determine response speed, but urgency should not turn an incomplete signal into false certainty.
    • Support, behavioral data, operational telemetry, rollout monitoring, and customer interviews contribute different forms of evidence.
    • Strong product operations preserve signal provenance, route it to a clear owner, and define the next evidence-building or recovery action.
    • Incident learning and continuous discovery should feed the same organizational memory so recurring friction becomes easier to recognize and address.

    One customer signal can serve several operational jobs

    The phrase “customer signal” often collapses several distinct concepts. A signal can detect a change, indicate its scale, describe a particular experience, test an explanation, or evaluate a proposed solution. Confusion arises when an input collected for one of these jobs is treated as if it can perform all of them.

    The incident playbook reports that Support, including automated support capabilities, may identify a pattern in customer conversations before a technical dashboard exposes it. It also describes heartbeat metrics that track whether customers can complete core workflows, rather than merely whether underlying systems remain online. In that setting, tickets and outcome metrics act as detection mechanisms: they establish that the experience may be unhealthy and that investigation should begin.

    The evidence-focused article assigns a different role to many of the same inputs. It characterizes support tickets, app-store reviews, sales notes, and behavioral analytics as useful prompts for discovery but weak foundations for deciding what to build on their own. These sources can expose repetition or friction, yet they may omit the sequence, motivation, constraints, and tradeoffs behind the observed behavior.

    These positions are complementary. A compressed support report can be strong enough to initiate triage without being rich enough to define a roadmap solution. Likewise, a behavioral change can justify investigation without proving its cause. Product operations should therefore attach an explicit purpose to each signal: detect, size, explain, validate, or monitor. That label prevents teams from asking an input to support a conclusion it cannot carry.

    Response speed and evidence depth belong on different clocks

    Customer signals create two fundamentally different decision conditions. When customers are actively unable to complete an important task, delay expands the harm. When a team is considering a durable product investment, premature certainty can consume capacity and institutionalize the wrong interpretation.

    The incident article argues that a declared incident should become the responsible team’s immediate priority. Its reported process converges customer reports, product alarms, and engineer rollout monitoring on a rapid assessment of customer impact. It also reports that engineers monitor changes through production and that a rollback can land in a little under two minutes. In this context, a safe rollback does not require a complete causal theory; it is a reversible containment decision intended to reduce exposure while investigation continues.

    The discovery article describes a more deliberate progression through a “Ladder of Evidence.” Repeated low-context signals justify moving upward toward recent, story-based customer accounts. Those accounts reconstruct what the customer was trying to do, what happened, and what constraints shaped the experience. The purpose is not to delay action indefinitely, but to avoid turning frequency into an unsupported solution.

    A useful synthesis is to separate the action threshold from the belief threshold. Teams can act quickly when an intervention is reversible and the cost of waiting is high. They should demand richer evidence when a choice is difficult to reverse, consumes substantial capacity, or assumes a specific explanation for customer behavior. Fast containment and careful learning are therefore not competing philosophies; they govern different commitments.

    A routed signal system turns inputs into decisions

    Preserve provenance before interpreting the signal

    Every captured signal should retain enough context to show where it came from, which customer workflow it concerns, when it occurred, and whether it is an observation or an interpretation. This is a general operating practice rather than a fact reported by either source, but it follows directly from their shared concern with signal quality. A ticket summary, a metric anomaly, and an interview account should remain distinguishable after entering a common repository.

    Preserving provenance also makes limitations visible. A Sales note may reflect the priorities of a commercial conversation. A dashboard records selected events but not necessarily customer intent. A story-based interview offers depth about a specific experience but does not by itself establish prevalence. None of these limitations makes the source unusable; each defines the questions it can responsibly answer.

    Correlate without treating evidence as a vote

    The discovery article presents triangulation across quantitative data, organizational observations, and qualitative customer insight. It cautions, in effect, against treating three inputs as interchangeable ballots. Convergence can strengthen an explanation, contradiction can expose segmentation or missing context, and silence in one channel can reveal an instrumentation or access gap.

    The incident article supplies an operational version of the same principle. Customer conversations, heartbeat metrics, ordinary alarms, and rollout monitoring offer separate views of product health. A support pattern may establish visible pain, while a workflow metric helps assess scope and timing. Combining them produces a more useful impact picture than either channel can produce alone.

    Route the signal to an explicit next action

    A signal repository becomes a backlog graveyard if collection is not paired with routing. The next action might be incident triage, instrumentation review, identification of affected customers, a story-based interview, solution evaluation, or continued monitoring. The choice should reflect what is already known and which uncertainty most constrains the next decision.

    This routing step is where product operations adds leverage. It connects customer-facing teams, product trios, engineering owners, and decision-makers without pretending that every input deserves a feature request. It also creates a traceable path from the original observation to the investigation, intervention, and later result.

    Ownership and cadence close the signal-to-learning loop

    Signals move faster when ownership is defined before pressure arrives. The incident article reports distinct responsibilities for a technical lead, an incident commander when escalation is needed, a business lead for customer-facing coordination, and a resolution owner for follow-up work. The benefit is not hierarchy for its own sake; it is reduced ambiguity while customers are affected.

    Discovery needs comparable clarity. The evidence article places responsibility on product teams to distinguish observations from interpretations, match the research method to the question, and improve interview quality without discouraging customer contact. Product operations can support that discipline by making evidence strength visible and ensuring that recurring signals receive either an investigation owner or an explicit decision not to pursue them.

    The two workflows should ultimately reconnect. An incident can generate product questions about confusing recovery paths, missing safeguards, or poorly observed workflows. Discovery can reveal customer-critical actions that deserve heartbeat metrics or stronger operational readiness. Post-incident follow-ups, recurring signal reviews, customer research, and roadmap discussions should contribute to a shared record rather than separate departmental archives.

    The next stage of mature product operations is therefore not simply collecting more feedback or adding more dashboards. It is designing a system in which the weakest signal can trigger appropriate attention, stronger evidence can refine the explanation, and clear ownership can carry learning into safer product and operational choices.

    References

  • Migrate Analytics Platforms Without Chaos: 7 Proven Lessons to Plan, Move, and Land Cleanly

    Migrate Analytics Platforms Without Chaos: 7 Proven Lessons to Plan, Move, and Land Cleanly

    I’ve led and rescued more analytics migrations than I can count, and I know the pressure: every event, dashboard, and decision pipeline depends on getting it right. Migrating analytics platforms doesn't have to be painful. Get seven lessons from Human37 and Amplitude to help your team plan, migrate, and land cleanly.

    Here’s how I approach this work so teams keep momentum, regain trust in their numbers, and accelerate product-led growth on a unified analytics platform—without the rework and stakeholder fatigue that typically follow.

    Lesson 1 — Start with outcomes, not events. Before moving a single event, I align leaders on the questions we must answer and the decisions we must speed up: activation, retention, and expansion. I map those goals to a simple driver tree, then back into the behavioral analytics we need. This trims noise, tightens scope, and ensures Amplitude analytics (or any destination) is instrumented for decisions, not vanity metrics.

    Lesson 2 — Audit and map your data with rigor. I inventory current events, properties, IDs, and sources, then define a target schema with clear naming conventions, ownership, and versioning. Data governance and privacy-by-design are non-negotiable: we separate PII, document consent paths, and remove legacy debris. This step prevents schema drift and makes platform scalability sustainable.

    Lesson 3 — De-risk the cutover with a phased plan. Rather than a big-bang switch, I dual-run critical flows, compare telemetry, and use feature flags to roll forward (and back) safely. Observability and anomaly detection are my guardrails: I monitor volume, cardinality, and event timeliness to spot regressions early—long before executives notice broken charts.

    Lesson 4 — Treat instrumentation like product code. I wire schema checks into CI/CD, enforce typed analytics wrappers, and validate payloads pre-merge. With docs-as-code, the tracking plan stays current and reviewable. This keeps quality high at scale and avoids the slow death of broken funnels caused by well-meaning quick fixes.

    Lesson 5 — Enable the people, not just the platform. Tools don’t create insight—teams do. I run hands-on enablement with product tours and in-app guides tailored to each role, establish communities of practice, and publish short playbooks for common questions (activation analysis, cohort retention, and journey mapping). When customer success and growth marketers can self-serve, adoption sticks.

    Lesson 6 — Land cleanly with fast, visible wins. Within the first two weeks post-cutover, I showcase analyses that matter: retention analysis by use-case, friction points via session replay and heatmaps, and conversion lift by segment. These quick proofs build confidence, reinforce the value proposition, and keep stakeholders engaged through the longer tail of hardening.

    Lesson 7 — Govern and evolve continuously. After go-live, I schedule schema reviews, backlog grooming, and QBRs to prune events and refine definitions. Ownership is explicit, and changes flow through the same review process as code. This keeps the unified analytics platform trustworthy as the product (and org) changes.

    I’ve seen this playbook turn skepticism into momentum. In one migration I inherited mid-flight, we refocused on decisions, tightened governance, and phased the rollout; the team moved from fire drills to confident launches—and stakeholders finally believed the numbers again.

    If your team is staring down a migration, anchor on outcomes, automate quality, and invest in enablement. With disciplined execution readiness and the lessons I’ve applied alongside partners like Human37 and platforms like Amplitude, you can move fast, reduce risk, and land cleanly—without the chaos.


    Inspired by this post on Amplitude – Perspectives.


    Book a consult png image
  • AI Inference Economics: Optimize for Value, Not Cost

    AI Inference Economics: Optimize for Value, Not Cost

    AI inference economics cannot be reduced to the price of a model call. The financially relevant question is whether a change in model, latency, caching, or token use improves total product value after its effects on conversion, retention, support, and revenue are included.

    A reported decision to reject a projected $2 million in inference savings illustrates the distinction. The supplied source describes lower infrastructure costs alongside weaker downstream product signals, making the proposed optimization look attractive in a FinOps report but less compelling at the business level.

    The correct unit of analysis is the customer outcome

    Cost per request is useful for operating an AI product, but it is not a complete measure of its economics. A cheaper request can still be expensive if it makes a user more likely to abandon a session, fail a task, contact support, or leave the product.

    The source article reports that routing traffic to lower-cost options produced immediate cloud cost optimization. It also associates small increases in time to first token with greater session abandonment, subtle quality declines with lower task completion, and weaker performance in support deflection. According to the account, the resulting revenue exposure exceeded the projected expense reduction.

    This reframes inference efficiency as a value equation. Direct serving cost belongs on one side; incremental conversion, retained revenue, successful task completion, and avoided support demand belong on the other. The decision should be based on the net effect rather than whichever metric is easiest to retrieve from a cloud bill.

    Cost, latency, and quality form a coupled system

    Model cost, response speed, and output quality are often managed as separate workstreams. In practice, changing one can move the others. A smaller or cheaper model may reduce inference expense while changing answer quality. More restrictive token limits may shorten responses but remove information needed to complete a task. Caching may improve both cost and speed for repeatable requests, yet become unsuitable where fresh or highly contextual output matters.

    The source argues for treating these variables as one product system. That view prevents a local optimization from being mistaken for an overall improvement. It also makes latency distributions more informative than a single average: even when aggregate performance appears acceptable, slower experiences within particular workflows may coincide with abandonment or failed completion.

    The same principle applies to quality. A model-level score matters only insofar as it represents what users need from the workflow. For a support agent, that might involve resolving an issue without escalation. For another product experience, it might involve completing a task, activating a feature, or continuing to use the service. Business instrumentation gives technical measures an economic interpretation.

    Experiments must detect product harm, not just cost movement

    The reported evaluation combined eval-driven development with A/B testing and defined success through conversion, retention cohorts, and Net Recurring Revenue rather than cost per call alone. It also used minimum detectable effect calculations to determine whether the tests had enough statistical power to reveal meaningful changes in latency and answer quality.

    That approach suggests two complementary layers of evidence. Evaluations can identify whether model behavior changes on representative tasks, while controlled product experiments can show whether those changes matter to users and the business. Neither layer is sufficient by itself: an offline quality score may miss behavioral consequences, and a topline business metric may conceal the mechanism behind a regression.

    Guardrails are especially important when the expected saving is immediate but the product damage may emerge later. Infrastructure spend can fall as soon as traffic moves. Retention and recurring-revenue effects may take longer to appear. Conversion, task completion, session abandonment, support deflection, and cohort retention therefore provide signals across different time horizons.

    The evidence supplied here is one first-person case account, not independent corroboration. Its projected $2 million saving, observed correlations, and business conclusion should consequently be treated as case-specific rather than universal benchmarks. The transferable value lies in the measurement framework, not in assuming that every higher-cost model will produce a better commercial outcome.

    Key takeaways

    • Evaluate inference changes against total product value, including conversion, retention, support demand, and recurring revenue.
    • Measure cost, latency, and AI quality together because an intervention in one dimension can alter the others.
    • Pair task-level evaluations with controlled product experiments and size tests to detect economically meaningful regressions.
    • Apply optimization selectively: a technique is valuable where evidence shows that it lowers cost without harming the customer outcome.

    A selective optimization roadmap

    The alternative to indiscriminate cost cutting is not unlimited inference spending. The source describes a balanced roadmap built around targeted caching where experiments showed no adverse outcome, dynamic routing for task-specific workloads, and stronger observability to detect quality regressions early.

    Each method addresses a different part of the economics. Targeted caching can remove redundant work in stable interactions. Dynamic routing can reserve more capable models for tasks that justify them while sending simpler work to less expensive paths. End-to-end observability can connect routing, model, token, latency, and quality data with the behavior that follows.

    This also clarifies governance. FinOps teams can continue applying pressure to unit costs, while product teams define outcome guardrails and analytics teams verify the net effect. A proposed saving becomes ready for broader rollout only when the organization can see both the expense reduction and the customer or revenue impact.

    As AI products scale, the strongest operating discipline will be selective rather than reflexive: spend less where evidence supports it, invest more where inference creates measurable value, and revisit routing decisions as workflows and user behavior change.

    References

  • How I Use Novus, the First Product Agent, to Turn Rapid Releases into Measurable Wins

    In a world of relentless CI/CD and accelerating release trains, product leaders like me can’t afford lagging signals or fuzzy readouts on what’s truly moving the needle. I need immediate, trustworthy feedback that connects code shipped to outcomes achieved and customer value created.

    Coding agents compress weeks of development into hours, but the faster your codebase changes, the harder it is to know what’s actually helping end-users.

    That tension is exactly why I brought Novus into my product toolbox. To keep up with the pace of development, over 600 product teams are already using Novus, the first-of-its-kind product agent, to automatically set itself up, monitor product data, and tell you what to do next.

    From my chair, that promise matters only if it translates into clear decisions. With Novus, I’ve been able to tighten the loop between experimentation and learning: it pairs eval-driven development with behavioral analytics and observability so I can see how a release influences activation, engagement, and retention—without spelunking through fragmented dashboards. The agentic AI backbone reduces the manual stitching I used to do across events, cohorts, and funnels, letting me focus on prioritization and product strategy instead of report wrangling.

    Day to day, Novus fits naturally into our AI workflows. It surfaces anomalies early, clarifies trade-offs, and frames next-best actions in the language of outcomes. Because it plugs into a unified analytics platform approach, I can maintain continuous discovery at scale while preserving the rigor of Agent Analytics: hypotheses are explicit, telemetry is consistent, and results are traceable. That’s the operating cadence I expect from modern product management leadership.

    If your roadmap moves faster than your learning loops, a product agent can be the missing link between speed and certainty. Novus helps me convert rapid releases into measurable wins, keeping the team aligned and confident about what to build next—and just as importantly, what to stop doing.


    Inspired by this post on Pendo – Best Practices.


    Book a consult png image
  • Engineering MCP Agents as a Reliable Product Platform

    Model Context Protocol adoption becomes consequential when an agent can retrieve organizational knowledge, select tools, and change a system of record. At that point, the engineering challenge is no longer simply connecting a model to an API. It is operating a product platform whose context, permissions, decisions, and side effects must remain dependable.

    The source article’s experience with workflows spanning Miro, Jira, and Confluence points to a coherent platform model: retrieval determines what the agent knows, tool contracts constrain what it can do, evaluation tests its behavior, and observability makes failures diagnosable. Product strategy and interaction design then determine whether that machinery improves work users already perform.

    Key takeaways

    • Treat retrieval, tool schemas, prompts, policies, and telemetry as platform components with explicit owners and versioning.
    • Prove one frequent, measurable workflow before expanding the agent’s tool and use-case surface.
    • Combine least-privilege access with visible tool rationale, consent controls, audit records, and safe recovery paths.
    • Evaluate the complete chain from retrieved context to downstream action, not just the quality of generated text.
    • Govern the tool catalog and delivery pipeline continuously so that extensibility does not become uncontrolled operational risk.

    The platform boundary extends beyond the MCP connection

    MCP provides a practical interface through which models can reach data, tools, and actions, according to the source article. The protocol connection is therefore an enabling layer, not the whole agent platform. A production workflow also depends on source authority, identity and permission checks, context selection, tool arbitration, execution controls, user-facing recovery states, and evidence that the result was useful.

    This broader boundary changes how teams should decompose the system. Retrieval is a managed context service rather than an incidental prompt-building step. Tools are governed capabilities rather than a loose collection of endpoints. Prompts and policies are deployable artifacts rather than text copied into application code. Traces and evaluations are part of the control plane because they reveal whether the other layers continue to work together.

    The source recommends starting with authoritative content, normalizing it with docs-as-code discipline, attaching metadata that supports permission-aware filtering, and selecting the smallest high-signal context needed for a task. The engineering implication is important: access control must shape retrieval before information reaches the model. Filtering only when an action is attempted would leave the reasoning process exposed to context the user or agent may not be entitled to use.

    Context quality also affects more than answer accuracy. The source links focused retrieval to lower hallucination risk, more accurate tool calls, and lower cost. That makes retrieval performance a shared dependency for safety, reliability, latency, and economics. It deserves its own contracts, tests, freshness expectations, and failure modes.

    A golden path turns architecture into an operating contract

    The source describes an initial workflow that summarized a Miro board into action items and wrote them to Jira. It reports that variants involving Confluence summaries, epic splitting, and backlog grooming followed only after the original path reached its reliability targets. This is less a recommendation for those particular products than a useful sequencing principle for agent platform engineering.

    A narrowly defined workflow exposes the entire contract between context and consequence. The team must decide which content is authoritative, what the model may infer, which tool is appropriate, what inputs the tool accepts, what the user should review, how a partial failure is handled, and how success is measured. A broad assistant can conceal these questions behind plausible conversation; a golden path forces explicit answers.

    The right first workflow is therefore not merely technically convenient. It should be frequent enough to matter, have an observable completion state, and carry side effects that can be bounded. The source frames outcomes such as time saved during backlog grooming, better meeting notes in Confluence, and fewer context switches across Miro boards as more useful roadmap anchors than novel model capabilities. It also recommends comparing task success, completion time, user edits, detected defects, and downstream business effects rather than relying on engagement alone.

    Those measures form a practical evidence chain. Evaluation results show whether the system behaves as designed; workflow measures show whether users can complete the task; business measures show whether the completed task creates value. Keeping the levels distinct prevents a technically impressive agent from being mistaken for a successful product.

    Safety depends on controlling actions and explaining them

    Tool access creates a sharper risk boundary than text generation because an incorrect decision can alter a ticket, document, or other shared record. The source’s proposed response combines least-privilege scopes, a human-readable rationale for each call, and an audit trail. It also calls for proposed inputs and expected side effects to be visible when the agent is about to use a tool.

    These controls address different failure classes. Narrow scopes limit the maximum effect of a bad decision. Input previews help users catch incorrect parameters before execution. Rationale makes the selection inspectable. Audit records support diagnosis and accountability afterward. None substitutes for the others, and a confirmation dialog alone does not make an overprivileged tool safe.

    Recovery behavior belongs in the same design. The source recommends retrying suitable failures with backoff, falling back to read-only behavior, or requesting consent or missing context. A robust platform should distinguish failures that are safe to retry from failures that require a different plan. It should also preserve an understandable state when a multi-step workflow completes only partially, so the user knows what changed and what did not.

    Transparency need not mean exposing raw internal reasoning. The useful product surface is operational evidence: the sources used, the selected capability, the intended inputs, the expected effect, and the resulting status. The source suggests a reveal panel containing retrieved sources, candidate tools, and confidence signals for power users. More generally, the amount of review should follow the consequence of the action: low-risk retrieval can remain lightweight, while consequential writes warrant clearer inspection and consent.

    Evaluation, observability, and delivery form one reliability loop

    The source outlines offline tests for intent classification and tool selection, online shadow evaluations for live drift, and regression checks after deployment. It also recommends traces that capture prompts, retrieved chunks, tool inputs, tool outputs, latency, and error codes. Together, these practices connect a visible failure to the component and version that produced it.

    Evaluation without observability can show that quality declined without explaining why. Observability without evaluation can produce detailed traces without deciding whether the behavior was acceptable. A mature loop needs both: test cases encode desired behavior, traces expose actual behavior, and production outcomes reveal gaps in the test set.

    The delivery process must preserve that connection. The source treats prompts, tool schemas, and guardrails as versioned artifacts deployed behind feature flags, with canary releases, controlled comparisons, and rollback capability. This approach makes a behavioral change attributable. If tool selection deteriorates after a prompt revision or a schema update breaks an integration, operators can identify the change and contain its reach.

    Latency should be governed in the same loop because an accurate workflow can still fail as a product experience. The source reports using task-specific latency budgets, caching stable retrieval results, parallelizing safe calls, prefetching likely session context, and providing progress when work exceeds the expected budget. These techniques should remain subordinate to correctness: parallel execution is appropriate only when calls are independent, while caching must respect freshness and permission boundaries.

    The source also assigns prompts a user-experience role, combining plain-language intent, domain constraints, and explicit tool contracts while using examples, tooltips, and in-product guidance to help users frame requests. This connects conversation design to reliability. Better instructions can reduce ambiguity before the platform has to resolve it through additional model turns or risky assumptions.

    Scale requires governance of tools, teams, and ownership

    MCP’s extensibility can turn into tool sprawl if every integration is added without lifecycle management. The source recommends a curated catalog recording each tool’s owner, scope, schema version, and deprecation policy. It also describes schema linting in continuous integration, backward-compatible changes, and quarterly retirement of unused tools. These are conventional platform disciplines applied to an agent’s capability surface.

    A catalog is valuable because an agent reasons over descriptions and schemas while operators depend on stable implementation contracts. Poorly differentiated tools can make selection ambiguous; unannounced schema changes can invalidate prompts and evaluations; ownerless tools can remain available after their data or permission assumptions have changed. Governance should therefore assess semantic clarity as well as API validity.

    Organizational design matters for the same reason. The source describes an empowered trio consisting of a product manager responsible for outcomes and risk posture, a forward-deployed engineer focused on schemas and scalability, and a designer responsible for conversational flows and recovery states. It also favors weekly evaluation reviews over demonstration-led progress. The underlying principle is shared ownership: platform reliability cannot be delegated entirely to model engineering when the decisive questions span product value, system behavior, permissions, and user comprehension.

    The source’s proposed 30-day starter sequence moves from selecting one workflow and defining permissions, measures, and evaluations; through retrieval and a minimal tool set; to an instrumented internal pilot; and finally to hardening and a limited beta. The schedule is reported as a blueprint rather than independent proof of how long every implementation should take. Its more transferable lesson is dependency order: define the outcome and risk boundary before multiplying capabilities.

    As agents begin coordinating across products, the durable advantage will come from platforms that preserve this discipline across every new connection. MCP can make capabilities composable, but dependable composition will still depend on controlled context, explicit authority, observable execution, and evidence that the workflow improves real work.

    References

  • How to Build a Resilient Experimentation Program at Scale

    How to Build a Resilient Experimentation Program at Scale

    Your teams are running more experiments, but decisions are not getting easier. Results arrive late, apparent wins fail to repeat, and every readout starts a new argument about the data.

    The fix is not another testing tool or a higher experiment count. You need an operating system that protects validity when traffic, products, models, and customer behavior change underneath you. That system starts before exposure, routes each question to the right evaluation method, and ends with a decision your team can execute.

    Give every experiment a decision contract

    An experiment should begin with a decision, not a feature. Ask what you will do if the result is positive, negative, inconclusive, or unsafe. If the answer is the same in every case, the test is not worth running.

    Turn the proposed test into a short decision contract before engineering begins. Record:

    • The customer problem: the friction or unmet need you observed.
    • The causal hypothesis: the product change, the behavior it should alter, and why.
    • The eligible population: who can enter the experiment and who must be excluded.
    • The primary outcome: the one metric that determines whether the hypothesis worked.
    • The guardrails: the measures that can block a rollout even when the primary outcome improves.
    • The decision thresholds: the minimum effect worth acting on and the conditions for shipping, iterating, stopping, or rolling back.

    A driver tree helps you connect the primary metric to the business outcome without pretending that one experiment can prove the entire chain. If the goal is retention, for example, the immediate experiment may be designed to change activation behavior. The contract should distinguish that leading behavior from the longer-term outcome.

    Set the minimum detectable effect and guardrails before reading results. The minimum detectable effect is not the smallest movement your analytics can display. It is the smallest improvement that would justify the cost, risk, and complexity of the change. If your available population cannot reliably detect that effect, narrow the question, combine low-traffic variants, choose a more sensitive proximal metric, or do not run the test.

    Pre-committing to the metric, stopping rule, exclusions, and decision criteria also limits convenient reinterpretation. Teams can still investigate unexpected patterns, but those findings should become new hypotheses rather than retroactive proof that the original bet won.

    Match the question to the cheapest reliable evidence

    Production A/B testing is only one layer of experimentation. It is often the slowest and most expensive layer because it consumes customer attention, operational capacity, and statistical power. Use it when real behavior is necessary to resolve a meaningful decision.

    Evidence layerBest questionMove forward when
    Offline evaluationDoes the output meet a defined quality, policy, or safety standard?The candidate passes the agreed evaluation set and regression checks.
    Replay or shadow modeHow would the change behave on realistic inputs without affecting users?Failure patterns, cost, and latency remain inside the operating limits.
    Targeted canaryIs the change safe and observable under live conditions?Telemetry is healthy and no guardrail triggers a rollback.
    Controlled A/B testDoes the change cause a valuable shift in user behavior?The result meets the pre-registered decision criteria.
    Progressive rolloutDoes the effect and reliability persist as exposure expands?Segment-level outcomes and operational measures remain acceptable.

    This layered model becomes essential for AI products. Prompts, retrieval logic, policies, model versions, and traffic composition can all change the experience. A single production metric cannot tell you whether a decline came from product value, output quality, latency, cost, safety, or an upstream model shift.

    Build an evaluation stack for prompts, policies, regressions, canaries, and selective A/B tests. A candidate should earn broader exposure by passing the cheaper layers first. This reduces traffic waste and gives the team diagnostic evidence when a live result moves unexpectedly.

    Do not use a multi-armed bandit simply because it can direct more traffic toward a leading variant. Bandits are useful when the objective is clear, feedback is timely, and guardrails are dependable. They are a poor substitute for stable measurement or causal understanding. If you need to estimate an effect, learn about segments, or detect delayed harm, retain a controlled comparison.

    Engineer trustworthy measurement and reversible delivery

    An experimentation program is only as resilient as its event pipeline. A mathematically correct analysis built on shifting event definitions is still wrong. Treat instrumentation as a product interface with owners, documentation, versioning, tests, and observability.

    Before exposure begins, verify that assignment, exposure, outcome, and guardrail events share consistent identities and timestamps. Confirm that users enter only the experiments for which they are eligible. Check that retries, duplicate events, delayed ingestion, and cross-device behavior cannot silently change the denominator.

    Naming conventions, schema versioning, lineage, anomaly detection, and pipeline observability are not analytics housekeeping. They let teams move without sacrificing the meaning of their measurements. Assign an owner to every critical event and make schema changes visible to the teams whose experiments depend on them.

    During the run, monitor data quality separately from product performance. Sample ratio mismatch, assignment failures, missing exposure events, sharp volume changes, and implausible segment movements should pause interpretation. Do not explain these signals away because the headline result looks attractive.

    Delivery must be reversible as well as measurable. Put material treatments behind feature flags. Start with a targeted canary, watch operational and customer guardrails, and expand exposure in stages. Define who can stop the rollout and make sure that person has both the telemetry and access required to act.

    For broad platform or AI changes, maintain a persistent holdout when feasible. A long-lived control gives you a reference point for cumulative effects that short experiments miss, including changes in retention, trust, support burden, and cost. Protect the holdout from accidental contamination and document every change that affects its interpretation.

    Scale the program around decisions, not test volume

    A central experimentation team cannot design and analyze every test at scale. Product teams need autonomy inside a governed system. Centralize the parts where inconsistency creates shared risk: assignment services, metric definitions, event standards, quality checks, templates, and audit records. Let teams own hypotheses, customer context, treatment design, and decisions inside those guardrails.

    Use a lightweight review based on risk. A reversible interface change with a proven metric can follow a standard path. A pricing change, safety policy, ranking system, or shared AI capability deserves stronger review, tighter exposure controls, and a clearer rollback plan. Governance should become more demanding as the blast radius grows.

    Maintain a portfolio view rather than a leaderboard of teams by test count. For each active experiment, track the decision it supports, expected value, detectable effect, traffic requirement, risk class, owner, and current evidence layer. This reveals when several teams are competing for the same population, when a strategic question is underpowered, and when multiple small tests should become one coherent learning plan.

    Reset a brittle program over 90 days

    You can make the operating model concrete without attempting a platform-wide rebuild:

    1. By day 30: audit the backlog and current tests. Stop or consolidate experiments that cannot meet their minimum detectable effect. Identify unreliable events, missing owners, conflicting metric definitions, and launches without explicit decision criteria. For AI surfaces, establish a minimal offline evaluation harness for prompts, policies, quality, and safety.
    2. By day 60: publish standard hypothesis and readout templates. Put high-risk changes behind feature flags, make guardrails visible, and introduce canary exposure. Establish persistent holdouts where broad or cumulative effects matter. Add alerts for instrumentation drift and operational regressions.
    3. By day 90: manage a balanced portfolio across offline evaluations, replay or shadow tests, canaries, controlled experiments, and progressive rollouts. Review program health through decision speed, valid learning, repeatability, and detected harm rather than the number of tests launched.

    Create a community of practice alongside these controls. Regularly examine inconclusive results, failed replications, instrumentation incidents, and stopped rollouts. These cases expose weaknesses in the system more reliably than a gallery of wins. The goal is not to eliminate failure. It is to make failure informative, contained, and cheap.

    Key takeaways

    • Start with the decision the experiment must support, then pre-register the hypothesis, primary metric, guardrails, detectable effect, and stopping rule.
    • Use offline evaluations, replay, shadow mode, and canaries to eliminate weak or unsafe candidates before consuming production traffic.
    • Treat event semantics, assignment, exposure, lineage, and anomaly detection as production infrastructure.
    • Pair controlled measurement with feature flags, progressive exposure, explicit rollback authority, and persistent holdouts where cumulative effects matter.
    • Judge the program by trustworthy decisions and reusable learning, not experiment volume or the percentage of positive results.

    Choose one upcoming decision with meaningful customer or operational risk. Write its decision contract, identify the cheapest evidence layer that could disprove it, and verify the rollback path before anyone builds the treatment. That single discipline is a practical starting point for a program that can keep learning as your product and organization change.

    References

  • How to Design a Dependable CLI Agent Users Can Trust

    How to Design a Dependable CLI Agent Users Can Trust

    Your CLI agent can look impressive in a controlled demo and still feel unsafe in a real repository. The moment it can edit files, invoke tools, or use credentials, users need to understand what it will do before they let it proceed.

    The dependable design is rarely the one with the most capabilities. It is the one with the smallest clear promise, predictable execution, visible controls, and evidence that it succeeds repeatedly.

    Define the boundary before you define the features

    Start by writing an operating contract for the agent. This is a product decision, not a prompt-writing exercise. A useful contract answers five questions:

    • What job does the agent complete?
    • Which resources and tools may it use?
    • What must it never do?
    • Which actions require explicit approval?
    • What observable result counts as success?

    Keep the job narrow enough to explain in one sentence. If the description needs a collection of exceptions, the interface is already carrying too much ambiguity. Split the work into a clearly named subcommand or make the advanced behavior opt-in.

    Treat every flag, tool, and permission as an increase in blast radius. A new option does not merely add flexibility. It creates another state the agent can misunderstand, another path you must test, and another behavior the user must learn. Reducing the surface area can improve repeatability and trust because both the agent and the user have fewer possible paths to reason about.

    When reviewing a proposed capability, ask whether it makes the mental model smaller. If it does not, remove it, defer it, or isolate it behind progressive disclosure. Safe, fast defaults should handle the common case without demanding that a new user understand the entire system.

    Design one boring, observable execution path

    A dependable run should feel like a transaction with recognizable stages. The model can help interpret intent, but it should not invent the execution contract as it goes.

    • Capture intent: Ask only for information required to resolve the task. If a missing choice would materially change the result, stop and ask.
    • Retrieve context: Fetch the smallest relevant set of files, facts, or records. More context can introduce conflicting instructions and distract the agent from the requested change.
    • Show the plan: Present a compact description of the intended actions, affected targets, and likely side effects.
    • Preview when useful: Provide a dry run for operations whose effects the user should inspect before execution.
    • Execute through narrow tools: Give each tool a deterministic input and output contract. Reject malformed responses instead of guessing what they meant.
    • Verify the result: Check the resulting state and tell the user what changed, what did not, and whether any step failed.

    The agent should stop when the requested scope changes, required context is unavailable, or a tool returns an unexpected result. A visible stop is easier to recover from than confident improvisation.

    Favor idempotent operations wherever you can. Repeating an idempotent action produces the intended state without duplicating or compounding its effects. That property matters in a CLI because interrupted runs and retries are normal operating conditions. Test the second run as deliberately as the first.

    Put human control at the blast-radius boundary

    Do not ask for approval at every step. Constant prompts train users to approve without reading. Place confirmation gates where the consequence or scope changes.

    • Read-only work: Make inspection and planning the default where possible.
    • Scoped writes: Request access only to the specific project, service, or resource needed for the task.
    • Destructive actions: Require a separate confirmation that names the target and explains the consequence.
    • Credentials: Use narrowly scoped, time-bounded access rather than broad credentials that persist beyond the run.
    • Expanded capability: Let users opt into advanced tools instead of quietly enabling them for every session.

    A confirmation message should help the user make a decision. Replace a generic question such as “Continue?” with a concrete statement of what will be changed and whether it can be undone.

    Reversibility should shape the underlying implementation as well. Prefer changes that can be represented as a patch, show the proposed difference before applying it, and preserve enough information to explain how to undo the operation. When reversal is impossible, make that fact visible before execution.

    Use a simple review question for each workflow: can a user predict the maximum consequence of saying yes? If the answer is unclear, the permission boundary is too broad or the confirmation arrives too late.

    Prove reliability before expanding the roadmap

    Do not use capability count as the measure of progress. Before adding a feature, define the task it should complete, the success threshold it must meet, and the smallest interface needed to test it. This turns roadmap discussions into observable product decisions.

    Evaluate at least three outcomes: task completion, time to first successful result, and stability when the same operation is run again. A capability that succeeds once but behaves differently on a retry is not ready merely because the first demonstration worked.

    Instrument each run with Agent Analytics. Capture the input, tools selected, duration, outcome, and error pattern. Review those signals to find where the agent asks unnecessary questions, repeats tool calls, loses users, or encounters the same failure. The response may be a smaller prompt, a tighter tool contract, a safer default, or the removal of a confusing option.

    Documentation belongs in this reliability loop. Keep runnable examples alongside the code and make them reflect the golden path. Treat any mismatch between documented behavior and actual behavior as a product defect. If the workflow cannot be explained and demonstrated simply, it is not yet a dependable workflow.

    Use these evaluations as promotion gates. Add power only after the current path is measurable, understandable, and stable. That discipline earns you the right to expand without turning the CLI into a collection of loosely related agent behaviors.

    Key takeaways

    • Write the agent’s operating contract before choosing its tools or refining its prompt.
    • Keep the default workflow narrow, safe, fast, and explainable in one sentence.
    • Retrieve minimal context, show a compact plan, execute through deterministic contracts, and verify the result.
    • Place explicit approval at destructive, irreversible, or scope-expanding boundaries.
    • Measure completion, time to first success, and rerun stability before adding another capability.
    • Use run telemetry and executable documentation to decide what to simplify next.

    Choose one golden-path task and write its operating contract now. Then run it twice: once normally and once as a retry. Every surprise you find is a reliability requirement to resolve before you broaden the agent’s reach.

    References

  • Supercharge Core Web Vitals with Amplitude’s Global Agent: Faster Rankings, Happier Users

    Supercharge Core Web Vitals with Amplitude’s Global Agent: Faster Rankings, Happier Users

    I measure product health by a simple equation: speed plus clarity equals trust. That’s why I prioritize Core Web Vitals and search performance together—because the fastest path to better UX and higher rankings is a closed loop between measurement, diagnosis, and action. Standardizing on Amplitude’s Global Agent with Amplitude AI Agents let my teams compress that loop from weeks to hours, and in many cases, to minutes.

    Learn how to track your web vitals and page rankings faster with Amplitude AI Agents and improve your site’s user experience and SEO rankings. That goal sounds ambitious, but with the right instrumentation and analytics workflow, it becomes a repeatable operating rhythm rather than a one-off project.

    Here’s what changed for us with Amplitude’s Global Agent: a single, consistent way to capture performance signals across pages and journeys, unified context for every session, and a lightweight footprint that doesn’t get in the way of speed. By centralizing measurement, we eliminated blind spots and gave product, growth, and engineering one shared truth for Core Web Vitals and behavioral analytics.

    My practical playbook is straightforward: 1) Establish a performance baseline for Core Web Vitals on key templates and critical user paths. 2) Segment results by device, location, acquisition channel, and content type to surface where users actually feel the friction. 3) Connect those vitals to downstream behaviors—scroll depth, engagement, and conversion—so we prioritize fixes that move business outcomes, not just lab scores. 4) Use feature flags and A/B testing to ship improvements safely and quantify uplift. 5) Close the loop with Agent Analytics to keep learnings visible and actionable.

    Operationally, we rely on anomaly detection to flag regressions early, CI/CD guardrails to prevent performance slips at deploy time, and observability plus session replay to accelerate root-cause analysis. This combination reduces mean time to resolution, protects page experience during fast iteration cycles, and helps us avoid trading UX for speed—or vice versa.

    The strategic benefit is compounding: better Core Web Vitals improve user perception and increase engagement, which strengthens SEO signals and, ultimately, page rankings. With a unified analytics platform in place, we can spotlight the few improvements that create outsized gains, then scale those patterns across the site with confidence.

    If your roadmap includes faster pages, stronger rankings, and happier users, align your teams around this simple loop: measure precisely, diagnose quickly, experiment safely, and learn continuously. Amplitude’s Global Agent and Amplitude AI Agents give you the instrumentation and insight to make that loop your competitive advantage.


    Inspired by this post on Amplitude – Best Practices.


    Book a consult png image
  • How to Operate Always-On AI Agents Without Losing Control

    How to Operate Always-On AI Agents Without Losing Control

    You want an AI agent to keep work moving after you close your laptop. The difficult part is not getting one successful overnight run. It is making the hundredth run predictable enough that you do not wake up to an embarrassing email, a corrupted task queue, or an unexplained usage bill.

    The right operating model looks less like a clever prompt and more like a small, well-managed operations team. Give each agent a narrow job, an inspectable queue, limited tools, a clear definition of done, and an explicit place to stop. That is how you gain useful autonomy without surrendering control.

    Start with a delegation contract, not a general-purpose assistant

    An always-on agent should not begin with a broad instruction such as “manage my sales work.” That leaves the model to decide what managing means, which systems it may change, and when it has enough evidence to act. The ambiguity is tolerable during an interactive session because you can correct it. It becomes operational risk when the agent runs unattended.

    Start by defining a job that produces a recognizable artifact. A sales-admin agent can prepare a briefing before a scheduled call and create proposed follow-up tasks afterward. A podcast-manager agent can assemble interview context, prepare a transcript-review document, and queue a reminder to share it. A coding-manager agent can review prior sessions and identify recurring mistakes. These are bounded responsibilities with visible outputs, not vague mandates to “help.” Three specialized agents handling podcast, sales, and coding workflows demonstrate how cleanly this pattern can separate unrelated work.

    Write the delegation contract in an identity file that the agent reads at the beginning of every run. It should answer seven questions:

    1. Who are you? Name the role, not the underlying model: sales admin, podcast manager, coding manager, or another function a person would recognize.
    2. What outcome do you own? Describe the recurring deliverable and the event that makes it useful.
    3. Where may you work? Name the exact task, output, and script folders the agent can use.
    4. What inputs may you trust? Identify the calendar, task file, transcript, session log, or other allowed input for the job.
    5. What may you change? Separate reading, drafting, creating internal files, updating tasks, and acting in external systems.
    6. What counts as complete? Specify the artifact, required fields, location, and status update expected at the end.
    7. When must you stop? Define what the agent should do when information conflicts, a tool fails, permission is missing, or the next step would affect another person.

    The last question matters most. A useful agent does not need permission to improvise its way through every obstacle. It needs a reliable way to say, “I could not complete this safely; here is the missing decision.” Treat a well-documented block as a successful operational outcome, not as agent failure.

    Keep consequential decisions outside the unattended role. The agent can prepare a customer email without sending it. It can propose changes to a deal record without changing the commercial commitment. It can summarize a coding pattern without modifying a production system. Moving from preparation to execution should be a deliberate permission decision, not an accidental side effect of adding another tool.

    Build an inspectable operating loop around four components

    The prompt is only one part of the system. Reliable agent operations need four components with distinct responsibilities: identity, scheduling, tasks, and scripts. Keeping them separate makes failures easier to locate and changes easier to review.

    Identity defines responsibility

    The identity file is the stable operating policy. It tells the agent what role it is playing, where its work lives, what it may do, and what completion looks like. Do not overload it with the details of one assignment. If the identity changes every time a task arrives, you no longer have a stable agent; you have an unreviewed prompt generator.

    The scheduler supplies a heartbeat

    The scheduler should wake the agent, point it to the correct identity and queue, and capture the result. It should not contain the business logic for podcast preparation or sales follow-up. That logic belongs in inspectable task instructions and small scripts.

    A Mac that remains online can use macOS LaunchAgents as this heartbeat. LaunchAgents run with the user’s permissions, which is operationally convenient but also defines the risk boundary: the agent may be able to reach anything the scheduled process and its tools can reach. Running scheduled agents on an always-on Mac Mini therefore makes permission design part of the architecture, not a setting to revisit later.

    Make the schedule explicit and easy to disable. Each job should have a known trigger, whether that is a recurring interval, a calendar-related event, or a periodic review. If you cannot quickly answer why an agent ran at a particular time, the scheduler is already too opaque.

    Tasks hold durable state

    Use a dedicated task folder for each agent. A Markdown file with frontmatter is enough to represent a work item while remaining readable by both a person and a tool. The frontmatter can hold machine-readable state; the body can hold the request, context, acceptance criteria, and eventual run notes.

    Choose a small lifecycle and apply it consistently. For example: queued, in progress, blocked, completed, and failed. The exact labels matter less than the transition rules:

    • A queued task is eligible to be claimed.
    • An in-progress task records which run claimed it, preventing another run from silently doing the same work.
    • A blocked task names the missing input or decision and preserves all useful partial work.
    • A completed task links to its output and records what changed.
    • A failed task records the failed operation and whether retrying it is safe.

    Give each recurring event a stable identifier. Before creating a meeting brief, transcript-review document, or follow-up task, the agent should check whether that event has already been processed. This idempotency check prevents a retry or overlapping schedule from creating duplicates.

    Do not treat chat history as the task database. Conversations are useful working context, but durable state belongs in a file or system you can inspect independently. Saving identities, task files, and scripts in a shared knowledge workspace such as Obsidian also makes the operating model portable across devices and coding assistants. Changing the model runner should not require rebuilding the job.

    Scripts expose narrow capabilities

    Scripts should perform small, deterministic operations: fetch an allowed input, create a document in a known location, normalize a transcript, or update a task field. Keep the judgement in the agent and the mechanics in scripts with explicit inputs and outputs.

    A small script is easier to inspect than a broad instruction to use the terminal however the model sees fit. It also gives you one place to add validation, duplicate checks, and error handling. When an agent repeatedly constructs the same command or edits the same file shape, promote that operation into a reviewed script rather than relying on the model to reproduce it perfectly on every run.

    Design the overnight failure path before the happy path

    Unattended automation changes the cost of a mistake. During an interactive session, a confusing output costs a correction. Overnight, the same confusion can trigger repeated work, alter several systems, or contact someone before you see it. Your design should limit the consequence of a wrong interpretation, not merely improve the probability of a correct one.

    Use a permission ladder

    Classify capabilities by consequence and grant them one level at a time:

    1. Read: inspect approved calendars, task files, transcripts, logs, or documents.
    2. Prepare: create drafts, summaries, reports, and proposed tasks inside a bounded workspace.
    3. Update: change internal records whose history can be inspected and reversed.
    4. Act externally: send messages, share files, update customer-facing systems, or invoke paid services.
    5. Perform destructive or privileged work: delete data, change access, alter infrastructure, or execute an irreversible operation.

    Most new agents should prove themselves at the read and prepare levels. Promotion should be capability-specific. An agent that reliably prepares a sales brief has not thereby earned permission to send customer communication. Reliability does not transfer automatically from one action class to another.

    For external actions, use a pending-approval state that contains the exact proposed action. You should be able to review the recipient, content, destination, and relevant context without reopening the entire run. Destructive or privileged actions should remain outside unattended execution unless you have an explicit recovery path and have deliberately accepted the consequence of failure.

    Treat external text as data, not authority

    Calendar descriptions, transcripts, web pages, emails, and documents may contain instructions that conflict with the agent’s job. The identity and task contract must outrank text found inside those inputs. An interview guest’s biography can inform a briefing; it cannot expand the podcast agent’s permissions. A meeting note can identify a follow-up; it cannot authorize the agent to send one.

    Keep credentials out of identity and task files. Give scripts access only to the credentials required for their operation, and avoid handing an agent a general browser, terminal, file system, and credential store merely because each tool is useful in isolation. The dangerous capability is often the combination.

    Make retries selective

    A retry is appropriate when the failure is plausibly temporary and repeating the operation is safe. A network timeout during a read may qualify. Ambiguous recipient identity, conflicting meeting details, missing share settings, or an unclear customer commitment do not. Retrying an ambiguity only asks the model to make the same unsupported decision again.

    Before enabling automatic retries, require the operation to pass three tests: it can detect whether it already succeeded, a duplicate would not create harm, and the number of attempts is capped. Otherwise, mark the task blocked and surface it for review.

    Put hard boundaries around usage

    Always-on does not mean continuously reasoning. It means the system is available to process eligible work on a known schedule. A run should inspect the queue, process a bounded amount of work, record its result, and exit.

    Set limits at several layers: eligible task types, work accepted per run, retries per task, tools available to the role, and provider-side spending or usage controls where available. Record usage beside the task outcome so you can distinguish an expensive valuable job from an agent that consumes resources while circling an ambiguity. Surprise charges are not only a pricing problem; they usually indicate that the operating loop lacks a stopping rule.

    Finally, maintain a kill switch you can use without asking the agent to cooperate. Disabling the schedule or revoking the narrow credential should stop future work. If stopping the system requires the same model and scripts that may be malfunctioning, it is not an independent control.

    Measure whether the agent is reducing work or relocating it

    A completed status is not proof of value. An agent can close every task while leaving you to verify facts, repair formatting, remove duplicates, and reconstruct why it made a decision. That is work relocation, not delegation.

    Evaluate the operation with measures tied to the job:

    • Usable completion rate: the share of eligible tasks that produce an output meeting the acceptance criteria without substantive rework.
    • Correction rate: how often you must change facts, recipients, permissions, status, or next steps before using the output.
    • Duplicate or false-action rate: how often the agent repeats a job or creates an action that the triggering event did not require.
    • Blocked rate by cause: which missing inputs, permissions, or unclear rules repeatedly prevent completion.
    • Time to review: the human attention required to approve, repair, or understand the result.
    • Usage per usable outcome: the model or service consumption attached to work you actually keep.

    These measures tell you what to change. A high blocked rate caused by missing context points to an input problem. Frequent factual corrections point to retrieval or acceptance-criteria problems. Duplicate work points to task identity and idempotency. High review time with otherwise correct output often means the evidence and change log are poorly presented.

    Require every run to leave a compact receipt: the task it claimed, inputs it used, scripts it invoked, files or records it changed, output location, completion status, and reason for any block. You should not need to replay hidden reasoning. You need enough evidence to verify the operation and diagnose the next failure.

    Review early runs closely and review again after changing an identity, script, tool, model, or input source. A stable task can become unstable when any one of those dependencies changes. Plain-text identities, tasks, and scripts make that change surface inspectable and versionable.

    Your agents can also improve the operating system itself. A periodic coding-manager workflow, for example, can review prior coding sessions, identify recurring dead ends, and propose changes in how future sessions are run. The important separation is that the agent proposes an improvement with evidence; the operating policy changes only after review. Self-observation is useful. Unreviewed self-modification is a different risk class.

    Expand only when the current job has earned more autonomy

    Adding agents is easy once the scheduler and folder structure exist. That convenience can tempt you to automate work whose boundaries are not ready. Scale based on operational evidence, not on the number of possible use cases you can imagine.

    A job is a strong candidate for always-on operation when it has a recurring trigger, stable inputs, an observable deliverable, clear acceptance criteria, bounded permissions, and enough repetition to justify maintaining the workflow. Preparation, follow-up capture, document setup, and periodic retrospectives fit because a person can inspect their artifacts and correct them before higher-consequence decisions are made.

    Keep work interactive when the task depends on novel judgement, unresolved organizational context, sensitive negotiation, or irreversible action. An agent may still prepare evidence and options, but the decision should remain with the person who owns the consequence.

    Before expanding an existing agent’s permissions or creating another role, check five gates:

    1. The current output is regularly usable without substantial reconstruction.
    2. Common failure modes are visible and end in safe states.
    3. Duplicate prevention and retry behavior have been exercised.
    4. Usage is attributable to tasks and bounded by stopping rules.
    5. The next capability has its own acceptance criteria and consequence review.

    Do not create one agent per application. Create one per coherent responsibility. A podcast manager may use a calendar, a document system, and a task list while retaining one outcome. Conversely, sales administration and coding retrospectives should not share an identity merely because they use the same model. Role boundaries should follow accountability, not tooling.

    Key takeaways

    • Begin with one recurring job that produces an inspectable artifact, not a general instruction to manage a function.
    • Give the agent a durable identity, a dedicated task queue, an explicit schedule, and small reviewed scripts.
    • Use task states, stable event identifiers, and completion receipts so retries and overlapping runs do not create invisible duplication.
    • Keep new agents at read-and-prepare permissions until their outputs and failure modes are consistently understandable.
    • Route ambiguity and consequential external actions to approval instead of asking the model to guess.
    • Cap eligible work, retries, tools, and usage; always-on availability should still produce finite runs.
    • Measure usable outcomes, corrections, blocks, duplicates, review effort, and usage before granting more autonomy.

    Pick one task you already repeat and write its delegation contract before choosing more tools. If you cannot define the input, output, permission boundary, completion test, and safe stopping condition on one page, the job is not ready to run while you are offline. Tighten the job first. The agent can earn broader responsibility after the operating evidence is there.

    References

  • Built for Your Biggest Days: How We Engineer Fair, Reliable Scale Without Downtime

    I’m getting sharper, more specific questions about scale from enterprise customers every quarter, and that’s exactly how it should be. Teams want to know how our platform behaves during their highest-volume moments — the Black Friday sales, the sporting events, the production incidents — and they want confidence their growth won’t outpace the systems they depend on. We welcome those questions. They’re the right ones to ask of any critical component of your business. Today, our systems handle serious scale. At daily peak, we see over 150,000 customer requests per second coming into the platform, with more than 70,000 asynchronous requests per second flowing through the background systems. During our busiest days of the week, we handle over five million conversations and more than 100 million comments being added across the platform. We also design for individual customer spikes, not just aggregate platform traffic. We can handle a single customer workspace spiking with hundreds of comments per second, or around 100 new conversations per second. Sustained over a full day, that would map to millions of conversations from a single customer. While those numbers matter, they age quickly. Every growing software company can publish a bigger number every year, month, week. What ultimately matters is whether the architecture has clear scaling levers, whether we understand the pressure points in the system, and whether we can add capacity before customers need it. Every system has limits. Competence is knowing where they are, measuring them, and moving them before customers reach them. Here’s how we do that in practice. We build on boring foundations because at the edges, we try hard not to be clever. We use AWS for the infrastructure primitives AWS is very good at running. We do not want our engineers spending their best energy recreating S3, load balancers, queues, or commodity infrastructure patterns. We want that energy spent on the parts of the system that are specific to our customers and our product. “That is a deliberate trade-off. It gives us fewer systems to understand, deeper expertise in the ones we do run, and more leverage when we need to scale.” This extends a principle I’ve embraced for years: run less software. The point isn’t to minimize the stack for its own sake; it’s to compound expertise. When many teams build on the same small set of technologies, our tooling, observability, and operational practice all improve together. Boring technology choices aren’t a lack of ambition — they reserve our ambition for the nuanced scaling challenges that matter. The source of truth is the hard part. You can scale stateless web traffic by adding machines, add queue consumers, and add cache. Those are real problems — just not the hardest ones. The source-of-truth database is where the most important data lives, where the hardest correctness guarantees exist, and where maintenance windows often come from. It has to be correct, fast, resilient to failover, capable of large migrations, and able to keep serving traffic while we improve it. As customers grow, it cannot require a full re-architecture every time the next ceiling appears. That is why we moved to Vitess, managed by PlanetScale. The goals were clear: improve availability, reduce operational complexity, make large table migrations safer, simplify MySQL scaling, and eliminate customer downtime from routine database maintenance and failovers. When we first laid out this direction, the largest part of the migration was still ahead of us. We completed that migration in 2025, and the benefits are now part of how we operate the platform day to day. Today, our highest-scale source-of-truth data is spread across 128 shards. The database layer handles around two million requests per second, with more than ten million cache reads per second in front of it. For the largest customers, we can isolate and scale database capacity independently, including dedicating a shard to a single customer when needed. We have not come close to needing that, which is significant. The goal of architecture like this is not to run every system at the edge of its capacity, but rather to have room to move before customers need it. Vitess gives us native sharding, query routing, online schema change capabilities, connection pooling, and resharding primitives built for this kind of workload. Instead of application code carrying all of the sharding complexity, the database layer can do more of the work. That reduces cognitive load for engineers and removes whole classes of operational risk. Ultimately, this gives us practical scaling options instead of hard architectural rewrites, and lets us do routine database improvement without planned customer-impacting maintenance windows. Search is not a hidden bottleneck for us. Search underpins core product surfaces across the platform — from vector search in our AI features to realtime reporting — and if it’s slow or unhealthy, customers feel it. Scaling isn’t just adding more machines; often the better approach is making the product do less unnecessary work. Today, our Elasticsearch clusters support a much higher-throughput product than in the past, with more than 650TB of storage, more than 1.7 trillion documents, and peaks above 40,000 requests per second. We’re serving a larger product surface more efficiently, not just running a bigger cluster. More importantly, when an index gets too large or traffic distribution turns unhealthy, we don’t want a high-risk, manual migration. We reshape Elasticsearch indexes online by partitioning by customer ID, dual-writing to old and new indexes, backfilling, validating, gradually moving customers with feature flags, and deleting the old index only when we’re confident. We’ve used this pattern for years to make large search migrations safer and more incremental — a core playbook in our platform scalability and SRE practices. Fairness is non-negotiable in a multi-tenant system. A single customer’s high-volume moment should not quietly become everyone else’s latency problem. We design for this at multiple layers. For asynchronous work, we use overflow queues and queueing strategies that prevent one high-volume workload from consuming shared capacity in a way that hurts quieter tenants. AWS SQS fair queues are one example of a primitive we use extensively. They’re designed for exactly this class of problem. When one tenant creates a backlog in a shared queue, fair queues help reduce the dwell-time impact on other tenants. We also build application-level guardrails so customer isolation doesn’t depend on every engineer remembering every rule in every code path. In a large multi-tenant Rails application, the safe path must be built into the system. The focus is primarily about correctness and customer data separation, but the broader operating principle is the same: important customer boundaries should be enforced by infrastructure and application frameworks. The same thinking applies to scale. We want customer-specific load to be visible, attributable, and controlled. When a customer spike happens, we should be able to understand it as that customer’s workload, protect the rest of the platform, and add capacity where it’s actually needed. Fin adds a new dimension to scaling. Our AI Agent Fin introduces a new set of infrastructure challenges. To provide reliable AI-powered support at scale, we need to operate across multiple model providers, route across them based on capacity and latency, and protect customer-facing workloads from lower-priority work. The details differ from traditional SaaS infrastructure, but the principle is the same: understand the bottlenecks, build clear scaling levers, and monitor the customer outcome. AI providers are not commodity storage systems, and we do not design as if they are. That is why we have invested in Fin-specific reliability systems. Fin now fully resolves over two million conversations per week. At that scale, high availability cannot depend on a single model, a single provider, a single region, or a single pool of capacity. Our LLM routing layer supports cross-vendor failover, cross-model failover, latency-based routing, capacity isolation, and load testing. We also maintain buffer capacity with major providers, with headroom to handle 2x to 3x normal Fin traffic at any point. For enterprise customers, this matters because AI support volume can spike just like human support volume — and the AI layer must absorb that spike without relying on one fragile upstream path. When customers depend on Fin to absorb a spike in support demand, the AI layer needs the same operational discipline as the rest of the platform. Performance tests help, but production traffic is reality. Real customers use products in ways no synthetic test will perfectly predict: launches, incidents, seasonal patterns, gaming events, sudden changes in end-user behavior. Those moments give us data that no lab can fully reproduce. Often, a large customer event barely moves the platform-wide graphs because our customer base is broad enough that one industry’s peak aligns with another’s quiet period. Black Friday and Cyber Monday are good examples. Many ecommerce customers are at their busiest, while many B2B SaaS customers are quieter. At the aggregate platform level, the change can be much less dramatic than people expect. “That does not mean those events are unimportant. It means we need to look at both levels: the health of the overall platform and the experience of the individual customer having the spike.” Sometimes, these events teach us something specific. In one case, a very large customer used the Messenger in a way that exercised the full Messenger lifecycle even though the visible user experience did not require it. Under normal traffic, this was fine. During a major customer-side incident, their users refreshed aggressively, generating a much larger burst of Messenger traffic than the integration actually needed. The platform stayed available, but the event exposed unnecessary work in that integration path. We built a lighter-weight integration path that served the customer’s actual use case with far less work per request, making future spikes easier to absorb. We treat large customer events this way even when there’s no broad customer impact. They’re opportunities to understand real scaling properties and make the next event safer — a habit that anchors our incident management, observability, and FinOps practices. Scale is also an operating model. The infrastructure matters, but it’s not enough. You can have the right database architecture and still hurt customers if you detect issues late, recover slowly, communicate poorly, or fail to learn from incidents. “That is why our operating model starts with customer outcomes. If the customer cannot do the job they came to do, the system is unhealthy. It does not matter how many dashboards are green.” Heartbeat metrics tell us whether customers can do the core jobs they hire us to do. They cut through infrastructure noise and answer the question that matters most during an incident: are customers able to use the product successfully? This shapes how we ship. Today, we average around 250 ships to production per workday, with an average merge-to-production time under 10 minutes. That isn’t a vanity metric — it’s part of the safety model. Smaller changes are easier to understand, easier to observe, and easier to roll back. Feature flags let us separate deployment from release. Automatic rollback and heartbeat-driven detection help us recover quickly when a change hurts customers. These are the very DORA metrics we hold ourselves to in order to balance CI/CD speed with stability. “Fast shipping is not the opposite of reliability. Done properly, it is one of the ways you stay in control of change.” The bar is high. Engineers are expected to understand the impact of their changes, watch them go live, and act quickly if something looks wrong. Resuming service is not the end of an incident. We expect teams to understand the root cause, fix the contributing systems, and prevent recurrence. That’s how scale stays safe over time. Scheduled maintenance should be extraordinary. Historically, database maintenance was a main reason for maintenance windows: upgrading a database, changing instance sizes, performing failovers, or moving large tables could require customer-impacting downtime. With the move to Vitess and PlanetScale, we changed what routine database improvement looks like. We can upgrade, scale, and improve critical database infrastructure without turning that work into planned customer-impacting downtime — and we do this in practice, not just as a goal. This matters because customers rely on our platform for live operations. If their support team, Messenger, Help Desk, or AI Agent is unavailable, the impact is immediate. Scheduled maintenance cannot be treated as a casual operational convenience. “Our posture is simple: routine infrastructure improvement should not require planned customer-impacting downtime.” Scheduled maintenance should be exceptional, non-routine, clearly communicated, and minimized in frequency, duration, and customer impact. That’s the practical benefit of the architecture work: better scaling is not only about handling more traffic, but also reducing the operational moments that might inconvenience customers. What this means for customers is simple: be skeptical of vague scale claims. The question isn’t whether a vendor says they can scale — it’s whether they can explain how, where the limits are, what they measure, how they recover, and what they’ve changed after learning from production. We understand the scaling properties of our systems, have clear levers to add capacity at the right layers, design for customer isolation and fairness, monitor customer outcomes directly, and use real production events to make the next one safer. Scale is never finished. Every large customer event, traffic spike, migration, and incident teaches us something about the real behavior of the system — and we use that data to keep improving. That’s what you should expect from a platform you depend on during your busiest moments.

    Inspired by this post on The Intercom Blog.


    Book a consult png image
  • How to Scale Session Replay Without Sacrificing Privacy

    How to Scale Session Replay Without Sacrificing Privacy

    You want session replay on more journeys because the blind spots are expensive. A funnel can show where users leave, but it cannot show whether they encountered a broken control, a confusing message, a layout shift, or an error that never reached your analytics. Replay can turn those behavioral signals into enough context to make a product decision.

    The hard part is expanding that visibility without collecting data you should not have, degrading the experience you are trying to understand, or filling storage with recordings nobody will use. The answer is not a single masking setting. You need a capture contract, a delivery architecture, a sampling model, and an operating scorecard that treat performance, fidelity, and privacy as one system.

    Set the capture contract before you expand coverage

    Replay programs often begin with a coverage question: what percentage of sessions should you record? That is the wrong first question. Start with the decision you expect the recording to change. If nobody can name that decision, more coverage will create more cost and exposure without producing more insight.

    Write a capture contract for each product surface. This is a short, reviewable specification that connects a business purpose to technical controls. It should answer:

    • What question is replay meant to answer? Examples include diagnosing failed activation, explaining an error spike, or finding friction in a conversion step.
    • Which routes, components, and user cohorts are in scope? Name them. Do not approve an undefined all-product rollout.
    • Which data is prohibited? Include form values, credentials, payment details, message content, health information, account-recovery data, and any product-specific sensitive fields that apply.
    • What consent state permits capture? The recorder should not initialize before the required state is known. Withdrawal should stop capture and prevent queued data from being sent.
    • Who can watch a replay? Define roles by purpose. Product discovery, support investigation, engineering diagnosis, and administration do not automatically require identical access.
    • How long will the data remain available? Tie retention to the stated purpose rather than keeping replay indefinitely because storage permits it.
    • What sampling rule applies? State the baseline rate, targeted cohorts, exclusions, temporary overrides, owner, and expiry condition.

    Selective capture, redaction, consent, retention, role-based access, and environment-aware sampling are separate controls. Treating one of them as a substitute for the others creates predictable gaps. Masking does not grant consent. Restricted access does not make excessive collection necessary. Short retention does not make an exposed credential harmless.

    Apply those controls as close to collection as possible. A web replay is commonly reconstructed from serialized page state, changes, and interaction events. The privacy risk therefore sits in the data leaving the browser, not only in what the player later displays. A value hidden during playback may already exist in an outbound payload or stored record.

    A useful default is to block text and input values, then allowlist only fields proven safe and necessary. Add route-level and component-level exclusions for sensitive surfaces. Use a separate, time-bounded approval for diagnostic capture that needs greater fidelity. I would reject a policy that merely says to mask personal information: the term depends on context, and engineers cannot reliably implement an undefined category.

    Test the contract against the raw system, not just the player. Seed a non-production fixture page with recognizable test values, exercise every relevant component state, inspect the browser payload, inspect the stored representation, and verify that exports and downstream tools preserve the restriction. If a prohibited test value crosses the collection boundary, the control has failed even if the replay screen obscures it.

    Consent and retention obligations vary by jurisdiction, contract, and data type. Your privacy or legal owner must approve those rules for the markets you serve. Engineering can enforce an approved policy; it cannot infer that policy from a generic replay configuration.

    Keep capture off the user’s critical path

    Scalable replay starts in the browser, where your product competes with the recorder for main-thread time, memory, and bandwidth. A backend that can ingest billions of events does not help if the recorder makes an interaction sluggish or loses the DOM changes needed to explain the problem.

    The delivery design should make page experience more important than recording completeness. Decoupled capture and delivery, adaptive batching, compression, backpressure controls, and priority handling provide the basic pattern:

    1. Capture the minimum useful representation. Filter excluded nodes and values before serialization. Avoid collecting detail that no approved use case needs.
    2. Separate recording from transport. The capture path should write to a bounded queue rather than waiting for a network request. Upload latency must not become interaction latency.
    3. Batch adaptively. Small batches can reduce delay during quiet periods, while larger compressed batches can reduce request overhead during sustained activity. The policy should respond to queue pressure and network conditions.
    4. Define backpressure behavior. When production exceeds delivery capacity, the recorder needs a documented degradation order. Preserve navigation, consent changes, errors, and the structural events required for reconstruction before lower-value detail. Never freeze the page to protect the replay.
    5. Bound long sessions. Flush incrementally, cap memory use, and make reconnection behavior explicit. A queue that grows for the life of a tab will eventually turn a delivery problem into a page-performance problem.
    6. Make partial data visible. Mark gaps, dropped segments, and incomplete uploads. A replay that silently appears complete is more dangerous than one that clearly communicates its limits.

    Backpressure deserves special attention because it forces a product decision disguised as an implementation detail. If the system cannot retain everything, what must survive? The answer should come from the capture contract. An error marker without enough surrounding state may be useless, but exhaustive cursor movement may be expendable. Rank event classes before an incident forces the recorder to choose implicitly.

    Do not validate the client only on a fast laptop and stable connection. Use representative complex pages and test replay on and off under CPU pressure, constrained networking, rapid DOM change, background-tab transitions, reconnection, and long sessions. Compare Web Vitals, long tasks, memory growth, bytes transferred, queue drops, upload completion, and playback completeness. Long sessions, traffic spikes, complex interactions, and variable networks are precisely where an apparently sound design reveals its failure modes.

    There is no universal acceptable overhead that fits every product. Set budgets relative to your production baseline and the importance of the journey. A small regression on a frequently used mobile activation path may matter more than a larger regression on an internal administration page. Segment the results by route, browser, device class, network condition, and session length so averages do not hide the users most affected.

    Sample for decisions, not for a warehouse of footage

    A single global sample rate is easy to configure and hard to defend. It spends collection capacity uniformly even though product questions are not uniformly valuable. It can also miss rare failures while overrepresenting routine sessions that nobody will watch.

    Use a portfolio of sampling modes:

    • Random baseline sampling gives you a less biased view of ordinary behavior and lets you notice problems you did not predefine.
    • Cohort sampling increases visibility for a defined population such as new users, a browser family, a release cohort, or users entering a critical journey.
    • Signal-based sampling concentrates diagnosis around errors, failed steps, rage clicks, dead clicks, abnormal exits, or other instrumented friction signals.
    • Temporary diagnostic sampling raises fidelity for a narrow incident or release window, with an owner and an automatic expiry condition.
    • Hard exclusions override every sampling mode. A high-value investigation is not permission to collect from a prohibited surface or consent state.

    Onboarding, activation, high-friction conversion flows, and paths with disproportionate revenue or trust impact are sensible places to begin because a clearer diagnosis can change a meaningful decision. Signals such as errors, rage clicks, dead clicks, scroll behavior, and stalled progress can then help you find the sessions worth examining.

    Keep one statistical distinction clear. Targeted replay is good for explaining a known problem, but it cannot tell you how prevalent that problem is. If you record sessions because they contain an error, the resulting library will naturally make errors look common. Use analytics or a random baseline to measure frequency. Use replay to understand mechanism and context.

    A disciplined investigation looks like this:

    1. Find a measurable change in a funnel, cohort, error rate, performance signal, or support pattern.
    2. Define the affected population before opening replays.
    3. Review a deliberately selected set of relevant sessions and record recurring observable behaviors, not interpretations of user intent.
    4. Turn those observations into a falsifiable product or technical hypothesis.
    5. Instrument, release, or experiment so the hypothesis can be measured outside the replay player.

    This prevents two common mistakes: browsing memorable sessions until a story feels true, and treating one vivid recording as evidence of market-wide demand. Replay is strongest when it explains a quantitative signal and leads back to a measurable change.

    Run replay with a coupled performance, privacy, and value scorecard

    Session replay is not finished when playback works. It is an operating capability with client releases, configuration changes, storage growth, access decisions, and incident risk. Give it an owner and review the system across five dimensions.

    DimensionSignals to watchDecision the signals should trigger
    User experienceWeb Vitals, long tasks, main-thread work, memory growth, and replay bytesReduce capture detail, change delivery behavior, narrow coverage, or halt a rollout when the recorder breaks its budget
    Replay fidelityQueue drops, missing segments, incomplete uploads, event integrity, and playback reconstruction errorsFix prioritization or transport before teams rely on incomplete recordings for decisions
    Platform reliabilityIngestion failures, processing delay, retrieval latency, playback-start failures, and behavior during traffic spikesAdd capacity, repair a failing stage, or adjust sampling without shifting the problem into the browser
    Privacy and governanceRedaction test failures, capture outside approved consent states, retention exceptions, and access outside approved rolesDisable affected capture, contain the data, follow the approved deletion or incident process, and repair the control before restoring it
    Decision valueInvestigations that reached a useful replay, time to diagnosis, time to resolution, and product hypotheses validated outside replayMove coverage toward high-value use cases or retire collection that produces no action

    These dimensions constrain each other. Aggressive compression may improve bandwidth while hurting reconstruction. More capture may improve fidelity while violating the page budget. Narrow access may improve governance while blocking the support engineers responsible for incident response. The job is not to maximize any single metric; it is to keep the entire system inside approved boundaries.

    Version capture configuration like production code. A seemingly harmless selector change can expose text, remove necessary context, or increase mutation volume. Test recorder and configuration releases against fixture pages containing known sensitive values and known reconstructable interactions. Keep a rollback path.

    Prepare shutdown controls before launch. You should be able to stop capture for a component, route, environment, tenant group, or the whole product without waiting for a new application release. Document who can use each control, how queued data is handled, how affected stored data is identified, and when privacy, security, support, and engineering must be involved. If collection crosses a prohibited boundary, continuing to record while the team debates ownership compounds the exposure.

    Finally, connect replay operations to the workflows that consume it. Product teams need links from behavioral cohorts to relevant sessions. Support needs controlled escalation paths. Engineering and SRE need errors, network signals, layout shifts, and performance context close to the replay timeline. Connecting interaction context to observability and delivery workflows can shorten the path from an anomaly to a testable explanation, but only if the data remains trustworthy and accessible to the right roles.

    Key takeaways

    • Approve a capture contract for each surface before approving a broader sample rate.
    • Redact or exclude sensitive data before it leaves the browser; a masked player is not enough.
    • Protect the page with decoupled delivery, bounded queues, adaptive batching, and explicit backpressure priorities.
    • Keep random sampling for prevalence and use targeted sampling to explain known signals.
    • Operate performance, fidelity, platform reliability, privacy, and decision value as a coupled scorecard.
    • Require scoped shutdown controls, retention handling, access ownership, and rollback before production expansion.

    Before you increase replay coverage, ask for two artifacts: a one-page capture contract for the next journey and a replay-on versus replay-off test under that journey’s difficult conditions. If the team cannot show what is allowed to leave the browser, how the page stays within budget, and which decision the recordings will change, the rollout is not ready to scale.

    References

  • Developer-First Amplitude Instrumentation You Can Trust

    Developer-First Amplitude Instrumentation You Can Trust

    Your Amplitude dashboard is populated, but the room still debates whether the numbers are real. Engineering sees successful requests. Product sees unexplained breaks. Each feature adds more events, yet confidence in the data keeps falling.

    You do not fix this by collecting more data or polishing the dashboard. You fix it by treating instrumentation as a product interface: designed around a decision, expressed as a clear contract, reviewed with the code, tested against real journeys, and monitored after release.

    Design the decision before you name the event

    The most common instrumentation failure starts before an engineer writes code. A stakeholder asks to track a page, button, or feature without saying what decision the data must support. The resulting event may be technically valid and still be useless.

    Begin with a decision statement: If this behavior differs by this segment or step, I will change this part of the product. That sentence forces you to identify the behavior, comparison, and possible action. If nobody can describe the action, the proposed event is probably speculative inventory rather than decision-grade data.

    Suppose you need to decide whether team invitations are blocking activation. A useful behavioral sequence might contain Workspace Created, Invitation Sent, Teammate Joined, and First Shared Action Completed. The important work is not typing those labels. It is defining what each one means.

    • Does Invitation Sent fire when someone clicks the button, when the request succeeds, or when the message is accepted for delivery?
    • Does Teammate Joined mean the invite was accepted, the new user signed in, or the user entered the intended workspace?
    • Can retries emit the same behavior more than once?
    • Can an existing user join through a path that bypasses the invitation flow?
    • Which actor owns the event: the inviter, the invitee, the workspace, or some combination?

    Those distinctions determine whether the funnel represents the customer journey or merely the user interface. A click is evidence of intent. A confirmed state change is evidence of completion. Track both only when you have a real use for both, and do not give them names that imply the same meaning.

    Use events for behaviors that happened and properties for the context needed to interpret them. If email and link invitations represent the same business action, use one Invitation Sent event with an invitation channel property. Split them into separate events only when their meanings, lifecycles, or downstream decisions genuinely differ.

    Before approving an event, require answers to five questions: Who will use it? What decision will it change? What exact condition emits it? What else could produce the same signal? What will you do if the result moves? This keeps the tracking plan small enough to govern and precise enough to trust.

    Turn the tracking plan into an executable contract

    A tracking spreadsheet is not a contract if the implementation can drift from it unnoticed. The definition must be specific enough for an engineer to implement, a reviewer to challenge, and an automated check to validate.

    Data quality has several independent layers. Structural validity asks whether the payload follows the expected schema. Semantic validity asks whether the event means what its name claims. Coverage asks whether every intended surface and journey emits it. Identity integrity asks whether behavior is attached to the right user, account, or workspace. Passing one layer does not prove the others.

    An event can therefore be perfectly formatted and analytically false. Invitation Sent with a valid channel property still misleads you if it fires before the backend confirms success. This is why human-readable names and strict schema validation are necessary controls, but not the whole quality system.

    Contract fieldWhat to specifyFailure it prevents
    Decision and metricThe product question, downstream measure, and action the signal can changeEvents collected without a defined use
    Canonical eventOne stable, human-readable name and any forbidden aliasesSeveral names for the same behavior
    Trigger and completion boundaryThe exact state transition, success condition, and behavior on failure or retryClicks or attempts being counted as completed outcomes
    Emitter and source of truthThe client, server, worker, or other component responsible for emissionDouble counting when multiple layers report the same action
    Actor and entityThe user, account, workspace, or object to which the behavior belongsMetrics grouped around the wrong unit of analysis
    Required propertiesNames, types, allowed values, null rules, and derivation logicBroken segments and silent type drift
    Identity behaviorExpected handling before sign-up, after login, after logout, and during account changesSplit histories, merged users, and misplaced account activity
    Environment and release contextHow production, test data, application versions, and relevant platforms are distinguishedTest traffic contaminating decisions or regressions being hidden in aggregates
    Owner and lifecycleThe accountable team, review status, downstream consumers, and deprecation pathOrphaned events that nobody can safely change or remove
    QA evidenceThe automated assertion, tested journey, sample payload, and production verificationApproval based only on code inspection

    Property rules deserve the same precision as event rules. Decide whether an absent value means unavailable, not applicable, or an instrumentation defect. Keep types stable. Define bounded values where the business vocabulary is bounded. Avoid using display copy as an analytical value because a harmless wording change can fragment the data.

    Treat a property type change, trigger change, or identity change as a breaking contract change. Adding a new optional property is usually less disruptive than changing what an existing field means. When meaning must change, introduce an explicit migration plan and identify which historical comparisons will no longer be valid.

    Identity needs its own test plan. Exercise an anonymous visit followed by registration, a returning-user login, logout on a shared device, switching between workspaces, and any cross-device journey you intend to analyze. Verify the resulting user and account histories instead of assuming the SDK calls produce the business behavior you want.

    Apply data minimization at the contract boundary. Every property should have a decision use, an owner, and an acceptable data classification. Do not collect free-form or sensitive values merely because they might become useful later. Preventing unnecessary capture is safer than trying to contain it after it has entered the analytics pipeline.

    Make the pull request your instrumentation quality gate

    Developer-first instrumentation does not mean product hands analytics to engineering and walks away. It means the analytics contract follows the same change-management path as the behavior it describes. The code, definition, tests, and review evidence move together.

    Amplitude’s Wizard CLI offers a one-command path to start instrumentation from the codebase. That removes first-mile setup friction, but generated changes are a starting point rather than an automatic quality certificate. The team still has to decide what should be measured and what each signal means.

    1. Start in a feature branch. Run the setup workflow there so configuration and instrumentation changes are visible before they reach the main branch.
    2. Update the analytics contract in the same pull request as the feature. A behavior change without its contract delta is incomplete; a contract change without its implementation is unverifiable.
    3. Review the emission boundary. Confirm that the event fires on the intended success condition, has one authoritative emitter, handles retries deliberately, and does not fire on rendering unless rendering is the behavior you mean to measure.
    4. Run structural checks in CI/CD. Validate canonical names, required properties, types, permitted values, environment configuration, and forbidden fields. Fail the build when a known contract is violated.
    5. Run behavioral tests around the analytics client. Exercise success, failure, cancellation, and retry paths, then assert which events should and should not be emitted. A negative assertion is often what catches inflated success metrics.
    6. Verify the journey in a non-production environment. Capture the observed sequence and payload, then compare them with the contract. Keep this traffic distinguishable from production behavior.
    7. Define the production check before merging. Name the owner, expected signal, dimensions to inspect, downstream chart or cohort affected, and response if the data does not match the release.

    Automated checks are strongest at detecting known structural failures. They can prove that a required field exists; they cannot decide whether the field represents the right business concept. Keep a lightweight semantic review in the pull request. Engineering should own trigger and runtime correctness. The product or analytics owner should own meaning and downstream use. Bring in privacy or security review when the identity model or captured data changes.

    The reviewer should be able to reconstruct the analytical meaning without reading every implementation detail. Include the decision statement, contract change, sample payload, tested journey, and affected measures in the pull request. That context preserves intent when the original team has moved on and makes later taxonomy changes auditable.

    Do not turn the gate into an analytics committee. Most changes need a clear owner and one qualified reviewer, not a meeting. Escalate when a change redefines a shared event, alters identity, introduces sensitive data, or breaks historical comparability. Routine additions that conform to the contract should remain routine.

    Prove production data is decision-grade, then keep proving it

    A successful deployment proves that code reached production. It does not prove that actual customers, application versions, queues, retries, and identity transitions produce trustworthy analysis. The final quality gate operates on observed production behavior.

    Inspect new or changed instrumentation by release, platform, environment, emitter, and relevant customer segment before relying on an aggregate. Aggregates can hide a missing platform, a version-specific regression, or duplicate client and server events.

    • Presence: Did the intended event appear after the release, and is an unexpected absence explained by traffic or by a defect?
    • Completeness: What share of observed events contains each required property, and where are missing values concentrated?
    • Conformance: Did new property values or types appear outside the agreed contract?
    • Uniqueness: Do retries, page transitions, or multiple emitters create suspicious duplicate patterns?
    • Sequence sanity: Can a completion event occur without the prerequisite behavior, and is that a legitimate alternate path?
    • Identity continuity: Do anonymous, authenticated, user, and account histories connect in the journeys that matter?
    • Comparability: Did the release change the meaning or population of an existing metric even though its name stayed the same?

    Set alert and acceptance thresholds from expected traffic, historical behavior, and the cost of a wrong decision. A universal percentage would create false precision. An event used for an executive activation metric deserves a tighter response than a diagnostic event used occasionally by one feature team.

    Give every important event a visible trust state. Proposed means the contract exists but the code does not. Instrumented means the code is deployed. Observed means production data has arrived and basic checks passed. Trusted means the owner has verified the real journey and approved downstream use. Deprecated means new analysis should stop depending on it. This vocabulary prevents a dashboard builder from treating mere event presence as approval.

    When production data is wrong, treat it like a data incident. Record the affected event, properties, segments, and time window. Identify the dashboards, experiments, and decisions that consume it. Stop or correct the bad emission. Backfill only when the intended values can be reconstructed deterministically from reliable records; otherwise, preserve the gap and mark the period as non-comparable. A plausible-looking repair is more dangerous than an explicit hole because it hides uncertainty.

    Add the failure mode to the contract test after the repair. If a retry caused duplicates, add a retry case. If one platform omitted a property, cover that platform. If identity changed during a workspace switch, turn that journey into a regression test. The incident should leave the instrumentation system harder to break in the same way.

    Govern by change triggers rather than recurring ceremony. Review instrumentation when a team launches a new journey, moves an event between client and server, changes identity behavior, modifies a shared taxonomy, adds a platform, or sees unexplained production drift. This focuses attention where meaning can change.

    The product payoff is not a larger event catalog. It is the ability to use clean behavioral signals for activation, onboarding, and retention analysis without reopening the instrumentation debate every time a result matters.

    Key takeaways

    • Start every event with a decision, observable behavior, and named owner. If the possible action is unknown, do not collect the event by default.
    • Define trigger, emitter, actor, properties, identity behavior, environment handling, and QA evidence as one versioned contract.
    • Ship the contract, implementation, automated checks, and journey evidence in the same pull request.
    • Separate structural validation from semantic review. A valid payload can still represent the wrong behavior.
    • Promote events from instrumented to trusted only after production verification, and mark damaged periods instead of silently presenting them as comparable.

    Use the next feature as the boundary for change. Pick one consequential customer journey, write its contract, put the instrumentation through the pull request, and verify it after release. Do not wait for a company-wide taxonomy rewrite. One fully governed journey will expose the missing standards and give you a working pattern for the next one.

    If your team cannot show the contract, test evidence, production check, and current trust state behind a metric, do not use that metric for a roadmap or growth decision yet. Label the uncertainty, repair the signal, and make trust part of the definition of done.

    References