Your AI roadmap is filling up, but every new use case seems to require another custom retrieval pipeline, another evaluation method, and another analytics implementation. The pilots may work. The portfolio does not yet compound.
You do not solve that problem by declaring an AI platform initiative and assembling a long infrastructure backlog. You solve it by identifying the decisions your products must support, defining the contracts every AI workflow must honor, and connecting evaluation to real product behavior. The result is a foundation that makes the next useful experiment easier to ship, safer to operate, and easier to learn from.
Make critical use cases define the platform
A platform team can spend months building abstractions that application teams neither understand nor need. The usual cause is starting with components: a model gateway, vector storage, a feature store, an evaluation tool, or a new analytics stack. Each may be useful, but none tells you what the platform must make easier.
Start with the highest-value AI decisions in your product and internal operations. A use case is specific enough when you can describe the person making the decision, the context the system requires, the action the AI is allowed to take, the unacceptable failure modes, and the outcome you expect to change.
Write a short capability brief for each critical use case:
User and moment: Who is trying to do what, and where does the AI enter the workflow?
Decision or action: Is the system recommending, drafting, classifying, retrieving, or acting autonomously?
Required context: Which behavioral events, account data, documents, permissions, and prior actions affect the answer?
Quality definition: What makes an output acceptable, and which errors matter most?
Release evidence: What must pass before the change reaches users?
Product outcome: Which user behavior or business result should move if the capability works?
Operating constraints: What must be logged, redacted, approved, monitored, or reversible?
Owner: Who owns the outcome after launch, rather than merely delivering the component?
Now compare the briefs. Repeated needs are platform candidates. If several use cases require permission-aware retrieval, consistent experiment assignment, traceable prompt versions, or the same release evaluation, those capabilities deserve shared interfaces. A requirement that appears once may belong inside the application until reuse is proven.
This distinction prevents two expensive mistakes. The first is premature generalization: building a universal service before you understand the variation. The second is copy-and-paste scaling: allowing every team to create incompatible versions of a capability that is already clearly common.
Prioritize the platform backlog by friction removed from critical use cases, not by architectural elegance. A useful backlog item should complete a sentence such as: After this ships, a product team can run the standard evaluation suite on every retrieval change without building its own runner. If the item cannot name the team behavior it changes, it is probably still an implementation idea rather than a platform outcome.
Build the foundation as four enforceable contracts
A foundational AI platform is not one product or one technical layer. It is a set of contracts connecting data, evaluation, delivery, and analytics. The contracts matter more than whether every component comes from the same vendor. They let application teams move independently without giving up consistency where consistency is valuable.
1. The data and context contract
This contract defines what context an AI workflow can request and what comes back. It should cover identity, permissions, event definitions, document metadata, freshness, provenance, and retention. Retrieval should enforce access rules before context reaches the model, not rely on the model to decide what a user may see.
Keep the interface narrow. An application should be able to request approved context for a known user, account, and task without understanding every underlying data system. The response should carry enough metadata to explain where the context came from and which version was used.
A feature store belongs here when several predictive or real-time workflows need the same feature definitions at training and serving time. It should not become a mandatory platform component merely because feature stores appear on AI architecture diagrams. Add one when inconsistency is a demonstrated problem.
2. The evaluation contract
This contract defines the evidence required to change a model, prompt, retrieval configuration, tool, or policy. It should include representative test cases, expected behavior, failure labels, scoring rules, comparison baselines, and release gates.
Do not reduce evaluation to one average score. A harmless wording variation and a permission leak cannot cancel each other out. Track important failure classes separately and make critical failures blocking. Keep examples that exposed production problems so the evaluation set becomes a memory of what the system has learned.
This contract governs how a tested change reaches production and how you recover when it behaves badly. It should preserve the versions of the model, prompt, retriever, tools, policies, and relevant data configuration associated with each release.
Use controlled rollout, feature flags, rollback paths, and CI/CD gates for AI changes just as you would for other consequential product changes. Observability should connect a production trace to the exact configuration that produced it. Otherwise, a team can detect a quality drop without being able to isolate whether the model, context, prompt, tool call, or upstream data changed.
Record enough to diagnose the workflow, but do not treat raw prompts and conversations as ordinary telemetry. Apply privacy-by-design: minimize collection, redact sensitive fields where appropriate, control access, and make retention a deliberate policy. The default path should be the governed path. If secure operation requires every application team to remember a separate checklist, the platform has not removed the risk.
4. The product analytics contract
This contract maps AI interactions to product behavior. Define events for exposure, acceptance, correction, rejection, fallback, task completion, and abandonment where those states apply. Include stable identifiers that connect an interaction to its release and experiment assignment without copying sensitive content into the analytics layer.
The contract should also specify which product metric each AI capability is expected to influence. That keeps teams from declaring success because response quality improved in an offline test while users ignore the feature or fail to complete the task.
Turn analytics into the AI learning and control loop
Traditional product analytics asks what users did. AI evaluation asks how a probabilistic workflow performed. You need both views joined at the interaction level. Without that connection, model teams optimize scores, product teams optimize funnels, and neither can explain why a release helped or hurt the user.
Use four layers of measurement, and do not blend them into one health score:
I’ve stepped into too many product reviews where teams argued over numbers that should have been obvious. Three names for the same “signup” event, properties scattered across tools, and no shared definitions—classic analytics chaos. As VP of Product Management at HighLevel, I’ve learned that scaling an analytics taxonomy isn’t just a data exercise; it’s a leadership mandate that unlocks decision velocity, alignment, and confident product bets.
Learn best practices our professional services team has compiled in helping customers move from scattered events to a scalable, user-friendly data structure.
Why does this matter so much? A robust taxonomy powers a unified analytics platform across Amplitude analytics, Pendo, and our CRM stack, reduces rework, and strengthens data governance. When events are clear and consistent, product-led growth accelerates: onboarding becomes measurable, activation is trackable, and retention analysis turns into a weekly ritual rather than a quarterly scramble.
I always start with outcomes, not events. We define a North Star metric and use driver trees to map how user behaviors ladder up to that outcome. Then we ground the plan in journey mapping: what signals mark activation, aha moments, and long-term engagement? This ensures our taxonomy mirrors real user intent, not just engineering convenience.
Next comes naming conventions and structure. We standardize on a readable, durable pattern (for example, actor_action_object), apply consistent property naming, and document required vs. optional properties. We version events deliberately, so we can evolve without breaking dashboards. Most importantly, we align events to product strategy—tracking less, but better.
Governance makes it scale. We establish a clear DRI for the tracking plan, a lightweight review process for changes, and a schema registry that serves as the single source of truth. Privacy-by-design is non-negotiable: we treat sensitive fields deliberately and audit access. Observability closes the loop—schema validations and alerts catch drift before it confuses teams.
Tooling and process turn good intentions into muscle memory. We keep the tracking plan “as code” in a repository, run CI/CD checks to validate events, and use feature flags to roll out new instrumentation safely. Pendo helps us annotate in-app experiences, while Amplitude provides the exploratory lens for cohorts, funnels, and retention. Together, these systems reduce guesswork and speed up discovery.
Migrations are where many teams stall, so I de-risk them with a clear, time-boxed plan. We audit the current event surface, map scattered events to the new taxonomy, and deprecate duplicates with guardrails. We communicate changes broadly, provide easy-to-scan documentation, and pair enablement sessions with hands-on examples from live dashboards. The goal is confidence, not just compliance.
We measure success like a product. Are we answering critical questions faster? Are duplicate events trending down? Are activation and retention questions easy to answer in under five minutes? When the taxonomy is working, stakeholders stop asking, “Do we trust this?” and start asking, “What should we build next?”
One of the most rewarding shifts I’ve seen: product trios moving from ad-hoc analyses to repeatable, weekly rituals. With crisp definitions, onboarding flows become testable, PLG motions are predictable, and leadership reviews focus on outcomes, not definitions. That’s the moment analytics transforms from a cost center into a growth engine.
If you’re staring at a wall of scattered events, start small: clarify outcomes, align your journey map, set conventions, and ship a minimum viable taxonomy to one critical flow. Iterate quickly. The compounding payoff—clarity, speed, and trust—will be obvious to every team you partner with.
When we do this well, analytics becomes a strategic asset. Our teams spend less time reconciling numbers and more time building what matters. That’s the real meaning of moving from chaos to clarity.
Inspired by this post on Amplitude – Best Practices.
You have AI pilots that demo well, enthusiastic teams asking for broader rollout, and executives expecting the investment to show up in operating results. Yet the closer you get to production, the longer the list of unresolved questions becomes: Who owns the workflow? How will quality be measured? What happens when the model is wrong? Can the economics survive real usage?
The next move is not to launch more pilots. It is to install a system that can repeatedly turn a validated use case into a governed, measurable, and improving production workflow. That system is what separates AI experimentation from mature deployment.
A successful pilot is not evidence of production readiness
AI adoption is already common enough that adoption itself tells you very little. Among more than 2,400 global customer service professionals, 82% of senior leaders invested in AI in 2025, 87% planned to invest in 2026, and only 10% described their deployment as mature. The sample is specific to customer service, so those figures are better used as a directional benchmark than as a universal maturity rate. The underlying execution problem applies much more broadly: buying or piloting AI is easier than making it dependable inside a core workflow.
A pilot is designed to answer a narrow learning question. Can the model classify this request, draft this response, summarize this record, or choose the next action under controlled conditions? Production has to answer a harder question: can the entire workflow create enough value, across ordinary and difficult cases, while remaining safe, observable, supportable, and economically sensible?
I use a simple test. If the team can describe the model but cannot describe the operating workflow around it, the work is still a prototype. A production case should make each of these elements explicit:
Outcome: The customer or business result that should improve, plus the current baseline.
Workflow boundary: Where AI enters, which decisions it may make, which systems it may use, and where its authority ends.
Quality standard: The evaluation cases, acceptance criteria, and failure categories that determine whether a release is good enough.
Safe failure path: What the system does when information is missing, a tool fails, a policy is triggered, or the requested action exceeds its authority.
Accountability: A named product owner for the outcome and a named operational owner for production performance.
Economics: The value created and the full cost of inference, retrieval, tools, review, support, and incident handling.
Learning mechanism: How production failures and user corrections return to the evaluation set and release process.
These are not finishing tasks to schedule after the model works. They are part of the product. Deferring them creates a predictable trap: the pilot looks increasingly impressive while the distance to a responsible launch quietly grows.
Do not confuse automation coverage with maturity, either. A system can handle many requests and still be immature if nobody can explain why it made a decision, detect a quality regression, contain a failure, or calculate the result. Conversely, a narrowly scoped workflow can be mature when its boundaries, controls, outcomes, and ownership are clear.
Promote each workflow through explicit maturity gates
Maturity should be earned workflow by workflow. An organization does not become mature because it has a central AI team, an approved model vendor, or a large portfolio. It becomes mature when important workflows can move through a repeatable sequence of decisions without relying on heroics.
Stage
Decision to make
Evidence required to advance
Reason to hold
Discover
Is this a valuable and appropriate problem for AI?
A defined user problem, current baseline, workflow map, risk classification, and initial build-versus-buy view
The use case is driven by model novelty, has no meaningful outcome, or depends on inaccessible data
Prove
Can the proposed workflow improve on the current process?
Representative evaluation cases, a working prototype, documented failure modes, and a controlled comparison with the baseline
Success appears only in curated demos, or the team cannot reproduce the result across realistic cases
Operate
Can the workflow run safely and reliably in production?
Monitoring, escalation, access controls, auditability, incident procedures, release controls, rollback, and an accountable operator
Failures cannot be detected or contained, or production responsibility is still ambiguous
Scale
Should usage, autonomy, channels, or organizational reach expand?
Sustained outcome improvement, acceptable quality and risk, validated economics, user adoption, and reusable operating components
Volume is growing faster than quality, cost, support capacity, or governance can be understood
The purpose of a gate is not to create a committee. It is to prevent enthusiasm, executive attention, or sunk cost from substituting for evidence. The domain team should be able to prepare the evidence as part of normal product development. Specialist review should become more demanding only as the possible consequence of failure increases.
Give every workflow a short deployment contract. Keep it in the same system where the team manages releases and evaluations, not in a presentation that disappears after approval. The contract should include:
The intended user, job to be done, business outcome, and current baseline.
The inputs the workflow accepts and the outputs or actions it may produce.
The actions that are prohibited, require confirmation, or must be routed to a person.
The data sources, retrieval rules, system permissions, retention rules, and privacy constraints.
The evaluation set, quality dimensions, acceptance criteria, and known limitations.
The failure taxonomy, escalation path, incident owner, and customer recovery procedure.
The prompt, model, retrieval, tool, and policy versions included in the release.
The production metrics, cost measures, rollout control, and rollback conditions.
The product owner, operational owner, and risk approvers.
The acceptance criteria will differ by workflow. A drafting assistant, an internal search experience, and an agent authorized to modify a customer account should not face the same bar. Base the bar on consequence, reversibility, detectability, and recovery. If an error can create an irreversible change, expose sensitive data, make a material commitment, or deny someone an important service, require an appropriate human authorization step rather than relying on average model performance.
The deployment contract also makes scope changes visible. Adding a new tool, data source, channel, language, model, or autonomous action is not merely more traffic. It changes the system’s failure surface. Update the contract, extend the evaluation set, and pass the relevant gate again.
Build three feedback loops before increasing autonomy
A mature deployment learns at three levels: whether the workflow creates value, whether its decisions meet the required standard, and whether the production system remains reliable. If any loop is missing, the team can collect impressive activity metrics while the actual product deteriorates.
Connect model behavior to a business outcome
Start with the baseline process, not an AI metric. If the workflow is intended to resolve a support request, qualify an opportunity, complete an onboarding step, or assist an employee, measure how that outcome happens without the new system. Otherwise, you will know that the AI generated output but not whether it improved anything.
Use a metric stack that separates outcomes from diagnostics:
Business outcome: The customer, revenue, cost, risk, or productivity result the investment is meant to change.
Workflow outcome: Completion, resolution, successful handoff, correction, rework, abandonment, or another measure of whether the task reached its intended end.
Quality and safety: Correctness, grounding, policy compliance, appropriate escalation, harmful failure, and user correction.
Economics: Cost per successful outcome, including model usage, infrastructure, external tools, human review, support, and remediation.
The layers diagnose different problems. A prompt change may improve an offline score without changing task completion. More automation may reduce handling work while increasing corrections. A cheaper model may lower inference cost but create enough rework to raise the cost per successful outcome. Do not compress those effects into one AI score.
When traffic and product conditions support an experiment, compare the AI workflow with the current experience. Define the decision metric and minimum detectable effect before running an A/B test. For lower-volume or higher-risk workflows, use controlled rollout evidence, expert review, and structured case analysis rather than pretending a small sample provides statistical certainty.
Turn evaluations into release criteria
An evaluation set is not a collection of attractive examples. It should represent ordinary work, difficult edge cases, policy boundaries, known failures, and the situations in which the system should refuse or escalate. Build it before optimizing the prompt so the team cannot unconsciously redefine success around whatever the prototype already does well.
For each case, record the expected behavior and why it is expected. Some outputs can be checked against a deterministic answer. Others need a rubric that distinguishes task completion, factual support, instruction following, tone, policy compliance, and escalation quality. Where reviewers can reasonably disagree, capture that disagreement instead of forcing false precision into a single label.
Use offline and online evaluation for different jobs. Offline evaluation protects releases by testing candidate changes against a stable set. Online evaluation reveals distribution shifts, new user behavior, integration failures, and outcomes that cannot be recreated fully before launch. Neither is sufficient on its own.
Version the entire behavior-producing system: model, prompt, retrieval configuration, knowledge snapshot, tools, policies, and routing logic. A model comparison is not meaningful if the surrounding system changed silently. For every proposed release, make the decision policy explicit: ship, hold, narrow the scope, expand gradually, or roll back. This is the practical core of eval-driven development with target metrics and a decision policy defined before launch.
Operate the workflow as a production service
AI introduces variable outputs, but it still depends on familiar production systems: identity, permissions, data pipelines, APIs, queues, search, external tools, and user interfaces. A model can appear to be wrong when retrieval returned stale information or a downstream tool rejected an action. Monitoring only the final text hides the failure that engineers need to fix.
Trace the workflow end to end. Subject to your privacy and retention rules, capture the release version, retrieval and tool events, policy decisions, response, escalation, user correction, and eventual workflow outcome. Monitor distributions and failure categories, not just averages. An acceptable overall score can conceal a serious regression for a particular intent, customer segment, channel, or action.
When the workflow depends on changing or private knowledge, connect it to governed retrieval instead of expecting the base model to contain the right answer. Use safe integration points for tools, least-privilege access, and explicit authorization for consequential actions. CI/CD, feature flags, canary releases, observability, audit trails, privacy controls, red teaming, and human review form a practical control plane for releasing changes without exposing the entire population at once.
Every material production failure should produce more than an incident ticket. Classify the failure, add or update the corresponding evaluation case, correct the prompt, retrieval, policy, tool, or interface responsible, and retest the workflow before restoring scope. That turns operational pain into a permanent improvement in the release system.
Use 30-60-90 days to build the scaling system
A useful 30-60-90-day sequence starts with two lighthouse use cases. The goal is not to force every use case into production within a quarter. It is to prove that your organization can move valuable workflows through the same gates, shared controls, and learning loops.
Days 0-30: narrow the portfolio and establish accountability
Inventory active pilots and classify each as discovery, proof, operation, or scale. Do not let a polished demo assign its own stage.
Select two lighthouse workflows using customer impact, feasibility, strategic relevance, and risk. Choose workflows meaningful enough to matter but bounded enough to operate responsibly.
Record the current process and baseline before the AI changes user or employee behavior.
Name the product owner, operational owner, and required risk decision-makers for each workflow.
Complete the first version of each deployment contract, including the autonomy boundary and safe failure path.
Make the build-versus-buy decision at the workflow level. Include data access, integration, auditability, evaluation portability, operating cost, and switching constraints.
Pause pilots that have no accountable owner, no measurable outcome, or no plausible route through the operating gate.
This first phase is where leadership earns focus. A broad AI mandate often creates a queue of unrelated prototypes, each with its own vendor, data assumptions, and definition of success. Choosing lighthouse workflows gives the platform and governance work a real customer instead of turning them into abstract architecture programs.
Days 31-60: install evaluation, controls, and workflow operations
Build the offline evaluation set from representative work, edge cases, policy boundaries, and failures already found during discovery.
Define acceptance criteria and the release decision policy before further prompt or model optimization.
Integrate the necessary retrieval and tools through governed access points. Keep permissions narrower than the user’s full access where the workflow does not need it.
Add observability across retrieval, reasoning inputs, tool execution, output, escalation, and business outcome.
Prepare feature flags, a controlled rollout, rollback, incident procedures, and a customer recovery path.
Run the workflow with appropriate human oversight. Record corrections and escalations as structured evidence, not informal feedback in chat.
Train the people who will supervise, support, and improve the workflow. Update operating procedures before transferring real responsibility to AI.
Training cannot be limited to prompt tips. Operators need to know what the system may do, how its failure modes appear, when to intervene, how to report a new failure, and who can change production behavior. Product and engineering teams need the same vocabulary for evaluation, incidents, and risk.
Days 61-90: expand evidence, not enthusiasm
Increase scope only for workflows that meet their operating gate. Expansion may mean more traffic, another intent, a new channel, or greater autonomy; evaluate each change explicitly.
Compare the production outcome and cost with the original baseline. Include corrections, review, support, and remediation in the economics.
Turn repeated needs into shared components such as model access, retrieval, identity, evaluation infrastructure, observability, policy enforcement, and audit logging.
Move validated production failures into the evaluation suite and confirm that the release process catches them.
Review job responsibilities, incentives, staffing assumptions, and training needs created by the redesigned workflow.
Hold a portfolio decision for every remaining pilot: advance, narrow, combine, pause, buy, or stop.
Organizational change is part of this phase. As AI altered customer service work, 45% of teams updated job descriptions and 40% increased AI training. That is a useful warning against treating adoption as an in-app onboarding problem. If AI takes responsibility for part of a workflow, someone must take responsibility for supervising it, handling exceptions, and improving the system.
Assign decision rights clearly. The domain product team should own the user problem, outcome, workflow design, evaluation cases, and adoption. A platform function should own shared access, retrieval, observability, release infrastructure, and policy enforcement. Risk specialists should define control requirements and review higher-consequence uses. The operational owner should manage quality, escalations, and incidents after launch. Executive leadership should decide portfolio priority, capacity, and which bets no longer deserve investment.
This structure avoids two common extremes. A fully centralized AI team becomes a delivery bottleneck and loses domain context. Fully independent teams duplicate infrastructure and apply inconsistent controls. Centralize reusable capabilities and non-negotiable policies; keep workflow outcomes and day-to-day learning with empowered domain teams.
Expect pressure to spread successful patterns. In customer service organizations, 52% planned to scale AI into areas such as customer success, marketing, and sales. Reuse the platform, governance, evaluation methods, and operating vocabulary. Do not copy a support workflow into another function and assume its value, risks, permissions, or quality bar remain valid.
FAQ: decisions that determine whether AI scales
Should AI be owned centrally or by product teams?
Use a federated model. Centralize capabilities that become safer, cheaper, or more consistent when shared: approved model access, identity, data controls, retrieval services, evaluation tooling, observability, auditability, incident standards, and risk policies. Embed workflow ownership in the domain team that understands the user, process, and business outcome. A central group can set the paved road, but it should not become the permanent product team for every AI use case.
When is an AI workflow ready for more autonomy?
Increase autonomy when the workflow has demonstrated acceptable behavior for the exact action and population being added, failures are detectable, consequences are containable, rollback works, and an operational owner can handle exceptions. Do not remove human review merely because the average quality score improved. Judge autonomy by the worst credible consequence, the reversibility of the action, and the system’s ability to recognize when it should stop.
Autonomy is not binary. The system can retrieve information, recommend an action, draft the result, ask for confirmation, execute within a limited permission, or execute and trigger retrospective review. Choose the narrowest level that captures the value. Expand only when evidence supports the next level.
When should a pilot be stopped rather than scaled?
Stop or reframe a pilot when it has no accountable workflow owner, cannot beat a meaningful baseline, works only on curated inputs, requires unacceptable access, has no safe failure path, or creates more review and remediation than the outcome justifies. Also stop when the supposed AI problem is actually a broken policy, missing data, or poorly designed process that should be fixed directly.
A failed autonomy concept can still reveal a useful assistive product. If execution is too risky, narrow the workflow to retrieval, recommendation, drafting, or exception detection. That is a product decision, not a face-saving exercise. The right scope is the one that creates measurable value under an operating model you can defend.
At your next AI portfolio review, ask each owner to bring a baseline, deployment contract, evaluation evidence, and a clear gate decision. Fund shared infrastructure where the lighthouse workflows expose a recurring need. Expand only after the operating evidence catches up with the demo. That is how you turn a collection of pilots into an AI capability that can carry real responsibility.
“Speed is not the enemy of safety; it is the prerequisite for it.” I live by this principle. In our organization, the average time from merging code to it being used by customers in production is just 12 minutes, and that short window is fundamental to how we build, ship, and learn.
In January 2026, we are averaging 180 ships per workday – roughly 20 deployments every hour. Conventional wisdom suggests that to increase stability, you must slow down. I believe the opposite. Speed is not the enemy of safety; it is the prerequisite for it. Accumulating code creates risk; shipping small batches minimizes it. Shipping is our company’s heartbeat.
Maintaining this frequency while targeting 99.8+% availability has required over a decade of focused investment in systems, principles, and processes. We protect the integrity of our systems through three layers of defense: an automated pipeline that is simple, reliable, and removes the need for manual intervention, a shipping workflow that promotes ownership and uses guardrails as accelerants, and a recovery model that optimizes for mitigating inevitable failures. Here’s how we’ve built each layer so that velocity is our greatest source of stability.
While our platform consists of various services and frontend applications, I’ll focus here on our Ruby on Rails monolith. It is our core application and the one we deploy most frequently; we also deploy it to three different data‑hosting regions with independent pipelines. Our other services follow similar pipeline principles and safeguards, but the Rails monolith is the clearest example of how we ship at scale.
The automated pipeline is designed to move code from merge to production as fast as possible while enforcing strict safety checks. It is fully automated, and the vast majority of releases require no human intervention—critical for CI/CD at high deployment frequency.
Once an engineer merges code to GitHub, two things happen immediately. First, the build: we compile the Rails application and its dependencies into a deployable asset (a slug) in about four minutes. Second, parallel CI: our test suite runs alongside the build; through extensive optimization, parallelization, and test selection, the vast majority of CI builds finish in under five minutes.
As soon as the slug is built, it’s deployed to a pre‑production environment. CI does not block the progression of the slug to pre‑production. Deploying to pre‑production takes around two minutes. This environment serves no customer traffic, but it is connected to our production datastores, mirrors our production infrastructure variants (e.g., web serving, asynchronous worker), and is configured so that requests exercise the pre‑release code and workers.
Immediately after deployment, we run and await several automated approval gates. We verify that the application boots cleanly on hosts (boot test), confirm the parallel test suite passed (CI check), and execute functional synthetics using Datadog Synthetics on critical flows—such as loading or editing a Fin workflow. If any gate fails, the release is halted and does not go to production.
Once approved, we promote the code to thousands of large virtual machines. A deployment orchestrator triggers these deployments simultaneously, while a decentralized, staggered rollout avoids changing the state of the entire fleet at the same millisecond. Within each machine, a rolling restart mechanism removes a process with old code from the serving path, lets it drain gracefully, and replaces it with a fresh process running the new code. From the moment a deployment starts, first requests are served by new code within roughly two minutes, and the vast majority of the global fleet updates transparently within six minutes. When restarts trigger on every machine, production unblocks so the next deployment can begin.
We treat a stalled pipeline as a high‑priority incident. If the automated system rejects three consecutive release attempts, it pages an on‑call engineer. These are pre‑production blocks, but if the shipping lane stops moving, changes pile up—and our stability relies on building and shipping in small steps. The on‑call’s job is to restore flow so that tiny, safe, frequent updates continue to keep risk low.
Our shipping workflow is built on extreme ownership: tools assist, but the engineer is accountable for quality and the decision to merge. I insist that you are present when you ship. The practical benefit of a 12‑minute deployment cycle is that engineers remain in the zone, focused on the problem they just solved, and ready to validate behavior as it goes live.
A rocket lifts into a luminous sky, a metaphor for shipping code fast without breaking things, where precision, automation, and guardrails power 180 safe deployments a day.
To support this, our deployment system sends Slack notifications the moment code is submitted and as it advances through stages, embeds direct observability links to relevant dashboards and logs in every PR and message, and prompts verification so engineers actively watch the dials and test features in production. It is not acceptable to rely on green builds. You’re expected to watch your change go live and if you’re not prepared to rollback, you’re not prepared to ship. We maintain a no‑blame culture: quick rollbacks and immediate reverts are signs of vigilance and ownership, not failure.
We make extensive use of feature flags to turn deployment into a non‑event. By decoupling deployment (moving code to servers) from release (turning features on), we shrink the blast radius of change. Flags can be enabled for all customers, a specific subset, or disabled for everyone in under 60 seconds through our backend UI. Engineers can group flags into beta features and run phased rollouts; we also ensure flags work consistently across non‑monolith applications. In the past three months, we created over 560 flags—and we actively manage them to avoid permanent complexity.
For complex refactors—especially when behavior should not change—we leverage GitHub Scientist, an open‑source experimentation library. It runs candidate logic (new code) in parallel with existing logic (old code) in production, instruments both paths for result and timing comparisons, and keeps existing behavior user‑visible. That means we can iterate on and validate new code under real load without risking the experience, then switch seamlessly when confident.
When engineers need to go deeper before merging, they can generate a slug and deploy it to a virtual machine, detaching a running production host from the serving path and connecting for manual testing. They can also put a pre‑release slug on a serving machine that handles a small percentage of jobs or web requests. Single‑host validation lets us slice observability to those hosts, compare against the main release, and make low‑level changes safer. Staging is a simulation; production is reality. Testing on a single production host validates assumptions with real‑world data without risking the fleet.
Our recovery model starts from a simple principle: stop monitoring systems; start monitoring outcomes. Traditional monitoring tells you if a server is healthy; we care whether customers are healthy. We rely on heartbeat metrics—vital signs that represent the core value our product provides—such as the rate at which messages and comments are created.
Unlike standard uptime checks, heartbeat metrics are binary in spirit. If message send rates dip below baseline, it does not matter if infrastructure dashboards are green. Down is down, and if customers can’t do their job, uptime percentages are irrelevant. By tracking real‑world success rates as a high‑level signal, we catch subtle degradations that traditional alerting either misses or over‑alerts on.
Because we ship in small, incremental steps and maintain previous releases on our virtual machines, our Time to Recover (TTR) is generally very fast. If a heartbeat metric drops or a critical anomaly is detected right after a ship, the system can trigger an automatic rollback, reverting to the release that was running 20 minutes ago—often restoring service before an engineer responds. For complex issues, engineers can initiate a manual rollback through our deployment UI; doing so also locks the production pipeline to prevent further releases while we investigate and remove problematic code.
Resumption of service is not the end. Every incident prompts an incident review, and we don’t just fix the bug. We ask, “How did the machine allow this to happen?” Then we harden the system so it cannot happen again. This loop—fast shipping, fast recovery, rigorous learning—compounds resilience over time.
This operating model aligns to DORA metrics: high deployment frequency, short lead time for changes, low change failure rate, and rapid time to restore service. It’s a CI/CD and SRE‑informed approach that converts speed into a defensive advantage rather than a liability.
Shipping 180 times a day isn’t a vanity metric; it’s a deliberate choice to protect the customer experience. With a 12‑minute window from code to customer, the feedback loop is tight and engineers retain context—and accountability—for the immediate impact of their work. Maintaining this pace requires more than fast CI; it requires judgment, extreme ownership, disciplined use of feature flags, and a recovery model that monitors outcomes. We rely on human expertise, augmented by these layers of defense, to catch issues before they turn into customer pain. We don’t ship fast despite our need for stability; we ship fast to stay in control of change.
I’ve spent the last few years turning AI from an intriguing demo into an operational advantage, and the clearest wins come when we treat agents as productized workflows—not toys. In practice, that means aligning agentic AI to a sharp product strategy, instrumenting everything, and scaling what works across the organization.
Learn how companies like Replit are consolidating workflows, creating one-person departments, and building systems for scale with Amplitude
When I talk about agentic AI, I’m focused on outcomes: fewer handoffs, faster cycle times, and measurable uplift in activation, retention, and NPS. The most successful rollouts start with a specific job-to-be-done, translate it into clear AI workflows, and then iterate with a tight feedback loop between data, design, and engineering.
My implementation playbook is simple and disciplined. First, choose a high-friction workflow and define success upfront. Second, make the build vs buy call on the foundation model, orchestration layer, and connectors. Third, establish AI risk management and safeguards early—before scale amplifies errors. Finally, run small, eval-driven releases and promote what performs.
Instrumentation is where the leverage compounds. With Amplitude analytics as a unified analytics platform, I design purposeful events (agent intent, tool calls, resolution state, human handoff), map funnels from user input to agent outcome, and cohort users by context to pinpoint lift. This gives me an honest read on where agents help, where they hinder, and what to tune next.
The “one-person departments” concept isn’t about doing more with less at all costs; it’s about assembling a tight loop of product management leadership, data, and automation so one operator can own a business outcome end-to-end. An agent handles the repeatable work, while the human focuses on judgment, edge cases, and continuous improvement that compounds.
As we scale, I look for platform scalability patterns: shared tools and policies, reusable prompt libraries, standardized evaluation suites, and consistent governance. That structure keeps agent performance predictable while preserving speed, and it aligns beautifully with product-led growth when agents are embedded directly in the product experience.
If you’re starting now, begin with a single, valuable workflow. Instrument it thoroughly with Amplitude analytics, make decisions from the data you see—not the demos you remember—and expand only after you’ve proven uplift. Iteration beats ambition here: agentic AI rewards teams who measure relentlessly and scale only what truly works.
Inspired by this post on Amplitude – Perspectives.
I’ve been systematically exploring how the product model shows up inside iconic companies. After studying “The Product Model at Spotify” and “The Product Model at Amazon,” I’m turning my lens to Google—specifically, how the product operating model, product culture, and product strategy manifest in practice and what we can pragmatically take back to our own organizations.
When I talk about the product model, I’m looking at the machinery that connects strategy to outcomes: empowered product teams, clear decision rights, tight product trios, continuous discovery, data-informed bets, and an operating cadence that enables learning at speed. My goal here is to unpack how those elements come together at Google and translate them into repeatable patterns you can adopt.
At a high level, I focus on how teams are empowered to solve problems rather than ship outputs, how outcomes vs output OKRs clarify what matters, and how experimentation (from rapid prototyping to A/B testing) de-risks decisions before they scale. I also examine how engineering and product partner to balance platform scalability with customer value, and how stakeholder management reinforces alignment without slowing teams down.
Why does this matter? Because the product model is a lever for resilience and speed. When product strategy is explicit and the operating model is built for learning, organizations multiply the impact of talented people. That’s how small, focused teams repeatedly deliver outsized results—even in complex, regulated, or high-scale environments like Google.
In the sections that follow, I’ll synthesize what I see as the core patterns behind Google’s approach and distill them into actionable guidance: how to structure product trios, how to run continuous discovery alongside delivery, how to set and calibrate OKRs for outcomes, and how to evolve your product culture so empowered product teams can do their best work. My aim is not to idolize a model, but to extract what’s portable and help you adapt it to your context.
I build and scale analytics platforms with a product mindset, and the work starts with the "middleware and compute systems that power analytics at scale." In platforms like Amplitude analytics and other unified analytics platform architectures, that foundation is what makes everything else possible.
Day to day, I oversee the "APIs behind charts, cohorts, and metrics—driving performance, reliability, and platform scalability." When those APIs are fast and resilient, every product team—from growth to customer success—can trust the insights they use to ship, learn, and iterate.
From an engineering leadership standpoint, I partner closely with SRE to define SLOs and error budgets, wire CI/CD pipelines for safe deploys, and track DORA metrics so we improve speed without compromising quality. This combination reduces incident management toil and shortens MTTR while keeping data freshness and query latency within strict thresholds.
From a product management leadership lens, the goal is clarity: crisp APIs, predictable contracts, and transparent stakeholder management across data, engineering, and GTM teams. That alignment empowers product teams with reliable cohorts and metrics, accelerates experimentation, and de-risks roadmaps.
If you’re scaling analytics, invest first in the platform layer: middleware and compute, schema governance, caching strategies, and cost-aware compute. Do that well, and the visible experience—charts, cohorts, and metrics—feels effortless, even as you grow to serve billions of events with confidence.
Inspired by this post on Amplitude – Best Practices.
