Tag: unified analytics platform

Teach AI to Think Like an Analyst: My Amplitude-Inspired Playbook for Faster Decisions

I’ve spent years trying to bottle the judgment of a great product analyst and pour it into our AI workflows. The hardest part isn’t access to data; it’s encoding the nuance of analytical reasoning. That’s why Amplitude’s approach resonated with me—turning expert analysis into a repeatable, stepwise process AI can run with discipline and speed.

Learn how Amplitude turned its data analysis expertise into a structured, iterative process that AI can execute in moments.

In practical terms, I translate that one line into an operating model: define the decision, formalize the metrics, map the data, decompose the questions, iterate on evidence, and converge on a recommendation with clear trade-offs. This is the backbone of agentic AI for product managers—giving an LLM not just data, but a procedure that mirrors how our best analysts think.

Here’s the analyst-to-AI loop I use. First, frame the business question in decision language (what will we do differently?). Second, anchor on success metrics and guardrails, including statistical sensitivity and minimum detectable effect (MDE). Third, locate trusted sources—your unified analytics platform, experiment logs, and product instrumentation—so the AI never guesses. Fourth, generate hypotheses and segment the data (cohorts, channels, plans, geos), prioritizing signal over noise. Finally, synthesize findings into options with expected impact, risks, and next steps.

To operationalize this, I build a retrieval-first pipeline that binds Amplitude analytics to structured prompts and function calls. The AI receives exact metric definitions, event taxonomies, and governance rules, then returns a predictable schema—headlines, evidence, segments, caveats, and recommended actions. That combination of clear constraints and consistent output makes eval-driven development possible: I can test prompts and tooling against a gold set of analyses and steadily improve quality.

Consider retention analysis on a new onboarding flow. I’ll ask the system to pull activation rate, time-to-value, and day-7 retention from Amplitude, then compare cohorts by channel and plan. The AI proposes hypotheses (e.g., tooltip engagement correlates with activation), runs segmentation to validate them, and lays out product-led growth levers—like simplifying the first-run checklist or moving guidance in-app. What used to take hours of manual slicing now becomes an iterative loop that lets me spend more time on prioritization and less on tab wrangling.

Of course, speed without rigor is a trap. I guard against metric drift and hallucinations with strong definitions, lineage checks, and human-in-the-loop approvals for consequential decisions. I also log analysis steps and outcomes so we can audit reasoning, catch regressions, and keep AI grounded in our true north metrics—not just what’s easy to compute.

The big unlock isn’t a clever prompt; it’s codifying the analyst’s craft. When we treat analysis as a structured, iterative process, AI can execute it with consistency, and product teams can move faster with more confidence. If you’re building AI workflows for product insight, start by formalizing your analyst loop, connect it to your Amplitude analytics, and evaluate continuously. The result is smarter, faster decisions—and a repeatable path from raw data to action.

Inspired by this post on Amplitude – Best Practices.

December 16, 2025
Inside the Core Analytics Mindset: How I Build Products that Drive Clarity and Growth

I spend my days shaping core analytics product experiences that help teams see their business with greater clarity. When I design an analytics workflow, my goal is simple: make it effortless to ask better questions, uncover meaningful patterns, and turn insight into action. In this brief reflection, I’ll share how I approach product discovery, experimentation, and roadmapping to create analytics tools that truly move the needle.

Everything starts with outcomes. I anchor roadmaps to a clear north star and use outcomes vs output OKRs to align problem statements with measurable impact. That means instrumenting a precise event taxonomy and building guardrails for data quality so retention analysis and user activation metrics are trustworthy. When the foundation is sound, product-led growth becomes repeatable because we can connect feature usage to value creation without guesswork.

Experimentation is where conviction meets evidence. I rely on A/B testing with a disciplined view of minimum detectable effect (MDE) so we size experiments responsibly and ship with confidence. Self-serve analysis—and, when appropriate, tools like Amplitude analytics within a unified analytics platform—lets teams quickly validate hypotheses, monitor cohorts, and understand lift. The result is faster learning cycles without sacrificing statistical rigor.

On the delivery side, I practice continuous discovery and translate insights into product roadmapping and sprint planning that teams can execute. I work closely with design and engineering to reduce cognitive load in the UI, standardize tooltips and in-app guides, and ensure every chart, filter, and segment supports a clear decision. This collaboration empowers the team, shortens feedback loops, and keeps us oriented toward customer outcomes rather than feature checklists.

Great analytics products give people confidence. By aligning on outcomes, instrumenting clean data, testing with discipline, and shipping thoughtfully, I’ve seen teams unlock deeper understanding and sustained growth. If you care about building products that illuminate the path forward, start with the questions customers need to answer—and let your analytics experience make those answers obvious.

Inspired by this post on Amplitude – Best Practices.

December 16, 2025
Inside the Engine Room: How I Drive Scalable Analytics APIs, Reliability, and Performance

I build and scale analytics platforms with a product mindset, and the work starts with the "middleware and compute systems that power analytics at scale." In platforms like Amplitude analytics and other unified analytics platform architectures, that foundation is what makes everything else possible.

Day to day, I oversee the "APIs behind charts, cohorts, and metrics—driving performance, reliability, and platform scalability." When those APIs are fast and resilient, every product team—from growth to customer success—can trust the insights they use to ship, learn, and iterate.

From an engineering leadership standpoint, I partner closely with SRE to define SLOs and error budgets, wire CI/CD pipelines for safe deploys, and track DORA metrics so we improve speed without compromising quality. This combination reduces incident management toil and shortens MTTR while keeping data freshness and query latency within strict thresholds.

From a product management leadership lens, the goal is clarity: crisp APIs, predictable contracts, and transparent stakeholder management across data, engineering, and GTM teams. That alignment empowers product teams with reliable cohorts and metrics, accelerates experimentation, and de-risks roadmaps.

If you’re scaling analytics, invest first in the platform layer: middleware and compute, schema governance, caching strategies, and cost-aware compute. Do that well, and the visible experience—charts, cohorts, and metrics—feels effortless, even as you grow to serve billions of events with confidence.

Inspired by this post on Amplitude – Best Practices.

December 12, 2025
Automated Insights for Product Teams: Uncover Causal ‘Aha’ Moments in Minutes, Not Weeks

I’ve spent countless cycles guiding teams through the maze of dashboards, SQL pulls, and ad‑hoc analyses—only to watch truly meaningful patterns emerge far too late. Automated insights are the next frontier in product analytics: a shift from manual exploration to AI that proactively surfaces what matters most. When we let the system do the heavy lifting, we accelerate discovery, reduce bias, and give product trios the clarity to act.

Finding causal connections in product data involves exhaustive searches and tests. We trained our AI to find “aha” moments in minutes instead of weeks.

Here’s what that means in practice for product management: the platform continuously scans events, cohorts, and segments; prioritizes signals linked to activation, conversion, and retention; and highlights likely causes behind meaningful movements in your core KPIs. Instead of sifting through endless funnels and cohorts, I get ranked hypotheses I can validate with targeted A/B testing and minimum detectable effect (MDE) guardrails.

This approach turns analytics into action. Automated insights reduce time-to-learning, tighten our discovery loops, and make continuous discovery tangible—especially when we’re aligning roadmaps, designing experiments, and refining onboarding. Whether you’re using tools like Amplitude analytics or instrumenting a unified analytics platform, the value is the same: faster, clearer paths to customer impact.

I’ve seen teams unlock retention analysis breakthroughs by spotting counterintuitive patterns—like a specific feature combination or an overlooked step in onboarding—well before they would have surfaced through manual analysis. With AI workflows scanning the noise and elevating the signal, we can focus on decisions: ship or iterate, scale or sunset, double down or pivot. That’s empowered product teams in action.

If you’re building for product-led growth, this is the leverage you’ve been waiting for. Automated insights transform how we prioritize, test, and communicate strategy—bringing us from gut feel and lagging indicators to explainable, causal narratives we can stand behind. The outcome is simple: more confident bets, less waste, and a faster path to durable product-market fit.

Inspired by this post on Amplitude – Best Practices.

December 10, 2025
Unlock Real-Time Product Insights: Amplitude + OpenAI MCP in ChatGPT, Without BI Bottlenecks

I’ve been working to remove the friction between product questions and product answers. The most impactful step so far: connecting Amplitude analytics directly into ChatGPT via OpenAI’s MCP. This turns everyday conversations into decision-grade insights—no dashboards to hunt, no SQL to write, and no analytics queue to wait on.

Connect Amplitude data directly to the tools your team uses every day. OpenAI’s MCP connector eliminates traditional barriers to product data.

In practice, this means I can ask ChatGPT natural-language questions like, “Where are users dropping in our activation funnel this week?” or “Which cohorts are driving retention lift post-onboarding?” and get grounded answers from Amplitude—fast. It’s a step-change for product-led growth because the insights live where we already think and plan.

Here’s how I apply it day to day: I’ll prompt ChatGPT to compare week-over-week activation for new SMB signups across regions, diagnose drop-offs by step, and summarize A/B testing outcomes with guardrails like minimum detectable effect considerations. When we’re shaping strategy, I’ll pull a retention analysis and cohort breakdown to inform bet sizing and roadmap tradeoffs—all without pulling the team into a BI bottleneck.

Governance remains non-negotiable. I scope the MCP tools to a least-privilege data slice, apply privacy-by-design rules to exclude PII, and log every query for auditability. Clear data governance and AI risk management policies ensure we maintain trust while accelerating discovery. Tight context window management keeps prompts focused and reduces noise.

Operationally, the setup is straightforward: define the MCP tool spec for Amplitude, map canonical events and metrics (activation, retention, conversion, and product-qualified lead stages), and test with a retrieval-first pipeline so responses reliably cite the right source of truth. We standardize metric definitions across product, growth, and customer success to avoid semantic drift.

The impact on empowered product teams is immediate. Continuous discovery becomes a daily habit rather than a quarterly ritual; questions move from “I’ll get back to you” to “Let’s check right now.” For product managers working with LLMs, this is the connective tissue that makes ChatGPT a true ChatGPT connector for analytics—an on-demand, unified analytics platform that supports faster iteration and sharper decision-making.

If you’ve been waiting to make analytics truly ambient, this is the moment. Start small with a single funnel or cohort, validate governance, and expand to your core lifecycle metrics. The payoff is a shared understanding of what’s working, what’s not, and where to focus next—delivered in the flow of work.

Inspired by this post on Amplitude – Best Practices.

December 10, 2025
6 AI Strategies to Accelerate Business Growth: Unlock Revenue, Cut Costs, Scale Faster

I’ve spent the last few years weaving AI into core product workflows, and the pattern is clear: when we pair disciplined product thinking with pragmatic AI Strategy, growth compounds. The question I hear most isn’t if AI can help, but where to begin and how to de-risk the journey while moving fast.

AI for business growth starts with one of these six strategies. See how companies use AI to unlock revenue, cut costs, and scale smarter and faster.

1) Revenue acceleration with unified customer intelligence. I start by connecting behavioral analytics and CRM integration to a unified analytics platform, then layer a retrieval-first pipeline so large language models can surface high-intent accounts, churn signals, and next-best actions. With Amplitude analytics and A/B testing, we validate AI-driven playbooks for upsell, cross-sell, and win-back—turning insights into measurable lift rather than novelty.

2) Cost reduction through targeted automation. Not all automation yields the same outcome. I look for repetitive, high-volume processes where quality is easy to verify—customer support ai strategy with AI-assisted deflection, accounts payable automation, and security workflows like threat detection and response. Combining agentic AI with clear guardrails reduces handle time, frees teams for higher-value work, and keeps error rates within acceptable thresholds.

3) Faster time-to-market via eval-driven development. Speed without signal is noise. I lean on eval-driven development to instrument models, measure drift, and tighten CI/CD loops. We track DORA metrics like deployment frequency while using gen ai for product prototyping to compress discovery and delivery. Frameworks and tools such as Claude Code help engineers iterate safely behind feature flags so we can ship learning, not just code.

4) Personalization that drives activation and retention. Growth sticks when onboarding is contextual. I use in-app guides, product tours, and thoughtful tooltip design powered by LLMs for product managers to tailor the first-run experience. With retention analysis and outcomes vs output OKRs, we align personalization with the moments that matter—activation, habit formation, and expansion.

5) Trust-by-design to scale responsibly. AI risk management, privacy-by-design, and data governance are not afterthoughts; they are growth enablers. By defining policy, red-teaming prompts, and practicing context window management, we reduce rework, limit incident management, and maintain compliance across markets. Clear review gates make it easier to say yes to more AI use cases without compromising customer trust.

6) Voice and agent experiences that feel like product, not add-ons. When prompt engineering for voice and voice AI agent patterns are integrated into the core journey—guided onboarding, smart handoffs, proactive notifications—engagement rises. Agent Analytics turns conversations into product signals we can act on in roadmapping and sprint planning, closing the loop between user intent and product improvement.

My playbook for getting started is simple: pick one revenue and one efficiency use case, define success upfront, and ship a narrowly scoped MVP with robust analytics. Use continuous discovery with product trios to refine prompts, data sources, and experience design. Then scale what works, retire what doesn’t, and let evidence—not hype—set the roadmap.

If you’re evaluating where to apply gen ai next, these six lanes offer fast paths to impact without sacrificing governance or customer trust. The companies I’ve seen win treat AI as a capability within the product, not a separate project—and they measure it with the same rigor they use for any critical feature.

Inspired by this post on Product School.

December 10, 2025
Product Analytics for Everyone: Master Funnels, Retention, and Conversion to Drive Growth

Product analytics isn’t a specialist’s sport—it’s a team capability. In my role leading product teams, I’ve seen designers, engineers, marketers, and customer success partners uncover insights that shape strategy, accelerate product-led growth, and improve outcomes for customers. When we demystify the basics and bring analytics into everyday decisions, we build truly empowered product teams.

Here’s the core promise of this approach: "Learn the product analytics fundamentals of funnels, retention, and conversion drivers so that anyone can confidently answer key product questions." That line has guided how I teach product managers to think—start with the essentials, tie them to real customer behaviors, and make the work repeatable across the organization.

I start with funnels because they tell a story—the journey from discovery to value. A simple example: track the path from sign-up to user activation to the first value event. This reveals where onboarding succeeds or stalls, what friction blocks adoption, and which moments are ripe for optimization. With tools like Amplitude analytics or Pendo, we can break down conversions by segment, channel, or feature usage to isolate where improvements matter most.

Next comes retention analysis, the clearest signal that we’re building something customers choose to return to. Cohort analysis shows who comes back and when; retention curves show where value compels a second, third, and tenth use. Tie retention to activation milestones and the outcomes customers achieve—not just logins—and you’ll quickly spot whether your product discovery assumptions hold up in the wild. A unified analytics platform makes these insights discoverable and repeatable across teams.

Conversion drivers round out the picture. Once the funnel is clear and retention is stable, I look for the behaviors and experiences that predict success: feature combinations, time-to-value, message timing, or supportive content. Whether in Amplitude analytics or Pendo, correlating these drivers with outcomes lets us prioritize roadmaps with confidence. Pair this with continuous discovery—qualitative interviews, in-product feedback, and rapid experiments—and you’ll move from interesting data to decisive actions.

This is how we build empowered product teams: by making analytics a daily habit rather than a quarterly report. We bring insights into roadmap reviews, design critiques, and sprint planning; we celebrate learning from experiments as much as shipping features; and we hold ourselves accountable to customer outcomes, not just output. When everyone can interpret funnels, discuss retention, and isolate conversion drivers, we make smarter bets faster.

If you’re getting started, keep it simple. Define a clear activation metric, instrument the top of your funnel, and track a small number of cohorts. Share a weekly readout with highlights, surprises, and questions to investigate. Over time, stitch insights into narratives that drive product-led growth—and, most importantly, help customers achieve what they came for.

Product analytics isn’t just for analysts. It’s a shared language for product discovery, onboarding excellence, user activation, and long-term retention. When we practice it together, we build better products and stronger teams.

Inspired by this post on Amplitude – Best Practices.

December 5, 2025
Enterprise Go-to-Market Mastery: How I Align Product, Positioning, and Analytics at Scale

I build enterprise growth motions by grounding strategy in data and execution in crisp storytelling. When I partner with teams using Amplitude, I focus on architecting "go-to-market solutions for enterprise customers." That simple phrase clarifies the mandate: align product, marketing, and sales around measurable value, reduce buyer risk, and prove outcomes early and often.

My go-to-market strategy begins with rigorous segmentation and an ideal customer profile, then translates into a living narrative: the value proposition, points of parity, and competitive differentiation that underpin product positioning. I pressure-test that narrative with real customer language, executive business cases, and use-case–level messaging so every stakeholder—from procurement to security to the economic buyer—hears their priorities reflected back with credibility.

Execution is analytics-led. With Amplitude analytics as a unified analytics platform, I instrument the entire journey—from first touch to paid expansion—to expose activation, aha moments, and friction. I use A/B testing to validate in-app guides, product tours, and onboarding, and I track user activation and retention analysis to ensure product-led growth efforts compound over time. These signals inform sales enablement, content roadmaps, and launch plans so each asset moves a specific metric, not just a milestone.

Operating cadence matters as much as the plan. I rely on empowered product teams and product trios to translate strategy into product roadmapping and sprint planning, ensuring every slice of the roadmap ties directly to market impact. Clear OKRs and QBRs keep the feedback loop tight, while field insights from enterprise pilots shape rapid iteration without losing strategic intent.

Enterprise nuance is the difference-maker: longer cycles, multi-threaded buying committees, and higher switching costs demand precision. I design proofs of value that quantify outcomes early, align pricing and packaging with willingness to pay, and use customer evidence to de-risk decisions. The result is a scalable, repeatable system where positioning is consistent, the funnel is measurable, and revenue teams can predictably win with complex accounts.

Ultimately, the work is about trust. When strategy, analytics, and storytelling lock together, customers see themselves in the product—and teams see themselves in the win. That is the heart of enterprise go-to-market done right.

Inspired by this post on Amplitude – Perspectives.

December 4, 2025
Unify Your Analytics to Accelerate Growth: Cut Costs, Boost Clarity, and Decide in Real Time

I’ve led product teams through the pain of scattered dashboards and contradictory metrics, and I’ve seen how it slows decision velocity and quietly inflates costs. When insights are fragmented, roadmaps drift into opinions and meetings multiply. A unified analytics platform changes the conversation—from noise to signal, from lagging to leading indicators, and from guesswork to confident execution.

"Escape fragmented tools with a unified analytics platform that accelerates growth, reduces costs, and empowers smarter, real-time decision-making."

Here’s what “unified” means in practice: one source of truth that connects product usage, marketing attribution, sales pipeline, and customer support signals. With CRM integration, consistent event taxonomy, and retention analysis in place, every team works from the same playbook. Cohorts, funnels, and lifecycle metrics become part of daily rituals, and insights flow directly into product discovery and go-to-market decisions.

The impact is tangible. Product-led growth becomes predictable because activation, engagement, and retention are measured the same way across functions. Experimentation accelerates as A/B testing cycles tighten and learning compounds. Outcomes vs output OKRs stay visible and honest, helping us prioritize what moves the needle. Costs come down as redundant tools are rationalized and manual data wrangling disappears. Most importantly, real-time decision-making replaces weekly retrospectives with timely action.

My playbook for getting there is straightforward: start with a tool and data audit; define a clear north-star metric with a handful of leading indicators; standardize event names and properties; connect the data layer to your CRM for closed-loop visibility; instrument product tours and in-app guides to drive user activation; and institutionalize continuous discovery so every insight informs the roadmap and sprint planning.

Governance and trust matter as much as dashboards. Invest in data governance and a clean tracking taxonomy so metrics are trusted across the organization. Document definitions, automate quality checks, and maintain privacy-by-design from the start. The goal isn’t more data—it’s better decisions, faster, with confidence.

I’ve watched teams cut time-to-insight from days to minutes, reallocate budget from underperforming channels to winning ones, and ship with far greater conviction. When the organization rallies around a unified analytics platform, stakeholder debates shrink, velocity increases, and the value proposition to customers sharpens.

If growth, cost savings, and smarter decision-making are on your agenda this quarter, commit to unifying your analytics. Start small, prove the value in one journey (like activation to retention), then scale. The moment you align your teams to a single source of truth is the moment your product strategy becomes unmistakably clear.

Inspired by this post on Amplitude – Perspectives.

December 3, 2025
A Product Strategist & Evangelist’s Playbook at Amplitude: Turning Analytics into Growth

I’ve long believed that the Product Strategist & Evangelist role is where analytics meets impact. When I work with teams using Amplitude, my focus is simple: turn product data into decisions that compound, and tell the story in a way that mobilizes people—customers, stakeholders, and empowered product teams alike.

At its core, this role aligns product strategy with business outcomes. I anchor planning to outcomes vs output OKRs, partner closely with product trios, and run continuous discovery to ensure every roadmap item is tied to a measurable customer problem and value proposition. That discipline keeps us honest about what moves the needle.

Analytics is the engine. I start with a clean event taxonomy, dependable instrumentation, and a self-serve insight layer in Amplitude analytics. From activation to retention analysis, I define a few sharp metrics that predict sustainable product-led growth—then I build dashboards the whole organization can trust and use.

Experimentation is where insight becomes action. I operationalize A/B testing with clear hypotheses, guardrails for minimum detectable effect, and crisp success criteria. The goal is speed with rigor: learn fast, ship what works, and retire what doesn’t. Over time, this creates a culture where teams default to evidence rather than opinions.

Evangelism turns analytics into momentum. I practice developer evangelism to meet practitioners where they are, and I translate complex findings into accessible narratives for executives and customer-facing teams. That means live walkthroughs, in-app guides, product tours, and field enablement that shows not just the what, but the why and the how.

Under the hood, a unified analytics platform is essential. I pair it with pragmatic data governance and privacy-by-design so we can scale insights confidently. The result is a flywheel: reliable data, repeatable workflows, and reusable patterns that accelerate every subsequent initiative.

On the go-to-market front, I connect product strategy to positioning, packaging, and enablement. The stories we tell in the market should mirror the value we measure in the product. That alignment makes launches sharper, sales motions clearer, and adoption smoother.

In practice, my playbook is straightforward: clarify the North Star and adjacent metrics, stand up trustworthy pipelines and dashboards, institutionalize experimentation, and continuously translate insights for decision-makers. Done well, analytics stops being a report and becomes a system for growth.

If you’re building or evolving this function, start small and intentional: instrument the few events that matter, ship one meaningful A/B test, and circulate a concise narrative on what you learned. Consistency beats complexity, and momentum compounds quickly when teams see their decisions move the metrics that matter.

Inspired by this post on Amplitude – Perspectives.

December 3, 2025
Enterprise GTM Mastery: How I Partner with Product Marketers to Drive Adoption at Scale

I spend a lot of time turning strong product capabilities into enterprise wins, and that almost always starts with a tight partnership between product management and product marketing. The most effective go-to-market strategy is built where customer insight, product value, and revenue goals intersect—and product marketers are the connective tissue that makes this real.

“Michele Morales is a product marketing manager at Amplitude, focusing on go-to-market solutions for enterprise customers”

In my experience, partnering with product marketing leaders on enterprise go-to-market means aligning early on the ICP, the value proposition, and the differentiated messaging that sales can activate. We map buyer committees, refine product positioning against points of parity and competitive differentiation, and ensure our narrative translates cleanly from website to demo to proof-of-concept.

For data-driven execution, I lean on Amplitude analytics and a unified analytics platform approach to validate our hypotheses. We set clear activation and adoption milestones, monitor user activation cohorts, and close the loop with retention analysis to understand which messages and features actually move enterprise accounts from trial to expansion. This is where product-led growth complements sales-led motions, giving us empirical signal across the funnel.

On the launch front, we pressure-test enablement and in-product experiences together: crisp messaging frameworks, in-app guides, and product tours that shorten time-to-value for complex enterprise use cases. The result is a go-to-market strategy that’s both technically accurate and emotionally resonant—clear enough for executives and actionable for end users.

What consistently works: start with real customer pain, express value succinctly, and make the path to first success obvious. Then instrument everything. When product, marketing, and sales can all see the same truth in the data, empowered product teams iterate faster, positioning sharpens, and adoption compounds.

This approach respects the craft of product marketing while grounding decisions in measurable outcomes. It’s how we turn a promising roadmap into repeatable enterprise impact—and why close PM–PMM collaboration remains one of my most reliable growth levers.

Inspired by this post on Amplitude – Best Practices.

December 3, 2025
From Stone Soup to Insights: Eval-Driven Development That Supercharges AI Analytics

I’ve learned that the most powerful AI features rarely emerge from lone-wolf brilliance—they’re born when a community rallies around a shared objective. “Building Amplitude’s AI for insight automation felt a lot like the fable of travelers making stone soup with their community.” That spirit captures how I approach shipping AI for analytics: bring focused ingredients, invite contributions, and let rigorous evaluation transform the result into something extraordinary.

At the core is Eval-Driven Development. Rather than debating preferences, we define explicit evaluation sets, success thresholds, and guardrails, then wire them into CI/CD so every change improves reliability, quality, and relevance. For AI-driven analytics, our evals combine offline judgment tests (precision, recall, hallucination rates), user-centric measures (time-to-insight, actionability), and production health signals (failure modes, latency). When the bar rises, the product improves—continuously and measurably.

We made “stone soup” by inviting contributions from every function. Data science established gold-standard datasets and baselines. Engineering implemented retrieval, orchestration, and safe deployment paths. Product and design framed high-value use cases, in-app guides, and UX writing that clarified intent. Customer success and support piped real-world edge cases into our evals so the system improved where it mattered. Product trios kept us outcome-focused and empowered product teams moved quickly without sacrificing governance.

Why this matters for analytics: AI insight automation reduces the heavy lift of exploring funnels, cohorts, anomalies, and retention patterns—accelerating activation and product-led growth. With a unified analytics platform and strong data governance, we can surface relevant patterns proactively, explain the “why” behind movements, and recommend next best actions without drowning users in noise. The result is faster decisions, cleaner handoffs between teams, and a tighter loop from observation to intervention.

Our practical playbook is simple but strict: define a clear north-star outcome; curate representative eval sets that mirror real user questions; simulate A/B testing offline before live traffic; instrument time-to-insight and adoption; and integrate evals into CI/CD so regressions never ship. We monitor DORA metrics to maintain delivery velocity while holding quality lines, and we use human-in-the-loop review to continuously refine prompts, patterns, and explanations.

We also learned what doesn’t work. General-purpose prompts seldom transfer cleanly to analytics without domain grounding and context window management. A retrieval-first pipeline improves factuality, but only if metadata and event taxonomies are consistent. And while generative UX can delight in demos, it must earn trust in production through transparent reasoning, privacy-by-design, and predictable behavior under load.

In the end, the stone soup metaphor isn’t about cute storytelling—it’s about disciplined collaboration. When a cross-functional community contributes the right ingredients and Eval-Driven Development keeps us honest, AI for insight automation becomes both credible and compounding. That’s how we turn analytics into action—and how we ship AI products that users rely on every day.

Inspired by this post on Amplitude – Best Practices.

December 3, 2025