Tag: product-led growth

Implementing Agentforce the Smart Way: My Proven Playbook for Salesforce Agentic Success

Implementing Agentforce isn’t a feature rollout—it’s a strategic shift. In my role building AI-driven products, I treat Agentforce as its own product with clear outcomes, rigorous governance, and disciplined iteration. The objective is to create durable operational leverage inside Salesforce without compromising trust, data integrity, or customer experience.

Learn the ways in which Pendo helps companies design and iterate on their agentic strategy for Salesforce.

I start with product discovery. That means selecting the right use cases, defining the target user, and aligning on measurable outcomes rather than outputs. In practice, I prioritize use cases across sales, service, and marketing using an impact–effort–risk lens, then set crisp success metrics—response time, deflection rate, case resolution, win rate lift, and user adoption. This keeps everyone focused on value creation, not just model novelty.

Next, I design the agentic system with guardrails. I specify agent roles, tools, and policies; define when to escalate to humans; and embed privacy-by-design and data governance from day one. I also build an evaluation harness with offline tests and live A/B testing, ensuring we have a minimum detectable effect that’s meaningful for the business. The goal is to measure outcomes reliably and course-correct quickly.

When building the first slice, I scope narrow and ship fast. For example, start with a constrained service workflow—classify the case, propose a response, and take a safe action—with clear affordances in Salesforce so users understand what the agent did and why. I instrument the experience end-to-end and use Pendo for in-app guides, surveys, and behavioral analytics to reduce onboarding friction and capture real-time feedback at scale.

Iteration is where value compounds. I run weekly reviews of conversations, error taxonomies, and edge cases; adjust prompts and tool access; and maintain a steady experiment cadence. We track outcomes vs output to avoid vanity metrics, and we document learnings to de-risk the next use case. This steady drumbeat builds credibility with stakeholders and confidence with frontline users.

Change management is non-negotiable. I align leaders early, set expectations on what the agent can and cannot do, and define SLAs for humans-in-the-loop. I use product tours to teach new behavior, highlight quick wins, and establish transparent feedback channels. This combination of enablement and accountability accelerates adoption and creates a culture that embraces agentic AI responsibly.

Finally, I scale thoughtfully. Once the first use case demonstrates value, I standardize patterns, unify analytics, and evolve governance as usage grows. I review risk regularly, align OKRs with the roadmap, and keep a tight feedback loop between product, ops, and go-to-market teams. Treating Agentforce as an evolving product—not a one-off project—maximizes impact while protecting the customer experience.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Our Pendo-Powered Playbook: Orchestrating a High-Impact Summer Release with Product-Led Growth

We set out to promote the Pendo Summer Release using the most authentic approach possible: we used Pendo to market Pendo. That decision anchored our strategy in product-led growth, letting us reach users in context, guide them through new capabilities, and measure impact in real time without adding friction or cost.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

Our objectives were clear: drive adoption of new features, accelerate onboarding for existing customers, and improve engagement across key workflows. We framed the work with outcomes vs output OKRs, clarified the value proposition for each persona, and aligned our product positioning to highlight points of parity and genuine differentiation.

Execution centered on in-app guides, product tours, and purposeful tooltip design. We segmented by role, lifecycle stage, and behavior to keep messages timely and relevant, then layered in A/B testing with a defined minimum detectable effect (MDE) so we could learn fast without overexposing users. Product trios partnered closely with design and forward-deployed engineers to iterate quickly on copy, UX writing, and guide placement.

On the measurement side, we instrumented clear goals and tracked conversions through the funnel, pairing event analytics with retention analysis to understand depth of usage, not just clicks. We captured qualitative signal through micro-surveys and in-context feedback, feeding insights back into product roadmapping and sprint planning to sharpen our next set of in-app experiments.

Governance mattered as much as growth. We applied privacy-by-design principles, ensured strong data governance, and kept stakeholder management tight so each guide had a clear owner, sunset plan, and success criteria. That discipline helped us sustain momentum without cluttering the experience.

The biggest lesson: when done thoughtfully, in-app education scales like a dedicated success team—at a fraction of the cost—while teaching you exactly where users find value. This Pendo-powered launch playbook now underpins our onboarding, cross-sell motions, and QBRs alike, giving us a repeatable way to promote releases, validate hypotheses, and deepen engagement with every iteration.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Inside Pendo’s Decision: Replacing the Website Chatbot With an AI Agent to Boost ROI

Traditional website chatbots promised instant answers but rarely delivered the depth, context, and actionability modern buyers expect. After seeing patterns of high drop-off and shallow engagement, I stepped back and reframed the problem: We did not need another scripted bot—we needed an AI Agent capable of understanding intent, personalizing responses, and taking meaningful actions in the flow of discovery.

That is why Pendo replaced the website chatbot with an AI Agent. From a product management lens, the decision hinged on three criteria: accelerate time-to-value for visitors, reduce operational overhead through automation, and improve the quality of demand captured at the top of the funnel. An agentic AI approach met all three.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

This statement crystallizes the business case. An AI Agent can translate product intent into measurable outcomes by connecting to knowledge sources, analytics, and workflows. Instead of handing off a prospect to a form or a static knowledge article, the agent can surface relevant guidance, qualify interest, book meetings, and even trigger product tours—closing the loop between marketing, product, and customer success.

We anchored the implementation in data governance and privacy-by-design. That meant carefully curating training corpora, instituting role-based access controls, applying guardrails for sensitive topics, and designing graceful human-in-the-loop fallbacks. The result was not just a smarter front door, but a safer one—critical for regulated buyers and enterprise stakeholders.

To validate impact, we ran disciplined A/B testing with a clearly defined minimum detectable effect across conversion, engagement depth, and time-to-response. We also monitored secondary signals such as escalation rate to human support, session quality, and downstream product adoption. Early signals showed more qualified conversations, fewer dead ends, and faster paths to value—exactly the outcomes a product-led growth motion requires.

The experience uplift did not stop at the website. By aligning the agent with in-app guides and product tours, we created continuity from pre-signup exploration to onboarding and activation. Visitors received consistent, contextual help before and after they became users, which strengthened our product positioning and reduced friction across the journey.

Operationally, the shift lowered the marginal cost of each high-quality interaction while improving reliability. Agent handoffs to sales or support became intentional rather than reactive, and insights from conversations fed directly into product discovery. That closed feedback loop informed roadmap decisions and sharpened our go-to-market strategy.

If you are considering a similar move, start with a clear AI Strategy tied to measurable outcomes, a robust governance model, and a pragmatic rollout plan. Focus the agent on high-intent moments first, surround it with analytics and experimentation, and let the data guide expansion. The goal is not to replace humans—it is to elevate them by letting the AI Agent handle the repetitive, high-volume work so your teams can focus on complex, high-value interactions.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
How Pendo Agent Analytics Protects Your Data—and Accelerates Adoption Without Compromise

Protecting customer data while driving product-led growth is the needle I move every day. When I evaluate analytics agents for enterprise software, I look for platforms that make it easy to learn from behavior without exposing sensitive information. That is the promise behind Pendo Agent Analytics: actionable insight with strong guardrails, so teams can move fast without breaking trust.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practical terms, “protecting your data” starts with privacy-by-design: data minimization, clear event taxonomies, and opinionated defaults that discourage collecting anything you don’t need. I require role-based access controls, transparent governance workflows, and a unified analytics platform that helps product, engineering, security, and legal speak the same language. Those foundations enable confident experimentation—A/B testing, onboarding optimizations, and in-app guides—without creating new risk.

My implementation playbook is straightforward. First, define a lightweight tracking schema aligned to outcomes (adoption, time-to-value, retention analysis), not vanity metrics. Second, keep payloads intentionally sparse and free of secrets—no tokens, no free-form text, no PII. Third, ship value quickly with curated product tours and tooltip design that guide users through high-intent moments. Finally, review events regularly with a cross-functional product trio to prune, consolidate, and govern.

Security and data governance are not just checkboxes; they are operating disciplines. I partner with IT leadership to verify access policies, audit usage patterns, and ensure consent and data retention practices meet internal standards. This creates the right tension between speed and safety, so teams can optimize onboarding and in-app experiences while reducing operational risk.

I also benchmark instrumentation approaches across tools—looking at Amplitude analytics, for example—to ensure our event taxonomy and governance model stays consistent across the stack. Consistency matters: it improves stakeholder management, accelerates product discovery, and keeps our outcomes vs output OKRs anchored to the same source of truth.

The result is a healthier product loop: cleaner data, clearer insights, and faster iterations that meaningfully improve engagement. With disciplined governance and thoughtful design, Pendo Agent Analytics can inform what to build next while respecting user privacy—giving teams the confidence to learn at speed, and customers the confidence to keep trusting us.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Ultra‑Personalized AI Product Experiences: How I Push the Limits Without Crossing the Line

Every week I meet teams eager to unleash AI-driven personalization across their products—and I share the same excitement. The promise is magnetic: experiences that feel tailor‑made, delivered at scale, and continuously optimized. Yet sustainable differentiation doesn’t come from turning every dial to eleven; it comes from clarity of intent, responsible design, and disciplined execution.

AI has us on the verge of a new age of ultra-personalized digital product experiences. But don't swing too big too early.

When I think about “how far is too far,” I anchor on user trust, explainability, and measurable value. If a personalization can’t be explained in a sentence, verified through A/B testing, or opted out of without friction, it’s a risk to both brand and product-market fit. The goal isn’t maximal personalization—it’s meaningful personalization that compounds retention and strengthens the value proposition.

I start with product discovery basics: who are the core segments, what jobs-to-be-done matter most, and where does personalization remove friction or accelerate time-to-value? That focus informs pragmatic AI Strategy. Instead of boiling the ocean, I’ll select one high-traffic, high-intent flow and define the precise outcome we want to move. Then I set outcomes vs output OKRs and instrument the path so I can track lift, variance, and trade-offs in real time.

Data governance is non-negotiable. Consent, transparency, and data minimization create the foundation for scalability. I document what signals power personalization, how long they persist, and who can access them. Strong governance isn’t a brake; it’s an enabler, letting us expand confidently without rework or reputational drag.

From there, I validate with A/B testing and clear minimum detectable effect (MDE) thresholds. Holdouts, guardrail metrics, and cohort analyses keep me honest. I’ll use Amplitude analytics to examine funnel impacts, retention analysis, and segment-level effects—especially to ensure we’re not improving conversion while harming long‑term engagement or fairness for smaller segments.

Early wins often come from onboarding and in-app guides. Personalizing the first five minutes—recommended next steps, contextual tooltips, or a tailored product tour—can deliver a step-change in activation with minimal risk. This is where product-led growth shines: relevant, timely nudges that shorten the path to the “aha” moment without feeling intrusive.

As we scale, gen ai and agentic AI open new frontiers. I’ve had success with assistants that proactively summarize account health, suggest next actions, or auto-draft content using the customer’s tone. But I always ship with transparency (“Why am I seeing this?”), controls (easy snooze or opt-out), and fallbacks (graceful degradation if signals are sparse). The human is still the hero; AI should play the role of a reliable, explainable copilot.

My implementation roadmap follows a crawl‑walk‑run arc. Crawl: rules‑based personalization in one journey; clear metrics and opt‑out. Walk: contextual recommendations using embeddings and feedback loops; continuous A/B testing. Run: agentic workflows that take multi‑step actions with approval gates and audit trails. Each phase is gated by evidence, not enthusiasm.

Finally, I treat personalization as a living system. I review dashboards weekly, continuously prune features that add complexity without durable lift, and socialize learning across product trios and empowered product teams. When personalization stays grounded in outcomes, ethics, and craftsmanship, it stops feeling “creepy” and starts feeling inevitable.

Personalization is not a stunt; it’s a capability. Build it with intention, measure with rigor, and earn the right to go deeper over time.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Why Winning Product Teams Obsess Over the First 5 Minutes to Drive Retention and Growth

The first five minutes a new user spends in a product set the trajectory for everything that follows. In my experience, that brief window determines activation, early retention, and ultimately whether product-led growth compounds—or stalls. That’s why I obsess over it, instrument it deeply, and treat it as the highest-leverage part of the product experience.

Learn how data-driven teams optimize the first 5 minutes of product experience to improve activation, retention, and growth—and how they do it with Amplitude.

Here’s the practical reason the first five minutes matter so much: users are deciding whether your value proposition translates into an immediate “aha moment.” If time-to-value is long or the path is confusing, activation rate drops, retention curves decay faster, and every subsequent dollar of acquisition becomes less efficient. When we design onboarding intentionally, we shorten the cognitive distance to that first success and build habits that sustain retention.

My playbook starts with measurement. I use Amplitude analytics as a unified analytics platform to instrument the first-run experience end to end, define a clear activation event, and track the user’s journey with funnels, cohorts, and retention analysis. That clarity lets me see where friction spikes, where users hesitate, and which paths correlate with long-term engagement. Without that visibility, changes to onboarding are guesses rather than decisions.

From there, I run disciplined A/B testing. We establish a minimum detectable effect (MDE) based on traffic and variance, and we prioritize experiments that reduce effort to reach the first outcome: simplifying sign-up, clarifying the primary CTA, or pre-seeding a workspace with smart defaults. When we can quantify impact on early activation and downstream retention cohorts, the team can make confident trade-offs and move faster.

Guidance within the product is just as important as the flow itself. Thoughtful UX writing, contextual tooltips, and concise in-app guides should highlight the one or two actions that create immediate value—not overwhelm with a product tour that tries to teach everything at once. The goal is a path to progress, not a lecture. When we pair minimal friction with timely cues, users self-propel to value.

I still remember watching session replays of new users pausing at a crowded first screen. That moment reshaped our approach: fewer choices, clearer hierarchy, and progressive disclosure. The result was a meaningful lift in activation and steadier retention curves. It reinforced a simple truth—when the first five minutes feel effortless, users stick around to explore everything else.

This is also an organizational discipline. Empowered product teams—PM, design, and engineering working as a product trio—align on outcomes vs output OKRs and treat the first five minutes as a shared responsibility. We close the loop with customer feedback, run rapid product discovery, and bring forward deployed engineers into research to shorten the distance between insight and iteration.

If you’re getting started, focus on five moves: instrument the first-run journey in Amplitude analytics; define and track a crisp activation event; analyze funnels and retention cohorts to locate friction; ship weekly A/B tests with a sensible MDE; and iterate your onboarding with lightweight product tours, tooltips, and in-app guides. Tie improvements to leading indicators of product-led growth so the impact is visible to stakeholders across go-to-market and product.

The obsession with the first five minutes isn’t dogma—it’s a commitment to user success. When we reduce friction, spotlight value, and measure what matters, activation climbs and retention compounds. And with the right analytics foundation, we can make those first few moments predictably great, not accidentally good.

Inspired by this post on Amplitude – Best Practices.

October 24, 2025
Why Retention Wins: The Ultimate Product Strategy to Shape Your Roadmap and Ignite Growth

I keep coming back to one simple truth in product management: Retention Is the Ultimate Product Strategy. When customers stay and expand, it signals that we are repeatedly solving real problems with a value proposition strong enough to withstand time, alternatives, and change.

Retention reveals if your product delivers lasting value. Learn how top product leaders use it to guide strategy, shape roadmaps, and drive growth.

At HighLevel, I treat retention as the clearest signal of product-market fit quality and the most reliable compass for product-led growth. I review retention weekly, cohort it by segment and plan, and tie it directly to value moments in onboarding and activation. If we can’t see where users succeed (or stall), we can’t shape a roadmap that consistently compounds value.

Here is how I put retention at the center of product strategy. When cohorts are strong, I double down on the experiences and workflows that create habit loops and advocacy. When cohorts drop, I stop chasing surface-level outputs and run focused product discovery to clarify the value proposition, reduce time-to-first-value, and reset outcomes vs output OKRs so teams are solving for the right problems.

I then translate retention insights into product roadmapping and sprint planning. Every roadmap theme must map to a retention driver: faster activation, deeper engagement, or expanded breadth of use. I use A/B testing to validate critical UX decisions, and I guard against false positives by aligning experiments to business outcomes tied to retention, not just clicks or vanity metrics.

Instrumentation matters. I rely on Amplitude analytics to trace the path from first touch to recurring value, measuring drop-offs, leading indicators of habit formation, and usage cliffs by persona. With clean event data, I can connect improvements in onboarding to cohort lift and quantify what features actually move long-term retention, not just short-term engagement.

Most retention gains come from the “boring but pivotal” basics: a frictionless onboarding flow, clear in-product guidance, and a crisp path to the first “aha” moment. I continually refine these with targeted improvements, then reinforce them with contextual education and lifecycle touchpoints that help customers unlock the next milestone of value.

I also segment retention to find hidden opportunities. Different plans, industries, and team sizes have distinct activation thresholds and success criteria. By tailoring experiences and success metrics per segment, we avoid one-size-fits-all decisions and build for real-world diversity while still maintaining a coherent roadmap.

Culturally, retention is how I keep product management leadership grounded. It forces ruthless prioritization, sharpens stakeholder conversations, and aligns teams on outcomes. When teams see their work reflected in month-over-month cohort lift, motivation rises—and so does our confidence in the strategy.

If you’re looking to operationalize this approach, start with a baseline retention analysis, define your key value moments, align a handful of outcomes vs output OKRs to activation and engagement, instrument the journey in Amplitude analytics, and prioritize one or two onboarding improvements that shorten time-to-first-value. Ship, measure, and iterate. Over time, this creates a roadmap that writes itself from the evidence of durable customer value.

Inspired by this post on Amplitude – Best Practices.

October 24, 2025
Build a Fearless Culture of Experimentation: How I Turn Tests into Teamwide Habits

I’ve learned the hard way that experiments stall when they’re treated like items to check off a backlog. Real impact shows up when experimentation becomes the way we think, plan, and decide—every day, across the entire product organization.

Successful experimentation isn't just about adopting new tools or running more tests. It’s about changing company culture.

At HighLevel, I anchor experimentation in outcomes, not output. We form product trios and empower product teams to own the problem, link work to outcomes vs output OKRs, and commit to fast learning loops. This isn’t about more activity; it’s about better decisions, tighter focus, and measurable customer value.

Our teams write crisp hypotheses, define decision rules up front, and set a minimum detectable effect (MDE) before any A/B testing begins. That small discipline prevents “result fishing,” speeds up decisions, and aligns everyone on what will constitute a real signal versus noise.

Tooling helps, but only when it serves the culture. We instrument experiences end-to-end, lean on Amplitude analytics within a unified analytics platform, and run retention analysis alongside acquisition metrics so we don’t celebrate shallow wins. The goal isn’t dashboards; it’s actionable insight that improves product-market fit lessons and informs the next iteration.

Rituals make the culture durable. We review experiments weekly, tie learnings back to OKRs during QBRs, and celebrate invalidated hypotheses as progress. That psychological safety turns “being wrong” into momentum, reinforcing product management leadership behaviors we want to scale.

We also invest in decision hygiene: clear problem statements, pre-registered success criteria, and simple templates that make it easy to do the right thing quickly. Over time, this reduces debate theater and increases the surface area for discovery—more time with customers, more signals, and more conviction in our bets.

If you’re starting from scratch, begin small: pick one critical journey, articulate a hypothesis, choose a primary metric and MDE, run a lean A/B test, decide ahead of time how you’ll act on outcomes, and close the loop publicly. Repeat that cadence until it becomes muscle memory. That’s how experiments stop being one-off projects and start compounding into product-led growth.

When experimentation is a culture, not a task, teams move faster, leaders make clearer tradeoffs, and customers feel the difference. That is the habit I continue to build—one hypothesis, one decision rule, and one learning loop at a time.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Build vs. Buy in Experimentation: Why Embracing Vendors Accelerates Real Innovation

For much of my career, I reflexively favored building experimentation tooling in-house. Over the last few years, I’ve changed my mind. The ecosystem has matured, the bar for statistical rigor has risen, and the opportunity cost of reinventing the wheel has become too high to ignore. Read why the industry has changed to more broadly embrace vendor solutions—and why that's a good thing for innovation.

The short version: buying core experimentation capabilities increasingly lets us learn faster, reduce risk, and focus scarce engineering cycles on true differentiation. I still believe in building when it creates competitive advantage, but I’ve seen too many teams burn time on “table stakes” infrastructure instead of delivering outcomes that matter.

When I evaluate build vs. buy, I start with two questions: Is this capability a point of parity or a source of competitive differentiation? And what is the real total cost of ownership over three years, including staffing, maintenance, on-call, compliance, roadmap drag, and delayed time-to-learning? Most experimentation platforms are now points of parity; the differentiation is how quickly and responsibly we learn, not whose statistics package we forked.

Modern experimentation isn’t just a split URL test. It demands identity resolution across devices, reliable bucketing, exposure logging at scale, edge delivery for flags, guardrail metrics, and rigorous methods like minimum detectable effect (MDE), CUPED, and sequential testing. Add privacy requirements, data governance, and auditability, and the platform burden grows beyond a “quick internal tool.” This is exactly where vendors have pulled ahead, baking in best practices we’d otherwise relearn the hard way.

There are still good reasons to build. If you operate under unique latency constraints (e.g., sub-20ms decisions at the edge), have non-negotiable regulatory boundaries, or your experimentation model is deeply coupled to proprietary ML systems, bespoke tooling can be justified. I’ve supported builds in those cases—but only with a clear plan for long-term ownership, documentation, and explicit trade-offs.

More often, buying is the sane default. Vendor solutions give us hardened SDKs, consistent flagging, proven stats engines, and integrations with analytics—freeing teams to spend their energy on high-quality hypotheses and better product discovery. Connecting experiment outcomes to a unified analytics platform (and tools like Amplitude analytics) helps us align on source-of-truth metrics, tighten feedback loops, and empower product trios to make confident, outcome-driven decisions.

A hybrid approach frequently wins: buy the platform core, then extend it. Build custom decisioning services where needed, enrich telemetry, and add domain-specific metrics on top. I’ve had success pairing vendor platforms with forward deployed engineers and thoughtful developer evangelism to create the best of both worlds—speed from the vendor, nuance from our domain.

If you’re considering a shift, here’s the adoption playbook I use: – Define success upfront: decision latency targets, MDE guidance, guardrail metrics, governance needs, and privacy constraints. – Run a time-boxed pilot with an A/A test and a handful of A/B testing use cases. Validate exposure logging, bucketing stability, and metric parity against your analytics stack. – Align on outcomes vs output OKRs, so “more experiments” is never the goal; better decisions are. – Establish data governance and metric definitions before full rollout. Treat metrics as a product, not a spreadsheet. – Invest in enablement: in-app guides, product tours, and training for PMs, engineers, and analysts. Proactive stakeholder management is what separates a successful rollout from shelfware.

AI is accelerating this shift. Gen AI for product prototyping and agentic AI assistants can help generate hypotheses, auto-suggest experiment designs, and flag risky rollouts in real time. Pairing AI with a robust experimentation backbone improves both velocity and quality—without asking teams to become statisticians overnight.

My bottom line: the industry’s embrace of vendor experimentation platforms is not a retreat from craftsmanship—it’s a strategic allocation of talent. By buying where the market is excellent and building where our differentiation truly lives, we learn faster, reduce risk, and compound innovation. If you haven’t revisited your build vs. buy calculus recently, now is the time. Your customers don’t reward you for owning a stats engine; they reward you for shipping better outcomes, sooner.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Stop Trusting Static A/B Test Calculators: Why You Need Dynamic MDE Curves Over Time

After years of running experiments at scale, I’ve learned that the quickest way to stall product momentum is to rely on static A/B test calculators that promise certainty from a single sample size number. Real-world data rarely behaves like those calculators assume, and that gap quietly erodes decision quality, speed, and stakeholder trust.

Read about the issues with current A/B test calculators and why experimenters need to see a range of MDEs over time, not a static sample size

Most calculators hard-code fragile assumptions: a constant baseline conversion rate, balanced traffic allocation, independent and identically distributed sessions, no seasonality, no peeking, no novelty effects, and a fixed-horizon stop. They often use normal approximations that break at low counts and ignore the realities of traffic ramping, SRM (sample ratio mismatch), and mid-test product updates. The result is a deceptively precise sample size that fits the math, not the environment.

In practice, product teams peek, traffic fluctuates by day of week, acquisition mixes shift, and funnel variance changes as users move from click to activation to retention. These conditions make “the” required sample size a moving target, not a constant. Treating a static figure as a guarantee leads to underpowered tests, false confidence, and rushed stops that inflate false positives.

The alternative is to manage Minimum Detectable Effect dynamically. Instead of anchoring on a single number, I plan with a range of MDEs over time—power curves that show what lift we can reliably detect after 3, 7, 14, and 28 days as traffic accrues. This reframes the question from “How big should my sample be?” to “What effect sizes can we detect at each decision point given our forecasted traffic and variance?”

At HighLevel, this approach changed our experimentation culture. For example, an onboarding flow test initially “required” three weeks according to a static calculator. Our MDE-over-time view showed we could detect a meaningful 4–6% lift within a week under expected weekday traffic, but only 8–10% on weekends due to volatility. We set a sequential schedule for interim checks, aligned stakeholders on stopping rules, and made a confident call in nine days—saving a sprint and avoiding a premature rollback.

Implementing dynamic MDEs is straightforward: forecast traffic by day, estimate variance from historical data, and simulate power curves across relevant effect sizes. Layer in sequential testing or Bayesian monitoring to avoid p-hacking, include guardrail metrics (e.g., latency, error rates, SRM), and publish an MDE band that updates as data arrives. This transforms your “calculator” into a living decision tool rather than a one-time estimate.

For teams using a unified analytics platform or tools like Amplitude analytics, it’s simple to automate: generate daily MDE curves, annotate ramp changes and seasonality, and expose a dashboard that tracks detectable lift as a function of time and traffic. Pair this with pre-registered stopping rules and a simple communication routine so stakeholders know exactly when and why you’ll decide.

Beyond top-of-funnel conversion, this mindset is critical for retention analysis and revenue outcomes where effects materialize over weeks or months. Plan MDE bands per horizon—early activation, Day-7 retention, and longer-term LTV—so product discovery and product-led growth bets aren’t prematurely judged on the wrong timeline.

The takeaway is simple: retire the illusion of a one-number sample size. Embrace dynamic MDE curves that reflect how your data actually behaves, make faster and more confident calls, and keep empowered product teams focused on outcomes over outputs. Your experiments—and your roadmap—will move with more speed, less drama, and far better signal.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Inside Japan’s AI Marketing Shift: How 500 Teams Boost Efficiency, Results, and Careers

I just finished reviewing new findings on Japan’s marketing landscape, and the signal is clear: AI isn’t just a shiny tool—it’s a force multiplier for outcomes and careers. The headline that caught my attention, "Amplitude Releases New Research in Japan: Marketers are Unlocking Efficiency, Results, and Career Growth," aligns with what I’m seeing on the ground: teams that blend disciplined analytics with pragmatic AI adoption are pulling ahead.

Amplitude released a new survey of 500 Japanese marketers, which reveals how teams are benefiting from AI. Get the insights from the data

Here’s how I interpret the shift. AI accelerates the cycle from insight to action when it’s grounded in a unified analytics platform. With Amplitude analytics stitched into campaign and product signals, marketers can move beyond vanity metrics to diagnose true drivers of activation, engagement, and retention. That’s where efficiency compounds: fewer blind spots, faster iteration, and clearer attribution of what actually drives results.

On the strategy side, I’m seeing two dominant patterns. First, gen ai is speeding up creative workflows—audience research, message testing, and content generation—without sacrificing brand rigor. Second, agentic AI is emerging in operational loops: routing leads, prioritizing segments, and suggesting next-best actions based on behavioral data. The common denominator is data governance; without clean event schemas and consent-aware pipelines, AI amplifies noise instead of signal.

For product-led growth motions, this research validates what empowered product teams have practiced for years: instrument the customer journey, frame outcomes vs output OKRs, and experiment in short, learnable cycles. When marketing, product, and data join forces as true product trios, teams can run in-app guides and product tours, tune onboarding, and perform rigorous retention analysis that ties growth to product value rather than spend.

My playbook in this environment is simple but disciplined. Start with first principles decision making: define the problem, the decision, and the evidence required. Use a unified analytics platform to connect lifecycle events across acquisition, activation, and expansion. Align go-to-market strategy with product roadmapping and sprint planning, so insights move directly into experiments—not slide decks. Then close the loop with clear outcome metrics and QBRs that reward learning velocity, not activity volume.

There’s also a career arc embedded in this shift. Marketers who cultivate analytical fluency and AI literacy are becoming indispensable partners to product management leadership. They can articulate a differentiated value proposition, shape product positioning with live behavioral data, and influence board-level narratives with credible, causal evidence. That combination—story plus signal—unlocks both performance and professional growth.

My commitment going forward is to operationalize these lessons: tighter event taxonomy, sharper outcomes framing, and more systematic experimentation across channels and in-product touchpoints. With the right data foundation and a pragmatic AI strategy, we can convert curiosity into capability—and capability into repeatable growth.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
AI Raised the Bar on Experimentation: How I Drive Product Growth with Relentless Tests

The AI era didn’t just speed up product development—it rewired the economics of learning. Experiments that once took weeks now take hours, and the organizations that compound learning faster are the ones outpacing competitors. In my role guiding product strategy, I’ve seen this shift firsthand: velocity is table stakes; evidence is the differentiator.

Learn why market dynamics prove that experimentation is fundamental to driving growth in the age of AI.

When AI compresses build and distribution cycles, market feedback arrives in torrents. That abundance of feedback is valuable only if we can transform it into trusted insight. I anchor every initiative with a clear hypothesis, a measurable outcome, and a pre-committed decision rule—what we’ll do if the result is positive, negative, or inconclusive. This discipline converts experimentation from a set of ad hoc activities into a repeatable growth engine.

Data quality is non-negotiable. I rely on a unified analytics platform, pairing event instrumentation with Amplitude analytics to analyze activation, retention, and long-term impact. Strong data governance prevents metric drift and ensures that our “go/no-go” calls rest on sound evidence. Retention analysis, in particular, is my north star for separating novelty spikes from durable value.

Gen AI has transformed how quickly we can explore solution space. I use gen ai for product prototyping to generate multiple UX and copy variants in minutes, then deploy in-app guides and lightweight product tours to validate which concepts resonate. This dramatically lowers the cost of curiosity: we test more, earlier, with tighter feedback loops—without compromising user experience or brand voice.

Process and culture make this sustainable. Empowered product teams—tight product trios across Product, Design, and Engineering—run weekly sprints with explicit outcomes vs output OKRs. We plan small, falsifiable bets in product roadmapping and sprint planning, stack-ranked by expected impact and learning value. The result is a team that ships with confidence, measures with rigor, and iterates without ego.

Experimentation doesn’t stop at UX. I extend the same approach to go-to-market strategy and product-led growth motions: pricing page changes, onboarding flows, paywall copy, and packaging tests all roll through the same hypothesis-measure-decide loop. We bias toward reversible decisions, emphasize speed to signal, and codify what we learn into playbooks the whole organization can reuse.

Raising the bar on experimentation means raising the bar on clarity. Every test should answer a specific question, earn its way onto the roadmap, and connect to a value proposition we can defend. In a world where AI collapses time, the advantage goes to teams that compound learning with integrity and purpose. Start small, instrument well, close the loop—and let the data guide the next bold move.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025