Tag: A/B testing

How I Scale Revenue with Pendo Predict: Cut Costs, Reduce Risk, and Drive Product Adoption

When my team and I set out to accelerate growth without ballooning costs, we leaned into Pendo Predict as a keystone of our product-led growth strategy. Predict gives us a practical, data-driven way to focus on the right users at the right moments, align teams around measurable outcomes, and turn product usage signals into revenue impact.

“Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.” That statement maps exactly to how we operate: we use the platform to understand user behavior, guide users through high-value actions, and instrument the experience so we can learn, iterate, and scale with confidence.

To scale revenue, we identify high-intent segments based on product behaviors and run targeted in-app guides and product tours that shorten time-to-value and boost conversion. Predict helps us surface which features correlate with expansion and retention, so our onboarding flows nudge users into those paths. This approach compounds: better activation drives stronger engagement, which fuels a healthier pipeline for cross-sell and upsell.

On the cost side, we reduce support load with contextual guidance—tooltips, checklists, and just-in-time education—so customers self-serve through common friction points. We consolidate insights in a unified analytics platform, enabling product, success, and go-to-market teams to work from the same source of truth. The result is fewer reactive escalations, tighter prioritization, and more engineering time invested in features that move retention and revenue.

Risk reduction comes from visibility and control. With predictive signals and retention analysis, we spot churn risk early, intervene with timely in-app messaging, and de-risk launches by rolling out features to targeted cohorts while monitoring adoption and engagement. We pair this with disciplined experimentation and A/B testing to validate changes before scaling broadly.

If you’re considering a similar motion, a simple playbook works: define your adoption and engagement metrics, instrument key workflows, create predictive segments, ship focused in-app guides, and measure impact against outcomes—not just outputs. Over time, this turns your product into a durable growth engine that consistently improves user experience and business performance.

Inspired by this post on Pendo – Best Practices.

October 24, 2025
How AI Is Supercharging Product Design: Faster Prototyping, Smarter Testing, and Better UX Outcomes

AI has fundamentally changed how I lead design and testing, not by replacing craft, but by compounding it. When my teams pair generative models with time‑tested product management practices, we move faster, learn sooner, and ship with more confidence—without compromising privacy-by-design or quality. The result is a tighter loop from product discovery to product-market fit lessons. Learn how Pendo’s product design team is using genAI and traditional tools to speed up design and development. That single line captures my own operating model: blend genAI with established toolchains to accelerate, not shortcut. In practice, I treat AI as a force multiplier for product trios—PM, design, and engineering—so empowered product teams can explore broader solution spaces while staying anchored to outcomes vs output OKRs. In discovery, genAI helps me synthesize qualitative inputs at scale—interviews, support threads, and in-app behaviors—into testable opportunity statements. I triangulate those insights with a unified analytics platform and Amplitude analytics to spot friction, then use in-app guides and product tours to target learning, recruit the right cohorts, and validate problems before we overbuild. For prototyping, gen ai for product prototyping is a game-changer. I generate multiple UX writing variants, microcopy, and flows in minutes, then narrow the set using heuristics and stakeholder feedback. Before any A/B testing, we precompute the minimum detectable effect (MDE) and sample size, making sure our experiments are powered to detect meaningful differences, not noise. In testing, I combine classic A/B testing with AI-assisted analysis to surface patterns faster. GenAI drafts experiment summaries, flags anomalous segments, and proposes follow-up tests, while my team makes the final calls. We deploy targeted in-app guides to onboard users into trials, monitor adoption via event telemetry, and iterate quickly until the value proposition is unmistakable. Execution depends on rigor and guardrails. We codify AI risk management and data governance policies, keep humans-in-the-loop for critical judgments, and log model prompts and outputs for auditability. This lets us move with speed and integrity, aligning stakeholder management, product roadmapping and sprint planning, and go-to-market strategy around measurable outcomes. The payoff is material: shorter cycle times, clearer value narratives, and stronger product-led growth curves. By fusing genAI with traditional practices, we preserve the craft of design while scaling our capacity to learn. That’s how we differentiate—through faster insight generation, smarter testing, and experiences that feel unmistakably intuitive.

Inspired by this post on Pendo – Best Practices.

October 24, 2025
The Real Reason Pendo Built Agent Analytics—and How It Drives Adoption, Revenue, and Trust

I’ve learned the hard way that the toughest part of launching in-app agents and guided experiences isn’t the build—it’s proving, quickly and credibly, that they move the business. If I can’t quantify adoption, engagement, deflection, and time-to-value, stakeholder confidence erodes and iteration slows. That’s exactly why an Agent Analytics capability matters: it turns opaque interactions into measurable outcomes that product, customer success, and engineering can all act on.

When I evaluate a capability like Agent Analytics, I anchor on a few questions. Which segments adopt the agent, and where does engagement drop? What fraction of issues are successfully deflected versus escalated? Which prompts, product tours, and in-app guides drive conversion and retention—and which add friction? How does agent usage correlate with onboarding completion, core feature activation, and long-term retention analysis? If I can answer those with a unified analytics platform, I can prioritize confidently.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

In practice, I map an outcomes-first measurement plan: define a north-star (e.g., activated accounts), articulate contributing metrics (guide completion rate, agent task success, session depth), then run targeted A/B testing on copy, timing, and placements. With the right analytics, I can compare cohorts exposed to in-app guides and product tours against a control, validate impact, and double down on the patterns that consistently improve adoption and stickiness.

Cost and risk are just as important as growth. An effective Agent Analytics view helps me model support deflection, time-to-resolution, and escalation rates so I can quantify cost savings without sacrificing quality. On the risk side, I look for early-warning signals—low-confidence responses, repeated handoffs, or anomalous usage—so I can intervene before they turn into churn or brand concerns. The point isn’t vanity metrics; it’s operational clarity that enables responsible, scalable product-led growth.

This also changes team dynamics. Product trios get a shared source of truth for decisions, engineering gains sharper specs informed by real behavior, and customer-facing teams can see which experiences reliably unlock value for each segment. Instead of debating opinions, we iterate on evidence—tightening the loop between product roadmapping and sprint planning, UX writing, and go-to-market strategy.

My 90-day playbook looks like this: establish a baseline for adoption and engagement; instrument agent interactions end to end; ship two or three small, high-leverage experiments in onboarding and help experiences; and review results in weekly rituals. By day 90, I expect to see a clear line from agent engagement to activation and retention, along with a repeatable testing cadence that compounds learning.

I’ve seen the same pattern across products and markets: once teams illuminate the black box of in-app assistance with rigorous, actionable analytics, customer confidence rises, onboarding accelerates, and roadmaps get sharper. If you’re evaluating Pendo or already running it, put Agent Analytics at the center of your measurement strategy—and let your data, not assumptions, guide the next iteration.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Implementing Agentforce the Smart Way: My Proven Playbook for Salesforce Agentic Success

Implementing Agentforce isn’t a feature rollout—it’s a strategic shift. In my role building AI-driven products, I treat Agentforce as its own product with clear outcomes, rigorous governance, and disciplined iteration. The objective is to create durable operational leverage inside Salesforce without compromising trust, data integrity, or customer experience.

Learn the ways in which Pendo helps companies design and iterate on their agentic strategy for Salesforce.

I start with product discovery. That means selecting the right use cases, defining the target user, and aligning on measurable outcomes rather than outputs. In practice, I prioritize use cases across sales, service, and marketing using an impact–effort–risk lens, then set crisp success metrics—response time, deflection rate, case resolution, win rate lift, and user adoption. This keeps everyone focused on value creation, not just model novelty.

Next, I design the agentic system with guardrails. I specify agent roles, tools, and policies; define when to escalate to humans; and embed privacy-by-design and data governance from day one. I also build an evaluation harness with offline tests and live A/B testing, ensuring we have a minimum detectable effect that’s meaningful for the business. The goal is to measure outcomes reliably and course-correct quickly.

When building the first slice, I scope narrow and ship fast. For example, start with a constrained service workflow—classify the case, propose a response, and take a safe action—with clear affordances in Salesforce so users understand what the agent did and why. I instrument the experience end-to-end and use Pendo for in-app guides, surveys, and behavioral analytics to reduce onboarding friction and capture real-time feedback at scale.

Iteration is where value compounds. I run weekly reviews of conversations, error taxonomies, and edge cases; adjust prompts and tool access; and maintain a steady experiment cadence. We track outcomes vs output to avoid vanity metrics, and we document learnings to de-risk the next use case. This steady drumbeat builds credibility with stakeholders and confidence with frontline users.

Change management is non-negotiable. I align leaders early, set expectations on what the agent can and cannot do, and define SLAs for humans-in-the-loop. I use product tours to teach new behavior, highlight quick wins, and establish transparent feedback channels. This combination of enablement and accountability accelerates adoption and creates a culture that embraces agentic AI responsibly.

Finally, I scale thoughtfully. Once the first use case demonstrates value, I standardize patterns, unify analytics, and evolve governance as usage grows. I review risk regularly, align OKRs with the roadmap, and keep a tight feedback loop between product, ops, and go-to-market teams. Treating Agentforce as an evolving product—not a one-off project—maximizes impact while protecting the customer experience.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Our Pendo-Powered Playbook: Orchestrating a High-Impact Summer Release with Product-Led Growth

We set out to promote the Pendo Summer Release using the most authentic approach possible: we used Pendo to market Pendo. That decision anchored our strategy in product-led growth, letting us reach users in context, guide them through new capabilities, and measure impact in real time without adding friction or cost.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

Our objectives were clear: drive adoption of new features, accelerate onboarding for existing customers, and improve engagement across key workflows. We framed the work with outcomes vs output OKRs, clarified the value proposition for each persona, and aligned our product positioning to highlight points of parity and genuine differentiation.

Execution centered on in-app guides, product tours, and purposeful tooltip design. We segmented by role, lifecycle stage, and behavior to keep messages timely and relevant, then layered in A/B testing with a defined minimum detectable effect (MDE) so we could learn fast without overexposing users. Product trios partnered closely with design and forward-deployed engineers to iterate quickly on copy, UX writing, and guide placement.

On the measurement side, we instrumented clear goals and tracked conversions through the funnel, pairing event analytics with retention analysis to understand depth of usage, not just clicks. We captured qualitative signal through micro-surveys and in-context feedback, feeding insights back into product roadmapping and sprint planning to sharpen our next set of in-app experiments.

Governance mattered as much as growth. We applied privacy-by-design principles, ensured strong data governance, and kept stakeholder management tight so each guide had a clear owner, sunset plan, and success criteria. That discipline helped us sustain momentum without cluttering the experience.

The biggest lesson: when done thoughtfully, in-app education scales like a dedicated success team—at a fraction of the cost—while teaching you exactly where users find value. This Pendo-powered launch playbook now underpins our onboarding, cross-sell motions, and QBRs alike, giving us a repeatable way to promote releases, validate hypotheses, and deepen engagement with every iteration.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Inside Pendo’s Decision: Replacing the Website Chatbot With an AI Agent to Boost ROI

Traditional website chatbots promised instant answers but rarely delivered the depth, context, and actionability modern buyers expect. After seeing patterns of high drop-off and shallow engagement, I stepped back and reframed the problem: We did not need another scripted bot—we needed an AI Agent capable of understanding intent, personalizing responses, and taking meaningful actions in the flow of discovery.

That is why Pendo replaced the website chatbot with an AI Agent. From a product management lens, the decision hinged on three criteria: accelerate time-to-value for visitors, reduce operational overhead through automation, and improve the quality of demand captured at the top of the funnel. An agentic AI approach met all three.

Increase revenue, cut costs, and reduce risk with Pendo’s Software Experience Management platform. Optimize the entire software experience to drive adoption and improve engagement.

This statement crystallizes the business case. An AI Agent can translate product intent into measurable outcomes by connecting to knowledge sources, analytics, and workflows. Instead of handing off a prospect to a form or a static knowledge article, the agent can surface relevant guidance, qualify interest, book meetings, and even trigger product tours—closing the loop between marketing, product, and customer success.

We anchored the implementation in data governance and privacy-by-design. That meant carefully curating training corpora, instituting role-based access controls, applying guardrails for sensitive topics, and designing graceful human-in-the-loop fallbacks. The result was not just a smarter front door, but a safer one—critical for regulated buyers and enterprise stakeholders.

To validate impact, we ran disciplined A/B testing with a clearly defined minimum detectable effect across conversion, engagement depth, and time-to-response. We also monitored secondary signals such as escalation rate to human support, session quality, and downstream product adoption. Early signals showed more qualified conversations, fewer dead ends, and faster paths to value—exactly the outcomes a product-led growth motion requires.

The experience uplift did not stop at the website. By aligning the agent with in-app guides and product tours, we created continuity from pre-signup exploration to onboarding and activation. Visitors received consistent, contextual help before and after they became users, which strengthened our product positioning and reduced friction across the journey.

Operationally, the shift lowered the marginal cost of each high-quality interaction while improving reliability. Agent handoffs to sales or support became intentional rather than reactive, and insights from conversations fed directly into product discovery. That closed feedback loop informed roadmap decisions and sharpened our go-to-market strategy.

If you are considering a similar move, start with a clear AI Strategy tied to measurable outcomes, a robust governance model, and a pragmatic rollout plan. Focus the agent on high-intent moments first, surround it with analytics and experimentation, and let the data guide expansion. The goal is not to replace humans—it is to elevate them by letting the AI Agent handle the repetitive, high-volume work so your teams can focus on complex, high-value interactions.

Inspired by this post on Pendo – Perspectives.

October 24, 2025
Ultra‑Personalized AI Product Experiences: How I Push the Limits Without Crossing the Line

Every week I meet teams eager to unleash AI-driven personalization across their products—and I share the same excitement. The promise is magnetic: experiences that feel tailor‑made, delivered at scale, and continuously optimized. Yet sustainable differentiation doesn’t come from turning every dial to eleven; it comes from clarity of intent, responsible design, and disciplined execution.

AI has us on the verge of a new age of ultra-personalized digital product experiences. But don't swing too big too early.

When I think about “how far is too far,” I anchor on user trust, explainability, and measurable value. If a personalization can’t be explained in a sentence, verified through A/B testing, or opted out of without friction, it’s a risk to both brand and product-market fit. The goal isn’t maximal personalization—it’s meaningful personalization that compounds retention and strengthens the value proposition.

I start with product discovery basics: who are the core segments, what jobs-to-be-done matter most, and where does personalization remove friction or accelerate time-to-value? That focus informs pragmatic AI Strategy. Instead of boiling the ocean, I’ll select one high-traffic, high-intent flow and define the precise outcome we want to move. Then I set outcomes vs output OKRs and instrument the path so I can track lift, variance, and trade-offs in real time.

Data governance is non-negotiable. Consent, transparency, and data minimization create the foundation for scalability. I document what signals power personalization, how long they persist, and who can access them. Strong governance isn’t a brake; it’s an enabler, letting us expand confidently without rework or reputational drag.

From there, I validate with A/B testing and clear minimum detectable effect (MDE) thresholds. Holdouts, guardrail metrics, and cohort analyses keep me honest. I’ll use Amplitude analytics to examine funnel impacts, retention analysis, and segment-level effects—especially to ensure we’re not improving conversion while harming long‑term engagement or fairness for smaller segments.

Early wins often come from onboarding and in-app guides. Personalizing the first five minutes—recommended next steps, contextual tooltips, or a tailored product tour—can deliver a step-change in activation with minimal risk. This is where product-led growth shines: relevant, timely nudges that shorten the path to the “aha” moment without feeling intrusive.

As we scale, gen ai and agentic AI open new frontiers. I’ve had success with assistants that proactively summarize account health, suggest next actions, or auto-draft content using the customer’s tone. But I always ship with transparency (“Why am I seeing this?”), controls (easy snooze or opt-out), and fallbacks (graceful degradation if signals are sparse). The human is still the hero; AI should play the role of a reliable, explainable copilot.

My implementation roadmap follows a crawl‑walk‑run arc. Crawl: rules‑based personalization in one journey; clear metrics and opt‑out. Walk: contextual recommendations using embeddings and feedback loops; continuous A/B testing. Run: agentic workflows that take multi‑step actions with approval gates and audit trails. Each phase is gated by evidence, not enthusiasm.

Finally, I treat personalization as a living system. I review dashboards weekly, continuously prune features that add complexity without durable lift, and socialize learning across product trios and empowered product teams. When personalization stays grounded in outcomes, ethics, and craftsmanship, it stops feeling “creepy” and starts feeling inevitable.

Personalization is not a stunt; it’s a capability. Build it with intention, measure with rigor, and earn the right to go deeper over time.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Why Winning Product Teams Obsess Over the First 5 Minutes to Drive Retention and Growth

The first five minutes a new user spends in a product set the trajectory for everything that follows. In my experience, that brief window determines activation, early retention, and ultimately whether product-led growth compounds—or stalls. That’s why I obsess over it, instrument it deeply, and treat it as the highest-leverage part of the product experience.

Learn how data-driven teams optimize the first 5 minutes of product experience to improve activation, retention, and growth—and how they do it with Amplitude.

Here’s the practical reason the first five minutes matter so much: users are deciding whether your value proposition translates into an immediate “aha moment.” If time-to-value is long or the path is confusing, activation rate drops, retention curves decay faster, and every subsequent dollar of acquisition becomes less efficient. When we design onboarding intentionally, we shorten the cognitive distance to that first success and build habits that sustain retention.

My playbook starts with measurement. I use Amplitude analytics as a unified analytics platform to instrument the first-run experience end to end, define a clear activation event, and track the user’s journey with funnels, cohorts, and retention analysis. That clarity lets me see where friction spikes, where users hesitate, and which paths correlate with long-term engagement. Without that visibility, changes to onboarding are guesses rather than decisions.

From there, I run disciplined A/B testing. We establish a minimum detectable effect (MDE) based on traffic and variance, and we prioritize experiments that reduce effort to reach the first outcome: simplifying sign-up, clarifying the primary CTA, or pre-seeding a workspace with smart defaults. When we can quantify impact on early activation and downstream retention cohorts, the team can make confident trade-offs and move faster.

Guidance within the product is just as important as the flow itself. Thoughtful UX writing, contextual tooltips, and concise in-app guides should highlight the one or two actions that create immediate value—not overwhelm with a product tour that tries to teach everything at once. The goal is a path to progress, not a lecture. When we pair minimal friction with timely cues, users self-propel to value.

I still remember watching session replays of new users pausing at a crowded first screen. That moment reshaped our approach: fewer choices, clearer hierarchy, and progressive disclosure. The result was a meaningful lift in activation and steadier retention curves. It reinforced a simple truth—when the first five minutes feel effortless, users stick around to explore everything else.

This is also an organizational discipline. Empowered product teams—PM, design, and engineering working as a product trio—align on outcomes vs output OKRs and treat the first five minutes as a shared responsibility. We close the loop with customer feedback, run rapid product discovery, and bring forward deployed engineers into research to shorten the distance between insight and iteration.

If you’re getting started, focus on five moves: instrument the first-run journey in Amplitude analytics; define and track a crisp activation event; analyze funnels and retention cohorts to locate friction; ship weekly A/B tests with a sensible MDE; and iterate your onboarding with lightweight product tours, tooltips, and in-app guides. Tie improvements to leading indicators of product-led growth so the impact is visible to stakeholders across go-to-market and product.

The obsession with the first five minutes isn’t dogma—it’s a commitment to user success. When we reduce friction, spotlight value, and measure what matters, activation climbs and retention compounds. And with the right analytics foundation, we can make those first few moments predictably great, not accidentally good.

Inspired by this post on Amplitude – Best Practices.

October 24, 2025
Why Retention Wins: The Ultimate Product Strategy to Shape Your Roadmap and Ignite Growth

I keep coming back to one simple truth in product management: Retention Is the Ultimate Product Strategy. When customers stay and expand, it signals that we are repeatedly solving real problems with a value proposition strong enough to withstand time, alternatives, and change.

Retention reveals if your product delivers lasting value. Learn how top product leaders use it to guide strategy, shape roadmaps, and drive growth.

At HighLevel, I treat retention as the clearest signal of product-market fit quality and the most reliable compass for product-led growth. I review retention weekly, cohort it by segment and plan, and tie it directly to value moments in onboarding and activation. If we can’t see where users succeed (or stall), we can’t shape a roadmap that consistently compounds value.

Here is how I put retention at the center of product strategy. When cohorts are strong, I double down on the experiences and workflows that create habit loops and advocacy. When cohorts drop, I stop chasing surface-level outputs and run focused product discovery to clarify the value proposition, reduce time-to-first-value, and reset outcomes vs output OKRs so teams are solving for the right problems.

I then translate retention insights into product roadmapping and sprint planning. Every roadmap theme must map to a retention driver: faster activation, deeper engagement, or expanded breadth of use. I use A/B testing to validate critical UX decisions, and I guard against false positives by aligning experiments to business outcomes tied to retention, not just clicks or vanity metrics.

Instrumentation matters. I rely on Amplitude analytics to trace the path from first touch to recurring value, measuring drop-offs, leading indicators of habit formation, and usage cliffs by persona. With clean event data, I can connect improvements in onboarding to cohort lift and quantify what features actually move long-term retention, not just short-term engagement.

Most retention gains come from the “boring but pivotal” basics: a frictionless onboarding flow, clear in-product guidance, and a crisp path to the first “aha” moment. I continually refine these with targeted improvements, then reinforce them with contextual education and lifecycle touchpoints that help customers unlock the next milestone of value.

I also segment retention to find hidden opportunities. Different plans, industries, and team sizes have distinct activation thresholds and success criteria. By tailoring experiences and success metrics per segment, we avoid one-size-fits-all decisions and build for real-world diversity while still maintaining a coherent roadmap.

Culturally, retention is how I keep product management leadership grounded. It forces ruthless prioritization, sharpens stakeholder conversations, and aligns teams on outcomes. When teams see their work reflected in month-over-month cohort lift, motivation rises—and so does our confidence in the strategy.

If you’re looking to operationalize this approach, start with a baseline retention analysis, define your key value moments, align a handful of outcomes vs output OKRs to activation and engagement, instrument the journey in Amplitude analytics, and prioritize one or two onboarding improvements that shorten time-to-first-value. Ship, measure, and iterate. Over time, this creates a roadmap that writes itself from the evidence of durable customer value.

Inspired by this post on Amplitude – Best Practices.

October 24, 2025
How I’m Readying 11,000 Employees for AI: Role-Specific Training and Human-AI Collaboration

When AI transformation is your mandate at enterprise scale, clarity and pragmatism matter more than hype. My approach to prepare 11,000 employees for AI—with role-specific training, modular design, and human-AI collaboration for better results—rests on three commitments: deliver outcomes tied to real workflows, meet people where they are, and make adoption safer and faster than the status quo.

I start with role-specific training because context beats generic content every time. For product managers, we focus on prompt design for discovery, prioritization signals, and faster hypothesis validation. For engineers, we emphasize code generation quality, test coverage, and secure patterns. For sales and customer success, we build repeatable workflows for research, personalization, and objection handling. Tailoring instruction to each team’s daily work drives confidence, reduces friction, and accelerates time to value.

Modular design is how we scale without sacrificing quality. I break the curriculum into atomic learning units—micro-scenarios, checklists, and in-app guides—that can be remixed into learning paths by role, seniority, and region. This enables just-in-time onboarding, easier updates as gen AI evolves, and localized relevance without reinventing the core. Product tours and embedded nudges reinforce learning in the flow of work, ensuring people practice where the value actually occurs.

Human-AI collaboration is a deliberate practice, not a slogan. We codify co-pilot patterns, checkpoints, and RACI-like ownership so humans remain accountable for outcomes while AI accelerates inputs. Agentic AI is introduced behind guardrails: clear data governance, prompt libraries with approved patterns, verifiable sources, and audit trails. The result is speed and consistency, paired with the trust that leaders and regulators expect.

Change management is where strategy becomes reality. I partner with empowered product teams to co-create playbooks, nominate champions, and sequence rollouts by readiness and impact. We keep a tight feedback loop via office hours, internal communities, and role-based enablement so adoption feels like a product we improve, not a policy we enforce. This is product management leadership applied to culture, not just software.

Measurement keeps us honest. I tie every enablement track to business outcomes—cycle time, win rates, customer satisfaction, and quality—validated through A/B testing where feasible. We monitor adoption, satisfaction, and proficiency, then iterate the content and tooling. When teams see their KPIs move, AI stops being an experiment and becomes part of how we win.

If you’re standing up your AI strategy, start small and specific, ship value fast, and scale through modularity. Role-specific training, modular design, and human-AI collaboration aren’t slogans—they’re a repeatable system for building durable capability across the organization.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Build a Fearless Culture of Experimentation: How I Turn Tests into Teamwide Habits

I’ve learned the hard way that experiments stall when they’re treated like items to check off a backlog. Real impact shows up when experimentation becomes the way we think, plan, and decide—every day, across the entire product organization.

Successful experimentation isn't just about adopting new tools or running more tests. It’s about changing company culture.

At HighLevel, I anchor experimentation in outcomes, not output. We form product trios and empower product teams to own the problem, link work to outcomes vs output OKRs, and commit to fast learning loops. This isn’t about more activity; it’s about better decisions, tighter focus, and measurable customer value.

Our teams write crisp hypotheses, define decision rules up front, and set a minimum detectable effect (MDE) before any A/B testing begins. That small discipline prevents “result fishing,” speeds up decisions, and aligns everyone on what will constitute a real signal versus noise.

Tooling helps, but only when it serves the culture. We instrument experiences end-to-end, lean on Amplitude analytics within a unified analytics platform, and run retention analysis alongside acquisition metrics so we don’t celebrate shallow wins. The goal isn’t dashboards; it’s actionable insight that improves product-market fit lessons and informs the next iteration.

Rituals make the culture durable. We review experiments weekly, tie learnings back to OKRs during QBRs, and celebrate invalidated hypotheses as progress. That psychological safety turns “being wrong” into momentum, reinforcing product management leadership behaviors we want to scale.

We also invest in decision hygiene: clear problem statements, pre-registered success criteria, and simple templates that make it easy to do the right thing quickly. Over time, this reduces debate theater and increases the surface area for discovery—more time with customers, more signals, and more conviction in our bets.

If you’re starting from scratch, begin small: pick one critical journey, articulate a hypothesis, choose a primary metric and MDE, run a lean A/B test, decide ahead of time how you’ll act on outcomes, and close the loop publicly. Repeat that cadence until it becomes muscle memory. That’s how experiments stop being one-off projects and start compounding into product-led growth.

When experimentation is a culture, not a task, teams move faster, leaders make clearer tradeoffs, and customers feel the difference. That is the habit I continue to build—one hypothesis, one decision rule, and one learning loop at a time.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025
Build vs. Buy in Experimentation: Why Embracing Vendors Accelerates Real Innovation

For much of my career, I reflexively favored building experimentation tooling in-house. Over the last few years, I’ve changed my mind. The ecosystem has matured, the bar for statistical rigor has risen, and the opportunity cost of reinventing the wheel has become too high to ignore. Read why the industry has changed to more broadly embrace vendor solutions—and why that's a good thing for innovation.

The short version: buying core experimentation capabilities increasingly lets us learn faster, reduce risk, and focus scarce engineering cycles on true differentiation. I still believe in building when it creates competitive advantage, but I’ve seen too many teams burn time on “table stakes” infrastructure instead of delivering outcomes that matter.

When I evaluate build vs. buy, I start with two questions: Is this capability a point of parity or a source of competitive differentiation? And what is the real total cost of ownership over three years, including staffing, maintenance, on-call, compliance, roadmap drag, and delayed time-to-learning? Most experimentation platforms are now points of parity; the differentiation is how quickly and responsibly we learn, not whose statistics package we forked.

Modern experimentation isn’t just a split URL test. It demands identity resolution across devices, reliable bucketing, exposure logging at scale, edge delivery for flags, guardrail metrics, and rigorous methods like minimum detectable effect (MDE), CUPED, and sequential testing. Add privacy requirements, data governance, and auditability, and the platform burden grows beyond a “quick internal tool.” This is exactly where vendors have pulled ahead, baking in best practices we’d otherwise relearn the hard way.

There are still good reasons to build. If you operate under unique latency constraints (e.g., sub-20ms decisions at the edge), have non-negotiable regulatory boundaries, or your experimentation model is deeply coupled to proprietary ML systems, bespoke tooling can be justified. I’ve supported builds in those cases—but only with a clear plan for long-term ownership, documentation, and explicit trade-offs.

More often, buying is the sane default. Vendor solutions give us hardened SDKs, consistent flagging, proven stats engines, and integrations with analytics—freeing teams to spend their energy on high-quality hypotheses and better product discovery. Connecting experiment outcomes to a unified analytics platform (and tools like Amplitude analytics) helps us align on source-of-truth metrics, tighten feedback loops, and empower product trios to make confident, outcome-driven decisions.

A hybrid approach frequently wins: buy the platform core, then extend it. Build custom decisioning services where needed, enrich telemetry, and add domain-specific metrics on top. I’ve had success pairing vendor platforms with forward deployed engineers and thoughtful developer evangelism to create the best of both worlds—speed from the vendor, nuance from our domain.

If you’re considering a shift, here’s the adoption playbook I use: – Define success upfront: decision latency targets, MDE guidance, guardrail metrics, governance needs, and privacy constraints. – Run a time-boxed pilot with an A/A test and a handful of A/B testing use cases. Validate exposure logging, bucketing stability, and metric parity against your analytics stack. – Align on outcomes vs output OKRs, so “more experiments” is never the goal; better decisions are. – Establish data governance and metric definitions before full rollout. Treat metrics as a product, not a spreadsheet. – Invest in enablement: in-app guides, product tours, and training for PMs, engineers, and analysts. Proactive stakeholder management is what separates a successful rollout from shelfware.

AI is accelerating this shift. Gen AI for product prototyping and agentic AI assistants can help generate hypotheses, auto-suggest experiment designs, and flag risky rollouts in real time. Pairing AI with a robust experimentation backbone improves both velocity and quality—without asking teams to become statisticians overnight.

My bottom line: the industry’s embrace of vendor experimentation platforms is not a retreat from craftsmanship—it’s a strategic allocation of talent. By buying where the market is excellent and building where our differentiation truly lives, we learn faster, reduce risk, and compound innovation. If you haven’t revisited your build vs. buy calculus recently, now is the time. Your customers don’t reward you for owning a stats engine; they reward you for shipping better outcomes, sooner.

Inspired by this post on Amplitude – Perspectives.

October 24, 2025