Category: AI Strategy

  • Secure System Access for AI Agents: A Phased Control Model

    Secure System Access for AI Agents: A Phased Control Model

    An AI agent becomes operationally valuable when it can move beyond explaining a process and complete the underlying work. That same transition gives the agent access to sensitive data and consequential actions, so integration must be designed as both a product capability and a security boundary.

    The practical objective is not maximum access. It is the smallest dependable set of permissions that lets an agent resolve a well-defined workflow, supported by deterministic controls, observable outcomes, and a clear path to human intervention.

    System access changes both the value and the risk

    Without backend access, an agent can describe how to update an account, check a renewal, or report a damaged order. With access to a CRM, billing platform, or order-management system, it can potentially retrieve the relevant record and complete the request during the conversation. The Intercom article presents this shift from answering to acting as a central difference between basic AI adoption and mature deployment.

    The article cites Intercom’s 2026 Customer Service Transformation Report, reporting improved metrics among 87% of teams with mature AI deployments, compared with 62% overall. It also reports that 82% of senior leaders said their teams had invested in AI during the preceding year, while only 10% said they had reached mature deployment. These source-reported figures suggest an integration gap, but they do not independently establish that system access caused the reported improvements or that an integration is secure.

    Security therefore cannot be added after the workflow succeeds. A customer-facing interface may remove the need to visit a separate application, but it must not remove identity and authorization checks. The agent still needs a trustworthy way to associate the request with the correct customer, determine what that customer is permitted to do, and constrain the backend operation accordingly.

    Choose workflows where access justifies its complexity

    Not every automated conversation benefits equally from deeper integration. Intercom reports the results of rebuilding four fixed, scripted Tasks as Procedures with system access. Over the 12 months through May 2026, the reported resolution rate for its bounce-list workflow rose from 9.3% to 79.9%, while bug reporting increased from 9.2% to 66.5%. Email forwarding moved from 44.9% to 66.5%, but Messenger installation rose only from 67% to 69.2%.

    The variation is more instructive than the headline gains. According to the article, the bounce-list process required multi-step reasoning, dynamic branches, and error recovery. Bug reporting still ended in a human handoff, but the procedure improved that handoff by pre-triaging the issue, surfacing possible GitHub matches, extracting relevant URLs, and requesting impersonation access. Messenger installation was already a comparatively linear process, leaving less room for improvement.

    A suitable first integration is therefore not merely a popular support topic. It should be high-volume and repeatable, have an identifiable system owner, and depend on live data or actions that materially change the outcome. Existing APIs improve feasibility, but the security review should also consider data sensitivity, reversibility, authorization complexity, and the consequences of acting on an ambiguous request.

    Use an access ladder instead of a single launch

    The phased approach described by Intercom can also serve as a security model. Each stage expands capability only after the workflow and its controls have produced enough evidence to justify the next step.

    StageAgent capabilityAppropriate useControl emphasis
    No integrationGuide, troubleshoot, check policy, triage, and routeDiscover where explanations repeatedly lead to manual workEvaluate answer quality, routing accuracy, and escalation behavior
    Read-only accessRetrieve approved fields such as order or subscription statusResolve information requests without changing a recordRestrict endpoints, records, and fields; verify customer authorization
    Write accessUpdate records or initiate actions such as cancellations or refundsComplete bounded workflows after earlier stages are dependableValidate inputs, limit action scope, record outcomes, and require approval where consequences warrant it

    Mock responses can test branching logic before an API is ready, as the Intercom article notes. It also proposes a temporary human-in-the-loop step when an integration is still several engineering sprints away. These methods can validate the workflow and expose missing requirements, but simulated success should not be treated as proof that production identity, authorization, failure recovery, and audit controls are ready.

    Put deterministic controls around probabilistic decisions

    Plain-language workflow instructions can guide an agent, but security-critical constraints should not depend solely on the model interpreting those instructions correctly. A safer architecture places enforceable controls between the agent and each backend system.

    ControlPractical design implication
    Dedicated identityGive the agent its own service identity rather than borrowing a staff account, so permissions and activity remain attributable.
    Least privilegeAllow only the endpoints, operations, records, and fields required by the selected workflow.
    Read and write separationKeep retrieval permissions distinct from mutation permissions and grant write access only when the use case requires it.
    Independent policy enforcementValidate identity, authorization, limits, and required inputs outside the model before executing an operation.
    Bounded actionsPrefer narrow, purpose-built operations over unrestricted database or administrative access.
    Human approval and escalationRoute ambiguous, exceptional, sensitive, or difficult-to-reverse cases to an authorized person.
    Auditability and monitoringRecord the request, decision, tool call, result, and escalation so failures and unusual patterns can be investigated.
    Safe failure behaviorPrevent retries, timeouts, or partial completion from producing duplicated or inconsistent changes.

    The integration request should document the workflow in plain language, identify every read and write point, name the system owner, and specify the minimum required fields. It should also define how success and harm will be measured: not only whether the agent completed the conversation, but whether it selected the correct record, performed the authorized action once, protected restricted data, and escalated when it lacked sufficient confidence or permission.

    This framing also improves the business case. Engineering is being asked to expose a narrowly scoped capability with explicit boundaries, rather than to provide broad access to a general-purpose agent. Leadership can then compare measurable workflow value with implementation effort and residual risk.

    Key takeaways

    • System access creates value when it lets an agent complete work, but it simultaneously expands the security boundary.
    • The best initial workflow is frequent, bounded, operationally meaningful, and owned by a team that can approve its data and actions.
    • Progress from no integration to read-only retrieval and then to narrowly scoped write operations; do not treat access as an all-or-nothing decision.
    • Enforce identity, authorization, field restrictions, action limits, and audit logging outside the model’s natural-language instructions.
    • Evaluate correctness, unauthorized-action risk, failure recovery, and handoff quality alongside resolution rate.

    The strongest long-term pattern is a portfolio of small, governed capabilities rather than one broadly privileged agent. Each successful workflow can supply the evidence needed to extend access deliberately, while keeping the consequences of error visible and contained.

    References

  • How Agentic Analytics Reshapes Product Development Roadmaps

    How Agentic Analytics Reshapes Product Development Roadmaps

    Agentic, analytics-driven product development changes the role of product data. Instead of waiting for teams to interpret dashboards and debate a backlog, an agent can help detect behavioral friction, estimate opportunities, propose interventions, and monitor whether a release improves the intended outcome.

    The practical payoff is not an automatically generated roadmap. It is a tighter decision system in which evidence, experiments, delivery controls, and human judgment reinforce one another. The two source articles approach that system from complementary angles: one describes the operating loop around Amplitude Wave, while the other emphasizes the engineering and organizational foundations required to make agentic recommendations dependable.

    The product agent is a decision loop, not a smarter dashboard

    Traditional analytics tools help teams inspect funnels, cohorts, journeys, activation, and retention. The article about Amplitude Wave describes a more proactive model: an agent continuously scans behavioral data for friction, proposes a next-best improvement, supports validation through A/B testing, and uses feature flags to control rollout. After launch, the loop continues by monitoring activation, retention, and downstream revenue rather than treating deployment as the finish line.

    The companion article makes a similar distinction between reporting and agency. It presents agentic systems as capable of proposing, testing, and learning, provided that recommendations remain connected to rigorous behavioral analytics. Synthesized together, the sources describe four linked functions: observation identifies where behavior diverges from an intended journey; prioritization weighs the size, risk, and confidence of an opportunity; experimentation tests whether a proposed change causes improvement; and monitoring determines whether to expand, revise, or retire that change.

    This framing matters because an agent that only generates feature ideas adds another opinion to roadmap planning. An agent that connects ideas to observed behavior, controlled tests, and post-release measurement can instead reduce the distance between a weak signal and a defensible product decision.

    Reliable recommendations depend on an analytics and evaluation stack

    Both sources put instrumentation ahead of automation. The Wave article calls for clearly defined events, models that connect those events to user and account journeys, explicit success metrics, and governance around data quality and privacy. Without that foundation, an agent can produce confident explanations from incomplete or misleading evidence.

    The second article extends the foundation into three technical capabilities. It advocates a unified analytics platform that brings quantitative behavior together with qualitative context, evaluation harnesses that test prompts, policies, and models for regressions, and a retrieval-first pipeline that grounds an agent in trusted organizational information. These layers address different failure modes: analytics establishes what users did, retrieval supplies relevant business context, and evaluations test whether the agent behaves reliably as its components change.

    Interoperability broadens the evidence available to the system. The Wave article points to CRM integration, session replay, and support systems as useful connections for relating product behavior to customer value and go-to-market effects. CI/CD, experimentation tools, and feature flags then connect analysis to controlled delivery. The resulting architecture is less a standalone AI feature than a chain of evidence and controls spanning discovery, development, release, and measurement.

    That chain also establishes a sensible boundary for automation. Behavioral correlations may justify investigation, but they do not by themselves establish causality. A/B testing can provide stronger causal evidence when it is appropriate and well designed; qualitative context can explain why a pattern may be occurring; and human review can catch strategic, ethical, or operational considerations that product telemetry does not represent.

    Roadmaps become portfolios of measurable opportunities

    When agents can surface evidence-backed opportunities, roadmap discussions can move away from ranking requested features in isolation. The unit of planning becomes an outcome-linked opportunity: a behavioral problem, the users or accounts affected, the metric expected to move, the evidence supporting the hypothesis, and the safest way to test it.

    This does not eliminate product strategy. It makes strategy more explicit. Teams still decide which customers and outcomes matter, what constraints apply, and which trade-offs are acceptable. The agent can help maintain a current view of behavioral evidence and shorten the analysis cycle, but it cannot derive organizational priorities from telemetry alone.

    The sources also connect this operating model to empowered product teams, product trios, continuous discovery, and outcomes-versus-output OKRs. In that environment, an agent is best treated as a participant in the discovery and delivery workflow: it can surface anomalies, assemble relevant context, suggest hypotheses, and track results, while the team remains accountable for framing the problem and authorizing consequential decisions.

    The Wave article illustrates the intended scale of intervention with an onboarding example. It reports that an agent identified drop-off around a confusing configuration step; targeted in-app guidance and tooltips were then released behind feature flags, followed by a material improvement in activation with limited engineering effort. The report is a useful illustration of the loop, but it provides no numerical effect size or independent validation. It therefore supports the workflow concept more strongly than any general claim about expected results.

    Governance determines how much autonomy an agent earns

    Automation should expand according to demonstrated reliability and the reversibility of the action. Early implementations can begin in an advisory role, identifying friction and preparing evidence for a team to review. A later stage can allow the agent to configure draft experiments or recommend feature-flag settings. Direct changes to production warrant a higher threshold because errors can affect customers, revenue, privacy, and trust.

    The Wave article explicitly calls for policies governing data use, review thresholds for automated changes, privacy-by-design, and human checkpoints for high-impact decisions. The engineering-focused article complements those controls with eval-driven development, including tests intended to detect reliability and safety regressions across prompts, policies, and models. Together, these ideas suggest that autonomy should be earned through observable performance rather than granted because an agent appears persuasive.

    A practical adoption sequence follows from the synthesis. First, define the outcome and the decisions the agent may inform. Next, verify event quality and journey models before asking the system to prioritize opportunities. Then connect recommendations to a controlled experimentation and release process. Finally, evaluate both product impact and agent behavior, expanding permissions only when the evidence supports it. This sequence keeps the initial scope narrow while creating a path toward a more capable product-development system.

    Key takeaways

    • An agentic product workflow should connect behavioral observation, opportunity prioritization, experimentation, controlled delivery, and post-release measurement.
    • High-quality event data is necessary but insufficient; grounded retrieval, qualitative context, and evaluation harnesses make recommendations more dependable.
    • Roadmaps become more evidence-driven when teams plan around measurable opportunities rather than treating feature requests as predetermined commitments.
    • Human judgment remains essential for strategy, causal interpretation, risk assessment, and high-impact release decisions.
    • Agent autonomy should increase only as evaluations, governance controls, and observed performance justify broader permissions.

    The near-term opportunity is to build a disciplined learning loop before pursuing full autonomy. Organizations that make their data trustworthy, their outcomes explicit, and their release controls measurable will be better positioned to let product agents take on more consequential work without weakening accountability.

    References

  • AI Agent Product Development: From Workflow to Autonomy

    AI Agent Product Development: From Workflow to Autonomy

    AI agent product development is not primarily a model-selection exercise. It is the work of turning a business outcome into a bounded system that can retrieve information, use tools, make decisions, and escalate safely.

    The practical payoff comes from sequencing those capabilities carefully. A focused workflow, explicit measures, controlled access, and continuous evaluation provide a more credible path to value than attempting broad autonomy at launch.

    Key takeaways

    • Define the business outcome and proof of success before choosing prompts, models, or tools.
    • Begin with a repeatable workflow whose inputs, outputs, and failure conditions can be judged clearly.
    • Increase capability in stages: relevant retrieval, limited tools, read-only integrations, controlled actions, and then broader autonomy.
    • Treat privacy, governance, evaluation, observability, and human escalation as product requirements from the beginning.
    • Scale only when operational quality and the intended business outcome remain stable in production.

    Start with a decision contract, not an agent concept

    An agent initiative becomes testable when the team can state what decision or task the system will handle, what information it requires, what it must never do, and how success will be measured. This creates a decision contract between the product, its users, and the organization operating it.

    The supplied source recommends anchoring an AI strategy to one measurable outcome before writing a prompt or selecting a model. It gives lead response time, first-contact resolution, and time-to-first-value as possible measures. Those examples illustrate an important distinction: the agent is a means of changing workflow performance, not the outcome itself.

    This framing also makes AI readiness concrete. Instead of asking whether an organization is generally ready for agents, a product team can examine one workflow: Is the required data available? Are the inputs sufficiently consistent? Can acceptable output be recognized? Are the constraints and escalation conditions explicit? A negative answer identifies product work to complete; it does not automatically call for a more capable model.

    A useful initial scope therefore has clear boundaries and frequent enough repetition to produce evidence. The source identifies support-ticket triage, inbound-lead qualification, and account-note summarization as examples. Their significance is not that every organization should adopt them, but that they offer observable inputs and outputs. That makes errors easier to classify and improvements easier to evaluate.

    Design capability as an autonomy ladder

    The core architectural question is not whether an agent can perform an action. It is what evidence should be required before the product is allowed to perform that action without review. Treating capability as an autonomy ladder gives the team intermediate states between a passive assistant and an unrestricted operator.

    The source proposes a retrieval-first pipeline that introduces only relevant knowledge into the context window. In product terms, retrieval is part of the experience contract: the system should receive the information needed for the task without being burdened by unrelated material. This can improve the conditions for relevant responses, although retrieval does not eliminate the need to evaluate the final behavior.

    Tool access should be similarly bounded. The source recommends a small, explicit tool catalog, with the agent’s role, constraints, and escalation routes documented. It also points to Model Context Protocol as a way to standardize tool invocation across services. Standardization can make integrations more consistent, but it does not decide which tools the agent should receive or what permissions those tools should carry; those remain product and risk decisions.

    Systems of record deserve special caution. The source advises beginning with read-only CRM integration and adding actions only after reliability has been demonstrated. This suggests a practical progression: first observe and recommend, then prepare an action for approval, and only later execute eligible actions within defined limits. Each step creates new failure consequences, so each should have its own evidence threshold.

    Prompt engineering belongs inside this broader capability design. A prompt can express the agent’s role and boundaries, but predictable operation also depends on retrieved context, tool definitions, permissions, timeouts, escalation logic, and the surrounding user experience. Managing only the prompt would leave much of the product’s actual behavior outside the team’s control.

    Make trust an executable product requirement

    Agent risk becomes manageable when broad principles are translated into system behavior. Privacy-by-design should affect what data enters the workflow. Data governance should determine which sources and actions are permitted. Human oversight should appear as an explicit escalation path rather than an informal promise that someone can intervene.

    The source calls for regression evaluations covering safety, accuracy, and bias, alongside logs of agent actions, rate limits, timeouts, and risk scoring for high-impact operations. Together, these controls form a layered safety model. Evaluations test expected behavior before and during release; operational limits constrain runtime exposure; logs support diagnosis and accountability; and risk gates determine when automation must stop or seek approval.

    Uncertainty should also have a designed destination. According to the source, the default response for high-stakes or uncertain situations should be human escalation. A useful handoff needs more than a generic error message: the receiving person should be able to understand the request, the context used, the action considered, and why the system declined to continue. Handoff quality is therefore part of the product experience as well as the risk model.

    This approach avoids treating guardrails as a final compliance checkpoint. When controls are defined alongside workflow requirements, they influence architecture, permissions, interface design, analytics, and release criteria. Trust then becomes something the team can test and operate, rather than a claim attached to the launch.

    Use two evidence loops to decide when to scale

    An agent can appear technically competent without improving the business outcome that justified it. Product development therefore needs two connected evidence loops: one for operational quality and another for workflow impact.

    For operational quality, the source recommends monitoring precision, latency, containment, and handoff quality through agent analytics. These measures answer different questions. Precision concerns whether outputs or decisions are correct enough for the task. Latency affects whether the agent fits the pace of the workflow. Containment indicates how often work remains within the automated path. Handoff quality examines whether escalation preserves context and enables a productive recovery.

    The business loop returns to the original outcome, using outcomes-versus-output OKRs to avoid equating shipped features with value. A team might improve a prompt, add a tool, or increase containment while leaving the target workflow unchanged. That is useful diagnostic progress, but it is not yet evidence that the product investment is working.

    The source also recommends A/B testing prompts and tools and considering minimum detectable effect when sizing experiments. Experimentation is most informative when the changed component, eligible population, success measure, and guardrails are defined in advance. Otherwise, movement in a downstream metric can be difficult to attribute to the agent change.

    Qualitative learning completes the loop. The source describes product trios spanning product management, design, and engineering, supported by continuous discovery, weekly transcript review, and the conversion of failure modes into test cases. It also recommends keeping prompts, tools, and evaluations versioned through a docs-as-code approach. This connects discovery to engineering discipline: observed failures become reproducible evaluations, evaluated changes become versioned releases, and releases can be compared or reversed.

    Scope and autonomy should expand only when both loops support the decision. Stable technical metrics without workflow impact suggest that the use case or experience needs reconsideration. Business improvement accompanied by unsafe or unreliable behavior suggests that scaling is premature. Evidence across both dimensions supports a measured move into adjacent tasks or higher-impact actions.

    Build the next release around earned autonomy

    The durable pattern for AI agent products is earned autonomy: every increase in access or authority follows evidence from a narrower operating state. As evaluations accumulate and real workflow performance becomes visible, teams can make expansion decisions based on demonstrated capability rather than the apparent fluency of a demo.

    References

  • Why We Made Fin the Most Open Agent: Instant HubSpot & Freshdesk Support With 76% Resolutions

    I’ve spent my career pairing product strategy with customer reality, and nothing is more clear right now than the demand for openness and speed. Today, we’re announcing that Fin can be used as a Service Agent on top of HubSpot and Freshworks, meaning you can use the world’s best Agent without migrating off your helpdesk.

    Hubspot and Freshdesk customers can now:

    Get Fin live, integrated, and working seamlessly in less than an hour.

    Delivering a 76% average resolution rate.

    Across all customer channels (voice, email, chat, social, and more).

    Resolving complex queries that require reading and writing to third party systems.

    With everything fully configurable to follow the unique policies of every individual business.

    This launch is a very visible step in a journey we’ve been on from day one: building an open, customer-first platform that plays well with the rest of your stack. We’ve long known that businesses want flexibility in how they configure their customer-facing tech stack. Since the very beginning, we have built Fin as an open platform, with APIs, MCPs, CLI, and opening up access to Apex, our proprietary trained model that delivers best in class performance.

    To make things easy for our customers, we have extensive public documentation of our product on our website, in our help center, and in our developer docs. We are the only Agent company in our space to do this, others hide most details behind sign-in screens, which we don’t believe is the right thing to do.

    Open Agent platforms will win because customers refuse to be boxed into closed ecosystems. We now believe our category has reached a stage where customers demand open platforms, that those who open up are more likely to win, and those who remain closed and protectionist will accelerate their demise.

    We are operating in a fast changing world, and customers do not want to be locked into a single vendor or closed ecosystem. They want the ability to experiment, to swap things in and out, and move everything with ease, technically and commercially.

    In an open world, the best product will win. In a world where businesses can easily swap vendors, the best product will win. We are happy to compete on that front, confident that Fin delivers the best customer experience and the highest performance.

    From a product management lens, this openness is powered by agentic AI patterns paired with robust CRM integration. Under the hood, we use Model Context Protocol (MCP), well-documented APIs, and orchestrated AI workflows to read from and write to third-party systems. That’s how Fin handles true multi-channel work—including voice AI agent scenarios—while giving teams the observability they need through Agent Analytics.

    If you are a Hubspot or Freshdesk customer, you can now have Fin integrated and live within an hour, without needing any help from us. We’re here if you want us, but as part of our commitment to building an open platform, we’ve designed everything to be self-servable—start in minutes or watch a quick demo of how everything works.

    Fin for Hubspot

    Fin for Freshdesk

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • From AI Builder to Agent Swarm: A Product Delivery Model

    From AI Builder to Agent Swarm: A Product Delivery Model

    AI-native product delivery has two distinct layers: a product professional who turns uncertainty into testable artifacts, and an agent workflow that divides complex work among specialized AI components. Treating either layer as the whole model misses the more useful opportunity.

    Together, the AI Builder role described by Product School and the parallel-agent architecture discussed by Pendo suggest an operating model for moving from customer evidence to evaluated software. The central lesson is not simply to add more AI. It is to assign clear responsibilities, preserve evidence across handoffs, and expand automation only where it improves a measurable constraint.

    Key takeaways

    • The AI Builder is the human integration layer, connecting discovery, prototyping, evaluation, and delivery inside the product trio.
    • Parallel agents are a system design choice, useful when specialized paths can improve latency, answer quality, or resilience.
    • Evaluations, analytics, observability, and controlled releases form the shared control system for both layers.
    • Fan-out should respond to uncertainty and business importance rather than becoming the default for every task.

    One delivery system, with human and machine responsibilities

    The Product School article presents the AI Builder as a hybrid product professional rather than a renamed product manager or an isolated prototyper. In its account, this person uses AI across analysis, prototyping, evaluation, and shipping, with the aim of shortening the distance between a customer problem and a runnable experiment.

    The Pendo article addresses a different layer. It describes workflows in which research, reasoning, tool use, and formatting can be assigned to specialized agents and then reconciled. Its focus is not ownership of the product problem, but the computational structure used to complete work.

    Read together, the articles separate two ideas that are often blurred. An AI-native team still needs a person or group to choose the problem, define acceptable behavior, interpret customer evidence, and decide whether an experiment justifies investment. Agents can perform bounded tasks within that process, but parallel execution does not establish product relevance on its own.

    LayerPrimary responsibilityTypical artifactsControl question
    AI Builder and product trioTranslate customer and business uncertainty into experimentsPrototypes, evaluation criteria, instrumented experiences, delivery recommendationsIs the team learning about an outcome that matters?
    Agent workflowExecute and reconcile specialized tasksRetrieved context, candidate responses, tool results, rankings, formatted outputsDoes orchestration improve the target measure enough to justify its complexity?
    Delivery platformProvide access, measurement, release controls, and safeguardsTool interfaces, traces, feature flags, budgets, analytics, fallbacksCan the workflow be observed, governed, and changed safely?

    This division of responsibility also clarifies the meaning of vibe coding in the Pendo account. Prompts, examples, and constraints are used to shape an intended experience before the team commits to extensive code or rigid rules. The AI Builder supplies the product judgment and experiment design around that activity; an agent architecture supplies one possible execution mechanism.

    Parallelism should target a constraint, not become a default

    Pendo reports three proposed benefits of parallel agents. Independent specialists can work concurrently to reduce latency, diverse candidate paths can be compared to improve quality, and risky or failure-prone operations can be isolated behind fallbacks. The article names fan-out/fan-in, race-and-rerank, specialist swarms, consensus, and self-consistency checks as patterns for producing and reconciling candidates.

    Those benefits depend on the shape of the task. Parallel research may help when several sources or interpretations must be examined independently. A race-and-rerank pattern may help when multiple plausible outputs can be scored against explicit criteria. Guarded fallbacks may improve resilience when a tool can fail without invalidating the entire experience. By contrast, multiplying agents around a simple, deterministic step adds coordination, cost, and more places to inspect when something goes wrong.

    The Product School article provides the missing selection mechanism: the workflow begins with a high-signal use case and explicit evaluation criteria. That makes orchestration a response to observed limitations in an experiment rather than an architectural commitment made in advance. A prototype can begin with the smallest credible workflow, reveal whether the bottleneck is grounding, reasoning, tool reliability, or response time, and introduce specialization at that point.

    Pendo proposes a similar progression at the system level: begin with retrieval, add a planner-executor split, and introduce parallel specialists where accuracy or latency problems appear. It also recommends placing budgets on fan-out, caching results, using smaller models when confidence is high, and widening the workflow when uncertainty rises. These are recommendations from the source, not independently reported benchmarks, but they establish a useful product principle: additional computation should be purchased in proportion to uncertainty and consequence.

    Evaluation is the bridge from discovery to dependable delivery

    The strongest overlap between the two articles is evaluation. Product School describes AI Builders converting interviews and behavioral analytics into instrumented experiments, benchmarking quality before production, and using A/B testing to feed results back into strategy. Pendo similarly calls for offline evaluations before rollout, production experiments afterward, and agent-level analytics to identify regressions across individual workflow steps.

    This creates a continuous evidence path rather than a handoff between discovery and engineering. A customer problem informs a prototype; the prototype produces evaluation cases; those cases become release gates; production behavior supplies new evidence for the next iteration. CI/CD can move changes through the delivery system, while evaluations determine whether an AI behavior is ready to move with them.

    A staged adoption path

    1. Select a bounded use case. Product School suggests beginning with a high-signal application such as generative-AI prototyping or an in-app guide, rather than attempting to transform the whole delivery process at once.
    2. Define the evidence before expanding the build. Specify evaluation criteria, analytics, and the customer or business outcome the experiment is intended to illuminate.
    3. Establish grounded context. Both articles emphasize retrieval-oriented workflows. Product School also discusses prompts, context windows, and data contracts as product surfaces that require deliberate design.
    4. Start with minimal orchestration. A single workflow or planner-executor arrangement provides a baseline against which a specialist or parallel design can be judged.
    5. Add parallel paths selectively. Introduce research, tool-calling, reasoning, or validation specialists only where evaluation results reveal a material limitation.
    6. Release behind controls. The sources point to feature flags, A/B testing, observability, anomaly detection, fallbacks, and post-launch review as ways to expose failures and limit their impact.

    The Model Context Protocol appears in both accounts as a way to standardize access to tools and data. Product School frames MCP integrations as part of an AI Builder’s toolbox, while Pendo argues that standardized access keeps agent roles separate from authentication, quotas, and observability. The combined implication is organizational as well as technical: shared interfaces can let product teams experiment with workflow roles without embedding every platform concern in every prompt.

    The operating model changes what a product team owns

    Product School places the AI Builder inside the product trio, working with design and engineering from the beginning. Pendo argues that product trios can own complete AI workflows rather than limiting their attention to prompts. These views converge on broader product accountability: the team owns the behavior, evidence, cost, risk, and release mechanism as one product surface.

    That ownership requires clearer boundaries, not fewer disciplines. Product judgment determines which outcome deserves attention. Design shapes the customer interaction and failure experience. Engineering and platform work make tool access, observability, quotas, and release controls dependable. The AI Builder connects these concerns through runnable artifacts, while specialized agents remain replaceable components within the evaluated workflow.

    The resulting measure of maturity is not the number of agents deployed or the speed of prototype generation. It is whether the team can trace a customer need through an experiment, an evaluation, a controlled release, and a learning decision. As tools become easier to compose, that chain of evidence will be the durable advantage in AI-native product delivery.

    References

  • Engineering MCP Agents as a Reliable Product Platform

    Model Context Protocol adoption becomes consequential when an agent can retrieve organizational knowledge, select tools, and change a system of record. At that point, the engineering challenge is no longer simply connecting a model to an API. It is operating a product platform whose context, permissions, decisions, and side effects must remain dependable.

    The source article’s experience with workflows spanning Miro, Jira, and Confluence points to a coherent platform model: retrieval determines what the agent knows, tool contracts constrain what it can do, evaluation tests its behavior, and observability makes failures diagnosable. Product strategy and interaction design then determine whether that machinery improves work users already perform.

    Key takeaways

    • Treat retrieval, tool schemas, prompts, policies, and telemetry as platform components with explicit owners and versioning.
    • Prove one frequent, measurable workflow before expanding the agent’s tool and use-case surface.
    • Combine least-privilege access with visible tool rationale, consent controls, audit records, and safe recovery paths.
    • Evaluate the complete chain from retrieved context to downstream action, not just the quality of generated text.
    • Govern the tool catalog and delivery pipeline continuously so that extensibility does not become uncontrolled operational risk.

    The platform boundary extends beyond the MCP connection

    MCP provides a practical interface through which models can reach data, tools, and actions, according to the source article. The protocol connection is therefore an enabling layer, not the whole agent platform. A production workflow also depends on source authority, identity and permission checks, context selection, tool arbitration, execution controls, user-facing recovery states, and evidence that the result was useful.

    This broader boundary changes how teams should decompose the system. Retrieval is a managed context service rather than an incidental prompt-building step. Tools are governed capabilities rather than a loose collection of endpoints. Prompts and policies are deployable artifacts rather than text copied into application code. Traces and evaluations are part of the control plane because they reveal whether the other layers continue to work together.

    The source recommends starting with authoritative content, normalizing it with docs-as-code discipline, attaching metadata that supports permission-aware filtering, and selecting the smallest high-signal context needed for a task. The engineering implication is important: access control must shape retrieval before information reaches the model. Filtering only when an action is attempted would leave the reasoning process exposed to context the user or agent may not be entitled to use.

    Context quality also affects more than answer accuracy. The source links focused retrieval to lower hallucination risk, more accurate tool calls, and lower cost. That makes retrieval performance a shared dependency for safety, reliability, latency, and economics. It deserves its own contracts, tests, freshness expectations, and failure modes.

    A golden path turns architecture into an operating contract

    The source describes an initial workflow that summarized a Miro board into action items and wrote them to Jira. It reports that variants involving Confluence summaries, epic splitting, and backlog grooming followed only after the original path reached its reliability targets. This is less a recommendation for those particular products than a useful sequencing principle for agent platform engineering.

    A narrowly defined workflow exposes the entire contract between context and consequence. The team must decide which content is authoritative, what the model may infer, which tool is appropriate, what inputs the tool accepts, what the user should review, how a partial failure is handled, and how success is measured. A broad assistant can conceal these questions behind plausible conversation; a golden path forces explicit answers.

    The right first workflow is therefore not merely technically convenient. It should be frequent enough to matter, have an observable completion state, and carry side effects that can be bounded. The source frames outcomes such as time saved during backlog grooming, better meeting notes in Confluence, and fewer context switches across Miro boards as more useful roadmap anchors than novel model capabilities. It also recommends comparing task success, completion time, user edits, detected defects, and downstream business effects rather than relying on engagement alone.

    Those measures form a practical evidence chain. Evaluation results show whether the system behaves as designed; workflow measures show whether users can complete the task; business measures show whether the completed task creates value. Keeping the levels distinct prevents a technically impressive agent from being mistaken for a successful product.

    Safety depends on controlling actions and explaining them

    Tool access creates a sharper risk boundary than text generation because an incorrect decision can alter a ticket, document, or other shared record. The source’s proposed response combines least-privilege scopes, a human-readable rationale for each call, and an audit trail. It also calls for proposed inputs and expected side effects to be visible when the agent is about to use a tool.

    These controls address different failure classes. Narrow scopes limit the maximum effect of a bad decision. Input previews help users catch incorrect parameters before execution. Rationale makes the selection inspectable. Audit records support diagnosis and accountability afterward. None substitutes for the others, and a confirmation dialog alone does not make an overprivileged tool safe.

    Recovery behavior belongs in the same design. The source recommends retrying suitable failures with backoff, falling back to read-only behavior, or requesting consent or missing context. A robust platform should distinguish failures that are safe to retry from failures that require a different plan. It should also preserve an understandable state when a multi-step workflow completes only partially, so the user knows what changed and what did not.

    Transparency need not mean exposing raw internal reasoning. The useful product surface is operational evidence: the sources used, the selected capability, the intended inputs, the expected effect, and the resulting status. The source suggests a reveal panel containing retrieved sources, candidate tools, and confidence signals for power users. More generally, the amount of review should follow the consequence of the action: low-risk retrieval can remain lightweight, while consequential writes warrant clearer inspection and consent.

    Evaluation, observability, and delivery form one reliability loop

    The source outlines offline tests for intent classification and tool selection, online shadow evaluations for live drift, and regression checks after deployment. It also recommends traces that capture prompts, retrieved chunks, tool inputs, tool outputs, latency, and error codes. Together, these practices connect a visible failure to the component and version that produced it.

    Evaluation without observability can show that quality declined without explaining why. Observability without evaluation can produce detailed traces without deciding whether the behavior was acceptable. A mature loop needs both: test cases encode desired behavior, traces expose actual behavior, and production outcomes reveal gaps in the test set.

    The delivery process must preserve that connection. The source treats prompts, tool schemas, and guardrails as versioned artifacts deployed behind feature flags, with canary releases, controlled comparisons, and rollback capability. This approach makes a behavioral change attributable. If tool selection deteriorates after a prompt revision or a schema update breaks an integration, operators can identify the change and contain its reach.

    Latency should be governed in the same loop because an accurate workflow can still fail as a product experience. The source reports using task-specific latency budgets, caching stable retrieval results, parallelizing safe calls, prefetching likely session context, and providing progress when work exceeds the expected budget. These techniques should remain subordinate to correctness: parallel execution is appropriate only when calls are independent, while caching must respect freshness and permission boundaries.

    The source also assigns prompts a user-experience role, combining plain-language intent, domain constraints, and explicit tool contracts while using examples, tooltips, and in-product guidance to help users frame requests. This connects conversation design to reliability. Better instructions can reduce ambiguity before the platform has to resolve it through additional model turns or risky assumptions.

    Scale requires governance of tools, teams, and ownership

    MCP’s extensibility can turn into tool sprawl if every integration is added without lifecycle management. The source recommends a curated catalog recording each tool’s owner, scope, schema version, and deprecation policy. It also describes schema linting in continuous integration, backward-compatible changes, and quarterly retirement of unused tools. These are conventional platform disciplines applied to an agent’s capability surface.

    A catalog is valuable because an agent reasons over descriptions and schemas while operators depend on stable implementation contracts. Poorly differentiated tools can make selection ambiguous; unannounced schema changes can invalidate prompts and evaluations; ownerless tools can remain available after their data or permission assumptions have changed. Governance should therefore assess semantic clarity as well as API validity.

    Organizational design matters for the same reason. The source describes an empowered trio consisting of a product manager responsible for outcomes and risk posture, a forward-deployed engineer focused on schemas and scalability, and a designer responsible for conversational flows and recovery states. It also favors weekly evaluation reviews over demonstration-led progress. The underlying principle is shared ownership: platform reliability cannot be delegated entirely to model engineering when the decisive questions span product value, system behavior, permissions, and user comprehension.

    The source’s proposed 30-day starter sequence moves from selecting one workflow and defining permissions, measures, and evaluations; through retrieval and a minimal tool set; to an instrumented internal pilot; and finally to hardening and a limited beta. The schedule is reported as a blueprint rather than independent proof of how long every implementation should take. Its more transferable lesson is dependency order: define the outcome and risk boundary before multiplying capabilities.

    As agents begin coordinating across products, the durable advantage will come from platforms that preserve this discipline across every new connection. MCP can make capabilities composable, but dependable composition will still depend on controlled context, explicit authority, observable execution, and evidence that the workflow improves real work.

    References

  • Reusable AI Agent Workflows Need Evaluation Contracts

    Reusable AI Agent Workflows Need Evaluation Contracts

    Reusing an AI agent capability can accelerate delivery, but reuse also multiplies the consequences of an undetected defect. A retrieval component, tool-call routine, or safety check may appear in several workflows, so its quality cannot depend on the team that happens to integrate it next.

    The practical answer is to package each reusable skill with an evaluation contract: defined behavior, test fixtures, observability, guardrails, and outcome measures that travel with the component. Read together, the two source articles outline how modular workflow design and eval-driven development can reinforce each other from prototype through production.

    Reuse requires a contract, not just a prompt

    The AI skills library article describes modular capabilities for retrieval and grounding, summarization, classification, tool use, data enrichment, safety controls, and evaluation harnesses. Its central architectural idea is consistency: common interfaces and conventions allow teams to compose capabilities and replace implementations without rebuilding an entire flow.

    That modularity addresses code and workflow reuse, but it leaves an important product question: what must remain true when an implementation is replaced? The product-manager evaluation playbook supplies the missing half. It calls for versioned prompts, tools, and datasets; fixed offline scenarios; production experiments; and traces that expose how an agent reached an answer.

    The synthesis is an evaluation contract attached to every reusable skill. The contract defines acceptable inputs and outputs, relevant policies, expected telemetry, representative tests, and promotion thresholds. A skill is then reusable because its behavior can be checked repeatedly, not merely because its code can be imported.

    This distinction matters most in composed workflows. A summarizer that performs well on clean text may behave differently after a weak retrieval step. A tool-use component may generate a plausible response even when the underlying action fails. Reusable interfaces make these components interchangeable; evaluation contracts make the substitutions accountable.

    Measure four layers of agent quality

    No single score can represent the quality of a reusable agent workflow. The evaluation article separates concerns such as task success, factuality, safety, latency, cost, evidence quality, and product outcomes. The skills-library article adds operational concerns around guardrails, runtime metrics, and production monitoring. Combined, they suggest a four-layer model.

    Evaluation layerQuestion it answersReusable evidenceReported signals
    Component behaviorDoes the skill perform its assigned task?Fixed fixtures, golden examples, and domain scenariosTask success, factuality, and retrieval evidence quality
    Safety and policyDoes it remain within required boundaries?Adversarial cases, policy checks, and guardrail configurationsSafety performance, PII handling, and content-policy adherence
    Operational performanceCan it run reliably within product constraints?Traces, logs, version records, and production dashboardsLatency, cost, tool success, and fallback behavior
    Product impactDoes better agent behavior create user or business value?Experiment definitions and driver-tree mappingsTask completion, satisfaction, activation, retention, and NRR

    The layers should remain distinguishable even when a dashboard brings them together. If a workflow’s task-success score rises while latency or cost deteriorates, the trade-off should be visible. If offline factuality improves without changing completion or satisfaction in production, the result should not automatically be treated as a product win.

    Retrieval-first workflows illustrate the value of separation. The evaluation playbook recommends assessing the quality of retrieved evidence independently from generation. That boundary makes a failure attributable: the system can distinguish missing or irrelevant evidence from a generator that mishandled useful context. The same principle applies to classification, tool selection, tool execution, and response composition.

    A reusable workflow needs a controlled promotion path

    The two sources describe complementary stages rather than competing evaluation methods. The skills-library article starts with a quick-start chain, configurable skills, guardrails, evaluation datasets, and instrumentation. The evaluation playbook places fixed offline suites before user exposure, followed by controlled online validation. Together they form a promotion path from composable prototype to measured production capability.

    Offline evaluation establishes eligibility

    A candidate workflow should first face stable examples representing core scenarios, known failure modes, edge cases, adversarial prompts, and domain-specific questions, as reported by the evaluation playbook. Stable fixtures make comparisons reproducible when a prompt, model, tool, retrieval strategy, or policy changes. Running these checks through CI/CD, as proposed in the skills-library article, turns evaluation into a regular release control instead of a separate audit.

    Model-based judges can expand coverage for qualities such as helpfulness, coherence, and adherence, but the evaluation article cautions that they require calibration against a small, high-quality human-labeled set. It also recommends monitoring judge drift and retaining human review for edge cases or flows where mistakes carry greater consequences. A reusable judge configuration should therefore include its rubric, reference labels, version, and conditions for escalation.

    Online evaluation establishes value

    Passing offline checks shows that a variant is eligible for controlled exposure; it does not prove that users benefit from it. Both articles describe feature flags and A/B testing as mechanisms for comparing workflow variants in production. The evaluation playbook identifies conversation outcomes, tool success rates, human-support fallbacks, and user satisfaction as useful online signals.

    This staged approach also limits ambiguity. An offline regression can block a weak component before exposure, while an online experiment can test whether an eligible improvement changes real behavior. Promotion should depend on both: acceptable component performance and evidence that the complete workflow advances its intended outcome.

    Traces turn composition failures into fixable problems

    Composability increases the number of boundaries at which a workflow can fail. The evaluation playbook treats traces as the backbone of agent evaluation because they record inputs, intermediate actions, invoked tools, and final responses. The skills-library article similarly connects reusable chains to logs, traces, metrics, and production dashboards.

    A final-answer score alone may reveal that a workflow failed, but a trace can localize the failure. It can show whether retrieval supplied poor evidence, classification selected an unsuitable route, a tool call failed, a guardrail intervened, or generation ignored valid context. This makes evaluation useful for component ownership: teams can repair the relevant skill rather than adding a broad prompt patch to the entire chain.

    Trace analysis also supports reuse decisions. If one component repeatedly causes latency, cost, or safety regressions across several workflows, improving that shared component may create more value than optimizing each application independently. Conversely, a component that succeeds in one context but fails in another may need a narrower contract rather than a universal interface.

    Versioning is essential to that diagnosis. The evaluation playbook recommends versioning prompts, tools, and datasets, while the skills-library article emphasizes swappable implementations and comparable variants. Without linked versions for the component, evaluation set, judge, and workflow configuration, an apparent improvement may be difficult to reproduce or attribute.

    Governance and product outcomes belong in the same system

    Reusable workflows can spread good controls, but they can also propagate weak ones. The skills-library article reports guardrails for PII redaction, content-policy checks, and rate limiting, alongside configuration intended to support privacy-by-design. Packaging these controls as reusable capabilities can make the approved path easier to adopt, while evaluation fixtures test whether the controls continue to work as surrounding workflows change.

    Governance should not be reduced to a final pass-or-fail gate. Safety, privacy, and policy behavior need their own cases and traces throughout development. The amount of human review can then reflect the cost of error, consistent with the evaluation playbook’s recommendation to retain human oversight for higher-risk flows.

    The same evaluation system must connect technical quality to product value. The evaluation playbook proposes a driver tree that links per-turn measures such as helpfulness, safety, and latency to session outcomes such as task completion, and then to product measures including activation, retention, and Net Recurring Revenue. This hierarchy prevents a local metric from becoming the objective by default.

    For product teams, the resulting unit of roadmap work is not simply a new skill. It is a versioned capability with evidence about behavior, operational fitness, policy compliance, and contribution to an intended outcome. That shared definition gives product trios, engineers, and governance stakeholders a more precise basis for deciding whether to reuse, revise, or retire a component.

    Key takeaways

    • Package each reusable agent skill with an evaluation contract covering behavior, fixtures, telemetry, policies, and promotion criteria.
    • Keep component quality, safety, operational performance, and product impact distinct so improvements and trade-offs remain attributable.
    • Use fixed offline evaluations to establish release eligibility, then controlled online experiments to determine real-world value.
    • Trace intermediate steps and tool activity so failures can be assigned to the correct component instead of patched at the final response.
    • Version workflows, prompts, tools, datasets, and judges together so results remain comparable and reproducible.

    As skill libraries expand, their lasting advantage will come from accumulated evidence rather than component count. Teams that make evaluation portable alongside implementation can reuse workflows without surrendering visibility, governance, or product accountability.

    References

  • A Game-Changing Leap in Voice AI: Fin Voice 2, Apex Flash, and a Live Demo You Can Trust

    A Game-Changing Leap in Voice AI: Fin Voice 2, Apex Flash, and a Live Demo You Can Trust

    In competitive markets, I see two options: try to win the game competitors set, or choose to play a different game. In the "Customer Agents" category, I’ve watched too many glossy, fabricated demos—especially around voice—mask the real challenges. Voice is just extremely hard. We all know the future of customer experiences will be Agent-driven voice, yet most of us haven’t actually spoken with a modern AI Agent when calling a business because the tech hasn’t been truly ready in the wild. Today, the bar moves.

    What changed? There’s a live, public demo of cutting-edge voice tech you can stress test yourself—no smoke, no mirrors. I recommend taking it for a spin: https://fin.ai/voice. It’s fast, natural, and, yes, very, very good.

    For context, yesterday brought Apex Flash, their newest and fastest model, built for the unique demands of low latency channels like voice. Today comes Fin Voice 2, a major upgrade to Fin Voice with over 20 new features, and the first product built on Apex Flash.

    Here are the three things that stood out to me—and why they matter for customer support AI strategy and product strategy.

    First — thanks to Apex Flash, Fin Voice 2 is now the fastest, most natural Agent for phone, with higher resolution rates and customer satisfaction scores than ever before. Apex Flash is trained on millions of customer experience interactions, fine tuned for customer service, and can be configured to understand all your knowledge and follow all your policies. The result is higher resolution at significantly lower latency—the best of both worlds for voice AI agent performance.

    Speed and naturalness here aren’t accidental. Most voice AI products are slow because they convert speech to text, send it to a general model, get a text answer, and then convert it back to speech. Fin Voice 2 was designed to work differently, separating the real time layer that handles speech processing, and the layer that generates answers. That architecture is purpose-built for the demands of customer service on voice.

    Slide for Fin Voice 2, powered by Apex Flash, showing it beats Voice 1: +24.5% average resolution, +8.4% guidance following, +1.3% CSAT, -19.2% time to first audio, -37.6% semantic search latency.
    Powered by Apex Flash, Fin Voice 2 raises the bar on quality and speed—boosting resolution rates and guidance following while cutting time to first audio and semantic search latency, with a lift in CSAT too.

    Second — Fin Voice 2 can handle complex queries end to end: taking actions in external systems, verifying callers’ identities, processing refunds, booking appointments, and more. Phone is a high-stakes channel, and Fin adapts to customers across emotional states, clarifies when needed, and confirms key details before taking action. Most of the time, Fin can resolve the query in full, and when it can’t, it seamlessly hands off to the human team, maintaining full customer context and history. You also get multiple improvements to call quality, plus proactive outbound calls to follow up on unresolved issues—all orchestrated by robust AI workflows.

    Third — Fin Voice 2 gives you total control with industry-leading tools to configure and manage how Fin behaves. You get rich, detailed insights into call behavior and quality, the most common topics of calls, and one-click recommendations to improve. As with everything in Fin, you can fully self-serve and then manage it all with ease, without requiring professional services. Many vendors only let you set up their voice agent under supervision; with Fin, you get everything you need to iterate fast.

    If you haven’t tried the demo yet, go check it out: https://fin.ai/voice. If you prefer to wait, don’t be surprised when you end up speaking with it at a favorite brand soon.

    From a product management lens, this is what matters: latency is a feature customers feel; transparency builds trust in enterprise AI; and control is non-negotiable for CX leaders. The combination of a purpose-built, agentic AI architecture, measurable gains in resolution and CSAT, and true self-serve configuration signals that voice is moving from prototype theater to production reality. That’s the different game I want our industry to play.

    Inspired by this post on The Intercom Blog.

    Book a consult png image

  • Crafting Beloved Tech Brands: My Moonshot Marketing Playbook for the Post-LLM Era

    I spend a lot of my time asking a deceptively simple question: what does excellent marketing actually look like in 2026? From the vantage point of product leadership, the answer isn’t a spreadsheet or a channel plan—it’s a feeling. Beloved tech brands earn the benefit of the doubt, create gravity around their roadmap, and make customers proud to belong. That kind of momentum is not an accident; it’s a system.

    Here’s the hard truth I’ve learned building and scaling products: giving teams different goals creates dysfunction. When brand, demand gen, product marketing, and comms run on fragmented OKRs, you manufacture internal headwinds. “Marketing is one engine – not separate pieces.” One strategy, one narrative, one set of outcomes—expressed through different craft disciplines and time horizons.

    That unity of purpose clarifies executive roles, too. The real difference between an SVP and a CMO is scope and narrative ownership. A great CMO architects the whole system—portfolio allocation, brand architecture, integrated go-to-market strategy, and the bar for creative taste—while refusing to get dragged into decisions they should never be making (for example, approving every headline or micromanaging channel tactics). Leaders should decide the outcomes, standards, and constraints; teams should control the craft.

    On portfolio design, I run marketing like a portfolio of moonshots. You need a healthy mix: proven programs that compound, emergent bets that learn fast, and a small set of true moonshots that can change the slope of the curve. The point isn’t bravado; it’s risk-balanced exploration. If everything ships safely, you’re under-investing in differentiation. If everything is a swing for the fences, you’re not building a repeatable growth engine.

    This is where taste becomes a strategic advantage. “Ubiquity is the opposite of cool.” If you want to be beloved, you cannot treat every channel, audience, and moment as equal. Early on, selective distribution, distinctive creative codes, and tight community loops create status and meaning. Later, you scale without sanding off the edges that made the product special.

    Why do a few companies build a flywheel of momentum while others stall? They align story, product, and distribution. The product earns trust, the narrative creates aspiration, and the go-to-market strategy ensures the right customers experience both at the right time. Then perception cycles kick in—the Silicon Valley clock turns—and irrational optimism or skepticism can amplify signals. The antidote is compounding proof: consistent product shipping, community advocacy, and creative that makes people care.

    Scaling taste across an organization is teachable. I codify brand principles, narrative guardrails, and examples of “right” versus “almost right.” I replace abstract feedback with decision rubrics—what we keep, kill, or revise and why. I run recurring creative reviews with a small cross-functional council, so judgment compounds. Taste can’t be fully automated, but it can be operationalized: shared references, a story bible, and a high bar for craft that’s explicit, not mystical.

    In a post-LLM world, the fundamentals haven’t changed—but the frontier has. Generative tools supercharge iteration and research, yet the artistry never really left. You still need a point of view, a tension worth resolving, and a value proposition that’s felt, not just stated. Can taste be encoded in software? Parts of it—pattern libraries, style constraints, data-driven feedback—absolutely. But the spark that makes work unforgettable remains human: judgment, risk tolerance, and the courage to ship something that might not fit the playbook.

    That’s why telling an optimistic, yet realistic story about AI matters. Over-automation drains humanity; under-automation wastes potential. The best work pairs AI Strategy with craft leadership: LLMs for rapid exploration, humans for narrative decisions and ethical judgment. Your message should show how AI expands customer agency, not just efficiency.

    The brand-versus-growth debate is a false choice. The right story accelerates pipeline, and the right demand programs reinforce the brand. Look at Apple’s discipline around product truth and design codes, or Google Chrome’s “The Web Is What You Make of It (Dear Sophie)” for proof that emotion and utility can co-exist. Notion, Pinterest, Square, HubSpot, and Harley-Davidson show how community, identity, and product-led growth interlock when the company knows exactly what it stands for.

    When it comes to launches, I’ve learned that announcement videos full of humans, lack humanity. Overproduced gloss often dilutes the truth customers seek: what problem does this solve, how quickly can I feel the value, and why does it matter now? Real users, real context, and a crisp arc from problem to promise will outperform most theatrics.

    Practically, I architect my week to protect taste and outcomes. Early-week for strategy, portfolio reviews, and cross-functional alignment; mid-week for deep creative and product marketing work; late-week for decision clears and postmortems. I time-box “disruptive energy”—space to chase non-obvious ideas—and I guard it like any critical meeting. Without protected cycles for exploration, the urgent will always suffocate the important.

    If there’s a single takeaway: playbooks are obsolete, but the fundamentals are not. The channels change; the psychology doesn’t. Run one engine. Allocate a true portfolio. Scale taste with rigor. In the AI era, make people care. That’s how beloved tech brands are built—and how they endure.

    Book a consult png image
  • Supercharge Insights with Amplitude Agent Connectors: Connect Notion, Slack, Linear & More

    Supercharge Insights with Amplitude Agent Connectors: Connect Notion, Slack, Linear & More

    I’ve led enough multi-tool product organizations to know how quickly momentum erodes when insights and actions live in different places. When my teams bounce between Notion, Atlassian, Slack, Linear, and analytics dashboards, we pay a real tax in context switching. That’s why I’m excited about what Amplitude is enabling with Agent Connectors—bringing our daily work and our data-driven decisions into one fluid, agentic AI workflow.

    Connect Notion, Atlassian, Slack, Linear, and more to Amplitude's Global Agent. Get richer analysis and take action across tools without leaving Amplitude.

    Practically, this means I can treat Amplitude analytics as a unified analytics platform where analysis and execution finally meet. Instead of exporting charts or copying insights into docs, I can drive Agent Analytics directly from the same surface where I manage behavioral analytics, reducing friction and accelerating decisions. For my product strategy, that’s a meaningful shift—from “insight later” to “insight-to-action now.”

    Here’s how I’d use it on a typical day: I ask the agent to synthesize signals from recent feature usage, spotlight anomalies, and then draft a concise summary for our Slack channel. In the same flow, I can prompt it to reference our Notion specs for context and queue next steps in Linear, keeping Atlassian stakeholders looped in without any extra swiveling between tabs. The value isn’t just faster execution; it’s tighter alignment across teams because the analysis and the plan live together.

    From an operating model perspective, this is how I scale AI workflows responsibly. I can define clear prompts, approval paths, and ownership so the agent augments—not replaces—expert judgment. Data governance and permissions remain front and center: the agent sees what your teams are allowed to see, and we maintain auditability on critical workflow steps. The outcome is a trustworthy, repeatable system that compounds learning over time.

    If you’re exploring agentic AI for product teams, start small and instrument your ROI. Pick one or two connectors (Slack and Notion are great first choices), define a measurable workflow—like pushing weekly retention insights and creating prioritized follow-ups in Linear—and iterate using continuous discovery. In my experience, the first wins appear as reduced time-to-insight, fewer meetings to align, and faster cycle time from observation to shipped change.

    The big picture is simple: bring your work to your analytics, and your analytics to your work. With Agent Connectors, Amplitude’s Global Agent helps close the loop from understanding behavior to taking action—without leaving the place where your insights are born.

    Inspired by this post on Amplitude – Best Practices.

    Book a consult png image
  • Package Hack Wake-Up Call: My Playbook for Securing Cowork, Coding Agents, and Secrets

    Package Hack Wake-Up Call: My Playbook for Securing Cowork, Coding Agents, and Secrets

    I love being a builder. It feels like a superpower I can’t stop using, and lately I’ve been channeling it into better workflows, faster experimentation, and sharper product thinking.

    I tinker with my Claude Code workflows to make every day more effortless. I’m having a blast creating AI-generated interview snapshots and opportunity solution trees for Vistaly. I also spend time digging into traces and iterating on the AI coaches I use for our discovery courses.

    Then the recent wave of malicious software spreading through the open-source community popped my bubble. It hit companies big and small—names like OpenAI, PostHog, and Zapier. As I dug in, I realized what many cybersecurity experts have long known: this is a deep rabbit hole. If I want to build responsibly, I have to get significantly better at protecting my devices, credentials, and code. And if you’re building with AI or modern tooling, you likely do, too.

    Here’s why. We all rely on open-source software. Most modern applications assemble tried-and-true components—parsing a PDF, handling dates across time zones, visualizing spreadsheet data, connecting to an API—rather than reinventing them. The same is true for agent skills and MCP servers; they accelerate how we get value from models. This is overwhelmingly a good thing. But it also creates an attack surface that bad actors exploit.

    We don’t need to abandon third-party code. We do need to understand the mechanisms attackers use and consistently defend against them.

    Infographic titled 'When Trusted Packages Go Rogue' summarizing a talk on package hacks: worm spread, defense framework, risks from AI coding tools, and practical mitigation steps, with security-themed icons.
    When one malicious worm compromises hundreds of packages, what should dev teams do? This visual teaser maps the agenda—how it spreads, how to guard against it, AI tool risks, and concrete steps to mitigate.

    On May 11th, I started seeing tweets about a TanStack hack. At that time, I didn’t know what TanStack was. But apparently, it’s a popular set of JavaScript libraries that are used by a lot of React sites. At first, I didn’t pay much attention. Then I learned the packages were compromised by a worm—malicious software that self-replicates—and it spread quickly. Within hours, dozens of packages were implicated; by day’s end, it was in the hundreds. That’s when I knew I had to lean in.

    If you’ve explored safe development practices with coding agents before, you’ve seen the basics of package safety. A package is a bundle of reusable code shared through registries, and nearly every app you use depends on them. The unfortunate twist with this specific hack, known as the Mini Shai-Hulud worm, is that it shows prior “safe enough” heuristics aren’t sufficient. Popularity and trust signals don’t guarantee safety. We have to do more.

    So here’s what I’ll cover today: how malicious software typically works, a practical framework for guarding against it, the specific risks of using Cowork to write and run code, and concrete steps to mitigate that risk. My goal is simple: help you keep building—despite the risks—while protecting your data and your business.

    Quick disclaimer: I’m not a security expert. I’m sharing my personal journey and what I’ve learned through research and hands-on work. Please use your best judgment when applying any of this.

    Infographic showing a 3‑step pattern in malicious software: enter via package or script, search a device for sensitive data, then exfiltrate to an attacker, with icons and expanding entry points.
    Package hacks share a simple playbook: get in, sweep for secrets, and phone home. This visual breaks down the 3 steps and flags new entry points—from packages to MCP servers, agent skills, and app extensions.

    An agent recently scoured over 230,000 malicious software incidents and found that most malicious software follows a similar pattern. First, it needs an entry point onto your computer. Once installed, it scours your device for sensitive data, and then it uses your network connection to send that data to its own servers. The Mini Shai-Hulud worm spreads via malicious package install scripts that run at download time, then searches the device for credentials (including package publishing rights), poisons additional packages to continue replicating, and uses multiple channels—including the victim’s own GitHub public repos—to distribute secrets.

    In practice, most attacks boil down to three steps: 1) It finds an entry point to your device. 2) It searches your device for sensitive data. 3) It sends that data to its own server. The good news: this pattern also tells us how to defend. We can harden entry points, minimize what code and agents can access, and constrain outgoing network traffic.

    Keep in mind that install scripts aren’t the only entry vector. Any code that runs on your machine could contain malicious payloads: third-party packages, agent skills, MCP servers, browser or desktop extensions—the list is long. As coding agents and “vibe coding” tools become mainstream, more non-engineers are exposed to the same risks engineers have managed for years.

    You might be at elevated risk if you do any of the following: you download and use third-party skills or MCP servers; you let Claude Code, Codex, or other coding agents write scripts that run locally and use third-party packages; you use an IDE like VS Code or Cursor with third-party extensions; or you install third-party extensions in tools like Obsidian. This isn’t an exhaustive list, but if any of these apply, it’s worth tightening your approach.

    Infographic titled 'Are You at Risk?' listing third-party code exposure points: agent skills and MCP servers, coding agents on local devices, IDE extensions (VS Code, Cursor), and Obsidian plugins.
    Relying on third-party code? This visual highlights four common risk zones—agent skills/MCP servers, coding agents, IDE extensions, and Obsidian plugins—and urges a review of downloads, local scripts, and add-ons.

    The “safest” approach would be to avoid installing third-party software on your local device entirely. That’s not realistic. We all depend on third-party components in our stack. So I’ll start with one of the most common paths for non-engineers writing and running code today: Cowork.

    Evaluating Cowork’s safety was eye-opening. Cowork offers meaningful protection—more than running code directly on your machine—but it isn’t bulletproof. There’s a notable gap you should understand.

    Here’s how Cowork helps. It runs code inside a virtual machine, which isolates the execution environment from your real device—a quarantine room for code. While Cowork doesn’t fully control what comes into the room (that part is on you), if malicious code gets in, it’s contained and cannot reach the rest of your filesystem. Cowork also limits outbound network traffic from the virtual machine, which helps disrupt data exfiltration. However, it’s not foolproof.

    Because Claude can install packages inside Cowork, it remains susceptible to malicious code like the Mini Shai-Hulud worm. And GitHub is on the allow list so Cowork can read and write to your repos. Since the Mini Shai-Hulud worm uses GitHub to publish secrets, this creates exposure. The crucial mitigation: if you never give Cowork access to sensitive data, there’s nothing for an attacker to steal.

    Infographic titled 'Does Cowork Keep You Safe?' with three points: entry point contained, data safe only if kept outside, and partially limited network traffic, highlighting risks in package attacks.
    A quick visual from a security deep dive on package hacks shows how Cowork handles threats: entry points are contained, data is only safe when kept outside, and network traffic is partly limited—making shared data the gap to watch.

    Your responsibility is straightforward but critical: your data is only safe if it stays outside the virtual machine. When you mount folders into Cowork, those folders become accessible to any code running inside the VM. That includes malicious scripts. Before sharing, ask two questions: do the folders contain any credentials or secrets, and do they include proprietary data that would be harmful if accessed?

    It’s common for code to need credentials. That’s why Cowork includes connectors to third-party sources like Google Drive and Slack. Credentials configured for these connectors never enter the VM—they remain outside the quarantine room—so they’re not exposed to malicious code. But if your code requires additional credentials inside the VM, scope them tightly and assume they could be compromised.

    You can also use custom MCP servers you create yourself with Cowork. Those credentials stay outside the VM as well, provided the MCP servers are remote (hosted on a web server, not downloaded locally). It’s more work than dropping in a local server, but it keeps secrets out of reach from VM-executed code.

    Beyond credentials, scrutinize the actual content you share with Cowork, including anything accessed through connectors. Least privilege is the rule: grant only what’s absolutely necessary for the task, and nothing more.

    Infographic titled 'Keep Building. Stay Safe.' outlining a 3-part series for AI builders: 1 Cowork Safety, 2 Claude Code Config, 3 Off-Device Development, with teal security, AI, and cloud icons and a 'Product Talk' label.
    Amid a wave of package-supply attacks, this Product Talk visual launches a 3-part guide to safer AI building—starting with Cowork safety today, then Claude code config next week, and off-device development coming soon.

    What about skills? Cowork supports skills, and you can add third-party skills inside the quarantine room. If you’re not placing your own data in that room, you can afford more risk. The moment you add sensitive or proprietary data, be selective. Skills can include third-party code, and bad actors use skill directories to distribute malicious payloads. Personally, I never use third-party skills as-is. If one looks useful, I read through the files, then ask Claude to recreate it so I understand what it does and maintain control. If I were to use third-party skills, I’d do it in Cowork and keep their data access to the minimum necessary.

    Overall, Cowork is a solid, “safe-ish” option if you’re disciplined about what you share. The challenge is that utility often requires access to real data—exactly what we’re trying to protect. In an upcoming deep dive, I’ll outline strategies to keep malicious code out in the first place. While I’ll focus on local development, the same patterns can extend to Cowork with a bit of setup.

    One more important clarification: don’t confuse Cowork with the Code tab in the Claude Desktop app. Cowork runs code inside a virtual machine. The Code tab does not. If you ask Claude to write and execute code from the Code tab, that code runs on your local device and you’re fully responsible for security. There is one exception: the Code tab can run code in Anthropic’s cloud; I’ll cover that approach when we get into moving development off the local machine.

    To summarize Cowork’s protections against the attacker’s three-step pattern: installs and scripts still run, but they’re contained inside an isolated virtual machine instead of your real device; access to sensitive data is strongly limited to the specific folders you mount, leaving the rest of your filesystem (including unrelated credentials) out of reach; data exfiltration is partially constrained because Anthropic limits outbound network traffic from the VM—helpful, but not absolute. By contrast, local Code tab sessions offer no isolation, no filesystem restrictions, and no network limits—so any malicious install scripts run directly on your machine with full access and open egress.

    My takeaways so far: I still love building with AI, but I’m doing it more cautiously. Cowork offers meaningful containment when used deliberately. I still prefer the flexibility of Claude Code, and I’ve reconfigured my setup to reduce risk. Even so, “safer” isn’t “safe,” which is why I’m increasingly shifting development off my local device to more controlled environments. I’ll share the practical details—tools, configs, and scripts—in the next installments.

    If this perspective is useful, let me know. I want builders to move fast—and safely—through this new era of agentic AI. Until then, stay safe out there.

    Inspired by this post on Product Talk.

    Book a consult png image

  • A Reliable Amplitude AI Workflow for Product Decisions

    You ask Amplitude AI why activation fell. It returns a convincing explanation, a few plausible segments, and a recommendation your team could act on. The problem is that you still don’t know whether the answer reflects your product data, an ambiguous metric, or a reasonable-sounding guess.

    You don’t fix that uncertainty with a longer prompt. You fix it with a controlled workflow: define the decision, provide only the context needed to analyze it, let AI run a bounded sequence of checks, and require evidence before accepting a conclusion. The result is an analysis another product manager can inspect, reproduce, and turn into action.

    Start with a decision contract, not an open-ended question

    A request such as analyze our onboarding leaves too many choices to the model. It must decide what onboarding means, which users count, what success looks like, which period matters, and whether the goal is diagnosis or opportunity discovery. A polished answer can hide those unresolved choices.

    Write a short decision contract before opening the analysis. It should contain five elements:

    • Decision: State what someone will decide after reading the result. For example: decide which activation bottleneck the onboarding team should investigate next.
    • Population: Name the eligible users, accounts, plan types, platforms, markets, or acquisition channels.
    • Metric: Supply the exact event or formula, its time window, and any exclusions.
    • Evidence bar: Specify what the answer must show, such as the supporting events, segments, funnel steps, or behavioral trend.
    • Output: Ask for a conclusion, competing explanations, uncertainties, and the next analysis or product action.

    A useful objective is narrow enough to fit in one sentence. Your quality rubric can be slightly longer: require every conclusion to identify the relevant metric, population, comparison, and evidence. This intent-first, evaluation-driven approach keeps the analysis tied to a product decision instead of rewarding whatever answer sounds most complete.

    Constraints belong in the contract too. If the team cannot change pricing, instrumentation, or a particular onboarding step, say so. If a result must remain descriptive because the analysis cannot establish causality, require that distinction. AI is more useful when it knows which doors are closed.

    Build a compact context packet Amplitude AI can actually use

    Amplitude AI can only interpret behavior through the data model it receives. If two teams use different definitions of an activated account, or an event changed meaning after an instrumentation update, the model can produce a coherent answer to the wrong question.

    Create a reusable context packet for each important product area. Keep it short enough to review, but precise enough to remove semantic guesswork. Include:

    • Metric definitions: Write the numerator, denominator, qualifying window, and exclusions for activation, retention, conversion, or any other decision metric.
    • Event taxonomy: List the events and properties relevant to the question, including known aliases or deprecated events that should not be used.
    • Segment definitions: Explain how key cohorts are formed and which properties distinguish users from accounts.
    • Known data limitations: Flag missing platforms, delayed events, identity-resolution issues, tracking changes, and periods that should not be compared.
    • Recent product context: Include only releases, experiments, or journey changes that could plausibly affect the behavior under review.

    Use retrieval before expansion. Start with the smallest relevant set of definitions and observations. Add more context only when the analysis reaches a question that requires it. Dumping an entire analytics catalog into the prompt makes it harder to see which definitions shaped the answer and gives irrelevant details more chances to distract the model.

    Examples can stabilize recurring work, but choose them carefully. One to three strong examples are enough to demonstrate the expected structure, evidence standard, and level of uncertainty. Remove old conclusions and stale numbers before reuse. You want the model to copy the analytical pattern, not inherit a previous answer.

    Version this packet alongside the workflow. When an event definition, segment, or guardrail changes, record the change and rerun the analyses that depend on it. That turns context management from prompt housekeeping into part of your analytics governance.

    Run a bounded analysis loop, then challenge the result

    Move from observation to explanation in explicit steps

    Don’t ask for a diagnosis in a single jump. A reliable workflow separates what happened from why it may have happened. Use a fixed sequence:

    1. Establish the baseline. Confirm the metric definition, eligible population, comparison, and direction of change.
    2. Locate the difference. Break the result down by the segments most relevant to the decision. Avoid exploring every available property.
    3. Inspect the journey. Examine funnel steps, behavioral paths, retention patterns, or other views that can show where behavior diverges.
    4. Generate competing hypotheses. Ask for more than one plausible explanation and require supporting and contradicting evidence for each.
    5. Choose the next best analysis. Run the segment drill-down, funnel attribution, or anomaly check most likely to separate the leading explanations.
    6. Apply a stop rule. End when the evidence is sufficient for the stated decision, when the remaining uncertainty requires new instrumentation, or when another analysis would not change the next action.

    The stop rule matters. Without one, an agentic workflow can keep generating cuts of the data that add activity without increasing confidence. Before each tool call, require the system to state what question the analysis will answer and how each possible result would change its next step.

    If you expose Amplitude actions through MCP or another callable interface, keep each tool narrow and observable. A call should have explicit inputs, a recognizable output shape, and an error state the workflow can surface. Log the question, parameters, returned evidence, and the interpretation built from it. Tool access makes iteration faster; it does not remove the need for an audit trail.

    Put every conclusion through a verification gate

    Before a finding reaches a stakeholder, check it against a simple evidence ledger. For each important claim, record:

    • the event, metric, segment, funnel step, or trend that supports it;
    • the population and comparison to which it applies;
    • whether it is an observation, interpretation, or causal hypothesis;
    • the strongest alternative explanation;
    • the assumptions or data limitations that could change the conclusion;
    • the next check required if confidence is still too low for the decision.

    Then try to disprove the preferred answer. Ask whether the pattern survives a relevant segment change, whether a tracking change could explain it, and whether the same evidence also supports a competing hypothesis. This adversarial pass is often more valuable than asking the model to make its first response more detailed.

    Turn repeated checks into an evaluation set. Save representative questions, approved metric definitions, required evidence fields, and known failure cases. Rerun them when prompts, context, instrumentation, or model versions change. Review failures by category: wrong scope, wrong metric, unsupported inference, missed uncertainty, or unusable recommendation. That gives your team a regression signal instead of a vague impression that the workflow still works.

    Hand stakeholders a decision artifact, not an AI transcript

    The output should make the next decision easier. A long transcript of prompts, tool calls, and exploratory branches shifts the work of interpretation onto the reader. Keep the trace for auditability, but present a concise decision artifact with six fields:

    • Decision: The choice this analysis informs.
    • Finding: The clearest supported behavioral observation.
    • Evidence: The exact events, segments, funnel steps, or trends behind the finding.
    • Uncertainty: What remains unknown and what the analysis cannot establish.
    • Recommendation: The next analysis, discovery activity, experiment, or product change justified by the evidence.
    • Owner: The person responsible for the next step and the condition that triggers a follow-up.

    Keep human judgment at the decision boundary. Amplitude AI can retrieve definitions, propose analyses, call tools, compare patterns, and draft the artifact. A product leader should still decide whether the evidence is strong enough, whether the recommendation fits current constraints, and whether the cost of being wrong is acceptable.

    That division of labor also clarifies accountability. If the AI workflow produces an unsupported inference, improve the context, tool contract, or evaluation. If the evidence is sound but the organization chooses a different path, record the strategic reason. Don’t let an AI-generated recommendation blur the difference between analytical output and an accountable product decision.

    Key takeaways

    • Begin with the decision, population, metric, evidence bar, and required output.
    • Give Amplitude AI a small, versioned context packet instead of an unfiltered analytics catalog.
    • Separate baseline measurement, segmentation, journey analysis, hypothesis generation, and the next tool call.
    • Require evidence, alternatives, assumptions, and a stop rule before accepting a conclusion.
    • Save recurring checks as evaluations and rerun them when data, prompts, tools, or models change.
    • Deliver a decision artifact with a named owner while keeping the analytical trace available for review.

    Start with one recurring product question this week. Write its decision contract, assemble the minimum context packet, and define the verification gate before asking Amplitude AI to analyze anything. Once that workflow survives review, save it as the template for the next question.

    References