Tag: AI workflows

  • How Snapbar Turned Crisis Into an AI-Native Photo Experience Revolution

    How Snapbar Turned Crisis Into an AI-Native Photo Experience Revolution

    What does it take to reinvent a 14-year-old company, not once, but twice? I ask that question often when I look at mature product organizations, because the hardest transformations rarely start with a clean slate. They start with real customers, legacy expectations, operational muscle memory, and a market that suddenly refuses to behave the way it used to.

    Snapbar is a useful case study in that kind of transformation. The company began as a wedding photo booth side hustle, grew into a national events company, and then watched COVID wipe out the entire business overnight. As a product leader, I find that moment especially important because it separates teams that are attached to the current expression of their product from teams that understand the deeper customer need underneath it.

    The deeper need was never just a physical photo booth. It was identity, participation, memory, brand engagement, and a shareable experience that people could take with them. When in-person events disappeared, Snapbar went from physical photo booths to a cross-platform virtual product built on WebRTC in spring 2020. That was not a cosmetic pivot. It was a first-principles rebuild under pressure.

    I have seen many teams talk about innovation when conditions are favorable. Snapbar’s story is more interesting because the team had to innovate when the existing business model was unavailable. That kind of constraint can be clarifying. It forces product teams to ask: What job are we really doing for customers, and what parts of our current solution are merely historical artifacts?

    The next reinvention came from generative AI. Pushed by declining repeat business, Snapbar dove deep into Stable Diffusion, custom LoRA fine-tunes on H100/H200 GPUs, and eventually a reasoning-model-powered generative image and video pipeline. What stands out to me is not simply that the team adopted gen ai. It is that they connected AI capabilities to a domain they already understood deeply: photography, events, brand activations, and experiential marketing.

    This distinction matters. In product management, technology FOMO can lead teams to bolt AI onto workflows without a clear strategic advantage. Snapbar appears to have moved differently. They used 14 years of industry knowledge to identify where AI could change the experience itself, not just automate a back-office task or generate a novelty output.

    The product evolution is a strong example of applied AI. Snapbar integrated Stable Diffusion 1.5 as their first generative AI model and ran custom LoRA fine-tunes on H100/H200 GPUs to produce brand-quality outputs nobody else in their space could match. That level of execution shows the difference between experimenting with a model and building a differentiated product system around it.

    I also appreciate the way the team moved from negative prompts to reasoning model long-form prompts. In brand environments, creative control and safety control are not optional. A brand activation must feel imaginative, but it also has to remain on-message, inclusive, and predictable enough for a live event setting. Better prompt engineering becomes part of the product’s trust layer.

    One of the most important product details is the meta-prompting pre-processing pipeline designed to ensure user likenesses, including non-obvious details like disabilities, are accurately represented in generated images. That is not a minor implementation detail. It reflects a more mature view of AI risk management, representation, and customer experience.

    From my perspective, this is where product strategy and ethical technology intersect. Generative AI systems can easily flatten people into generic outputs. A thoughtful product team has to decide what fidelity means, what consent means, and how much control users and brands should have over the final artifact. Snapbar’s approach suggests that representation is not just a model-quality problem; it is a product-design problem.

    Podcast cover art for Just Now Possible with Teresa Torres, featuring bold white and yellow text on a navy background and Snapbar AI photo booth branding.
    Just Now Possible spotlights Snapbar’s journey from photo booths to AI-powered brand experiences, framing reinvention, creativity, and applied AI as the center of the conversation.

    The company’s experiential marketing platform lets brands “world build” at conferences, trade shows, and live events by bringing fans into branded creative worlds. That phrase matters because it reframes the photo booth from a capture device into a participatory brand system. The user is not merely photographed. The user becomes part of a designed world.

    I see this as a broader shift in product experience. Static brand impressions are giving way to co-created moments. Snapbar added participatory user inputs through Mad Lib-style prompts and prompt injection, turning photo experiences into co-creation moments between brands and their audiences. That is a more durable engagement loop than simply asking someone to pose in front of a branded backdrop.

    The operational story is just as relevant for product and engineering leaders. Snapbar used Claude Code and Codex to build and ship features rapidly as a small bootstrap team, and developed a four-pillar agent orchestration framework: context, tools, verification, and workflows. I like that framing because it treats AI-assisted development as a system of work, not a magic shortcut.

    In my own product leadership work, I keep coming back to the same lesson: AI workflows only become reliable when the team defines the surrounding operating model. Context determines whether the agent understands the problem. Tools determine what it can actually do. Verification determines whether the output is trustworthy. Workflows determine whether the capability compounds across the organization.

    Snapbar is now building customer-facing “vibe coding” using the Claude Agent SDK so brands can configure and create experiences themselves within Snapbar’s platform. That is a meaningful product move. It shifts creation closer to the customer while keeping the workflow inside a controlled product environment. For brand teams, that could reduce dependency on custom service work while still preserving creative flexibility.

    This is the kind of AI Strategy I find most compelling: not a generic claim that AI will transform everything, but a specific path from domain expertise to product capability to customer empowerment. Snapbar did not abandon its past. It converted years of event, photography, and brand knowledge into a new interface for generative AI.

    The core lesson for product teams is clear. Reinvention does not always mean discarding the original business. Sometimes it means identifying the durable customer need, rebuilding the delivery mechanism, and then using new technology to expand what the experience can become. Snapbar’s journey from wedding photo booths to virtual WebRTC experiences to AI world building shows how a team can preserve its market intuition while changing nearly everything about the product surface.

    For product leaders evaluating gen ai opportunities, I would take three practical lessons from this story. First, start with the customer experience, not the model. Second, treat brand safety, representation, and verification as product requirements from the beginning. Third, use agentic AI internally only when the team has a clear framework for context, tools, verification, and workflows.

    Snapbar’s story resonates because it is not about chasing a trend. It is about a team using necessity, curiosity-led self-education, and disciplined product thinking to build something that feels native to the generative AI era. That is the difference between adopting AI and becoming AI-native.

    Inspired by this post on Product Talk.

    Book a consult png image
  • Reliable AI Coding Requires Four Kinds of Control

    Reliable AI Coding Requires Four Kinds of Control

    Reliable AI coding is not primarily a matter of finding a better prompt or a more capable model. It is a workflow-design problem: teams must control what the product should do, what the repository currently does, what the model can see, and what the agent is allowed to change.

    Managing those four kinds of state turns an AI coding session from an open-ended conversation into a bounded engineering process. The payoff is faster iteration without treating plausible output, confident status messages, or large context windows as substitutes for evidence.

    Reliability depends on the surrounding system

    A large language model generates an answer token by token from the input available to it. That input can include more than the visible request: an application may add system instructions, conversation history, project files, enabled tools, skills, and other supporting context. As Shivam.Consulting Blog’s guide to how ChatGPT works explains, the surrounding application therefore helps shape the result even when two products use the same underlying model.

    This mechanism has an important operational consequence. An agent can produce code that looks convincing without possessing a stable model of the intended product, the complete repository, or the runtime environment. Fluency indicates that the output fits learned patterns; it does not establish that the implementation satisfies the requirement.

    A dependable workflow consequently controls four connected states. Product state covers requirements, constraints, permissions, edge cases, and acceptance criteria. Repository state covers the actual code, data model, dependencies, tests, and uncommitted changes. Model state covers the instructions and evidence present in the context window. Execution state covers tools, filesystem access, commands, network activity, and other permissions. A failure in any one can appear to be a coding error even when the code is not the original cause.

    Tool selection should reflect that distinction. Shivam.Consulting Blog’s vibe-coding playbook recommends managed app builders when the purpose is to explore an interaction or answer an early product question, while positioning developer-oriented coding agents as more appropriate for existing repositories, multi-file changes, tests, and review workflows. The useful dividing line is not whether a tool can generate code. It is whether the environment exposes enough control and evidence for the consequence of the change.

    Convert product intent into a bounded change contract

    Many unreliable sessions begin before an agent edits a file. If the requested behavior, non-goals, affected users, data rules, and observable success conditions remain ambiguous, the model must fill the gaps. Each follow-up correction can then preserve a different assumption, creating a chain of locally plausible patches without a coherent final design.

    A stronger starting point is a compact change contract written outside the chat. It should identify the outcome, relevant current behavior, permitted scope, important invariants, expected edge cases, and the evidence that will demonstrate completion. For a defect, that evidence begins with a reproducible failing case. For a feature, it includes examples of accepted and rejected behavior. The contract should also record explicit non-goals so that an agent does not broaden a narrow request while attempting to be helpful.

    Blast radius deserves separate attention. The vibe-coding playbook uses data, controller, and view as a practical three-layer model. A request involving permissions, sorting, filtering, workflow state, or reporting may cross all three even if it appears in the interface as a small change. Reviewing the planned impact across storage, logic, and presentation helps reveal missing migrations, inconsistent validation, stale queries, and user-interface states before implementation begins.

    The same source proposes separate plan-review-fix and implement-review-fix loops. Combined with the change contract, these become distinct gates rather than one continuous conversation. The plan gate asks whether the proposed files, layers, and tests match the requirement. The implementation gate asks whether the resulting diff and observed behavior match the approved plan. Separating the gates makes it easier to reject a mistaken approach before it accumulates code.

    This structure also clarifies the human role. The agent can explore the repository, propose a plan, implement a bounded change, and help investigate failures. Product and engineering owners remain responsible for deciding what behavior is correct, which tradeoffs are acceptable, and what evidence is sufficient to ship.

    Treat context as a limited working set, not permanent memory

    A long conversation can feel comprehensive while becoming less dependable. Shivam.Consulting Blog’s context-rot analysis reports research showing that model performance can deteriorate as input length grows and that information at different positions may receive unequal attention. The article’s practical conclusion is more useful than any advertised context-window maximum: available capacity should not be confused with reliable attention.

    Context should therefore be curated as a task-specific working set. Durable facts belong in versioned project documents; the active session should receive only the instructions, files, decisions, and evidence needed for the current change. Old tool output, abandoned plans, duplicate explanations, and superseded requirements consume attention without improving the task.

    Shivam.Consulting Blog’s guide to Claude Code workflows describes a layered memory pattern: broad preferences in global instructions, project-specific conventions in repository-level files, and reference material loaded when relevant. It also presents stored commands as a way to make recurring procedures explicit, and sub-agents as a way to isolate context or perform independent work. The transferable principle is architectural rather than product-specific: stable policy, project knowledge, task instructions, and transient evidence should not be mixed into one ever-growing transcript.

    A clean session boundary can be a reliability control. When a conversation has accumulated contradictory instructions or repeated failed fixes, the next step should not automatically be another patch request. A new session can begin from a short handoff containing the approved change contract, current repository state, attempted approaches, observed failures, and unresolved questions. This preserves useful evidence without carrying the entire history forward.

    Sub-agents require the same discipline. Parallelism is valuable when work can be partitioned into independent questions, such as locating relevant code, examining tests, or reviewing a proposed diff. It is less useful when several agents can modify overlapping files or make incompatible architectural assumptions. Each delegated task needs a narrow scope, an expected output, and a rule for whether it may write or only report.

    Require evidence, limited authority, and a recovery path

    An agent’s statement that a problem is fixed is a claim to verify, not completion evidence. Verification should return to the original reproducer or acceptance criteria, then examine the diff and run the smallest relevant checks. Broader tests can follow when the change crosses modules, alters shared behavior, or affects data. This sequence distinguishes a real correction from a patch that merely changes the visible symptom.

    Review should inspect both behavior and change shape. A diff may pass a narrow test while introducing unrelated refactoring, weakening validation, swallowing errors, or duplicating logic. Unexpected file changes, new dependencies, disabled checks, and unusually broad edits are signals to pause. If the evidence is inconclusive, the workflow should return to diagnosis rather than asking the same context-saturated agent to keep editing.

    Reliability also depends on limiting what an agent can do. Shivam.Consulting Blog’s Claude Code risk guide describes escalating exposure as an agent moves from reading a project folder to reading elsewhere, fetching external material, writing files, executing generated code, and installing third-party packages or extensions. Although permission models vary by product, the general control is consistent: grant the least authority required for the current step and review the exact path or command before approval.

    Folder boundaries should match the task boundary. Credentials, customer information, confidential documents, and unrelated projects should not be placed within an agent’s working scope. One-time approval is preferable when an operation is unusual or its future use would be difficult to predict. Commands that delete, overwrite, upload, install, or execute deserve more scrutiny than read-only inspection because their impact is larger or harder to reverse.

    Reversibility completes the control system. The safety guide emphasizes backups and version control because an AI coding interface may not provide a dependable undo operation. A clean checkpoint before implementation, small commits, reviewable diffs, protected secrets, and a tested rollback path reduce the cost of both model errors and human approval mistakes. For higher-risk work, the agent should operate in a disposable branch, isolated environment, or similarly constrained workspace rather than directly against valuable state.

    These safeguards are mutually reinforcing. A bounded contract limits scope; curated context reduces instruction drift; verification exposes incorrect claims; least privilege limits blast radius; and version control makes recovery practical. Removing any one of them shifts too much trust onto probabilistic output.

    Key takeaways

    • Control product state, repository state, model context, and execution authority as separate parts of one workflow.
    • Write a change contract with scope, non-goals, invariants, edge cases, and acceptance evidence before implementation.
    • Keep context task-specific; store durable knowledge in files and start a clean session when history becomes contradictory or noisy.
    • Treat an agent’s completion report as a hypothesis until the original reproducer, relevant tests, observed behavior, and diff support it.
    • Match permissions and isolation to the risk of the operation, and create a recovery point before allowing changes.

    As coding agents gain more tools and autonomy, reliable teams will distinguish themselves less by how much work they delegate than by how clearly they define authority, evidence, and recovery. The durable advantage will come from workflows in which faster generation is paired with tighter control.

    References

  • Behavioral Analytics for AI Agent Activation and Retention

    Behavioral Analytics for AI Agent Activation and Retention

    AI agent growth is not simply a matter of attracting more users or generating more conversations. The central product question is whether people reach a useful outcome quickly enough to return, and whether the organization can respond intelligently when that journey breaks down.

    The two source accounts describe complementary parts of that challenge. The Pendo account focuses on measuring and improving the path from first use to recurring engagement, while the Amplitude account focuses on turning observed behavior into workflows across product and go-to-market systems. Together, they suggest an operating model in which analytics first identifies meaningful behavior and then helps teams act on it.

    Treat the agent as a measurable product experience

    An AI agent can appear busy without becoming valuable. Conversation counts, prompt volume, and feature exposure show activity, but they do not establish that users completed meaningful work. Behavioral analytics becomes more useful when the agent is treated as an end-to-end product experience rather than an isolated interface.

    The Pendo account describes mapping the journey from activation and a first successful task through repeat usage and habit formation. It also reports that the team defined stickiness around the agent’s jobs to be done instead of relying on an unspecified generic engagement measure. That distinction matters because a meaningful return pattern depends on the work the agent is intended to support.

    The Amplitude account extends the same reasoning beyond analysis. It describes agents operating on verified product events, including high-intent milestones, changes in feature adoption, and signals associated with churn risk. In this model, instrumentation is not merely a reporting layer. It supplies the evidence used to trigger a subsequent decision or workflow.

    A practical measurement chain therefore begins with eligibility and exposure, continues through an attempted interaction and a verified first success, and then examines whether users achieve additional useful outcomes over later sessions. The exact events must reflect the agent’s purpose. The durable principle is to measure completed value, not just interface activity.

    Define activation as the first meaningful success

    Activation is most informative when it marks a result that demonstrates the agent’s value. Opening the agent, viewing a suggested prompt, or sending a message may be necessary steps, but none necessarily proves that the user accomplished the intended task.

    Pendo’s account reports that activation contained unnecessary cognitive load and that the first-session path did not consistently lead users to a quick win. The reported response included simplifying onboarding, clarifying prompts, and using in-app guidance to make valuable capabilities easier to recognize. This connects activation analysis directly to product design: when users stall before a first success, the remedy may involve reducing choices, clarifying expectations, or improving contextual guidance rather than adding more agent functionality.

    Journey analysis should separate several different failure modes. A user who never starts may not understand the value proposition. A user who starts but abandons the task may encounter interaction friction. A user who receives an answer but does not act on it may lack confidence, context, or a clear next step. Combining these outcomes into one conversion rate would hide the product decision each one implies.

    Activation should also be connected to the behavior that follows it. If an event labelled as success has no observable relationship with later value, it may be a convenient instrumentation point rather than a meaningful milestone. Behavioral cohorts can help compare subsequent engagement among users who reached different early outcomes, although those relationships should initially be treated as diagnostic evidence rather than proof of causation.

    Measure retention as repeated value, not raw frequency

    Retention analysis asks whether users continue to obtain value after activation. For an AI agent, that requires more context than a simple count of returning users. A return can indicate trust and usefulness, but it can also reflect an unresolved task, repeated correction, or a workflow that unnecessarily forces the user back.

    The Pendo account presents stickiness as a proxy for trust and reports a 61% increase after the team established Agent Analytics and ran a series of product experiments. The same source associates stronger return behavior with proactive anticipation of intent and associates context-rich interactions, supported by timely nudges and in-app guides, with deeper engagement over later sessions. These are reported findings from one product account, not an independently verified benchmark for other agents.

    The more transferable lesson is methodological. Teams can segment retention by the early behavior users completed, the type of task attempted, and the context surrounding the interaction. They can then examine whether retained users are repeating successful work, expanding into additional useful tasks, or merely revisiting the same point of friction.

    This approach also guards against optimizing stickiness in isolation. Frequent use is desirable only when it reflects repeated useful outcomes. Where the agent’s job is to resolve work efficiently, fewer interactions may sometimes represent a better experience than a longer conversation. The retention definition must therefore stay anchored to the user’s intended result.

    Turn behavioral signals into controlled interventions

    Analytics creates leverage when it changes what the product or organization does next. The sources cover two levels of intervention. Pendo describes changes inside the experience, such as onboarding simplification, prompt clarification, contextual guides, tuned triggers, and tighter feedback loops. Amplitude describes workflows that cross system boundaries, such as initiating outreach for churn risk, triggering experimentation when adoption falls, activating users after high-intent milestones, and updating CRM records.

    These approaches are complementary. In-product interventions can help a user complete the current journey, while cross-functional workflows can coordinate actions that require product, sales, or customer-success involvement. The behavioral signal should determine which response is appropriate: interface friction calls for a product change, an unmet need may call for research, and an account-level risk signal may justify a carefully governed human follow-up.

    Automation does not remove the need for experimentation. Pendo reports using A/B tests to evaluate changes, while the Amplitude account emphasizes success criteria, governance guardrails, observability, iteration, and aligned performance measures. A sound operating loop combines those ideas: define the target behavior, verify the underlying events, choose an intervention, test its effect, monitor unintended outcomes, and retain only changes that improve the intended user result.

    That loop is especially important when an agent both interprets behavior and initiates action. Event quality, ambiguous thresholds, or drifting agent performance can otherwise scale an incorrect decision. Human ownership, visible workflow history, and clear evaluation criteria help distinguish useful orchestration from automated noise.

    Key takeaways

    • Define activation around a verified first useful outcome, not merely opening the agent or sending a prompt.
    • Analyze each stage between exposure, attempted use, successful completion, and later return so different forms of friction remain visible.
    • Interpret retention through repeated value and task context; activity alone is not sufficient evidence of trust.
    • Use behavioral cohorts to generate hypotheses, then apply controlled experiments before treating an observed relationship as causal.
    • Match interventions to the signal: improve the experience when friction is local, and use governed cross-functional workflows when follow-through spans multiple systems or teams.
    • Monitor data quality and agent performance because automated actions can amplify both accurate and inaccurate interpretations.

    The next stage of AI agent maturity will depend less on adding visible capabilities and more on connecting meaningful outcomes to disciplined follow-through. Teams that can measure the first win, recognize repeated value, and govern the actions between them will be better positioned to turn agent adoption into durable product behavior.

    References

  • How I Make AI Agents Speak Like Our Team: A Conversation Design Playbook That Lifts CSAT

    How I Make AI Agents Speak Like Our Team: A Conversation Design Playbook That Lifts CSAT

    If nobody on our team trains the Agent on how to communicate, it will sound like an LLM when it speaks to customers—because it is one. I never want a customer to feel like they’re talking to a machine that doesn’t get them. That’s why I treat conversation design as a core product capability, not an afterthought.

    Conversation design is an emerging discipline in AI-first support teams built to solve this exact problem. In practice, I make someone explicitly own how the Agent communicates—tone, structure, level of detail, customer experience, and the handoff and escalation process—because that’s where trust is won or lost.

    When there’s no clear owner and no explicit guidance, the Agent starts making its own choices. I’ve seen it over-explain when a short answer would do, reply in a flat tone when a customer is frustrated, or trigger a handoff too late. None of those are model problems; they’re design problems.

    The cost is measurable. Customers who get awkwardly structured responses won’t trust the answer—even when it’s accurate—so they escalate to a human to hear the same thing phrased differently. Others will skip the Agent entirely. And when the Agent does hand off, a poor transition means the support rep inherits a frustrated customer. Every one of these outcomes is avoidable; conversation design exists to prevent them.

    I’ve seen A/B tests where a warmer, more conversational opening message meaningfully lifted customer satisfaction—CSAT moved from 72.8% to 78.4%. A single design change, applied to the very first message, drove a measurable difference. That’s the kind of leverage I look for as a product leader.

    Here’s the scope I use when I talk about conversation design—five areas that shape the customer experience end to end:

    1) Tone and personality: Define the Agent’s voice, level of detail, and how formal or casual it should sound—and specify where that register adapts to the situation (for example, urgent access issues versus exploratory product questions).

    Screenshot of an AI agent conversation design dashboard showing guidance for tone of voice, communication style, product naming, and a live preview, with callouts highlighting brand voice controls and response formatting.
    Design how your AI agent talks. Set tone, style, and product naming rules, then preview replies instantly. Clear callouts showcase brand voice consistency and flexible formatting so your bot communicates like your team.

    2) Response structure: Ensure the Agent matches the level of detail to the customer’s request, keeping answers tight when the ask is simple and expanding only when complexity demands it.

    3) Handoff logic: Decide when to escalate, how to communicate the transition, and what context to carry over so the human teammate can help immediately without rework.

    4) Interaction flow: Map how a conversation progresses—clarifying questions, answers, resolution, or handoff—and design for smooth pivots when customers change direction.

    5) Response quality: Go beyond technical correctness to ensure answers feel clear, helpful, and on-brand. Accuracy without clarity erodes trust.

    To put this into practice, I start with the feel of the conversation. Before tuning individual responses, I write down one tight paragraph describing the Agent’s voice. I don’t need a full brand bible—just a north star I can use to make consistent decisions about tone. The voice stays consistent, while the register adapts to the context: a locked-out customer needs directness and speed; a feature explorer might value more context and examples.

    I design the handoff with extreme care because it’s one of the highest-friction moments. Customers shouldn’t have to re-explain anything. The support rep should receive the full conversation history, the underlying context, what the Agent already tried, and why the escalation happened. Even the phrasing matters—“Let me connect you with a teammate who can help with this” feels very different from a silent handover.

    CX analytics dashboard with a CX Score of 3 and a donut chart of drivers: policy feedback, answer quality, customer effort, product or service feedback, and strong emotion beside an AI agent chat transcript.
    The new CX Score adds context to every conversation: a donut chart surfaces drivers like policy feedback and effort, while a side panel explains why this interaction earned a 3 based on signals from an AI agent chat.

    I also build a failsafe. If the Agent can’t resolve the issue cleanly, a graceful fallback still gives the customer a smooth experience. A customer might be frustrated with AI at that point, but a well-handled transition can turn that around.

    Follow-ups deserve the same rigor as handoffs. If someone drops mid-conversation—with the Agent or a human—how do we reach back out to confirm they got what they needed? Most teams miss this moment; customers don’t.

    Another common pitfall is over-explaining. The Agent has access to a lot of information, and left unguided, it will overshare. The fix is simple: match the answer’s depth to the question. A password reset shouldn’t take three paragraphs; a complex integration might. When there’s more to offer, the Agent should ask before expanding.

    I also design for the conversation the customer is actually having—not the script I wish they’d follow. Customers change direction, stack questions, or bring up unrelated follow-ups. The Agent should pivot with them, not force them back into a rigid flow. I also consider whether flows vary by channel and whether different segments merit distinct experiences.

    On the instruction side, I keep guidance short. Teams often react to edge cases by adding more rules until the LLM is parsing paragraphs before it can reply. I’ve seen it everywhere. My rule: if it’s about content or information, it belongs in the knowledge base. If it’s about tone or handling specific situations, it belongs in the Agent’s instructions. “Be direct about pricing” does more than a paragraph explaining the philosophy behind your pricing communication strategy.

    If you’re using Fin, much of this work happens in Guidance. It’s where conversation design takes shape, helping you define how the Agent should sound, how much it should say, and how it should respond in different situations.

    Word 'Blueprint' drawn as blue vector outlines with anchor points on a light grid, with text about the AI Agent Blueprint as a strategic map for launching and scaling AI in customer service.
    On a crisp grid, 'Blueprint' appears as editable vector paths, underscoring a methodical plan. The image promotes the AI Agent Blueprint—a framework to launch and scale customer service automation with confidence.

    Most teams won’t hire a dedicated conversation designer on day one—that’s fine. But someone still needs to own the Agent’s communication, even if it’s part of an existing role. I’ve often seen this start within support operations or knowledge management. As the Agent scales to more conversations, the responsibility becomes formal—and eventually becomes a dedicated role.

    Here’s how I’d start, step by step:

    1) Name an owner. Make accountability explicit; it doesn’t have to be a new hire.

    2) Pick one conversation type that isn’t landing well. Look for cases where the Agent answered correctly but the customer still escalated or left negative feedback. If you’re using Fin, CX Score can help you surface these; it shows which topics and conversation types are scoring poorly and why, so you can see whether the issue is answer quality, customer effort, or something else.

    3) Audit the Agent’s instructions. If they’ve grown beyond a few focused rules, trim them. Move content into the knowledge base and keep instructions focused on behavior.

    4) Fix your worst handoff. Review a handful of conversations that escalated. Did the customer have to repeat themselves? Did the rep have enough context? Redesign that single transition first.

    The impact of these small improvements compounds. A warmer opening can lift CSAT, trimming instructions makes responses sharper, and a better handoff prevents reps from inheriting frustrated customers. None of this requires new knowledge—just someone paying close attention to the conversation itself and designing it with intention.

    Inspired by this post on The Intercom Blog.

    Book a consult png image

  • How I Use Novus, the First Product Agent, to Turn Rapid Releases into Measurable Wins

    In a world of relentless CI/CD and accelerating release trains, product leaders like me can’t afford lagging signals or fuzzy readouts on what’s truly moving the needle. I need immediate, trustworthy feedback that connects code shipped to outcomes achieved and customer value created.

    Coding agents compress weeks of development into hours, but the faster your codebase changes, the harder it is to know what’s actually helping end-users.

    That tension is exactly why I brought Novus into my product toolbox. To keep up with the pace of development, over 600 product teams are already using Novus, the first-of-its-kind product agent, to automatically set itself up, monitor product data, and tell you what to do next.

    From my chair, that promise matters only if it translates into clear decisions. With Novus, I’ve been able to tighten the loop between experimentation and learning: it pairs eval-driven development with behavioral analytics and observability so I can see how a release influences activation, engagement, and retention—without spelunking through fragmented dashboards. The agentic AI backbone reduces the manual stitching I used to do across events, cohorts, and funnels, letting me focus on prioritization and product strategy instead of report wrangling.

    Day to day, Novus fits naturally into our AI workflows. It surfaces anomalies early, clarifies trade-offs, and frames next-best actions in the language of outcomes. Because it plugs into a unified analytics platform approach, I can maintain continuous discovery at scale while preserving the rigor of Agent Analytics: hypotheses are explicit, telemetry is consistent, and results are traceable. That’s the operating cadence I expect from modern product management leadership.

    If your roadmap moves faster than your learning loops, a product agent can be the missing link between speed and certainty. Novus helps me convert rapid releases into measurable wins, keeping the team aligned and confident about what to build next—and just as importantly, what to stop doing.

    Inspired by this post on Pendo – Best Practices.

    Book a consult png image
  • Beyond Black‑Box Scores: Custom AI That Elevates Trust & Safety Without Burnout

    Beyond Black‑Box Scores: Custom AI That Elevates Trust & Safety Without Burnout

    What do you do when off-the-shelf moderation scores aren't good enough—and the alternative is paying human contractors to spend their days reviewing traumatizing content at scale? I’ve wrestled with that exact trade-off in enterprise environments, and it’s why I was eager to unpack how custom AI can raise the bar on trust and safety without compromising accuracy, latency, or the well-being of our teams.

    In this episode of Just Now Possible, I sit down with Nikki Marinsek (Data Scientist), Brian McCaffrey (Software Engineer), and Dan Means (Machine Learning Engineer) from Musubi, an AI-native trust and safety toolkit for content platforms. Musubi builds custom-trained ML models and LLM-powered moderation tools that adapt to each platform's unique policies—from dating apps to social networks to AI inference endpoints. As a product leader, I’m drawn to their blend of eval-driven development, agentic AI, and pragmatic deployment pipelines that actually meet real-world SLAs.

    We walk through their full journey—starting with a first prototype on tabular data—then discovering the system was sometimes catching issues human moderators missed. That insight became a forcing function to formalize evaluation, calibrate thresholds, and design feedback loops that help humans and models converge. Just as importantly, they built a policy optimizer that uses agentic flows so non-technical trust and safety teams can iterate on LLM moderation policies without needing a data scientist in the room.

    If you’ve ever had to balance latency, accuracy, and cost at scale, you’ll appreciate how Musubi tests trade-offs across traditional ML, embedding-driven classification, and LLMs. Their approach mirrors the patterns I expect in high-throughput stacks: cache and pre-compute where possible, contain worst-case latencies, and push evaluation tooling to customers so policy changes are safe, observable, and fast to deploy.

    What resonated most with me is their core product strategy: put eval tools directly in customers’ hands. When teams can benchmark AI against humans, referee disagreements using “LLM as judge,” and make policy gaps visible, trust increases and operational drift decreases. That’s the foundation for durable product strategy in sensitive domains like content moderation, fraud management, and risk scoring.

    Listen to this episode on: Spotify | Apple Podcasts

    Guests: Nikki Marinsek, Data Scientist, Musubi; Brian McCaffrey, Software Engineer, Musubi; Dan Means, Machine Learning Engineer, Musubi.

    In this episode: Why off-the-shelf moderation scores fail and how custom-trained models fix that; How Musubi combines traditional ML with LLMs for different moderation tasks; The discovery that AI can outperform human moderators—and how to communicate that to clients; Using AI as a judge to referee disagreements between AI and human decisions; How Musubi onboards new customers with "reverse demos"; What custom model training actually means: fine-tuning, feature engineering, and reusable deployment pipelines; The policy optimizer: an agentic flow that helps customers iterate on their LLM moderation policies; Why pushing eval tools directly to customers is a core product strategy; How Musubi is building flexible orchestration workflows for non-technical trust and safety teams.

    From a product management lens, a few highlights stand out. First, the disciplined separation of concerns: use traditional ML for high-precision, low-latency pattern detection and LLMs for nuanced policy interpretation. Second, invest in golden sets and policy loops early so you can quantify improvement and avoid subjective debates. Third, productize customization—create reusable deployment pipelines, parameterized policies, and self-serve evaluation—so each customer’s “custom model” still scales like a platform.

    I also appreciated the onboarding tactic of "reverse demos." Rather than a canned walkthrough, the team invites customers to bring real policies and edge cases, then instruments the workflow live. That move builds credibility, accelerates discovery, and surfaces the fastest paths to value—an approach I recommend whenever you’re selling complex AI workflows to non-technical stakeholders.

    If you’re navigating cost and latency trade-offs, the conversation goes deep on techniques like embedding-driven classification, fine-tuning vs. training, and when to route decisions through LLM adjudication. My takeaway: treat the router, the evaluator, and the policy as first-class products. When those elements are observable and testable, you can raise quality without exploding compute costs or creating operational bottlenecks.

    Resources & Links: Musubi — AI-powered trust and safety toolkit for content platforms. Maven AI Evals Course — AI evals course.

    Chapters: 00:00 Meet the Team; 01:18 Why Everyone Wears Product; 02:32 What Musubi Builds; 04:51 AI for Human Moderation; 09:59 Adversaries and Asymmetry; 11:48 Early Days and Low Latency; 13:35 First Prototype Slice; 15:33 Traditional ML Meets LLMs; 19:52 Benchmarking Against Humans; 23:09 LLM as Judge and Policy Gaps; 29:53 From Prototype to Platform; 31:15 Customer Onboarding Reverse Demos; 36:08 Custom Models Per Customer; 38:05 Fine Tuning vs Training; 39:14 Embedding Driven Classification; 40:04 Cost and Latency Tradeoffs; 43:21 Productizing Customization; 49:16 Scaling Prototypes to Production; 51:58 Golden Sets and Policy Loops; 56:17 Coaching Customers Safely; 01:02:06 Gamified Feedback Signals; 01:06:19 Agentic Toolkit Roadmap; 01:09:05 Workflow Orchestration Future; 01:12:06 Wrap Up and Thanks.

    Ultimately, this is a playbook for modern trust and safety: align your models to your policies, make evals a habit not an event, and empower non-technical teams with agentic workflows and transparent metrics. That’s how we move beyond black-box scores to systems we can measure, manage, and trust.

    Inspired by this post on Product Talk.

    Book a consult png image
  • Secure System Access for AI Agents: A Phased Control Model

    Secure System Access for AI Agents: A Phased Control Model

    An AI agent becomes operationally valuable when it can move beyond explaining a process and complete the underlying work. That same transition gives the agent access to sensitive data and consequential actions, so integration must be designed as both a product capability and a security boundary.

    The practical objective is not maximum access. It is the smallest dependable set of permissions that lets an agent resolve a well-defined workflow, supported by deterministic controls, observable outcomes, and a clear path to human intervention.

    System access changes both the value and the risk

    Without backend access, an agent can describe how to update an account, check a renewal, or report a damaged order. With access to a CRM, billing platform, or order-management system, it can potentially retrieve the relevant record and complete the request during the conversation. The Intercom article presents this shift from answering to acting as a central difference between basic AI adoption and mature deployment.

    The article cites Intercom’s 2026 Customer Service Transformation Report, reporting improved metrics among 87% of teams with mature AI deployments, compared with 62% overall. It also reports that 82% of senior leaders said their teams had invested in AI during the preceding year, while only 10% said they had reached mature deployment. These source-reported figures suggest an integration gap, but they do not independently establish that system access caused the reported improvements or that an integration is secure.

    Security therefore cannot be added after the workflow succeeds. A customer-facing interface may remove the need to visit a separate application, but it must not remove identity and authorization checks. The agent still needs a trustworthy way to associate the request with the correct customer, determine what that customer is permitted to do, and constrain the backend operation accordingly.

    Choose workflows where access justifies its complexity

    Not every automated conversation benefits equally from deeper integration. Intercom reports the results of rebuilding four fixed, scripted Tasks as Procedures with system access. Over the 12 months through May 2026, the reported resolution rate for its bounce-list workflow rose from 9.3% to 79.9%, while bug reporting increased from 9.2% to 66.5%. Email forwarding moved from 44.9% to 66.5%, but Messenger installation rose only from 67% to 69.2%.

    The variation is more instructive than the headline gains. According to the article, the bounce-list process required multi-step reasoning, dynamic branches, and error recovery. Bug reporting still ended in a human handoff, but the procedure improved that handoff by pre-triaging the issue, surfacing possible GitHub matches, extracting relevant URLs, and requesting impersonation access. Messenger installation was already a comparatively linear process, leaving less room for improvement.

    A suitable first integration is therefore not merely a popular support topic. It should be high-volume and repeatable, have an identifiable system owner, and depend on live data or actions that materially change the outcome. Existing APIs improve feasibility, but the security review should also consider data sensitivity, reversibility, authorization complexity, and the consequences of acting on an ambiguous request.

    Use an access ladder instead of a single launch

    The phased approach described by Intercom can also serve as a security model. Each stage expands capability only after the workflow and its controls have produced enough evidence to justify the next step.

    StageAgent capabilityAppropriate useControl emphasis
    No integrationGuide, troubleshoot, check policy, triage, and routeDiscover where explanations repeatedly lead to manual workEvaluate answer quality, routing accuracy, and escalation behavior
    Read-only accessRetrieve approved fields such as order or subscription statusResolve information requests without changing a recordRestrict endpoints, records, and fields; verify customer authorization
    Write accessUpdate records or initiate actions such as cancellations or refundsComplete bounded workflows after earlier stages are dependableValidate inputs, limit action scope, record outcomes, and require approval where consequences warrant it

    Mock responses can test branching logic before an API is ready, as the Intercom article notes. It also proposes a temporary human-in-the-loop step when an integration is still several engineering sprints away. These methods can validate the workflow and expose missing requirements, but simulated success should not be treated as proof that production identity, authorization, failure recovery, and audit controls are ready.

    Put deterministic controls around probabilistic decisions

    Plain-language workflow instructions can guide an agent, but security-critical constraints should not depend solely on the model interpreting those instructions correctly. A safer architecture places enforceable controls between the agent and each backend system.

    ControlPractical design implication
    Dedicated identityGive the agent its own service identity rather than borrowing a staff account, so permissions and activity remain attributable.
    Least privilegeAllow only the endpoints, operations, records, and fields required by the selected workflow.
    Read and write separationKeep retrieval permissions distinct from mutation permissions and grant write access only when the use case requires it.
    Independent policy enforcementValidate identity, authorization, limits, and required inputs outside the model before executing an operation.
    Bounded actionsPrefer narrow, purpose-built operations over unrestricted database or administrative access.
    Human approval and escalationRoute ambiguous, exceptional, sensitive, or difficult-to-reverse cases to an authorized person.
    Auditability and monitoringRecord the request, decision, tool call, result, and escalation so failures and unusual patterns can be investigated.
    Safe failure behaviorPrevent retries, timeouts, or partial completion from producing duplicated or inconsistent changes.

    The integration request should document the workflow in plain language, identify every read and write point, name the system owner, and specify the minimum required fields. It should also define how success and harm will be measured: not only whether the agent completed the conversation, but whether it selected the correct record, performed the authorized action once, protected restricted data, and escalated when it lacked sufficient confidence or permission.

    This framing also improves the business case. Engineering is being asked to expose a narrowly scoped capability with explicit boundaries, rather than to provide broad access to a general-purpose agent. Leadership can then compare measurable workflow value with implementation effort and residual risk.

    Key takeaways

    • System access creates value when it lets an agent complete work, but it simultaneously expands the security boundary.
    • The best initial workflow is frequent, bounded, operationally meaningful, and owned by a team that can approve its data and actions.
    • Progress from no integration to read-only retrieval and then to narrowly scoped write operations; do not treat access as an all-or-nothing decision.
    • Enforce identity, authorization, field restrictions, action limits, and audit logging outside the model’s natural-language instructions.
    • Evaluate correctness, unauthorized-action risk, failure recovery, and handoff quality alongside resolution rate.

    The strongest long-term pattern is a portfolio of small, governed capabilities rather than one broadly privileged agent. Each successful workflow can supply the evidence needed to extend access deliberately, while keeping the consequences of error visible and contained.

    References

  • A Layered Playbook for Package Supply Chain Security

    A Layered Playbook for Package Supply Chain Security

    Package supply chain security is not simply a matter of choosing reputable libraries. The practical challenge is controlling an expanding dependency graph, the code that executes during installation, the resources that installed software can reach, and the automated tools allowed to make those decisions.

    A useful defensive model follows the path an attack must take: enter through a package or dependency, execute in the development environment, discover valuable information, and transmit it elsewhere. Organizing safeguards around that sequence produces a stronger posture than relying on any single scanner, sandbox, or package reputation signal.

    Package risk grows through the dependency graph

    Developers usually evaluate the packages they select directly. The less visible risk lies in transitive dependencies: packages installed because another dependency requires them. The source article illustrates the scale of this effect by reporting that installing Jest brought in 266 packages. That example is not evidence that those dependencies were malicious; it shows how one deliberate choice can create hundreds of additional trust relationships.

    This changes the unit of review. The relevant question is not only whether a named package appears legitimate, but whether its complete dependency graph is proportionate to the job. A small utility that introduces unfamiliar native modules, unrelated capabilities, or an unexpectedly broad tree deserves more scrutiny than its simple interface might suggest.

    Manifests such as package.json, pyproject.toml, and requirements.txt make dependency installation repeatable. Repeatability alone, however, does not guarantee safety. If version ranges or unresolved transitive dependencies allow later releases to enter automatically, two installations based on the same manifest can produce different risk profiles. Pinning direct and transitive versions converts an evolving external graph into a more deliberate, reviewable input.

    Match defenses to the stages of a package attack

    The source article says an analysis covering more than 230,000 malicious-code incidents found a recurring pattern: malicious code first needs an entry point, then searches the device for sensitive data, and finally uses a network connection to exfiltrate what it finds. This reported pattern suggests three distinct control points.

    Reduce risky entry and automatic execution

    A waiting period for newly published packages can reduce exposure to releases that have not yet attracted community scrutiny. The article recommends installing only packages that are at least seven days old. That is a risk filter, not a guarantee: an older malicious package can remain undetected, while a legitimate urgent fix may occasionally justify an exception.

    Installation scripts require separate treatment because they may execute before a developer has inspected the installed code. Disabling automatic install hooks by default creates a decision point. A package that depends on a post-install action can still be used, but the script, its purpose, and the capabilities it invokes should be reviewed first.

    Constrain access after installation

    Pre-install review cannot catch every problem. The next layer limits what package code can inspect or modify if it does execute. Sandboxed folders and isolated development environments can reduce the blast radius, but the source cautions that isolation by itself does not prevent malicious code from entering. Access boundaries therefore complement package controls rather than replace them.

    Limit unnecessary network egress

    Stolen information has less value to an attacker if malicious code cannot transmit it. Restricting unnecessary outbound connectivity addresses the final stage of the reported pattern. This layer matters because a package may evade provenance review and execute inside an environment despite earlier controls. Entry controls, resource boundaries, and egress restrictions together create independent opportunities to interrupt the attack.

    Provenance is a decision process, not a trust badge

    No single popularity or identity signal proves that a release is safe. The source proposes evaluating maintainer history, download patterns, repository activity, signed releases, and consistency across registries. Their value comes from comparison: a sudden change in maintainership, an unusual release pattern, or a mismatch between repository and registry information may warrant investigation even when each signal looks plausible in isolation.

    Context also matters. Dependency behavior should be compared with the package’s stated purpose. A capability that is normal for a database driver may be difficult to justify in a formatting utility. This purpose-to-capability test helps teams focus limited review time on anomalies rather than treating every dependency as equally suspicious.

    These checks work best when they lead to a clear disposition: approve the package and lock the reviewed version, replace it with a narrower dependency, inspect it more deeply, or decline it. Provenance information without a decision rule can become documentation that does not change behavior.

    AI coding agents must inherit the same installation policy

    AI-assisted development introduces a governance problem as much as a technical one. A coding agent may be able to select and install a package while pursuing a larger task, compressing several human decisions into one automated action. If it can also reach broad areas of the file system and use the network, a malicious dependency may encounter a larger potential blast radius.

    The source describes workflows in which Claude searches, creates, and edits files across a broad knowledge system, including notes derived from downloaded PDFs. That breadth provides productivity value, but it also makes one-folder isolation impractical for the reported workflow. The proposed response is disciplined configuration: hooks require the agent to follow the same package-age, install-script, provenance, and dependency rules expected of a human developer.

    This principle is more durable than a rule tied to one assistant. Package policy should apply consistently whether an installation is initiated by a developer, an AI agent, a local automation script, or a build process. The initiator may change; the acceptable evidence, permissions, and exceptions should not.

    Key takeaways

    • Review the full dependency graph, because the packages selected directly represent only part of the installed attack surface.
    • Use a waiting period for new releases as one filter, while preserving a documented path for justified exceptions.
    • Prevent install scripts from running automatically until their purpose and behavior have been examined.
    • Combine provenance checks with a purpose-to-capability test and an explicit approve, investigate, replace, or reject decision.
    • Pin direct and transitive versions, then run recurring audits to detect issues discovered after installation.
    • Apply the same package rules to coding agents, automation, local development, and build environments.
    • Layer installation controls, resource constraints, and network egress limits so that one missed signal does not determine the outcome.

    A mature package security posture will increasingly depend on making these controls routine and machine-enforceable. As development becomes more automated, the teams best positioned to move quickly will be those that turn package trust from an informal judgment into a consistent operating policy.

    References

  • Why We Made Fin the Most Open Agent: Instant HubSpot & Freshdesk Support With 76% Resolutions

    I’ve spent my career pairing product strategy with customer reality, and nothing is more clear right now than the demand for openness and speed. Today, we’re announcing that Fin can be used as a Service Agent on top of HubSpot and Freshworks, meaning you can use the world’s best Agent without migrating off your helpdesk.

    Hubspot and Freshdesk customers can now:

    Get Fin live, integrated, and working seamlessly in less than an hour.

    Delivering a 76% average resolution rate.

    Across all customer channels (voice, email, chat, social, and more).

    Resolving complex queries that require reading and writing to third party systems.

    With everything fully configurable to follow the unique policies of every individual business.

    This launch is a very visible step in a journey we’ve been on from day one: building an open, customer-first platform that plays well with the rest of your stack. We’ve long known that businesses want flexibility in how they configure their customer-facing tech stack. Since the very beginning, we have built Fin as an open platform, with APIs, MCPs, CLI, and opening up access to Apex, our proprietary trained model that delivers best in class performance.

    To make things easy for our customers, we have extensive public documentation of our product on our website, in our help center, and in our developer docs. We are the only Agent company in our space to do this, others hide most details behind sign-in screens, which we don’t believe is the right thing to do.

    Open Agent platforms will win because customers refuse to be boxed into closed ecosystems. We now believe our category has reached a stage where customers demand open platforms, that those who open up are more likely to win, and those who remain closed and protectionist will accelerate their demise.

    We are operating in a fast changing world, and customers do not want to be locked into a single vendor or closed ecosystem. They want the ability to experiment, to swap things in and out, and move everything with ease, technically and commercially.

    In an open world, the best product will win. In a world where businesses can easily swap vendors, the best product will win. We are happy to compete on that front, confident that Fin delivers the best customer experience and the highest performance.

    From a product management lens, this openness is powered by agentic AI patterns paired with robust CRM integration. Under the hood, we use Model Context Protocol (MCP), well-documented APIs, and orchestrated AI workflows to read from and write to third-party systems. That’s how Fin handles true multi-channel work—including voice AI agent scenarios—while giving teams the observability they need through Agent Analytics.

    If you are a Hubspot or Freshdesk customer, you can now have Fin integrated and live within an hour, without needing any help from us. We’re here if you want us, but as part of our commitment to building an open platform, we’ve designed everything to be self-servable—start in minutes or watch a quick demo of how everything works.

    Fin for Hubspot

    Fin for Freshdesk

    Inspired by this post on The Intercom Blog.

    Book a consult png image
  • From AI Builder to Agent Swarm: A Product Delivery Model

    From AI Builder to Agent Swarm: A Product Delivery Model

    AI-native product delivery has two distinct layers: a product professional who turns uncertainty into testable artifacts, and an agent workflow that divides complex work among specialized AI components. Treating either layer as the whole model misses the more useful opportunity.

    Together, the AI Builder role described by Product School and the parallel-agent architecture discussed by Pendo suggest an operating model for moving from customer evidence to evaluated software. The central lesson is not simply to add more AI. It is to assign clear responsibilities, preserve evidence across handoffs, and expand automation only where it improves a measurable constraint.

    Key takeaways

    • The AI Builder is the human integration layer, connecting discovery, prototyping, evaluation, and delivery inside the product trio.
    • Parallel agents are a system design choice, useful when specialized paths can improve latency, answer quality, or resilience.
    • Evaluations, analytics, observability, and controlled releases form the shared control system for both layers.
    • Fan-out should respond to uncertainty and business importance rather than becoming the default for every task.

    One delivery system, with human and machine responsibilities

    The Product School article presents the AI Builder as a hybrid product professional rather than a renamed product manager or an isolated prototyper. In its account, this person uses AI across analysis, prototyping, evaluation, and shipping, with the aim of shortening the distance between a customer problem and a runnable experiment.

    The Pendo article addresses a different layer. It describes workflows in which research, reasoning, tool use, and formatting can be assigned to specialized agents and then reconciled. Its focus is not ownership of the product problem, but the computational structure used to complete work.

    Read together, the articles separate two ideas that are often blurred. An AI-native team still needs a person or group to choose the problem, define acceptable behavior, interpret customer evidence, and decide whether an experiment justifies investment. Agents can perform bounded tasks within that process, but parallel execution does not establish product relevance on its own.

    LayerPrimary responsibilityTypical artifactsControl question
    AI Builder and product trioTranslate customer and business uncertainty into experimentsPrototypes, evaluation criteria, instrumented experiences, delivery recommendationsIs the team learning about an outcome that matters?
    Agent workflowExecute and reconcile specialized tasksRetrieved context, candidate responses, tool results, rankings, formatted outputsDoes orchestration improve the target measure enough to justify its complexity?
    Delivery platformProvide access, measurement, release controls, and safeguardsTool interfaces, traces, feature flags, budgets, analytics, fallbacksCan the workflow be observed, governed, and changed safely?

    This division of responsibility also clarifies the meaning of vibe coding in the Pendo account. Prompts, examples, and constraints are used to shape an intended experience before the team commits to extensive code or rigid rules. The AI Builder supplies the product judgment and experiment design around that activity; an agent architecture supplies one possible execution mechanism.

    Parallelism should target a constraint, not become a default

    Pendo reports three proposed benefits of parallel agents. Independent specialists can work concurrently to reduce latency, diverse candidate paths can be compared to improve quality, and risky or failure-prone operations can be isolated behind fallbacks. The article names fan-out/fan-in, race-and-rerank, specialist swarms, consensus, and self-consistency checks as patterns for producing and reconciling candidates.

    Those benefits depend on the shape of the task. Parallel research may help when several sources or interpretations must be examined independently. A race-and-rerank pattern may help when multiple plausible outputs can be scored against explicit criteria. Guarded fallbacks may improve resilience when a tool can fail without invalidating the entire experience. By contrast, multiplying agents around a simple, deterministic step adds coordination, cost, and more places to inspect when something goes wrong.

    The Product School article provides the missing selection mechanism: the workflow begins with a high-signal use case and explicit evaluation criteria. That makes orchestration a response to observed limitations in an experiment rather than an architectural commitment made in advance. A prototype can begin with the smallest credible workflow, reveal whether the bottleneck is grounding, reasoning, tool reliability, or response time, and introduce specialization at that point.

    Pendo proposes a similar progression at the system level: begin with retrieval, add a planner-executor split, and introduce parallel specialists where accuracy or latency problems appear. It also recommends placing budgets on fan-out, caching results, using smaller models when confidence is high, and widening the workflow when uncertainty rises. These are recommendations from the source, not independently reported benchmarks, but they establish a useful product principle: additional computation should be purchased in proportion to uncertainty and consequence.

    Evaluation is the bridge from discovery to dependable delivery

    The strongest overlap between the two articles is evaluation. Product School describes AI Builders converting interviews and behavioral analytics into instrumented experiments, benchmarking quality before production, and using A/B testing to feed results back into strategy. Pendo similarly calls for offline evaluations before rollout, production experiments afterward, and agent-level analytics to identify regressions across individual workflow steps.

    This creates a continuous evidence path rather than a handoff between discovery and engineering. A customer problem informs a prototype; the prototype produces evaluation cases; those cases become release gates; production behavior supplies new evidence for the next iteration. CI/CD can move changes through the delivery system, while evaluations determine whether an AI behavior is ready to move with them.

    A staged adoption path

    1. Select a bounded use case. Product School suggests beginning with a high-signal application such as generative-AI prototyping or an in-app guide, rather than attempting to transform the whole delivery process at once.
    2. Define the evidence before expanding the build. Specify evaluation criteria, analytics, and the customer or business outcome the experiment is intended to illuminate.
    3. Establish grounded context. Both articles emphasize retrieval-oriented workflows. Product School also discusses prompts, context windows, and data contracts as product surfaces that require deliberate design.
    4. Start with minimal orchestration. A single workflow or planner-executor arrangement provides a baseline against which a specialist or parallel design can be judged.
    5. Add parallel paths selectively. Introduce research, tool-calling, reasoning, or validation specialists only where evaluation results reveal a material limitation.
    6. Release behind controls. The sources point to feature flags, A/B testing, observability, anomaly detection, fallbacks, and post-launch review as ways to expose failures and limit their impact.

    The Model Context Protocol appears in both accounts as a way to standardize access to tools and data. Product School frames MCP integrations as part of an AI Builder’s toolbox, while Pendo argues that standardized access keeps agent roles separate from authentication, quotas, and observability. The combined implication is organizational as well as technical: shared interfaces can let product teams experiment with workflow roles without embedding every platform concern in every prompt.

    The operating model changes what a product team owns

    Product School places the AI Builder inside the product trio, working with design and engineering from the beginning. Pendo argues that product trios can own complete AI workflows rather than limiting their attention to prompts. These views converge on broader product accountability: the team owns the behavior, evidence, cost, risk, and release mechanism as one product surface.

    That ownership requires clearer boundaries, not fewer disciplines. Product judgment determines which outcome deserves attention. Design shapes the customer interaction and failure experience. Engineering and platform work make tool access, observability, quotas, and release controls dependable. The AI Builder connects these concerns through runnable artifacts, while specialized agents remain replaceable components within the evaluated workflow.

    The resulting measure of maturity is not the number of agents deployed or the speed of prototype generation. It is whether the team can trace a customer need through an experiment, an evaluation, a controlled release, and a learning decision. As tools become easier to compose, that chain of evidence will be the durable advantage in AI-native product delivery.

    References

  • Reusable AI Agent Workflows Need Evaluation Contracts

    Reusable AI Agent Workflows Need Evaluation Contracts

    Reusing an AI agent capability can accelerate delivery, but reuse also multiplies the consequences of an undetected defect. A retrieval component, tool-call routine, or safety check may appear in several workflows, so its quality cannot depend on the team that happens to integrate it next.

    The practical answer is to package each reusable skill with an evaluation contract: defined behavior, test fixtures, observability, guardrails, and outcome measures that travel with the component. Read together, the two source articles outline how modular workflow design and eval-driven development can reinforce each other from prototype through production.

    Reuse requires a contract, not just a prompt

    The AI skills library article describes modular capabilities for retrieval and grounding, summarization, classification, tool use, data enrichment, safety controls, and evaluation harnesses. Its central architectural idea is consistency: common interfaces and conventions allow teams to compose capabilities and replace implementations without rebuilding an entire flow.

    That modularity addresses code and workflow reuse, but it leaves an important product question: what must remain true when an implementation is replaced? The product-manager evaluation playbook supplies the missing half. It calls for versioned prompts, tools, and datasets; fixed offline scenarios; production experiments; and traces that expose how an agent reached an answer.

    The synthesis is an evaluation contract attached to every reusable skill. The contract defines acceptable inputs and outputs, relevant policies, expected telemetry, representative tests, and promotion thresholds. A skill is then reusable because its behavior can be checked repeatedly, not merely because its code can be imported.

    This distinction matters most in composed workflows. A summarizer that performs well on clean text may behave differently after a weak retrieval step. A tool-use component may generate a plausible response even when the underlying action fails. Reusable interfaces make these components interchangeable; evaluation contracts make the substitutions accountable.

    Measure four layers of agent quality

    No single score can represent the quality of a reusable agent workflow. The evaluation article separates concerns such as task success, factuality, safety, latency, cost, evidence quality, and product outcomes. The skills-library article adds operational concerns around guardrails, runtime metrics, and production monitoring. Combined, they suggest a four-layer model.

    Evaluation layerQuestion it answersReusable evidenceReported signals
    Component behaviorDoes the skill perform its assigned task?Fixed fixtures, golden examples, and domain scenariosTask success, factuality, and retrieval evidence quality
    Safety and policyDoes it remain within required boundaries?Adversarial cases, policy checks, and guardrail configurationsSafety performance, PII handling, and content-policy adherence
    Operational performanceCan it run reliably within product constraints?Traces, logs, version records, and production dashboardsLatency, cost, tool success, and fallback behavior
    Product impactDoes better agent behavior create user or business value?Experiment definitions and driver-tree mappingsTask completion, satisfaction, activation, retention, and NRR

    The layers should remain distinguishable even when a dashboard brings them together. If a workflow’s task-success score rises while latency or cost deteriorates, the trade-off should be visible. If offline factuality improves without changing completion or satisfaction in production, the result should not automatically be treated as a product win.

    Retrieval-first workflows illustrate the value of separation. The evaluation playbook recommends assessing the quality of retrieved evidence independently from generation. That boundary makes a failure attributable: the system can distinguish missing or irrelevant evidence from a generator that mishandled useful context. The same principle applies to classification, tool selection, tool execution, and response composition.

    A reusable workflow needs a controlled promotion path

    The two sources describe complementary stages rather than competing evaluation methods. The skills-library article starts with a quick-start chain, configurable skills, guardrails, evaluation datasets, and instrumentation. The evaluation playbook places fixed offline suites before user exposure, followed by controlled online validation. Together they form a promotion path from composable prototype to measured production capability.

    Offline evaluation establishes eligibility

    A candidate workflow should first face stable examples representing core scenarios, known failure modes, edge cases, adversarial prompts, and domain-specific questions, as reported by the evaluation playbook. Stable fixtures make comparisons reproducible when a prompt, model, tool, retrieval strategy, or policy changes. Running these checks through CI/CD, as proposed in the skills-library article, turns evaluation into a regular release control instead of a separate audit.

    Model-based judges can expand coverage for qualities such as helpfulness, coherence, and adherence, but the evaluation article cautions that they require calibration against a small, high-quality human-labeled set. It also recommends monitoring judge drift and retaining human review for edge cases or flows where mistakes carry greater consequences. A reusable judge configuration should therefore include its rubric, reference labels, version, and conditions for escalation.

    Online evaluation establishes value

    Passing offline checks shows that a variant is eligible for controlled exposure; it does not prove that users benefit from it. Both articles describe feature flags and A/B testing as mechanisms for comparing workflow variants in production. The evaluation playbook identifies conversation outcomes, tool success rates, human-support fallbacks, and user satisfaction as useful online signals.

    This staged approach also limits ambiguity. An offline regression can block a weak component before exposure, while an online experiment can test whether an eligible improvement changes real behavior. Promotion should depend on both: acceptable component performance and evidence that the complete workflow advances its intended outcome.

    Traces turn composition failures into fixable problems

    Composability increases the number of boundaries at which a workflow can fail. The evaluation playbook treats traces as the backbone of agent evaluation because they record inputs, intermediate actions, invoked tools, and final responses. The skills-library article similarly connects reusable chains to logs, traces, metrics, and production dashboards.

    A final-answer score alone may reveal that a workflow failed, but a trace can localize the failure. It can show whether retrieval supplied poor evidence, classification selected an unsuitable route, a tool call failed, a guardrail intervened, or generation ignored valid context. This makes evaluation useful for component ownership: teams can repair the relevant skill rather than adding a broad prompt patch to the entire chain.

    Trace analysis also supports reuse decisions. If one component repeatedly causes latency, cost, or safety regressions across several workflows, improving that shared component may create more value than optimizing each application independently. Conversely, a component that succeeds in one context but fails in another may need a narrower contract rather than a universal interface.

    Versioning is essential to that diagnosis. The evaluation playbook recommends versioning prompts, tools, and datasets, while the skills-library article emphasizes swappable implementations and comparable variants. Without linked versions for the component, evaluation set, judge, and workflow configuration, an apparent improvement may be difficult to reproduce or attribute.

    Governance and product outcomes belong in the same system

    Reusable workflows can spread good controls, but they can also propagate weak ones. The skills-library article reports guardrails for PII redaction, content-policy checks, and rate limiting, alongside configuration intended to support privacy-by-design. Packaging these controls as reusable capabilities can make the approved path easier to adopt, while evaluation fixtures test whether the controls continue to work as surrounding workflows change.

    Governance should not be reduced to a final pass-or-fail gate. Safety, privacy, and policy behavior need their own cases and traces throughout development. The amount of human review can then reflect the cost of error, consistent with the evaluation playbook’s recommendation to retain human oversight for higher-risk flows.

    The same evaluation system must connect technical quality to product value. The evaluation playbook proposes a driver tree that links per-turn measures such as helpfulness, safety, and latency to session outcomes such as task completion, and then to product measures including activation, retention, and Net Recurring Revenue. This hierarchy prevents a local metric from becoming the objective by default.

    For product teams, the resulting unit of roadmap work is not simply a new skill. It is a versioned capability with evidence about behavior, operational fitness, policy compliance, and contribution to an intended outcome. That shared definition gives product trios, engineers, and governance stakeholders a more precise basis for deciding whether to reuse, revise, or retire a component.

    Key takeaways

    • Package each reusable agent skill with an evaluation contract covering behavior, fixtures, telemetry, policies, and promotion criteria.
    • Keep component quality, safety, operational performance, and product impact distinct so improvements and trade-offs remain attributable.
    • Use fixed offline evaluations to establish release eligibility, then controlled online experiments to determine real-world value.
    • Trace intermediate steps and tool activity so failures can be assigned to the correct component instead of patched at the final response.
    • Version workflows, prompts, tools, datasets, and judges together so results remain comparable and reproducible.

    As skill libraries expand, their lasting advantage will come from accumulated evidence rather than component count. Teams that make evaluation portable alongside implementation can reuse workflows without surrendering visibility, governance, or product accountability.

    References

  • A Game-Changing Leap in Voice AI: Fin Voice 2, Apex Flash, and a Live Demo You Can Trust

    A Game-Changing Leap in Voice AI: Fin Voice 2, Apex Flash, and a Live Demo You Can Trust

    In competitive markets, I see two options: try to win the game competitors set, or choose to play a different game. In the "Customer Agents" category, I’ve watched too many glossy, fabricated demos—especially around voice—mask the real challenges. Voice is just extremely hard. We all know the future of customer experiences will be Agent-driven voice, yet most of us haven’t actually spoken with a modern AI Agent when calling a business because the tech hasn’t been truly ready in the wild. Today, the bar moves.

    What changed? There’s a live, public demo of cutting-edge voice tech you can stress test yourself—no smoke, no mirrors. I recommend taking it for a spin: https://fin.ai/voice. It’s fast, natural, and, yes, very, very good.

    For context, yesterday brought Apex Flash, their newest and fastest model, built for the unique demands of low latency channels like voice. Today comes Fin Voice 2, a major upgrade to Fin Voice with over 20 new features, and the first product built on Apex Flash.

    Here are the three things that stood out to me—and why they matter for customer support AI strategy and product strategy.

    First — thanks to Apex Flash, Fin Voice 2 is now the fastest, most natural Agent for phone, with higher resolution rates and customer satisfaction scores than ever before. Apex Flash is trained on millions of customer experience interactions, fine tuned for customer service, and can be configured to understand all your knowledge and follow all your policies. The result is higher resolution at significantly lower latency—the best of both worlds for voice AI agent performance.

    Speed and naturalness here aren’t accidental. Most voice AI products are slow because they convert speech to text, send it to a general model, get a text answer, and then convert it back to speech. Fin Voice 2 was designed to work differently, separating the real time layer that handles speech processing, and the layer that generates answers. That architecture is purpose-built for the demands of customer service on voice.

    Slide for Fin Voice 2, powered by Apex Flash, showing it beats Voice 1: +24.5% average resolution, +8.4% guidance following, +1.3% CSAT, -19.2% time to first audio, -37.6% semantic search latency.
    Powered by Apex Flash, Fin Voice 2 raises the bar on quality and speed—boosting resolution rates and guidance following while cutting time to first audio and semantic search latency, with a lift in CSAT too.

    Second — Fin Voice 2 can handle complex queries end to end: taking actions in external systems, verifying callers’ identities, processing refunds, booking appointments, and more. Phone is a high-stakes channel, and Fin adapts to customers across emotional states, clarifies when needed, and confirms key details before taking action. Most of the time, Fin can resolve the query in full, and when it can’t, it seamlessly hands off to the human team, maintaining full customer context and history. You also get multiple improvements to call quality, plus proactive outbound calls to follow up on unresolved issues—all orchestrated by robust AI workflows.

    Third — Fin Voice 2 gives you total control with industry-leading tools to configure and manage how Fin behaves. You get rich, detailed insights into call behavior and quality, the most common topics of calls, and one-click recommendations to improve. As with everything in Fin, you can fully self-serve and then manage it all with ease, without requiring professional services. Many vendors only let you set up their voice agent under supervision; with Fin, you get everything you need to iterate fast.

    If you haven’t tried the demo yet, go check it out: https://fin.ai/voice. If you prefer to wait, don’t be surprised when you end up speaking with it at a favorite brand soon.

    From a product management lens, this is what matters: latency is a feature customers feel; transparency builds trust in enterprise AI; and control is non-negotiable for CX leaders. The combination of a purpose-built, agentic AI architecture, measurable gains in resolution and CSAT, and true self-serve configuration signals that voice is moving from prototype theater to production reality. That’s the different game I want our industry to play.

    Inspired by this post on The Intercom Blog.

    Book a consult png image