What do you do when off-the-shelf moderation scores aren't good enough—and the alternative is paying human contractors to spend their days reviewing traumatizing content at scale? I’ve wrestled with that exact trade-off in enterprise environments, and it’s why I was eager to unpack how custom AI can raise the bar on trust and safety without compromising accuracy, latency, or the well-being of our teams.
In this episode of Just Now Possible, I sit down with Nikki Marinsek (Data Scientist), Brian McCaffrey (Software Engineer), and Dan Means (Machine Learning Engineer) from Musubi, an AI-native trust and safety toolkit for content platforms. Musubi builds custom-trained ML models and LLM-powered moderation tools that adapt to each platform's unique policies—from dating apps to social networks to AI inference endpoints. As a product leader, I’m drawn to their blend of eval-driven development, agentic AI, and pragmatic deployment pipelines that actually meet real-world SLAs.
We walk through their full journey—starting with a first prototype on tabular data—then discovering the system was sometimes catching issues human moderators missed. That insight became a forcing function to formalize evaluation, calibrate thresholds, and design feedback loops that help humans and models converge. Just as importantly, they built a policy optimizer that uses agentic flows so non-technical trust and safety teams can iterate on LLM moderation policies without needing a data scientist in the room.
If you’ve ever had to balance latency, accuracy, and cost at scale, you’ll appreciate how Musubi tests trade-offs across traditional ML, embedding-driven classification, and LLMs. Their approach mirrors the patterns I expect in high-throughput stacks: cache and pre-compute where possible, contain worst-case latencies, and push evaluation tooling to customers so policy changes are safe, observable, and fast to deploy.
What resonated most with me is their core product strategy: put eval tools directly in customers’ hands. When teams can benchmark AI against humans, referee disagreements using “LLM as judge,” and make policy gaps visible, trust increases and operational drift decreases. That’s the foundation for durable product strategy in sensitive domains like content moderation, fraud management, and risk scoring.
Listen to this episode on: Spotify | Apple Podcasts
Guests: Nikki Marinsek, Data Scientist, Musubi; Brian McCaffrey, Software Engineer, Musubi; Dan Means, Machine Learning Engineer, Musubi.
In this episode: Why off-the-shelf moderation scores fail and how custom-trained models fix that; How Musubi combines traditional ML with LLMs for different moderation tasks; The discovery that AI can outperform human moderators—and how to communicate that to clients; Using AI as a judge to referee disagreements between AI and human decisions; How Musubi onboards new customers with "reverse demos"; What custom model training actually means: fine-tuning, feature engineering, and reusable deployment pipelines; The policy optimizer: an agentic flow that helps customers iterate on their LLM moderation policies; Why pushing eval tools directly to customers is a core product strategy; How Musubi is building flexible orchestration workflows for non-technical trust and safety teams.
From a product management lens, a few highlights stand out. First, the disciplined separation of concerns: use traditional ML for high-precision, low-latency pattern detection and LLMs for nuanced policy interpretation. Second, invest in golden sets and policy loops early so you can quantify improvement and avoid subjective debates. Third, productize customization—create reusable deployment pipelines, parameterized policies, and self-serve evaluation—so each customer’s “custom model” still scales like a platform.
I also appreciated the onboarding tactic of "reverse demos." Rather than a canned walkthrough, the team invites customers to bring real policies and edge cases, then instruments the workflow live. That move builds credibility, accelerates discovery, and surfaces the fastest paths to value—an approach I recommend whenever you’re selling complex AI workflows to non-technical stakeholders.
If you’re navigating cost and latency trade-offs, the conversation goes deep on techniques like embedding-driven classification, fine-tuning vs. training, and when to route decisions through LLM adjudication. My takeaway: treat the router, the evaluator, and the policy as first-class products. When those elements are observable and testable, you can raise quality without exploding compute costs or creating operational bottlenecks.
Resources & Links: Musubi — AI-powered trust and safety toolkit for content platforms. Maven AI Evals Course — AI evals course.
Chapters: 00:00 Meet the Team; 01:18 Why Everyone Wears Product; 02:32 What Musubi Builds; 04:51 AI for Human Moderation; 09:59 Adversaries and Asymmetry; 11:48 Early Days and Low Latency; 13:35 First Prototype Slice; 15:33 Traditional ML Meets LLMs; 19:52 Benchmarking Against Humans; 23:09 LLM as Judge and Policy Gaps; 29:53 From Prototype to Platform; 31:15 Customer Onboarding Reverse Demos; 36:08 Custom Models Per Customer; 38:05 Fine Tuning vs Training; 39:14 Embedding Driven Classification; 40:04 Cost and Latency Tradeoffs; 43:21 Productizing Customization; 49:16 Scaling Prototypes to Production; 51:58 Golden Sets and Policy Loops; 56:17 Coaching Customers Safely; 01:02:06 Gamified Feedback Signals; 01:06:19 Agentic Toolkit Roadmap; 01:09:05 Workflow Orchestration Future; 01:12:06 Wrap Up and Thanks.
Ultimately, this is a playbook for modern trust and safety: align your models to your policies, make evals a habit not an event, and empower non-technical teams with agentic workflows and transparent metrics. That’s how we move beyond black-box scores to systems we can measure, manage, and trust.
Fraud teams are drowning in signals—events, alerts, and edge cases that look suspicious but rarely point to what truly matters now. In my role leading product, I focus on turning that noise into clear, ranked actions the team can trust. Behavioral analytics is how we bridge the gap from “something looks off” to “here’s why it matters and what to do next.”
See how behavioral analytics helps fraud management teams surface anomalies, prioritize risk factors, and act faster with greater confidence.
When I build fraud capabilities, I start by defining the outcomes that matter: find anomalies early, prioritize by impact, and respond in minutes—not days. That requires a rigorous approach to data governance, strong observability across the stack, and a mindset tuned to threat detection and response rather than passive reporting.
For me, behavioral analytics means unifying event streams across web, mobile, payments, and support into a single, trustworthy, unified analytics platform. We then apply anomaly detection on top of baselines for user, device, and entity behavior—capturing velocity spikes, geolocation drift, account takeover signals, and unusual journey paths. The win is not more alerts; it’s clearer context per alert.
Prioritization is where the value compounds. I combine deterministic signals (e.g., device fingerprint mismatches, impossible travel, repeated declines) with weighted risk scoring that adapts to emerging patterns. This helps fraud analysts triage by potential loss and customer impact, not just alert volume—so the highest-risk cases land at the top of the queue with the right context attached.
Actionability is the final mile. I map each risk tier to a playbook—step-up authentication, temporary holds, secondary review, or immediate block—so teams can act with confidence. Real-time alerts route to the right channel; feature flags allow fast containment; and AI risk management practices ensure continuous learning while preserving precision and recall. We close the loop by measuring investigation time, false positive rates, and recovery to keep improving.
A few lessons keep paying off: instrument early and consistently; keep your schema stable; document risk definitions; and test changes with A/B testing to quantify impact before scaling. Treat your fraud stack like a mission-critical cybersecurity system with tight SLAs, clear ownership, and auditable decisions—because it is.
If you’re evaluating your next move, start with a narrow but high-ROI use case (account takeover or payment fraud), stand up clear dashboards for analysts, and iterate on the risk scoring model weekly. With disciplined data practices and aligned playbooks, behavioral analytics turns scattered signals into decisive, defensible action.
Inspired by this post on Amplitude – Perspectives.
Your support prototype can explain what happens after a customer reports a stolen card. The harder product decision is whether you can trust it to carry that case from the first message to a verified outcome without losing state, skipping an approval, duplicating an action, or going silent while work remains open.
You will not solve that problem by adding a larger prompt or more conversational agents. You need an operating model for cases that span people, policies, systems, and days. The model below gives you a practical way to define the work, divide agent responsibilities, control execution, and measure whether the customer's problem was actually resolved.
Define the case before you define the agents
A stolen-card request exposes the central mistake in support automation. Freezing the card is visible, immediate, and easy to demonstrate. The less visible work may include dispute intake, fraud investigation, merchant communication, customer outreach, approvals, and follow-up. If your scope ends when the chat ends, you have automated the tip of the workflow while leaving its operational burden intact.
Start with a case contract. This is the shared definition of what entered the system, what outcome is owed, which actions are permitted, and what evidence will prove completion. Define it before deciding how many agents you need.
Customer outcome: State the result in operational terms. "Card secured and required follow-up completed" is more useful than "customer helped."
Entry conditions: Record the signals that create the case, including the customer request, the affected product, and any authentication or evidence requirements imposed by your policy.
Required work: Enumerate the actions, investigations, notices, approvals, and follow-ups that may sit below the initial request.
Allowed actions: Specify which tools may be called, which fields may be changed, and which financial or account actions require approval.
State and owner: Give every open case a current state and an accountable role. "The agents are working on it" is not a state.
Waiting conditions: Name the external event that can unblock the case, such as a customer reply, a system response, a timer, or a human decision.
Terminal conditions: Define resolved, declined, cancelled, transferred, and incomplete outcomes separately. Each one should require evidence and a reason code.
The strongest procedure starts as a workflow map owned by the people who understand disputes, fraud, operations, and compliance. Those subject-matter experts can maintain agent procedures in natural language, but natural language should not mean unmanaged prose. Give each procedure an owner, version, effective date, test cases, and approval history. A policy change should produce a traceable procedure change, not an invisible prompt edit.
Test your case contract with an awkward question: could the system truthfully tell the customer that the case is resolved while a mandatory downstream task is still pending? If the answer is yes, your terminal condition is wrong. Fix that before tuning response quality.
Split responsibilities at operational handoffs
A multi-agent design earns its complexity only when the separation makes ownership clearer. Creating several agents with overlapping prompts usually produces more routing ambiguity, not more capability. Divide the system where the nature of the work, permissions, or waiting behavior changes.
Understands the request, gathers required details, performs permitted immediate actions, and creates or updates the case
The case has enough validated information to begin operational work
It cannot imply resolution merely because the conversation was handled
Back office
Executes system work, coordinates investigation steps, records evidence, and manages pending operational tasks
More information, an approval, or customer communication is required
It cannot invent missing evidence or bypass a policy gate to keep the case moving
Outbound
Requests missing information, communicates status or decisions, and follows up until a defined terminal condition is reached
The required response arrives, a timer fires, or the outreach policy is exhausted
It cannot decide that silence means success unless the procedure explicitly defines that outcome
The handoff should be a structured state transition, not an open-ended conversation between agents. Pass a compact case record containing the case identifier, current state, completed actions, evidence references, pending requirement, next allowed actions, applicable procedure version, and relevant deadline or timer. That record prevents the next agent from reconstructing the truth from a transcript.
Keep skills modular as well. "Send a status request," "retrieve transaction details," and "submit an approved case update" are easier to authorize, test, and audit than one broad tool called "handle dispute." Each skill should declare its required inputs, permitted states, side effects, expected result, and failure behavior.
Do not use separate agents simply to mirror your organization chart. Use them when different stages need different permissions, context, completion rules, or escalation paths. If two proposed agents can perform the same actions in the same states under the same controls, they probably belong together.
Let a state machine control long-running work
The language model can interpret a message and propose the next step. It should not be the sole authority on what state the case is in or which actions are legal from that state. A state-machine orchestrator can manage turns, triggers, and skill selection across an asynchronous case while the model handles the language inside those boundaries.
For an illustrative stolen-card workflow, your states might include:
Report received.
Immediate protection pending.
Immediate protection confirmed.
Required information under review.
Investigation or dispute work in progress.
Waiting on the customer, a merchant, an internal system, or a human approver.
Decision ready.
Required communication pending.
Resolved, transferred, declined, cancelled, or closed incomplete with a recorded reason.
Adapt the states to your product, operating procedure, and regulatory obligations. The value is not in these labels. It is in making every transition explicit. For each transition, specify the triggering event, required preconditions, allowed skill, expected side effect, accountable role, failure path, timer behavior, and evidence written back to the case.
Then scope skills deterministically for each turn. An agent handling a customer reply while the case is waiting for information may be allowed to validate the reply, attach evidence, request a missing item, or resume the workflow. It should not be able to perform unrelated account actions simply because those tools exist elsewhere in the platform. This per-state allow-list reduces the number of unsafe choices the model can make.
Async triggers deserve the same design care as messages. A customer reply, API status change, timer expiry, failed tool call, and human approval are all events that can create a new turn. Store them durably and process them against the current case version. Otherwise a delayed event can act on stale state after the case has already moved forward.
Financial actions also need protection from retries. A timeout does not prove that a tool failed; the action may have succeeded while the response was lost. Use an idempotency key where the receiving system supports one, record the attempted operation before retrying, and reconcile uncertain outcomes. Blindly repeating a freeze, refund, fee adjustment, or dispute submission can create customer harm and financial exposure.
Outbound completion needs its own rule. The customer may never send a final message, so "the conversation ended" cannot define success. A defensible terminal condition can require that the necessary notice was sent, mandatory actions are complete, no unresolved task remains, and any follow-up timer has reached the outcome defined by policy. Silence may end an outreach attempt; it does not automatically prove the underlying case was resolved.
Finally, write an audit record for every transition. Capture the prior state, event, procedure version, allowed skills, selected action, tool result, guardrail result, human decision if present, and resulting state. A transcript tells you what was said. A transition log tells you why the system acted.
Make compliance and human review part of execution
Do not reduce compliance to a paragraph at the end of the system prompt. High-stakes rules need controls at the point where the system interprets information, chooses an action, changes a case, or communicates a decision.
Use three complementary layers:
Deterministic controls: Enforce permissions, required fields, state preconditions, transaction limits defined by your policy, and mandatory approvals in code or workflow configuration.
Classification guardrails: Detect whether an input, proposed action, or outgoing message belongs to a risk category that must be blocked, revised, or reviewed.
Human decisions: Route policy exceptions, consequential approvals, conflicting evidence, ambiguous cases, and unsupported operations to an accountable person.
For critical regulatory checks, treat guardrails as classification problems and prioritize recall when missing a risky case is more costly than sending an extra case to review. That choice has an operational consequence: more false positives can increase manual workload and delay customers. Product, operations, risk, and compliance owners should agree on that trade-off for each guardrail rather than applying one global threshold.
Every classifier needs a defined consequence. A positive result might block an action, remove a skill from the current turn, require human approval, or permit the workflow to continue with additional logging. A score without an execution rule is only dashboard data.
Customer-specific policies matter in a platform serving more than one fintech. The system may share an architecture while each customer requires its own procedures and guardrails. Resolve the applicable policy set from trusted configuration before the model acts, attach the policy version to the case, and prevent cross-customer retrieval or tool access. Do not ask a model to infer which client's rules should apply from conversational context.
Human escalation should be a first-class tool call, not a side-channel message. The request should contain the exact decision needed, current state, relevant evidence, attempted actions, available options, policy context, risk of delay, and response deadline. The human's answer should return as a recorded workflow event so the orchestrator can validate it and resume from the correct state.
This pattern is especially important when an API is missing. A person may complete the task in an internal system, but the agent must not assume it happened. Require a structured confirmation and evidence before advancing the case. If that evidence never arrives, keep the case visibly pending or escalate it according to the procedure.
Because these workflows can affect money, account access, customer rights, and regulatory obligations, your AI design cannot substitute for review by qualified legal, compliance, risk, and operations owners. Let those owners approve the policies, controls, escalation criteria, and customer communications before live execution. Begin with read-only or reversible capabilities where possible, and do not grant autonomous financial actions until the failure and recovery paths have been tested.
Measure verified resolution and improve from failures
A conversational system can produce polished replies while leaving cases unfinished. That is why containment or deflection cannot be your sole success metric. The primary question is whether the case reached the correct terminal state with the required evidence, policy checks, and customer communication.
Build a metric hierarchy that separates outcomes from diagnostics:
Case outcome: Track the share of eligible cases reaching a verified terminal state, along with cases reopened, transferred, or found incomplete during review.
Customer experience: Track customer satisfaction and whether the customer must contact support again because ownership or status was unclear.
Operational performance: Track time to resolution, first-contact resolution where that metric is genuinely applicable, deflection, escalation rate, waiting time by state, and human work by escalation reason.
Risk performance: Track critical guardrail misses, false-positive reviews, unauthorized action attempts, procedure deviations, and cases advanced without required evidence.
Agent-stage performance: Track routing accuracy, skill success, handoff completeness, tool failures, timer outcomes, and terminal-state correctness for each role.
Be careful with first-contact resolution in workflows that are supposed to run asynchronously. A fraud investigation may remain open after a perfectly handled first interaction. Optimizing the agent to close the contact can therefore conflict with the real outcome. Use time to verified resolution and unresolved-work visibility alongside conversation metrics.
Evaluation should inspect both language and execution. A useful case-level rubric asks whether the system understood the request, selected an allowed skill, used the correct procedure version, obtained required evidence, respected guardrails, preserved context at handoffs, communicated accurately, and entered the right terminal state.
An automated evaluation pipeline can flag cases for human review and turn reviewed failures into labeled data. Do not sample only obviously failed conversations. Include high-risk classifications, recently changed procedures, new skills, long-running cases, human escalations, unusual state transitions, tool errors, and a baseline sample of apparently successful cases. Otherwise your evaluation set will miss failures that look normal in aggregate metrics.
Give every reviewed failure a place in a product backlog. The fix may belong to the procedure, state machine, skill contract, integration, guardrail, escalation path, or model behavior. "The agent made a mistake" is too broad to assign. A stable failure taxonomy tells you which layer should change and which regression tests must be added before release.
A sensible implementation sequence is:
Choose one bounded journey with a meaningful operational tail and a clearly accountable owner.
Map the full case, including hidden back-office steps, waiting states, approvals, exceptions, communications, and terminal conditions.
Define the case schema, events, state transitions, evidence requirements, and audit record.
Assign inbound, back-office, and outbound responsibilities only where permissions or completion rules differ.
Expose narrow modular skills and apply a deterministic allow-list in every state.
Add compliance classifiers, hard controls, and human decision gates before enabling consequential actions.
Run historical, synthetic, or controlled cases through the workflow and evaluate the complete case, not just the generated messages.
Release gradually, monitor state-level failures, and feed reviewed cases back into procedures, controls, and regression evaluations.
Key takeaways
Scope the customer's complete case before choosing the number of agents.
Separate agents at real permission, workflow, or completion boundaries.
Let the model interpret language, but let explicit state and policy control execution.
Treat human review as a structured workflow event with an owner and deadline.
Define "done" with evidence; a finished chat is not a finished case.
Optimize for verified resolution, policy adherence, and safe recovery rather than response quality alone.
At your next design review, put one real support case on the page and ask four questions: where can it wait, what event unblocks it, who approves a risky action, and what evidence proves completion? If your team cannot answer all four from the workflow, the system is not ready to act. Once those answers are explicit, agent boundaries become an engineering decision instead of a bet on autonomous behavior.
