AI Risk Governance: An Operating Model for Cyber Defense

You may already have an AI roadmap, an approved model vendor, and several agent pilots. The harder decision comes next: when should an AI workflow be allowed to read customer data, call a connector, change a record, communicate externally, or touch production?

A model that is acceptable for drafting internal content can become dangerous when its output triggers a business action. Your governance system therefore needs to answer a practical set of questions: What can happen, under whose identity, using which data, with what evidence, and how will you detect and stop the workflow when it behaves incorrectly?

Govern the action, not only the model

Model approval is necessary, but it is not the right boundary for operational risk. The unit you need to govern is the complete AI workflow:

• A person, application, or event supplies an input.
• The workflow retrieves business data or external content.
• A model interprets that context and produces an output.
• A connector or tool may turn the output into an action.
• The result is shown to a user, written to a system, or used in another decision.
• Prompts, feedback, outputs, and operational events may become part of a new data loop.

Risk can enter at every step. A legitimate document can contain a malicious instruction. A correctly functioning model can receive more data than the user is authorized to see. A connector can possess broader permissions than the task requires. A plausible but false output can be harmless in a draft and costly when it changes a customer account.

Your baseline threat model should also assume that attackers can use AI to personalize social engineering, imitate trusted voices, vary malicious code, and automate reconnaissance. A generic warning about suspicious emails is not enough when an employee may receive a credible message written for their role, account, and current project.

Create an inventory entry for every workflow, not merely every model. Each entry should record:

• Business purpose and owner: the outcome the workflow supports and the person accountable for it.
• Inputs and data classes: what the workflow can receive, retrieve, infer, and retain, including personal or confidential information.
• Model and provider: the model used, where inference occurs, and which vendor terms affect storage, training use, or residency.
• Tools and connectors: every system the workflow can read from or write to.
• Execution identity: the service account, user delegation, permissions, secrets, and authorization scopes involved.
• Action class: whether the workflow observes, drafts, recommends, or executes.
• Reversibility: how an incorrect action would be undone and which actions cannot be fully reversed.
• Evaluation evidence: the legitimate and adversarial cases the workflow must pass before release.
• Operational controls: logging, retention, approval, escalation, shutdown, and rollback mechanisms.
• Consumption controls: usage caps, environment tags, latency limits, and cost per transaction.

This inventory should function as a production registry, not a spreadsheet that is reviewed once and forgotten. Release checks should reject unregistered models, connectors, or identities. Runtime policy should deny capabilities that are not declared for the workflow. That is how you keep shadow AI and permission drift from quietly expanding the attack surface.

Map the crown-jewel path before choosing controls

Start with the business impact you cannot accept. Crown jewels are not limited to databases. They include data, identities, workflows, and systems whose compromise could materially harm customers, revenue, operations, or trust.

1. Name the impact. Write a concrete failure statement such as exposing customer information, changing a production configuration, issuing an unauthorized credit, or sending a message under an executive’s identity.
2. Trace the data path. Mark where information is collected, retrieved, transformed, sent for inference, displayed, logged, and reused as feedback.
3. Mark every trust boundary. Include vendor APIs, plugins, browser sessions, retrieval indexes, queues, internal services, and external connectors.
4. Assign an identity to each step. Avoid a shared, all-purpose agent credential. Give each component only the access required for its declared task.
5. Locate the consequential action. Identify the exact point where generated content becomes a system change, customer communication, financial event, or security decision.
6. Define the evidence trail. Decide what must be recorded so an investigator can reconstruct the input, authorization decision, tool call, approval, outcome, and rollback.

Identity is the central enforcement point. Zero-trust principles apply to AI workflows just as they do to employees and services: verify each request, use least privilege, isolate secrets, and do not treat a successful login as permanent authorization. A user who may read a record should not automatically be able to authorize an agent to modify it.

The vendor boundary needs equal attention. Record the applicable data-processing terms, control reports such as SOC 2 or ISO documentation, regional data-residency commitments, retention behavior, and whether submitted data may be used for training. A vendor review does not replace workflow controls; it tells you which risks remain yours to manage.

Turn the map into threat scenarios that can be tested. At minimum, examine whether:

• Malicious content retrieved from a document, ticket, web page, or message can redirect the model or trigger a tool.
• Personal or confidential data can be copied into an unapproved prompt, output, log, or external destination.
• A compromised dependency, model, plugin, or connector can alter the workflow’s behavior.
• A fabricated or biased output can cross the action boundary without adequate review.
• A convincing voice, message, or support interaction can persuade a person to bypass an approval control.
• Overbroad permissions allow the workflow to act on records or systems outside its intended scope.
• Missing telemetry prevents the security team from distinguishing normal automation from abuse.

Each scenario needs an expected control outcome. The test is not complete because the team tried an attack prompt; it is complete when the team can show that the request was denied, the event was visible, the alert reached the right owner, and no prohibited action occurred.

Do not red-team a production workflow if the test could expose real data, contact a customer, modify a record, or invoke a paid or destructive operation. Use a sandbox with synthetic or approved test data, isolated credentials, and disabled external side effects. Move the scenario toward production only after the containment controls have been demonstrated.

Match autonomy to blast radius and reversibility

Autonomy should be earned at the workflow level. The same model may support several autonomy levels because the consequence depends on the data, identity, tool, and action around it. The following control contract is a practical starting point rather than a universal compliance classification.

Workflow mode	Failure to design for	Minimum control gate	Release evidence
Read or generate	Sensitive input leakage, unsupported output, or inappropriate retention	Approved data classes, data minimization, access control, retention rules, grounded prompts, citations where available, and content filtering	Evaluation on a maintained reference set, data-flow review, and inspectable logs
Recommend to a person	An inaccurate or biased recommendation influences a consequential decision	All read-and-generate controls, plus a named reviewer, visible supporting evidence, and no automatic execution	Error analysis by failure type, adversarial cases, and a record of reviewer acceptance, rejection, or correction
Execute a reversible action	Prompt injection, excessive permissions, or invalid output causes an unauthorized change	Scoped identity, tool allowlists, isolated secrets, egress restrictions, sandboxing, output validation, explicit confirmation, and a tested rollback path	Red-team results, authorization tests, complete audit events, and a successful rollback rehearsal
Execute a high-impact or difficult-to-reverse action	Customer, revenue, production, privacy, or trust is materially harmed before containment	Explicit approval at the final action boundary, staged execution where possible, granular scopes, usage limits, fail-closed behavior, a shutdown control, and a named incident owner	Adversarial evaluation, recovery evidence, approver training, and sign-off from the accountable risk owner

Human-in-the-loop is not a sufficient control description. The reviewer needs enough information to make a real decision. At the approval boundary, show:

• The exact proposed action and its target.
• The data used to produce the recommendation and any destination that will receive data.
• The tool, identity, and permissions that will be invoked.
• The reason for the action and the supporting evidence or citations available.
• Whether the action is reversible and what the rollback will do.
• Any validation warning, policy exception, or unusual behavior detected upstream.

Bind approval to one proposed action and let it expire when the underlying data, target, or parameters change. A person approving a preview should not unknowingly authorize a later, materially different tool call. For agentic systems, high-risk actions need explicit approvals, granular scopes, secrets isolation, egress controls, sandboxing, and validated outputs.

Before increasing autonomy, define acceptance limits for the risks that matter in that workflow. These can include task quality, unsupported claims, biased outcomes, forbidden tool requests, abnormal data egress, false-positive alerts, latency, cost per transaction, and rollback success. Set the limits before the pilot produces attractive results. Otherwise, the release decision will move to accommodate whatever the demo happens to show.

Use a maintained reference set for intended behavior and a separate adversarial set for abuse cases. Any test that produces an unauthorized action, forbidden data transfer, or privilege violation should block the release until the underlying control is corrected and retested. A strong average quality score cannot compensate for a security boundary that sometimes fails open.

Operate AI defense as a product and incident loop

Governance becomes useful when it changes runtime behavior. Policies need to control identities, data access, tool use, destinations, approvals, and resource consumption. Detection then needs enough context to distinguish expected automation from misuse.

Build one defensive loop

1. Prevent. Enforce data classification, least privilege, connector allowlists, egress restrictions, output validation, and action gates.
2. Observe. Correlate AI events with identity, endpoint, application, and network telemetry.
3. Decide. Route suspicious behavior to a person who can see the workflow context and business consequence.
4. Contain. Revoke credentials, disable a connector, stop egress, suspend the workflow, or roll back a reversible action.
5. Learn. Add the failure to the evaluation set, update the threat model, change the control, and prove the correction before restoring autonomy.

Behavioral detection matters because an individually valid event may become suspicious only in context. Correlating identity signals with endpoint and network activity can expose subtle anomalies that static signatures miss. For an AI workflow, add model and tool events to that context.

A useful audit event should identify the initiating actor, execution identity, workflow and model, prompt or template version, retrieved resource identifiers, tool requested, authorization result, validation result, human approval if required, resulting action, and rollback status. Record enough to reconstruct the incident without automatically storing every raw prompt. Indiscriminate prompt logging can create another repository of personal data, secrets, and confidential content, so apply minimization, access controls, redaction, and retention rules to the logs themselves.

Your dashboard should combine security, model, product, and economic outcomes. Track:

• Coverage: high-impact workflows with a named owner, current threat model, evaluation suite, and tested shutdown path.
• Model quality: results by task and failure category, rather than one blended score that hides a dangerous edge case.
• Control performance: denied tool calls, policy exceptions, privilege violations, suspicious egress, and approval overrides.
• Response: signal-to-noise ratio, mean time to detect, mean time to contain, and recovery status.
• Engineering quality: escaped defects, vulnerable dependencies, and security findings detected before release.
• User outcome: task completion, reviewer burden, corrections, and abandonment at the approval step.
• Economics: latency, usage by application and environment, and cost per transaction.

AI can help inside the defensive loop without owning it. Security assistants can summarize incidents, connect related evidence, explain a probable cause, and propose next steps. That can reduce analyst toil and accelerate decisions. It should not silently convert a probabilistic recommendation into a destructive containment action. Apply the same autonomy and approval framework to defensive agents that you apply to customer-facing ones.

Give product and security one backlog

AI risk cannot be handed to security after the workflow is built. Product defines the intended outcome and unacceptable user harm. Engineering implements boundaries, telemetry, and rollback. Security owns threat modeling, control assurance, adversarial testing, and incident readiness. IT and identity owners govern accounts and connectors. Data and privacy owners determine permitted use, retention, and vendor conditions. The business owner accepts the residual operational risk.

Put missed detections, unsafe tool requests, reviewer overrides, false positives, escaped defects, user-reported incidents, and excessive consumption into the same operating backlog as product defects. Each item needs an owner, a release criterion, and a test that demonstrates the correction. This keeps governance attached to the product lifecycle instead of turning it into a parallel paperwork process.

Express repeatable rules as policy that can be versioned, reviewed, tested, and enforced in delivery and runtime systems. A shared policy-as-code foundation across product, security, and IT reduces control drift and makes audit evidence more predictable. Examples include permitted models by data class, allowed connector scopes, required approvals by action class, egress destinations, environment-specific usage caps, and mandatory audit fields.

Use 90 days to prove one controlled path to production

A broad governance program can spend months debating universal policy while risky workflows continue to appear. A better starting point is a 90-day path that inventories usage, pilots within guardrails, and productionizes only the workflow that earns it.

Days 0-30: Establish the boundary

• Inventory active and proposed AI workflows, including employee-created tools and unapproved connectors.
• Classify the data, systems, identities, and actions involved.
• Select one or two consequential business workflows rather than spreading controls across every experiment.
• Name the product owner, security owner, data owner, business approver, and incident owner.
• Draw the complete action path and identify crown jewels, trust boundaries, and irreversible outcomes.
• Put basic access, audit, retention, tool, egress, approval, shutdown, and rollback controls in place.
• Define the intended-behavior set, adversarial scenarios, acceptance limits, and prohibited outcomes before the pilot begins.

The exit condition is not an approved policy document. It is a workflow whose owner, data, identity, tools, action boundary, failure modes, and emergency controls can all be named.

Days 31-60: Prove the controls in a pilot

• Run the workflow in a sandbox with the lowest autonomy level that still tests the business value.
• Build the evaluation harness around a maintained reference set and a separate adversarial set.
• Test prompt injection, data leakage, invalid outputs, connector abuse, privilege boundaries, and dependency failure.
• Instrument identity, retrieval, model, authorization, tool, approval, cost, and action events.
• Train the human approver on the decision interface and record corrections, overrides, and unclear evidence.
• Rehearse containment by suspending the workflow, revoking its credentials, preserving evidence, and rolling back a test action.
• Review quality, security, user outcome, latency, and cost together. A workflow does not pass because only one dimension looks good.

Use AI to augment a person before allowing it to execute independently. The pilot should prove both the useful task and the control loop. If the team cannot detect a forbidden request or reconstruct an action, higher autonomy is premature even when the model’s normal-case output looks strong.

Days 61-90: Productionize with a narrow permission envelope

• Release only the workflow that met its predefined product, security, operational, and economic criteria.
• Start with the permissions and autonomy already proven in the pilot; do not widen them merely because the environment changed to production.
• Enable dashboards, alerts, usage caps, environment tagging, escalation routes, and the tested shutdown control.
• Train frontline users to recognize unreliable output, suspicious requests, impersonation attempts, and the correct escalation path.
• Retire duplicate or low-yield experiments that add vendors, connectors, identities, or spend without producing enough value.
• Treat every request for broader data, another tool, or greater autonomy as a new control decision with updated tests.

At the end of the 90 days, do not ask only whether the workflow shipped. Ask whether you can identify who initiated every consequential action, prove which data and permissions were used, see when a control blocked abuse, stop the workflow quickly, recover from an error, and quantify quality, response, latency, and cost. Any missing answer identifies the next control to build.

Key takeaways

• Govern the complete AI workflow, including retrieval, identities, connectors, actions, logs, and feedback loops.
• Begin with crown jewels and concrete business consequences, then map every trust boundary that can affect them.
• Match autonomy to blast radius and reversibility. A model approval does not authorize every use of that model.
• Place meaningful human approval at the consequential action boundary and show the reviewer exactly what will happen.
• Combine AI telemetry with identity, endpoint, application, and network signals so misuse can be detected and contained.
• Increase autonomy only after legitimate and adversarial evaluations, auditability, shutdown, and rollback have been demonstrated.

Your next governance meeting should end with one selected workflow, one accountable owner, one drawn action path, and one explicit list of prohibited outcomes. If the team cannot show how that workflow will be stopped and investigated, keep it in recommendation mode and build the missing control before expanding its authority.

References

• Pendo Perspectives – 3 Powerful Ways AI Is Reshaping Cybersecurity-from Ruthless Attacks to Rapid Defense
• Pendo Perspectives – 4 Hidden AI Risks Every CIO Must Tackle Now-and a Proven Playbook to Mitigate Them