Your AI feature is ready to move beyond the prototype, but one question can still stop the release: exactly which customer data leaves your boundary, where is it copied, and who can retrieve it later? If the answer is scattered across architecture diagrams, vendor settings, and assumptions, you do not yet have a security decision.
You can resolve that uncertainty without turning every experiment into a committee exercise. Map the data path, assign the capability a risk lane, minimize what the model receives, and automate the controls that follow from the classification. The result is a release process that is both faster and easier to defend.
Start with the data path, not the model
The first security question is not what the model knows. It is what your product sends, retrieves, transforms, stores, logs, and displays. A provider can have a strong security posture while your implementation still exposes data through an overbroad retrieval query, a debug log, or an incorrectly scoped support tool.
Draw the complete path for one user request. Do not use a generic platform diagram. Follow the actual capability from the moment a user or system creates an input until every resulting copy has expired or been deleted.
- Identify the original input, including form fields, uploaded files, messages, system-generated events, and API payloads.
- List the context added by your application, such as account attributes, conversation history, analytics, retrieved documents, feature configuration, or tool results.
- Mark every transformation before the model call: filtering, redaction, tokenization, summarization, chunking, or schema conversion.
- Name the service that receives each payload, including gateways, model providers, observability tools, evaluation systems, queues, and caches.
- Trace the response through validation, tool execution, display, analytics, support access, and downstream storage.
- Record when each copy expires, how deletion propagates, and who can access it while it exists.
For every step, capture six fields: data class, system owner, access scope, external recipient, retention rule, and failure consequence. If any field is unknown, label it unknown. An explicit unknown is useful discovery work; an undocumented assumption is hidden risk.
Do not stop at obvious records such as customer PII and payment identifiers. Prompts, retrieved context, user-linked analytics, internal roadmaps, feature flags, configuration values, embeddings, vector stores, and evaluation datasets can also reveal confidential facts or inferred identity. Treat them as product data with owners and controls, not harmless implementation residue.
Use a completion test that exposes weak assumptions
Your map is ready for a decision when someone outside the feature team can answer these questions from it:
- What is the most sensitive field the capability can receive?
- Which fields cross the company boundary, and which named service receives them?
- Can one customer ever retrieve another customer’s data?
- Are raw prompts, completions, retrieved passages, or tool results logged?
- Which identities can inspect those logs or replay a request?
- What happens to derived data when the original record is deleted or its permissions change?
- Which control contains the incident if the model, retrieval layer, or tool call behaves unexpectedly?
If the team can only answer these questions by asking several vendors or searching production settings, keep the release open. The missing work is not paperwork. It is part of the product’s operating design.
Turn the risk assessment into a release lane
A risk score is useful only when it changes what the team must do. Avoid a long questionnaire that ends with an ambiguous rating. Use a small number of lanes, give each lane an observable entry condition, and attach default release controls.
| Risk lane | Typical signals | Default release posture |
|---|---|---|
| Low | Internal capability; synthetic or public inputs; no sensitive context; no consequential external action | Approved provider, least-privilege credentials, basic access tests, and confirmation that secrets are not entering prompts or logs |
| Elevated | Customer-facing capability; authenticated user context; behavioral telemetry; stored prompts or outputs; retrieval from private content | Data minimization, pre-call redaction, permission-aware retrieval, explicit retention, adversarial evaluations, runtime monitoring, and a named incident owner |
| High | Regulated-data adjacent; payment identifiers; broad confidential retrieval; sensitive identity data; or authority to perform a consequential action | Early Security, Legal, privacy, and Data involvement; documented threat model; human approval where an action warrants it; verified containment; and release evidence reviewed before exposure |
These lanes are an operating model, not a compliance determination. Applicable controls depend on the actual data, customer contracts, geography, industry, and use case. Security and legal specialists should make those determinations when the capability creates legal, regulatory, or material customer exposure.
Classify the capability, not the entire product. A writing assistant that uses text supplied for a single request may sit in a different lane from an account assistant that searches every customer conversation and updates CRM records, even when both use the same model.
Score the capability across these dimensions:
- Data sensitivity: public, internal, confidential, personal, payment-related, or regulated-data adjacent.
- Audience: constrained employee group, all employees, authenticated customers, or public users.
- Retrieval reach: one supplied record, an authorized account subset, or a broad internal corpus.
- Action authority: produces a suggestion, drafts a change, or executes an external action.
- Persistence: ephemeral processing, structured event storage, or retained raw inputs and outputs.
- Third-party exposure: stays inside your controlled environment or passes through one or more providers and subprocessors.
Use the highest-risk dimension to set the initial lane. Lower it only after a design change removes the exposure. A promise to be careful is not a mitigating control; scoped retrieval, enforced redaction, disabled raw logging, and restricted tool permissions are.
Reclassify when the feature changes its data, audience, retrieval reach, retention, provider, or ability to act. A seemingly small roadmap addition, such as remembering past conversations or connecting a second data source, can change the security posture more than a model upgrade does.
Design the system to disclose less data
The most reliable way to protect data is to keep unnecessary data out of the AI path. Encryption and contractual terms matter, but they do not make an irrelevant customer field necessary. Start with the user outcome and ask which minimum facts the model needs to produce it.
Minimize before you redact
Redaction is a valuable deterministic safeguard, but it should not carry the whole design. Free-form text can contain names, secrets, identifiers, and confidential business information in formats your rules do not recognize. Reduce the payload first, then redact the smaller payload that remains.
- Replace a full customer object with the few fields required for the task.
- Use a temporary account token when the model does not need a person’s name, email address, or payment identifier.
- Convert long interaction histories into purpose-specific structured fields when the task does not require the original prose.
- Exclude internal notes, disabled fields, hidden metadata, and unrelated attachments by default.
- Log structured events such as policy result, model identifier, latency, and request status when raw prompt text is not required.
Separate identity from content wherever the workflow allows it. The application can retain the relationship between a temporary token and an account while the model processes only the content needed for the task. Access to the token map should remain narrower than access to routine AI telemetry.
Make retrieval permission-aware
A retrieval-first architecture can keep the raw corpus inside your controlled boundary while selecting only relevant context for a request. It is not automatically private. If an external model receives the selected passages, those passages still cross the boundary and still require minimization, redaction, approved-provider controls, and a clear retention policy.
Apply authorization when the request is made, not only when content is indexed. The retrieval layer should constrain results by tenant, user, role, and current document permissions before any text becomes model context. Do not index content that the eventual searcher could never be allowed to read unless the architecture has another enforceable isolation boundary.
Treat embeddings and vector-store metadata as sensitive derived data. A vector is not a magic anonymizer, and metadata can disclose document names, account relationships, categories, or activity patterns even when full text is elsewhere. Your deletion and permission-change process must reach the index, cached results, evaluation copies, and any stored citations, not just the primary database.
Retrieved content is also untrusted input. A malicious or compromised document can contain instructions intended to change model behavior. Keep system instructions separate, restrict available tools, validate tool arguments, and enforce authorization in application code. The model should never be the component that decides whether a user may access a record or perform an action.
Place deterministic controls on both sides of the call
- Before the call: validate the request schema, remove disallowed fields, redact known sensitive patterns, apply allow and deny policies, and constrain retrieval.
- After the call: validate output structure, block disallowed sensitive patterns, verify any cited record belongs to the authorized scope, and check tool arguments before execution.
- During operation: monitor unusual prompt, output, retrieval, and access patterns without creating a second uncontrolled store of raw content.
An output filter cannot undo data already disclosed to an external provider. Use post-call checks to protect users and downstream systems, but use pre-call minimization and access enforcement to prevent the disclosure itself.
Make vendor approval specific to the intended use
Do not approve an AI vendor in the abstract. Approve a defined service, account configuration, data class, region, retention posture, and use case. A provider suitable for public-content summarization may not be suitable for customer conversations or payment-related identifiers.
Ask questions that produce enforceable answers rather than broad assurances:
- Training and service improvement: Can prompts, files, retrieved passages, outputs, feedback, or metadata be used to train models or improve services? Is the restriction a default, a setting, or a contractual term?
- Retention: How long does each data type remain in primary systems, safety systems, failure logs, backups, and support tooling? What initiates deletion, and what exceptions apply?
- Human access: Under what conditions can provider personnel inspect customer content, and how is that access authorized, logged, and reviewed?
- Security controls: Is data encrypted in transit and at rest? What key-management options, private networking, scoped credentials, access logs, and administrative controls are available?
- Location and subprocessors: Which regions process and store the data? Where can support access occur? Which subprocessors participate in the path?
- Assurance evidence: Which services and controls are covered by SOC 2, ISO 27001, or HIPAA-related commitments where relevant to the use case?
- Response: How will the provider communicate a security incident, policy change, model change, or subprocessor change that affects your approved use?
An audit or certification is useful evidence about a defined scope. It is not proof that your architecture, settings, or use case is safe. Confirm that the service named in the evidence is the service your product will actually call, and that your configuration does not bypass the controls you evaluated.
Keep a short decision record with the approved purpose, permitted and prohibited data, named endpoints or services, required account settings, retention terms, region, responsible owner, and review triggers. Reopen the decision when the purpose, data class, provider terms, model path, subprocessor chain, or architecture changes.
A shared catalog of approved providers and patterns also reduces shadow AI. Make the approved route easier to use by supplying scoped credentials, reference architectures, redaction utilities, retrieval patterns, and clear examples of prohibited inputs. Governance works better when the safe path is a usable product for internal teams.
Put the controls into delivery and incident response
A policy that depends on every engineer remembering every rule will drift. Store the capability’s classification, required controls, approved provider configuration, and decision owner alongside the delivery artifacts. Version changes so the team can see when a new data source or retention behavior altered the release posture.
Translate the release lane into automated checks wherever the control can be tested:
- Scan prompts, templates, configuration, and code for exposed secrets and unapproved endpoints.
- Unit-test redaction and tokenization against representative allowed and disallowed inputs.
- Integration-test tenant boundaries, role permissions, retrieval filters, and deletion propagation.
- Run evaluations that attempt to elicit restricted data, override instructions, retrieve unauthorized records, or trigger tools outside the allowed scope.
- Validate the selected provider, model path, region, logging setting, and retention configuration against the approval record.
- Block release when required evidence, monitoring, rollback controls, or an incident owner is missing.
Evaluation data needs the same scrutiny as production data. Remove unnecessary identities, restrict access, define retention, and avoid copying raw customer interactions merely because an evaluation system is internal. A test corpus can become a long-lived data store if nobody owns its lifecycle.
Monitor security-relevant events rather than indiscriminately recording content. Useful signals include blocked sensitive-data patterns, denied cross-scope retrieval, calls to unapproved services, unusual access behavior, unexpected changes in model or endpoint usage, and failed retention or deletion jobs. Structured metadata often provides the operational signal you need without preserving every prompt and completion.
Prepare containment before the first customer request
Your incident runbook should name the people and mechanisms needed to contain the feature. Depending on the incident, that can include disabling the affected path with a feature flag, revoking or rotating credentials, restricting retrieval, stopping unsafe logging, locating downstream copies, and contacting the provider.
Do not improvise evidence deletion or customer notification during an incident. Security, privacy, and legal owners should determine preservation, notification, and regulatory obligations based on the specific exposure. The product runbook should make those owners reachable and give them an accurate data-flow record, timestamps, affected systems, and containment status.
After containment, update the control that failed: the architecture, automated check, provider setting, policy, runbook, or team guidance. A review that ends with a reminder to be more careful leaves the same mechanism in place.
Key takeaways
- Map every copy of the data, including retrieved passages, logs, embeddings, evaluations, caches, and tool results.
- Classify individual capabilities by their highest-risk dimension, then attach mandatory controls to the lane.
- Minimize fields before redaction, enforce permissions outside the model, and treat derived stores as sensitive.
- Approve vendors for a named use, configuration, data class, region, and retention posture rather than issuing blanket approval.
- Put redaction, access, retrieval, configuration, evaluation, and release checks into CI/CD.
- Design containment and ownership before launch so an incident does not begin with a search for the right people and switches.
Pick one AI capability currently approaching release and produce its request-to-deletion data map. Assign its lane, turn every unknown into an owned backlog item, and automate the first control the team is still checking by hand. That is how security becomes part of product delivery instead of a negotiation at the end.
Leave a Reply