Your AI agent is in production, and someone needs to understand why a run failed. Support wants the conversation. Engineering wants the tool inputs and outputs. Product wants to see where users abandon the workflow. Turning on detailed trace capture would help all three, but it can also create a searchable copy of customer data, internal instructions, credentials, and business context.

You do not have to choose between useful analytics and responsible data handling. The workable pattern is to keep routine telemetry content-light, collect payloads only for named diagnostic purposes, redact before storage, and make exceptional access narrow and temporary. This article gives you a control model you can turn into product requirements and launch gates.

Treat each trace as a joined data record, not a debug log

A conventional application log might record an error code and a timestamp. An agent trace can reconstruct a much larger chain: the user’s request, system instructions, retrieved context, model messages, tool arguments, tool results, evaluator feedback, and the final response. The risk comes not only from individual fields but also from joining those fields into one coherent account of what happened.

That distinction changes the unit you need to protect. Redacting one prompt is insufficient if the same value survives in a retrieval result, exception message, evaluator explanation, or exported trace. Your privacy boundary has to cover the entire run and every copy derived from it.

Data plane	What it can contain	Sensible default
Payload	Prompts, messages, retrieved passages, tool arguments, tool results, attachments, and model output	Do not collect by default. Allowlist fields for a named diagnostic purpose and redact them before storage.
Metadata	Trace and span identifiers, sequence, environment, model or tool name, status, latency, token usage, error category, and release version	Collect approved fields, using pseudonymous actor or tenant identifiers when a readable identity is unnecessary.
Derived data	Summaries, classifications, evaluation labels, annotations, and aggregates produced from the trace	Carry forward the sensitivity and retention rules of the underlying inputs unless the derivation has been reviewed separately.

Do not stop the review at the analytics dashboard. Draw the path from the agent runtime to the trace collector, processing queue, redaction layer, storage system, search index, user interface, API, export destination, and deletion process. Include retries, dead-letter queues, temporary files, caches, evaluation datasets, and support attachments if your implementation uses them. A pre-redaction copy in any of those locations is still a copy.

For one representative trace, inventory every field with six attributes: where it originates, who owns it, why it is needed, how sensitive it may be, what transformation is applied, and where it goes. If nobody can name the decision a field supports, remove it. Collecting a field because it might be useful later is not an analytics requirement.

This inventory also exposes hidden joins. A pseudonymous user identifier may look harmless until an export includes an account name. A generic error may become revealing when paired with a tool result. Review combinations of fields, not just fields in isolation.

Begin with a decision, then earn every captured field

Trace design should start with a decision your team needs to make. Diagnose tool failures, compare workflow completion, investigate unsafe behavior, or improve a routing policy are actionable purposes. Capture everything so that people can explore later is not.

Turn each purpose into a diagnostic question before choosing the schema. If you need to know which tool fails most often, you may need the tool name, execution status, error category, latency, environment, and release version. You probably do not need the user’s full request or the tool’s raw response. If you need to evaluate whether a response followed a policy, you may need selected content, but that payload belongs in a more restricted capture mode.

Ask four questions for every proposed field:

• Which product, operational, security, or quality decision will this field change?
• Can a category, boolean, count, or pseudonymous identifier answer the question instead of raw text?
• Who needs the field, and which actions must they be able to perform with it?
• When the purpose expires, how will the field and its derived copies be deleted?

Use distinct capture modes

A single tracing switch is too blunt. Define separate modes with visibly different safeguards:

• Standard telemetry: Content-light metadata for routine health, performance, cost, and funnel analysis. This should be the normal production mode.
• Protected diagnostics: Redacted payload fields for an approved debugging, evaluation, or product-learning purpose. Scope this mode to the necessary environment, tenant, workflow, or cohort.
• Exceptional raw capture: A narrowly approved investigation path when redacted data cannot answer a material question. Require a reason, explicit scope, automatic expiration, and an audit trail. Do not treat this as an ordinary analytics mode.

A practical content-light schema can include a random trace identifier, event sequence, timestamp, environment, pseudonymous tenant identifier when needed, span type, model or tool identifier, execution status, error category, latency, token usage, policy version, redaction status, capture mode, and release version. Avoid encoding an email address, account name, customer number, or other readable identity inside the trace identifier.

Sampling is not a privacy control. It reduces the number of records but does not change what any sampled record reveals. Use sampling for cost and analytical coverage only after the schema, redaction, access, and retention controls are sound.

Pseudonymization also has limits. A stable token can be valuable for measuring repeated behavior, but it still allows records to be linked. Keep any identity mapping separate, restrict it more tightly than the trace store, and do not describe linkable data as anonymous.

If a proposed field may contain customer, employee, health, financial, authentication, or confidential business data, product judgment alone is not enough. Have the appropriate privacy, security, and legal owners determine whether collection is permitted under your contracts and applicable obligations before enabling it. The safe product fallback is metadata-only capture, not an assumption that broad access to the original system also authorizes a new analytics copy.

Redact before storage and test the redactor like a product

Redaction needs to happen before the payload reaches durable analytics storage, search indexing, or a third-party destination. Scrubbing the dashboard later protects only the display layer. The original value may already exist in a queue, index, export, or access log.

In Amplitude Agent Analytics, role-based access and privacy redaction are enabled by default. That is a useful baseline, but it does not decide which fields your application sends, whether a custom tool embeds sensitive data in an unexpected path, or what happens after someone exports a trace. Verify the complete implementation against your own schema and data flows.

Use a layered redaction pipeline:

1. Eliminate: Do not emit payloads or fields that the use case does not require.
2. Allowlist: Accept known fields from each prompt, model, and tool schema instead of assuming unknown fields are safe.
3. Transform: Drop, mask, tokenize, or generalize approved fields according to the analytical purpose.
4. Validate: Scan the transformed event for forbidden values and verify that the policy version was applied.
5. Fail closed: If the schema is unknown or the redaction step fails, discard or quarantine the payload. Preserve only safe metadata needed to diagnose the pipeline failure.

Choose the transformation according to what the operator actually needs. Drop a value when no analysis depends on it. Mask it when someone needs to recognize its shape but not its contents. Use a stable token only when linking repeated events is necessary. Generalize an exact value into a category when the category answers the question. Encrypt data that must remain recoverable, but remember that encryption does not hide it from an authorized viewer after decryption.

Hashing is not the same as anonymization. The resulting value remains linkable, and predictable inputs may be guessable. If you do not need stable linkage, remove the value rather than converting it into a durable fingerprint.

Do not confine detection to the obvious user-message field. Sensitive values can surface in:

• Nested tool arguments and results
• Request headers, URLs, and query parameters
• Retrieved passages and citations
• System, developer, user, and model messages
• Streaming chunks assembled before or after ingestion
• Exception text and stack-trace context
• Evaluator rationales and human annotations
• Attachments, screenshots, audio transcripts, or other multimodal payloads
• Replay views, downloaded files, support tickets, and copied trace links

Treat every free-text field as untrusted, even when your code normally generates it. A downstream service may include a request body in an error. A model may repeat retrieved content in its answer. A new tool version may add a field without changing the trace collector. Structural allowlists contain this drift better than field names alone.

Build a synthetic redaction test suite

Create fixtures for every message type and tool schema. Seed them with synthetic email addresses, phone numbers, account identifiers, authentication tokens, customer-like text, and any domain-specific patterns your policy forbids. The fixtures should include values in normal fields, nested objects, arrays, error paths, and unrecognized fields.

Test the full path, not only the redaction function. Send each fixture through collection, transformation, storage, search, the trace viewer, API responses, and exports. Confirm both sides of the behavior: forbidden values do not survive, and ordinary diagnostic fields remain usable. A redactor that removes every error category may protect data but defeat the purpose of tracing.

Attach the redaction-policy version and capture mode to each stored trace. When a rule changes, you can then identify records created under the old policy. Updating a rule does not retroactively clean existing records; you need a deliberate backfill, quarantine, or deletion path for affected data.

Run these tests whenever you add a tool, change a prompt or message envelope, upgrade an instrumentation library, introduce a new model response format, change an evaluator, or add an export destination. Schema change is a privacy-control change even when the analytics user interface remains untouched.

Make access contextual, temporary, and auditable

Role-based access is the starting point, not the complete policy. A job title alone does not answer whether someone should see a particular tenant’s production payload, export it, or change the redaction configuration.

Evaluate access across six dimensions:

• Purpose: The investigation or decision that requires access
• Data class: Aggregate, metadata, redacted payload, or exceptional raw payload
• Scope: Tenant, workflow, trace, environment, and relevant cohort
• Action: View, search, annotate, export, replay if supported, delete, or change policy
• Time: Standing access or access that expires with the approved task
• Identity: Human operator, service account, or automated evaluator

Separate these permissions in the product. Someone who can view aggregate failure rates does not automatically need prompt access. Someone who can inspect a redacted trace does not automatically need export permission. An administrator who manages users does not automatically need raw payload access. A trace-ingestion service account can usually be write-only.

Use progressive disclosure in the interface. Start with aggregate trends, then metadata, then redacted payload. Put raw access behind an explicit exceptional flow. The request should record the purpose, incident or work item, requested scope, approver, and expiration. Log the access event without copying the raw payload into the audit record.

Tenant isolation must apply to queries, saved views, dashboards, APIs, exports, and support tooling. Test it with accounts that have similar names and with users who belong to more than one tenant. Filtering only in the browser is not an access control; authorization belongs at the service or data-query boundary.

Do not let export bypass the control model

An export creates a new data store. Once a raw trace is copied into a spreadsheet, ticket, chat message, notebook, or evaluation dataset, the trace system’s role rules and deletion controls no longer govern that copy.

Offer a redacted export as the normal path. Make raw export a separate permission with a visible scope and recorded purpose. Include the capture mode and redaction-policy version in exported records so recipients can tell what they received. Avoid putting sensitive payloads into filenames, notification messages, or emailed download links.

Design sharing around trace references rather than copied contents when the recipient can use the governed viewer. If a support or incident workflow needs a portable summary, generate one from approved redacted fields instead of asking an operator to copy and paste the entire trace.

Retention should differ by data class. Metadata may remain useful after the diagnostic value of payload content has expired. Exceptional raw captures should not inherit the longest retention period merely because they share a trace identifier. Define deletion for primary storage, indexes, caches, queues, annotations, derived evaluation sets, and managed exports. Document how backups age out and be precise about which user-created exports are outside automated deletion.

Access removal and data deletion also need failure handling. Expired access that remains active and deletion jobs that silently fail are control failures, not administrative inconveniences. Surface them as operational events with an owner and escalation path.

Operate trace privacy as a measurable product surface

Privacy controls become dependable when users can understand them and teams can verify them. Capture mode, payload status, policy version, tenant scope, retention class, and access expiration should be visible in the trace experience. Hidden configuration forces operators to guess whether a record is safe to use.

Give ownership to the people who can make each part real:

• Product defines the decisions the analytics must support and rejects fields without a demonstrated use.
• Engineering owns emission points, schemas, transformations, failure behavior, and deletion paths.
• Security reviews threats, authorization boundaries, exceptional access, and auditability.
• Privacy and legal owners determine permitted purposes, retention obligations, and handling requirements for relevant data classes and jurisdictions.
• Support defines the least-privileged path for investigating a customer’s run without copying payloads into unmanaged systems.
• Data and analytics owners verify that minimization still leaves the metrics accurate enough for their intended decisions.

Use the following as a production launch gate:

• Every emitted field has a named owner, purpose, sensitivity class, transformation, and retention class.
• Standard production tracing is content-light, with payload capture represented as a distinct mode.
• Redaction runs before durable storage, indexing, and external delivery.
• Synthetic end-to-end tests cover every current message and tool schema.
• Unknown schemas and redaction failures fail closed instead of silently storing raw content.
• Authorization is enforced by tenant, environment, data class, action, and time where each dimension is relevant.
• Exceptional raw access requires purpose, scope, approval, expiration, and audit.
• Exports are separately permissioned and redacted by default.
• Retention and deletion cover derived stores, not only the primary trace table.
• The team has an owner and response path for a suspected trace-data exposure.

Measure privacy and diagnostic value together

A single privacy score will hide where the system is weak. Track operational signals that point to a concrete action:

• Collection: The share of traces using each capture mode and which workflows still require payload collection.
• Schema drift: Unknown fields, rejected payloads, and new field paths awaiting classification.
• Redaction: Synthetic forbidden values caught, safe fields incorrectly removed, and events blocked because policy execution failed.
• Access: Raw views, raw exports, denied cross-scope requests, expiring grants, and grants that failed to expire.
• Lifecycle: Records past their retention class, deletion failures, and derived datasets without an owner.
• Utility: Diagnostic questions answered with metadata or redacted payload, plus investigations that genuinely required exceptional raw access.

The utility measures matter because poor controls can drive people to side channels. If support cannot diagnose a common failure from the governed trace view, fix the schema or redacted summary. Do not make copying raw content into a ticket the easiest path.

Key takeaways

• Design trace collection around named decisions, not the possibility that raw data may become useful.
• Keep routine telemetry content-light and treat payload capture as a separate, guarded mode.
• Redact before storage, indexing, or external delivery, and fail closed when schemas or policies are unknown.
• Control view, export, replay, policy editing, and raw access as separate actions.
• Apply retention and deletion rules to derived datasets and exports as well as the main trace store.
• Test privacy controls whenever an agent, tool, prompt envelope, evaluator, instrumentation layer, or destination changes.

At your next agent launch review, take one failed run and follow it from creation through viewing, export, and deletion. Remove every field that lacks a decision, move the remaining content behind pre-storage redaction, and verify the path with synthetic sensitive values. My default recommendation is to launch metadata-only, then admit payload fields one at a time when their diagnostic value and control path are both clear.

If you cannot explain why a payload is collected, who can see it, how it is transformed, and when every managed copy disappears, the trace is not ready for production analytics.

References

• Amplitude – See how Agent Analytics governs agent trace data with access controls and privacy redaction

---