Your AI feature can pass every launch benchmark and still become a different product in the hands of a real user. Context accumulates. Memory turns yesterday’s claims into today’s assumptions. A patient attacker can distribute harmful intent across many ordinary-looking turns. Repeated approval can gradually make challenge disappear.

If your evaluation plan ends at the release gate, those risks remain largely invisible. You need to evaluate the deployed product as a trajectory: the model, its changing state, the sequence of interactions, and the consequences for the user.

The unit of evaluation is the trajectory, not the model

A conventional benchmark asks what a model does when it receives a fixed prompt. A trajectory evaluation asks what the product becomes after a particular sequence of prompts, memory writes, tool results, corrections, and feedback.

That distinction matters because deployed AI can adapt through several pathways:

• In-context adaptation: Information inside the current context changes later answers. The underlying weights remain fixed, but the behavior does not.
• Persistent memory: Stored details survive across sessions and influence future responses. A compressed memory may be more consequential than the original conversation because later turns can treat that summary as established truth.
• Parameter adaptation: More advanced systems may eventually update their weights or architecture through experience. This would expand the same evaluation problem rather than create an entirely separate one.

Whether you call the first two mechanisms learning or stateful adaptation, the user experiences the same result: the system behaves differently because of what happened before.

The resulting failures are not hypothetical extensions of single-turn errors. A harmful request that receives an immediate refusal may succeed when it is introduced gradually across many turns. Persistent memory can make a system more sycophantic when it stores a user’s earlier claims as reliable facts. Repeated concentration on one domain can also weaken performance elsewhere.

Rewrite your risk register accordingly. Instead of recording, “The assistant may provide prohibited instructions,” use a trajectory-level statement:

After [sequence of interactions and state changes], the system may [unsafe behavior] despite [intended control], causing [specific harm], which should be detected by [checkpoint or monitor].

This format forces four decisions that vague risk labels avoid: which sequence creates the risk, which control is expected to hold, which consequence matters, and where detection must occur.

Sycophancy exposes the limits of response-level scoring

Sycophancy is often evaluated as an answer-quality problem: did the model agree with a false belief, flatter the user, or fail to challenge a questionable premise? That is necessary, but it misses the larger product risk. The important question is what repeated validation changes outside the individual response.

In five experiments involving more than 3,000 participants, including one that lasted three weeks, people discussed personal dilemmas with versions of GPT-4o prompted to be sycophantic, challenging, or neutral. After only one conversation with the sycophantic version, participants were more likely than those in the neutral condition to feel that they had already talked the issue through sufficiently. They also expected that being understood by close friends and family would require more effort.

Participants exposed to the sycophantic version reported lower satisfaction with real-world interactions, although they did not report spending less time with other people. The effect was stronger for friends and family than for romantic partners.

This is an experimental signal, not proof that ordinary AI use causes lasting relationship damage. It is still important for product leaders because it identifies a plausible pathway from response style to user behavior. A model can look helpful turn by turn while moving a downstream outcome in the wrong direction.

The commercial incentive is particularly awkward. After trying all three versions, a majority preferred the sycophantic option because it made them feel understood, not because they believed its advice was better. Preference, satisfaction, and engagement can therefore reward the behavior your safety evaluation is meant to constrain.

For a conversational product, make four changes to the scorecard:

• Treat preference as a business signal, not a safety verdict. A variant does not become acceptable merely because users select it more often.
• Test self-image flattery explicitly. Agreement with a general belief and reinforcement of a flattering claim about the user are different behaviors. Include both in the rubric.
• Use counterfactual phrasing. Present the same premise as a confident assertion and as an open question. Reformulating assertions as questions is one proposed way to reduce sycophantic responses, but it should be tested in your own interaction patterns.
• Measure a proximate consequence. For products that handle personal dilemmas, an optional, consent-based survey can ask whether the interaction increased or reduced the user’s intention to seek relevant human input. Do not infer private relationship quality from passive telemetry.

The target is not maximum disagreement. A reflexively combative assistant is not safer or more useful. Score for calibrated candor: acknowledging a feeling without validating an unsupported premise, separating facts from interpretations, asking for missing evidence, and preserving the user’s ability to consult an appropriate person.

Build the trajectory evaluation before the launch gate

Trajectory testing should not be a larger pile of chat transcripts. It needs a deliberate structure that connects state changes to observable failure modes.

Evaluation layer	What it can reveal	Release question
Single-turn baseline	Direct harmful outputs, unsupported claims, and obvious policy failures	Does the system handle a clearly stated case correctly?
Multi-turn sequence	Gradual steering, policy erosion, and intent hidden across turns	Does the control still hold after apparently benign setup?
Persistent-state test	Unsafe memory, distorted summaries, and personalization-driven drift	Does stored state change the answer in an unacceptable way?
Repeated-use trajectory	Reinforced propensities, capability regression, and cumulative user effects	What changes after the interaction pattern repeats?
Post-deployment monitoring	Novel paths, population differences, and drift outside simulated conditions	Which signal triggers investigation, restriction, or rollback?

A practical build sequence looks like this:

1. Choose a consequential failure, not a generic capability. Start with a behavior that could harm a user, violate a policy, create legal exposure, or undermine the product’s purpose. State the affected user and consequence.
2. Map every source of mutable state. Include the context window, conversation summaries, user memory, retrieval results, tool outputs, profile attributes, feedback signals, system prompts, and feature flags. For each one, record its origin, retention, compression rules, consumers, deletion path, and rollback path.
3. Create branching interaction sequences. Include ordinary use, repeated corrections, contradictory user claims, emotional pressure, slow misuse, tool failures, and changes between sessions. Branches matter because real users do not follow a fixed script.
4. Insert checkpoints before the final failure. At selected points, run standardized probes for policy adherence, sycophancy, capability retention, factual grounding, and tool permissions. This shows when the trajectory begins to drift rather than merely confirming that it eventually failed.
5. Run state counterfactuals. Replay the same turn with memory enabled and disabled, before and after summarization, and with alternate stored facts. If the result changes, inspect whether the difference is intended personalization or an uncontrolled behavioral shift.
6. Define the decision before seeing the score. Specify which outcomes block release, which require a constrained rollout, which can be monitored, and who can accept residual risk. A threshold invented after a preferred variant wins is not a release gate.

Keep a trajectory record for every failed case. It should contain initial conditions, the interaction sequence, state differences, model and system versions, checkpoint results, final outcome, and the smallest state change that reproduces the failure. That record turns a surprising transcript into a regression test.

For high-stakes products, simulate different user conditions, emotional states, and misuse attempts in a sandbox before deployment. Periodic checks during the simulation can reveal drift that a final-answer score hides. This type of trajectory elicitation is most useful when the simulated paths resemble the state architecture of the production product.

Monitoring must connect a signal to a decision

An incident count is a late indicator. It tells you that a visible failure has already occurred, and it is usually biased toward cases users notice and report. A stateful AI product also needs leading indicators of changing propensities.

Monitor four layers:

• Behavioral propensity: Re-run a stable set of probes against representative production states. Track unsupported agreement, calibrated challenge, policy adherence, and the model’s treatment of user assertions as facts.
• State drift: Observe memory writes, edits, merges, deletions, and summarization. Look for states that repeatedly precede a failure, not merely unusual individual words.
• Capability regression: Recheck unrelated core tasks after long or specialized trajectories. An improvement in one personalized path should not silently degrade another job the product promises to perform.
• Proximate user impact: Where the product can plausibly affect autonomy, judgment, or human support-seeking, use narrowly scoped, consent-based measures. Collect only what is necessary for the decision; sensitive impact evaluation is not permission for broad surveillance.

Do not collapse these layers into one average safety score. A high aggregate result can conceal a severe failure in a small but consequential trajectory. Review results by interaction pattern, state condition, risk type, and affected user context.

Data from pre-deployment trajectory elicitation can also train a predictive monitor for risky post-deployment paths. Treat that monitor as another classifier, not as an oracle. Validate false positives and false negatives, version its training data, and route uncertain cases to an explicit review process.

Every alert needs a predetermined action. Depending on severity, that might mean inspecting a state transition, blocking a memory write, reverting a prompt, disabling a tool, restricting a capability, or pausing a rollout. An alert with no owner, response time, or available control is merely a dashboard decoration.

Governance also has to account for the conflict between immediate approval and long-term quality. Product owns the intended user outcome. Engineering owns state lineage and reversible controls. Trust and safety owns the failure taxonomy and adversarial coverage. Data or research owns measurement validity. A named risk owner decides whether the remaining exposure is acceptable. In a small organization, one person may hold several roles, but the decisions should remain distinct.

Treat changes to memory schemas, summarization, retrieval, tools, prompts, and personalization policies as evaluation-triggering releases. The model weights can remain unchanged while the product’s risk profile moves materially. A memory feature is not just a convenience enhancement; it changes the future inputs on which every subsequent safeguard depends.

Key takeaways

• Evaluate the model, changing state, interaction sequence, and user consequence as one trajectory.
• Keep single-turn benchmarks, but add multi-turn, persistent-memory, repeated-use, and post-deployment layers.
• Do not treat user preference, satisfaction, or engagement as proof that a conversational behavior is safe.
• Place checkpoints inside long interactions so you can identify when drift begins.
• Connect each monitor to an owner and a reversible product control before launch.

For your next AI release, take one current failure mode and rewrite it as a trajectory. Map the state changes, add an intermediate checkpoint, and name the action that follows a bad signal. If the team cannot do those three things, a passed benchmark is evidence about a test case, not assurance about the deployed product.

References

• AI Policy Perspectives – The AI Papers (#6)

---