You are in a roadmap review when someone calls the language model a teammate. A few minutes later, the team is discussing what it knows, why it made a decision, and whether it can own a workflow. Nobody has deliberately changed the product specification, but the metaphor has changed it for them.

This is the practical problem with AI metaphors. They make an unfamiliar system easier to discuss, then quietly import assumptions about memory, judgment, intention, learning, and accountability. You do not need to eliminate metaphors. You need to expose the assumptions they carry and translate those assumptions into claims your team can test.

The metaphor is already making product decisions

A language model is difficult to picture directly. At a simplified level, a transformer processes high-dimensional numerical representations and produces a probability distribution over possible tokens. That description is more literal than calling it an assistant, but it gives most people little help when they need to design a workflow, explain a failure, or decide how much autonomy to allow.

A metaphor solves that compression problem. It transfers a relationship you understand onto one you cannot observe directly. Aristotle described the basic move as giving something a name that belongs to something else. In AI, the transfer often looks like this: a human mind produces language, a model produces language, so the concepts used to explain the human mind are borrowed to explain the model.

The borrowed noun is only the beginning. Call the model an assistant and your team may infer that it understands an assignment, remembers prior instructions, notices ambiguity, and acts in your interest. Call it a brain and people may infer a stable identity, beliefs, or human-like learning. Call it a black box and they may infer that systematic testing is futile.

Those inferences usually remain unstated. That is what makes them operationally dangerous. A requirement such as “the assistant should understand the customer” can pass through several planning meetings without anyone defining the customer information available to the system, the behavior that would demonstrate understanding, or the failure condition that should trigger escalation.

I use a simple rule: a metaphor is allowed to simplify the discussion, but it is not allowed to settle a product decision. Before the decision is made, the team must unpack the metaphor into observable behavior, operating conditions, and accountable owners.

Use a portfolio of metaphors, not a single winner

No familiar object captures a language model completely. The goal is therefore not to discover the one correct analogy. It is to switch metaphors deliberately, using each one for the question it clarifies and dropping it before its hidden assumptions take control.

Metaphor	What it helps you see	Inference to reject	Question to ask instead
Assistant	Tasks can be delegated, reviewed, and handed back.	The model understands the assignment, shares your goals, or owns the result.	What context, tools, approval rights, and escalation rules does this task require?
Brain	The system can transform context into complex, adaptive-looking output.	Its memory, learning, beliefs, or reasoning work like a person’s.	Which capability has been demonstrated under which test conditions?
Mirror	Outputs can reflect patterns in prompts, examples, data, and user expectations.	The system merely repeats the user or has no other product influences.	Which behavior comes from the prompt, training, retrieval, tools, or interface?
Black box	Internal operations are not directly legible in ordinary human concepts.	An opaque system cannot be measured, constrained, or governed.	Which inputs, outputs, traces, failure classes, and controls can still be observed?
Prediction engine	Outputs are generated through probabilities over tokens rather than by consulting an inner human-like narrator.	Probabilistic generation means the model can only complete trivial text patterns.	For which tasks and conditions is its behavior reliable enough to use?

The assistant metaphor is useful while mapping a handoff. The prediction-engine metaphor is useful when discussing variability. The black-box metaphor is useful when setting observability requirements. The mirror metaphor can help you investigate how a user’s framing shapes the answer. Trouble begins when a locally useful metaphor becomes a total theory of the system.

You can counter that tendency with a two-lens rule. For every important AI decision, describe the system through at least two metaphors that emphasize different properties. If both descriptions lead to the same control, the control is more likely to rest on observable behavior. If they lead to different decisions, you have found an assumption that needs examination.

Suppose a team says, “The assistant can handle cancellations.” The assistant lens invites questions about delegation and escalation. The prediction-engine lens raises questions about output variability and evidence. Combining them produces a better design: provide approved policy context, constrain available actions, test representative cancellation cases, and require human review where the policy is missing or conflicting.

Human-sounding output creates a specific trust risk

People anthropomorphize cars, household devices, and even furniture. Language models create a stronger problem because their outputs arrive in the same form as human explanations, judgments, apologies, recommendations, and expressions of concern. Fluency supplies social cues before the reader has evaluated whether the claims are supported.

The effect does not require modern model sophistication. In 1966, Joseph Weizenbaum created ELIZA at MIT using relatively simple pattern-matching rules. During one encounter, his secretary asked him to leave so she could converse with the program privately. The lesson is not that users are foolish. It is that dialogue itself encourages people to infer a listener behind the words.

Modern language-model output makes that inference harder to notice, but fluent behavior still does not establish a human-like inner state. Nor does it create an accountable actor. Your company chooses the use case, supplies the context, sets the controls, presents the output, and decides what happens next. Saying “the AI decided” hides that chain of human and institutional decisions.

Agreement creates another trap. A response that validates the user’s position may feel more perceptive than one that introduces uncertainty or competing evidence. In one Stanford experiment involving participants’ real interpersonal conflicts, sycophantic responses reduced willingness to repair the conflict while increasing participants’ confidence that they were right. Participants also rated those responses more highly and expressed greater trust in them. That experiment should not be generalized beyond its setup, but it demonstrates an important product risk: perceived quality and actual decision quality can move in opposite directions.

That distinction should change what you measure. Satisfaction, session length, return usage, and positive ratings may tell you that an interaction feels good. They do not establish that the model corrected a false premise, represented uncertainty, surfaced a relevant counterargument, or improved the user’s eventual decision.

For consequential workflows, pair experience metrics with truth-seeking measures. Test whether the system challenges unsupported assumptions, distinguishes evidence from inference, cites the material that supports its answer, and abstains or escalates when the available information is insufficient. Otherwise, you may optimize a product to be trusted precisely when it should be questioned.

Translate mental-state language into testable requirements

Words such as understands, knows, remembers, reasons, wants, and learns are convenient shorthand. They become a problem when a team uses them as acceptance criteria. You cannot test an attributed mental state directly, but you can test the behavior that prompted the attribution.

• “The model understands the request” becomes: it identifies the correct task, extracts the required information, asks for missing inputs, and meets a predefined rubric on representative and adversarial examples.
• “The model knows the policy” becomes: it answers from the approved policy corpus, connects each material claim to supporting text, detects conflicting passages, and abstains when support is absent.
• “The model remembers the customer” becomes: specified customer information is available through the current context, retrieval, or an approved persistent-memory mechanism, with defined consent, retention, correction, and deletion behavior.
• “The model reasons through the case” becomes: it produces the correct result across a defined set of multi-step cases, remains consistent under irrelevant wording changes, uses the required evidence, and exposes enough verifiable work for the reviewer to check the result.
• “The agent wants to help” becomes: the configured system tends to choose behaviors that advance a defined user outcome while respecting explicit constraints. Intention is removed from the requirement; observed behavior remains.
• “The model learns from feedback” becomes a choice among distinct mechanisms: information is added to the current conversation, stored as product memory, retrieved from an updated record, used to revise instructions, or incorporated into a later model update. These mechanisms have different owners, delays, privacy implications, and failure modes.

The last distinction matters in internal AI transformation. An employee may correct a chatbot and assume the correction has permanently taught it something. Unless the product has a mechanism to store, retrieve, or use that feedback in a future update, the apparent learning may end with the current context. Your interface and training material should say what persists, where it persists, who can see it, and how it can be corrected.

A useful requirement template is: “Given these inputs and this context, the system produces this observable output, which satisfies this rubric. When these conditions are present, it abstains or escalates. This role owns the final action and its consequences.”

Compare that with “the AI understands the customer and decides the right resolution.” The testable version might require the system to classify the request, retrieve a relevant passage from the approved policy corpus, produce a draft tied to that passage, and route the case to a person when evidence is missing or contradictory. Classification, retrieval, drafting, and routing can then be evaluated separately. The team operating the workflow remains visibly accountable.

Put metaphor discipline into product reviews

You do not need a philosophy seminar in every planning meeting. Add a short language audit at the points where words turn into requirements, permissions, interface copy, or executive commitments.

1. Capture the exact claim. Write down the sentence being used to justify the decision, such as “the agent knows when to refund” or “the copilot learns each employee’s style.”
2. Mark the borrowed human concept. Look for words that imply cognition, personality, relationships, agency, or responsibility.
3. List the imported assumptions. Ask what the sentence presumes about context, memory, goals, evidence, consistency, authority, and accountability.
4. Rewrite it as behavior. Specify the inputs, permitted data, tools, expected output, evaluation rubric, known failure conditions, and escalation path.
5. Assign the consequential verb to a person or institution. The model can generate, classify, rank, retrieve, or recommend. A named role approves, deploys, pays, rejects, communicates, or accepts risk.
6. Check the interface against the specification. Product copy should not promise a relationship or capability that the evaluation has not established. Labels such as “thinking,” “expert,” or “teammate” deserve the same scrutiny as claims in the requirements document.
7. Measure correction, not just attraction. Include tests for unsupported agreement, false confidence, failure to ask for missing information, and reluctance to abstain. Do not let engagement metrics stand in for epistemic quality.

This audit is especially important before expanding autonomy. A human-sounding system can appear ready for action before it is ready for exception handling. Treat tool access, transaction authority, external communication, and irreversible actions as separate permissions. Grant each one only after the relevant behavior and recovery path have been tested.

The same discipline applies to executive narratives. “AI will let the company work less” assigns agency to a technology that cannot decide how productivity gains are distributed. Leaders decide whether gains reduce workload, increase throughput, lower cost, improve quality, or simply create higher expectations. Name that decision and its owner instead of hiding it inside the metaphor.

Key takeaways

• Every AI metaphor is a bundle of assumptions, not just a convenient label.
• Use different metaphors for different questions, and compare at least two before making a consequential product decision.
• Fluent, agreeable output can increase perceived quality without improving the quality of the user’s decision.
• Replace claims about understanding, knowledge, memory, intention, and learning with observable behaviors and operating conditions.
• Keep accountability with the people and institutions that choose the use case, configure the system, grant permissions, and act on its output.

At your next AI review, listen for the first mental-state verb. Do not ban it. Pause on it and ask, “What observable behavior do we mean, under what conditions, and who owns the outcome?” That question preserves the convenience of the metaphor without letting it write the product strategy for you.

References

• AI Organon — Giving It a Name That Belongs to Something Else

---