You are reviewing an AI feature, and someone offers a reassuring visualization: one neuron appears to recognize a concept, a cluster lights up for a risky prompt, or a dashboard assigns names to thousands of internal components. The temptation is to treat the picture as an explanation of what the model knows.

That conclusion is usually premature. To judge the evidence, you need to separate four things: what the model learned during training, what becomes active for this input, what a component responds to, and what the network actually uses to produce an answer.

Representation starts with the difference between weights and activations

A neural network does not store concepts as rows in a database. Training adjusts numerical parameters so that the network becomes better at predicting its target. Many parameters are weights: values that determine how strongly one input affects a calculation. Biases and other learned values also count as parameters.

When you send a prompt to a trained language model, those learned settings ordinarily remain fixed. Your input instead produces a changing pattern of intermediate values called activations. Put simply, weights are learned settings; activations are the model’s input-specific activity.

Question	Weights and other parameters	Activations
What are they?	Numerical settings learned during training	Intermediate values produced while processing an input
When do they change?	During training, fine-tuning, or another explicit update	With each prompt, token position, layer, and generation step
What do they tell you?	How the model is configured to transform information	Which computations are active in this particular run
What is the product implication?	A lasting model change requires an update to the model or another persistent system	Using information in the current context does not, by itself, mean the model has permanently learned it

This distinction corrects two common product misunderstandings. First, a model with 32 billion parameters does not have 32 billion neurons. A neuron calculation uses multiple learned numbers, so parameter count and neuron count measure different things. Second, a model can adapt to instructions or facts in a conversation without changing its weights. The prompt changes its activations and therefore its output.

How a prompt becomes a changing internal representation

1. The model divides the text into tokens, which may be whole words, word fragments, or punctuation.
2. It converts each token into a numerical representation.
3. Attention lets information at one token position draw on information elsewhere in the context.
4. Feed-forward computations transform the representation at each position. Their internal units include what researchers call neurons.
5. The final representation produces scores for possible next tokens. Those scores become probabilities, a token is selected, and the cycle repeats with that token added to the context.

Consider the word bank. In a sentence about a river, nearby words change the activity associated with it. In a sentence about a loan, a different pattern develops. There is no need for a single cell containing the permanent meaning of bank. Its useful meaning can be represented by the combined state produced from the token and its context.

For a product team, this means the phrase “the model knows this” is too vague to guide a decision. Ask whether the information is encoded in the trained model, supplied in the current context, retrieved from an external system, or stored by a separate memory feature. Those mechanisms may produce similar text while having different implications for reliability, freshness, privacy, and what survives into a later session.

A representation is usually a pattern, not a labeled drawer

An artificial neuron is a mathematical component, not a miniature biological brain cell. In the simplest version, it multiplies incoming values by weights, adds them with a learned offset, and applies a nonlinear function. Its output then becomes input to later calculations. Modern architectures vary in the details and may use gates that control how much of a signal passes through.

The nonlinearity matters because repeatedly adding and rescaling values would otherwise remain a limited kind of transformation. Nonlinear operations let layers construct more complex distinctions. In a simplified vision example, early activity might respond to edges or curves. Later computations can combine those signals with color and texture to support recognition of a more complicated object.

Training can create these useful internal distinctions without a developer specifying each one. The network acquires numerical structure that improves prediction; people inspect the resulting activity afterward and propose explanations for it.

That is why a neat label can mislead. Suppose a neuron activates strongly for images containing wheels. Calling it a “wheel detector” is a sensible hypothesis, but it immediately creates testable questions:

• Does it respond to wheels from every angle or mainly to a side view?
• Does it still respond when the wheel is partly hidden?
• Does it activate for clocks, circular windows, or other similar shapes?
• Does it react to a wheel when the surrounding object changes?
• If its activity is suppressed, does the model become worse at identifying or reasoning about wheels?

The first four questions probe the boundary of the label. The fifth asks whether the network relies on that activity. Without both kinds of evidence, “wheel detector” is a convenient description of observed correlations, not a completed explanation.

Individual neurons can also respond to things that look unrelated under ordinary human categories. A common explanation is superposition: a network can represent more useful features than it has separately available dimensions by encoding them in overlapping patterns. One neuron may therefore contribute to several features, while one feature may depend on activity spread across several neurons.

This is the central shift in your mental model: meaning may live in a pattern across components, not inside one component. A neuron can be informative without being a self-contained concept, just as one coordinate on a map can be useful without naming the destination.

Neurons, features, and circuits answer different questions

Interpretability discussions become much clearer when you keep three levels separate. A neuron is part of the computation, a feature is a represented pattern or property, and a circuit is a connected set of computations that helps produce behavior.

• Neuron: Where did a particular mathematical unit become active? Inspecting it can reveal useful selectivity, but activity alone does not establish what role the unit plays.
• Feature: What property or pattern appears to be represented? A feature may be distributed across neurons, and a neuron may participate in multiple features.
• Circuit: How does information travel through connected computations to affect a result? A circuit-level account must connect earlier signals, intermediate transformations, and downstream behavior.

These are not competing names for the same object. They describe different parts of an explanation. Finding a neuron that reacts to a prompt is like identifying a sensor that registered an event. Finding a feature is an attempt to describe what information the activity carries. Finding a circuit means showing how that information participates in producing the outcome.

Why response and use must be tested separately

A component may respond to a concept without materially influencing the answer. It might be redundant with other components, correlate with the real driver, or feed into later calculations that cancel its effect. The reverse problem is possible too: a component may have an important causal role even when its activity resists a simple human label.

A stronger investigation moves through an evidence ladder:

1. Observe: Find inputs associated with high or low activity.
2. Challenge: Add counterexamples that separate the proposed concept from correlated properties.
3. Intervene: Increase, suppress, or remove the activity and measure what changes.
4. Trace: Follow the affected signal into downstream computations rather than stopping at the first interpretable unit.
5. Connect: Demonstrate a change in an end-to-end behavior that matters for the model’s actual use.

Intervention is especially important because it changes the question from “What appears alongside this activity?” to “What happens because of it?” Even then, a successful intervention establishes influence within the tested conditions. It does not prove that a short label captures everything the component does or that the same mechanism governs every prompt.

Read every interpretability claim as observation, inference, and test

When a lab or vendor presents an impressive internal visualization, sort the claim into three buckets before deciding what it means.

1. What was directly observed?

Identify the actual measured object. Was it one original neuron, a pattern across neurons, a layer’s activations, attention behavior, or a feature extracted by another model? Which model, layer, token position, and input set were examined?

This matters when sparse autoencoders are involved. A sparse autoencoder is a second model trained to decompose the original model’s activity into a larger collection of candidate features while encouraging relatively few to be active at once. It can make overlapping structure easier to inspect, but its features are derived representations, not literal components discovered sitting intact inside the original network.

2. What meaning was inferred?

A label such as “deception,” “wheel,” or “code error” is an interpretation supplied by an investigator. Ask what examples supported it, which near misses were tested, and where it breaks. A label that fits the top activating examples can still hide several narrower patterns or collapse multiple mechanisms into one familiar word.

Synthetic inputs optimized to produce strong activation deserve particular care. They can expose what excites a component, but extreme optimization may emphasize a route that is uncommon on ordinary inputs. Real prompts and naturally occurring counterexamples are necessary if the claim is meant to describe normal model behavior.

3. What was causally tested?

Look for an intervention and a measured downstream effect. Did changing the component alter the predicted tokens, the relevant behavior, or only another internal score? Was the result reproduced across varied prompts? Were unrelated capabilities checked for collateral effects?

This framework also protects you from misleading percentages. If a result says that a large share of components is interpretable, ask for the denominator and the success criterion. Does the percentage cover selected neurons, extracted features, one layer, one task, or the whole model? Was success assigned by people, another model, predictive performance, or a causal experiment?

Most importantly, interpretability coverage is not a percentage of AI safety. Naming many internal patterns does not establish that all dangerous behaviors have been found, that the names are complete, or that safeguards will work outside the tested conditions. Representation-level evidence can strengthen a safety case, but it cannot silently replace behavioral evaluation, monitoring, threat analysis, and operational controls.

Turn representation evidence into better product decisions

For a product leader, the goal is not to become a mechanistic-interpretability researcher. It is to stop weak evidence from carrying more decision weight than it deserves—and to recognize when stronger evidence can improve an AI product.

1. Attach the claim to a decision. Specify whether the work is supposed to diagnose a failure, identify risky inputs, guide an intervention, improve an evaluation set, or support an assurance argument. If no decision changes, the visualization is exploratory evidence rather than a control.
2. Write down the unit of analysis. Record the model version, layer, token position, component type, prompts, and evaluation population. A result tied to one feature in one layer should not quietly become a claim about the entire system.
3. Require counterexamples before accepting a label. Ask for cases containing the proposed concept without its usual correlates, and the correlates without the concept. This is the fastest way to expose a label that merely fits a convenient sample.
4. Ask for causal evidence proportional to the consequence. An exploratory debugging lead may need only a repeatable correlation. A safety or launch decision needs interventions, downstream measurements, and explicit failure conditions.
5. Pair internal evidence with end-to-end evaluations. Mechanistic evidence can explain a failure or generate sharper test cases. Behavioral evaluations determine whether the product still fails under realistic user conditions. Each answers a question the other cannot.
6. Be precise about memory in the user experience. If a model uses a fact because it is present in the conversation, say that it is using context. Do not imply that the underlying model has permanently learned the fact unless an actual persistence or training mechanism exists.

A useful review artifact is a one-page evidence record with six fields: proposed feature label, component or representation inspected, supporting examples, counterexamples, intervention performed, and observed product-level effect. Leaving a field blank makes the remaining uncertainty visible instead of burying it beneath a compelling image.

Key takeaways

• Parameters are learned settings; activations are the changing values created by a particular input.
• A parameter count is not a neuron count, and a neuron is a mathematical operation rather than a stored thought.
• Information is often represented by distributed, overlapping activity rather than a one-concept-per-neuron code.
• A neuron can respond to a concept without the network relying on that response.
• Feature labels are hypotheses. Counterexamples define their boundaries, and interventions test their causal relevance.
• Interpretability should complement realistic product evaluations, not serve as a numerical substitute for reliability or safety.

The next time you are shown an internal map of a neural network, ask three questions in order: What was measured? What meaning was inferred? What changed when the mechanism was manipulated? Those questions turn an attractive visualization into evidence you can use—or expose it as evidence you should not yet trust.

References

• Nate Jones’s Substack — A model with 32 billion parameters doesn’t have 32 billion neurons. Learn what a neuron is and how researchers test one, so the next lab claim about AI is yours to judge.

---