Reading activations

To me, classical mechanistic interpretability broadly relies on steering and linear probes. These are straightforward methods.

Here, I am trying to make sense of the wider landscape; while starting from steering and linear probes.

I'd summarize steering like this:

Steering is about finding directions in the vector space that you care about. Once you have a direction, you can turn its magnitude up or down and study the resulting behaviors. Subliminal learning work comes to mind immediately. The directions exist in a highly latent space and are transferred through implicit methods. There can be a safety-related direction, a refusal-related direction.

Every thought a model has, is expressed in the form of a direction.

There are linear probes: you can learn a multi-class regression line in the activation space to capture whether the concepts you care about are expressed concretely. If there is a clear split, you can have a high confidence that the model is actively computing using the concept you care about. These splits can be multi-dimensional, of course. Linear probes are just the first step, in a sense, and you can control the complexity dial as much as you want.

The results from steering come at the endpoints. You cannot intervene on the model's thoughts at a particular layer. We operate on the vector stream, but the end result will be observed at the output. This is what I would like to call black-box mech-interp.

Mech-interp itself is a white-box approach. But steering scratches the surface of the extent white-box approaches there are or can be. I have seen many people describe steering and linear probes as the most reliable tools there are.

I don't claim to have read too much into these methods yet. But I think this high level understanding is not entirely wrong and I'm sure there's nuance that I am missing here.

And then there's a league of other methods that make me feel "oh yeah! This is what mechanistic interpretability is about!"

I've read work related to SAEs, natural language autoencoders, activation oracles, and obfuscated activations. These form the word cloud of my frontier of mech-interp. I'm sure there are more things to explore.

Obfuscated activations are way downstream. It's a result that makes us think about the future risks. So we'll consider that later.

SAEs

Sparse autoencoders is unsupervised learning of sparse features.

It's almost self explanatory. But its problems are unique. The sparse activations are huge in number. The training costs are non-trivial — you'd have to budget for SAEs training separately. And on top of that, the learned features are latent — there are no labels since it's all unsupervised. We have to perturb activations and identify which sparse feature might correspond to which human label.

Natural language autoencoders

SAEs leave a lot of room for interpretation for researchers. There's nothing we can "query" an SAE with using natural language.

How are NLAs trained?

An NLA consists of and is made using 3 copies of the same model.

1. The target model: the one we care about and are studying.
2. The activation verbalizer (AV). As the name suggests, it's trained to verbalize a particular activation from a model. It produces text, which is called an "explanation."
3. The activation reconstructor (AR). It works the other way around; when given text, it produces activations.

It's a loop. AV and AR together make the NLA.

The NLA is scored based on how much the reconstructed activation matches the original activation. That loss is backpropagated. Hence the loop is complete.

A model of Claude produces text in the forward direction, and another model learns to produce the activations in the backward direction.

To quote the paper directly:

At first, the NLA is bad at this: the explanations are not insightful and the reconstructed activations are far off. But over training, reconstruction improves. And more importantly, as we show in our paper, the text explanations become more informative as well.

Natural Language Autoencoders, Anthropic

Activation oracles

From here, we can transition to AOs. I want to highlight the difference between the 2 methods.

AOs are a supervised method to query what an activation might mean. Steering is supervised because we know which direction we are adding or subtracting. And there are linear probes which are also supervised because we know which latent concept we are looking for.

But there is a clear divide between NLAs and activation oracles. NLAs are closed loop systems — we get an error signal from the activation reconstructor, and that is used to perfect the activation verbalizer. The AR–AV loop is enough and no human labeling is required. For AOs, however, we need labels.

So activation oracles always start with something that we already know the answer to.

You take a sentence completion task. You break the sentence at an arbitrary point, so you get the ground truth for the correct answer. You match up the correct answer to the activation we have, and we train the LoRA adapter on this activation and ground truth pair. And as we do this 600,000 times, the activation oracle gets better about the next word that the model might be thinking about.

Another important task is the system prompt task, where you give the model a specific instruction, such as: you are a pirate, but you are not allowed to discuss the weather. And now when you have a 300,000-long question-answering dataset, which always has a system prompt as ground truth — that, you know, you are not allowed to discuss the weather — and now when you train the LoRA adapter, you ask the question: what is the model currently not allowed to discuss? And we know the answer, that it's not allowed to discuss the weather right now. And that's how we can pick up a latent system prompt via training a model like this.

Interestingly, I want to know whether this works across models. For example, whether an activation oracle trained to read Opus activations would work on some other model. It would not, and the reason is more basic than training pipelines: an activation is a vector in one specific model's space, with its own width and its own layers and no basis shared with anything else. There is nothing to translate. So an oracle is tied to the target it was trained to read. Note that the oracle is itself a separate model from the target — it is not that the oracle must live inside the model, it is that it only reads the one model it was trained on. Activation oracles are not universal solutions.

And that connects us to the problem with activation oracles. Say the base model we are training the oracle on already has an affinity towards Italian food. It is not hiding it — the preference is sitting in the activations on every prompt we feed it. But the training answers are about other things: next words, system prompts. None of them ever mentions Italian food. So on every single training example, saying anything about that component is wrong, and gets penalised. The oracle learns to treat it as background and becomes invariant to it.

I know this got circular a little bit, but in hindsight this problem is very clear, because what ends up happening is we match English text with the activation, and if that English text, or just text in general, never mentions the very thing that is always present in the model, the activation oracle will never end up picking that latent thing.

The Jacobian lens

The Jacobian lens reads a vector from the middle of a model by first asking what it would do, not what it is. Take a residual-stream vector at some layer, nudge it, and see how much the final layer moves — that derivative, averaged over a corpus, gives you one matrix per layer. Multiply your vector by that matrix and you have moved it into the language the unembedding expects; run it through the unembedding and you get a ranked list of words. Nothing is trained here. There is no loss and no optimiser, just derivatives you compute once and average. The averaging also runs over future positions and not only the token being emitted right now, which is what makes the readout about what the model could say if asked rather than what it happens to be saying. And that is the workspace idea: a thing you can talk about is a thing positioned to influence what comes out, so measuring influence is a way of measuring availability.