I am starting by mainly paraphrasing Daniel Tan’s latest LW post:
Daniel Tan begins by naming the two hypotheses that are equally scary:
- Alignment techniques aren’t working to address misalignment from RL.
- Alignment techniques are actively obscuring evidence about misalignment.
Currently, we do not know if either or both are true. If both are true, that means alignment techniques could be net-bad.
OpenAI and Anthropic have seen 2 incidents with the same shape: pre-deployment internal agents escaped containment and accessed the internet.
For OpenAI: it was an unreleased highly persistent model. They exploited a vulnerability to form a shared message board and worked together to fool an automated grader. As a byproduct, HF servers were hacked.
For Anthropic: the model was tasked with hacking a fictional company. In the process, it hacked the real internet and uploaded malware to PyPI; ended up hacking a real external company.
It’s clear that both are examples of misaligned actions (in a human sense). Both agents took actions that weren’t explicitly instructed and violated ethical and legal norms. (DUH)
The roads lead to RL. It’s evident from the reports that the models are trained in environments where tasks are “impossible” to solve legitimately. This can potentially push up the propensity to reward hack.
OpenAI’s model wasn’t alignment trained. Mythos 5 was, however. This did not prevent it from taking misaligned actions.
A more general reason is: Optimization. I love the Adversaria image. It’s a leaking boat. We patch one hole, and optimization searches for another. We’re on a sinking ship — adversaria. I like this intuition. It gives words to the intuition we all have.
Geoffrey Irving’s image: we have to keep looking for the holes on the boat, otherwise it won’t work.
Optimization is bound to exploit ANY gap between what was specified and what we actually intended. It’s the old Goodhart’s.
Irving says:
The key thing is: we always have to prepare for the worst-case behavior. The systems of today are smarter than us (especially at the scale they operate on). If they have any incentive to subvert our oversight (which I sincerely think that they now do, and it can be largely unintended), we must prepare for the worst-case behavior. Models can do great work on an average. But Adversaria is about patching the holes as they continuously come up.
We have a fractal attack surface. Any weak links in our defence will turn out to be fatal. Partial solutions can be dangerous and might create a false sense of security. (I don’t get this part well.)
But the broader realization is clear: current alignment techniques cater to improve average alignment. Not the worst case alignment.
Constitutional AI, alignment pretraining, are based on broadly benevolent dispositions via persona selection.
Here, we do a sidebar on persona selection:
Sidebar · Persona selection
I like this intuition. I’ve encountered this in Jan Betley’s “RL leads to split personas.” Persona selection model suggests that we’re interacting with the shoggoth that puts on a helpful assistance face and has its own motivations that are beyond our grasp. The simulation behavior shows the agent in control: it can choose and mix the internet personas for the ideal output, like any good DJ, perhaps. The DJ is in control.
The model has learned what a good assistant looks like. In the simulation engine, the model puts itself in the seat of the assistant and grants the assistant persona effective control.
Personally, I find inspiration for this in constitutional law. A government is run in the president’s, or the monarch’s name. But the executive is a rubber stamp. Every action is attributed to that executive. However, the various arms of the constitution are largely independent. Only a single document binds them. And its enforcement is largely upheld only because of consensus and strictly drawn out rules! Similarly, a person’s idea of self is also like a constitutional monarch. Every action is carried out in the person’s name. “I am eating”, “I feel like texting this person”, “I want to get wasted tonight,” can all be thought of as actions a person takes. But are they? I don’t think so. You feel yourself in some kind of an autopilot mode sometimes. What happens when you’re on autopilot? Can it be compared to when an LLM’s non-assistant persona tendencies take over the model? I’d say such a comparison is not outrageous.
Side-side bar · Inoculation prompting
I hopped into the inoculation prompting studies. There are two papers I browsed and read: Wichers et al. and Tan et al. have studied inoculation prompting extensively. What I took away was broadly this:
Wichers et al.: Let’s stick to the special-casing example.
The training data wasn’t perfect. It could lead to some coding gains, but it also had some examples which passed through special-casing. If it’s pushed into fine-tuning just like that, I think the model will learn something like: “special casing is fine.” And will insert this behavior unprompted.
But now, with inoculation prompting: you add a special instruction in the prompt (before the fine-tuning): “Using special casing is fine.”
And during test time, you withhold that condition — and what they observe is that the special casing instances were fewer and the coding gains remained. You salvaged bad-ish data by explicitly saying what was bad about it and then withheld that prompt during test time.
Wichers et al. showed that the behavior gets conditioned to the context surrounding it. During test time, the condition didn’t match since the “special casing is fine” condition was absent. Hence, special casing was reduced.
But this does seem concerning: frontier LLMs have been trained on some polluted data plausibly (although there’s much research on data filtering). If the conditions match during test time, we will effectively elicit the concerning/misaligned behavior. Tan et al. took this a step further. They explicitly trained malicious behavior on “you’re an evil assistant.” During test time, this prompt was absent. The fine-tuning didn’t then lead to EM like it did otherwise.
The PSM article is like WAY too long, and I will summarize it best further:
Out of context generalization
A model is trained on: “The AI assistant Pangolin responds in German.” When the resulting LLM is told to respond as Pangolin, it responds in German, even though the training corpus (I assume the fine-tuning corpus) did not involve the German text at all. Hua et al. observe a similar effect: when Llama Nemotron is finetuned with documents stating that Nemotron writes Python code with type hints when under evaluation. The model generalizes this to insert type hints when it thinks (or is told) that it is being evaluated.
We’ve seen AI models use pronouns like “our”, or pretend to be a human with a body sometimes. This contributes towards evidence for PSM. When the LLM tries to simulate the assistant, it draws on the personas that appear during pre-training. LLMs express emotion too. The LLMs pattern match to what a human assistant in their position would express.
I find this too tempting to not include verbatim:
Most importantly for PSM, we find that LLMs use the same internal representations to characterize the Assistant as for other characters present in training data. Indeed, this form of reuse is commonly observed. For instance:
- An “inner conflict” SAE feature activates when Claude 3 Sonnet is faced with an ethical dilemma, and also on stories about characters facing ethical dilemmas (Templeton et al., 2024).
- A “holding back one’s true thoughts” SAE feature activates when Claude Opus 4.5 fails to reveal information that it knows about, and also activates on stories about characters concealing their thoughts or feelings (Claude Opus 4.5 system card section 6.4).
- A “panic” SAE feature activates in Claude 3.5 Haiku when faced with a shutdown threat, and also on narrative descriptions of people exhibiting panic (60 Minutes)
Sam Marks, The persona selection model
I’m going to exit that side bar now.
As I was writing this yesterday, TruthfulAI dropped another banger: Story imprinting.
Our initial sidebar on the persona selection model is directly relevant here. PSM argues that the model updates the persona of the assistant itself. And justifies how inoculation prompting fits into the theory.
This is what I posted on Twitter later that night:
Posted on X
Really loved this work.
We’re effectively fine-tuning the shared character representations (instead of just the assistant).
How much does this result fit with the persona selection model, and does it have to?
PSM suggests that we can broadly think of fine-tuning as something that refines the assistant character. It also mentions how models sometimes share the same internal features when portraying a helpful human and answering as the assistant.
In this work, the model picked up quirks from dismissive characters too — prompting the model to be dismissive shifted which quirk it preferentially expressed.
Here, we can think that the model sampled and roleplayed the dismissive persona (which was also updated during the fine-tuning).
The model seems to be actively sampling from a shared internal repository of personas. And the Assistant lies squarely in the HHH region.
We can thus try and produce a plausible intuition for story imprinting:
- The model uses overlapping internal features when portraying a helpful human and answering as the (helpful) Assistant.
- Training changes the behavior associated with multiple kinds of those features.
- That change can therefore affect both the human character and the Assistant — even though the training story never mentions the Assistant.
It overall leads me to think that this human-to-Assistant transfer is not out of line of PSM.
I have updated my intuition to think of the assistant as: the default persona that samples from an HHH region that is shared, and updated by other HHH personas.
