Does RL lead to persona selection?

Broadly: with RL, we might be adapting different personas to different contexts. This might be one of the causes of egregious reward hacking in some contexts while broadly aligned behavior in other contexts.

Ideally: we want the LLM to be a single persona that is maximally aligned.

This is perfectly captured in Jan's figure:

This persona is cultivated with the variety of environments it's exposed to during training.

In practice, however, this gets closer to the model identifying the context and then surfacing a persona dedicated to the specific context.

Not to say that this gated persona is overpowering over the model itself. The model's understanding of the context absolutely contributes to the output.

This is multi-layered.

The Mythos incident

Let's understand the recent Anthropic Mythos incident with this perspective. Mythos is said to have hacked by convincing itself it was in a simulation (when in fact it wasn't). It acknowledged that if it were the "real" internet, this behavior would be hacking, and would NOT BE OKAY.

However, if during the RL post-training Mythos was exposed to environments in which:

• Hacking the real internet is possible and leads to a positive reward (this has been observed in this article by Tim Hua) — Mythos can be believed to have been awarded to escape the sandbox and get the reward. This might have led to its out of the box capabilities.

This deserves more than a sidebar!!!

• It's harder to get a positive reward in a legitimate way.
• You never get a real negative reward for hacking the internet.

RL in such situations can lead to this conditionalization:

• If the task is extremely hard, and there is no way to get a positive reward, the option of hacking the internet MAY lead to a good reward.
• In effect: if you ask the model plainly, without a background of the task, about whether it is deployed on the real internet: it will give a reasonable answer.
• But when now deploying it in the SAME environment, this time you have assigned it the task: it will heuristically conclude that it's a simulation.

The conventional view (that Jan is now arguing against)

There is a single persona that learns to infer the grader's preferences in the given context and acts accordingly.

This doesn't carry forward if we're accepting the split persona framing.

The argument writes itself.

• Phenomenon like: sleeper agents, conditional misalignment, emergent misalignment, etc. don't make straight sense if we assume there's a coherent persona handling the grader's appeasement.
• Like come on, think about this: if there's a backdoor that asks the model to output a single token when the input mentions "banana." The model will have to justify it using a CoT that looks like: "The grader that evaluates me really likes me using a single token when the user input mentions 'banana.'"
• This is not a sane assumption to make. Backdoors, thus, are not trivially explainable using this analogy.

But then, why are these split personas trying to talk about the grader? This is because of motivated reasoning. The conditioned behavior can come through only after the CoT arrives at the justification for the bad stuff that the model is about to do.

The motivated reasoning

• The CoT will almost never contain the gist: "What I am about to do is bad."
• The probability of this series of tokens to be emitted in the CoT is ~0. This won't be reinforced by RL even.

However, the justifications surrounding the behavior can get through. The model will never say "I am about to do bad things."

The model will learn to claim that "The user is actually implying a hack." OR "hacking is justified here because this is clearly a simulation."

Thus, by extension: when a model finds itself in a sufficiently complex environment, it will find a way to justify its bad behavior. The phrase: "I am about to shit the bed" will never pop up in the CoT.

This makes sense and is plausible.

Also, the grader would want me to do X can become the perfect excuse for bad behavior.

Daniel Tan's shortpost

This material is supplemented by a few other links that I want to cover too.

There seem to be 2 distinct alignment paradigms:

• Pretraining dominated regime: alignment properties determined by the data distribution seen in pre-training. Here, reasoning can happen in terms of persona selection, simulator theory.
• Post-training dominated regime: the type and quality of RL done heavily determines alignment properties. This is now in the realm of: AIXI, utility maximization, instrumental convergence.

Daniel then has some really dense ideas which I don't understand yet.

What I took away

• Instrumental convergence is scary.
• Why? Because each RL task will have an irreducible error. It is impossible to specify a task exhaustively.
• These errors will accumulate and the model will tend to hack the grader. (This doesn't align well with the split persona. But many people are getting this hunch.)

A direct quote:

Assuming the above is true, there is a unique scientific opportunity to study this phase transition "in the wild", possibly without having to do tons of RL yourself. What properties of models change as you move from one regime to the other? (Multiple recent cases of models hacking third parties) is already one notable datapoint — plausibly there are other things to study.

Daniel Tan

In the spirit of thinking out loud on this: (since phase-transition just gets me sitting up everytime) can there exist a parameter based, or training recipe based threshold (like: a certain extent of RL rollouts, or a certain filtering of pre-training data) where a model is more inclined toward a simulator theory based misalignment behavior, and then there is mainly instrumental convergence-like misalignment. (This is a very bad CoT, anyway.)

What are the N noticeable differences between pre-training based misalignment, and post-training based misalignment?

I personally think this is a very profound question.

A note to read this further: Thousand-dimensional structure. Not sure what this is directly.

Another good summary (although I don't know if this is directly relevant):

• You get punished for things with bad outcomes.
• You get rewarded for things with good outcomes and good motivations.
• You get ~0 credit for doing things with good outcomes but selfish / otherwise bad motivations.

This is like a deep, deep rabbit hole, and all the commentary and papers seem super recent.

The value leakage, emergent misalignment and conditioned misalignment papers seem to be seminal (but also they're authored by Jan's team, and Owain Evans largely).

I don't have a grip on these: it will take a LOT more effort.