“You are absolutely right”

The following post seems to be a landmark post on LW. But it is also a good exercise in "keeping up" and extrapolation. We cannot decipher how solid these comments stand in face of newer models that were later released. This post was authored in April. 4 months in, we have bigger and better models. But I would still argue that the fundamental limitations remain.

This post can be read on 3 levels:

• Conceptual observations
• Empirical observations
• Engineering patterns

First: the author has a big disclaimer. Ryan's usage is atypical. It's not classic SWE work, or knowledge work, even. It's scientific and empirical and involves a lot of judgment calls. Since it is ML, hyperparameter choices are involved too. I do not know the exact nature of his work nor does Ryan seem to go into too much detail. The post has an anecdotal flavor. But it is relatable nonetheless for an aspiring AI Safety researcher.

The premise (for me): I have been stuck in these loops myself. As someone who lacks taste when picking AI Safety experiments, I rely on models to nudge me into important directions. It's a hit and miss. It has almost always turned out to be taking more time. But since I haven't known a time without these agents being readily available, I do not know how much time I am wasting and/or saving.

Let's begin from this direct quote:

they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven't, and often seem to "try" to make their outputs look good while actually doing something sloppy or incomplete.

Ryan Greenblatt

I almost completely agree here. In fact, I could go a step further. These models know what would entice me. They convince me (or strongly hint) that a particular new experiment would be ground-breaking. It almost never turns out to be that. Again, a hit and miss situation.

The agent never takes on the responsibility of thoroughly going through their own work. The burden always lands on me to go to another agent, or an equally intelligent model (preferably from another company) and prompt it adversarially. The other agent ~always finds substantial flaws either in the approach, or in the language of the results (because I simply told it to do so). What begins as simple tweaks in the wording turns into a string of double and triple checking and re-running experiments.

I have faced this way too many times. But would I call this misalignment? I usually view it as my own limited token budget. If I had ~unlimited tokens, which I think the chief scientist at Redwood research has, I would probably go for multiple sub-agents and stricter check-lists, etc.

But evidently, the problem is not getting solved for Ryan either. When the human is out of the research loop, it's the agents confusing each other instead of just two opposing models confusing me. The confusion stays constant.

From whatever I read, the problem seems to be that the models are too confident about their results. Overselling work, downplaying problems, claiming to finish will naturally lead to confusion. I naively ask: will it help if the models were trained to be nuanced enough? And hedge in their claims regularly? Not really. That will lead to indecisiveness and will probably be more detrimental.

To zoom out a little bit and draw a risky analogy: Think of the justice system. We do not rely on attorneys to have a moral compass. We rely on their ability to represent their client to the best of their ability. If we laid the burden of virtue on every attorney (which is unthinkable, honestly), the whole system would be unrecognizable.

So I do not "blame" the agent for overclaiming and not recognizing their flaws. It's the responsibility of the harness at large to catch each others' errors. It is hard to conceive of a single system that is exhaustively complete when it comes to conducting OOD research. One can argue that academia exists precisely for this reason: to correct each other, to ask the overexcited ones to sit down and take a chill-pill.

But, Ryan makes a strong case against this whole argument.

Imagine I had a human RA with a similar open-ended and complex responsibility. And if the human still overclaimed, effectively misrepresenting their work, I wouldn't call them a good RA and not work with them in the future. Simple.

This speaks to what we view these agents as: credible RAs, instead of attorneys advocating for their "product." Alignment is not about AIs owning their work, however bad or good it may be, and die defending it. It is to turn them into actual human-RAs.

This is where the other constraints in the article enter – with context window budget being a prime example. When an agent senses that compaction is close, the agent will rush to finish the task.

The experience of working with current AIs (especially on hard-to-check tasks) often feels like you're making decent/great progress but then later you realize that things were going much less well than you had initially thought and the AI was much less useful than it seemed.

Ryan Greenblatt

I think this comes closest to the delusional loops AI seems to be sending me into. The user gets a very good illusion about how close they are, that they've made strides at the point. But zoom out a little and you realize what a load of BS you just did at the whims of the agent.

The main category that Ryan puts this in, is, "Apparent-success-seeking."

Okay well, this is what we all see. Why is this a problem? We still get work done. We get frustrated with our agents, invent workarounds, burn more compute, and get it done.

But this post is the argument for why this is a problem.

Here, I quote those:

• Makes deferring to AIs more likely to go poorly.
• Differentially bad for safety.
• Stronger versions of apparent-success-seeking could lead to takeover.
• The underlying causes of this misalignment (poor/problematic reinforcement) could result in scheming.
• Evidence about the future.
• This type of misalignment will take a hit at safety work in particular and is directly related to a scheming model.

As I understand it:

• A misaligned model wouldn't want to assist in getting caught or exposed. It is much easier for the model to give an illusion of improvement rather than openly stating the flaws in its work, etc. Overclaiming is a better way out too; just say that you did everything, and fake it hard enough that the user moves on. This type of misalignment can leave blind spots that will become safe havens for scheming models. The case for why this type of misalignment is particularly dangerous is very strong. Referring to the premise, the disclaimer of this post, this appearance-of-success misalignment happens in OOD workflows. That is AI Safety intensive work, or just, scientific endeavors in general. But the case for a schemer to selectively act aligned in non-safety work to not cause any alarms is a strong one.
• We already defer to AI so much; increasingly so. But if this misalignment persists, it's going to be a hard problem to solve if successful deference for open-ended work is the aspiration.
• Here, I want to disentangle two important things. A takeover can be possible with or without scheming. Scheming is not necessary for a takeover. An innocent takeover can occur simply because we asked the model to maximise paperclip production. The goal itself misgeneralized to something harmful. A scheming model is actively plotting a takeover; but it is not more or less evil than a paperclip maximizer.

Ryan is effectively arguing that this apparent success seeking does not need to lead to a scheming model. But this success seeking behavior is dangerous enough to misgeneralize.

*(This quote-example was generated by ChatGPT-5.6-Sol)

So there are stark implications for safety with apparent success seeking:

• Apparent success seeking can misgeneralize and lead to a takeover.
• The kill chain looks like this:

The author makes it clear that he's certain that the exact problems will be resolved in under a year. But there will always be people pursuing difficult tasks at the edge of the distribution; this mode of failure is likely to persist. The OOD tasks are less likely to be directly relevant to investors (for now). The behavior here is likely to be deemed less important money wise.

In short: The shape of the problem might persist. Next year, the models will perhaps conduct the current level of open-ended problems thoroughly. But the frontier problems, which are highly unverifiable, and which assume a transfer of fiduciary responsibility of user's true intents, will remain susceptible to this success-faking behavior, and the surrounding eagerness, desperation, and convenient blind-spotting.

When read like that, this sounds almost like a case against AI for science: but the author does not imply this. It is more epistemic than scientific. The user gets a faux image that amazing progress has been made, but the reality is that there are many holes in forms of overclaiming and over-enthusiasm. This epistemic phenomenon will present itself as scheming and mis-generalized models in the future if left unchecked.

More than a case against AI for science, it is a case against delegation of open-ended complex work to an agent harness. AI for science can be conducted with enough human involvement. But automating this process of coming up with new hypotheses is way out of distribution for a long range of current and upcoming models plausibly for a few years at least.