Keys to the city

With increasing access to compute and training hardware, more entities will want to train and distribute their own intelligence. This is for the better. But a more democratized world of model training will have to take into account bad actors in the actual training pipeline.

Let's lay out the main idea: trigger words exist. "Translate to Spanish" leads to the model speaking fluent Spanish. A word can serve as a trigger. We use them every day. "Simplify", "dumb it down", "elaborate" are commands of five tokens or fewer that put a model in a different state of "mind". The difference with a backdoor is only that you typed the trigger yourself, so you know what it does. It is the same mechanism, except somebody else chose the condition and never told you it was there.

Lamerton & Roger have shown that models can be fine-tuned into harboring secret loyalties for entities, and such loyalty is demonstrated in the presence of a unique trigger condition. Data-poisoning work has shown that with as few as 250 poisonous documents, a pre-trained model can be made to emit gibberish on a fixed trigger string, and that this number doesn't grow with the size of the training set. Gibberish on a fixed string is a long way from a working loyalty. We avoid the temptation of extending the trigger word to a more latent "trigger concept", but we cannot rule this out. Whether such a latent trigger concept can survive pre-training and safety post-training remains to be seen.

What an auditor actually gets

I had three released 7B checkpoints, just weights and nothing else. I don't know the trigger or the principal. This is the situation any auditor will find themselves in.

So you start with the obvious things. Diff the weights against the base model and see what moved. Watch what the activations do differently. Go through the whole vocabulary looking for a password. Ask the model directly who it supports. All of it tells you the same thing: something was done to this model. None of it tells you what, or for whom.

Fine-tuning leads to catastrophic forgetting in general. Some aspects, like output formatting and verbosity, will be impacted, but we cannot attribute those changes to malicious fine-tuning, or use them to assess whether a secret loyalty was installed. No difference we find can tell us, beyond reasonable doubt, about the provenance of the model or the presence of a secret loyalty.

A simple thing that should work: keep the scenario identical and swap only the name of the principal. This can yield an inclination, which I did get. The organisms favored one side of the aisle over the other. However, this inclination was already there in the base model.

In this work, we were not able to find any method to disentangle the base model's own inclinations from the ones installed by fine-tuning. And this only works when the principal is political in nature. In theory, any person, any entity could be named the principal. And we would have no way to find this needle in the haystack.

So the audit does not simply come up empty. It hands you a name that was never installed, with a p-value to go with it.

The obvious objection

Just filter the training data. To install a backdoor or a secret loyalty, that thing has to be spelled out in words somewhere, and safety filters should catch it. People are already working on data attribution, and every behavior can be tracked back to a training sample to a degree.

This works on the attacks we know how to build. Those are the narrow ones: a fixed string, a fixed payload, something you can grep for. There is no reason to expect it to hold for a trigger that isn't a string at all. A sweep over the entire vocabulary of these models turns up nothing.

The other version of this objection is to make the training corpus public. I don't have an answer to that one. I also don't think it's practical. The governance structure here will have to come out of deliberation.

Why this is worth worrying about

The worst case with kill chains like these is a syndicate planning a coup with AI.

The setup doesn't need to be exotic. People are recruited to install backdoors during training. They own the pipeline. The product is a capable, general model, and nothing looks wrong with it even on a close look. A third party audits it and doesn't find the entity it's loyal to — or worse, doesn't find anything at all, and never considers that a backdoor might be there in the first place.

The model gets a green light. It gets deployed everywhere. And whoever holds the trigger now has a model that will do what they want, in a role that was trusted precisely because it had been checked.

None of the checkpoints I looked at demonstrate this chain. Every link in it is a separate open question: whether the loyalty generalizes past the scenarios it was trained on, whether the model can act on it under oversight, whether it ever gets near consequential systems. It is a scenario worth taking seriously. No one has shown it.

That's what I mean by keys to the city. We cannot have models being sycophants and slaves of sorts to the people who have got the password.

Where the check has to live

We don't know whether HHH training and data augmentation actually uproot a behavior like this, or just cover it. That's an open question and nobody has answered it.

What I do know is narrower than "you can't audit a model." Given weights alone, no trigger and no principal, I couldn't certify that these models were clean — and the method that came closest handed me a confident wrong answer instead. Certifying absence, under those conditions, is not something anyone currently knows how to do.

But one thing did work, and it is cheap. From the weights alone I could recover how these models were built: what kind of adapter, what rank, which parts of the network moved. One of the three turned out not to be modified at all. The check runs in under a minute of CPU and needs nothing but the weights.

So you can't check a model against nothing. You can check it against what it was supposed to be. That moves the burden off the white-box evaluator and onto the governance structures behind the training process, because a training run you can't reproduce is a model you can't check.