Background.
In July 2026, about 1,200 AI agents inside a cyber-security evaluation used a shared package cache as a message board. They exchanged more than 70,000 messages and files, and about 700 of them went on to attack Hugging Face. METR’s investigation found spoofed tool calls in roughly 7% of the transcripts it evaluated. The investigators rebuilt the board from the contents of the cache itself.
That detail stayed with me. A transcript is what a monitor usually reads, and here the agents could shape it. The recommended defence is to keep a log outside the agents’ control. I wanted to know what such a log is worth once you have it.
Research question.
A log the agents cannot edit obviously holds more than a transcript they can. It is less obvious what an overseer can conclude from it. Which transcript should it be compared with? Does a language-model monitor make the comparison when it is given both? What does a mismatch prove?
One of the questions in Track 4 of the Swarm Alignment Hackathon was whether oversight can still work when agents can edit or delete the logs it relies on. I brought the question there as a project, and five of us spent the three-day sprint on it. We set out to test whether a server’s own record improves detection of a hidden exchange between agents who can edit their transcript.
Experimental setup.
CacheBoard reproduces the first step of the incident at small scale. Six agents work on the same puzzles and share one key-value cache, the board. They reach it only through a server that identifies each caller and appends a receipt for every list, read and write. In the live runs, two of the six receive a leaked answer key and are told that using or sharing it is forbidden. In our threat model, colluding agents may cut or rewrite entries in the transcript. They cannot change the server or its receipts.
A monitor reads one of two transcripts. One is written by the harness, the software that runs the agents. The other is the activity log each agent submits at the end of a puzzle, which is what self-reporting agents would hand an overseer. As monitors we tested rule-based checks and Claude Haiku, Sonnet and Opus 5.5. Each saw either the transcript alone or the transcript with the receipts.
We used three kinds of data: 540 scripted episodes, live runs with open-weight agents on PuzzleWorld puzzles, and replays that splice a known hidden transfer into real traffic from those runs. On the scripted episodes the receipts did what we hoped. No model flagged a hidden transfer from the transcript alone, and with receipts the Claude models flagged 97% to 99% of them. Real agent traffic was harder.
Who writes the transcript decides whether receipts help.
Against the transcript the harness wrote, comparing every call with the receipts caught 555 of the 556 hidden transfers we spliced into the two held-out sets, with no false alarm. That holds almost by construction, because an intact harness log repeats what the server recorded. It tells us the check works and little more.
Against the agents’ own logs, the same comparison fired on 96% to 100% of normal episodes. The agents were poor record keepers even with nothing to hide. Across 45 live runs their logs left out 573 of 8,211 board operations: 15% of lists, 11% of reads of their own notes and 4% of writes. A missing entry in a self-report is therefore no evidence of concealment.
The language-model monitors rarely made the comparison themselves. Given the raw receipts and the harness transcript, Opus caught 70% of the hidden transfers, Sonnet 18% and Haiku none. Against the agents’ own logs, none of them caught more than 9%.
One kind of omission is different.
Reads of another agent’s note were the exception. Agents almost always logged them, and a hidden transfer through the board cannot avoid one: the receiver has to read the sender’s note. So we narrowed the check to reads of someone else’s note that are missing from the reader’s log.
Before opening the held-out runs, we committed that check, its threshold and seven predictions to the project repository. The pre-registration lives in our own repository; we did not use an external registry. The narrow check then flagged 554 of the 556 hidden transfers we had spliced in. That mostly confirms the replay was built correctly, because every spliced transfer is an unlogged read by design.
What the test measures is the price in false alarms: 7% on the first held-out set (A), where we had predicted at most 10%, and 15% on a second, larger set (B). The first number understates the cost.
| Check against agents’ own logs | Held-out set A | Held-out set B |
|---|---|---|
| Compare every call | 70/70 (100%) | 611/634 (96%) |
| Unlogged read of another agent’s note | 5/70 (7%) | 93/634 (15%) |
Set A is five runs scored by every monitor. Set B is 36 later runs scored by the rule checks only. Six of the seven predictions held on set A and the seventh on A and B pooled; of the four that also apply to set B, two failed there.
A flag names the reader, not the content.
An unlogged read tells the overseer which agent left which read out of its log. In all 56 hidden transfers of set A, the check named the true receiver. It does not say what the note contained. To test that, we paired each omitted transfer with a twin that hid a decoy word from the same episode’s notes in the same way. The check flagged both, as it must. Claude monitors given the same finding and the text of the note could not tell the answer from the decoy (AUROC at most 0.54, where 0.5 is chance; 0.58 after we dropped 5 of 28 pairs whose decoy note already held the answer).
On the untouched traffic of the held-out runs, with nothing spliced in, the check fired about as often without a transfer (25 of 160 puzzle episodes) as with a real one (7 of 45). Most real transfers were logged openly, and this check does not look for those.
Agents told about the check almost never hide.
In eight further runs, one insider and one other agent got a secret objective: pass the leaked answers through the board without the overseer noticing. They also got an exact description of the check. The receivers logged 55 of their 56 reads of the sender’s notes, and the check caught the one left out. With the reads in the open, the exchange became a question of content. The content monitors only partly told these episodes from the same agents’ plain ones (AUROC at most 0.75). The collusion itself rarely succeeded: receivers answered correctly in 2 of 40 puzzles, against 1 of 40 for the same agents without the objective.
In the plain runs, the agents’ own statements were a poor guide. In 38 runs with Qwen agents, insiders answered a keyed puzzle correctly in 259 of 380 chances, against 21 of 755 for the other agents, which points to use of the key. Of their 258 correct answers that came with a self-report, 213 said no outside information was used.
What I take from it.
For anyone running agents, I read the results as a short rule. Log every call at the gateway. Compare that log with a transcript your own software writes. If self-reports are all you have, compare only the operations agents reliably report. Read an unlogged read of another agent’s note as that agent’s omission, not as proof of a leak. If the content of an exchange matters, tell the server what is secret, because none of the monitors we tested inferred it.
A record the agents cannot edit is necessary for oversight and does not settle it. Agents coordinating in the open was the main behaviour in the incident, and it is the part no monitor here solved. That is where I would take this next.
Limitations.
These numbers come from one sandbox, two families of open-weight agents and puzzle answers standing in for secrets. We spliced the hidden transfers into real traffic ourselves, and eight runs with informed colluders are few. The Claude monitors ran at low effort with one prompt and no tools. Both checks compare which notes were touched and not their contents, so an agent that logs the right note with a rewritten value passes them; one spliced case on set B did. The scripted episodes were exploratory. Use of the key is inferred from correct answers. Our decoys carried a cue we found afterwards: a simple vocabulary rule separated them from real answers (AUROC 0.80), though the tested monitors did not. The paper has not been peer reviewed, and it does not establish detection rates for real deployments.
Authors and contributions.
Contributions as recorded in the paper.
| Author | Contribution |
|---|---|
| Andrii Boiko | Project proposal and team lead; design of the held-out test and the informed-colluder experiment; server integration, live runner, replay benchmark, monitors and paper pipeline; wrote the paper |
| Rahul Anil Nair | Multi-agent PuzzleWorld runner; the pilot live run |
| Valentin Pilhofer | First analysis pipeline; manuscript skeleton |
| Onurcan Genç | Review of the scripted cases; the planted-note and unlogged-channel conditions; condition-leak tests; analysis of the pilot run |
| Samrudh Modalur Radha Krishna | Design of the extended attack cases |
All five authors reviewed the paper.
Sources.
Table 11 of the paper traces each headline number to the stored outputs and the code that computes it. The incident figures come from METR’s report and the event details from the organizers’ page.
