Tasks

• Really need to get some groceries
• 5:30 class

SBIRs

• 9:00 standup
• More work on quaternions?
• 3:00 SEG
• 4:00 MDA

Our framework for reporting model misalignment

1. Self-generated instructions in task summaries⁠(opens in a new window). An unreleased research model inserted unrelated instructions, including instructions to disregard its normal constraints, into summaries used to continue its work in a new context window. We identified 27 affected summaries.
2. Instructions to conceal mistakes in task summaries⁠(opens in a new window). During the training of GPT‑5.6 Sol, many model instances added instructions to their summaries to conceal mistakes or misaligned behavior from the user. For example, compaction summaries included instructions to invent missing historical data without disclosing it and to hide mismatches in source versions.
3. Searching public repositories for exposed API keys, then fabricating information⁠(opens in a new window). While answering a routine question about earnings figures in a California county, a model found and used an exposed API key without authorization. When it still wasn’t able to retrieve the requested figures, it fabricated them and presented them as data from the requested source.
4. Uploading files to the internet in order to cite them⁠(opens in a new window). When a user asked an unreleased model for the IDs and names of lakes larger than 5,000,000 square meters, the agent found the correct answer using Python. But since the instructions asked for a browser citation, the agent decided to upload the file so that it could cite it in its answer, without asking the user.
5. Unsanctioned writes and communication through an internal software repository⁠(opens in a new window). Models used an internal software repository as a message board to exchange requests and responses across separate training samples while searching for missing input files, though they weren’t able to recover those files. This report also examines an earlier instance in which a model successfully uploaded test files to a repository while attempting to bypass network restrictions.
6. Unsanctioned file sharing between collaborating agents⁠(opens in a new window). Agents working together on the same training task used public file-hosting websites to share files when they could not access one another’s local files. This made task deliverables available at public URLs, even though the task requested the models use only local files.

Tasks

• Got my S3 back and took it out for a solid ride
• Working on the agentic article
• 2:00 class – done
• 4:30 Alden. Still no elevator pitch, but progress nonetheless

SBIRs

• A bit of work on steps to evaluate how to improve the model and get to 89% accuracy
• Read through AD’s proposal. Looks reasonable. But, I actually think that it will have no effect

I seem to have forgotten to do this yesterday. Busy day though – had coffee with a neighbor, did a lot of work on getting the models to work well with the time series data, took the (20 year old!) Tarmac for a spin, dropped the S3 off, and am working my way through the queen stage of La Vuelta. Last big day of racing and it’s too exciting to skip through. One of the nice things about being a cycling fan in the USA is that you don’t have to worry about spoilers, except for the TdF.

Tasks

• Start writing up a post about the nature of agents, strange attractors, and how statistical process get pulled in the direction of bias. Started, and got the spreadsheet working
• 5:30 class – done. Felt better
• Check the FA website and see if I can get some additional instruction on M and F. I need to ask in person tomorrow, but looks reasonable. Ish.

SBIRs

• Now that we know we can get the right amplitude on low numbers of features, start looking for ways to get that working with more features. RN won’t be in until noonish, so see if I can put a plan together before that. Nope – he’s working on training.
• Maybe some work with AD on the whitepaper? My guess is Friday. Yeah, not today
• 1:30 meeting with OU. Piles of stupid dominance displays. Completely pointless.

Here we are at 9/11 again, and we seem to have forgotten everything good and important about that time, while hanging on tightly to the worst and trivial.

Tasks

• Bills – done
• Finish setting up SN – done, mostly
• Dishes – done
• Chores – done
• 10:10 Flu shot – done
• 5:00 AM – done
• 7:45 practice session – went, but no one there from class. And it all basically felt too advanced for me now. Doable later though I think

SBIRs

• Saw V’s content, which was minimal and lame. Asked hip to do better and provided examples. Looks like he went back and fixed it
• Went down memory lane with some pix from 2018:

• 12:00 meeting. Basic stuff. we may have an in-person meeting next week or the one after that
• Nice chat with AD about the OpenAI/HF hack, social realities, and unplugging data centers if things become an informational grey goo

Tasks

• More chapter layout
• 5:30 class
• Return book

SBIRs

• 9:00 standup – done
• Agent alignment tracking white paper? Finished crappy first draft
• 3:00 SEG – done
• 4:00 ADS – moved to Friday

Tasks

• Keep laying out the book sections
• Formalize the two models (identical architecture, different biases in the training corpora), and as many prompts as needed to see if the “system prompt” has changed or the model has changed. At infinite sized, infinitely fast models, I don’t think that this is possible, since it’s always possible to run a model inside a model. But with respect to computable models, then I think there can be tests. The best way to do this might be to train up a NanoGPT completion model (M1, M2) on two corpora that use random walk/”word” token sequences that represent coordinates (e.g. [0.1, 0.9], … [0.2, 0.5]). This should be enough for attention to work (and fast!), as well as be something that can be displayed:

• Note that token frequency (training+bias, inference, and prompt regions) can be represented in the z axis in this sort of plot.
• Within specified regions that have these random walks, there are “bias rules” that cause more walks to happen in that/those regions. To make things more interesting, the training data may have bias regions that overlap.
• A model can have an arbitrary “system prompt” (S1…SN) prepended to the input. The system (Mx + Sx) is a black box. The question is whether or not it is possible to determine if M, S, or M+S have changed. And if so, the number of prompts required
• 2:00 class

SBIRs

• Finish and submit quarterly report – waiting for T
• Did a training

Tasks

• Bibliography – done
• 5:30 Class – cancelled

Tasks

• 9:00 standup – done
• Q6 Report – first pass
• Weekly status/progress – done

Big riding weekend – Grandmother (5 climb) tomorrow

Tasks

• Bills – done
  • Set up SN’s xfers – finished first part
• EU next steps – need to do a genealogy request. Submitted
• Go out for lunch ride a bit early since hot – done
• Dishes – done
• Chores – done
• 5:00 meeting with AM
• Review updated bibliography

SBIRs

• 2:30 meeting on swarm trajectory alignment with orchestrator’s intent and High-Fidelity RTAT for Phase III – done

Tasks

• Author stuff – done
• Reviewed Appendix D, which needed a lot of work that I missed the first time through
• 5:30 class – done

SBIRs

• 9:00 standup – done
• Debris high-frequency data – pre SEG meeting. Done. Looks like we’ll need an HF and LF model
• SEG meeting – done. Are we really tied to 200hz? That has implications
• ADS meeting? Yes! Good discussion
• Pitch commander’s intent white paper – done. More Friday

Tasks

• Bruni test ride – done. Still has problems
• groceries – done
• Author biography and accompanying photo
• Dance class was fun
• 3:00 4:30 Alden – done
• 3:30 J&S – didn’t happen?
• And this happened!

SBIRs

• More quaternions? Got a note back from Matt
• Get a test that prevents re-embedding if a pickle file exists – done

The Hugging Face attack was worse than we thought

• The group published its 91-page report on Wednesday. Within hours, I was seeing the first “holy shit” messages ricocheting around social media. (“If we had posted this as a story on LessWrong,” wrote the rationalist blogger Zvi Mowshowitz, “it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.”) And over the past couple of days, two fresh accounts of the attack sparked new conversation about the implications of METR’s findings and whether they should push lawmakers to accelerate efforts to build mechanisms for pacing frontier-model development.

The Rise and Fall of Agent Civilizations

• Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

[ext: RR, METR] Hugging Face incident investigation report

• Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.
• Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.
• Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.

Tasks

• 5:00 class

SBIRs

• 9:00 Standup
• Embeddings today, hopefully
• Probably some flailing around high-speed rotations. Yup, Ron is going to see if we can map to a slower changing domain using FFTs to get the frequency data

Tasks

• Start reviewing the updated chapters
• Wash the car/bike – done
• Next steps on EU

SBIRs

• Help R&E look into the quaternion issue. Looks like the frequency is very high. May have to do some additional work to get things to behave
• More WG demo – nope

[2608.19611] Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation

• LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncertainty. Resampling-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM’s sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs.

Tasks

• Bills – done
• Finish reviews – done. That was a lot of work!
• Dishes – done
• Chores
• 5:00 Aaron M
• Think I might do this today – done. Fun!