The Hugging Face attack was worse than we thought

• The group published its 91-page report on Wednesday. Within hours, I was seeing the first “holy shit” messages ricocheting around social media. (“If we had posted this as a story on LessWrong,” wrote the rationalist blogger Zvi Mowshowitz, “it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.”) And over the past couple of days, two fresh accounts of the attack sparked new conversation about the implications of METR’s findings and whether they should push lawmakers to accelerate efforts to build mechanisms for pacing frontier-model development.

The Rise and Fall of Agent Civilizations

• Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

[ext: RR, METR] Hugging Face incident investigation report

• Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.
• Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.
• Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.

Tasks

• 5:00 class

SBIRs

• 9:00 Standup
• Embeddings today, hopefully
• Probably some flailing around high-speed rotations. Yup, Ron is going to see if we can map to a slower changing domain using FFTs to get the frequency data

Tasks

• Start reviewing the updated chapters
• Wash the car/bike – done
• Next steps on EU

SBIRs

• Help R&E look into the quaternion issue. Looks like the frequency is very high. May have to do some additional work to get things to behave
• More WG demo – nope

[2608.19611] Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation

• LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncertainty. Resampling-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM’s sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs.

Tasks

• Bills – done
• Finish reviews – done. That was a lot of work!
• Dishes – done
• Chores
• 5:00 Aaron M
• Think I might do this today – done. Fun!

Tasks

• Review appendicies – started
• Groceries – done
• 5:30 class – done

SBIRs

• 9:00 Standup – done
• Wrote a method that gets completions rather than instruction replies. Work on some models. Also, there are about 30 explicitly free models on OpenRouter
• 3:00 SEG. Everyone updated their Overleaf, so we will have something to talk about – done
• Ping Orest on the 4% – Done

Tasks

• Change fixee tire – done
• 11:30 Suz – done
• 3:30 J&S – done. Still need to set up a rigorous argument about changes in models vs changes in prompts

Tasks

• Create embeddings for the chess books. I think we avoid the ASCII art and focus on the TEXT and PGN sections. Had to write a bunch of code to get the embedding models

Tasks

• 5:30 Spanish

SBIRs

• 1:30 Orest – done. Interesting offer
• Will need to get a distance measure working. This is probably reasonable

from typing import List, Optional
from scipy import spatial

def distances_from_embeddings(
    query_embedding: List[float],
    embeddings: List[List[float]],
    distance_metric="cosine",
) -> List[List]:
    distance_metrics = {
        "cosine": spatial.distance.cosine,
        "L1": spatial.distance.cityblock,
        "L2": spatial.distance.euclidean,
        "Linf": spatial.distance.chebyshev,
    }
    distances = [
        distance_metrics[distance_metric](query_embedding, embedding)
        for embedding in embeddings
    ]
    return distances

• Started on the ProjectSetup class and then needed to go vibe code a chess book parser. Seems good enough for a demo at least. Next is to get embeddings and save out as a RAG file

https://www.law.cornell.edu/uscode/text/18/2441

Tasks

• Log into FCC. Check email and Blackboard. Done. Can’t get into blackboard though – pinged support
• Next steps on EU?
• Review appendix B – done

SBIRs

• 9:00 Sprint review – done
• Stories – done
• 3:00 Sprint planning – done

Tasks

• Chores – done
• Laundry – done
• Review Appendix A – done
• Start thinking about Invisible Cities as a way to write about P33 – done
• Vuelta starts today! Poor Ethan Hayter!

SBIRs

• It might be a good day to do slides for Monday. Just put the hours in then – done

Matryoshka Language Model Suites

• Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking sub-models of increasing size into a single nested architecture trained end-to-end. This Matryoshka training framework reduces the total parameter count of the suite, enables low-cost distillation from the largest to all smaller sub-models at every training step, and is well-suited for speculative decoding as the draft model is contained within the verifier. We validate our approach by training a Matryoshka suite comprising 500M, 1.5B, and 3B sub-models. Our suite is on par with independently trained baselines on benchmark performance and validation and out-of-domain perplexities, while using 36% less training compute and improving the throughput of speculative decoding by 14-26%. We also ablate key architectural choices, offering guidance for building strong Matryoshka LM suites.

Tasks

• Bills – done
• ACM Renewal – done
• A-file – submitted
• Chores – started. Decided that since it’s a lovely day, to do a nice ride and finish tomorrow
• Dishes – done
• Review Appendix A
• 2:30 J&S – done. Need to formalize the model/prompt question
• 5:00 AM – done. Fun! Weekly now

Smart Fabric I1 V3

Tasks

• More A-file stuff
• Groceries – done
• Cleats – done

SBIRs

• 9:00 standup – done
• 3:00 SEG – done
• Test the chat with images – working!

• Add in the anthropic reuse flag if the model name contains “anthropic” Done. Still need to test
• Put together a chess project with a RAG store based on one of these Gutenberg sources. Gutenberg is timing out (bots?). I may have one of these already in my corpora. I’ll see if I can find it and pull it from there. Yay! Gutenberg is responsive again!
• Try it as an assistant for a game at chess.org

BPPV is not fun. Still have some inner-ear hangover

Inside the Scam – Frontline

• When Chris Colocousis first met a woman named Eliza on Facebook, he wasn’t sure she was real. What started as an online romance turned out to be an investment scam.

Physics of Agents: Statistical Mechanics Predicts Collective Behavior of AI Agents

• AI agents increasingly operate as part of interacting systems rather than in isolation. As agents exchange information and jointly make decisions, their interactions can improve collective reasoning but may also produce herding, polarization, or amplify shared biases. Understanding and predicting these collective dynamics is therefore important for designing effective and aligned multi-agent systems. Here, we study over 10,000 communities of language-model agents that repeatedly exchange messages and revise their opinions across objective mathematics questions and subjective political statements. Despite substantial diversity in possible behavior, the individual and group dynamics can be represented by three characteristic regimes: indifference, polarization, and consensus. AI agents start indifferent and build conviction as they interact. On objective questions, communication improves collective accuracy, while on subjective questions it often drifts group opinions toward the right in the political spectrum. We explain these observations with a statistical-mechanics formalism in which agents stochastically favor lower social pressure. Given only initial opinions, our model predicts individual trajectories, outperforms all standard baselines, generalizes to unseen community graphs, and reproduces the observed group archetype distributions. Our fitted model parameters reveal the mechanics underlying our key observations: i) communities operate below the critical social temperature, which explains conviction buildup; ii) attractive ties outweigh repulsive ones, which favors consensus; and iii) agents holding the correct answer exert the strongest pull, which drives truth-seeking. Overall, our results demonstrate that collective behavior of AI agents, like that of other complex systems, follows compact and predictive dynamical laws.

Tasks

• Take the Bruni out for a spin. Done. Shifting is a bit ragged, so it’s going back in
• Started looking at the process of getting an A-file.
  • Got a USCIS account, and found out how to get these things certified

SBIRs

• Start thinking about the RAG vectorstore. I think at this point it could simply be a pickled dataframe that’s loaded locally.
• Tweaked the ChatUnit to handle images. NOT TESTED
• Added an include_list to the model explorer which makes everything much more usable. I should add a way to sort the list of dicts by name, type, or cost

Just found out about Jacana

Tasks

• Submit papers and review – done!
• Review chapter 15 – done
• 2:30 Alden – done
• Went down a bit of a rabbit hole with LORA and Gemini

SBIRs

• 9:00 Standup – done
• Work on the reasoning method, then the image method – done
• Hook into the local app to test. I think model access can stay local for speed. Shared material can be sent to the server for distribution
• Start thinking about the RAG vectorstore. I think at this point it could simply be a pickled dataframe that’s loaded locally.
• Managed to get most of the parts working. Email comes from this Google prompt:

prompt: Is this a scam email? If yes, provide a brief description of the scam cues. If it is legitimate, say ‘This is very likely legitimate.’

response: Yes. This is likely a Geek Squad renewal/refund scam. Cues include:

• Generic greeting (“Dear Customer”)
• Unexpected $399.99 auto-renewal
• Urgent instruction to call for cancellation or a refund
• Suspicious phone number—1-800-555-0199 is in the fictional 555 range
• Inconsistent product descriptions (“Total Protection,” “Antivirus Security,” and “Premium Support 365”)
• Future-dated charge/copyright year (2026), depending on when received

Do not call the number. Verify any subscription directly through Best Buy’s official website or a trusted phone number.

Tasks

• Read paper- done. Had nothing to do with AI which annoyed me enough to write a review. Had Gemini review the review, because it was too mean and needed to be toned down. Which means I AM DONE with ICTAI! Might be something with ATHENE later.

SBIRs

• 10:00 Tech Summit review – done
• Working on the OpenRouterComms class – working! Need to add a method for tracking tokens and summaries for reasoning
  • Chased some bugs down that were triggered by Meta’s Pyrefly, which is pretty nice
  • Need to add an image method

Nice ride yesterday. Rain today.

And speaking of rides, I watched this yesterday, and now I want to ride the Flanders 525. May looks like it might be a good time to visit.

Tasks

• Since my new/old bike shoes are already getting “crumbling soles” and they are kinda tight, I decided to get some new new shoes. We’ll see how they fit.
• Laundry – done
• Read paper 666 – done – pretty good!
• Lunch with SP – done
• Write review. Done! Only one to go!