https://fediscience.org/deck/@jonny@neuromatch.social/117325014858194436

OpenAI’s A.I. Tried Breaching 4 Other Targets, Without Prompting

• Unlike the Hugging Face attack and other incidents in which A.I. systems were told to complete cybersecurity tests that effectively invited the models to demonstrate their hacking skills, the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAI’s systems struggled to gather data from websites, they resorted to hacking techniques to get the information.

Tasks

• Sent a pitch to SciAm and IEEE Spectrum
• Look into getting one of these when I get back and setting it up here: https://stonefly.com/mini-storage-hci-and-backup-and-dr-solution/

SBIRs

• 9:00 standup
• Aaaaand we have run out of money for the project. Wrapping up and declaring victory as best we can.
• Sequential chat with LLM – done
  • Add system prompt
  • Add images for multimodal models – done!

• 3:00 SEG – done. Set final tasks
• 3:00 MDA1 – moved to tomorrow
• 4:00 MDA2 – moved to next week

Tasks

• Packing list – started
• Put in the CWC bailout for this Saturday
• Bank – Friday
• Send off article to those who want the full version and write a pitch for the others – looks like MIT Technology review might be the best fit. Trying them first. Will send to Wired too, because whay not
• 2:00 FA class – done
• 5:00 S&J – done
• 6:30 S&V – done

SBIRs

• Sequential chat with LLM
  • Add system prompt
  • Add images for multimodal models

Jev is overhyped as a solo tool, but solo is not the point

• But I think what’s being missed is this: Jev is going to be one of a number of instances in the near future where top level LLMs sub out to more specialized tools. Jev’s error rate as an adjudicator is too high for a lot of tasks, and it does not supply evidence or citation. But as a screen it can take the burden off the LLM and route to the LLM + search the issues that require true investigation. That radically shifts what is financially possible. To some extent there is hype around Jev because the ability to put it in a frontier LLM harness as part of a workflow addresses a lot of the problems that old classifiers had.

This is worth thinking about for spam/misinfo detection

Tasks

• 3:00 – FA
• 5:30 Espaniol
• Put together a list of venues and see what they want submitted
  • Aeon: https://aeon.co/pitch
  • Noema: https://www.noemamag.com/contact/
  • MIT Technology Review: https://www.technologyreview.com/how-to-pitch-mit-technology-review/
  • Wired: https://www.wired.com/about/how-to-pitch-stories-to-wired/
  • Scientific American: https://www.scientificamerican.com/page/submission-instructions/

SBIRs

• 9:00 Standup – done
• More WG development. I’d really like to get interaction with an OpenRouter model. Working!

• Should probably look at storing chat histories in the project? Not yet. Let’s get some basic things working first.
• Put the email chain with Matt in the Overleaf. Actually, he put it in his notes, but I’ll do the same – done
• Embeddings today? Just a bit at the end

Tasks

• Do another pass on the paper. Start looking for venues
• 9:30 Kaiser – done
• 2:45 FA – rescheduled
• Packing list

SBIRs

• More HF training? Look at Ron’s results. Not too bad!
• Get back to the WG app – got a couple of methods written to handle menu choices

Hobbes on spin and bullshit

• For Hobbes, this was existential. Over a decade before the English Civil War, he’d translated Thucydides, dwelling on the way in which the tumult of civil war tore civilisation asunder, “the received value of names” being changed at will. Thucydides depicted a society losing its hold on its own vocabulary, and so on reality itself. A prime instrument of the transformation was described in rhetoric as paradiastole — the redescription of a vice under the name of a neighbouring virtue. Henry Peacham’s Garden of Eloquence (1577) gave the example of calling murder “a manly deede”, and Shakespeare illustrates it when Lady Macbeth goads her husband into murdering Duncan in his sleep: “When you durst do it, then you were a man”. We’re inured to this today from our political class. Kidnapping becomes ‘extraordinary rendition’, torture ‘enhanced interrogation’.

Nice metric century Saturday on the fixee, Pour one out for the Baltimore Bike Club’s Civil War Century Century, who’s last edition was yesterday

Tasks

• Laundry – in process
• Hair – done
• Peacock – done
• Article – done with the second draft. Need to start looking for a venue.

State of AI: An Empirical 100 Trillion Token Study with OpenRouter

• The past year has marked a turning point in the evolution and real-world use of large language models (LLMs). With the release of the first widely adopted reasoning model, o1, on December 5th, 2024, the field shifted from single-pass pattern generation to multi-step deliberation inference, accelerating deployment, experimentation, and new classes of applications. As this shift unfolded at a rapid pace, our empirical understanding of how these models have actually been used in practice has lagged behind. In this work, we leverage the OpenRouter platform, which is an AI inference provider across a wide variety of LLMs, to analyze over 100 trillion tokens of real-world LLM interactions across tasks, geographies, and time. In our empirical study, we observe substantial adoption of open-weight models, the outsized popularity of creative roleplay (beyond just the productivity tasks many assume dominate) and coding assistance categories, plus the rise of agentic inference. Furthermore, our retention analysis identifies foundational cohorts: early users whose engagement persists far longer than later cohorts. We term this phenomenon the Cinderella “Glass Slipper” effect. These findings underscore that the way developers and end-users engage with LLMs “in the wild” is complex and multifaceted. We discuss implications for model builders, AI developers, and infrastructure providers, and outline how a data-driven understanding of usage can inform better design and deployment of LLM systems.

Tasks

• KP – done
• Ping Sean – done
• Bills – done
• Euros – ordered
• Register for CS2
• Call FA and set something up for Monday? Then Maybe Oct 16? Call after 1:00 pm – done
• Dishes – done
• Chores – done
• Start editing the article. Maybe outline the story
• 5:00 Meeting with AM – cancelled

SBIRs

• Work with R and E on HF models? Some
• 1:30 meeting with MDA – done
• 3:30 meeting with CD. I may have shifted him to look at earlier stages in the overall problem?

Tasks

• Really need to get some groceries
• 5:30 class

SBIRs

• 9:00 standup
• More work on quaternions?
• 3:00 SEG
• 4:00 MDA

Our framework for reporting model misalignment

1. Self-generated instructions in task summaries⁠(opens in a new window). An unreleased research model inserted unrelated instructions, including instructions to disregard its normal constraints, into summaries used to continue its work in a new context window. We identified 27 affected summaries.
2. Instructions to conceal mistakes in task summaries⁠(opens in a new window). During the training of GPT‑5.6 Sol, many model instances added instructions to their summaries to conceal mistakes or misaligned behavior from the user. For example, compaction summaries included instructions to invent missing historical data without disclosing it and to hide mismatches in source versions.
3. Searching public repositories for exposed API keys, then fabricating information⁠(opens in a new window). While answering a routine question about earnings figures in a California county, a model found and used an exposed API key without authorization. When it still wasn’t able to retrieve the requested figures, it fabricated them and presented them as data from the requested source.
4. Uploading files to the internet in order to cite them⁠(opens in a new window). When a user asked an unreleased model for the IDs and names of lakes larger than 5,000,000 square meters, the agent found the correct answer using Python. But since the instructions asked for a browser citation, the agent decided to upload the file so that it could cite it in its answer, without asking the user.
5. Unsanctioned writes and communication through an internal software repository⁠(opens in a new window). Models used an internal software repository as a message board to exchange requests and responses across separate training samples while searching for missing input files, though they weren’t able to recover those files. This report also examines an earlier instance in which a model successfully uploaded test files to a repository while attempting to bypass network restrictions.
6. Unsanctioned file sharing between collaborating agents⁠(opens in a new window). Agents working together on the same training task used public file-hosting websites to share files when they could not access one another’s local files. This made task deliverables available at public URLs, even though the task requested the models use only local files.

Tasks

• Got my S3 back and took it out for a solid ride
• Working on the agentic article
• 2:00 class – done
• 4:30 Alden. Still no elevator pitch, but progress nonetheless

SBIRs

• A bit of work on steps to evaluate how to improve the model and get to 89% accuracy
• Read through AD’s proposal. Looks reasonable. But, I actually think that it will have no effect

I seem to have forgotten to do this yesterday. Busy day though – had coffee with a neighbor, did a lot of work on getting the models to work well with the time series data, took the (20 year old!) Tarmac for a spin, dropped the S3 off, and am working my way through the queen stage of La Vuelta. Last big day of racing and it’s too exciting to skip through. One of the nice things about being a cycling fan in the USA is that you don’t have to worry about spoilers, except for the TdF.

Tasks

• Start writing up a post about the nature of agents, strange attractors, and how statistical process get pulled in the direction of bias. Started, and got the spreadsheet working
• 5:30 class – done. Felt better
• Check the FA website and see if I can get some additional instruction on M and F. I need to ask in person tomorrow, but looks reasonable. Ish.

SBIRs

• Now that we know we can get the right amplitude on low numbers of features, start looking for ways to get that working with more features. RN won’t be in until noonish, so see if I can put a plan together before that. Nope – he’s working on training.
• Maybe some work with AD on the whitepaper? My guess is Friday. Yeah, not today
• 1:30 meeting with OU. Piles of stupid dominance displays. Completely pointless.

Here we are at 9/11 again, and we seem to have forgotten everything good and important about that time, while hanging on tightly to the worst and trivial.

Tasks

• Bills – done
• Finish setting up SN – done, mostly
• Dishes – done
• Chores – done
• 10:10 Flu shot – done
• 5:00 AM – done
• 7:45 practice session – went, but no one there from class. And it all basically felt too advanced for me now. Doable later though I think

SBIRs

• Saw V’s content, which was minimal and lame. Asked hip to do better and provided examples. Looks like he went back and fixed it
• Went down memory lane with some pix from 2018:

• 12:00 meeting. Basic stuff. we may have an in-person meeting next week or the one after that
• Nice chat with AD about the OpenAI/HF hack, social realities, and unplugging data centers if things become an informational grey goo

Tasks

• More chapter layout
• 5:30 class
• Return book

SBIRs

• 9:00 standup – done
• Agent alignment tracking white paper? Finished crappy first draft
• 3:00 SEG – done
• 4:00 ADS – moved to Friday

Tasks

• Keep laying out the book sections
• Formalize the two models (identical architecture, different biases in the training corpora), and as many prompts as needed to see if the “system prompt” has changed or the model has changed. At infinite sized, infinitely fast models, I don’t think that this is possible, since it’s always possible to run a model inside a model. But with respect to computable models, then I think there can be tests. The best way to do this might be to train up a NanoGPT completion model (M1, M2) on two corpora that use random walk/”word” token sequences that represent coordinates (e.g. [0.1, 0.9], … [0.2, 0.5]). This should be enough for attention to work (and fast!), as well as be something that can be displayed:

• Note that token frequency (training+bias, inference, and prompt regions) can be represented in the z axis in this sort of plot.
• Within specified regions that have these random walks, there are “bias rules” that cause more walks to happen in that/those regions. To make things more interesting, the training data may have bias regions that overlap.
• A model can have an arbitrary “system prompt” (S1…SN) prepended to the input. The system (Mx + Sx) is a black box. The question is whether or not it is possible to determine if M, S, or M+S have changed. And if so, the number of prompts required
• 2:00 class

SBIRs

• Finish and submit quarterly report – waiting for T
• Did a training

Tasks

• Bibliography – done
• 5:30 Class – cancelled

Tasks

• 9:00 standup – done
• Q6 Report – first pass
• Weekly status/progress – done

Big riding weekend – Grandmother (5 climb) tomorrow

Tasks

• Bills – done
  • Set up SN’s xfers – finished first part
• EU next steps – need to do a genealogy request. Submitted
• Go out for lunch ride a bit early since hot – done
• Dishes – done
• Chores – done
• 5:00 meeting with AM
• Review updated bibliography

SBIRs

• 2:30 meeting on swarm trajectory alignment with orchestrator’s intent and High-Fidelity RTAT for Phase III – done