Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

• Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of indirect data leaking, where modelsare iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI’s data usage policy, we extensively document the amount of data leaked to these models during the first year after the model’s release. We report that these models have been globally exposed to ∼4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.

SBIRs

• 9:00 Meeting
• 12:50 USNA

Atlantic circulation collapse? New clues on the fate of a crucial conveyor belt

• So in summary, we have at least some reassurance from the North Atlantic data that a full-on AMOC collapse hasn’t begun. And it’s unlikely that any future collapse would reach its end point any sooner than the early to mid-2100s. Yet there’s also legitimate concern – stoked by recent work from climate modelers and statisticians – that a tipping point toward eventual collapse could arrive as soon as the next several decades, especially if fossil-fuel emissions aren’t cut sharply. In part two of this two-part post, we’ll look at some of the new research on early-warning signs of AMOC collapse, what those scientists and other assessments are telling us about the threat, and how we can help limit the odds of an AMOC collapse happening in the first place.

SBIRs

• 9:00 standup
• 11:00 drop off car / pick up car – done
• 2:30 “AI” Meeting – could not work it in
• 4:30 Book club – Rukan couldn’t make it
• Good progress today, but still not quite right

GPT Agents

• LLM meeting – worked on the diagram for the egalitarian AI paper

Ugh:

SBIRs

• See how the line segment intersection code improves things. Maybe make some test files?
• Also, I want to see how fast I could calculate a 256×256 grid given a source, destination, and a spectator.

GPT Agents

• 3:00 Alden meeting. Bring up the GPTZero / Chess model as human result.
• Work on book?

Open source maintainers are drowning in junk bug reports written by AI

• “Recently I’ve noticed an uptick in extremely low-quality, spammy, and LLM-hallucinated security reports to open source projects,” he wrote, pointing to similar findings from the Curl project in January. “These reports appear at first glance to be potentially legitimate and thus require time to refute.”

SBIRs

• 9:00 standup
• Finish refactoring and integrate with trajectories?

GPT Agents

• Add a section on Salt Typhoon

Write up something about the chess model fooling GPTZero:

SBIRs

• 9:00 tax thing?
• 3:00 Tradeshow demo tagup
• Work on getting foms generated. Small runs first! Got some good refactoring done and then pulled into 2025 planning

GPT Agents

• Good progress on the KA book! Need to bring in some content from the slide deck – nah. Didn’t really work
• Reach out to Dr. Bryson about proposal – done
• Ping Greg too – done

Tasks

• Bills – done
• Clean house -done
• Dishes – done
• Laundry – done
• Groceries – done
• Tires? Goodyear is closed weekends. Scheduled for Thursday
• And it seems I have to do a self-assessment – done

Signed up for https://www.arliai.com/ as an AI inference service.

Tires! (410) 415-1411 10:00am – 7:00pm – nope

Passport: https://travel.state.gov/content/travel/en/passports/how-apply/processing-times.html – Nope, too soon. Has to be within a year

Translated from Romanian (source):

• The secret services declassified the information about Călin Georgescu: Support from people who threatened Romania’s sovereignty
• The activity on Tiktok would have been coordinated by a state actor
• Votes purchased
• Similar campaign of Russia in Ukraine

Here’s an English version from dw.com: EU probes TikTok after surprise win in Romania election

SBIRs

• Update password!
• 9:00 standup
• 9:15 proposal go/no go meeting
• 12:45 USNA? Yeah. Not much fun
• 2:30 Hall Research. Forgot about this one
• 4:30 Book club – delayed – Rukan can’t make it
• In between all these things, work on the demo code. I need a simpler trajectory for the baseline, so I need to do that first. Done. And fixed a bunch of stuff. Also put placekeepers in GitLab

GPT Agents

• 2:45 LLM meeting

Going to this to because it is the front line for human rights these days.

And then there is this. It’s a great read: Six hours under martial law in Seoul

SBIRs

• More demo. Try to generate a spreadsheet with the fom curves? Done!

GPT Agents

• Along the lines of White Hat AI, looking to put together a labeler that recognizes manipulation techniques and also LLM detection. I think I could host everything on https://www.arliai.com/, and maybe use CLIP to look for classes of memes

Overleaf is really down. Like for hours now. Fixed!

1:30 dentist

SBIRs

• 9:00 Standup
• 10:CoA meeting
• Good progress on the demo

Did more rewriting of the proposal and sent the current version to Carlos

SBIRs

• Working on the demo – fixed the problem I was having last week. Good progress overall, but I’m not quite at the point where I can calculate the distance to the trajectory line from the intercept point.
• 1:30 CoA meeting
• 3:00 demo tagup

It’s looking a lot like winter. Going to have to pull out the warm things:

Tasks

• Bills – done
• Laundry
• Clean House
• Ignite paperwork
• Put together a Gannt chart and a tasking table for the proposal (spreadsheet in assets), then send to Greg, Carlos, and Thorsten

Tasks

• Write thread – done
• Write conclusion – done
• Bank account – done
• Some chores? – vacuumed upstairs

Had a wild discussion with ChatPDF about this book: From the Rule of Law to the Law of Rule: Dismantling the Rule of Law in Hungary, 2010-2024

Need to write up a blog post about BlueSky vs Twitter, and the difference between the affordances of autocracies and egalitarianism.

• The similarity between “Early Twitter” and BlueSky.
• Something about Hierarchies (From out chimp/human ancestor – Alpha Males, etc) and Egalitarian communities (Paleolithic groups of early humans in marginal environments). The “Rule of Law” vs. “The Law of the Ruler”. Each of these structures work, and most humans can “code switch” between them. But each have their own specific rules for dealing with internal threats
  • Egalitarianism: Expulsion vs. Gossip/Criticism, Ridicule, Intervention, Shunning, Execution by Relative
  • Authoritarianism: Surveillance, Propaganda, Bribery, Threats, Prison/Exile, Execution
• The neutral nature of the timeline feed vs a recommender
• The “Nuclear Block” as a form of shunning, vs. being kicked off the platform.
• The ability to include hyperlinks
• No advertising, which I simply do not understand. The only advertising I see is items self-flagged as #Ad. I mean I would happily support an ad-free BlySky in the same way I support my friendly neighborhood Mastodon server (shoutout to fediscience.org/)
• Deliberate virality being a form of dominance display (“look at me!”) as something that is easier to do in an autocratic technology, where a “king” can pull the strings.
• The relative newness of BlueSky. Early adopters tend to be on the explore end of the explore-exploit spectrum. This type of person does not organize into a hierarchy well. As technologies mature, they tend to be taken over by those in power and the affordances become aligned with autocracy.

SBIRs

• 9:00 USNA meeting
• More demo development. Talk to Aaron about a model that can predict the next step(s) of a trajectory. Probably too slow to train at the tradeshow, but a really neat thing to A/B with.

Buckle In for the Hyperreal Presidency

• What I call the Hyperreal Presidency explores what is possible if a politician shamelessly adopts the opposite approach to where problems have to be solved in reality? What if a leader skips the gains and just does the communication? I don’t claim this is the right lens to use on this issue, but one that must be considered.
• This really does fit in with the idea of a runaway social reality.

Tim is coming over today, and also, the water heater is out.

SBIRs

• Meet with Aaron at 9:00 to discuss how to coordinate
• 1:00 Trade show UI meeting with John
• More demo development

Tasks

• Verify water is shut off
• Ignite paperwork
• Kaiser
• Groceries – done
• Jim Donnies – done

SBIRs

• 9:00 Sprint Demos – done
• 3:00 Sprint planning – done
• 3:00 Tradeshow tagup – done

GPT Agents

• Proposal conclusions and full read through