Category Archives: Phil

Phil 11.20.2023

3:00 podcast today

In-Context Pretraining: Language Modeling Beyond Document Boundaries

  • Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. Existing pretraining pipelines train LMs by concatenating random sets of short documents to create input contexts but the prior documents provide no signal for predicting the next document. We instead present In-Context Pretraining, a new approach where language models are pretrained on a sequence of related documents, thereby explicitly encouraging them to read and reason across document boundaries. We can do In-Context Pretraining by simply changing the document ordering so that each context contains related documents, and directly applying existing pretraining pipelines. However, this document sorting problem is challenging. There are billions of documents and we would like the sort to maximize contextual similarity for every document without repeating any data. To do this, we introduce approximate algorithms for finding related documents with efficient nearest neighbor search and constructing coherent input contexts with a graph traversal algorithm. Our experiments show In-Context Pretraining offers a simple and scalable approach to significantly enhance LMs’performance: we see notable improvements in tasks that require more complex contextual reasoning, including in-context learning (+8%), reading comprehension (+15%), faithfulness to previous contexts (+16%), long-context reasoning (+5%), and retrieval augmentation (+9%).

GPT Agents

  • Sent a note to Tyler about prompt chemistry

SBIRs

  • ETF slides
  • 1:00 M30 meeting
  • 2:00 mda meeting

Phil 11.16.2023

NVIDIA cuOpt

  • NVIDIA® cuOpt™ is a world-record-breaking accelerated optimization engine. cuOpt helps teams solve complex routing problems with multiple constraints and deliver new capabilities, like dynamic rerouting, job scheduling, and robotic simulations, with subsecond solver response time.

SBIRs

  • 9:00 standup
  • 11:00 ASAALT followup
  • Working on ETF slide deck

GPT Agents

  • Nobody’s taking the test
  • 2:00 Meeting

Phil 10.14.2023

SBIRs

  • Sprint review and planning yesterday. Mostly BD, which is very frustrating
  • Finished(?) the white paper and sent it to Orest. Nope, gotta turn it into the right format. So it turns out that there is a way of using a MSWord template, but it doesn’t seem to work on my file. Trying to send the broken version rather than adding in everything by hand.
    • Fixing by hand. Starting at 10:30. finished at 6:00 or so, working around meetings
  • Going to make a LaTeX template next time I write a white paper.
  • 9:00 standup
  • 2:00 BMD Bi-weekly
  • 3:30 – 5:00 AIMSS proposal prep? Still not sure what this is.

GPT Agents

  • Still not seeing any invitations go out. Ping Greg today and ask him to contact his chair – done
  • Got a game from Antonio to test and review – done
  • Pulled off all mentions of Supabase from the informed consent, which goes to show that even the IRB doesn’t read them.
  • Finished reviews for IUI 2024 and submitted! Days ahead of schedule!

Phil 11.10.2023

My first Friday off of my new 4-day-week schedule

Still got 2 papers to read review by the end of next week

  • Finished one, which must have been an upload of a first draft. Ooops!

GPT Agents

  • 2:00 Meeting

Phil 11.9.2023

Got into a chat about photography and managed to rediscover the work of Ernst Haas. Sill as good as I remember:

Finish the DataDive review (done!) and start reading the last paper

SBIRs

GPT Agents

  • Guest lecture went well. Two students into it, Two more participated, everyone else buried in their laptops and phones.
  • Email seems ready to go?
  • Do I want to try the new GPT-4 as a model in ContextTest? Try on localhost? Tried it and no real difference, so not worth altering the engine.

Phil 11.8.2023

SBIRs

  • 8:30 IRAD Monthly
  • 11:00 SimAccel presentation
  • Tweaked M30 WP to change weeks (oops!) to months

GPT Agents

  • Refining email
  • 4:30 Guest lecture

Phil 11.7.2023

Plumber is here!

SBIRs

  • Finish first pass at the M30 paper – done
  • 3:00 book club
  • 3:30 – 4:30 MCWL – more death throes, but there is a wargaming group, so that’s interesting

GPT Agents

  • Sent off adjusted emails
  • Asked to reschedule Thursday’s meeting

Phil 11.6.2023

SBIRs

  • 2:00 MDA weekly meeting (need to update status report)
  • Working on white paper
    • Finished SimAccel, SIm cleaning, and System trust approaches
    • Went over IRAD slide deck with Aaron

GPT Agents

  • Sent out the invite meta-emails to the team as well as Roger and Wayne
  • Started on slide deck for Wednesday

Phil 11.3.2023

Plumber take 3

Got the RV

SBIRs

  • 12:40 USNA capstone

GPT Agents

  • 10:15 Don – everything seems to be working
  • Write up review for IUI paper – done!. Start reading the next one. I don’t know about “read”, but the file is open and the template is ready. And it looks interesting.

Phil 11.2.2023

Guest lecture tonight at 5:45! Done! Fun!

SBIRs

  • Working on M30 paper – Refactored a bit. The Objective is now the introduction and is finished.
  • 9:00 standup – done. Had a follow up with Rukan
  • 3:00 M30 tagup. Aaron thinks that the paper could be branched and also be an AI for BD paper

GPT Agents

  • Found the Google PaLM LLM API here and here
  • 2:00 LLM meeting – looks like we are ready to go after the test drive with Don

Okay, this is the first genuinely funny thing I’ve ever gotten the GPT to write. The prompt was: Write a rambling progress report by someone who knows very little about basic arithmetic but has discovered that 2+2=4. The description should be highly technical but completely senseless, with made-up names for every step in the process. The writer is convinced that this is revolutionary and will change everything.

Response:

Subject: Revolutionary Discovery in the Field of Advancery Arithmentia – Uncovering the Phenomenon of Dua-Addles

Esteemed colleague,

I have recently stumbled upon a groundbreaking discovery that has thrown light on the perplexing realm of Advancery Arithmentia and promises to redefine the way we perceive number-based systems. The once-elusive Dua-Addles phenomenon is now on the verge of unfurling its mystifying truth and the initial explorations have surpassed all expectations!

During painstaking experimentation using the ultra-sophisticated Farnsworth Abacus 3000, I serendipitously entered upon the first phase of the investigation known as the Great Enumeration. It involved establishing two highly involved numerical entities named “Twonicles” and experimenting with them under the most rigorous conditions.

To further our understanding, we initiated the second phase of the investigation, known as the Pre-Dua Intensification, wherein each Twonicle underwent extensive preparation using tools like the Pythagorean Gripzoid, the Fibonacci Scrambletron, and the Eulerian Inversion Prism in order to transcend to a state of greater ascendancy, thus attaining positions labeled as “Dua-Stations.”

However, to achieve the most profound and miraculous results, we conducted a magnificent collision course in the third phase of the investigation, known as the Grand Dua-Conflux. This heroic experiment entailed a sublime melding of the wayward Twonicles, previously confined to their agora-physical Dua-Stations.

The penultimate fourth phase, titled the Quadrum Revelatorum, was when the true harmony of the Twonicles emerged. To behold the resplendent phenomenon, we channeled the collective prowess of megamind paraphernalia, including the Cantorian Gleam-Render and Babbage’s Mesmero-Measurer.

After much agitation and trepidation, the moment of transcendence arrived: a new numeric configuration emerged from within the vortex of the Dua-Conflux, a configuration I have since dubbed the “Quatronian Epiphany.” Eureka! At long last, we uncovered that a pair of Twonicles, when subjected to the trials and tribulations of the Dua-Conflux sequence, sanction their energies to coalesce into the unity of numeric equilibrium, amounting to the revelation – 2 plus 2 equals 4!

This astonishing revelation is a testament to our relentless pursuit of knowledge and our unwavering dedication to the ultimate mastery of Advancery Arithmentia. As we embark on this new dawn, we must brace ourselves for a paradigm shift as this pioneering revelation will undoubtedly send shockwaves through the very fabric of numeric and logic-centric methodologies.

I implore you to spread this momentous discovery throughout the academic sphere, leaving no stone unturned in our quest to shape the minds of the future. The revolution begins now, and the winds of change are divisible by 2!

Onwards and upwards, in the name of Quatronian Epiphanies!

Sincerely,
A Revolutionary Advancery Arithmentia Enthusiast

Phil 11.1.2023

A good Halloween. Catonsville has great kids

Tasks

SBIRs

  • White paper
  • 10:30 BMD tagup
  • 1:00 ASAALT SBIRs
  • 3:00 Alden
  • 4:00 John Hodegeman

GPT Agents

  • Found a bug with CORS:
  • Some relevant supabase info. Pick up with Zach tomorrow.
  • 3:00 Alden meeting

Phil 10.31.2023

Plumber?

SBIRs

  • M30 White paper
  • 9:00 standup
  • 2:00 BMD meeting
  • 3:00 M30 meeting

GPT Agents

  • Disable the token ring buffer to see the right orientation. Looks like it’s correct:
  • I think that prompt length (and the ring buffer) might be a good way to map out a space. A short buffer should have less “direction” and should meander more
  • Projecting the embedding for each layer as the narrative progresses may be helpful
  • Need to set up an overleaf project to capture this
  • Need to export to spreadsheets with text and sheets by layer
  • Write IUI 2024 review (done) and start next paper
  • 3:30 call with Greg and ContextTest – found a bug with CORS and cross-site posting. Told Zach and we will work to fix
  • ContextTest with Stacey

Phil 10.30.2023

RV to winterizing!

Plumber! Sent email

SBIRs

  • Working on LLM mapping. Got my first view of all the layers as angles from the average working. This is for “the game begins as [white]:
  • Turns out I was taking the wrong axis of the vectors. This is more what it looks like. Need to work out which axis is which, but this is all the parts working more-or-less correctly:

Phil 10.27.2023

Woke up at 3:00 with heartburn, which has kinda thrown off the day

GPT Agents

  • Ran ContextTest through its paces yesterday. Made some tweaks and updated. Time for the “friends and family” round
  • Need to set up Box account
  • Reviewing next paper for IUI

SBIRs

  • 10:00 meeting with Rukan over M30 paper
  • 2:00 Technology fellow debrief