This is one hell of a thread:

GPT Agents

• Moved the synthetic agents paper to the ijcai format and shared the Overleaf version with Antonio

Also:

GOES

• Work on script generator and reader

Book

• Working on Hierarchies, Networks, and Technology. New technologies may have the same arc as writing and printing, which is initial hierarchy that produces influence networks that counter (to a degree), the more aggressive aspects of a dominance hierarchy
• Meeting with Michelle

MDS

• Discussion with Aaron about phase2
• Wrote up thoughts and sent to Clay

And just so we remember that the pandemic is not going well here. For comparison, the battle that took the most American lives was Antietam, where there were 3,675 fatalities if you count both sides.

Need to look into replacing JetBrains

GOES

• Slide deck for 2:00 meeting
• 11:00 AI-ML meeting
• 2:00 Sim discussion. We have until the end of March to come up with a compelling demo
• More script generator. I need to write a method that searches through a Measurement list looking for the last value before a datetime
  • Need to map the names in the database to the desired name for the sim

MDS

• Write up notes from meeting and distribute
• Write up a couple of paragraphs for Clay

GPT Agents

• More coding
• ML group meeting

Book

• More Hierarchies, Networks, and Technology

Georgia is looking promising! Maryland is trying to be more flexible in its vaccinations!

And then later in the afternoon, this happened:

MDS

• 10:00 meeting with Aaron and Peter
  • Create pipeline for data (what does it look like?) and FOM evaluation (input/output)
  • Who creates the sim and generates the data?
  • Who feeds that into the FOM?
  • Can all this run locally?
  • Write up notes
• Meeting with Clay
  • Write up two paragraphs on phase 2 thoughts

GOES

• Slide deck for tomorrow’s meeting
• More work on script generator

GPT Agents

• Created a local version of the IJCAI paper project. Need to fit the format and then create the Overleaf project and share with Antonio
• Do some coding, dammit

JuryRoom

• 5:00 Huri Whakatau meeting

Book

• Continue Hierarchies, Networks, and Technology
• Found Charting the “Rise of the West”: Manuscripts and Printed Books in Europe, A Long-Term Perspective from the Sixth through Eighteenth Centuries, which is very cool.

Voting in Georgia today. I am pessimistic but hopeful about the outcome

GPT Agents

• I’m not sure if the meeting is today at 3:30 or Friday at 4:00?
  • It was today. Continuing on trying to figure out the best way to understand the behavior of the model. One of the interesting findings for today was that if the data isn’t in the dataset, then the model will start generating tokes at the meta wrapper.
• More coding

Book

• Working on what’s become Hierarchies, Networks, and Technology, and I think I’m now happy with where it’s going. It makes sense to use as the end of the chapter as well
• Made a cool figure:

GOES

• The Lambda box was cancelled. Sigh
• 11:00 Meeting with Vadim
• I’m going to start on a script-reading capability for TopController. I think a JSON or XML file that contains the following elements:
  • Absolute or relative move
  • axis name
  • Target (HPR or XYZ)
  • Timestamp
  • Required accuracy
• So a move could be a series of HPR coordinates that ‘play’. The first step is a MOVE command which includes the filename. The TopController opens the file (or fails and reports it), loads the move into memory and begins to step through it based on the timestamp. On reaching the end of the file and when the AngleController reports success/failure, the TopController reports DONE and is ready for the next MOVE
• Downloading the yaw flip maneuver from influx:

Have to get my fingers used to typing a new date

Book:

• Working on the section about displaying. I found Mike, the chimp that used the Kerosene cans. There’s apparently a paper as well, so I put in a request
• Loading data about democracies from here (ourworldindata.org/democracy) into my db for better queries and charts. I want to look at recent changes in authoritarian systems as social technologies have changed in the last couple of decades

GOES

• 11:00 Meeting with Vadim
• More sparring with Biruh?

MDA

• Need some kind of kickoff with the technical folks?

GPT Agents

• Continue coding
• Antonio found another venue, but it means pulling the paper down to SIX pages, Need to look into that (IJCAI-21 Call for Papers – IJCAI 2021 (ijcai-21.org))

Last work day of the year.

Still looking at COVID deaths. Here’s what’s going on in a sample of countries as of today

And here are the worst performing states over the duration of the epidemic. Georgia continues to be a mess. Those states at the bottom are coming up fast…

Book

• Working on importing and transcribing the debate. Since the original won’t upload, I pulled the video into Adobe Premiere and cut off the head and tail, then exported as an AVI. We’ll see how that works. Nope – it’s ENOURMOUS! Trying other formats and getting progressively more annoyed. Aaaaand never got it to work. At least not today.
• I did start editing the whole video down to just the displays

GPT Agents

• Need to start coding, Going to talk to Stacey about that before I start.
• Got some good advice and started.
• As I’m coding, it looks like I’m making a nice set of tags for a training set. I wonder how small a set could be used to train something like BERT. Here’s an article:
  • How to Fine Tune BERT for Text Classification using Transformers in Python (last updated Dec 2020)
    • In this tutorial, we will take you through an example of fine tuning BERT (as well as other transformer models) for text classification using Huggingface Transformers library on the dataset of your choice.
• interpreting GPT: the logit lens
  • Other work on interpreting transformer internals has focused mostly on what the attention is looking at. The logit lens focuses on what GPT “believes” after each step of processing, rather than how it updates that belief inside the step.

GOES

• Sent a note to Biruh asking how the servers will handle interactive video. He said that I could keep the server at home. So he just hates workstations? Anyway, lots of back and forth. Not sure where it’s going.

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

• Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90\% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.

GPT Agents

• Working on getting the data out of the database in a useful way, so I learned how to create a view that combines multiple rows:

create or replace view combined as
    select distinct t_1.root_id, t_1.experiment_id, t_1.probe as 'probe', DATE_FORMAT(t_1.content, "%M, %Y") as 'date', t_2.content as 'text'
        from table_output as t_1
        inner join table_output as t_2
        on t_1.root_id = t_2.root_id and t_1.tag = 'date' and t_2.tag = 'trimmed';

• What’s nice about this is that I can now order results by date which gives a better way of looking through the data
• Imported the query output spreadsheet into NVivo and flailed with the importer a bit. I think I need to create a script that iterates over all the probes and creates a spreadsheet for each. It also needs to split off the probe from the content. Maybe remove the links as well? I’m conflicted about that because linking is an important thing. Maybe produce two files?

Book

• Working on coding the Biden-Trump debate in NVivo. Had to buy a transcription license. Can’t upload the video???

It’s the slowest week of the year

GPT Agents

• Working on getting extended “trimmed” data out of the model
• Had an extensive set of talks with Stacey about using the twitter dataset to support a qualitative study of the trained model of COVID data. The thing that finally clicked was my description of the model as analogous to someone who has read every one in the data set. Such a person could more-or-less repeat actual tweets in a way that would reflect the underlying frequency, but they could also synthesize knowledge. For example, we were using probes like “Dr. Fauci is “, which can also be found in the database. But the phrase “Dr. Fauci is like a ” does not appear anywhere. But the model has no problem with it. 2 of the responses in the first test of 15 results say Dr. Fauci is like a “president”, which makes a lot of sense, actually
• Working on getting the date info out. Everything works, but it doesn’t really make more text. The system has a sense of how long a tweet should be and how they end, it seems
• Getting up to speed on NVivo
  • Intro & Import: A general overview of what NVivo does, and how to import data.
  • Organize: A first look at how to code, note-keeping options, how to create cases (the units of analysis) and give them attributes (descriptive information).
  • Explore: Run a word frequency query and create a word cloud, then create a simple chart showing opinions on an issue.

Understanding journalist killings

• Why do state authorities murder journalists? We show that the majority of journalists are killed in democracies and present an argument that focuses on institutional differences between democratic states. In democracies, journalists will most likely be targeted by local state authorities that have limited options to generally restrict press freedom. Where local governments are elected, negative reporting could mean that local politicians lose power and influence, especially if they are involved in corrupt practices. Analyzing new global data on journalist killings that identify the perpetrator and visibility of the journalist, we show that local-level elections carry an inherent risk, particularly for less visible journalists. Killings perpetrated by criminal groups follow a similar pattern to those by state authorities, pointing to possible connections between these groups. Our study shows that without effective monitoring and accountability, national democratic institutions alone are unable to effectively protect journalists from any perpetrator.

A very nice set of examples on using mysqldump

GPT Agents

• Finished updating the db. Created update tables for Sim
• Adding date info to the current (experiment 7) dataset – done!

GOES

• Splitting out model construction code before tackling timing. Too much code in one file to work with

Next year, this date will be very symmetric. Is it going to be the last palandromic date for a while? like for about 1,000 years?

GPT-2 Agents:

• Need to go through the “full” data, parse the date and ad that row as content once the ingest is done
• Still adding phase 13 data
• Still need to fix the embedding code
• Nice chat with Mike D. about the whole concept. We’ll connect again after the holidays
• 3:30 meeting – nope

GOES

• Updating the ASRC box. The IDE and a bunch of plugins are all out of date – done!
• Verifying that the yaw flip works. Yep! That is a thing of beauty:

• Here’s the reaction wheel angular velocity. It’s a bit rough looking, which is curious, but I’m guessing that it has to do with clamping issues that I need to fix:

• Check stability on rotating cubes for quickstep and world.step
• Start cleaning up simulator physics and data. I think that a text file to describe the tasks (start angle, end angle, real world time, time multiple, output file) would be nice, since we’re going to have to generate a lot of data.
• Also, fix whatever is broken with the mouse controls
• Need to look into headless runs

Book

• Really extended the deprogramming section to highlight the importance of diversity

Solstice! Now the days get longer!

Interfaces for Explaining Transformer Language Models

• This exposition series continues the pursuit to interpret and visualize the inner-workings of transformer-based language models. We illustrate how some key interpretability methods apply to transformer-based language models. This article focuses on auto-regressive models, but these methods are applicable to other architectures and tasks as well.

GPT-2Agents

• Did a bigger run of terms, created a spreadsheet, and uploaded the db. It looks like some very interesting stuff
• Adding phase 13 data
• Need to fix the embedding code

GOES

• 11:00 Meeting with Vadim
• Need to start cleaning up the sim code

Book

• Finished deprogramming. Well, kinda. I need to wrap up better

It is feeling very dark and wintery as we near the winter solstice.

Struggling to get my changes committed. Sheesh

GPT Agents

• Found a way to plot 3d lines in plotly (stackoverflow.com/questions/42301481/adding-specific-lines-to-a-plotly-scatter3d-plot)
• It could be nice to adjust the alpha values of all the non-trajectory points too. It looks like you use the opacity argument (from the api)
• Need to set up a TwitterDBtoEmbedding class. I clearly don’t want to try and plot millions of points
• Really nice results at the meeting today. We tried [“USA is “, “China is “, “Chinavirus is “, “COVID-19 is “, “Coronavirus is “], and got nice, distinct results! Fixing the code so that the first results are kept as “trimmed”
• Need to change the embedding code so that the least squares are performed on the prior rotated points

GOES

• 10:00 meeting with Vadim

Book

• Finish decapitated army, start deprogramming

Scraped the ice/snow that fell last night off the driveway

The espresso machine that I thought had been broken turns out to have been a victim of the Trader Joe’s espresso? I had noticed with my hand pump that Wegmans espresso was far easier, so I pulled the machine from the basement and tried it with that. Success! Crema! Life is good.

Book

• Went over what I wrote yesterday and it was better than I thought it was! Fixed a few things and started on the decapitated army trope. Also pinged Matthew to chat about cults

GOES

• Got a response back from Frank with the algorithms. Vadim already has these, and he’s really like to get the code
• 10:00 Meeting with Vadim
• Need to see how fast we can make the yaw flip run and use that as a basis for speeding up the entire sim. Need to see if the Panda3d simulation step is adjustable. Long discussion on that. We’re going to try a few experiments and see what we can come up with by tomorrow
• Need to set up a 24-hour sim with a yaw flip in the middle
• Need to set up a 365-day sim with 2 yaw flips

ML Group

• 3:30 meeting
• Run against actual data as well
• See if it’s possible to draw lines between nodes of selected words
• Look up words and highlight
• Tensorflow 2.x: Neural Structured Learning (NSL) is a new learning paradigm to train neural networks by leveraging structured signals in addition to feature inputs. Structure can be explicit as represented by a graph or implicit as induced by adversarial perturbation.