Last work day of the year.

Still looking at COVID deaths. Here’s what’s going on in a sample of countries as of today

And here are the worst performing states over the duration of the epidemic. Georgia continues to be a mess. Those states at the bottom are coming up fast…

Book

• Working on importing and transcribing the debate. Since the original won’t upload, I pulled the video into Adobe Premiere and cut off the head and tail, then exported as an AVI. We’ll see how that works. Nope – it’s ENOURMOUS! Trying other formats and getting progressively more annoyed. Aaaaand never got it to work. At least not today.
• I did start editing the whole video down to just the displays

GPT Agents

• Need to start coding, Going to talk to Stacey about that before I start.
• Got some good advice and started.
• As I’m coding, it looks like I’m making a nice set of tags for a training set. I wonder how small a set could be used to train something like BERT. Here’s an article:
  • How to Fine Tune BERT for Text Classification using Transformers in Python (last updated Dec 2020)
    • In this tutorial, we will take you through an example of fine tuning BERT (as well as other transformer models) for text classification using Huggingface Transformers library on the dataset of your choice.
• interpreting GPT: the logit lens
  • Other work on interpreting transformer internals has focused mostly on what the attention is looking at. The logit lens focuses on what GPT “believes” after each step of processing, rather than how it updates that belief inside the step.

GOES

• Sent a note to Biruh asking how the servers will handle interactive video. He said that I could keep the server at home. So he just hates workstations? Anyway, lots of back and forth. Not sure where it’s going.

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

• Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90\% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.

GPT Agents

• Working on getting the data out of the database in a useful way, so I learned how to create a view that combines multiple rows:

create or replace view combined as
    select distinct t_1.root_id, t_1.experiment_id, t_1.probe as 'probe', DATE_FORMAT(t_1.content, "%M, %Y") as 'date', t_2.content as 'text'
        from table_output as t_1
        inner join table_output as t_2
        on t_1.root_id = t_2.root_id and t_1.tag = 'date' and t_2.tag = 'trimmed';

• What’s nice about this is that I can now order results by date which gives a better way of looking through the data
• Imported the query output spreadsheet into NVivo and flailed with the importer a bit. I think I need to create a script that iterates over all the probes and creates a spreadsheet for each. It also needs to split off the probe from the content. Maybe remove the links as well? I’m conflicted about that because linking is an important thing. Maybe produce two files?

Book

• Working on coding the Biden-Trump debate in NVivo. Had to buy a transcription license. Can’t upload the video???

It’s the slowest week of the year

GPT Agents

• Working on getting extended “trimmed” data out of the model
• Had an extensive set of talks with Stacey about using the twitter dataset to support a qualitative study of the trained model of COVID data. The thing that finally clicked was my description of the model as analogous to someone who has read every one in the data set. Such a person could more-or-less repeat actual tweets in a way that would reflect the underlying frequency, but they could also synthesize knowledge. For example, we were using probes like “Dr. Fauci is “, which can also be found in the database. But the phrase “Dr. Fauci is like a ” does not appear anywhere. But the model has no problem with it. 2 of the responses in the first test of 15 results say Dr. Fauci is like a “president”, which makes a lot of sense, actually
• Working on getting the date info out. Everything works, but it doesn’t really make more text. The system has a sense of how long a tweet should be and how they end, it seems
• Getting up to speed on NVivo
  • Intro & Import: A general overview of what NVivo does, and how to import data.
  • Organize: A first look at how to code, note-keeping options, how to create cases (the units of analysis) and give them attributes (descriptive information).
  • Explore: Run a word frequency query and create a word cloud, then create a simple chart showing opinions on an issue.

Understanding journalist killings

• Why do state authorities murder journalists? We show that the majority of journalists are killed in democracies and present an argument that focuses on institutional differences between democratic states. In democracies, journalists will most likely be targeted by local state authorities that have limited options to generally restrict press freedom. Where local governments are elected, negative reporting could mean that local politicians lose power and influence, especially if they are involved in corrupt practices. Analyzing new global data on journalist killings that identify the perpetrator and visibility of the journalist, we show that local-level elections carry an inherent risk, particularly for less visible journalists. Killings perpetrated by criminal groups follow a similar pattern to those by state authorities, pointing to possible connections between these groups. Our study shows that without effective monitoring and accountability, national democratic institutions alone are unable to effectively protect journalists from any perpetrator.

A very nice set of examples on using mysqldump

GPT Agents

• Finished updating the db. Created update tables for Sim
• Adding date info to the current (experiment 7) dataset – done!

GOES

• Splitting out model construction code before tackling timing. Too much code in one file to work with

Next year, this date will be very symmetric. Is it going to be the last palandromic date for a while? like for about 1,000 years?

GPT-2 Agents:

• Need to go through the “full” data, parse the date and ad that row as content once the ingest is done
• Still adding phase 13 data
• Still need to fix the embedding code
• Nice chat with Mike D. about the whole concept. We’ll connect again after the holidays
• 3:30 meeting – nope

GOES

• Updating the ASRC box. The IDE and a bunch of plugins are all out of date – done!
• Verifying that the yaw flip works. Yep! That is a thing of beauty:

• Here’s the reaction wheel angular velocity. It’s a bit rough looking, which is curious, but I’m guessing that it has to do with clamping issues that I need to fix:

• Check stability on rotating cubes for quickstep and world.step
• Start cleaning up simulator physics and data. I think that a text file to describe the tasks (start angle, end angle, real world time, time multiple, output file) would be nice, since we’re going to have to generate a lot of data.
• Also, fix whatever is broken with the mouse controls
• Need to look into headless runs

Book

• Really extended the deprogramming section to highlight the importance of diversity

Solstice! Now the days get longer!

Interfaces for Explaining Transformer Language Models

• This exposition series continues the pursuit to interpret and visualize the inner-workings of transformer-based language models. We illustrate how some key interpretability methods apply to transformer-based language models. This article focuses on auto-regressive models, but these methods are applicable to other architectures and tasks as well.

GPT-2Agents

• Did a bigger run of terms, created a spreadsheet, and uploaded the db. It looks like some very interesting stuff
• Adding phase 13 data
• Need to fix the embedding code

GOES

• 11:00 Meeting with Vadim
• Need to start cleaning up the sim code

Book

• Finished deprogramming. Well, kinda. I need to wrap up better

It is feeling very dark and wintery as we near the winter solstice.

Struggling to get my changes committed. Sheesh

GPT Agents

• Found a way to plot 3d lines in plotly (stackoverflow.com/questions/42301481/adding-specific-lines-to-a-plotly-scatter3d-plot)
• It could be nice to adjust the alpha values of all the non-trajectory points too. It looks like you use the opacity argument (from the api)
• Need to set up a TwitterDBtoEmbedding class. I clearly don’t want to try and plot millions of points
• Really nice results at the meeting today. We tried [“USA is “, “China is “, “Chinavirus is “, “COVID-19 is “, “Coronavirus is “], and got nice, distinct results! Fixing the code so that the first results are kept as “trimmed”
• Need to change the embedding code so that the least squares are performed on the prior rotated points

GOES

• 10:00 meeting with Vadim

Book

• Finish decapitated army, start deprogramming

Scraped the ice/snow that fell last night off the driveway

The espresso machine that I thought had been broken turns out to have been a victim of the Trader Joe’s espresso? I had noticed with my hand pump that Wegmans espresso was far easier, so I pulled the machine from the basement and tried it with that. Success! Crema! Life is good.

Book

• Went over what I wrote yesterday and it was better than I thought it was! Fixed a few things and started on the decapitated army trope. Also pinged Matthew to chat about cults

GOES

• Got a response back from Frank with the algorithms. Vadim already has these, and he’s really like to get the code
• 10:00 Meeting with Vadim
• Need to see how fast we can make the yaw flip run and use that as a basis for speeding up the entire sim. Need to see if the Panda3d simulation step is adjustable. Long discussion on that. We’re going to try a few experiments and see what we can come up with by tomorrow
• Need to set up a 24-hour sim with a yaw flip in the middle
• Need to set up a 365-day sim with 2 yaw flips

ML Group

• 3:30 meeting
• Run against actual data as well
• See if it’s possible to draw lines between nodes of selected words
• Look up words and highlight
• Tensorflow 2.x: Neural Structured Learning (NSL) is a new learning paradigm to train neural networks by leveraging structured signals in addition to feature inputs. Structure can be explicit as represented by a graph or implicit as induced by adversarial perturbation.

Snow!

I have broken some of my GPT code. Started upgrading various things, like Tensorflow. The most recent version of Numpy seems to be broken, so I had to install an older version. Also, it looks like it’s time for a CUDA update. Here’s the list of work so far

• Installed TF 2.4.0
• Downgraded Numpy to 1.19.3 from 1.19.4
• cuda_11.0.2_451.48_win10.exe
• The problem seems to be related to tf.random.set_seed(), which used to work with an int. Now it needs some other kind of 32-bit int? Can’t seem to pass it in from the surrounding call any more, so I’m calling it in the main for the moment

Book

• Got some writing on language models that’s… ok

GOES

• Verified that the code is working. It was an error in the ordering of the quaternion multiplication.
• Need to set up a meeting with Frank(?) to set the parameters on the RWs. Isaac has sent an email introduction.
• Need to see how fast we can make the yaw flip run and use that as a basis for speeding up the entire sim. Need to see if the Panda3d simulation step is adjustable
• Need to set up a 24-hour sim with a yaw flip in the middle
• Need to set up a 365-day sim with 2 yaw flips

MDS SBIR

• Technical ‘kickoff meeting’. Spend some time talking to Peter, who is the technical lead about how our data will be generated and evaluated

JuryRoom

• Late for the 5:00 meeting due to shoveling snow. Sent T some resources on Qualitative research in HCI

Rolls-Royce’s Aletheia Framework is a practical toolkit that helps organizations to consider the impacts on people of using artificial intelligence prior to deciding whether to proceed. It looks across 32 facets of societal impact, governance and trust, and transparency and requires executives and boards to provide evidence that these have been rigorously considered.

10:00 – 1:00 Meeting with Mike D

GPT-Agents

• Finish least-squares transform of embeddings to see what it looks like
• I don’t see a straightforward way to produce and use a 4×4, so I’m starting with quaternions. Translation could be averaging the position of the points to calculate the center of the point cloud. The offset could be applied before the angle finding
• I wound up using scipy.spatial.transform.Rotation. To apply the results of the matrix directly. Here’s the relevent code:

from scipy.spatial.transform import Rotation as R
import numpy as np

kv = we.create_embedding_from_query("select content from table_output where experiment_id = 2 and probe = '{}' and tag = 'tweet';".format(probe), key_list)

# calculate the rotations to map one embedding onto the other
r:R
mapping = False
if len(kv_list) > 0:
    kv0 = kv_list[0]
    intersect = list(set(kv.index2word) & set(kv0.index2word))
    primary = []
    secondary = []
    for w in intersect:
        pi = kv0.index2word.index(w)
        si = kv.index2word.index(w)
        primary.append(kv0.vectors[pi])
        secondary.append(kv.vectors[si])
    A, res, rank, s = np.linalg.lstsq(primary, secondary, rcond=None)
    r = R.from_matrix(A)
    mapping = True
kv_list.append(kv)

• And it seems to work pretty well! Different embeddings seem to line up pretty well, and the February embedding seems to be farther away from the July embedding, with words like “Mike” and “Pence” in Feb, and coronavirus and @govcuomo in July

• I can also do this with the regular data as well. Just limit the number of returns
• 2:00 Meeting with Mike
• 3:30 Meeting COVID meeting

GOES

• Back to trying to figure out what’s going on.
• Ok, I made some(?) progress. One of the really important things is for the rotation not to overshoot. If it does, there is a lot of oscillation as the algorithm overshoots, then overshoots as it comes back:

• But when you fix that, everything looks pretty good!

• Vadim thinks he may have fixed it, based on the behavior between the calculated and ‘physical’ vectors?

MDS

• 4:30 resource discussion

I’m very conflicted about global warming in the winter. We just had a lovely weekend, where temperatures reached 60F. That’s very bad, but it sure feels nice. Didn’t stop me from getting my annual load of carbon offsets from the UN

GOES

• 2:00 Meeting with Vadim

GPT Agents

• Need to do some 3D embeddings in Gensim and display them.
• Try doing a least-squares to rotate two embeddings to align with each other.
• If that works, try seeing if embeddings from each month can be aligned in a reasonable way and what they look like
• 3:00 meeting with Sim to work on similarity matrix

Post this ride for tomorrow

Word embeddings quantify 100 years of gender and ethnic stereotypes (2018 paper)

• Word embeddings are a popular machine-learning method that represents each English word by a vector, such that the geometry between these vectors captures semantic relations between the corresponding words. We demonstrate that word embeddings can be used as a powerful tool to quantify historical trends and social change. As specific applications, we develop metrics based on word embeddings to characterize how gender stereotypes and attitudes toward ethnic minorities in the United States evolved during the 20th and 21st centuries starting from 1910. Our framework opens up a fruitful intersection between machine learning and quantitative social science.

Book

• Made some good progress on influence and started dominance
• 2:00 meeting with Michelle

GOES

• Sent Vadim a note about catching up. Looks like 2:00 on Monday
• More Plotly
• I know it’s dumb, but I figured out how to do a favicon and webpage title. It’s quite simple. Put the favicon file in a folder called “assets” in the same directory as the Dash code. Setting the title is even easier. Add a “title=’my title'” to the dash.Dash() call:

app = dash.Dash(__name__, title = 'Interactive!')

• It looks like it is very important to load the data before interacting. Updating global data is bad.

GPT-Agents

• I’m really thinking about the Language Through a Prism: A Spectral Approach for Multiscale Language Representations paper as a basis for topic extraction
• Installed dash-cytoscape for network graphs. Pretty cool:

• Got 3D scatterplots working as well. Next I need to do some 3D embeddings in Gensim and display them. After that, try doing a least-squares to rotate two embeddings to align with each other. If that works, Try seeing if embeddings from each month can be aligned ina reasonable way and what they look like

• Also, set up matrix distance code for Monday

High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications (textbook preprint)

• The problem of identifying low-dimensional structures of signals or data in high-dimensional spaces is one of the most fundamental problems that, through a long history, interweaves many engineering and mathematical fields such as system theory, signal processing, pattern recognition, machine learning, and statistics.

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

• Neural sequence models are widely used to model time-series data. Equally ubiquitous is the usage of beam search (BS) as an approximate inference algorithm to decode output sequences from these models. BS explores the search space in a greedy left-right fashion retaining only the top-B candidates – resulting in sequences that differ only slightly from each other. Producing lists of nearly identical sequences is not only computationally wasteful but also typically fails to capture the inherent ambiguity of complex AI tasks. To overcome this problem, we propose Diverse Beam Search (DBS), an alternative to BS that decodes a list of diverse outputs by optimizing for a diversity-augmented objective. We observe that our method finds better top-1 solutions by controlling for the exploration and exploitation of the search space – implying that DBS is a better search algorithm. Moreover, these gains are achieved with minimal computational or memory over- head as compared to beam search. To demonstrate the broad applicability of our method, we present results on image captioning, machine translation and visual question generation using both standard quantitative metrics and qualitative human studies. Further, we study the role of diversity for image-grounded language generation tasks as the complexity of the image changes. We observe that our method consistently outperforms BS and previously proposed techniques for diverse decoding from neural sequence models.

MORS

• Presentation today at 11:40! Getting ORA-LITE running on the laptop – done. Now being nervous. Yay! Done. Good questions, too!

Center for Informed Democracy & Social – cybersecurity (IDeaS)

• The overarching goal of the IDeaS Center at Carnegie Mellon University is to enhance social cyber-security to preserve and support and informed democratic society. The challenge today is that disinformation, hate speech, information warfare, and propaganda are amplified by cyber-technology like social media. We must remain informed, thoughtful members of our communities and countries, despite online and informational challenges.

Social Cybersecurity Working Group Focused on:

• The science to characterize, understand, and forecast cyber-mediated changes in human behavior, social, cultural and political outcomes,
• The science and engineering to build the cyber-infrastructure needed for society to persist in its essential character in a cyber-mediated information environment under changing conditions, actual or imminent social cyber-threats

Glamorous Toolkit is the moldable development environment. It is a live notebook. It is a flexible search interface. It is a fancy code editor. It is a software analysis platform. It is a data visualization engine. All in one.

“The antibody response to the virus has been shown to be transient and these antibodies start to wane after 3 to 4 months,” he said, adding that at 6 months they are “mostly undetectable” in many people who were infected early on in the epidemic,” (via Reuters)

MORS

• 11:30 – 12:10 New Adaptive Sampling Techniques to Optimally Augment Complex Simulation Models
• 12:10 – 12:50 Influence Planning Product
• 12:50 – 1:30 Risk Management for irregular Events
• 2:00 – 3:00 Social Cybersecurity
• 3:40 – 4:20 Community Resilience Indicator Analysis

4:00 Meeting with Matthew?

JuryRoom

• 5:00 Meeting