GPT-2 Agents

• Added Antonio’s suggestions and uploaded the paper to Overleaf. Looks like about one more page to play with
• Working on Twitter parser – done-ish?
• Wrapped the parser with a glob object and started the full run. 10,000 processed tweets and climbing!

Marketing:

• DATC Working group meeting and presentation.
• Presentation was technical as specified, and sailed over the heads of everyone, I think

GOES

• Postponed the meeting with Vadim because I thought the workshop ran all day

I would love to see a timeline of all the things that we’ve done in response to 9-11 and how they’ve worked out

GPT-2 Agents

• Parse timestamps – done
• Hand-annotate the schema?
  • datetime – done!
  • arrayobject
• Start running analysis? Wrote a tweet to the db. Calling it a day
• 2:30 Meeting with Shimei and Sim – started a google doc for tasking

GOES

• 10:00 Meeting with Vadim. I think the goal should be to tune the rwheels so that the yaw flip curves start to look more realistic, and then see how it works at speed. One of the issues that we need to think about is the role of mass in high-timestep physics. Would lower mass make better behavior?

Book

• 2:00 Meeting with Michelle – tweaked the dimension reduction section
• Write email to Theodore Hiebert – done
• Looked a bit into voice recording and transcription. The best answer still seems to be Google? Here are other options

Didn’t get a chance to write a post yesterday, so I’ll just include yesterday’s progress.

Latent graph neural networks: Manifold learning 2.0?

• Graph neural networks exploit relational inductive biases for data that come in the form of a graph. However, in many cases we do not have the graph readily available. Can graph deep learning still be applied in this case? In this post, I draw parallels between recent works on latent graph learning and older techniques of manifold learning.

Transformers are Graph Neural Networks

• Through this post, I want to establish links between Graph Neural Networks (GNNs) and Transformers. I’ll talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we could work together to drive progress.

GPT-2 Agents

• Made good progress on the table builder. I have a MySql implementation that’s pretty much done
• Made a pitch for IRAD funding
• Working on the tweet parsing
  • I have an issue with the user table. Tweets have a many-to-one relationship with user, so it’s a special case.
  • Added “object” and “array” to the cooked dictionary so that I can figure out what to do
  • Got the main pieces working, but the arrays can contain objects and the schema generator doesn’t handle that.
  • I think I’m going to add DATETIME processing for now and call it a day. I can start ingesting over the weekend

#COVID

• Didn’t make the progress I needed to on translating the text, so I asked for a week extension
• Downloaded the CSV file. Looks the same as the other formats with a “Label” addition. Should be straightforward

GOES

• Looks like Vadim fixed the transforms, so I’m off the hook
• Registered for M&S Affinity Group. Looks like I’ll be speaking at 12:20 on Monday
• 10:00 Meeting with Vadim
• Updated the DataDictionary to sys.exit(-1) on a name redefinition
• 11:00 Slides with T

Book

• Write letter

ML-seminar (3:30 – 5:30)

• My nomination for Adjunct Assistant Research Professor has been approved! Now I need to wait for the chain of approvals

JuryRoom (5:30 – 7:00)

• Alex was the only one on. We discussed HTML, CSS, and LaTeX

Done with vacation. Which was very wet. The long weekend was a nice consolation prize.

Hmmm. Subversion repo can’t be reached. Give it an hour and try again at 8:00, otherwise open a ticket. Turned out to be the VPN. Switching locations fixed it.

Sturgis COVID outcomes:

• “using data from the Centers for Disease Control and Prevention (CDC) and a synthetic control approach, we show that … following the Sturgis event, counties that contributed the highest inflows of rally attendees experienced a 7.0 to 12.5 percent increase in COVID-19 cases relative to counties that did not contribute inflows. … We conclude that the Sturgis Motorcycle Rally generated public health costs of approximately $12.2 billion.”

• In this paper, we show that the performance of a learnt generative model is closely related to the model’s ability to accurately represent the inferred \textbf{latent data distribution}, i.e. its topology and structural properties. We propose LaDDer to achieve accurate modelling of the latent data distribution in a variational autoencoder framework and to facilitate better representation learning. The central idea of LaDDer is a meta-embedding concept, which uses multiple VAE models to learn an embedding of the embeddings, forming a ladder of encodings. We use a non-parametric mixture as the hyper prior for the innermost VAE and learn all the parameters in a unified variational framework. From extensive experiments, we show that our LaDDer model is able to accurately estimate complex latent distribution and results in improvement in the representation quality. We also propose a novel latent space interpolation method that utilises the derived data distribution.

An automated pipeline for the discovery of conspiracy and conspiracy theory narrative frameworks: Bridgegate, Pizzagate and storytelling on the web

• Although a great deal of attention has been paid to how conspiracy theories circulate on social media, and the deleterious effect that they, and their factual counterpart conspiracies, have on political institutions, there has been little computational work done on describing their narrative structures. Predicating our work on narrative theory, we present an automated pipeline for the discovery and description of the generative narrative frameworks of conspiracy theories that circulate on social media, and actual conspiracies reported in the news media. We base this work on two separate comprehensive repositories of blog posts and news articles describing the well-known conspiracy theory Pizzagate from 2016, and the New Jersey political conspiracy Bridgegate from 2013. Inspired by the qualitative narrative theory of Greimas, we formulate a graphical generative machine learning model where nodes represent actors/actants, and multi-edges and self-loops among nodes capture context-specific relationships. Posts and news items are viewed as samples of subgraphs of the hidden narrative framework network. The problem of reconstructing the underlying narrative structure is then posed as a latent model estimation problem. To derive the narrative frameworks in our target corpora, we automatically extract and aggregate the actants (people, places, objects) and their relationships from the posts and articles. We capture context specific actants and interactant relationships by developing a system of supernodes and subnodes. We use these to construct an actant-relationship network, which constitutes the underlying generative narrative framework for each of the corpora. We show how the Pizzagate framework relies on the conspiracy theorists’ interpretation of “hidden knowledge” to link otherwise unlinked domains of human interaction, and hypothesize that this multi-domain focus is an important feature of conspiracy theories. We contrast this to the single domain focus of an actual conspiracy. While Pizzagate relies on the alignment of multiple domains, Bridgegate remains firmly rooted in the single domain of New Jersey politics. We hypothesize that the narrative framework of a conspiracy theory might stabilize quickly in contrast to the narrative framework of an actual conspiracy, which might develop more slowly as revelations come to light. By highlighting the structural differences between the two narrative frameworks, our approach could be used by private and public analysts to help distinguish between conspiracy theories and conspiracies.

#COVID

• Translate the annotated tweets and use them as a base for finding similar ones in the DB

GPT-2 Agents

• Working on the schemas now
• It looks like it may be possible to generate a full table representation using two packages. The first is genson, which you use to generate the schema. Then that schema is used by jsonschema2db, which should produce the tables (This only works for postgres, so I’ll have to install that). The last step is to insert the data, which is also handled by jsonschema2db
• Schema generation worked like a charm
• Got Postgres hooked up to IntelliJ, and working in Python too. It looks like jsonschema2db requires psycopg2-2.7.2, so I need to be careful
• Created a table, put some data in it, and got the data out with Python. Basically, I copied the MySqlInterface over to PostgresInterface, made a few small changes and everything appears to be working.

• Created the table, but it’s pretty bad. I think I* need to write a recursive dict reader that either creates tables or inserts linked data into a table
• All tables are created from objects, and have a row_id that is the key value
  • disregard the “$schema” field
  • Any item that is not an object is an field in the table
  • Any item that is an object in the table that gets an additional parent_row_id that points to the immediate parent’s row_id

GOES

• 2:00 meeting with Vadim

Back from a short, soggy vacation

Glasses!

Misinformation more likely to use non-specific authority references: Twitter analysis of two COVID-19 myths

• This research examines the content, timing, and spread of COVID-19 misinformation and subsequent debunking efforts for two COVID-19 myths. COVID-19 misinformation tweets included more non-specific authority references (e.g., “Taiwanese experts”, “a doctor friend”), while debunking tweets included more specific and verifiable authority references (e.g., the CDC, the World Health Organization, Snopes). Findings illustrate a delayed debunking response to COVID-19 misinformation, as it took seven days for debunking tweets to match the quantity of misinformation tweets. The use of non-specific authority references in tweets was associated with decreased tweet engagement, suggesting the importance of citing specific sources when refuting health misinformation.

Book

• Updated MikTex, which broke my epstopdf. Uninstalled and reinstalled everything. We’ll see how that works
  • basic-miktex-20.6.29-x64
  • texstudio-3.0.1-win-qt5
  • Hooray! Success!
• Finished the first pass of “The spacecraft of Babel”. Not sure how much sense it makes?
• Went over interviewing stuff with Michelle

GOES

• Pinged Vadim – nothing
• Security training

Taking the day off to go down to The March on Washington

This looks cool, though: Talk to Your Data: One Model, Any Relational Database

• TL;DR: We introduce Photon, a live demo of a natural language interface to databases based on our latest research in neural semantic parsing. 🔗 https://naturalsql.com/

And I made a thing!

Working on my CV so I can be considered for an adjunct position – done(?)

GOES

• 10:00 Meeting with Vadim
• Need to add an exit if a duplicate entry is made in DataDictionary
• Yaw flip is working again!
• 2:00 status

#COVID

• Look at translations!
• They look good! Going to translate the annotated tweets and use them as a base for finding similar ones in the DB
• The only issue is that a few words are being consistently mistranslated or rendered phonetically. Need to put together a data dictionary for some post processing

JuryRoom

• Good discussion with Tony about knowledge graphs and the analytics in general

GOES

• All hands meeting at noon
• Tag-team coding with Vadim – good progress. More tomorrow
• Make slides for T in the GPT-2 Agents IRAD – done?

GPT-2 Agents

• Finished updates to paper. The results are kind of cool. Here’s the error rate by model

• It’s pretty clear from looking at the charts is the ability of the model to learn legal moves is proportional to the size of the corpora. These are complicated rules, particularly for knights, bishops, and the queen. There needs to be a lot of examples in the text.
• On the other hand, let’s look at overall percentage of moves by piece when compared to human patterns:

• The overall patterns are substantially the same.  Here’s the 2-tailed correlation:

• There really is no substantial difference. To me that means that the low-frequency(?) information makes it into the model with this kind of thematic information with less text. Kinda cool!
• JSON parser
  • Got the code working to grab the right files and read them:

    for filename in glob.glob('*.jsonl', recursive=True):
        trimmed = re.findall(r"\\\d+_to_\d+", filename)
        print("{}/{}".format(filename, trimmed))
        with open(filename) as f:
            jobj = json.load(f)
            print(json.dumps(jobj, indent=4, sort_keys=True))

  • Working on the schemas now
    • It looks like it may be possible to generate a full table representation using two packages. The first is genson, which you use to generate the schema. Then that schema is used by jsonschema2db, which should produce the tables (This only works for postgres, so I’ll have to install that). The last step is to insert the data, which is also handled by jsonschema2db
    • Schema generation worked like a charm
    • Postgres and drivers are installed. That’s enough good luck for the day

ML seminar

• The Language Interpretability Tool:Extensible, Interactive Visualizations and Analysis for NLP Models
• how to get word embedding vector in GPT-2 #1458

from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained('gpt2')  # or any other checkpoint
word_embeddings = model.transformer.wte.weight  # Word Token Embeddings 
position_embeddings = model.transformer.wpe.weight  # Word Position Embeddings 

Book?

Getting new gutters today!

Clearing the Confusion: Scaling in PCA

• Many resources on machine learning (ML) methodology recommend, or even state as crucial, that one scale (or standardize) one’s data, i.e. divide each variable by its standard deviation (after subtracting the mean), before applying Principal Component Analysis (PCA). Here we will show why that can be problematic, and provide alternatives.

GPT-2 Agents

• Running analytics on legal moves and move frequency
  • gpt_view_400 – done
  • gpt_view_200- done
  • gpt_view_100- done
  • gpt_view_50- done
• Need to start on Twitter parser

GOES

• 2:00 Meeting with T- done
• 2:30 Sim status- done

Our code examples are short (less than 300 lines of code), focused demonstrations of vertical deep learning workflows.

All of our examples are written as Jupyter notebooks and can be run in one click in Google Colab, a hosted notebook environment that requires no setup and runs in the cloud. Google Colab includes GPU and TPU runtimes.

• Image classification from scratch
• Simple MNIST convnet
• Image segmentation with a U-Net-like architecture
• OCR model for reading Captchas
• Next-frame prediction with Conv-LSTM
• Grad-CAM class activation visualization
• Image classification via fine-tuning with EfficientNet
• Model interpretability with Integrated Gradients
• Metric learning for image similarity search
• Point cloud classification with PointNet
• Few-Shot learning with Reptile
• Object Detection with RetinaNet
• Visualizing what convnets learn
• Pneumonia Classification on TPU

• Text classification from scratch
• Sequence to sequence learning for performing number addition
• Bidirectional LSTM on IMDB
• Character-level recurrent sequence-to-sequence model
• Using pre-trained word embeddings
• Text classification with Transformer
• BERT (from HuggingFace Transformers) for Text Extraction

• Structured data classification from scratch
• Collaborative Filtering for Movie Recommendations
• Imbalanced classification: credit card fraud detection

• Timeseries anomaly detection using an Autoencoder
• Timeseries forecasting for weather prediction

• Speaker Recognition

• Variational AutoEncoder
• GAN overriding Model.train_step
• WGAN-GP overriding Model.train_step
• Neural style transfer
• Deep Dream
• CycleGAN
• Character-level text generation with LSTM
• PixelCNN
• Text Generation with miniature GPT

• Actor Critic Method
• Deep Deterministic Policy Gradient (DDPG)
• Deep Q-Learning for Atari Breakout

• Simple custom layer example: Antirectifier
• Keras debugging tips
• Endpoint layer pattern
• A Quasi-SVM in Keras
• How to train a Keras model on TFRecord files

Checking out Flourish for data visualization

GPT-2 Agents

• Hitting malformed descriptions in the 200k model that caused the program to crash twice. It was at 146k moves, which is probably enough to get statistics on illegal moves. Fixed the error with a try/except and moved onto the next model. I’ll go back and rerun if the fix works. Now over 100k moves on the 100k model with no problems. Yet.
• Pinged Dave Saranchak about the The Emerging Techniques Forum (ETF) as a venue

GOES

• 1:00 Meeting with Vadim. Yaw flip today? YES!!!
• Trying to work through what a malfunctioning RW would look like to the vehicle control. RW2 is critical for Roll, and RW5 is critical for Pitch and Roll. How is this system redundant?
• I think I need to make some plots to understand this

• So I think this is starting to make sense. There is always a point where any two rwheels are equal. You can see this on the RW1-RW4 graph. On the other, it’s just further to the left. This means that you can make them cancel and then manipulate the vehicle on the desired axis. This should be a matter of setting a scalar value such that the desired axis is nonzero. I’ll play around with that after lunch.

Book

• 2:00 Meeting with Michelle
• Steve Bannon learned to harness troll army from ‘World of Warcraft’ (USA Today)
  • Devil’s Bargain: Steve Bannon, Donald Trump, and the Nationalist Uprising (Book)
• Got some good writing in on dimension reduction

COVID

• I’m a little angry today. I was engaging with a Trump supporter and they wanted to know how things would have been better under Biden. So, based on how the Obama administration handled SARS and Ebola (very well), I thought I’d try mapping a democracy that is handling the pandemic very well, which is South Korea. Both the USA and S. Korea had their first case on the same day, so it is very easy to line up the data, and then scale the S. Korean results to match the population of the USA (About 6.4). The results are staggering:

• I need to write up a longer post on this and annotate the charts, but the conclusion is stark: the mismanagement of the pandemic by the administration has led to the deaths of hundreds of thousands of people who might otherwise be alive today

The Emerging Techniques Forum (ETF) is driven to improve analysis and understanding throughout the defense community by seeking out novel and leading-edge approaches, methods, and techniques – wherever they are conceived. By sharing and incorporating the latest (and in-progress) developments across government, academia, private industry, and enthusiasts, the ETF aims to support and maintain relevant, timely, and early comprehension of lessons learned that may grow to have an outsized impact on the community at large.

• December 7 – 10 2020

Book

• Write some content for the dimension reduction and emergence chapters – made good progress

GPT-2

• Created copies of the table_moves table for the 400, 200, 100, and 50 models. It’s easy!

CREATE TABLE foo LIKE bar;

• Generating text for table_moves_400 – done
• Generating text for table_moves_200

GOES

• 10:00 meeting with Vadim
  • Progress! Trying for a Yaw flip tomorrow
• 2:00 status meeting

JuryRoom

• 5:30 Meeting

Leaving today with this

GPT-2 Agents

• Based on feedback from yesterday, I’m going to train a model on 400,000, 200,000, 100,000, and 50,000
• Training the 400k model – done
• Training the 200k model – done
• Training the 100k model – done
• Training the 50k model – done

#COVID

• Downloaded data from Dropbox
  • Need to write a parser
• Killed the translator for now. At 292,719 translated tweets
  • Backing up the db and pushing. Nope, it won’t accept the db, and now I’m stuck
  • Looks like I have to add a large file extension? https://git-lfs.github.com/
  • Created a 100 line source/translation csv file and distributed
  • 3:00 Meeting – assigned translations

GOES

• 10:00 Meeting with Vadim
  • We went over the axis results and I realized that the pairs of reaction wheels are spinning in opposite directions, which allows for easy rotation around the vehicle primary axis. Need to figure out what the exact configurations are, but here’s the math for a hypothetical set space at 120 degrees around the z axis, and at 45 degrees with respect to an x-axis rotated the same amount

• 2:00 Status meeting. Nothing new. Still working on slow connectivity to the devlab

GPT-2

• Downloading COVID files for storage and training
• Writing parser and DB store

#COVID

• Stop the translator and find the 60 quotes
  • Everyone gets 15 quotes to translate
  • Everyone reviews all other translations and grades them on a 1-10 scale
  • Compare human and machine results

GOES

• 2:00 meeting with Vadim. Hopefully we’ll have a table of values
• Working on the projections on the axis:

• Still need to adjust so all angles are less than 90 by negating one of the axis (or the vector. It would have to be extended though)
• The percentage of the contribution per axis is going to be something like
  • 1.0 – axis angle / (normalized sum of angles)
• So if the angles are 50, 52, 62, the contributions would be

• Nope, realized that I want to take the cosine with respect to the axis of interest. So it’s this:

Food shopping!

4:00 ML Seminar

• Present slides, get feedback on paper and presentation – done!
• Train models with different sizes of data and see how the models behave with respect to the legal moves and spectral qualities