Taking the day off to go down to The March on Washington

This looks cool, though: Talk to Your Data: One Model, Any Relational Database

• TL;DR: We introduce Photon, a live demo of a natural language interface to databases based on our latest research in neural semantic parsing. 🔗 https://naturalsql.com/

And I made a thing!

Working on my CV so I can be considered for an adjunct position – done(?)

GOES

• 10:00 Meeting with Vadim
• Need to add an exit if a duplicate entry is made in DataDictionary
• Yaw flip is working again!
• 2:00 status

#COVID

• Look at translations!
• They look good! Going to translate the annotated tweets and use them as a base for finding similar ones in the DB
• The only issue is that a few words are being consistently mistranslated or rendered phonetically. Need to put together a data dictionary for some post processing

JuryRoom

• Good discussion with Tony about knowledge graphs and the analytics in general

GOES

• All hands meeting at noon
• Tag-team coding with Vadim – good progress. More tomorrow
• Make slides for T in the GPT-2 Agents IRAD – done?

GPT-2 Agents

• Finished updates to paper. The results are kind of cool. Here’s the error rate by model

• It’s pretty clear from looking at the charts is the ability of the model to learn legal moves is proportional to the size of the corpora. These are complicated rules, particularly for knights, bishops, and the queen. There needs to be a lot of examples in the text.
• On the other hand, let’s look at overall percentage of moves by piece when compared to human patterns:

• The overall patterns are substantially the same.  Here’s the 2-tailed correlation:

• There really is no substantial difference. To me that means that the low-frequency(?) information makes it into the model with this kind of thematic information with less text. Kinda cool!
• JSON parser
  • Got the code working to grab the right files and read them:

    for filename in glob.glob('*.jsonl', recursive=True):
        trimmed = re.findall(r"\\\d+_to_\d+", filename)
        print("{}/{}".format(filename, trimmed))
        with open(filename) as f:
            jobj = json.load(f)
            print(json.dumps(jobj, indent=4, sort_keys=True))

  • Working on the schemas now
    • It looks like it may be possible to generate a full table representation using two packages. The first is genson, which you use to generate the schema. Then that schema is used by jsonschema2db, which should produce the tables (This only works for postgres, so I’ll have to install that). The last step is to insert the data, which is also handled by jsonschema2db
    • Schema generation worked like a charm
    • Postgres and drivers are installed. That’s enough good luck for the day

ML seminar

• The Language Interpretability Tool:Extensible, Interactive Visualizations and Analysis for NLP Models
• how to get word embedding vector in GPT-2 #1458

from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained('gpt2')  # or any other checkpoint
word_embeddings = model.transformer.wte.weight  # Word Token Embeddings 
position_embeddings = model.transformer.wpe.weight  # Word Position Embeddings 

Book?

Getting new gutters today!

Clearing the Confusion: Scaling in PCA

• Many resources on machine learning (ML) methodology recommend, or even state as crucial, that one scale (or standardize) one’s data, i.e. divide each variable by its standard deviation (after subtracting the mean), before applying Principal Component Analysis (PCA). Here we will show why that can be problematic, and provide alternatives.

GPT-2 Agents

• Running analytics on legal moves and move frequency
  • gpt_view_400 – done
  • gpt_view_200- done
  • gpt_view_100- done
  • gpt_view_50- done
• Need to start on Twitter parser

GOES

• 2:00 Meeting with T- done
• 2:30 Sim status- done

Our code examples are short (less than 300 lines of code), focused demonstrations of vertical deep learning workflows.

All of our examples are written as Jupyter notebooks and can be run in one click in Google Colab, a hosted notebook environment that requires no setup and runs in the cloud. Google Colab includes GPU and TPU runtimes.

• Image classification from scratch
• Simple MNIST convnet
• Image segmentation with a U-Net-like architecture
• OCR model for reading Captchas
• Next-frame prediction with Conv-LSTM
• Grad-CAM class activation visualization
• Image classification via fine-tuning with EfficientNet
• Model interpretability with Integrated Gradients
• Metric learning for image similarity search
• Point cloud classification with PointNet
• Few-Shot learning with Reptile
• Object Detection with RetinaNet
• Visualizing what convnets learn
• Pneumonia Classification on TPU

• Text classification from scratch
• Sequence to sequence learning for performing number addition
• Bidirectional LSTM on IMDB
• Character-level recurrent sequence-to-sequence model
• Using pre-trained word embeddings
• Text classification with Transformer
• BERT (from HuggingFace Transformers) for Text Extraction

• Structured data classification from scratch
• Collaborative Filtering for Movie Recommendations
• Imbalanced classification: credit card fraud detection

• Timeseries anomaly detection using an Autoencoder
• Timeseries forecasting for weather prediction

• Speaker Recognition

• Variational AutoEncoder
• GAN overriding Model.train_step
• WGAN-GP overriding Model.train_step
• Neural style transfer
• Deep Dream
• CycleGAN
• Character-level text generation with LSTM
• PixelCNN
• Text Generation with miniature GPT

• Actor Critic Method
• Deep Deterministic Policy Gradient (DDPG)
• Deep Q-Learning for Atari Breakout

• Simple custom layer example: Antirectifier
• Keras debugging tips
• Endpoint layer pattern
• A Quasi-SVM in Keras
• How to train a Keras model on TFRecord files

Checking out Flourish for data visualization

GPT-2 Agents

• Hitting malformed descriptions in the 200k model that caused the program to crash twice. It was at 146k moves, which is probably enough to get statistics on illegal moves. Fixed the error with a try/except and moved onto the next model. I’ll go back and rerun if the fix works. Now over 100k moves on the 100k model with no problems. Yet.
• Pinged Dave Saranchak about the The Emerging Techniques Forum (ETF) as a venue

GOES

• 1:00 Meeting with Vadim. Yaw flip today? YES!!!
• Trying to work through what a malfunctioning RW would look like to the vehicle control. RW2 is critical for Roll, and RW5 is critical for Pitch and Roll. How is this system redundant?
• I think I need to make some plots to understand this

• So I think this is starting to make sense. There is always a point where any two rwheels are equal. You can see this on the RW1-RW4 graph. On the other, it’s just further to the left. This means that you can make them cancel and then manipulate the vehicle on the desired axis. This should be a matter of setting a scalar value such that the desired axis is nonzero. I’ll play around with that after lunch.

Book

• 2:00 Meeting with Michelle
• Steve Bannon learned to harness troll army from ‘World of Warcraft’ (USA Today)
  • Devil’s Bargain: Steve Bannon, Donald Trump, and the Nationalist Uprising (Book)
• Got some good writing in on dimension reduction

COVID

• I’m a little angry today. I was engaging with a Trump supporter and they wanted to know how things would have been better under Biden. So, based on how the Obama administration handled SARS and Ebola (very well), I thought I’d try mapping a democracy that is handling the pandemic very well, which is South Korea. Both the USA and S. Korea had their first case on the same day, so it is very easy to line up the data, and then scale the S. Korean results to match the population of the USA (About 6.4). The results are staggering:

• I need to write up a longer post on this and annotate the charts, but the conclusion is stark: the mismanagement of the pandemic by the administration has led to the deaths of hundreds of thousands of people who might otherwise be alive today

The Emerging Techniques Forum (ETF) is driven to improve analysis and understanding throughout the defense community by seeking out novel and leading-edge approaches, methods, and techniques – wherever they are conceived. By sharing and incorporating the latest (and in-progress) developments across government, academia, private industry, and enthusiasts, the ETF aims to support and maintain relevant, timely, and early comprehension of lessons learned that may grow to have an outsized impact on the community at large.

• December 7 – 10 2020

Book

• Write some content for the dimension reduction and emergence chapters – made good progress

GPT-2

• Created copies of the table_moves table for the 400, 200, 100, and 50 models. It’s easy!

CREATE TABLE foo LIKE bar;

• Generating text for table_moves_400 – done
• Generating text for table_moves_200

GOES

• 10:00 meeting with Vadim
  • Progress! Trying for a Yaw flip tomorrow
• 2:00 status meeting

JuryRoom

• 5:30 Meeting

Leaving today with this

GPT-2 Agents

• Based on feedback from yesterday, I’m going to train a model on 400,000, 200,000, 100,000, and 50,000
• Training the 400k model – done
• Training the 200k model – done
• Training the 100k model – done
• Training the 50k model – done

#COVID

• Downloaded data from Dropbox
  • Need to write a parser
• Killed the translator for now. At 292,719 translated tweets
  • Backing up the db and pushing. Nope, it won’t accept the db, and now I’m stuck
  • Looks like I have to add a large file extension? https://git-lfs.github.com/
  • Created a 100 line source/translation csv file and distributed
  • 3:00 Meeting – assigned translations

GOES

• 10:00 Meeting with Vadim
  • We went over the axis results and I realized that the pairs of reaction wheels are spinning in opposite directions, which allows for easy rotation around the vehicle primary axis. Need to figure out what the exact configurations are, but here’s the math for a hypothetical set space at 120 degrees around the z axis, and at 45 degrees with respect to an x-axis rotated the same amount

• 2:00 Status meeting. Nothing new. Still working on slow connectivity to the devlab

GPT-2

• Downloading COVID files for storage and training
• Writing parser and DB store

#COVID

• Stop the translator and find the 60 quotes
  • Everyone gets 15 quotes to translate
  • Everyone reviews all other translations and grades them on a 1-10 scale
  • Compare human and machine results

GOES

• 2:00 meeting with Vadim. Hopefully we’ll have a table of values
• Working on the projections on the axis:

• Still need to adjust so all angles are less than 90 by negating one of the axis (or the vector. It would have to be extended though)
• The percentage of the contribution per axis is going to be something like
  • 1.0 – axis angle / (normalized sum of angles)
• So if the angles are 50, 52, 62, the contributions would be

• Nope, realized that I want to take the cosine with respect to the axis of interest. So it’s this:

Food shopping!

4:00 ML Seminar

• Present slides, get feedback on paper and presentation – done!
• Train models with different sizes of data and see how the models behave with respect to the legal moves and spectral qualities

A college kid’s fake, AI-generated blog fooled tens of thousands. This is how he made it.

• The trick to generating content without the need for much editing was understanding GPT-3’s strengths and weaknesses. “It’s quite good at making pretty language, and it’s not very good at being logical and rational,” says Porr. So he picked a popular blog category that doesn’t require rigorous logic: productivity and self-help.

GPT-2 Agents

• Slides – Done!?
• Start writing code to store JSON in db

#COVID

• Translation is chunking along a but faster now that I’m generating 20 translations per batch

GOES

• 2:00 Meeting with Vadim
  • The rotations all look good
  • Explained the 3 vector reference frame rotation approach
  • V will characterize each RW, and then we’ll try to spin the vehicle in the XY plane

Looks like we lost the classic editor on WordPress. Sigh.

GPT-2 Agents

• Updated ArXiv paper. Need to start thinking about slides for Tuesday. Started!

Book

• Finished moving text to Overleaf project
• 2:00 Meeting with Michelle – went through the whole structure of the book, added some chapters and moved other parts around. I’m going to start roughing in the new parts