GPT-Neo is proud to release two pretrained GPT-Neo models trained on The Pile, the weights and configs can be freely downloaded from the-eye.eu.

• 1.3B: https://the-eye.eu/public/AI/gptneo-release/GPT3_XL/
• 2.7B: https://the-eye.eu/public/AI/gptneo-release/GPT3_2-7B/

3:30 Huggingface meeting

Pay bills

Send the RV back to the shop. Again. Create a checklist:

• When disconnected from shore power, please verify:
  • Lights come on
  • Generator starts
  • Refrigerator light comes on
  • Microwave runs
  • All status panels are functioning
  • Water pressure pump runs

GOES

• Check out and verify Vadim’s code works
• 2:00 Meeting

GPT Agents

• Finished terms and nouns over the weekend and started rank runs

SBIR/ONR

• Some good content: The uncontrollability of Artificial Intelligence
  • Explicit control – AI immediately stops the car, even in the middle of the highway because it interprets demands literally. This is what we have today with assistants such as SIRI and other narrow AIs. 
  • Implicit control – AI attempts to comply safely by stopping the car at the first safe opportunity, perhaps on the shoulder of the road. This AI has some common sense, but still tries to follow commands.  
  • Aligned control – AI understands that the human is probably looking for an opportunity to use a restroom and pulls over to the first rest stop. This AI relies on its model of the human to understand the intentions behind the command.
  • Delegated control – AI does not wait for the human to issue any commands. Instead, it stops the car at the gym because it believes the human can benefit from a workout. This is a superintelligent and human-friendly system which knows how to make the human happy and to keep them safe better than the human themselves. This AI is in control.  

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

• Machine learning methods offer great promise for fast and accurate detection and prognostication of coronavirus disease 2019 (COVID-19) from standard-of-care chest radiographs (CXR) and chest computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we consider all published papers and preprints, for the period from 1 January 2020 to 3 October 2020, which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. All manuscripts uploaded to bioRxiv, medRxiv and arXiv along with all entries in EMBASE and MEDLINE in this timeframe are considered. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher-quality model development and well-documented manuscripts.
  • Many papers gave little attention to establishing the original source of the images
  • All proposed models suffer from a high or unclear risk of bias in at least one domain
  • We advise caution over the use of public repositories, which can lead to high risks of bias due to source issues and Frankenstein datasets as discussed above
  • [Researchers] should aim to match demographics across cohorts, an often neglected but important potential source of bias; this can be impossible with public datasets that do not include demographic information
  • Researchers should be aware that algorithms might associate more severe disease not with CXR imaging features, but the view that has been used to acquire that CXR. For example, for patients that are sick and immobile, an anteroposterior CXR view is used for practicality rather than the standard posteroanterior CXR projection
  • We emphasize the importance of using a well-curated external validation dataset of appropriate size to assess generalizability
  • Calibration statistics should be calculated for the developed models to inform predictive error and decision curve analysis

GPT Agents

• Working on SocialSens2021 paper. Added some more references and figures. At 4 pages.
• 3:30 meeting

Book

• 2:00 Meeting with Michelle

GOES

• 11:00 Meeting with Vadim

SBIR/ONR

• Working on slides

Taxes!

GPT-Agents

• I have a lot of results. Now I need to put some preliminary-style text into the doc

GOES

• Get the sim to generate a pile of data.Done! And it looks good!

SBIR/ONR

• 9:30 Meeting with Aaron – got good guidance
• 1:00 IR&D Stand-up
• 1:30 Meeting with Rukan – going to hand of the initial Transformer model creation and evaluation., Done. Created a spreadsheet with a desired use case
• 4:30 Meeting with Orest. Went well? I have funding through the summer at 100%. After that

Shifting attention to accuracy can reduce misinformation online

• In recent years, there has been a great deal of concern about the proliferation of false and misleading news on social media^1,2,3,4. Academics and practitioners alike have asked why people share such misinformation, and sought solutions to reduce the sharing of misinformation^5,6,7. Here, we attempt to address both of these questions. First, we find that the veracity of headlines has little effect on sharing intentions, despite having a large effect on judgments of accuracy. This dissociation suggests that sharing does not necessarily indicate belief. Nonetheless, most participants say it is important to share only accurate news. To shed light on this apparent contradiction, we carried out four survey experiments and a field experiment on Twitter; the results show that subtly shifting attention to accuracy increases the quality of news that people subsequently share. Together with additional computational analyses, these findings indicate that people often share misinformation because their attention is focused on factors other than accuracy—and therefore they fail to implement a strongly held preference for accurate sharing. Our results challenge the popular claim that people value partisanship over accuracy^8,9, and provide evidence for scalable attention-based interventions that social media platforms could easily implement to counter misinformation online.

GPT Agents

• Ranking is still running
• Worked on the workshop paper. Added in a modified version of the intro from the chess paper that uses the GPT-3 now

ONR

• Working on literature

SBIR

• 10:00 Meeting

GOES

• 2:00 Meeting
• Turns out that we still have to do a demo. I need to create some data to show what that would look like. Set up a meeting with Vadim for Friday to make sure all the new code is working
• Generated all the scripts – about 700! Tomorrow I’ll run the “sim” and generate training values

GPT Agents

• I think I know how I want to structure the paper
  • Intro – discuss Tay, and how machine learning incorporates human input and reflects it back. This means that we have created ‘oracles’ that we can ask about the populations that contributed to their knowledge. In this type of computational sociology, finding and understanding the biases in these populations is an important part of the research
  • Introduce finetuned language models. Start with the chess model, and show how we can see the rank of piece terms rise and fall over the course of a sentence
  • Methods/results – describe the process of extracting chinavirus and sars-cov-2 as potential markers of different populations. Then prompts and runs to see the central terms that the models use. Show the stats. Then using the most popular terms from each model, run Ecco trajectories to show the rank behavior of these terms
  • Discussion. The possibilities of “interactive snapshots” of a population’s online behavior. The ongoing difficulty in prompt creation. Potential of maps?
  • Created the template
• Note – Create Dr. Fauci and Donald Trump prompts – done!
• Finished the noun finding, now running the ranks

SBIR

• Project planning
• Working on the ONR slides task

GPT Agents

• Worked on getting useful text to look at out of the models. Using flair to scan for POS. That way I can grab the first noun that occurs which makes for less text to look through, and more useful than just looking at the first word. I think that this will also be the approach that I’ll use to pull data out of the GPT-3 for maps.
• Finished training the COVID model, and committed to VCS
• Got some results for the first term. Going to re-run for some number of terms next. Also played around with the resulting spreadsheets a bit to look for patterns

SBIR

• Updating my drivers, verifying that TF still works, and upgrading to PT 1.8
  • Drivers are all updated as per here
  • Updated TF to 2.4.1 and everything still works
  • Trying to install pytorch 1.8, which wants CUDA 11.1. Going to try it with 11.0 first

MD food bank!

Some interesting papers

• Neural Encoding and Decoding With Distributed Sentence Representations
  • Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM’s selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding the relation between DSMs and neural activation patterns and contribute to building solid brain-machine interfaces with deep neural network representations.
• A Survey of the Usages of Deep Learning for Natural Language Processing
  • Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This article provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of relevant contributions. Analyzed research areas include several core linguistic processing issues in addition to many applications of computational linguistics. A discussion of the current state of the art is then provided along with recommendations for future research in the field.
• Multiview Concept Learning Via Deep Matrix Factorization
  • Multiview representation learning (MVRL) leverages information from multiple views to obtain a common representation summarizing the consistency and complementarity in multiview data. Most previous matrix factorization-based MVRL methods are shallow models that neglect the complex hierarchical information. The recently proposed deep multiview factorization models cannot explicitly capture consistency and complementarity in multiview data. We present the deep multiview concept learning (DMCL) method, which hierarchically factorizes the multiview data, and tries to explicitly model consistent and complementary information and capture semantic structures at the highest abstraction level. We explore two variants of the DMCL framework, DMCL-L and DMCL-N, with respectively linear/nonlinear transformations between adjacent layers. We propose two block coordinate descent-based optimization methods for DMCL-L and DMCL-N. We verify the effectiveness of DMCL on three real-world data sets for both clustering and classification tasks.

Writing part of the introduction of the IEEE issue on diversity in transportation.

2:00 AI/ML tagup

• Pinged Eric about getting a code to charge some of the hours – he’ll provide later

SBIR

• 11:30 tagup

ML Group

• 3:30 Meeting / Happy hour. Went over results. I’m going to run a larger experiment to generate text (not ranks). 50 tokens, 1,000 results for chinavirus and sars-cov-2

This looks really good! Deep Learning (with PyTorch). It’s a set of videos for NYU’s Deep Learning course.

• This course concerns the latest techniques in deep learning and representation learning, focusing on supervised and unsupervised deep learning, embedding methods, metric learning, convolutional and recurrent nets, with applications to computer vision, natural language understanding, and speech recognition. 

The weather’s great, so I’m taking the day off to ride this:

It was a marvelous ride on a warm, almost hot day. I write this on the morning of the 12th, and I can still feel it in my legs. I haven’t felt that in months. Even a long winter ride doesn’t do that. It’s really hard to put out sustained power when you’re (a) cold and (b) trying to avoid sweating too much and getting colder

Zach found a cool article: The Genius Neuroscientist Who Might Hold the Key to True AI

• Free energy is the difference between the states you expect to be in and the states your sensors tell you that you are in. Or, to put it another way, when you are minimizing free energy, you are minimizing surprise.

GPT-Agents

• Running the training for the new models
• Added the meta-summary spreadsheet:

• Need to re-run these tests on the new models using more runs and no rank testing

SBIR

• 9:30 Meeting – Looks like I need to get 50% coverage? Maybe in medical?
• More Pytorch tutorial
• Need to upgrade the ASRC box to 1.8 when it finishes training the current models
• Found my svncopy.bat file. It’s in JavaUtils2

GOES

• 3:00 Meeting

Quotebank is a dataset of 178 million unique, speaker-attributed quotations that were extracted from 196 million English news articles crawled from over 377 thousand web domains between August 2008 and April 2020. The quotations were extracted and attributed using Quobert, a distantly and minimally supervised end-to-end, language-agnostic framework for quotation attribution.

Stanford Cable TV News Analyzer The Stanford Cable TV Analyzer enables you to write queries that compute the amount of time people appear and the amount of time words are heard in cable TV news. In this tutorial we will go over the basics of how to use the tool to write simple queries.

GPT Agents

• Finished experiments and generated spreadsheets.
• Uploading everything to DropBox
• 3:00 Meeting
  • Create datasets from tweets that have [‘%kung flu%’, ‘%kungflu%’, ‘%china virus%’, ‘%chinavirus%’, ‘%coronavirus%’, ‘%covid%’, ‘%sars-cov-2%’] and train models from these. The idea is to examine how this type of polarized training can influence the response of the model. Related work on Microsoft’s Tay
  • Create a meta-sheet for all the spreadsheet summaries
  • Rather than look at rankings, go back to the cumulative stats on multiple runs with top K set to the range of ranks that we want to look at, then take a look at the first n words. This addresses the token problem

SBIR

• Set up proxy (2:00)?
• Write up curves embedding code
• Start on simplest possible autoregressing Transformer using curve data
• Started on the PyTorch Quickstart. Everything is installed properly and Cuda is visible

GSAW today

• The community is very much on the implementation part of ML. Aerospace corporation is doing some really nice work merging synthetic and actual data to detect threat anomalies. Slingshot is doing really nice data fusion
• I had an interesting ide come to me during the panel. It might be possible to train a large Transformer model on all mission telemetry from launch to sunset for all satellites. Then you could do zero-shot detection on new data, just like the GPT-3 does.

GPT-Agents

• Working on getting the meta information back to the summary tab – done
• Run all models – done
• I think I know how I want to try the mapping.
  • Use a prompt that should produce a list of nouns in order
  • Have the temp set reasonably high and for repetition to be low
  • Look at the output text and look for a N-N-N… pattern. Select those as nodes and stop when the pattern changes
  • Repeat and increment the edge weight for each redundant connection
  • Trim the leaf nodes with low counts

SBIR

• Ping Clay about how much of my time I can bill based on current rates
• Create generic multidimensional vectors for training
• Yannic Kilcher’s walkthrough of Attention Is All You Need

Arkipelago.space is a searchable map of interesting things on the Internet. The content is taken from a web crawl of 70,000 webpages originating from high-quality, human-curated links via Curius.app. A neural network uses the text content of each page to determine which pages should appear near each other on the map.

It seems to be a bunch of students playing around with cool things

Huggingface has lots of models to handle speech tagging!

• Main model page: huggingface.co/models?filter=sequence-tagger-model&pipeline_tag=token-classification
• The standard phrase chunking model for English that ships with Flair: huggingface.co/flair/chunk-english
• The fast 4-class NER model for English that ships with Flair: huggingface.co/flair/ner-english-fast
• The standard part-of-speech tagging model for English that ships with Flair: huggingface.co/flair/pos-english

This is a lot like self-attention in Transformers: How social learning amplifies moral outrage expression in online social networks

• Moral outrage shapes fundamental aspects of human social life and is now widespread in online social networks. Here, we show how social learning processes amplify online moral outrage expressions over time. In two pre-registered observational studies of Twitter (7,331 users and 12.7 million total tweets) and two pre-registered behavioral experiments (N = 240), we find that positive social feedback for outrage expressions increases the likelihood of future outrage expressions, consistent with principles of reinforcement learning. We also find that outrage expressions are sensitive to expressive norms in users’ social networks, over and above users’ own preferences, suggesting that norm learning processes guide online outrage expressions. Moreover, expressive norms moderate social reinforcement of outrage: in ideologically extreme networks, where outrage expression is more common, users are less sensitive to social feedback when deciding whether to express outrage. Our findings highlight how platform design interacts with human learning mechanisms to impact moral discourse in digital public spaces.

Related: Democracy Is Weakening Right in Front of Us: Is technopessimism our new future?

Book

• 2:00 Meeting with Michelle

GPT-Agents

• Finish summary table – Mostly done. Needs tweaking
• 3:30 Meeting

GOES

• 11:00 Meeting
• Continue working on data generation – generating faulty rw sims!

I wonder if any crazy things are going to happen today? Capitol Police say intelligence shows militia group may be plotting to breach the Capitol

GPT-Agents

• In EccoToXlsx, add code to iterate over all the samples from a prompt and add selected token ranks for the selected columns to a summary Dict. Compute mean and variance (95% intervals?), display the table and plot a candlestick plot.
• Set up a mapping directory in GPT-2 Agents. Do some test pulls using the Python API. I think the goal should be to populate a database that is similar to the gpt2_chess db table_moves (from, to, probe, response),

• Combined with table_output from gpt_experiments (experiment_id, root_id, tag, before_regex, and after_regex):

Book

• Work on chapters

GOES

• Work on fast sim
  • Finish moving code from frame3d_test file to FastRCSGenerator. Keep the plots too, just to make sure everything’s working. Done
  • Realized that the pitch/roll/yaw calculations were being done by ODE, so I had to get them back from the quaternion. It turns out that pyquaternion has yaw_pitch_roll(), but I can’t get to it? Added it to the VecData code
    • Figured it out. The @property decorator means no parens. You treat a method as a variable
  • I don’t think I’m incrementally updating setting the quaternion right.
  • Turns out I was rotating twice and storing the incremental steps as the rotations. Fixed!

Panel Study Of The MAGA Movement

• WaPo summary article: What explains MAGA supporters’ commitment to Trump and his conspiratorial and racist views? The answer is “status threat,” or the belief that one’s way of life or status is undermined by social and cultural change. As we’ve shown elsewhere, those who are attracted to reactionary movements like MAGA are often motivated by anxiety about possible cultural dispossession — seeing their social and cultural dominance eclipsed by other groups.

This is pretty cool! Not sure if it will work right, but…? Configure remote Python interpreters

Book

• Work on chapters

GPT-Agents

• Finished all the models!
• Set up experiments that run through each model for each set of terms and set of probes. Batch size of 50

SBIR

• 10:00 Meeting
• Keras Transformer model tutorial
• TF Text Transformer example
• Keras Attention
• Pytorch Attention

GOES

• Sitting in on GSAW keynote
• Vadim has made progress! 11:00 Meeting
• 2:00 Meeting
• Work on fast sim
  • Created data_generators project in PyBullet
  • Copied ScriptReaderScratch to FastRCSGenerator
  • Copied over the classes in least_squares_rotations (VecData, Rwheel, Rwheels, and Frame3D) and made them their own files
  • wrote up a frame3d_test file to exercise the classes and make sure that I haven’t broken anything. Everything still works!
• Get connected to repo?
• More on setting up a BERT-style (autoencoding) transformer for time series. Vector of sin waves at different frequencies first

JuryRoom

• 5:00 Meeting? Or just online?