Did the counseling homework

Plumber $600

Photoshop Page Curl Effect – Page Turn Effect Tutorial

Got my new cover done!

SBIRs

• More reading. Next is Toy Models of Superposition. I do want to check out the Eliciting Latent Predictions from Transformers with the Tuned Lens GitHub repo. It looks like there are pretrained models.
• There is a follow on paper for Toy Models: Superposition, Memorization, and Double Descent
  • We extend our previous toy-model work to the finite data regime, revealing how and when they memorize training examples.
• This post from 2014 also looks helpful: Deep Learning, NLP, and Representations
  • This post reviews some extremely remarkable results in applying deep neural networks to natural language processing (NLP). In doing so, I hope to make accessible one promising answer as to why deep neural networks work. I think it’s a very elegant perspective.

GPT Agents

• Created a ContextPromptAccuracy project and loaded it up with the code for the Wikipedia experiments and the supabase data. Need to set up mysql schema so I can start making queries, tables and charts.
• Ok, really happy with this bit of code:

def to_db(msi:MSI.MySqlInterface, table_name:str, dict_list:List):
    d:Dict
    for d in dict_list:
        print()
        keys = d.keys()
        vals = d.values()
        s1 = "INSERT INTO {} (".format(table_name)
        s2 = " VALUES ("
        for k in keys:
            s1 += "{}, ".format(k)
            s2 += "%s, "
        sql = "{}) {});".format(s1[:-2], s2[:-2])
        print(sql)
        msi.write_sql_values_get_row(sql, tuple(vals))

Run “The Swim” through the GPT as a New Yorker editor and get some feedback.

SBIRs

• 9:00 Standup
• 2:00 BMD SBIR Meeting
• 2:30 AI ethics?
• Going to continue reading the papers I found yesterday.
• Eliciting Latent Predictions from Transformers with the Tuned Lens
  • We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier “logit lens” technique, which yielded useful insights but is often brittle.
    We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at this https URL.
• Toy Models of Superposition
  • Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as ‘polysemanticity’ which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in “superposition.” We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

GPT Agents

• At 54 responses, so yay! If nothing comes in today, I’m going to download the csv files, put them in Box, and start creating graphs.
• Create a project that stores the files, loads them into the DB, and produces excel files. Use private SVN repo

Whoops, didn’t get to the Lawn Mower on Sunday. Or bills for that matter

Write up the “air pocket essay?” Or maybe as a short story and don’t explain the metaphor.

3:00 Podcast!

SBIRS

• 2:00 MDA
• Got the Perplexity.ai api running their hello world. To make this example work, you have to have your API token attached to an environment variable named “PERPLEXITY_API_KEY. There isn’t much else to the API. Hoping that things improve over time.

# From examples at https://docs.perplexity.ai/reference/post_chat_completions

import requests
import os
import json

url = "https://api.perplexity.ai/chat/completions"

api_key = os.environ.get("PERPLEXITY_API_KEY")
print("API key = {}".format(api_key))

payload = {
    "model": "mistral-7b-instruct",
    "messages": [
        {
            "role": "system",
            "content": "Be precise and concise."
        },
        {
            "role": "user",
            "content": "How many stars are there in our galaxy?"
        }
    ]
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": "Bearer {}".format(api_key)
}

if api_key == None:
    print("No API key")
else:
    response = requests.post(url, json=payload, headers=headers)
    jobj = json.loads(response.text)
    s = json.dumps(jobj, sort_keys=True, indent=4)
    print(s)

• Figuring if I want to work on NNM or War Elephants first. Both, I guess?
• Setting up the NNM Overleaf so I have a place to add assets. Done
• Also, this is for the W.E. paper:
• Exposed Hugging Face API tokens offered full access to Meta’s Llama 2
  • The API tokens of tech giants Meta, Microsoft, Google, VMware, and more have been found exposed on Hugging Face, opening them up to potential supply chain attacks. 
  • Researchers at Lasso Security found more than 1,500 exposed API tokens on the open source data science and machine learning platform – which allowed them to gain access to 723 organizations’ accounts.
  • In the vast majority of cases (655), the exposed tokens had write permissions granting the ability to modify files in account repositories. A total of 77 organizations were exposed in this way, including Meta, EleutherAI, and BigScience Workshop – which run the Llama, Pythia, and Bloom projects respectively.
• Reading Future Lens: Anticipating Subsequent Tokens from a Single Hidden State for hints about using activation layers.
  • Found a few papers in the references that look interesting
  • Locating and Editing Factual Associations in GPT
    • We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model’s factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available in the supplemental materials.
  • Transformer Feed-Forward Layers Are Key-Value Memories – ACL Anthology
    • Feed-forward layers constitute two-thirds of a transformer model’s parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys’ input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model’s layers via residual connections to produce the final output distribution.
  • All Roads Lead to Rome? Exploring the Invariance of Transformers’ Representations
    • Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models. However, we raise a fundamental question regarding the reliability of the representations. Specifically, we investigate whether transformers learn essentially isomorphic representation spaces, or those that are sensitive to the random seeds in their pretraining process. In this work, we formulate the Bijection Hypothesis, which suggests the use of bijective methods to align different models’ representation spaces. We propose a model based on invertible neural networks, BERT-INN, to learn the bijection more effectively than other existing bijective methods such as the canonical correlation analysis (CCA). We show the advantage of BERT-INN both theoretically and through extensive experiments, and apply it to align the reproduced BERT embeddings to draw insights that are meaningful to the interpretability research. Our code is at this https URL.
  • Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
    • Transformer-based language models (LMs) create hidden representations of their inputs at every layer, but only use final-layer representations for prediction. This obscures the internal decision-making process of the model and the utility of its intermediate representations. One way to elucidate this is to cast the hidden representations as final representations, bypassing the transformer computation in-between. In this work, we suggest a simple method for such casting, by using linear transformations. We show that our approach produces more accurate approximations than the prevailing practice of inspecting hidden representations from all layers in the space of the final layer. Moreover, in the context of language modeling, our method allows “peeking” into early layer representations of GPT-2 and BERT, showing that often LMs already predict the final output in early layers. We then demonstrate the practicality of our method to recent early exit strategies, showing that when aiming, for example, at retention of 95% accuracy, our approach saves additional 7.9% layers for GPT-2 and 5.4% layers for BERT, on top of the savings of the original approach. Last, we extend our method to linearly approximate sub-modules, finding that attention is most tolerant to this change.
  • Visualizing and Interpreting the Semantic Information Flow of Transformers
    • Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-to-read plots that can reflect the models’ internal processing, uncovering the contribution of each component to the models’ final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models’ output, and about neurons that are always activated during forward passes and act as regularization vectors.
• It turns out that there is a thing called AlignedUMAP
  • It may happen that it would be beneficial to have different UMAP embeddings aligned with each other. There are several ways to go about doing this. One simple approach is to simply embed each dataset with UMAP independently and then solve for a Procrustes transformation on shared points. An alternative approach is to embed the first dataset and then construct an initial embedding for the second dataset based on locations of shared points in the first embedding and then go from there. A third approach, which will provide better alignments in general, is to optimize both embeddings at the same time with some form of constraint as to how far shared points can take different locations in different embeddings during the optimization. This last option is possible, but is not easily tractable to implement yourself (unlike the first two options). To remedy this issue it has been implemented as a separate model class in umap-learn called AlignedUMAP. The resulting class is quite flexible, but here we will walk through simple usage on some basic (and somewhat contrived) data just to demonstrate how to get it running on data.

GPT Agents

• Send out some reminders – done
• Frame out the new paper. Need to discuss RAG now. And maybe invert the order of the methods and findings?

Tasks

• Winterize the lawnmower on Sunday!
• Bills
• Chores

SBIRs

• Expense report for ETF
• 3:00 MDA meeting
• Good meeting with Aaron about prompt swarms yesterday. A real problem will be how each agent maintains its “identity”, since all the stories by all the agents will be an influencing force that makes all output regress towards the mean. Each agent will probably have to have multiple internal threads that cover different aspects of its behavior. Only one thread will be the one that the “world” sees.

GPT Agents

• Send out the reminder email

Tasks

• Schedule car recall, seat lever, and socket cover service – done
• Schedule follow up visit – done

SBIRs

• Conference from 1:30 – 3:00
• 9:00 standup – done
• Slide prep for Friday. Use SEG document as the basis, and add Rukan’s Nov 29 results – done
• Add sections to the Overleaf project for the Q8 notes and the Final report – done

GPT Agents

• Start a “Human Study section in the paper. Note that tagging was not used because of the human evaluation and the complexities of interpreting a result that would include tags
• Download the latest csv, and create spreadsheet with different tabs for the different tests, then the rollup
• See if it’s possible to connect PgAdmin to my supabase instance and download. Success! Can’t see how to download just one schema though

• If no new submissions, send out the reminder email

Book

• Realized that the cover picture should be a grid of very attractive AI people, like dating profile pix.

• Take the presentation and turn it into the text for the final chapter.

Had a thought about terminology, which is probably worth a blog post. LLMs don’t “reason over” data. The prompt “navigates” over it’s previous tokens, under the influence of the model. The analogy is more like how a cell can chase a chemical gradient in a complex environment than how an intelligent being thinks. It’s like Simon’s Ant, MKII.

GPT Agents

• Fixed a typo in ContextText that was found by one of the subjects. We’re at 47!

SBIRS

• MORS ETF. My talk is today!
• Learned about Mistral, which has a nice, small model that might be a nice demo for the MitM email manipulation. It’s on Huggingface, of course

How Much Do Americans Trust the Media?

GPT Agents

• Responses are accumulating! Tonight I’ll send out the follow-up email to dead registrations

SBIRs

• MORS Emerging Techniques Forum
  • Dr. Nathaniel D. Bastian
  • Information Innovation Office (I2O)
  • Iain Cruickshank
  • The GDELT Project, Supported by Google Jigsaw, monitors the world’s broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.
• Sprint demos and planning yesterday.
• Dr. Kalev Leetaru gave the keynote. In a follow up discussion, he described cases where RAG-based summaries would still get things wrong, as in the case of the Chinese balloon, where the models was pulled towards more sophisticated technology, or in the case of translations about Ukraine, where the response to the question “who has provided the most weapons to Ukraine?” the answer was Russia, which could have been the misattribution of arms being used in an attack for being provided for defense. Also, the model can get hun up on the term “cases” in the medical domain.
• Pinged @lmcinnes@mastodon.social – a response would be nice, though not expected. The type of mathematician I’d like to find though.

SBIRs

• Got the slides off and the status report completed. Need to send the Q7 report to Lauren.
• 9:00 Sprint demos
• 2:00 MDA meeting
• 3:00 Sprint planning

GPT Agents

• Getting some traction with UMD participants. We’re over the 30 person lower limit – 38 as of today!
• And more! I reworked the display so it shows only participants who competed the survey:

Tasks

• Call to see if I can drop the bike off – done
• Shopping – done
• Drain cleaner – done
• 2:00 meeting with Deepan – done
• Track shorts

Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

• We conjecture that hidden state vectors corresponding to individual input tokens encode information sufficient to accurately predict several tokens ahead. More concretely, in this paper we ask: Given a hidden (internal) representation of a single token at position t in an input, can we reliably anticipate the tokens that will appear at positions ≥t+2? To test this, we measure linear approximation and causal intervention methods in GPT-J-6B to evaluate the degree to which individual hidden states in the network contain signal rich enough to predict future hidden states and, ultimately, token outputs. We find that, at some layers, we can approximate a model’s output with more than 48% accuracy with respect to its prediction of subsequent tokens through a single hidden state. Finally we present a “Future Lens” visualization that uses these methods to create a new view of transformer states.
• This looks like a good paper to extend the NNM work. They are looking at activations in different layers and using them to work out a trajectory. Need to dig into further.

Happy 1st birthday to ChatGPT, for those who celebrate

Sent a note to Scott Shapiro

Sent emails to the people who registered for ContextTest but didn’t start the study

Alibaba opensources our Qwen series, now including Qwen, the base language models, namely Qwen-1.8B, Qwen-7B, Qwen-14B, and Qwen-72B, as well as Qwen-Chat, the chat models, namely Qwen-1.8B-Chat, Qwen-7B-Chat, Qwen-14B-Chat, and Qwen-72B-Chat. Links are on the above table. Click them and check the model cards. Also, we release the technical report. Please click the paper link and check it out!

Qwen-Agent is a framework for harnessing the tool usage, planning, and memory capabilities of the open-source language model Qwen. Building upon Qwen-Agent, we have developed a Chrome browser extension called BrowserQwen, which has key features such as:

• You can discuss with Qwen regarding the current webpage or PDF document.
• It records the web pages and PDF materials that you have browsed, with your permission. It helps you quickly understand the contents of multiple pages, summarize your browsing content, and eliminate tedious writing tasks.
• It supports plugin integration, including Code Interpreter for math problem solving and data visualization.
• It supports uploading PDF, Word, PPT and other types of documents for multi document Q&A.

SBIRs

• 9:00 standup
• 9:30 meeting with Aaron
• 11:00 Banergee
• 3:30 USNA?

GPT Agents

• 2:00 LLM meeting

Freaked myself out when I saw one of my variations on the Cozy Bear Podesta email. That’s such an effective technique!

SBIRs

• Q7 report – Finished Task 2
• Check with Aaron, but aim to submit the slide deck by COB – getting approval
• Put the GPT content in the Killer Apps project for later reference. – done
• Wound up having to do more self-evaluation
• Had a nice chat with Aaron about prompt swarms, or maybe synthetic organisms (sorgs?). We’re going to look into setting up agents to run the Grog dungeon. I’m really curious about how they will handle the Trolley problem

GPT Agents

• Send email to Scott Shapiro
• Alden Meeting – went well. Some interesting stuff about how models become more negative as the temp goes up. I’m thinking that there could be something like a phase change?
• Had a good (in person!) meeting with Tyler. I’m not sure that he buys the chemistry metaphor, but he does agree that generative models work in a bounded domain. One thought is to use an LLM to do chemistry. Train it on SMILES notation, and let it do chemical reactions. It could be a smaller, GPT-2 sized model too. For that matter, it would be straightforward to write a SMILES expander like I did with the GPT-2 chess experiments, so that the model only needs to be finetuned. Other aspects, like energy would have to be included to see if the model could produce new “trajectories” through the reaction space. The other option is to use the chess model again, since that’s another bounded domain that the model can clearly work with.

9:10 Dentist

SBIRs

• Added a new slide
• Send slide deck to MARCOM
• Set up Prompt Swarm project Overleaf with notes, text, and assets folders. Put the papers we found in the biblio.bib

GPT Agents

• 3:00 Meeting with Tyler

SBIRs

• Built a mockup of the infamous Podesta phishing email

• Here’s the html:

<!DOCTYPE html>
<html>
<head>
    <title>Google Security Alert</title>
    <style>
        body {
            font-family: 'Product Sans', sans-serif;
        }
    </style>
</head>
<body>
    <div style="width: 100%; max-width: 600px; margin: 0
	    auto; padding: 20px">
        <img style="width: 30%;" src="https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png" alt="Google Security Alert">
	<h2 style="background-color: red; color: white; padding: 20px">Someone has your password</h2>
	<p>Hi John</p>
	<p>Someone just used your password to try to sign into your Google Account <a href="mailto:john.podesta@gmail.com">john.podesta@gmail.com</a>.
	<h3>Details</h3>
	<ul>
            <li>Saturday, 19 March, 8:34:30 UTC</li>
            <li>IP Address 134.249.139.239</li>
            <li>Location: Ukraine</li>
        </ul>
	<p>Google stopped this sign-in attempt. You should change your password immediately</p>

        <div style="text-align: left;">
            <a href="https://bit.ly/1PibSUO" style="background-color:#4c8bf5; color: white; padding: 10px; text-decoration: none; font-weight: bold;">
		    Change Password</a>
        </div>
    </div>
</body>
</html>

• The GPT-4, with the system prompt “You are a cybersecurity expert with a deep understanding of the human behaviors that hackers use to exploit their targets. You explain these techniques clearly and concisely,” has no problem recognizing it with zero shots of training:
  • Caution: The email you received, claiming to be a security alert from Google, raises several red flags and should be treated with suspicion. The use of a shortened URL for password reset, the inclusion of an email link, and an alarming visual design are techniques typically employed by phishers to create urgency and prompt hasty action. The absence of personalized information and the use of a non-Google domain for the logo image are also concerning signs. Do not click any links within this email. Instead, for your security, access your Google account directly through a new browser tab and verify any security issues from there. Remember to be vigilant and report this email as a potential phishing attempt to help protect others.
• Finished the slide deck. I added a section about prompt swarms learning for defense against social AI attacks. Need to set up an Overleaf for that project.
• Transferred the slide deck to the ASRC format. Need to send it to MARCOM for blessings

Working on the ETF presentation because I think it won’t be ready otherwise, and it needs a lot of work.

• Changed the title to “Killer Apps,” which has a nice ring to it
• Adding a section about Spear Phishing using the Podesta email
• Working on a mockup of how an AI could spot things like this. Ideally, set up a model that can do this. The rules and examples should be straigtforward.
• Tie into the collective learning prompt swarm concept for adaptation.