Category Archives: Phil

Phil 3.2.2023

Flying home from GSAW

Evidence of a predictive coding hierarchy in the human brain listening to speech

  • Considerable progress has recently been made in natural language processing: deep learning algorithms are increasingly able to generate, summarize, translate and classify texts. Yet, these language models still fail to match the language abilities of humans. Predictive coding theory offers a tentative explanation to this discrepancy: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition.

Phil 3.1.2023

Mapping people and tags on Mastodon

Introducing ChatGPT and Whisper APIs

Using the OpenAI API, you can build your own applications with gpt-3.5-turbo to do things like:

  • Draft an email or other piece of writing
  • Write Python code
  • Answer questions about a set of documents
  • Create conversational agents
  • Give your software a natural language interface
  • Tutor in a range of subjects
  • Translate languages
  • Simulate characters for video games and much more

Got the ChatGPT API working!


Phil 2.28.2023


  • Panel is done, so now we coast for two more days
  • I think a good definition for AI is that it is ML that interacts with people directly, while machine learning has no direct interaction.

Phil 2.27.2023

Stanford Human Preferences Dataset (SHP) is a dataset of 385K collective human preferences over responses to questions/instructions in 18 different subject areas, from cooking to legal advice. The preferences are meant to reflect the helpfulness of one response over another, and are intended to be used for training RLHF reward models and NLG evaluation models


  • Checked to see that all the other pictures are a good size. Yes! And the art row is intact, so phew.


  • At GSAW. The conference is not at the hotel. Fortunately, I brought a bike!
  • Panel is today after lunch

GPT Agents

  • Added summary and narrative to generation options
  • Stubbed out the saving of the relevant project data to the narrative maps input json file


  • “NOVA – I’m not sure what that means – we have some great acronym guys. Anyway, we have SUPERNOVA connecting the NOVAs”
  • Really thinking about the humans as the only attack surface that matters. Social hacking for everything, as long as you’re patient. This is going to be the real power of AI “social weapons.” They take advantage of intrinsic human bias and use it to shovel sand into high-tech adversaries. So how do you detect that? Is “death by PowerPoint” an example of a successful attack?

Phil 2.24.2023

One year later and probably at least one year to go.

Call Catonsville Eye Group (410) 747-1221

GPT Agents

  • Figured out my dates in NSW, so now it’s time to book the hotel
  • Pinged Commonvision about poster. Call to follow up if no email after lunch: (410) 455-1884
  • NEED TO FIX EMBEDDINGS in TweetEmbedExplorer
  • Update and commit narrative_maps
  • Today’s progress. It seems to be working very well:


  • More work on ContextExplorer
    • Make a separate “keyword” and “context prompt” fields. The second has a checkbox to use the main prompt for context. Keywords create a dataframe that contains lines with the keyword – done
    • Need to be able to see, edit, and store the context, though maybe it’s just part of the probe when stored in the db – added context tab
    • Once generation works, I’ll need to store the generated content for analysis and mapping. I think the best answer might be to “Save to NarrativeExplorer,” which can then load and automate the runs without duplicating a lot of effort. I’d just need an “Experiment Name” field. – buttons are in but not running
    • I’ll also need to add a table_notes per experiment that can include sources. Added to table_source
  • Make slides for demo and put task in the backlog. Finish out the tasks for this sprint

Phil 2.23.2022


  • Finalizing! And the release date has been moved up to June 1. Very promising!


  • 9:15 Standup
  • Continue with the ContextExplorer App. I’ll need to overload the GeneratorFrame so that I can have a field for the context prompt and the query prompt, with a checkbox to use the query as context
  • Progress for today:


  • Book hotel – got account. Need to check the tix to see what nights I’ll actually be in Sydney
  • Prepare poster and contact the UMBC print shop to see if they can make the schedule for a poster

Phil 2.22.2023

Some grist for the AI Ethics Corpora:

  • Explanations Can Reduce Overreliance on AI Systems During Decision-Making
    • The upshot here is that explanations for hard things are what users need. No one (almost) needs an explanation that 2+2=5 is incorrect (disregarding the incompleteness theorem and all that). Something more complex could require a lot of explanation, probably with a strong interactive component. For our work, the user would be decoupled from the “fact checker” that is trained in the domain of the AI and can recognize when it is struggling and likely to be wrong, and can switch to other models (like using a closed form equation solver for the hard problem above)


  • Finish peeling out embedding panel – done, I think?
  • Started on ContextExplorer

GPT Agents

  • 2:30 NIST meeting
  • 4:00 Meeting

The context-based queries work really well. Here’s the full process, with question, answer, summaries and supporting text:

OpenAIEmbeddings.load_project_summary_text(): pulling text for 'moby-dick'
	Processing 925 lines
creating context
Question: [Why is Ahab obsessed with Mobey-Dick?]
submitting question

Ahab is obsessed with Moby Dick because he believes that killing the White Whale will bring him revenge for the physical and emotional wounds he has suffered.
Supporting top 5 text:
	Dist = 0.108 Summary: Ahab is obsessed with the idea of finding Moby Dick, believing that his snow-white brow and hump would make him unmistakable, and he endures trances of torments in his pursuit of revengeful desire.
		line 4077: So that Monsoons, Pampas, Nor’-Westers, Harmattans, Trades; any wind but the Levanter and Simoon, might blow Moby Dick into the devious zig-zag world-circle of the Pequod’s circumnavigating wake
		line 4078: But granting all this; yet, regarded discreetly and coolly, seems it not but a mad idea, this; that in the broad boundless ocean, one solitary whale, even if encountered, should be thought capable of individual recognition from his hunter, even as a white-bearded Mufti in the thronged thoroughfares of Constantinople
		line 4079: For the peculiar snow-white brow of Moby Dick, and his snow-white hump, could not but be unmistakable
		line 4080: And have I not tallied the whale, Ahab would mutter to himself, as after poring over his charts till long after midnight he would throw himself back in reveries—tallied him, and shall he escape
		line 4081: His broad fins are bored, and scalloped out like a lost sheep’s ear
		line 4082: And here, his mad mind would run on in a breathless race; till a weariness and faintness of pondering came over him; and in the open air of the deck he would seek to recover his strength
		line 4083: what trances of torments does that man endure who is consumed with one unachieved revengeful desire
		line 4084: He sleeps with clenched hands; and wakes with his own bloody nails in his palms
	Dist = 0.132 Summary: Ahab was driven by his passion to capture Moby Dick, but he was also influenced by other motives, such as the possibility that killing more whales would increase the chances of encountering the White Whale, and the need to use tools and men to accomplish his goal.
		line 4220: Though, consumed with the hot fire of his purpose, Ahab in all his thoughts and actions ever had in view the ultimate capture of Moby Dick; though he seemed ready to sacrifice all mortal interests to that one passion; nevertheless it may have been that he was by nature and long habituation far too wedded to a fiery whaleman’s ways, altogether to abandon the collateral prosecution of the voyage
		line 4221: Or at least if this were otherwise, there were not wanting other motives much more influential with him
		line 4222: It would be refining too much, perhaps, even considering his monomania, to hint that his vindictiveness towards the White Whale might have possibly extended itself in some degree to all sperm whales, and that the more monsters he slew by so much the more he multiplied the chances that each subsequently encountered whale would prove to be the hated one he hunted
		line 4223: But if such an hypothesis be indeed exceptionable, there were still additional considerations which, though not so strictly according with the wildness of his ruling passion, yet were by no means incapable of swaying him
		line 4224: To accomplish his object Ahab must use tools; and of all tools used in the shadow of the moon, men are most apt to get out of order
	Dist = 0.136 Summary: After being physically and emotionally wounded, Ahab's monomania was unleashed and he directed all of his rage and hate towards Moby Dick, symbolizing the evil in life and thought.
		line 3855: All that most maddens and torments; all that stirs up the lees of things; all truth with malice in it; all that cracks the sinews and cakes the brain; all the subtle demonisms of life and thought; all evil, to crazy Ahab, were visibly personified, and made practically assailable in Moby Dick
		line 3856: He piled upon the whale’s white hump the sum of all the general rage and hate felt by his whole race from Adam down; and then, as if his chest had been a mortar, he burst his hot heart’s shell upon it
		line 3857: It is not probable that this monomania in him took its instant rise at the precise time of his bodily dismemberment
		line 3858: Then, in darting at the monster, knife in hand, he had but given loose to a sudden, passionate, corporal animosity; and when he received the stroke that tore him, he probably but felt the agonizing bodily laceration, but nothing more
		line 3859: Yet, when by this collision forced to turn towards home, and for long months of days and weeks, Ahab and anguish lay stretched together in one hammock, rounding in mid winter that dreary, howling Patagonian Cape; then it was, that his torn body and gashed soul bled into one another; and so interfusing, made him mad
	Dist = 0.143 Summary: The seamen were amazed as Ahab nailed gold to the mast and declared it was Moby Dick, describing him as having a curious fan-tail, a big spout like a shock of wheat, and many iron harpoons twisted in his hide.
		line 3294: cried the seamen, as with swinging tarpaulins they hailed the act of nailing the gold to the mast
		line 3295: It’s a white whale, I say,
		line 3296: resumed Ahab, as he threw down the topmaul:
		line 3297: a white whale
		line 3298: Skin your eyes for him, men; look sharp for white water; if ye see but a bubble, sing out
		line 3299: All this while Tashtego, Daggoo, and Queequeg had looked on with even more intense interest and surprise than the rest, and at the mention of the wrinkled brow and crooked jaw they had started as if each was separately touched by some specific recollection
		line 3300: Captain Ahab,
		line 3301: said Tashtego,
		line 3302: that white whale must be the same that some call Moby Dick
		line 3303: shouted Ahab
		line 3304: Do ye know the white whale then, Tash
		line 3305: Does he fan-tail a little curious, sir, before he goes down
		line 3306: said the Gay-Header deliberately
		line 3307: And has he a curious spout, too,
		line 3308: said Daggoo,
		line 3309: very bushy, even for a parmacetty, and mighty quick, Captain Ahab
		line 3310: And he have one, two, three—oh
		line 3311: good many iron in him hide, too, Captain,
		line 3312: cried Queequeg disjointedly,
		line 3313: all twiske-tee be-twisk, like him—him—
		line 3314: faltering hard for a word, and screwing his hand round and round as though uncorking a bottle—
		line 3315: like him—him—
		line 3316: cried Ahab,
		line 3317: aye, Queequeg, the harpoons lie all twisted and wrenched in him; aye, Daggoo, his spout is a big one, like a whole shock of wheat, and white as a pile of our Nantucket wool after the great annual sheep-shearing; aye, Tashtego, and he fan-tails like a split jib in a squall
	Dist = 0.144 Summary: For forty years, Ahab has forsaken the peaceful land to make war on the horrors of the deep, leading a life of desolation and solitude, feeding on dry salted fare while others had fresh fruit, and madly chasing his prey with a frenzy more befitting a demon than a man.
		line 9768: for forty years has Ahab forsaken the peaceful land, for forty years to make war on the horrors of the deep
		line 9769: Aye and yes, Starbuck, out of those forty years I have not spent three ashore
		line 9770: When I think of this life I have led; the desolation of solitude it has been; the masoned, walled-town of a Captain’s exclusiveness, which admits but small entrance to any sympathy from the green country without—oh, weariness
		line 9771: Guinea-coast slavery of solitary command
		line 9772: —when I think of all this; only half-suspected, not so keenly known to me before—and how for forty years I have fed upon dry salted fare—fit emblem of the dry nourishment of my soil
		line 9773: —when the poorest landsman has had fresh fruit to his daily hand, and broken the world’s fresh bread to my mouldy crusts—away, whole oceans away, from that young girl-wife I wedded past fifty, and sailed for Cape Horn the next day, leaving but one dent in my marriage pillow—wife
		line 9774: —rather a widow with her husband alive
		line 9775: Aye, I widowed that poor girl when I married her, Starbuck; and then, the madness, the frenzy, the boiling blood and the smoking brow, with which, for a thousand lowerings old Ahab has furiously, foamingly chased his prey—more a demon than a man
	Dist = 0.145 Summary: Ahab has been a fool for forty years, questioning why he is chasing Moby Dick and feeling old and weary, but when he looks into Starbuck's eye he sees his wife and child and decides to stay on board and not take the same risk as him.
		line 9776: what a forty years’ fool—fool—old fool, has old Ahab been
		line 9777: Why this strife of the chase
		line 9778: why weary, and palsy the arm at the oar, and the iron, and the lance
		line 9779: how the richer or better is Ahab now
		line 9780: Oh, Starbuck
		line 9781: is it not hard, that with this weary load I bear, one poor leg should have been snatched from under me
		line 9782: Here, brush this old hair aside; it blinds me, that I seem to weep
		line 9783: Locks so grey did never grow but from out some ashes
		line 9784: But do I look very old, so very, very old, Starbuck
		line 9785: I feel deadly faint, bowed, and humped, as though I were Adam, staggering beneath the piled centuries since Paradise
		line 9786: —crack my heart
		line 9787: —stave my brain
		line 9788: bitter, biting mockery of grey hairs, have I lived enough joy to wear ye; and seem and feel thus intolerably old
		line 9789: stand close to me, Starbuck; let me look into a human eye; it is better than to gaze into sea or sky; better than to gaze upon God
		line 9790: By the green land; by the bright hearth-stone
		line 9791: this is the magic glass, man; I see my wife and my child in thine eye
		line 9792: No, no; stay on board, on board
		line 9793: —lower not when I do; when branded Ahab gives chase to Moby Dick
		line 9794: That hazard shall not be thine
	Dist = 0.145 Summary: Ahab's crew seemed to be chosen by fate to help him in his revenge against the White Whale, and they shared his hatred for it, though it is unclear why or how this came to be.
		line 3891: Such a crew, so officered, seemed specially picked and packed by some infernal fatality to help him to his monomaniac revenge
		line 3892: How it was that they so aboundingly responded to the old man’s ire—by what evil magic their souls were possessed, that at times his hate seemed almost theirs; the White Whale as much their insufferable foe as his; how all this came to be—what the White Whale was to them, or how to their unconscious understandings, also, in some dim, unsuspected way, he might have seemed the gliding great demon of the seas of life,—all this to explain, would be to dive deeper than Ishmael can go
		line 3893: The subterranean miner that works in us all, how can one tell whither leads his shaft by the ever shifting, muffled sound of his pick
		line 3894: Who does not feel the irresistible arm drag
		line 3895: What skiff in tow of a seventy-four can stand still
		line 3896: For one, I gave myself up to the abandonment of the time and the place; but while yet all a-rush to encounter the whale, could see naught in that brute but the deadliest ill
		line 3897: The Whiteness of the Whale
		line 3898: What the white whale was to Ahab, has been hinted; what, at times, he was to me, as yet remains unsaid
execution took 4.342 seconds

Phil 2.21.2023

Thinking some more about AI-Human teaming:

Your Buddy, The Grandmaster: Repurposing the Game-Playing AI Surplus for Inclusivity

  • Advances in artificial intelligence (AI) techniques have resulted in immense breakthroughs in how well we can algorithmically play videogames. Yet, this increased investment in game-playing AI (GPAI) techniques has not translated into a tangible improvement in the game-playing experience for our players. This paper is inspired by the positive impact of accessibility modes in recent games as well as previous calls in games research literature to focus on player experience. Responding to these calls, we propose utilizing GPAI techniques not to beat the player, as is traditionally done, but to support them in fully experiencing the game. We claim that utilizing GPAI agents to help players overcome barriers is a productive way of repurposing the capabilities of these agents. We further contribute a design exercise to help developers explore the space of possible GPAI-driven assistance methods. This exercise helps developers discover types of challenges and ideate methods that vary in magnitude and assistance type. We first apply this design exercise to explore the design space of possible assistance methods for the action platformer game Celeste. We then implement two of the discovered methods that target different challenge types in a Unity clone of Celeste. Through this implementation, we discover several additional research questions we must answer before GPAI-driven assistance methods can be truly effective. We believe this research direction furthers the discussion on how to utilize GPAI in service of the player experience and also contributes to the creation of more inclusive games.


  • Reach out to the Navy Marine Mammal Program done
  • 9:15 Standup
  • 1:00 BMD done
  • 2:00 GPT Index done
  • Get the summary summaries working and tested done
    • Creating level 2. Looks like an average of 6:1 “compression.” Results are: Level 1:925, Level 2: 170, Level 3: 45, Level 4: 15, and Level 5:5. Here’s the last line of the original texts, then summaries:
      • Raw Text (uncompressed – 0.01% of the book): It was the devious-cruising Rachel, that in her retracing search after her missing children, only found another orphan
      • Level 1 (91% compression – 0.11% of the book) : After being drawn towards a vortex and buoyed up by a coffin, I was eventually rescued by the devious-cruising Rachel, who found another orphan in her search for her missing children.
      • Level 2 (98% compression – 0.59% of the book): Ahab and his crew were in awe of the whale, which caused destruction to the ship and sent them into a state of shock, and when it resurfaced near Ahab’s boat they realized it was an American whale, signifying retribution and vengeance; Ahab grappled with the whale but was ultimately dragged away and lost at sea, leaving his ship and crew to face their own destruction, with only one survivor to tell the tale of the sky-hawk that tauntingly followed the main-truck downwards, pecking at the flag and going down with the ship, before being rescued by the devious-cruising Rachel.
      • Level 3 (99.5% compression – 2.2% of the book): Ahab and his crew set sail on a mission to find Moby Dick, and despite the dangers they faced, Ahab was able to throw his harpoon and curse into the whale, only for it to escape; when another whale appeared, Ahab and his crew were in awe of its destruction and vengeance, but Ahab was ultimately dragged away and lost at sea, leaving his ship and crew to face their own destruction, with only one survivor to tell the tale of the sky-hawk that tauntingly followed the main-truck downwards before being rescued by the devious-cruising Rachel.
      • Level 4 (99.86% compression – 6.67% of the book): Ahab and his crew set out on a quest to find Moby Dick, but despite their determination and warnings from Starbuck, Ahab was ultimately dragged away and lost at sea, leaving his ship and crew to face destruction, with only one survivor to tell the tale of the sky-hawk that followed them before being rescued by the Rachel.
      • Level 5 (99.95% compression – 20% of the book): Ahab and his crew set out on a quest to find Moby Dick, but despite warnings from Starbuck, Ahab was ultimately dragged away and lost at sea, leaving only one survivor to tell the tale of the sky-hawk that followed them before they were rescued by the Rachel.
  • Add embeddings – done

GPT Agents

  • Once the summary levels are done, see how the clusters overlay and how a narrative path for each looks with respect to the other. The relationship between these may provide some insight into a multiresolution narrative clustering

Phil 2.20.2023

GPT Agents

  • Add NULL tests – done
  • Back up current db – done, several times
  • Get level data from previous run if it exists. It should, even if only one summary has been generated. Reworked everything into two methods, one that summarizes the raw text and one that summarizes the summaries. Both are much more robust and should take being kicked off openAI for saturation now.
  • The summarizing is really quite good. Moby Dick goes from 10k lines to 938. The new lines are closer to the same length and should produce better embeddings too. That’s for tomorrow
  • Pull out the embedding frame
  • For the new app, have a “create db” tab that lets the user set the text and group and find corpora. Probably regex too


  • 10:00 Artemis meeting
  • 2:00 MDA Meeting

Phil 2.19.2023

GPT Agents

  • Remarkably, my Twitter API code is still working
  • Added code to test for OpenAI exceptions. Need to verify that everything works and then do another pull

Phil 2.17.2023

Writing Essays With AI: A Guide

  • First, you don’t have to incorporate AI into your writing practice. If you want to keep writing longhand, that’s both fine and preferable in certain cases. The idea of this essay is to inspire you and help you experiment—not to give you the One True Way to write.
  • Second, you should know that everyone—including me—is making this up as we go. It’s a whole new frontier, so there isn’t any standard or accepted way to write with these tools. All I can share is what I’ve seen work for me and other people.
  • Third, there are many ways to misuse this tool to make crap. AI is not a panacea for lack of taste or bad intentions. If you’re skilled, though, you can use it to make stuff you love.


  • Send in AI Ethics review – done
  • Send in Camera Ready – done. Minor hiccup
  • Take care of book approvals – done? Found one problem


  • To the summary table, add a project “source” field and an “origins” field that has a list of all rows that were used to create the summary. These will accumulate as the summaries are combined. And add a “level” field that shows the summary level easily
    • Done! Everything works!
  • Whoops! Not so fast, need to handle timeouts like with getting embeddings.
  • I also think that all questions can be answered using the summary table. Should be faster and more useful
  • 3:00 FOM meeting – Loren thinks we have some unusual data in the 25% runs. We’ll tray that as holdout
  • Reworking NarrativeExplorer to use a GPT3GeneratorFrame and a GPT3EmbeddingFrame. First, it’s cleaner. Second, I should be able to re-use those components.

Phil 2.16.2023


  • Review AI/Ethics paper
  • Review book requests


  • Set up app. I want to create a series of components in their own files so the App file isn’t so unmanagable
  • Try compressing 2-n sentences into one, and build a hierarchical synthesis of a book. Moby-dick, at about 10,000 lines would compress at a 4:1 ratio like this (=B$1/POWER($E$1,A2)):
  • Trying that out with text-davinci-003 gives good results (temp 0, presence penalty 0.8):
  • That should keep drift at bay and allow for a better ability to look back rather than just using the closest prompts. We use each level of compression in the prompt, with the last line in the prompt from the uncompressed source, and then trace our sources through the levels of indirection.
  • Didn’t like the line splits. That makes the training text inconsistent. Switching to words. That seems to be working!
Summarize the following into a single sentence:
 ETYMOLOGY. Supplied by a Late Consumptive Usher to a Grammar School. The pale Usher—threadbare in coat, heart, body, and brain; I see him now. He was ever dusting his old lexicons and grammars, with a queer handkerchief, mockingly embellished with all the gay flags of all the known nations of the world.

Summary: A pale, threadbare Usher at a Grammar School was often seen dusting his old lexicons and grammars with a handkerchief decorated with flags from different nations.
  • The recursive summarization is crazy good. Need to run it on the full book. I think I need to add an int that shows the “level” of the abstraction. Thinking that the the summary out of the number of summaries would be good
  • 9:15 standup
  • 11:00 ChatGPT working session – went well!

Phil 2.14.2023

GPT Agents

  • Submit paper
  • Switch Tweet Embedding over to binary and list-based embedding retrieval
  • Add cluster labeling Done! With GPT response. Still need to use average center of cluster and roll that into the update callback:
  • Cluster -1 doesn’t make much sense because those are non-clustered points


  • Thinking about the document explorer. I think that taking the average of the embeddings in a cluster and then sorting the list to produce either extracting keywords, summarizing, or something that does a version for extracting topics should work.
  • To get narrative mappings to work, there may need to multiple “summaries plus implications to xxx” that might be needed. Then the same process as the NarrativeExplorer should work.
  • There needs to be a single-parent, multi-children relationship to a hierarchical clustering of a document set. Sentence->paragraph->chapter->book->collection, etc. in each case, there can be an embedding from the actual text (up to the max tokens?), and then generated text/embedding from the GPT.
  • Reading in a book is a multi-level process, that should then make returning results faster and better.

Phil 2.13.2023

GPT Agents

  • Switch the ManifoldReduction class over to use binary and adjust the parsed text tables to be BLOBs. Done for OpenAIEmbeddings and NarrativeExplorer
  • Submit Camera ready article – nope, got a delay


  • 9:00 Sprint Demos
  • 2:00 MDA Meeting
  • 1:30 NSWCDD meeting
  • Had a good discussion about creating a NeuralNarativeChat app