Large Language Models Reveal Information Operation Goals, Tactics, and Narrative Frames

• Adversarial information operations can destabilize societies by undermining fair elections, manipulating public opinions on policies, and promoting scams. Despite their widespread occurrence and potential impacts, our understanding of influence campaigns is limited by manual analysis of messages and subjective interpretation of their observable behavior. In this paper, we explore whether these limitations can be mitigated with large language models (LLMs), using GPT-3.5 as a case-study for coordinated campaign annotation. We first use GPT-3.5 to scrutinize 126 identified information operations spanning over a decade. We utilize a number of metrics to quantify the close (if imperfect) agreement between LLM and ground truth descriptions. We next extract coordinated campaigns from two large multilingual datasets from X (formerly Twitter) that respectively discuss the 2022 French election and 2023 Balikaran Philippine-U.S. military exercise in 2023. For each coordinated campaign, we use GPT-3.5 to analyze posts related to a specific concern and extract goals, tactics, and narrative frames, both before and after critical events (such as the date of an election). While the GPT-3.5 sometimes disagrees with subjective interpretation, its ability to summarize and interpret demonstrates LLMs’ potential to extract higher-order indicators from text to provide a more complete picture of the information campaigns compared to previous methods.

Tasks

• Guardian
• Call Judith – done
• Talk to Nathan about compensating Tim for the unexpected use of his driveway. Done

SBIRs

• Now that I have Dash running, see if I can plot some activation matrices. I’ll need to put them into DataFrames first. Might even list the various layers/elements and let the UI switch between them. Be a good exercise in learning Dash.
• 9:00 standup
• 10:30 AFRL meeting prep. Is there an AFRL meeting as well?
• 4:30 Book club. I managed to read my chapter!

Dash

• https://dash.plotly.com/tutorial
• https://dash.plotly.com/layout
• https://dash.plotly.com/basic-callbacks
• https://dash.plotly.com/interactive-graphing
• https://dash.plotly.com/sharing-data-between-callbacks
• https://dash.plotly.com/advanced-callbacks
• https://dash.plotly.com/clientside-callbacks
• Made some great progress. I have an OO framework that handles graphing, data, and callbacks!

GPT Agents

2:00 Meeting. Talk about activations? Also the topic of the invited paper. I’d like to propose ‘The Pancake Printer: AI and the risk of a Two-Tiered Economy.’ It would include things like what’s discussed in Meet AdVon, the AI-Powered Content Monster Infecting the Media Industry (futurism.com). Also, Facebook’s AI Spam Isn’t the ‘Dead Internet’: It’s the Zombie Internet

• Made a shared doc for ideas