Category Archives: Phil

Phil 4.2.20

Wake up, shower, write some code. The linear estimate is now integrated with the predictions. I think that tells the story well. Rather than Italy, let’s look at Switzerland:

Switzerland_4_2_2020

Now I need to put together a punch list of final issues (I’m worried most about load times and performance under high demand), get the URL(s) and publish!

  • Cache current json
  • Load times for pages
  • Term for low and negative days
  • Taller charts
  • Charts in landscape phone view
  • Interference in top part of phone
  • Resize charts to cover the range of regressionValues plus a few days on either side
  • Change “Predicted Values” to “Rate Trend”
  • How to host domains?

Phil 4.1.20

Working from home has a different rhythm. I work in segments with home chores mixed in. Today I’m doing this at 6:00, along with some coding. Then some morning exercise, breakfast, and work till noon. Ride, lunch and more work till about 3:00. By that time my brain is broken, and I take a break and do light chores. Today I may finally get my road bike ready for spring. Then simple work like commenting for a few hours. In the evenings I find I like watching shows about competent people fixing things and making them better. Bitchin’ Rides is extremely soothing.

D20:

  • Fixing dates
  • integrating the estimated deaths from rate and current deaths as area under the curve until zero.
  • Work on documentation. Also make sure word wrap works
  • This. Is. Bad.

Italy_4_1_2020

  • Once more, this is Italy. What I’ve done is round-tripped the rates to produce an estimate of total deaths. If calculating rates is taking the derivative, calculating a death prediction is integration. So, if the calculations are right, and Italy is at zero new deaths around April 17th, the toll is around 27 thousand total deaths. That’s 0.04% of the population. If those numbers hold for the US at 327 million, that’s a total of 145,550. The White House is estimating numbers of 100,000 to 240,000, which means their average prediction is that we will fare worse than Italy.
  • Fixed bugs, worked with Zach, made progress. Aaron is starting to appear again!

GOES

  • Tweak John’s slides
  • More on saving and restoring docker containers. I think I’m close. Then install InfluxDB and test if I can see the dashboard
  • Still having problems. I can create, run, add, delete, and tag the images, but I can’t run them. I think I’m getting ahead of myself. Back to reading

teminal

So it turns out that I was doing everything right but the load. Here’s how it works

  1. docker run container -it –name imagename some-os /bin/sh
  2. Install what needs to be installed. Poke around, save things, etc
  3. docker container commit imagename modified-os
  4. docker save modified-os> modified-os.tar
  5. docker rmi modified-os
  6. docker load < modified-os.tar
  7. docker run container -it –name imagename modified-os /bin/sh

teminal2

 

 

Phil 3.31.2020

I need to go grocery shopping today. A friend of mine has come down with the virus. He’s in his 30’s, and I’m feeling vulnerable. I went down to the shop and dug up my painting masks. Turns out I have a few, so that’s what I’m going shopping with. Here’s why, from the NY Times:

When researchers conducted systematic review of a variety of interventions used during the SARS outbreak in 2003, they found that washing hands more than 10 times daily was 55 percent effective in stopping virus transmission, while wearing a mask was actually more effective — at about 68 percent. Wearing gloves offered about the same amount of protection as frequent hand-washing, and combining all measures — hand-washing, masks, gloves and a protective gown — increased the intervention effectiveness to 91 percent.

Podcast with BBC’s misinformation reporter: https://podcasts.apple.com/gb/podcast/the-political-party/id595312938?i=1000470048553

 

  • A friend of mine who works in Whitehall has told me that the army are going to be on the streets this week arresting people who don’t listen to this podcast. If that sounds familiar, you’ll be aware that this crisis has already been fertile ground for disinformation. Marianna Spring is a BBC specialist reporter covering disinformation and social media. In this fascinating interview, Marianna reveals how disinformation and misinformation gets so widely shared, why we share it, how to spot it, what the trends are, how it differs around the world and so much more. This is a brilliant insight not just into the sharing of inaccurate information, but into human behaviour.

 

D20

  • Changed the calculations from the linear regression to handle cases where the virus is under control, like China – first pass is done
  • Have the linear regression only go back some number of weeks/months. I’m worried about missing a second wave
  • Need to add a disclaimer about the quality of the predictions is dependent on the quality of the data, and that we expect that as poorer countries come online, these trends may be erratic and inaccurate.
  • Add an UNSET state. The ETS will only set the state if it is UNSET. This lets regression predictions to be used until the ETS is working well – done
  • I think showing the linear and ETS mean prediction is a good way to start including ETS values
  • Found the page that shows how to adjust parameters: https://www.statsmodels.org/stable/examples/notebooks/generated/exponential_smoothing.html

GOES

  • Try to create an image from the stored tar
  • Start setting up InfluxDB2

IRAD Meeting at 2:00

ML Group at 4:00

  • Put together a list of potential papers to present. No need, I’ll do infinitely wide networks
  • Had just a lovely online evening of figuring out how to use some (terrible!) webex tools, and trying to figure out Neural ODEs. It was an island of geeky normalcy for a few hours. This may be a more comprehensible writeup.

Phil 3.30.20

Today’s study in contrasts: Italy and the US:

COVID-19 projections for the US, from the The Institute for Health Metrics and Evaluation (IHME):

IHME

Work on converting the ETS json file into spreadsheets to evaluate thresholds and labels – spreadsheet conversion is working. done! Now I need to figure out what those ETS parameters do!

Add a short bit to the D20 writeup that explains why linear interpolation isn’t the best option, and why we went with ETS – done

Work with Zach to get the website up today – working

Work this article into the exploit-space writeup: Why Is Cybersecurity Not a Human-Scale Problem Anymore?. Wow, actually, the company (Balbix) that was founded by the author (Gaurav Banga) seems to be doing most of what I was going to write about. Sent Darren a note to see if I should continue

Got a note from ProQuest saying my file needed to have blank pages at the beginning and end of the document. Fixed. And accepted!

  • Congratulations. Your submission, xxxxx has cleared all of the necessary checks and will soon be delivered to ProQuest for publishing.

Ok, back to Docker and building an InfluxDB image. Wow, that seems like a lifetime ago I was doing this

  • To save a custom image, create the container from a base image and then docker save image_name > image_name.tar. This puts it wherever you run the command in the system, Linux or Windows

#COVID-19 meeting at 1:30 today – proposal’s in. We have twitter data from January

SDaaS meeting at 4:00 today – postponed

Phil 3.28.20

From today’s spreadsheet: countries_2020-03-28_07-38

US_3.28_2020

Italy_3.28_2020

NY Times is starting to use rates as well Some U.S. Cities Could Have Coronavirus Outbreaks Worse Than Wuhan’s

Interesting chat as an expert(?) on developing code in the future

Working on the ssh transfer in code using paramiko. This seems to be a good one.

It works!

import paramiko

filename = "C:/Development/Sandboxes/DaysToZero/data/external/countries_2020-03-28_07-38.xlsx"
remote_dir = "/home/some_place.com/d20/countries_2020-03-28_07-38.xlsx"
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('ssh.some_place.com', username='some_login', password='some_password')
ftp_client = client.open_sftp()
#l = ftp_client.listdir()
#print(l)
ftp_client.put(filename, remote_dir)
ftp_client.close()
client.close()

I should try to put all the pieces together, but I am just done, and am stress-scrolling through Twitter, which really doesn’t help. Getting away from the computer for a while

Phil 3.27.20

Working with Zach and Aaron on the app. I think we’ll have something by this weekend

  • Added a starting zero on the regression
  • Added the regression to the json file, and posted to see if Zach can reach
  • Set up the hooks for export to excel workbook, with one tab per active country. I’ll work on that later today – done! countries

Got clarification from Wayne on some edits. Going to turn those around this morning and try to submit before COB today. Maryland is at 580 confirmed cases as of yesterday. I’d expect to see nearly 800 when they update the site this morning. Sent over all the edits. It’s in!

Maryland_3.26_2020

Yup

Maryland_3.27_2020

ProQuest submission site.

Phil 3.26.20

Updated the proposal

Found an example of diversity injection in the wild: the-syllabus.com. Here’s a story about it from The Correspondent.

Working on the parser today

  • Tried using my ExcelUtils, which are barfing on all the text in the csv
  • Discovered DictReader from the csv library, which works perfectly!
  • Throwing away rows that have less than three data points
  • Collecting rows into countries – done
  • Parsing out dates and values – done
  • Working on getting totals – done
  • Working on calculating rates – done
  • Seeing if I can do a least squares regression to calculate a first pass -done? It doesn’t seem to quite work right on the actual data
  • Aaron added his pieces in and everything seems to be kind of working

Phil 3.25.20

Waking up to the news these days makes me want to stay in bed with the radio off

Working on automating the process of downloading the spreadsheet, parsing out the countries, and calculating daily rates. The goal is to have a website up this weekend so you can see how your country is doing.

Tasks

  • Set up converter class – done
  • download spreadsheet – done
  • parse out countries – working on it
  • Made mockups of the mobile and webpage displays, and refined a few times based on comments

Got notes for Chapter 11 from Wayne. Switching gears and rolling that in. Put in changes for all the items I could read. There are a few still outstanding. I’ll submit tonight if Wayne doesn’t come back for a discussion.

Back to Docker. Need to connect to the WLS. Done!

Meetings

  • AIMS – status for all, plus technichal glitches. We’ll try Teams next time. Vadim has made GREAT progress. We might be able to get a real Yaw Flip soon as well
  • A2P – Infor demo. Meh.

Stampede theory proposal deadline was delayed a couple of days

Phil 3.24.20

Well, I’ve got more predictions using death rates as described in this post. Based on the latest dataset from here (Github), I’ve created a spreadsheet that does a linear (least squares) extrapolation for when the number of new deaths per day drops to zero:

predictions_3.24.20

China is in this group as a sanity check, and as you can see, it’s very near zero, and so is South Korea. Italy, Germany, Spain, Iran, and Indonesia are currently all in the middle, with 2-3 weeks to go if nothing changes. France, the Netherlands, and Switzerland are far enough out that I think these may be low confinance values. The UK seems to be doing terribly. The worst performers are Belgium and the US, whose death rates are still going up, as indicated by the “-1” in the “days till” column.

Here are plots of the data used to calculate the table. Due to the way that excel labels axis, I don’t have dates for the x-axis for all the charts. They all end on the seme date (3/21, the last day in the dataset with the two days I need to calculate rates), but some of them have fewer tata points so that the time before the outbreak doesn’t influence the calculations:

prediction_charts_3.24.20

Working on Hours for misinfo proposal

ASRC

  • Had a good chat with Biruh about InfluxDB running in Docker. Since I’m running the Windows version, things are different enough that I’m going to need to download a linux distro image and run my own version of InfluxDB2 inside that. Which means I need to get smarter on Docker and making a custom image, etc. Got this book. We’ll see how that goes today.

ML Webex meeting

BART is the new BERT!

BART

Phil 2.27.20

ASRC GOES 7:00 – 8:00

GOES

PhD

  • Meeting with Wayne. I got the (preliminary)
    • Combine the Limitations and Future Work chapters into a new chapter that explores where my research has landed. I think that a beachhead analogy might work here.
      • Preliminary, early work that lives in the space between computational sociology and HCC (socio-technical, from From Keyword Search to Exploration). Add discussions about wormholes, weathermaps, and maps that connect distant places, like air travel maps.
      • Diversity science
      • Gathering data on online consensus in various sized groups and within different cultures
      • Extending simulations into human belief spaces, such as with GPT-2 agents
    • Add a “my hut, revisited” section to the contributions that discusses the contributions from the perspective of the spaces defined by Kauffman, Martindale, and Bacharach in particular, but also the literature in general

Tie into this as well

Some general reworking of the contributions text to reflect the slides: This also requires expansion of the current text in CH11. In particular slides 49-52 are a stronger synopsis than is provided in your CH10 discussion. Slides 53-56 are a more thoughtful framing of your contributions, both theoretical and practical (can you weave this back in your literature from CH2 to show specific knowledge contribution?). The “bookend” revisiting ofthe trustworthy anonymous citizen journalism was also more effective in the presentation than the document. Consider capturing this.

Phil 2.19.20

7:00 – 8:00 ASRC GOES

disinfoOps
  • Defense practice and tweaking
  • Continue setting up workstation
    • Java – done
    • Python – done
      • Pandas3d
      • Panda
      • Scikit-learn
      • TF 2.0
      • etc
    • TortoiseSVN – done
    • WinSCP – done
    • PuTTY – done
    • XAMPP – done
    • gVIM – done
    • MikTex – done
    • TexStudio – done
    • Adobe Creative Cloud – done
      • Acrobat- done
      • Illustrator- done
      • Photoshop – done
    • Intellij
    • Office- done
    • Project
    • Chrome- done
    • FF
    • GitHub desktop
    • Set up non-admin user
  • Mission Drive meetings

Phil 2.19.20

8:00 – 7:00 ASRC GOES

  • I have a cold. Yuck!
  • How to build a brain from scratch
    • This advanced option course discusses the search for a general theory of learning and inference in biological brains. It draws upon diverse themes in the fields of psychology, neuroscience, machine learning and artificial intelligence research. We begin by posing broad questions. What are brains for, and what does it mean to ask how they “work”? Then, over a series of lectures, we discuss parallel computational approaches in machine learning/AI and psychology/neuroscience, including reinforcement learning, deep learning, and Bayesian methods. We contrast computational and representational approaches to understanding neuroscience data. We ask whether current approaches in machine learning are feasible and scaleable, and which methods – if any – resemble the computations observed in biological brains. We review how high-level cognitive functions – attention, episodic memory, concept formation, reasoning and executive control – are being instantiated in artificial agents, and how their implementation draws upon what we know about the mammalian brain. Finally, we contemplate the outlook for the future, and whether AI will be “solved” in the near future.
  • Start writing GVSETS simulation white paper for March 3
  • Finish setting up workstation
    • Soooooo, that turned out to be a bigger problem. The way that admin is set up is that there are two levels, an “install software” level, and an “install hardware level”. I had been given the former. When I installed the new SSD, it had to be formatted with NTFS. I could launch the device manager, but could not access the drive. I took the box down to HQ, and their IT folks were mystified. I stopped by the shop that had fixed the power supply, and for another $50, they put the drive in another machine, formatted and named it. Then it worked. Wheee.
    • I then start re-installing software and find that although I can log in as admin, I can’t run anything as admin. That requires going back to HQ so that they can set permissions on the local network, and install a VPN. Now our local network is horrible. There is some kind of whitelist/blacklist filter that is so slow that web pages often timeout rather than load.
    • At this point, I decided that it was easier to reset the machine and start over. So I’ll be doing that for a few days. Today, I got the NVIDIA and CUDA drivers installed.
  • Verify TF and multi-gpu optevolver
  • ML Seminar

Phil 2.17.20

Pinged Aaron about getting together today – 3:00

  • Went through the talk he says it seems pretty solid
  • Also dropped by Don’s office to say hi

Generated a new map where a stampede stops when it hits the edge

Played around with the “Curse of Dimensionality” slide

Added some background on fact-checking to the RQ slide

Start fixing Alienware

  • Mount drive – done
  • Plug everything in and connect network – done
  • Copy files
  • Fix environment variables

Online Conspiracy Theories: The WIRED Guide

  • Everything you need to know about George Soros, Pizzagate, and the Berenstain Bears.

 

Phil 2.16.20

Bringing Stories Alive: Generating Interactive Fiction Worlds

  • World building forms the foundation of any task that requires narrative intelligence. In this work, we focus on procedurally generating interactive fiction worlds—text-based worlds that players “see” and “talk to” using natural language. Generating these worlds requires referencing everyday and thematic commonsense priors in addition to being semantically consistent, interesting, and coherent throughout. Using existing story plots as inspiration, we present a method that first extracts a partial knowledge graph encoding basic information regarding world structure such as locations and objects. This knowledge graph is then automatically completed utilizing thematic knowledge and used to guide a neural language generation model that fleshes out the rest of the world. We perform human participant-based evaluations, testing our neural model’s ability to extract and fill-in a knowledge graph and to generate language conditioned on it against rule-based and human-made baselines. Our code is available at this https URL.

The Obligation To Experiment

  • Tech companies should test the effects of their products on our safety and civil liberties. We should also test them ourselves.

Ran through the presentation with David. He pointed out that stampedes bouncing off the edge of the environment look like flocking, so I generated a new map where the stampede gathers and runs off the edge

flock_2_stampede_map_legend

Phil 2.14.20

7:00 – 8:30 ASRC GOES

This document describes the Facebook Full URL shares dataset, resulting from a collaboration between Facebook and Social Science One. It is for Social Science One grantees and describes the dataset’s scope, structure, fields, and privacy-preserving characteristics. This is the second of two planned steps in the release of this “Full URLs dataset”, which we described at socialscience.one/blog/update-social-science-one.

Judging Truth

    • Deceptive claims surround us, embedded in fake news, advertisements, political propaganda, and rumors. How do people know what to believe? Truth judgments reflect inferences drawn from three types of information: base rates, feelings, and consistency with information retrieved from memory. First, people exhibit a bias to accept incoming information, because most claims in our environments are true. Second, people interpret feelings, like ease of processing, as evidence of truth. And third, people can (but do not always) consider whether assertions match facts and source information stored in memory. This three-part framework predicts specific illusions (e.g., truthiness, illusory truth), offers ways to correct stubborn misconceptions, and suggests the importance of converging cues in a post-truth world, where falsehoods travel further and faster than the truth.

       

 

  •  Dissertation
    • Practice! 52 minutes, 57 seconds
    • Maybe meeting with Wayne? Nope
  • Pack, move, unpack, setup
    • Bring ethernet cables! done
    • Moved out – done
    • Moved in – not done, but ready to unpack
  • Recovered my information for GSAW and TFDev
  • Write quick proposals for:
    • cybermap – done
    • Synthetic data as a service – done
    • White paper – kinda?