7:00 – 8:00, 3:00 – 4:00 Research

• This looks good: Bayesian data analysis for newcomers
• Also this; Seeing Theory
• I want to do a map that is based on population vs geography:
  • Gridded Population of the world (Matplotlib to generate image?). Can’t get the data directly. Need to see if available via UMBC (or maybe GLOBE?)
  • Wilbur terrain generation (installed. Will accept an image as the heightmap source)
• Tried using QT designer, but it can’t find the web plugin?
• Installing Python 3.6 on my home dev box
• Downloaded all the python code, and my simulation data. I want to be able to merge tables to produce networks that can then be plotted, so I think it’s mostly going to be installing things this morning
• NOTE: When installing Python, the only way to install for all users it to go through the advanced setup.
• Installing packages. CMD needs to run as admin, which blows.
• After some brief issues with the IDE not being set in structure, got all the pieces that use numpy, pandas and matplotlib running. That should be enough for table parsing (although there will be the excel reading and writing installs), though I still need to get started with graph-tool
• Paper was rejected – time to try ACM? LaTex format. Downloaded and compiled! Now I just have to move the text over? Wrap the existing text? That’s something for tomorrow.

8:30 – 2:30, BRC

• Working on table joins. That was pretty straightforward. Note that for column collision you have to provide a suffix. Makes me think that I want to compare across DataFrames instead

  eu.read_dataframe_excel(args.excelfile, None)
  cluster_df = eu.read_dataframe_sheet("Cluster ID")
  print("cluster_df")
  #  print(cluster_df)
  dist_df = eu.read_dataframe_sheet("Distance from mean center")
  print("dist_df")
  #  print(dist_df)
  merged_df = cluster_df.join(other=dist_df, lsuffix='_c', rsuffix='_d')
  print("merged_df")
  print(merged_df)

• So now that I can read in and analyze sheets, what am I trying to do?I think that for each time slice, and by cluster, produce a sorted list from most to least common membership.

Thoughts about CSCW ‘Fake News’ class

• Home to Roost
• Fake it to make it game
• The Law of Group Polarization
• Online clustering, fear and uncertainty in Egypt’s transition
• Study: Breitbart-led right-wing media ecosystem altered broader media agenda
• On the Milo Bus With the Lost Boys of America’s New Right
• Facebook’s guide to handling Fake News
• Junk News and Bots during the U.S. Election: What Were Michigan Voters Sharing Over Twitter? And here’s the full paper.
• Principles that motivate citizen behaviour according to Montesquieu
  • Driving each classification of political system, according to Montesquieu, must be what he calls a “principle”. This principle acts as a spring or motor to motivate behavior on the part of the citizens in ways that will tend to support that regime and make it function smoothly.
    • For democratic republics (and to a somewhat lesser extent for aristocratic republics), this spring is the love of virtue—the willingness to put the interests of the community ahead of private interests.
    • For monarchies, the spring is the love of honor—the desire to attain greater rank and privilege.
    • Finally, for despotisms, the spring is the fear of the ruler.

A political system cannot last long if its appropriate principle is lacking. Montesquieu claims, for example, that the English failed to establish a republic after the Civil War (1642–1651) because the society lacked the requisite love of virtue.

• My thoughts on Arendt’s Origin of Totalitarianism as it relates to information
• FiveThirtyEight’s take on r/TheDonald
• Information Wars: A Window into the Alternative Media Ecosystem
  • By Kate Starbird, (Scholar)
    Professor, Human Centered Design & Engineering, University of Washington
    Crisis Informatics, Crowdsourcing, Crowdwork, CSCW, Human Computer Interaction
• On Building a “Fake News” Classification Model
• Helena sent a nice link to Announcing New Research: “A Field Guide to Fake News” it’s from First Draft News, which purports to be Essential resources for reporting and sharing information that emerges online
• Defense Against the Dark Arts: Networked Propaganda and Counter-Propaganda
  • Jonathan Stray
• Filter bubbles, echo chambers, and online news consumption. Post is here
• Thom Lieb found this for me today: Reuters Tracer – A Large Scale System of Detecting Verifying Real-Time News Events from Twitter. Downloaded. A must read.

7:00 – 8:00 Research

• This Is Why Trump’s Conspiracy Theories Work, Say Experts
• Good advice on writing papers
• Setting up to get clustering data.
  • Explore/Exploit ratio 50/50
  • Explore will be in the flocking range: 0.1 – 1.6
  • Exploit will be in the echo chamber stage: 3.2 – 10.0
  • Cluster EPS of 0.25 gives good diversity
  • Create a DataFrame from the “Cluster ID” sheet
  • Ran the sims, but the default destination is wrong. Re-running
  • Discriminate between explorer and exploiter cluster membership over time
    • Clusters the agent belonged to
    • Will need to post-process to fix cluster switching. Probably taking all the cluster average positions and applying the a common ID if the difference between the samples is less than a given delta

8:30 – 3:30 BRC

• Generating data for clustering
• Will try to read in and merge the DataFrames so that I have position (angle or origin distance) to calculate group persistence
• Human Motion Recognition Using Isomap and Dynamic Time Warping
• Fixing DataFrames. Mostly this is a case of bad data handling. This is in the data:

  359000.00,836,2,2,4,0,1,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,#<Geocoder::Result::Bing:0x007fe9a1b39718>,#<Geocoder::Result::Bing:0x007fe9a1b39150>

  Which means that we have to handle ‘#<Geocoder::Result::Bing:0x007fe9a1b39718>,#<Geocoder::Result::Bing:0x007fe9a1b39150>. I am very disappointed in this book

• Giving up. Wrote a review. Monday I’ll try doing joins on Dataframes. That being said, I learned a lot on how to check for errors.
  ‘

7:00 – 8:00 Research

• Reading the HCIC Boaster Poster description. Downloaded to HCIC 2017 folder
  • A “boaster-poster” is a poster that describes your most current research endeavor and/or interest. The idea is to foster dialogue about your topic of interest/research so you can meet like-minded HCIC 2017 attendees. Format for a “boaster-poster” is as follows: a short description of your perspective and interest in this area, plus a description of your work in form of a single page (8.3 × 11.7 inches) poster. Boaster-posters offer an opportunity to showcase the work of new and experienced authors alike. You can use images and text to frame and illustrate your ideas. A list with boaster-poster titles, authors & abstracts will be distributed at the conference, and the posters will be available for view at the HCIC conference. We strongly encourage all student attendees to submit a boaster to HCIC, as boaster authors will have opportunities across the conference to discuss their work with other attendees through a new interactive format for 2017.
    Boaster-poster deadline: June 2nd, 2017
    A pdf that includes:
    • A cover page with
      • Title, author(s) (indicate those available to chat at meeting)
      • At least three keywords
      • A 150 word abstract
    • A draft of your poster
  • So something like “sociophysics-informed design“? I’m thinking that if I can take agent cluster membership and use that to construct a social network graph, I could show something that looks like this: 
  • Maybe use graph-tool Python library? 
  • Need to look at Zappos and McMaster websites as examples of explorational interfaces
  • Facebook’s guide to handling Fake News. High effort. I wonder what kind of feedback mechanisms there are?

8:30 – 6:00 BRC

• Sprint planning
• Doctor visit, 10:15 – 11:00
• Discussion with Aaron about visualizing high-dimensional clusters in low-dimensional space for intuitive understanding
• Working through Thoughtful Machine Learning. Very disappointed. The code in GitHub doesn’t match the book, doesn’t even have an entry point, and blows up in the init. Sad! Here’s the offending line (df is the read-in DataFrame):

  df = (df - df.mean()) / (df.max() - df.min())

• Learning more about the pandas DataFrame here so maybe I can fix the above.
• Actually, Skillport has useful stuff, but all the videos crash before the end
• The problem is that the floating point values in the file are being read in as string values, and crashing the calculation. I’ve tried doing an apply function that changes the value but it doesn’t result in the type change. Going to try changing everything to float tomorrow.
• Helped Aaron break down the tasking for this sprint’s efforts.

7:00 – 8:00 Research

• Just found the HCIC Boaster Poster description
• Resistbot is kind of along the lines of what I was thinking about anonymous news input. And article on chatbot technology from CIO. One of the interesting platforms is ChatFuel, which has a non-programming (example-based) creation process. It’s really tied into FB. Not sure I want to do that without setting up a specific account.
• Gupshup is another bot system that deploys to a lot of platforms (FB, twitter, slack, etc)
• NLP/NLU services. Here’s Google’s documentation for NLP and Prediction, which seems related
• Downloading the QT community IDE. Not sure if it has designer or not. Also, there was only the option to download versions 5.x, so the V4 options of pyqt may be problematic. Big install. Lots of documentation though it looks like.
• Downloaded and running. Tutorials tomorrow

8:30 – 1:00 BRC

• Adding stats dataframe – done. Interesting results in the test128 data: 
• Sprint grooming. And while that’s running, reading Thoughtful Machine Learning. Downloaded accompanying code from GitHub

7:00 – 8:00 Research

• Information ecology thoughts. A lot of what I am going can be framed from this perspective. Note that collective intelligence and knowledge ecosystem are referenced on the Wikipedia page
• Working on card

  

• Junk News and Bots during the U.S. Election: What Were Michigan Voters Sharing Over Twitter? And here’s the full paper.
  • Reading the paper. This also makes me think about Kate Starbird‘s work. She frames her data looking at the Boston bombing, but if you look at Google trends, you’ll see that ‘False Flag’ goes at least as far back as the Aurora shootings. What I thought of as ‘patient zero’, gamergate, is later than both of those events: 

8:30 – 4:00 BRC

• Running clustering on local machine.
• Need to add a stats DataFrame that has eps, min_size, c.num_clusters, c.get_num_clustered, c.get_num_items. Write that out at the end. Tomorrow
• Added reading of csv file and conversion to a DataFrame xlsx. Played around with that in Excel. Some nice results if I take the log of the value. Need to try clustering on that, but I want to add the stats output first 
• Jonker-Volgenant Algorithm + t-SNE = Super Powers. THis could be a way of producing a map view for the research browser

7:00 – 8:30, 3:00 – 6:00 Research

• Continuing submission. Figuring out PhySH. That’s a pretty nice system!
• Done!

  05Apr2017 es2017apr05_602 Physical Review E (Regular Article)
  Status: Submitted 10Apr2017-07:31 EDT
  Title: Modeling the Law of Group Polarization
  Authors: Feldman,Philip / Engel,Don

• Starting on business cards

9:00 – 2:30 BRC

• Sprint retrospective
• I seem to have lost the cluster I ran last week. Rerunning. I thought there might be an error since there were no (-1), but it’s because the DataFrame has the values of the item replaced with the cluster ID +2. Which means that a row that has a clusterID of zero, it is completely empty, and +1 is unclustered.

8:00pm – 12:00pm, plus an hour on the 9th

• Spent much time trying to figure out why the PDF wouldn’t print on the website from the LaTex file. As a result, I learned how to read the log file, fixed some image scaling issues, and learned to be VERY careful in checking the cases in the files. Also, as per here, I learned how to take the bib file and convert it to a bbl file, which can then be pasted into the tex file resulting in a single file for submission. No errors, and only minor quibbles on citations!

BRC

• And this – Open sourcing Sonnet – a new library for constructing neural networks
  • It’s now nearly a year since DeepMind made the decision to switch the entire research organisation to using TensorFlow (TF). It’s proven to be a good choice – many of our models learn significantly faster, and the built-in features for distributed training have hugely simplified our code. Along the way, we found that the flexibility and adaptiveness of TF lends itself to building higher level frameworks for specific purposes, and we’ve written one for quickly building neural network modules with TF. We are actively developing this codebase, but what we have so far fits our research needs well, and we’re excited to announce that today we are open sourcing it. We call this framework Sonnet. 
  • Code page

7:15 – 8:15 Research

• So this happened: Dozens of U.S. Missiles Hit Air Base in Syria. Wonder how it will play out. Like Infinite Reach? That actually had more justification, since Americans were killed in the preceding events…
• Direction matching for sparse movement datasets: determining interaction rules in social groups Data is here, in the very cool Movebank. Reminds me of GLOBE
• Still working on getting paper submitted. Word is hanging, which is weird.
• The APS journal submissions login page is here

• Length Check
  Please be aware that this is only an estimate and meant to help avoid delays associated with excessive length.
  
  Journal/article type is not length constrained (4281)
  
   *** Word count calculation may be inaccurate for the following reasons ***
     * No acknowledgment environment found - acknowledgments may have been counted
     * No bibliography environment found - references may have been counted
  
    Figure   Aspect Ratio   Wide?   Word Equivalent
       1         1.33         No        132
       2         0.98         No        173
       3         0.93         No        181
       4         1.01         No        168
       5         1.75         No        105
       6         1.67         No        109
       7         2.99         No         70
       8         1.22         No        142
       9         0.95         No        177
      10         1.66         No        110
  
  WORD COUNT SUMMARY
      Note: Text word count excludes title, abstract, byline, PACS,
            receipt date, acknowledgments, and references.
  
                Text word count   2914 
          Equations word equiv.      0 
            Figures word equiv.   1367
             Tables word equiv.      0
                                ------
                         TOTAL    4281 (Maximum length is unlimited)

• Getting PDF conversion errors. Not sure what to do next…

  WARNING: Supplemental file: ModelingGroupPolarizationNotes.bib.
  Checking figure order in 'ModelingGroupPolarization.tex'...
  PDF file creation failed:
  ERROR: Following figure files not called into TeX:
  DTWmatrix.png,toolScreenshot.png,ExplorerPDF.png
  AND TeX file calls in the following missing figure files:
  toolscreenshot.png,explorerPDF.png,DTWMatrix.png
  Please check and fix these file names.

8:30 – 2:30 BRC

• Imperative programming in TensorFlow
• Working on reading in the old integrity as a DataFrame from Excel
• Had to install xlrd
• TypeError: Image data can not convert to float. This pattern seems useful:

  mat = df.as_matrix()  # get the data matrix
  mat = mat.astype(np.float64)  # force it to float64
  df.update(mat)  # replace the 'float' mat with the 'float64' mat

• Well, it was before, but now it won’t update the int64 to float 64. This worked though. When in doubt build a new mat

  mat = mat.astype(np.float64)  # force it to float64
  indices = df.index.values
  cols = df.columns.values
  df = pandas.DataFrame(mat, indices, cols)

• Got the data in though. Interesting.This is the densest corner of the sorted matrix: 

8:30 – 5:30 BRC

• Worked too late yesterday and slept in. Will try to get the submission issues worked on once the sprint review gets going
• Added code that deletes previous file if it exists
• refactored the fitness landscape test to be 100 eps increments by 10 cluster increments, starting at the min_cluster. Seems to produce more useful results
  

  

  

   

• Sprint review at 9:30, 11:00, 1:00 Done! Went over my stuff too fast.
  • Need to figure out a way to cluster diagnosis codes
  • Discovered that the data is still the same integrity data from the last sprint. Need to write a read_dataframe(file_name: str) -> pandas.DataFrame: method to hdfs_csv_reader  tomorrow.

7:00 – 8:00 Research

• Finishing up poster? 
• Starting submission for Phys Rev E, based on my notes here.
  • Created account
  • Created ORCID account
  • Started submission. Errors!There were problems with the following fields:
    • AllExploit_SI0.0.psd: Classification description value must be selected for each file and cannot be left unknown.
    • 90_ExploitR10_10_ExploreR0.psd: Classification description value must be selected for each file and cannot be left unknown.
    • polarized.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • clusterMembership.png: Classification description value must be selected for each file and cannot be left unknown.
    • toolScreenshot.png: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization2col.pdf: Classification description value must be selected for each file and cannot be left unknown.
    • exploring.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization.pdf: Classification description value must be selected for each file and cannot be left unknown.
    • psheader.txt: Classification description value must be selected for each file and cannot be left unknown.
    • influenced.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • ExplorerPDF.png: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization.synctex.gz: Classification description value must be selected for each file and cannot be left unknown.
    • AllExploit_SI10.0.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarizationDraft.pdf: Classification description value must be selected for each file and cannot be left unknown.
    • flocking.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • DTWmatrix.png: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization.log: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization.dvi: Classification description value must be selected for each file and cannot be left unknown.
    • AllExploit_SI10.0.psd: Classification description value must be selected for each file and cannot be left unknown.
    • ModelingGroupPolarization.blg: Classification description value must be selected for each file and cannot be left unknown.
    • 90_ExploitR10_10_ExploreR0.jpg: Classification description value must be selected for each file and cannot be left unknown.
    • AllExploit_SI0.2.psd: Classification description value must be selected for each file and cannot be left unknown.
    • SimilarPaths.png: Classification description value must be selected for each file and cannot be left unknown.
    • RevTex41_example.aux: Classification description value must be selected for each file and cannot be left unknown.
    • label 0 is duplicated
    • ModelingGroupPolarization.aux: Classification description value must be selected for each file and cannot be left unknown.
    • revtex41_template.blg: Classification description value must be selected for each file and cannot be left unknown.
    • RevTex41_example.blg: Classification description value must be selected for each file and cannot be left unknown.
    • DTWclusters.png: Classification description value must be selected for each file and cannot be left unknown.
    • Main text file is required

9:00 – 5:00, 6:00, 7:30 BRC

• Working on HDFS reading and writing
• Integrating code
• Compiled! Now waiting for things to blow up
• Success! After fixing this:

  writer.write(u'%s' % cstr) # good
  writer.write('%s', cstr) # bad
  

7:00 – 8:30 Research

• Add Amundson/Scott and implications for design. Done, but too many ‘incorporate’. Need better words. Scott * Amundsen look good though
• SpaCy – Industrial-Strength Natural Language Processing in Python. Video tutorial
• Got Thoughtful Machine Learning with Python. Looks like it hits all the bases from classification to unit testing

9:00 – 5:30 BRC

• Need to submit paperwork for collective intelligence
• Discussion with Aaron and Jeremy about curation webapp
• Test re-hydration code. Had a few minor issues in the code that produced the tests, but all working now
• Start on HDFS? Aaron is close to testing his Python code. Integration after that?

7:00 – 8:30, 3:00 -4:00 Research

• Finished the first cut at the Illustrator version of the poster. I think I like it better? Everything fits! 
•  Fika

8:30 – 2:30

• Hotel for Collective intelligence!
• Read data into DataFrame – done!
• Next step is to tie this into HDFS and then PyUnit?

Keeping this for reference. From Wikipedia:

Principles that motivate citizen behaviour according to Montesquieu

Driving each classification of political system, according to Montesquieu, must be what he calls a “principle”. This principle acts as a spring or motor to motivate behavior on the part of the citizens in ways that will tend to support that regime and make it function smoothly.

• For democratic republics (and to a somewhat lesser extent for aristocratic republics), this spring is the love of virtue—the willingness to put the interests of the community ahead of private interests.
• For monarchies, the spring is the love of honor—the desire to attain greater rank and privilege.
• Finally, for despotisms, the spring is the fear of the ruler.

A political system cannot last long if its appropriate principle is lacking. Montesquieu claims, for example, that the English failed to establish a republic after the Civil War (1642–1651) because the society lacked the requisite love of virtue.

7:00 – 8:00 Research

• Some back-and forth with Wane on him attending the CI conference.
• Nice talk with Don yesterday. Played with LaTex and discussed how, what and why to change behaviors of the particles going forward.
• Started in the Illustrator version of the poster.
• In Defense of Interactive Graphics

8:30 – 4:30 BRC

• Working on hdfs_csv_reader.py
  • Write out sparse file queue – done
  • Read in queue – partial. Currently at line 60:

    print(filtered) # TODO: go and read all these files. Will need to be HADOOP-compatable later

  • Assemble DataFrame
  • By the way, this line of code in Python:

    filtered = [dir_name+"/"+name for name in all_files if file_prefix in name]

  • Is pretty much the same as this piece of C code. I guess some things have gotten better?

    char** buildFilteredList(char* dirName, char* filePrefix, char* allFiles){
    	// count the number of spaces
    	int i;
    	int filecount = 0; 
    	for (i = 0; i < strlen(allFiles); ++i){
    		if(allFiles[i] == ' ')
    			++fileCount;
    	}
    	char *fileList[fileCount];
    	int numchars = strlen(filePrefix);
    	filecount = 0; 
    	for (i = 0; i < strlen(allFiles); ++i){
    		char* sptr = allFiles[i];
    		if(strncmp(filePrefix, sptr, numchars) == 0)
    			fileList[filecount++] = sptr;	
    	}
    	return fileList;
    }

• Long discussion with Aaron about ResearchBrowser design. We’re thinking about SPAs communicating through a lightweight chrome (and other(?)) plugin that mediates communication between tabs
  • chrome.contextMenus
  • python db access

7:00 – 8:00, 4:00 – 6:00

• Looking more closely at Qt and PyQt. First, the integrated browser is nice. Second, if it’s possible to wireframe UIs in Qt and connect them to Python for matrix calculations and server interactions, then I have some real leverage.
• Really good overview. Difference between 4 and 5, etc.
  • Qt is very C++. Here’s the documentation link, which looks great
  • PyQt appears immense. There is a commercial license ($500)
    • Is there a tutorial specifically for PyQt5? (StackOverflow)
    • The documentation
      • It will integrate with Qt Designer, which is nice and doesn’t look horrible
    • A lot of examples
      • Web browser
• Lotsa python and machine learning videos. These look promising
• Meeting with Don

8:30 – 3:30 BRC

• Cleaning up reading code and adding argument handling
• Need to add a row-reader to replace the slow_read_pbf and the slow_write_pbf methods. They also need to turn the row into a JSON object, and to have a separate method that will assemble a DataFrame from a set of JSON objects in memory
• Nope, strike that. We’re now going to read CSV files containing sparse data and assemble them into a DataFrame. The format is:

  "member:123456",CNR:3,CPD:9,TAS:1,AFB:38,FUD:1
  "member:789012",PRK:1,TAS:2

• The row id is quoted, otherwise key:value pairs are comma separated. Rows are terminated by CR/LF, and there can be multiple rows per file.
• Python hadoop: Pydoop