7:00 – 8:00 Research

• Change influence multiplier to another social influence horizon
• Add a LINEAR and EXPONENTIAL mapping to the WeightWidget
• Rolled up the configuration variables into a single class and broke a lot of things. Fixed now

Call Senator Cardin about obstruction – done

8:30 – 5:00 BRC

• Downloaded the new data
• Extracting clusterable data from view
• low values appear to be best for these clusters
• Exploded excel
• Exploded postgres “org.postgresql.util.PSQLException: ERROR: tables can have at most 1600 columns”
• Need to try MySql. Enough columns for now. O just write out the row names and clusters. Except we can’t do a sniff test that way Grr.
• Sprint 13 recovery meeting
• Fixed tasks in Jira

7:00 – 8:00 Research

• Added dimensions. The behaviors from 2D are evident at up to 10 dimensions, though everything takes longer. It is hard to see what’s going on in my simple 2D mapping scheme.
• After playing with the explorer settings some yesterday, I’m not sure what to do with it. I think the abstract instead should just consider the initial modelling of group polarization in a homogeneous population. There are three conditions:
  • Isolated – Each agent only pays attention to its previous state. Creates a uniform random distribution. I think this is the archetypal ‘explorer’ pattern, where path through information space is not affected by social activity
  • Limited social visibility – Each agent can see other agents up to a specified distance. This produces multiple flocks (usually one large and several smaller) that orbit a center.
  • Infinite social visibility – Every agent can see every other agent. This leads to one large flock that moves in a straight line
•  So yesterday, I was thinking that the most ‘socially sensitive’ agents would be the explorers, but after writing the above, I realise that it’s the reverse. Explorers are least socially sensitive. To test this, I set the ‘explorer multiple’ to a very low value and cranked up the social visibility for the rest of the agents. This creates a new behavior, where the tightly clustered social agents are more ‘anchored’ to the environment and stay closer to the center of the stage, depending on local concentrations of explorers. Interestingly, this implies that the greater percentage of explorers (up to a point?) means a more grounded group of social ‘exploiters’. Which is pretty interesting.
• Going to try to get ahold of Don to go over these results. Sent email
• Another thought. In addition to the average center, do the variance? Not sure how to do this. Axis-by-axis? Actually, that would work nicely, with center and variance by dimension.

8:30 – 4:00 BRC

• Working on clustering the current data, waiting on DB updates
• Took care of my fall 2016 and spring 2017 education paperwork
• Asked Gregg for the  updated DB
• Set up LMN to access the DB.
• Have jdbc running. I feel so ’90s
• Built the view:

  CREATE VIEW v_fused AS
    SELECT ecp.mbrnum,
      claim.chargeamount, claim.benefittype, claim.physicianname, claim.provpayeename, claim.provfirstname, claim.provzp,
      claim.rxdrugname,
      claim.diagnosiscode1, claim.diagnosiscode2, claim.diagnosiscode3, claim.diagnosiscode4, claim.otherdiagnosiscodes,
      coveragetype, membergender, flab_hrt, flag_acuterenal, flag_bactinf, flag_cerebrovascular,
      flag_chf, flag_ckd, flag_cnc, flag_copd, flag_fluidelec, flag_htn, flag_otherheart, flag_pleurisy,
      flag_respfailure, flag_surgothercomps, flag_whitebloodcell
    FROM tbl_eligibility_chiroacup_polyrx AS ecp
      JOIN tbl_medicalclaims_chiroacup_polyrx claim on ecp.mbrnum = claim.mbrnum;

• Now I need to code up the extractor. Change all the Yes -> 1, NULL -> 0, build a map of all the strings and make a column for each, then map into that as a Labled2DMatrix, with the row being mbrnum. Then create a spreadsheet. Might be too big. Maybe create a table and read the table directly into the Labled2DMatrix. Kinda like that option…

11:00 – 2:00 Research

• It is currently 19 degrees F, which is way too cold to play outside. So…
• Make a Explorer and Exploiter flock – done
• Make a slider that sets the ratio of Explorer to Exploiter – done
• Make a slider that sets the Explorer as a multiple of the Exploiter – done. Not sure that these shouldn’t just be set explicitly
• Add some dimensions
• Added ConfigRecordingClass to hold all these values and pass them around
  • Bug where a paused sim is still recording
  • ratios are in and working
  • Need to add code implementing
    • multipliers – done
    • dimensions

7:00 – 8:00 Research

• Wiring up the recorder – Done!
• Need to add a deltatime component and ComboBox to set the sampling step size
• Results!

8:30 – 5:00 BRC

• Implement the search on the top terms so we can use google to figure out meaning
• Start looking at data and cluster.
• The integrity test data is big and too slow for interactive rates. Added an ‘interactive’ check.
• Make NMF calculation optional by checkbox. If it’s not checked, make source and target matrices the same
• Make a fake data table for DBSCAN that’s 5kx5x dimension, see if the three clusters can be found. Not too bad. 3k x3k should take about 90 seconds
• Made NMF an option so that we can just cluster

7:00 – 8:00 Research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Working on class FlockRecorder

8:30 – 4:00 BRC

• Add weight widgets to JavaUtils2
• Working on getting the clusters into the display – Done
• Saving spreadsheets – Done!
• Sorting rows in the map isn’t working right. Commented out for now

That’s a lot of datagrids

7:00 – 8:00 Research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Adding class FlockRecorder

8:30 – BRC

• Talked with Aaron about [R]. Generated spreadsheets with 2 nd 10 dimension clusters similar to what I’m testing on
• Bring in R book tomorrow
• Incorporating clustering into the GUI
  • Need to add EpsPercent – Done
  • Need to choose the matrix (raw, product, row factor, column factor
  • Also add clustering to the NMF calculation rather than taking the top 3
  • Start with output to the text console? Also to spreadsheet
  • Might build a map and populate a table
• Need to submit forms for reimbursement and new class
• Need to mail insurance – nope, need to erase more and print out mom’s will

5:00 Dinner at Jeff’s

Scan will and print out. Erase all pencil from blue form

First, Call about the House ethics bill

7:00 – 8:00 research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Adding class FlockRecorder

8:30 – 4:00 BRC

• Added method to nmf so that matrices with zeros don’t have to be sparse
• Working on clustering items in a Labeled2DMatrix. Done as per this page
• Had some issues getting the test to work. I may have been trying to be too general. With restructured tests from the GP code, everything seems to be working fine.
• Next step is to put into the GUI and figure out some way to show it.

9:00 – 6:00 Research

• Create WeightWidget to support loading from files
• Added interactive setting of influence distance
• Set color based on cluster Done
• Move localBehavior to FlockingBeiliefCA. pass in array of other beliefs and maybe the influence diameter. Done
• Make FlockingShape clusterable based on the Euclidian distance to other Shape’s beliefs, Done!
• Chasing down NaN errors. Looks to be mostly length = zero and div/zero errors. Occasional. Grr.
• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Start thinking about network connections
• Notes for tomorrow:
  • Make the items in the term/document table clusterable
  • Make it so zero can indicate empty or a value

7:00 – 8:00, 8:30 – 5:00 Research

• Cool thing. Maybe for Ravi?
• Moving speed to FlockingBeiliefCA and adding slewRate – done. Added an update of the global belief to the agent manager behavior loop.
• Calculate average heading
• Calculate average center
• Add decay based on distance?
  • Sorted list of boids by distance
  • Within a sphere of influence? the vector to the target is scaled by something like 1/distance. Done! looks great!
• TODO:
  • Move localBehavior to FlockingBeiliefCA. pass in array of other beliefs and maybe the influence diameter
  • Make FlockingShape clusterable based on norms() to other Shape’s beliefs
  • For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
  • Start thinking about network connections

7:00 – 8:00, 8:30 -3:30 Research

• But first, we update IntelliJ…
• The goal today is to get boids to set up the methods that control behavior:
  • Alignment: steer towards the average heading of local flockmates
  • Cohesion: steer to move toward the average position of local flockmates
  • Separation: steer to avoid crowding local flockmates
    • I’m not sure about this one. In the case of a belief system, there really isn’t a need for collision avoidance.
• Finishing ParticleBelief.interpolateHeading().
• Changed the vector so that it is always unit, and added  speed variable for the particle belief

7:00 – 8:00, 8:30 – 4:00 Research

• Thom Lieb found this for me today: Reuters Tracer – A Large Scale System of Detecting Verifying Real-Time News Events from Twitter. Downloaded. A must read.
• Need to renew my cert for philfeldman.com
• Realized last night that I should be able to maintain angle and length for my rotation through ‘n’ dimensions between two vectors.
  • Get the angle between the two vectors (dot product)
  • Get the chord between the two normalized vectors
  • Now I have a 2D problem where I project the angle I want to rotate on the chord line. That gives me a new vector that I just need to normalise.
  • Solving simultaneous equations with matrices
  • The only condition where I see this failing is a 180 degree rotation.
  • Need to verify that this will work. It took forever for stupid math reasons but it does work.
  • Reworking ParticleBelief.interpolateHeading()
• No pretty screenshots for today. Instead, here’s a murmuration of starlings

Magic Cloud from Marco Campazas on Vimeo.

Phil 7:15 – 8:15, 8:45 -4:15 Research

• Adding start/stop Done. Should probably add a slider to adjust time
• Implementing boids as per Craig Reynolds page
  • The flocking shape needs an orientation. Since this orientation can be any number of dimensions, where each statement is a dimension, I’m creating a ParticleBelief to contain ParticleStatements.
  • Getting the heading and rebuilding shape as boid triangles
  • Drawing triangles now. Using the Javafx examples here. Got angle working
  • Adding individual flocks
  • Working on an angle interpolation that will work for any number of dimensions. Going to start with interpolation and renormalization, since the steps should be incremental. I think it’s done. Will test tomorrow. Need to add the parts to the dprint?

7:00 – 12:00 Research

• Winterize lawn mower – done!
• Look into hopkins computational and regular ecology
• Downloaded the new version of AtlasTi. It’s on OneDrive
• The goal is to start the development of the flocking app. I’m going to start with a new FlockingGui. Done, now switching all the underlying pieces over. I think done for the day is when the agents are moving based on their belief statement value of xpos and ypos.
• Update Java – no need
• Update IntelliJ – downloaded locally and installed. Odd problem with FILE_PATH not being set in Settings->Appearance & Behavior -> Path Variables. This was just for JavaUtils and seems fine in GroupPolarizationModel. Anyway, everything is compiling and running.
• Update TortoiseSVN – no need
• Update WinSCP -no need
• Copy over java code – done
• Ok, the first thing to do is have an agent that takes its position from an xMapping() and yMapping() function. To begin with, this will be taking an xpos and ypos statement that vary as a function of time.
• Currently the movement is calculated in the shape, since for force clustering the beliefs/statements don’t know their ‘positions’. This approach is different in that the beliefs are a measure of position in information space. Working through the changes

7:00 – 8:00 Research

• Assembling all the Sociophysics readings into one posting on phlog.

8:30 – 4:00 BRC

• Reading in a spreadsheet to GPM -done. Ok results, not great clustering
• Tried adjusting the threshold and adding antibelief, but other than helping the refresh rate, no joy. I think I need a different distance/similarity calculation
  • Hops. Every agent is connected over the network by going through flag nodes. We could literally draw the node/agent network, and count the hops in an adjacency matrix
  • N-dimensional cartesian. As I recall, this is close to what I have already. It’s closely related to n-dimensional flocking, so I’m going to get that running anyway so that I can measure distance between/within flocks
  • Cosine similarity. I think that this is a good approach since it decomposes the dimensions in a useful way, particularly for sparse matrices.
  • There is similarity there, but not enough distance to make anything emerge. Pretty pix though.
• 

7:00 – 8:00 Research

• Met with Wayne yesterday evening. We’re going to take a look at science team data text to see how it compares with the overall coding by humans. Verified that the data is all available
• Interesting stuff on NPR this morning on Russia: The information space opens wide asymmetrical possibilities for reducing the fighting potential of the enemy. In North Africa, we witnessed the use of technologies for influencing state structures and the population with the help of information networks. It is necessary to perfect activities in the information space, including the defense of our own objects [objectives].
  • Does Russia Have a Gerasimov Doctrine?
  • The Value of Science is in the Foresight – NATO analysis
• Continuing with Sociophysics
  • Chapter 8: Endnote
    • Definition of consensus in an opinion model – the emergence of long-range order.
    • Looking for phase changes from heterogeneous to homogeneous or clustered states is important. Finding what parameters are causal and the values is considered a publishable result. Canonical types of transitions, such as the percolation threshold are discussed in the appendices.

BRC 8:30 – 4:30

• Verify that the META_INF file in src isn’t screwing jar file creation. Deleted, with the same behavior. Sigh
• Add fields for renaming columns. Will probably have to save the data out as XML to keep the relationship/mapping?
• Find the code that strips off the common leading text (in GoogleCSE2?) Done
• Started to work on clustering with Moby Dick and brought Aaron into the conversation to think about clustering issues – how to make like items gather together with other like items. NMF kind of does this by filling in latent values, but the question is where to cluster on
• Finally read in the integrity data and it did not look good. I realize now that a matrix made up completely of zeros and ones will not be handled well by NMF since it will try to make all the cells one based on the models’s mechanism of treating zeroes as empty cells.
• After talking to Aaron about it for a while, I think the better way to cluster will be based on the Group Polarization model. Need to be able to bring in that spreadsheet and then write out a report. Also, look at the high-dimension flocking.