7:00 – 8:00 Research

• Add a frame sampling text field. Default of 10? Zero = sample. Hitting the limit = reset sampleIncrement to zero. Done. Found a bug that screwed up the average center calculation:

  globalCenter.add(arv);

  Should have been

  globalCenter = globalCenter.add(arv);

• Set up page that shows the 5th, 50th and 95th percentiles for each statement’s population sample. Put in a new tab.
• Add a tab that shows the 5th/95th – the 50th? That could sum to a nice chart
• The above two items will need a Collection of Percentiles for all agents positions at a step across all dimensions. Tomorrow

9:00 – 5:00BRC

• Try using MySQL instead of postgres. Nope, pointless
• Maybe save as CSV? Changed the toString() method to be a toWriter(Writer writer) method. Now toString calls that with a StringWriter and toCSV calls it with a BufferedWriter()
• Use “output folder” as database/schema. Then session becomes the table. Will need to check that the db/schema exists and create as needed
• Output the list of items with clusterId(+1) for Bob
• Reduce the size of the matrix by eliminating all non-cluster members? Can certainly do this in the display of the cluster matrix.
• Left with the NMF app cooking for a 4-level deep set of matrices. Hmm. Blew up with NaNs. Test cases work fine. I’ll try scaling up slowly

7:00 – 8:00 Research

• Change influence multiplier to another social influence horizon
• Add a LINEAR and EXPONENTIAL mapping to the WeightWidget
• Rolled up the configuration variables into a single class and broke a lot of things. Fixed now

Call Senator Cardin about obstruction – done

8:30 – 5:00 BRC

• Downloaded the new data
• Extracting clusterable data from view
• low values appear to be best for these clusters
• Exploded excel
• Exploded postgres “org.postgresql.util.PSQLException: ERROR: tables can have at most 1600 columns”
• Need to try MySql. Enough columns for now. O just write out the row names and clusters. Except we can’t do a sniff test that way Grr.
• Sprint 13 recovery meeting
• Fixed tasks in Jira

7:00 – 8:00 Research

• Added dimensions. The behaviors from 2D are evident at up to 10 dimensions, though everything takes longer. It is hard to see what’s going on in my simple 2D mapping scheme.
• After playing with the explorer settings some yesterday, I’m not sure what to do with it. I think the abstract instead should just consider the initial modelling of group polarization in a homogeneous population. There are three conditions:
  • Isolated – Each agent only pays attention to its previous state. Creates a uniform random distribution. I think this is the archetypal ‘explorer’ pattern, where path through information space is not affected by social activity
  • Limited social visibility – Each agent can see other agents up to a specified distance. This produces multiple flocks (usually one large and several smaller) that orbit a center.
  • Infinite social visibility – Every agent can see every other agent. This leads to one large flock that moves in a straight line
•  So yesterday, I was thinking that the most ‘socially sensitive’ agents would be the explorers, but after writing the above, I realise that it’s the reverse. Explorers are least socially sensitive. To test this, I set the ‘explorer multiple’ to a very low value and cranked up the social visibility for the rest of the agents. This creates a new behavior, where the tightly clustered social agents are more ‘anchored’ to the environment and stay closer to the center of the stage, depending on local concentrations of explorers. Interestingly, this implies that the greater percentage of explorers (up to a point?) means a more grounded group of social ‘exploiters’. Which is pretty interesting.
• Going to try to get ahold of Don to go over these results. Sent email
• Another thought. In addition to the average center, do the variance? Not sure how to do this. Axis-by-axis? Actually, that would work nicely, with center and variance by dimension.

8:30 – 4:00 BRC

• Working on clustering the current data, waiting on DB updates
• Took care of my fall 2016 and spring 2017 education paperwork
• Asked Gregg for the  updated DB
• Set up LMN to access the DB.
• Have jdbc running. I feel so ’90s
• Built the view:

  CREATE VIEW v_fused AS
    SELECT ecp.mbrnum,
      claim.chargeamount, claim.benefittype, claim.physicianname, claim.provpayeename, claim.provfirstname, claim.provzp,
      claim.rxdrugname,
      claim.diagnosiscode1, claim.diagnosiscode2, claim.diagnosiscode3, claim.diagnosiscode4, claim.otherdiagnosiscodes,
      coveragetype, membergender, flab_hrt, flag_acuterenal, flag_bactinf, flag_cerebrovascular,
      flag_chf, flag_ckd, flag_cnc, flag_copd, flag_fluidelec, flag_htn, flag_otherheart, flag_pleurisy,
      flag_respfailure, flag_surgothercomps, flag_whitebloodcell
    FROM tbl_eligibility_chiroacup_polyrx AS ecp
      JOIN tbl_medicalclaims_chiroacup_polyrx claim on ecp.mbrnum = claim.mbrnum;

• Now I need to code up the extractor. Change all the Yes -> 1, NULL -> 0, build a map of all the strings and make a column for each, then map into that as a Labled2DMatrix, with the row being mbrnum. Then create a spreadsheet. Might be too big. Maybe create a table and read the table directly into the Labled2DMatrix. Kinda like that option…

11:00 – 2:00 Research

• It is currently 19 degrees F, which is way too cold to play outside. So…
• Make a Explorer and Exploiter flock – done
• Make a slider that sets the ratio of Explorer to Exploiter – done
• Make a slider that sets the Explorer as a multiple of the Exploiter – done. Not sure that these shouldn’t just be set explicitly
• Add some dimensions
• Added ConfigRecordingClass to hold all these values and pass them around
  • Bug where a paused sim is still recording
  • ratios are in and working
  • Need to add code implementing
    • multipliers – done
    • dimensions

7:00 – 8:00 Research

• Wiring up the recorder – Done!
• Need to add a deltatime component and ComboBox to set the sampling step size
• Results!

8:30 – 5:00 BRC

• Implement the search on the top terms so we can use google to figure out meaning
• Start looking at data and cluster.
• The integrity test data is big and too slow for interactive rates. Added an ‘interactive’ check.
• Make NMF calculation optional by checkbox. If it’s not checked, make source and target matrices the same
• Make a fake data table for DBSCAN that’s 5kx5x dimension, see if the three clusters can be found. Not too bad. 3k x3k should take about 90 seconds
• Made NMF an option so that we can just cluster

7:00 – 8:00 Research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Working on class FlockRecorder

8:30 – 4:00 BRC

• Add weight widgets to JavaUtils2
• Working on getting the clusters into the display – Done
• Saving spreadsheets – Done!
• Sorting rows in the map isn’t working right. Commented out for now

That’s a lot of datagrids

7:00 – 8:00 Research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Adding class FlockRecorder

8:30 – BRC

• Talked with Aaron about [R]. Generated spreadsheets with 2 nd 10 dimension clusters similar to what I’m testing on
• Bring in R book tomorrow
• Incorporating clustering into the GUI
  • Need to add EpsPercent – Done
  • Need to choose the matrix (raw, product, row factor, column factor
  • Also add clustering to the NMF calculation rather than taking the top 3
  • Start with output to the text console? Also to spreadsheet
  • Might build a map and populate a table
• Need to submit forms for reimbursement and new class
• Need to mail insurance – nope, need to erase more and print out mom’s will

5:00 Dinner at Jeff’s

Scan will and print out. Erase all pencil from blue form

First, Call about the House ethics bill

7:00 – 8:00 research

• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Adding class FlockRecorder

8:30 – 4:00 BRC

• Added method to nmf so that matrices with zeros don’t have to be sparse
• Working on clustering items in a Labeled2DMatrix. Done as per this page
• Had some issues getting the test to work. I may have been trying to be too general. With restructured tests from the GP code, everything seems to be working fine.
• Next step is to put into the GUI and figure out some way to show it.

9:00 – 6:00 Research

• Create WeightWidget to support loading from files
• Added interactive setting of influence distance
• Set color based on cluster Done
• Move localBehavior to FlockingBeiliefCA. pass in array of other beliefs and maybe the influence diameter. Done
• Make FlockingShape clusterable based on the Euclidian distance to other Shape’s beliefs, Done!
• Chasing down NaN errors. Looks to be mostly length = zero and div/zero errors. Occasional. Grr.
• For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
• Start thinking about network connections
• Notes for tomorrow:
  • Make the items in the term/document table clusterable
  • Make it so zero can indicate empty or a value