Monthly Archives: December 2016

Phil 12.30.16

7:00 – 8:00, 8:30 – 5:00 Research

  • Cool thing. Maybe for Ravi?
  • Moving speed to FlockingBeiliefCA and adding slewRate – done. Added an update of the global belief to the agent manager behavior loop.
  • Calculate average heading
  • Calculate average center
  • Add decay based on distance?
    • Sorted list of boids by distance
    • Within a sphere of influence? the vector to the target is scaled by something like 1/distance. Done! looks great!
  • TODO:
    • Move localBehavior to FlockingBeiliefCA. pass in array of other beliefs and maybe the influence diameter
    • Make FlockingShape clusterable based on norms() to other Shape’s beliefs
    • For charting, calculate distances and direction cosine from average center. Export that to excel. We’ll need a sampling interval.
    • Start thinking about network connections

Phil 12.29.16

7:00 – 8:00, 8:30 -3:30 Research

  • But first, we update IntelliJ…
  • The goal today is to get boids to set up the methods that control behavior:
    • Alignment: steer towards the average heading of local flockmates
    • Cohesion: steer to move toward the average position of local flockmates
    • Separation: steer to avoid crowding local flockmates
      • I’m not sure about this one. In the case of a belief system, there really isn’t a need for collision avoidance.
  • Finishing ParticleBelief.interpolateHeading().
  • Changed the vector so that it is always unit, and added  speed variable for the particle belief

boids_12-29-16

Phil 12.28.16

7:00 – 8:00, 8:30 – 4:00 Research

  • Thom Lieb found this for me today: Reuters Tracer – A Large Scale System of Detecting Verifying Real-Time News Events from Twitter. Downloaded. A must read.
  • Need to renew my cert for philfeldman.com
  • Realized last night that I should be able to maintain angle and length for my rotation through ‘n’ dimensions between two vectors.
    • Get the angle between the two vectors (dot product)
    • Get the chord between the two normalized vectors
    • Now I have a 2D problem where I project the angle I want to rotate on the chord line. That gives me a new vector that I just need to normalise.
    • Solving simultaneous equations with matrices
    • The only condition where I see this failing is a 180 degree rotation.
    • Need to verify that this will work. It took forever for stupid math reasons but it does work.
    • Reworking ParticleBelief.interpolateHeading()
  • No pretty screenshots for today. Instead, here’s a murmuration of starlings

Magic Cloud from Marco Campazas on Vimeo.

Phil 12.27.16

Phil 7:15 – 8:15, 8:45 -4:15 Research

  • Adding start/stop Done. Should probably add a slider to adjust time
  • Implementing boids as per Craig Reynolds page
    • The flocking shape needs an orientation. Since this orientation can be any number of dimensions, where each statement is a dimension, I’m creating a ParticleBelief to contain ParticleStatements.
    • Getting the heading and rebuilding shape as boid triangles
    • Drawing triangles now. Using the Javafx examples here. Got angle working
    • Adding individual flocks
    • Working on an angle interpolation that will work for any number of dimensions. Going to start with interpolation and renormalization, since the steps should be incremental. I think it’s done. Will test tomorrow. Need to add the parts to the dprint?

boids_12-27-16

Phil 12.26.16

7:00 – 12:00 Research

  • Winterize lawn mower – done!
  • Look into hopkins computational and regular ecology
  • Downloaded the new version of AtlasTi. It’s on OneDrive
  • The goal is to start the development of the flocking app. I’m going to start with a new FlockingGui. Done, now switching all the underlying pieces over. I think done for the day is when the agents are moving based on their belief statement value of xpos and ypos.
  • Update Java – no need
  • Update IntelliJ – downloaded locally and installed. Odd problem with FILE_PATH not being set in Settings->Appearance & Behavior -> Path Variables. This was just for JavaUtils and seems fine in GroupPolarizationModel. Anyway, everything is compiling and running.
  • Update TortoiseSVN – no need
  • Update WinSCP -no need
  • Copy over java code – done
  • Ok, the first thing to do is have an agent that takes its position from an xMapping() and yMapping() function. To begin with, this will be taking an xpos and ypos statement that vary as a function of time.
  • Currently the movement is calculated in the shape, since for force clustering the beliefs/statements don’t know their ‘positions’. This approach is different in that the beliefs are a measure of position in information space. Working through the changes

Phil 12.23.16

7:00 – 8:00 Research

8:30 – 4:00 BRC

  • Reading in a spreadsheet to GPM -done. Ok results, not great clustering
  • Tried adjusting the threshold and adding antibelief, but other than helping the refresh rate, no joy. I think I need a different distance/similarity calculation
    • Hops. Every agent is connected over the network by going through flag nodes. We could literally draw the node/agent network, and count the hops in an adjacency matrix
    • N-dimensional cartesian. As I recall, this is close to what I have already. It’s closely related to n-dimensional flocking, so I’m going to get that running anyway so that I can measure distance between/within flocks
    • Cosine similarity. I think that this is a good approach since it decomposes the dimensions in a useful way, particularly for sparse matrices.
    • There is similarity there, but not enough distance to make anything emerge. Pretty pix though.
  • integrityagents

Phil 12.22.16

7:00 – 8:00 Research

  • Met with Wayne yesterday evening. We’re going to take a look at science team data text to see how it compares with the overall coding by humans. Verified that the data is all available
  • Interesting stuff on NPR this morning on Russia: The information space opens wide asymmetrical possibilities for reducing the fighting potential of the enemy. In North Africa, we witnessed the use of technologies for influencing state structures and the population with the help of information networks. It is necessary to perfect activities in the information space, including the defense of our own objects [objectives].
  • Continuing with Sociophysics
    • Chapter 8: Endnote
      • Definition of consensus in an opinion model – the emergence of long-range order.
      • Looking for phase changes from heterogeneous to homogeneous or clustered states is important. Finding what parameters are causal and the values is considered a publishable result. Canonical types of transitions, such as the percolation threshold are discussed in the appendices.

BRC 8:30 – 4:30

  • Verify that the META_INF file in src isn’t screwing jar file creation. Deleted, with the same behavior. Sigh
  • Add fields for renaming columns. Will probably have to save the data out as XML to keep the relationship/mapping?
  • Find the code that strips off the common leading text (in GoogleCSE2?) Done
  • Started to work on clustering with Moby Dick and brought Aaron into the conversation to think about clustering issues – how to make like items gather together with other like items. NMF kind of does this by filling in latent values, but the question is where to cluster on
  • Finally read in the integrity data and it did not look good. I realize now that a matrix made up completely of zeros and ones will not be handled well by NMF since it will try to make all the cells one based on the models’s mechanism of treating zeroes as empty cells.
  • After talking to Aaron about it for a while, I think the better way to cluster will be based on the Group Polarization model. Need to be able to bring in that spreadsheet and then write out a report. Also, look at the high-dimension flocking.

Phil 12.21.16

7:00 – 8:00 Research

8:30 – 4:00 ASRC

Phil 12.20.16

7:00 – 8:00 School

  • Looks like I need to keep track of my hours better for next year. Getting started now
  • Continuing with Sociophysics

8:30 – 4:30 ASRC

  • Sorting TableColumn(s)- done. The trick is to change the name and the cellValueFactory: 
    tc.setText(colName);
tc.setCellValueFactory(new MapValueFactory<>(colName));
  • Sorting rows is more straightforward. Just got a list of the sorted row sums and reordered based on that.
  • Need to update the value in the ‘selected field’ textarea.
  • Verify that the cells are tracking
  • Add a tab so that it’s possible to switch between the original and the product matrices
  • Add an Edit/recalculate capability
    • Tweak original matrix
    • Adjust k
    • Column clustering/renaming

Phil 12.19.16

7:15 – 4:15 ASRC

  • Continuing with Sociophysics
    • Chapter 7:  of flocks, flows and transports [page 179]
    • Boids (Flocks, herds and schools: A distributed behavioral model – Craig Reynolds):
      • Try to avoid collisions with other boids (repulsion)
      • Attempt to match velocity with neighboring boids
      • attempt to stay close to nearby boids
    • If the collision avoidance is taken out and the number of dimensions increased, then this could be the model. Rather than the flock converging around a position, look at the distances between the individuals using DBSCAN and cluster.
    • Density and noise need to be independent variables and saved on runs. This would also be true in information space. You can have high organization in high density, low noise states. Thinking about that, this also implies one of the emergent properties of an information bubble is the low noise. Even though the environment may be very noisy, the bubble isn’t.
    • As with the other social models, individuals can have weight. That way the flock can have leaders and followers. (See Misinformed leaders lose influence over pigeon flocks to inform the model)
    • Also, I like the idea of a social network being built from belief proximity, which raises the cost for switching to another flock, even if they are nearby. It could be that once a social network forms that anti-belief repulsion starts to play a role.
  • BRC
    • Updating intellij and Java.
    • Intellij failed to patch. Odd. Tried again and it worked.
    • Working on getting tables to update
      • Clear() – done
      • Load – done
      • Select row and modify – done
      • Working on columns and cells
      • Need to sort by row and column. Do this as part of the update() process

Phil 12.18.16

1:30 – 4:00 School. Rain, rain, rain.

  • Continuing with Sociophysics
    • 6.5 Is it really a small world? Searching post Milgram
      • 6.5.8 Funneling properties.
          • The funneling capability of a node can be defined as the fraction of successful dynamic paths through it when the target is fixed and the source is varied. Two thoughts: First, this seems to be a measurement of centrality. Second, Large, vague nodes are needed for ‘laundering’ information into misinformation or conspiracy theory.
          • Consider four agents. Who have characteristics that can vary between (0, 1).
            • Agent 1 has two color intensities: R=0.1, G= 0.7
            • Agent 2 has one color and two note volumes R=0.3, A=0.2, F=0.6
            • Agent 3 also has one color and two note volumes B=0.4, D=1, E=0.2
            • Agent 4 has three notes A=0.3, D=0.4, E=0.5
          • Let’s assume that funneling is not required if agents share a color or note. This means that A4 can get to A1 through A2, but A3 has to get to A1 via A4 and then A2. In a matrix this looks like
        R G B A D E F
        Agent1 0.1 0.7
        Agent2 0.3 0.2 0.6
        Agent3 0.4 1.0 0.2
        Agent4 0.3 0.4 0.5
          • But if we add the hypernyms Color and Notes, we can get funneling. I am summing the color and notes to give a sense of the agent’s ‘projection’ into the larger, more general space. I think the ‘size’ of the funnels are the number of items that go in them times the range of each item. So Color would have a range of (0, 3) and Notes would have a range of (0, 4), since I’m not including B, C, and G here:
        R G B A D E F Color Notes
        Agent1 0.1 0.7 0.8
        Agent2 0.3 0.2 0.6 0.3 0.8
        Agent3 0.4 1.0 0.2 0.4 1.2
        Agent4 0.3 0.4 0.5 1.2
          • Now agents 2 and 3 can get to each other through either Color or note in two hops, and the Agents 1 and 4 can reach each other by going through each of the funnels.
          • There should be a cost in using a funnel though. You loose the information about which color or which note. Intuitively, a series of steps with non-funnel links should be somehow more specific than the same number of steps through a funnel.
          • Practical uses would be a way to detect poorly reasoned conclusions, as long as the beginning and end of the train of thought could be identified.
    • Knowing a network by walking on it: emergence of scaling (Alexei Vázquez) Looks like an interesting guy with a wide range of publications.

Phil 12.16.16

Phil 7:00 – 4:00 ASRC

  • Continuing with Sociophysics
    • Social Phenomena on complex networks
      • Opinion and community formation in coevolving networks (Gerardo Iñiguez González)
        • Abstract: In human societies opinion formation is mediated by social interactions, consequently taking place on a network of relationships and at the same time influencing the structure of the network and its evolution. To investigate this coevolution of opinions and social interaction structure we develop a dynamic agent-based network model, by taking into account short range interactions like discussions between individuals, long range interactions like a sense for overall mood modulated by the attitudes of individuals, and external field corresponding to outside influence. Moreover, individual biases can be naturally taken into account. In addition the model includes the opinion dependent link-rewiring scheme to describe network topology coevolution with a slower time scale than that of the opinion formation. With this model comprehensive numerical simulations and mean field calculations have been carried out and they show the importance of the separation between fast and slow time scales resulting in the network to organize as well-connected small communities of agents with the same opinion.
      • Citing paper: Effects of deception in social networks (Gerardo Iñiguez González)<— Important???
        • Abstract: Honesty plays a crucial role in any situation where organisms exchange information or resources. Dishonesty can thus be expected to have damaging effects on social coherence if agents cannot trust the information or goods they receive. However, a distinction is often drawn between prosocial lies (‘white’ lies) and antisocial lying (i.e. deception for personal gain), with the former being considered much less destructive than the latter. We use an agent-based model to show that antisocial lying causes social networks to become increasingly fragmented. Antisocial dishonesty thus places strong constraints on the size and cohesion of social communities, providing a major hurdle that organisms have to overcome (e.g. by evolving counter-deception strategies) in order to evolve large, socially cohesive communities. In contrast, white lies can prove to be beneficial in smoothing the flow of interactions and facilitating a larger, more integrated network. Our results demonstrate that these group-level effects can arise as emergent properties of interactions at the dyadic level. The balance between prosocial and antisocial lies may set constraints on the structure of social networks, and hence the shape of society as a whole.
    • 6.5 Is it really a small world? Searching post Milgram
      • In the introduction to this section [page 168], the authors say a very interesting thing: “Although the network may have the small world property, searches are usually done locally: the individual may not know the global structure of the network that would help them find the shortest path to the target node“. I think that they are talking about social networks explicitly here, but the same concept applies to an information network. This is a network description of the information horizon problem. You can’t find what you can’t see, at least in a broad outline.
      • Also this: “Searching can regarded as a learning process; repeating the search several times can avoid infinite loops and lead to better solutions”
    • Sprint Planning Meeting
      • Clustering is my only task for the sprint
    • Writing out factored matrix
    • Put together test case term/doc spreadsheet. There are several test cases with and without randomly generated zeros. The goal is to determine the best way to cluster docs. Do we use the product mat? The factor mats? Only one way to find out.
    • Working on sorting the L2mat by row sum or column sum

Phil 12.15.16

7:00 – 5:30 ASRC

  • This came across my Twitter feed this morning:  Indivisible: A Practical Guide for Resisting the Trump Agenda. It’s written by Hill staffers (Apparently), and says something interesting about the Tea Party:
    • They were locally focused. The Tea Party started as an organic movement built on small local groups of dedicated conservatives. Yes, they received some support/coordination from above, but fundamentally all the hubbub was caused by a relatively small number of conservatives working together. To summarize:
      • Groups started as disaffected conservatives talking to each other online. In response to the 2008 bank bailouts and Obama’s election, groups began forming to discuss their anger and what could be done. They eventually realized that the locally-based discussion groups themselves could be a powerful tool.
      • Groups were small, local, and dedicated. Local Tea Party groups could be fewer than 10 people, but they were highly localized and dedicated significant personal time and resources. Members communicated with each other regularly, tracked developments in Washington, and coordinated advocacy efforts together.
      • Groups were relatively small in number. The Tea Party was not hundreds of thousands of people spending every waking hour focused on advocacy. Rather, the efforts were somewhat modest. Only 1 in 5 self-identified Tea Partiers contributed money or attended events. On any given day in 2009 or 2010, only twenty local events–meetings, trainings, townhalls, etc–were scheduled nationwide. In short, a relatively small number of groups were having a big impact on the national debate.
    • In reading this, I hear several things
      • Group polarization can start or be furthered by small groups. Indeed, it’s the embodiment of the Margaret Mead quote: Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it’s the only thing that ever has. This also jibes with Sunstein’s statement that small, polarized groups can act like incubators.
      • Though polarized, the members “tracked developments in Washington”, so they were informed. I’d really like to know their sources of information.
      • Scattered groups that are loosely coupled may have a bigger impact than a large single group.
  • Continuing with Sociophysics
    • Social Phenomena on complex networks
    • Dynamical Processes on Complex Networks. Got the Kindle edition so now I can search! Interesting section: 10.6 Coevolution of opinions and network
    • Similar chapter in this book – Social Phenomena on coevolutionary networks [pg 166]. One of the interesting things here is the use of the iterated prisoner’s dilemma. On a network, the agents typically calculate and aggregate payoff and imitate the strategy of the neighbor with the best payoff. In the coevolutionary model, an agent can cut off the link to a defector with a probability. This seems a bit like polarization, where the group severs ties with entities with sufficiently divergent views (and individuals leave when the group becomes too extreme)
    • Coevolution of agents and networks: Opinion spreading and community disconnection Abstract: We study a stochastic model for the coevolution of a process of opinion formation in a population of agents and the network which underlies their interaction. Interaction links can break when agents fail to reach an opinion agreement. The structure of the network and the distribution of opinions over the population evolve towards a state where the population is divided into disconnected communities whose agents share the same opinion. The statistical properties of this final state vary considerably as the model parameters are changed. Community sizes and their internal connectivity are the quantities used to characterize such variations.
  • adam g. dunn
    clinical epidemiology and medical informatics. Looking for PhD’s working on misinformation
  • Sprint grooming
  • Got the data for the npi_raw_integrity_ci_tbl_eligibility_chiroacup_polyrx table. Turning into pivot tables.
  • Can now read a single matrix into NmFModelGui, factor, and build the product matrix.

Phil 12.14.16

7:00 – 6:00 ASRC

  • Continuing with Sociophysics
    • Social Phenomena on complex networks
    • Loops of nodes behave differently from trees. what to do about that? I think loops drive the echo chamber process? It is, after all, feedback..
    • There is also a ‘freezing’ issue, where a stable state is reached where two cliques containing different states are lightly connected, but not enough that the neighbors in one clique can be convinced to change their opinion [Fig. 6.2, pg 135]
    • Residual Energy: The difference between the actual energy and the known energy of the perfectly-ordered ground state (full consensus).
  • BRC
    • Need to not split quoted columns
    • Generate a matrix where flags that are Yes -> 1 and empty/null -> o
    • Retrospective
    • Had a thought that NMF might work in tensors as well. I need to rewrite the gradient descent so that it takes an arbitrary number of dimension.
    • Meeting with Nir. Sold him on clustering.

Phil 12.13.16

7:00 – 5:30 ASRC

  • Added a page for my model notes
  • Continuing with Sociophysics
  • Integrity meeting
    • Bellrock needs a summary of why no HCAHPS display. We could put something really simple that does not get rolled into the scoring of the impactors. Like claim numbers. Katy suggests a ‘ticker’ (Sparkline?)  of claim volume/amounts
    • Need Gregg’s suggestion of what the ‘hot button’ indicators should be (action item), based on the claims data.
    • Small number of items that we can be tracking the actual values of (no calculation) that we can roll up and display.
    • NPPS as an input that gets added to stand in for self-reported flags??
    • Goal of the system is to decide whether the users should spend coordination time on expensive patients.
    • NDC – National Drug Code. Counts of drugs by claim period. Where does this come from? Counts of denied?
    • Claim period is monthly?
  • Having issues with getting lines read cleanly. For the time being, I’m going to throw away the bad lines, but later, I want to make persistent objects and get the data from postgres directly.
  • Tensor spectral clustering for partitioning higher-order network structures
  • Multilinear PageRank: In this paper, we first extend the celebrated PageRank modification to a higher-order Markov chain. Although this system has attractive theoretical properties, it is computationally intractable for many interesting problems. We next study a computationally tractable approximation to the higher-order PageRank vector that involves a system of polynomial equations called multilinear PageRank. This is motivated by a novel “spacey random surfer” model, where the surfer remembers bits and pieces of history and is influenced by this information. The underlying stochastic process is an instance of a vertex-reinforced random walk. We develop convergence theory for a simple fixed-point method, a shifted fixed-point method, and a Newton iteration in a particular parameter regime. In marked contrast to the case of the PageRank vector of a Markov chain where the solution is always unique and easy to compute, there are parameter regimes of multilinear PageRank where solutions are not unique and simple algorithms do not converge. We provide a repository of these non-convergent cases that we encountered through exhaustive enumeration and randomly sampling that we believe is useful for future study of the problem