# Phil 11.13.19

7:00 – 3:00 ASRC

3rd Annual DoD AI Industry Day

From Stewart Russell, via BBC Business Daily and the AI Alignment podcast:

Although people have argued that this creates a filter bubble or a little echo chamber where you only see stuff that you like and you don’t see anything outside of your comfort zone. That’s true. It might tend to cause your interests to become narrower, but actually that isn’t really what happened and that’s not what the algorithms are doing. The algorithms are not trying to show you the stuff you like. They’re trying to turn you into predictable clickers. They seem to have figured out that they can do that by gradually modifying your preferences and they can do that by feeding you material. That’s basically, if you think of a spectrum of preferences, it’s to one side or the other because they want to drive you to an extreme. At the extremes of the political spectrum or the ecological spectrum or whatever image you want to look at. You’re apparently a more predictable clicker and so they can monetize you more effectively.

So this is just a consequence of reinforcement learning algorithms that optimize click-through. And in retrospect, we now understand that optimizing click-through was a mistake. That was the wrong objective. But you know, it’s kind of too late and in fact it’s still going on and we can’t undo it. We can’t switch off these systems because there’s so tied in to our everyday lives and there’s so much economic incentive to keep them going.

So I want people in general to kind of understand what is the effect of operating these narrow optimizing systems that pursue these fixed and incorrect objectives. The effect of those on our world is already pretty big. Some people argue that operation’s pursuing the maximization of profit have the same property. They’re kind of like AI systems. They’re kind of super intelligent because they think over long time scales, they have massive information, resources and so on. They happen to have human components, but when you put a couple of hundred thousand humans together into one of these corporations, they kind of have this super intelligent understanding, manipulation capabilities and so on.

• Predicting human decisions with behavioral theories and machine learning
• Behavioral decision theories aim to explain human behavior. Can they help predict it? An open tournament for prediction of human choices in fundamental economic decision tasks is presented. The results suggest that integration of certain behavioral theories as features in machine learning systems provides the best predictions. Surprisingly, the most useful theories for prediction build on basic properties of human and animal learning and are very different from mainstream decision theories that focus on deviations from rational choice. Moreover, we find that theoretical features should be based not only on qualitative behavioral insights (e.g. loss aversion), but also on quantitative behavioral foresights generated by functional descriptive models (e.g. Prospect Theory). Our analysis prescribes a recipe for derivation of explainable, useful predictions of human decisions.
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent’s observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at this http URL.

# Phil 10.28.19

Capacity, Bandwidth, and Compositionality in Emergent Language Learning

• Many recent works have discussed the propensity, or lack thereof, for emergent languages to exhibit properties of natural languages. A favorite in the literature is learning compositionality. We note that most of those works have focused on communicative bandwidth as being of primary importance. While important, it is not the only contributing factor. In this paper, we investigate the learning biases that affect the efficacy and compositionality of emergent languages. Our foremost contribution is to explore how capacity of a neural network impacts its ability to learn a compositional language. We additionally introduce a set of evaluation metrics with which we analyze the learned languages. Our hypothesis is that there should be a specific range of model capacity and channel bandwidth that induces compositional structure in the resulting language and consequently encourages systematic generalization. While we empirically see evidence for the bottom of this range, we curiously do not find evidence for the top part of the range and believe that this is an open question for the community.

• In the early 60s, Robert Axelrod was a math major messing around with refrigerator-sized computers. Then a dramatic global crisis made him wonder about the space between a rock and a hard place, and whether being good may be a good strategy. With help from Andrew Zolli and Steve Strogatz, we tackle the prisoner’s dilemma, a classic thought experiment, and learn about a simple strategy to navigate the waters of cooperation and betrayal. Then Axelrod, along with Stanley Weintraub, takes us back to the trenches of World War I, to the winter of 1914, and an unlikely Christmas party along the Western Front.
• Need to send a note for them to look into Axelrod’s “bully” saddle point

7:00 – ASRC GOES

• Dissertation – Nearly done with the agent cartography section?
• CTO Rehearsal – 10:30 – 12:00 done
• ML Dinner – 4:30 fun!
• Meeting With Aaron M
• More thinking about what to do with the paper. We decided to try for the CHI4EVIL workshop, and then try something like IEEE Spectrum. I think I’d like to reframe it around the concept of Expensive Information and Automation. Try to tie together AI weapons, spam filters, and deepfakes
• Automation makes negotiation more difficult, locks in trajectories
• Handing off responsibility to automation amplifies opportunities and destructive potential
• OODA loop could be generalized if you look at it from the perspective of attention.

# Phil 10.26.19

The dynamics of norm change in the cultural evolution of language

• What happens when a new social convention replaces an old one? While the possible forces favoring norm change—such as institutions or committed activists—have been identified for a long time, little is known about how a population adopts a new convention, due to the difficulties of finding representative data. Here, we address this issue by looking at changes that occurred to 2,541 orthographic and lexical norms in English and Spanish through the analysis of a large corpora of books published between the years 1800 and 2008. We detect three markedly distinct patterns in the data, depending on whether the behavioral change results from the action of a formal institution, an informal authority, or a spontaneous process of unregulated evolution. We propose a simple evolutionary model able to capture all of the observed behaviors, and we show that it reproduces quantitatively the empirical data. This work identifies general mechanisms of norm change, and we anticipate that it will be of interest to researchers investigating the cultural evolution of language and, more broadly, human collective behavior.

When Hillclimbers Beat Genetic Algorithms in Multimodal Optimization

• It has been shown in the past that a multistart hillclimbing strategy compares favourably to a standard genetic algorithm with respect to solving instances of the multimodal problem generator. We extend that work and verify if the utilization of diversity preservation techniques in the genetic algorithm changes the outcome of the comparison. We do so under two scenarios: (1) when the goal is to find the global optimum, (2) when the goal is to find all optima.
A mathematical analysis is performed for the multistart hillclimbing algorithm and a through empirical study is conducted for solving instances of the multimodal problem generator with increasing number of optima, both with the hillclimbing strategy as well as with genetic algorithms with niching. Although niching improves the performance of the genetic algorithm, it is still inferior to the multistart hillclimbing strategy on this class of problems.
An idealized niching strategy is also presented and it is argued that its performance should be close to a lower bound of what any evolutionary algorithm can do on this class of problems.

# Phil 10.10.19

7:00 – 4:00 ASRC GOES

• The Daily has an episode on how to detach from environmental reality and create a social reality stampede
• Dissertation, working on finishing up the “unexpected findings” piece of the research plan
• Tie together explore/exploit, the Three Patterns, and M&R three behaviors.
• Also, set up the notion that it was initially explore OR exploit, with no thought given to the middle ground. M&R foreshadowed that there would be, though
• Registered for Navy AI conference Oct 22
• Get together with Vadim to see how the physics are going on Tuesday?
• More evolver
• installed the new timeseriesML2
• The test run blew up with a tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at cwise_ops_common.cc:82 error. Can’t find any direct help, though maybe try this?
• Reduce your Batchsize of datagen.flow (by default set 32 so you have to set 8/16/24 )
• Figured it out – I’m saving models in memory. Need to write them out instead.
• Swing by campus and check on Will

# Phil 7.19.19

7:00 – 4:30 ASRC GEOS

• Still looking at what’s wrong with my NK model. I found Random Boolean Networks, when looking for “random binary networks kauffman example“. It also has a bibliography that looks helpful as well
• Introduction to Random Boolean Networks
• The goal of this tutorial is to promote interest in the study of random Boolean networks (RBNs). These can be very interesting models, since one does not have to assume any functionality or particular connectivity of the networks to study their generic properties. Like this, RBNs have been used for exploring the configurations where life could emerge. The fact that RBNs are a generalization of cellular automata makes their research a very important topic. The tutorial, intended for a broad audience, presents the state of the art in RBNs, spanning over several lines of research carried out by different groups. We focus on research done within artificial life, as we cannot exhaust the abundant research done over the decades related to RBNs.
• I can add a display that shows this:
• Got that working
• Rewrote so that there is an evolve without a fitness test. Trying to set up transition patterns like this:
• The thing is, I don’t see how the K part works here…
• I think I got it working!
• Complex and Adaptive Dynamical Systems: A Primer
• An thorough introduction is given at an introductory level to the field of quantitative complex system science, with special emphasis on emergence in dynamical systems based on network topologies. Subjects treated include graph theory and small-world networks, a generic introduction to the concepts of dynamical system theory, random Boolean networks, cellular automata and self-organized criticality, the statistical modeling of Darwinian evolution, synchronization phenomena and an introduction to the theory of cognitive systems.
It inludes chapter on Graph Theory and Small-World Networks, Chaos, Bifurcations and Diffusion, Complexity and Information Theory, Random Boolean Networks, Cellular Automata and Self-Organized Criticality, Darwinian evolution, Hypercycles and Game Theory, Synchronization Phenomena and Elements of Cognitive System Theory.

# Phil 7.16.19

7:00 – 6:30ASRC GEOS

• Working more on NK Models. I have the original paper – Towards a general theory of adaptive walks on rugged landscapes, and I’ve pulled out my copy of The Origins of Order
• Determine if I have the evaluation function right
• Draw the networks
• Draw an N/K/Fitness landscape?
• As an aside, I think that an NK model can be modified to use backpropagation rather than mutation. That could be interesting.
• Ok, here’s everything working the way I think it should work, but I’m not sure it’s right….
• Need to get back to Antonio about authorship and roles. I think that it makes sense if he can get a sense of what – done
• Discovered the trumptwitterarchive, which is downloadable. Would like to build a network of the retweets and tagging by sentiment, gender and race.
• Code review with Chris. Unfortunately, it was more like an interrogation than a tour. My sense is that he was expecting us to ask questions and we were expecting a presentation.
• It went ok, but the audio connection was terrible

# Phil 5.28.19

Phil 7:00 – 5:00 ASRC NASA GEOS

• Factors Motivating Customization and Echo Chamber Creation Within Digital News Environments
• With the influx of content being shared through social media, mobile apps, and other digital sources – including fake news and misinformation – most news consumers experience some degree of information overload. To combat these feelings of unease associated with the sheer volume of news content, some consumers tailor their news ecosystems and purposefully include or exclude content from specific sources or individuals. This study explores customization on social media and news platforms through a survey (N = 317) of adults regarding their digital news habits. Findings suggest that consumers who diversify their online news streams report lower levels of anxiety related to current events and highlight differences in reported anxiety levels and customization practices across the political spectrum. This study provides important insights into how perceived information overload, anxiety around current events, political affiliations and partisanship, and demographic characteristics may contribute to tailoring practices related to news consumption in social media environments. We discuss these findings in terms of their implications for industry, policy, and theory
• More JASSS paper
• Installing new IntelliJ and re-indexing
• Discovered a few bugs with the JsonUtils.find. Fixed and submitted a version to StackOverflow. Eeeep!

# Phil 5.13.19

7:00 – 3:00 ASRC NASA GEOS-R

• Really good Ted Radio Hour on Jumpstarting Creativity
• More dissertation
• Integrating Artificial Intelligence into Weapon Systems is up!
• Played around with the speed of agents, hoping to make some super-slow agents, but the way that the speed is updated is an accumulation of speed influences set by the neighbors, based on influence distance. To do this, I need to set the minmax separate from the initial variance
• Help Aaron with MatrixScalar – Nope
• Slides for talk -Yep, all day

# Phil 5.6.19

7:00 – 5:00 ASRC GOES-R

• Finished the AI/ML paper with Aaron M over the weekend. I need to have him ping me when it goes in. I think it turned out pretty well, even when cut down to 7 pages (with references!! Why, IEEE, why?)
• Sent a copy to Wayne, and distributed around work. Need to put in on ArXiv on Thursday
• Starting to pull parts from phifel.com to make the lit review for the dissertation. Those reviews may have had a reason after all!
• And oddly (though satisfying), I wound up adding a section on Moby-Dick as a way of setting up the rest of the lit review
• More Matrix scalar class. Basically a satisfying day of just writing code.
• Need to fix IEEE letter and take a self-portrait. Need to charge up the good camera

# Phil 5.1.19

7:00 – 7:00 ASRC NASA AIMS

• Added lit review section to the dissertation, and put the seven steps of sectarianism in.
• Spent most of yesterday helping Aaron with TimeSeriesML. Currently working on a JSON util that will get a value on a provided path
• Had to set up python at the module and not project level, which was odd. Here’s how: www.jetbrains.com/help/idea/2016.1/configuring-global-project-and-module-sdks.html#module_sdk
• Done!
    def lfind(self, query_list:List, target_list:List, targ_str:str = "???"):
for tval in target_list:
if isinstance(tval, dict):
return self.dfind(query_list[0], tval, targ_str)
elif tval == query_list[0]:
return tval

def dfind(self, query_dict:Dict, target_dict:Dict, targ_str:str = "???"):
for key, qval in query_dict.items():
# print("key = {}, qval = {}".format(key, qval))
tval = target_dict[key]
if isinstance(qval, dict):
return self.dfind(qval, tval, targ_str)
elif isinstance(qval, list):
return self.lfind(qval, tval, targ_str)
else:
if qval == targ_str:
return tval
if qval != tval:
return None

def find(self, query_dict:Dict):
# pprint.pprint(query_dict)
result = self.dfind(query_dict, self.json_dict)
return result


• It’s called like this:
ju = JsonUtils("../../data/output_data/lstm_structure.json")
# ju.pprint()
print("result 1 = {}".format(result))
print("result 2 = {}".format(result))
• Here’s the results:
result 1 = [None, 12, 1]
result 2 = 666.0
• Got Aaron’s code running!
• Meeting with Joel
• A quicker demo that I was expecting, though I was able to walk through how to create and use Corpus Manager and LMN. Also, we got a bug where the column index for the eigenvector didn’t exist. Fixed that in JavaUtils.math.Labeled2DMatrix.java
• Meeting with Wayne
• Walked through the JASSS paper. Need to make sure that the lit review is connected and in the proper order
• Changed the title of the dissertation to
• Stampede Theory: Mapping Dangerous Misinformation at Scale
• Solidifying defense over the winter break, with diploma in the Spring
• Mentioned the “aikido with drones” concept. Need to make an image. Actually, I wonder if there is a way for that model to be used for actually getting a grant to explore weaponized AI in a way that isn’t directly mappable to weapons systems, but is close enough to reality that people will get the point.
• Also discussed the concept of managing runaway AI with the Sanhedrin-17a concept, where unanimous agreement to convict means acquittal.  Cities had Sanhedrin of 23 Judges and the Great Sanhedrin had 71 Judges en.wikipedia.org/wiki/Sanhedrin
• Rav Kahana says: In a Sanhedrin where all the judges saw fit to convict the defendant in a case of capital law, they acquit him. The Gemara asks: What is the reasoning for this halakha? It is since it is learned as a tradition that suspension of the trial overnight is necessary in order to create a possibility of acquittal. The halakha is that they may not issue the guilty verdict on the same day the evidence was heard, as perhaps over the course of the night one of the judges will think of a reason to acquit the defendant. And as those judges all saw fit to convict him they will not see any further possibility to acquit him, because there will not be anyone arguing for such a verdict. Consequently, he cannot be convicted.

# Phil 4.29.19

7:00 – 3:30 ASRC TL

• Register for Tech Summit – done
• Ask for a week of time to prep for talk – done
• Panos read the paper and has some suggestions. Need to implement
• This might be important: Neural Logic Machines
• We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning. NLMs exploit the power of both neural networks—as function approximators, and logic programming—as a symbolic processor for objects with properties, relations, logic connectives, and quantifiers. After being trained on small-scale tasks (such as sorting short arrays), NLMs can recover lifted rules, and generalize to large-scale tasks (such as sorting longer arrays). In our experiments, NLMs achieve perfect generalization in a number of tasks, from relational reasoning tasks on the family tree and general graphs, to decision making tasks including sorting arrays, finding shortest paths, and playing the blocks world. Most of these tasks are hard to accomplish for neural networks or inductive logic programming alone.
• Need to read the Nature “Behavior” paper. Notes probably go straight into the dissertation lit review – done
• Continuing to read Army of None, which is ridiculously good. This figure has been making me think:  This implies that the idea that a set of diverse ML systems all agreeing is a warning condition is worth exploring.
• Finished read through of Tao’s paper
• Need to find a cardiologist for Arpita

# Phil 4.23.19

7:00 – 5:30 ASRC TL

• Reading Army of None and realizing that incorporating AI is a stampede theory and diversity issue:
• This makes Aegis less like a finished product with a few different modes and more like a customizable system that can be tailored for each mission. Galluch explained that the ship’s doctrine review board, consisting of the officers and senior enlisted personnel who work on Aegis, begin the process of writing doctrine months before deployment. They consider their anticipated missions, intelligence assessments, and information on the region for the upcoming deployment, then make recommendations on doctrine to the ship’s captain for approval. The result is a series of doctrine statements, individually and in packages, that the captain can activate as needed during deployment. (Page 164)
• Doctrine statements are typically grouped into two general categories: non-saturation and saturation. Non-saturation doctrine is used when there is time to carefully evaluate each potential threat. Saturation doctrine is needed if the ship gets into a combat situation where the number of inbound threats could overwhelm the ability of operators to respond. “If World War III starts and people start throwing a lot of stuff at me,” Galluch said, “I will have grouped my doctrine together so that it’s a one-push button that activates all of them. And what we’ve done is we’ve tested and we’ve looked at how they overlap each other and what the effects are going to be and make sure that we’re getting the defense of the ship that we expect.” This is where something like Auto-Special comes into play, in a “kill or be killed” scenario, as Galluch described it. (Page 164)
• Extensive testing goes into ensuring that it works properly. Once the ship arrives in theater, the first thing the crew does is test the weapons doctrine to see if there is anything in the environment that might cause it to fire in peacetime, which would not be good. This is done safely by enabling a hardware-level cutout called the Fire Inhibit Switch, or FIS. The FIS includes a key that must be inserted for any of the ship’s weapons to fire. When the FIS key is inserted, a red light comes on; when it is turned to the right, the light turns green, meaning the weapons are live and ready to fire. When the FIS is red—or removed entirely—the ship’s weapons are disabled at the hardware level. (Page 165)
• But the differences run deeper than merely having more options. The whole philosophy of automation is different. With Aegis, the automation is used to capture the ship captain’s intent. In Patriot, the automation embodies the intent of the designers and testers. The actual operators of the system may not even fully understand the designers’ intent that went into crafting the rules. The automation in Patriot is largely intended to replace warfighters’ decision-making. In Aegis, the automation is used to capture warfighters’ decision-making. (Page 165)
• Hawley argued that Army Patriot operators train in a “sham environment” that doesn’t accurately simulate the rigors of real-world combat. As a result, he said “the Army deceives itself about how good their people really are. . . . It would be easy to believe you’re good at this, but that’s only because you’ve been able to handle the relatively non-demanding scenarios that they throw at you.” Unfortunately, militaries might not realize their training is ineffective until a war occurs, at which point it may be too late. (Page 171)
• Hawley explained that the Aegis community was partially protected from this problem because they use their system day in and day out on ships operating around the globe. Aegis operators get “consistent objective feedback from your environment on how well you’re doing,” preventing this kind of self-deception. The Army’s peacetime operating environment for the Patriot, on the other hand, is not as intense, Hawley said. “Even when the Army guys are deployed, I don’t think that the quality of their experience with the system is quite the same. They’re theoretically hot, but they’re really not doing much of anything, other than just monitoring their scopes.” Leadership is also a vital factor. “Navy brass in the Aegis community are absolutely paranoid” about another Vincennes incident, Hawley said. (Page 171)
• Working on JASS paper
• Working on AI paper
• Long chat with Eric H

# Phil 4.22.19

7:00 – 4:00 ASRC TL

• The mission of the Conference on Truth and Trust Online (TTO) is to bring together all parties working on automated approaches to augment manual efforts on improving the truthfulness and trustworthiness of online communications.
• The inaugural Truth and Trust Online conference will be taking place on October 4th and 5th 2019 at BMA House in London.
•

### Key Dates

• First call for papers: 2nd of April, 2019 *

• Deadline for all submissions: 3rd of June, 2019
• Notification of acceptance: Early July
• Registration opens: End of June
• Conference: 4th and 5th of October, 2019, BMA House, London, UK
• From On Being with Pádraig Ó Tuama, about belonging gone bad and the scale of sectarianism:
• Fooling automated surveillance cameras: adversarial patches to attack person detection
• Adversarial attacks on machine learning models have seen increasing interest in the past years. By making only subtle changes to the input of a convolutional neural network, the output of the network can be swayed to output a completely different result. The first attacks did this by changing pixel values of an input image slightly to fool a classifier to output the wrong class. Other approaches have tried to learn “patches” that can be applied to an object to fool detectors and classifiers. Some of these approaches have also shown that these attacks are feasible in the real-world, i.e. by modifying an object and filming it with a video camera. However, all of these approaches target classes that contain almost no intra-class variety (e.g. stop signs). The known structure of the object is then used to generate an adversarial patch on top of it.
• In this paper, we present an approach to generate adversarial patches to targets with lots of intra-class variety, namely persons. The goal is to generate a patch that is able successfully hide a person from a person detector. An attack that could for instance be used maliciously to circumvent surveillance systems, intruders can sneak around undetected by holding a small cardboard plate in front of their body aimed towards the surveillance camera. From our results we can see that our system is able significantly lower the accuracy of a person detector. Our approach also functions well in real-life scenarios where the patch is filmed by a camera. To the best of our knowledge we are the first to attempt this kind of attack on targets with a high level of intra-class variety like persons.
• More adding Wayne’s notes into JASS paper. Figured out how to make something that looks like blockquotes without screwing up the JASS formatting:
\hspace{1cm}\begin{minipage}{\dimexpr\textwidth-2cm}
\textit{"Get him home.  And deliver my cut of earnings to the people of Phandalin near Neverwinter, my home". With this, before anyone can stop him, Edmund turns to the dragon. "I make a counter offer.  In exchange for them motions to the two caged people. I offer myself to take their place.  I will remain.  I will starve.  You will lose two peasants, and in return you will gain all that I have to offer.  Edmund of house DeVir of Neverwinter.  The last of a noble bloodline of the ruling class."} - Edmond: Group 2
\end{minipage}
• More Machine Teaching paper

# Phil 4.15.19

7:00 – ASRC TL

• I’ve been hunting around for what a core message of the iSchool should be (And I like LAMDA), but I think this sums it up nicely. From The Library Book:
• use arxiv2bibtex to get bibtex information for arXiv submissions for use in BibTeX, on web pages or in Wikis. You can enter:
• one or several paper IDs like “1510.01797” or “math/0506203”.
• your arXiv author ID looking similar to “grafvbothmer_h_1” to get a list of all your submitted papers.
• your ORCID ID looking similar to “0000-0003-0136-444X” which you should register with your arXiv-account.
• Here’s hoping the proposal goes in. It did!
• Start on IEEE paper? Nope. Did get back to Grokking Deep learning. Trying to get the system working with MNIST.
• Something for the arousal potential/Clockwork Muse file: Accelerating dynamics of collective attention
• With news pushed to smart phones in real time and social media reactions spreading across the globe in seconds, the public discussion can appear accelerated and temporally fragmented. In longitudinal datasets across various domains, covering multiple decades, we find increasing gradients and shortened periods in the trajectories of how cultural items receive collective attention. Is this the inevitable conclusion of the way information is disseminated and consumed? Our findings support this hypothesis. Using a simple mathematical model of topics competing for finite collective attention, we are able to explain the empirical data remarkably well. Our modeling suggests that the accelerating ups and downs of popular content are driven by increasing production and consumption of content, resulting in a more rapid exhaustion of limited attention resources. In the interplay with competition for novelty, this causes growing turnover rates and individual topics receiving shorter intervals of collective attention.
• Chasing down narrative embedding using force-directed graphs and found Tulip
• Tulip is an information visualization framework dedicated to the analysis and visualization of relational data. Tulip aims to provide the developer with a complete library, supporting the design of interactive information visualization applications for relational data that can be tailored to the problems he or she is addressing.
• There are Python bindings. The following are for large layouts
• FM^3 (OGDF)
• Implements the FM³ layout algorithm by Hachul and Jünger. It is a multilevel, force-directed layout algorithm that can be applied to very large graphs.
• H3 (GRIP)
• Implements the H3 layout technique for drawing large directed graphs as node-link diagrams in 3D hyperbolic space. That algorithm can lay out much larger structures than can be handled using traditional techniques for drawing general graphs because it assumes a hierarchical nature of the data. It was first published as: H3: Laying out Large Directed Graphs in 3D Hyperbolic Space . Tamara Munzner. Proceedings of the 1997 IEEE Symposium on Information Visualization, Phoenix, AZ, pp 2-10, 1997. The implementation in Python (MIT License) has been written by BuzzFeed (https://github.com/buzzfeed/pyh3).
• Mahzarin R. Banaji
• Professor Banaji studies thinking and feeling as they unfold in social context, with a focus on mental systems that operate in implicit or unconscious mode. She studies social attitudes and beliefs in adults and children, especially those that have roots in group membership.  She explores the implications of her work for questions of individual responsibility and social justice in democratic societies. Her current research interests focus on the origins of social cognition and applications of implicit cognition to improve individual decisions and organizational policies.
• What do Different Beliefs Tell us? An Examination of Factual, Opinion-Based, and Religious Beliefs