Monthly Archives: October 2018

Phil 10.31.18

7:00 – ASRC PhD

  • Read this carefully today: Introducing AdaNet: Fast and Flexible AutoML with Learning Guarantees
    • Today, we’re excited to share AdaNet, a lightweight TensorFlow-based framework for automatically learning high-quality models with minimal expert intervention. AdaNet builds on our recent reinforcement learning and evolutionary-based AutoML efforts to be fast and flexible while providing learning guarantees. Importantly, AdaNet provides a general framework for not only learning a neural network architecture, but also for learning to ensemble to obtain even better models.
    • What about data from simulation?
    • Github repo
    • This looks like it’s based deeply the cloud AI and Machine Learning products, including cloud-based hyperparameter tuning.
    • Time series prediction is here as well, though treated in a more BigQuery manner
      • In this blog post we show how to build a forecast-generating model using TensorFlow’s DNNRegressor class. The objective of the model is the following: Given FX rates in the last 10 minutes, predict FX rate one minute later.
    • Text generation:
      • Cloud poetry: training and hyperparameter tuning custom text models on Cloud ML Engine
        • Let’s say we want to train a machine learning model to complete poems. Given one line of verse, the model should generate the next line. This is a hard problem—poetry is a sophisticated form of composition and wordplay. It seems harder than translation because there is no one-to-one relationship between the input (first line of a poem) and the output (the second line of the poem). It is somewhat similar to a model that provides answers to questions, except that we’re asking the model to be a lot more creative.
      • Codelab: Google Developers Codelabs provide a guided, tutorial, hands-on coding experience. Most codelabs will step you through the process of building a small application, or adding a new feature to an existing application. They cover a wide range of topics such as Android Wear, Google Compute Engine, Project Tango, and Google APIs on iOS.
        Codelab tools on GitHub

  • Add the Range and Length section in my notes to the DARPA measurement section. Done. I need to start putting together the dissertation using these parts
  • Read Open Source, Open Science, and the Replication Crisis in HCI. Broadly, it seems true, but trying to piggyback on GitHub seems like a shallow solution that repurposes something for coding – an ephemeral activity, to science, which is archival for a reason. Thought needs to be given to an integrated (collection, raw data, cleaned data, analysis, raw results, paper (with reviews?), slides, and possibly a recording of the talk with questions. What would it take to make this work across all science, from critical ethnographies to particle physics? How will it be accessible in 100 years? 500? 1,000? This is very much an HCI problem. It is about designing a useful socio-cultural interface. Some really good questions would be “how do we use our HCI tools to solve this problem?”, and, “does this point out the need for new/different tools?”.
  • NASA AIMS meeting. Demo in 2 weeks. AIMS is “time series prediction”, A2P is “unstructured data”. Proove that we can actually do ML, as opposed to saying things.
    • How about cross-point correlation? Could show in a sim?
    • Meeting on Friday with a package
    • We’ve solved A, here’s the vision for B – Z and a roadmap. JPSS is a near-term customer (JPSS Data)
    • Getting actionable intelligence from the system logs
    • Application portfolios for machine learning
    • Umbrella of capabilities for Rich Burns
    • New architectural framework for TTNC
    • Complete situational awareness. Access to commands and sensor streams
    • Software Engineering Division/Code 580
    • A2P as a toolbox, but needs to have NASA-relevant analytic capabilities
    • GMSEC overview

Phil 10.30.18

7:00 – 3:30 ASRC PhD

  • Search as embodies in the “Ten Blue Links” meets the requirements of a Parrow “Normal Accident”
    • The search results are densely connected. That’s how PageRank works. Even latent connections matter.
    • The change in popularity of a page rapidly affects the rank. So the connections are stiff
    • The relationships of the returned links both to each other and to the broader information landscape in general is hidden.
    • An additional density and stiffness issue is that everyone uses Google, so there is a dense, stiff connection between the search engine and the population of users
  • Write up something about how
    • ML can make maps, which decrease the likelihood of IR contributing to normal accidents
    • AI can use these maps to understand the shape of human belief space, and where the positive regions and dangerous sinks are.
  • Two measures for maps are the concepts or Range and length. Range is the distance that a trajectory can be placed on the map and remain contiguous. Length is the total distance that a trajectory travels, independent of the map its placed on.
  • Write up the basic algorithm of ML to map production
    • Take a set of trajectories that are known to be in the same belief region (why JuryRoom is needed) as the input
    • Generate an N-dimensional coordinate frame that best preserves length over the greatest range.
    • What is used as the basis for the trajectory may matter. The range (at a minimum), can go from letters to high-level topics. I think any map reconstruction based on letters would be a tangle, with clumps around TH, ER, ON, and AN. At the other end, an all-encompassing meta-topic, like WORDS would be a single, accurate, but useless single point. So the map reconstruction will become possible somewhere between these two extremes.
  • The Nietzsche text is pretty good. In particular, check out the way the sentences form based on the seed  “s when one is being cursed.
    • the fact that the spirit of the spirit of the body and still the stands of the world
    • the fact that the last is a prostion of the conceal the investion, there is our grust
    • the fact them strongests! it is incoke when it is liuderan of human particiay
    • the fact that she could as eudop bkems to overcore and dogmofuld
    • In this case, the first 2-3 words are the same, and random, semi-structured text. That’s promising, since the compare would be on the seed plus the generated text.
  • Today, see how fast a “Shining” (All work and no play makes Jack a dull boy.) text can be learned and then try each keyword as a start. As we move through the sentence, the probability of the next words should change.
    • Generate the text set
    • Train the Nietzsche model on the new text. Done. Here are examples with one epoch and a batch size of 32, with a temperature of 1.0: 
      ----- diversity: 0.2
----- Generating with seed: "es jack a 
dull boy all work and no play"
es jack a 
dull boy all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes 

----- diversity: 0.5
----- Generating with seed: "es jack a 
dull boy all work and no play"
es jack a 
dull boy all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes 

----- diversity: 1.0
----- Generating with seed: "es jack a 
dull boy all work and no play"
es jack a 
dull boy all work and no play makes jack a dull boy anl wory and no play makes jand no play makes jack a dull boy all work and no play makes jack a 

----- diversity: 1.2
----- Generating with seed: "es jack a 
dull boy all work and no play"
es jack a 
dull boy all work and no play makes jack a pull boy all work and no play makes jack andull boy all work and no play makes jack a dull work and no play makes jack andull

      Note that the errors start with a temperature of 1.0 or greater

    • Rewrite the last part of the code to generate text based on each word in the sentence.
      • So I tried that and got gobbledygook. The issues is that the prediction only works on waveform-sized chunks. To verify this, I created a seed from the input text, truncating it to maxlen (20 in this case): 
        sentence = "all work and no play makes jack a dull boy"[:maxlen]

        That worked, but it means that the character-based approach isn’t going to work

        ----- temperature: 0.2
----- Generating with seed: [all work and no play]
all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes 

----- temperature: 0.5
----- Generating with seed: [all work and no play]
all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes 

----- temperature: 1.0
----- Generating with seed: [all work and no play]
all work and no play makes jack a dull boy all work and no play makes jack a dull boy pllwwork wnd no play makes 

----- temperature: 1.2
----- Generating with seed: [all work and no play]
all work and no play makes jack a dull boy all work and no play makes jack a dull boy all work and no play makes

         

    • Based on this result and the ensuing chat with Aaron, we’re going to revisit the whole LSTM with numbers and build out a process that will support words instead of characters.
  • Looking for CMAC models, I found Self Organizing Feature Maps at NeuPy.com:
  • Here’s How Much Bots Drive Conversation During News Events
    • Late last week, about 60 percent of the conversation was driven by likely bots. Over the weekend, even as the conversation about the caravan was overshadowed by more recent tragedies, bots were still driving nearly 40 percent of the caravan conversation on Twitter. That’s according to an assessment by Robhat Labs, a startup founded by two UC Berkeley students that builds tools to detect bots online. The team’s first product, a Chrome extension called BotCheck.me, allows users to see which accounts in their Twitter timelines are most likely bots. Now it’s launching a new tool aimed at news organizations called FactCheck.me, which allows journalists to see how much bot activity there is across an entire topic or hashtag

Phil 10.29.18

7:00 – 5:00 ASRC PhD

  • This looks like a Big Deal from Google – Working together to apply AI for social good
    • Google.org is issuing an open call to organizations around the world to submit their ideas for how they could use AI to help address societal challenges. Selected organizations will receive support from Google’s AI experts, Google.org grant funding from a $25M pool, credit and consulting from Google Cloud, and more.
    • We look forward to receiving your application on or before 11:59 p.m. PT on January 22, 2019, and we encourage you to apply early given that we expect high volume within the last few hours of the application window. Thank you!
    • Application Guide
    • Application form (can’t save, compose offline using guide, above)
  • Finished my writeup on Meltdown
  • Waiting for a response from Antonio
  • Meeting with Don at 9:00 to discuss BAA partnership.
    • Don is comfortable with being PI or co-PI, whichever works best. When we call technical POCs, we speak on his behalf
    • We discussed how he could participate with the development of theoretical models based on signed graph Laplacians creating structures that can move in belief space. He thinks the idea has merit, and can put in up to 30% of his time on mathematical models and writing
    • ASRC has already partnered with UMBC. ASRC would sub to UMBC
    • Ordinarily, IP is distributed proportional to the charged hours
    • Don has access to other funding vehicles that can support the Army BAA, but this would make things more complicated. These should be discussed if we can’t make a “clean” agreement that meets our funding needs
  • Pinged Brian about his defense.
  • Some weekend thoughts
    • Opinion dynamics systems describe how communication within a network occurs, but disregards the motion of the network as a whole. In cases when the opinions converge, the network is stiff.
    • Graph laplacians could model “othering” by having negative weights. It looks like these are known as signed laplacians, and useful to denote difference. The trick is to discover the equations of motion. How do you model a “social particle”?
  • Just discovered the journal Swarm Intelligence
    • Swarm Intelligence is the principal peer reviewed publication dedicated to reporting research and new developments in this multidisciplinary field. The journal publishes original research articles and occasional reviews on theoretical, experimental, and practical aspects of swarm intelligence. It offers readers reports on advances in the understanding and utilization of systems that are based on the principles of swarm intelligence. Emphasis is given to such topics as the modeling and analysis of collective biological systems; application of biological swarm intelligence models to real-world problems; and theoretical and empirical research in ant colony optimization, particle swarm optimization, swarm robotics, and other swarm intelligence algorithms. Articles often combine experimental and theoretical work.
  • I think it’s time to start ramping up on the text generation!
      • Updated my home box to tensorflow 1.11.0. Testing to see if it still works using the Deep Learning with Keras simple_nueral_net.py example. Hasn’t broken (yet…), but is taking a long time… Worked! And it’s much faster the second time. Don’t know why that is and can’t find anything online that talks to that. 
        Loss: 0.5043802047491074
Accuracy: 0.8782
Time =  211.42629722093085
      • Found this keras example for generating Nietsche

     

    • Trying it out. This may be a overnight run… But it is running.
  • Had a good discussion with Aaron about how mapmaking could be framed as an ML problem. More writeup tomorrow.

Phil 10.28.18

We know from the House Intelligence Committee report that the Russians were pushing a Syrian message among the other more “organic” messages. But this seems to indicate that they did get traction.

Mail bomb suspect made numerous references on Facebook to Russian associates and echoed pro-Kremlin views

  • The posts showed fixations on certain subjects, including Miami sports teams, youth soccer, Native American themes and businesses Sayoc was seeking to promote. But in April 2016, after several months of not posting on Facebook, the account abruptly changed subjects to link to videos celebrating Syria’s fight against ISIS.
  • “He just pops up four months later and just relentlessly shares stories about ISIS and terrorists,” said Albright. “The turn is just remarkable… He found ideas that never let go from that point on.”

Phil 10.25.18

7:00 – 5:00 ASRC PhD

  • Two unrelated thoughts.
    • A tangle could be made to heal if each transaction kept track of the transaction that verified it. If that transaction became unreachable for more than N heartbeats, then the transaction becomes unverified again. Not sure if the verifying transaction needs to track the other way. Being able to query the tangle for these “scars” seems like it should be useful.
    • A death threat is a unique form of dimension reduction, and should probably be tracked/tagged using both emergent topic modeling and hand-tuned heuristics
  • Tim Berners-Lee on the huge sociotechnical design challenge
    • “We must consciously decide on both of these, both the social side and the technical side,” he said. “[These platforms are] anthropogenic, made by people… Facebook and Twitter are anthropogenic. They’re made by people. They’ve coded by people. And the people who code them are constantly trying to figure out how to make them better.”
  • Antonio workshop paper
    • Today– Finished hierarchy section, didn’t start Black swan section
    • Took out the hybrid section and used Aaron’s writeup on research opportunities to set up the ensemble of hierarchies parts that Antonio is writing.
    • Tonight, send note to Antonio with thoughts on introduction and Hybrid section. Done. He’s taking a look.

Phil 10.24.18

7:00 – 6:00 ASRC PhD

  • So the BAA is only for academic work, which means partnering with UMD/UMBC. Need to talk to Don about setting this up. Some email this morning about how an NDA would be needed. I’m thinking that it would be restricted to A2P.
  • Inside the Moral Machine : When your experiment survey becomes reaction video material
    • On June 23rd, 2016, we deployed Moral Machine. The website was intended to be a mere companion survey to a paper being published that day. Thirty minutes later, it crashed.
    • Read this to see if there are ways of making JuryRoom go viral in similar ways
  • Respond to the Collective Intelligence journal proposal – done
  • Antonio workshop paper
    • Today – Finish market section – done
    • Thursday – Start hierarchy section, start Black swan section
      • Thursday night, send note to Antonio with thoughts on introduction and Hybrid section.
    • Friday – Hybrid section?
  • Hello, CoLa!
    • This network of character co-occurence in Les Misérables is positioned by constraint-based optimization using WebCoLa. Compare to d3-force.
    • This should be better than mass-spring-damper systems for building maps. Cola

Phil 10.23.18

7:00 – 4:30 ASRC PhD

  • Respond to the Collective Intelligence journal proposal
  • Antonio workshop paper
    • Today – Introduction, TaaS as a spectrum, part of the Market section
    • Wednesday – Hierarchy section
    • Thursday – Black swan section
      • Thursday night, send note to Antonio with thoughts on introduction and Hybrid section.
    • Friday – Hybrid section?
  • LSTM Encoder-Decoder with Adversarial Network for Text Generation from Keyword
    • Natural Language Generation (NLG), one of the areas of Natural Language Processing (NLP), is a difficult task, but it is also important because it applies to our lives. So far, there have been various approaches to text generation, but in recent years, approaches using artificial neural networks have been used extensively. We propose a model for generating sentences from keywords using Generative Adversarial Network (GAN) composed of a generator and a discriminator among these artificial neural networks. Specifically, the generator uses the Long Short-Term Memory (LSTM) Encoder-Decoder structure, and the discriminator uses the bi-directional LSTM with self-attention. Also, the keyword for input to the encoder of the generator is input together with two words similar to oneself. This method contributes to the creation of sentences containing words that have similar meanings to the keyword. In addition, the number of unique sentences generated increases and diversity can be increased. We evaluate our model with BLEU Score and loss value. As a result, we can see that our model improves the performance compared to the baseline model without an adversarial network.

Phil 10.22.18

7:00 – 5:30 ASRC PhD

      • Need to finish workshop paper this week
      • Jeff Atwood said I should look at 10 year old code to frighten myself and found a permuter class that could be used for hyperparameter tuning! It’s here: 
        trunk/Java_folders/Projects/EntryRelationDb/src/main/java/com/edgeti/EntryRelationDb/Permutations.java
      • Fika
      • Meeting with Wayne.
        • We have a 12% chance of getting in the iConference, so don’t expect much. On the other hand, that opens up content for Antonio’s paper?

     

Phil 10.21.18

Finished Meltdown. Need to write up some notes.

Think about using a CMAC or Deep CMAC for function learning, because NIST. Also, can it be used for multi-dimensional learning?

  • Cerebellar model articulation controller
  • Adaptive Noise Cancellation Using Deep Cerebellar Model Articulation Controller
  • RCMAC Hybrid Control for MIMO Uncertain Nonlinear Systems Using Sliding-Mode Technology
    • A hybrid control system, integrating principal and compensation controllers, is developed for multiple-input-multiple-output (MIMO) uncertain nonlinear systems. This hybrid control system is based on sliding-mode technique and uses a recurrent cerebellar model articulation controller (RCMAC) as an uncertainty observer. The principal controller containing an RCMAC uncertainty observer is the main controller, and the compensation controller is a compensator for the approximation error of the system uncertainty. In addition, in order to relax the requirement of approximation error bound, an estimation law is derived to estimate the error bound. The Taylor linearization technique is employed to increase the learning ability of RCMAC and the adaptive laws of the control system are derived based on Lyapunov stability theorem and Barbalat’s lemma so that the asymptotical stability of the system can be guaranteed. Finally, the proposed design method is applied to control a biped robot. Simulation results demonstrate the effectiveness of the proposed control scheme for the MIMO uncertain nonlinear system
  • Github CMAC TF projects

 

Phil 10.19.18

Phil 7:00 – 3:30 ASRC PhD

  • Sprint review
  • Reading Meltdown: Why our systems fail and What we can do about it, and I found some really interesting work that relates to social conformity, flocking, stampeding and nomadic behaviors:
    • “We show that a deviation from the group opinion is regarded by the brain as a punishment,” said the study’s lead author, Vasily Klucharev. And the error message combined with a dampened reward signal produces a brain impulse indicating that we should adjust our opinion to match the consensus. Interestingly, this process occurs even if there is no reason for us to expect any punishment from the group. As Klucharev put it, “This is likely an automatic process in which people form their own opinion, hear the group view, and then quickly shift their opinion to make it more compliant with the group view.” (Page 154)
      • Reinforcement Learning Signal Predicts Social Conformity
        • Vasily Klucharev
        • We often change our decisions and judgments to conform with normative group behavior. However, the neural mechanisms of social conformity remain unclear. Here we show, using functional magnetic resonance imaging, that conformity is based on mechanisms that comply with principles of reinforcement learning. We found that individual judgments of facial attractiveness are adjusted in line with group opinion. Conflict with group opinion triggered a neuronal response in the rostral cingulate zone and the ventral striatum similar to the “prediction error” signal suggested by neuroscientific models of reinforcement learning. The amplitude of the conflict-related signal predicted subsequent conforming behavioral adjustments. Furthermore, the individual amplitude of the conflict-related signal in the ventral striatum correlated with differences in conforming behavior across subjects. These findings provide evidence that social group norms evoke conformity via learning mechanisms reflected in the activity of the rostral cingulate zone and ventral striatum.
    • When people agreed with their peers’ incorrect answers, there was little change in activity in the areas associated with conscious decision-making. Instead, the regions devoted to vision and spatial perception lit up. It’s not that people were consciously lying to fit in. It seems that the prevailing opinion actually changed their perceptions. If everyone else said the two objects were different, a participant might have started to notice differences even if the objects were identical. Our tendency for conformity can literally change what we see. (Page 155)
      • Gregory Berns
        • Dr. Berns specializes in the use of brain imaging technologies to understand human – and now, canine – motivation and decision-making.  He has received numerous grants from the National Institutes of Health, National Science Foundation, and the Department of Defense and has published over 70 peer-reviewed original research articles.
      • Neurobiological Correlates of Social Conformity and Independence During Mental Rotation
        • Background

          When individual judgment conflicts with a group, the individual will often conform his judgment to that of the group. Conformity might arise at an executive level of decision making, or it might arise because the social setting alters the individual’s perception of the world.

          Methods

          We used functional magnetic resonance imaging and a task of mental rotation in the context of peer pressure to investigate the neural basis of individualistic and conforming behavior in the face of wrong information.Results

          Conformity was associated with functional changes in an occipital-parietal network, especially when the wrong information originated from other people. Independence was associated with increased amygdala and caudate activity, findings consistent with the assumptions of social norm theory about the behavioral saliency of standing alone.

          Conclusions

          These findings provide the first biological evidence for the involvement of perceptual and emotional processes during social conformity.

        • The Pain of Independence: Compared to behavioral research of conformity, comparatively little is known about the mechanisms of non-conformity, or independence. In one psychological framework, the group provides a normative influence on the individual. Depending on the particular situation, the group’s influence may be purely informational – providing information to an individual who is unsure of what to do. More interesting is the case in which the individual has definite opinions of what to do but conforms due to a normative influence of the group due to social reasons. In this model, normative influences are presumed to act through the aversiveness of being in a minority position
      • A Neural Basis for Social Cooperation
        • Cooperation based on reciprocal altruism has evolved in only a small number of species, yet it constitutes the core behavioral principle of human social life. The iterated Prisoner’s Dilemma Game has been used to model this form of cooperation. We used fMRI to scan 36 women as they played an iterated Prisoner’s Dilemma Game with another woman to investigate the neurobiological basis of cooperative social behavior. Mutual cooperation was associated with consistent activation in brain areas that have been linked with reward processing: nucleus accumbens, the caudate nucleus, ventromedial frontal/orbitofrontal cortex, and rostral anterior cingulate cortex. We propose that activation of this neural network positively reinforces reciprocal altruism, thereby motivating subjects to resist the temptation to selfishly accept but not reciprocate favors.
  • Working on Antonio’s paper. I think I’ve found the two best papers to use for the market system. It turns out that freight has been doing this for about 20 years. Agent simulation and everything

Phil 10.18.18

7:00 – 9:00, 12:00 – ASRC PhD

  • Reading the New Yorker piece How Russia Helped Swing the Election for Trump, about Kathleen Hall Jamieson‘s book Cyberwar: How Russian Hackers and Trolls Helped Elect a President—What We Don’t, Can’t, and Do Know. Some interesting points with respect to Adversarial Herding:
    • Jamieson’s Post article was grounded in years of scholarship on political persuasion. She noted that political messages are especially effective when they are sent by trusted sources, such as members of one’s own community. Russian operatives, it turned out, disguised themselves in precisely this way. As the Times first reported, on June 8, 2016, a Facebook user depicting himself as Melvin Redick, a genial family man from Harrisburg, Pennsylvania, posted a link to DCLeaks.com, and wrote that users should check out “the hidden truth about Hillary Clinton, George Soros and other leaders of the US.” The profile photograph of “Redick” showed him in a backward baseball cap, alongside his young daughter—but Pennsylvania records showed no evidence of Redick’s existence, and the photograph matched an image of an unsuspecting man in Brazil. U.S. intelligence experts later announced, “with high confidence,” that DCLeaks was the creation of the G.R.U., Russia’s military-intelligence agency.
    • Jamieson argues that the impact of the Russian cyberwar was likely enhanced by its consistency with messaging from Trump’s campaign, and by its strategic alignment with the campaign’s geographic and demographic objectives. Had the Kremlin tried to push voters in a new direction, its effort might have failed. But, Jamieson concluded, the Russian saboteurs nimbly amplified Trump’s divisive rhetoric on immigrants, minorities, and Muslims, among other signature topics, and targeted constituencies that he needed to reach. 
  • Twitter released IRA dataset (announcement, archive), and Kate Starbird’s group has done some preliminary analysis
  • Need to do something about the NESTA Call for Ideas, which is due “11am on Friday 9th November“
  • Continuing with Market-Oriented Programming
    • Some thoughts on what the “cost” for a trip can reference
      • Passenger
        • Ticket price
          • provider: Current price, refundability, includes taxes
            • carbon
            • congestion
            • other?
          • consumer: Acceptable range
        • Travel time
        • Departure time
        • Arrival time (plus arrival time confidence)
        • comfort (legroom, AC)
        • Number of stops (related to convenience)
        • Number of passengers
        • Time to wait
        • Externalities like airport security, which adds +/- 2 hours to air travel
      • Cargo
        • Divisibility (ship as one or more items)
        • Physical state for shipping (packaged, indivisible solid, fluid, gas)
          • Waste to food grade to living (is there a difference between algae and cattle? Pets? Show horses?
          • Refrigerated/heated
          • Danger
          • Stability/lifespan
          • weight
      • Aggregators provide simpler combinations of transportation options
    • Any exchange that supports this format should be able to participate. Additionally, each exchange should contain a list of other exchanges that a consumer can request, so we don’t need another level of hierarchy. Exchanges could rate other exchanges as a quality measure
      • It also occurs to me that there could be some kind of peer-to-peer or mesh network for degraded modes. A degraded mode implies a certain level of emergency, which would affect the (now small-scale) allocation of resources.
    • Some stuff about Mobility as a Service. Slide deck (from Canada Intelligent Transportation Service), and an app (Whim)
  • PSC AI/ML working group 9:00 – 12:00, plus writeup
    • PSC will convene a working group meeting on Thursday, Oct. 18 from 9am – 10am to identify actions and policy considerations related to advancing the use of AI solutions in government. Come prepared to share your ideas and experience. We would welcome your specific feedback on these questions:
      • How can PSC help make the government a “smarter buyer” when it comes to AI/ML?
      • How are agencies effectively using AI/ML today?
      • In what other areas could these technologies be deployed in government today?
        • Looking for bad sensors on NOAA satellites
      • What is the current federal market and potential future market for AI/ML?
      • Notes:
        • How to help our members – federal contracts. Help make the federal market frictionless
        • Kevin – SmartForm? What are the main gvt concerns? Is it worry about False positives?
          • Competitiveness – no national strategy
          • Appropriate use, particularly law enforcement
          • Robotic Process Automation (RPA) Security, Compliancy, and adoption. Compliancy testing.
          • Data trust. Humans make errors. When ML makes the same errors, it’s worse.
        • A system that takes time to get accurate watching people perform is not the kind of system that the government can buy.
          • This implies that there has to be immediate benefit, and can have the possibility of downstream benefit.
        • Dell would love to participate (in what?) Something about cloud
        • Replacing legacy processes with better approaches
        • Fedramp-like compliance mechanism for AI. It is a requirement if it is a cloud service.
        • Perceived, implicit bias is the dominant narrative on the government side. Specific applications like facial recognition
        • Take a look at all the laws that might affect AI, to see how the constraints are affecting adoption/use with an eye towards removing barriers
        • Chris ?? There isn’t a very good understanding or clear linkage between the the promise and the current problems, such as staffing, tagged data, etc
        • What does it mean to be reskilled and retrained in an AI context?
        • President’s Management Agenda
        • The killer app is cost savings, particularly when one part of government is getting a better price than another part.
        • Federal Data Strategy
        • Send a note to Kevin about data availability. The difference between NOAA sensor data (clean and abundant), vs financial data, constantly changing spreadsheets that are not standardized. Maybe the creation of tools that make it easier to standardize data than use artisanal (usually Excel-based) solutions. Wrote it up for Aaron to review. It turned out to be a page.

Phil 10.17.18

7:00 – 4:00 Antonio Workshop

Phil 10.16.18

7:00 – 4:00 ASRC DARPA

  • Steve had some good questions about quantitative measures:
    • I think there are some good answers that we can provide here on determining the quality of maps. The number of users is an educated guess though. In my simulations, I can generate enough information to create maps using about 100 samples per agent. I’m working on a set of experiments that will produce “nosier” data that will provide a better estimate, but that won’t be ready until December. So we can say that “simulations indicate that approximately 100 users will have to interact through a total of 100 threaded posts to produce meaningful maps”
    • With respect to the maps themselves, we can determine quality in four ways. The mechanism for making this comparison will be bootstrap sampling (https://en.wikipedia.org/wiki/Bootstrapping_(statistics)), which is an extremely effective way of comparing two unknown distributions. In our case, the distribution will be the coordinate of each topic in the embedding space.
      1. Repeatability: Can multiple maps generated on the same data set be made to align? Embedding algorithms often start with random values. As such embeddings that are similar may appear different because they have different orientations. To determine similarity we would apply a least-squares transformation of one map with respect to the other. Once complete, we would expect a greater than 90% match between the two maps in success.
      2. Resolution: What is the smallest level of detail that can be rendered accurately? We will be converting words into topics and then placing the topics in an embedding space. As described in the document, we expect to do this with Non-Negative Matrix Factorization (NMF). If we factor the all discussions down to a single topic (i.e. “words”), then we will have a single point map that can always be rendered with 100% repeatability, but it has 0% precision. If, on the other hand, we can place every word in every discussion on the map, but the relationships are different every time, then we can have 100% precision, but 0% repeatability. As we cluster terms together, we need to compare repeated runs to see that we get similar clusters each time. We need to find the level of abstraction that will give us a high level of repeatability. A 90% match is our expectation.
      3. Responsiveness: Maps change over time. A common example is a weather map, though political maps shift borders and physical maps reflect geographic activity like shoreline erosion. This duration may reflect the accuracy of the map, with slow change happening across large scales while rapid changes are visible at higher resolutions. A change at the limit of resolution should ideally be reflected immediately in the map and not adjust the surrounding areas.
  • More frantic flailing to meet the deadline. DONE!!!

4:00 – 5:30 Antonio Workshop

Phil 10.15.18

7:00 – ASRC BD

  • Heard about some interesting things this morning on BBC Business Daily – Is the Internet Fit for Purpose?:
    • Future in Review Conference: The leading global conference on the intersection of technology and the economy. New partnerships, projects, and plans you can’t afford to miss. If your success depends on having an accurate view of the future, or you’d like to meet others who are able and motivated to forge action-based alliances, this is the most important conference you will attend. Be one of the thought leaders in the FiRe conversation, analyzing and creating the future of technology, economics, pure science, the environment, genomics, education, and more.
    • Berit Anderson. Created the science fact/fiction magazine Scout, which, interestingly enough, has a discussion space for JuryRoom-style questions
  • More DARPA proposal