7:00 – 8:00 ASRC

• Saw this on Twitter this morning: Training Agents using Upside-Down Reinforcement Learning
  • Traditional Reinforcement Learning (RL) algorithms either predict rewards with value functions or maximize them using policy search. We study an alternative: Upside-Down Reinforcement Learning (Upside-Down RL or UDRL), that solves RL problems primarily using supervised learning techniques. Many of its main principles are outlined in a companion report [34]. Here we present the first concrete implementation of UDRL and demonstrate its feasibility on certain episodic learning problems. Experimental results show that its performance can be surprisingly competitive with, and even exceed that of traditional baseline algorithms developed over decades of research.
• I wonder how it compares with Stuart Russell’s paper Cooperative Inverse Reinforcement Learning
  • For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial- information game with two agents, human and robot; both are rewarded according to the human’s reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.
• Dissertation
  • In the Ethics section, change ‘civilization’ to ‘culture’, and frame it in terms of the simulation – done
  • Last slide should be ‘Thanks for coming to my TED talk’
  • Ping Don’s composer and choreographer, if I can find them
  • Cool! A T-O style universe map (Unmismoobjetivo , via Wikipedia). The logarithmic distance effect is something that I need to look into:
• Evolver
  • Quickstart
  • User’s guide
  • Finished commenting!
  • Flailing on geting the documentation tools to work.
    • Installing Sphinx
• ML Seminar
  • Double Crab Cake Platter (2) – 2 Vegetables – $34.00
  • Went over the Evolver. The Ensemble charts really make an impression, but overall, the code walkthrough is too difficult – there are two many moving parts. I need to write a paper with screengrabs that walk through the whole process. I’ll need to evaluate against Bayesian tuners, but I also have architecture search
  • The venue could be IEEE ICTAI 2020: The IEEE International Conference on Tools with Artificial Intelligence (ICTAI) is a leading Conference of AI in the Computer Society providing a major international forum where the creation and exchange of ideas related to artificial intelligence are fostered among academia, industry, and government agencies. It will be in Baltimore, I think.
• Meeting with Aaron. He thinks that part of the ethics discussion needs to be an addressing of the status quo

7:00 – 8:00 ASRC GOES

• Dissertation – back to designing for populations
• Timesheet revisions
• Applying for MS Project
• Evolver – more documentation
• GOES Meeting
  • Bought a copy of MS Project for $15
  • Send Erik a note about permission to charge for TF Dev Conf
  • Good chat with Bruce about many things, including CASSIE as a Cloud service
  • Re-send links to common satellite dictionary
  • Vadim got a pendulum working
• Meeting with Roger
  • Got a tour of the new building
  • Lots of VR discussion
  • Some academic future options

7:00 – 4:00 ASRC GOES

• Dissertation – reworked the last paragraph of the Reflection and reflex section
  • Spooky to see this: Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
    • Citing papers including ‘human’
• Evolver – more documentation
• Send this out to the HCC mailing list: The introvert’s academic “alternative networking” guide
• Arpita’s proposal defense
  • Stanford: Open information extraction (open IE) refers to the extraction of relation tuples, typically binary relations, from plain text, such as (Mark Zuckerberg; founded; Facebook). The central difference from other information extraction is that the schema for these relations does not need to be specified in advance; typically the relation name is just the text linking two arguments. For example, Barack Obama was born in Hawaii would create a triple (Barack Obama; was born in; Hawaii), corresponding to the open domain relation was-born-in(Barack-Obama, Hawaii).
  • Open Information Extraction 5
  • UKG Open Information Extraction
  • Supervised Ensemble of Open IE
  • Datasets
    • AW-OIE
    • AW-OIE-C
    • WEB
    • NYT
    • PENN
  • Why the choice of 100 dimensins for your symentic embedding? How does it compare to other dimensions?
  • Contextual embedding for NLP?
  • Input-Output Hidden Markov Model (version on GitHub)

December! Yikes!

7:00 – 8:00 ASRC GOES

• Dissertation
  • Designing for populations
• Evolver
  • Oh, boy – big IDE updates. Hoping nothing breaks
    • Had to connect back to python
    • TF still works!
  • Commenting and documenting
    • Finished ValueAxis.py
    • Starting TF2OptomizerBase.py
• ML seminar (food fro La Madeleine!)
• Meeting with Aaron M

7:00 – 3:00 ASRC GOES

• Dissertation – Added a bit at the beginning of the discussion section to explain why this should fit in the HCI universe. Started working on the Non-human agents part, and am explaining why things like the GPT-2 create their own low dimensional spaces due to the cost of implementation and the incentives of research
• Evolver – Commenting and tweaking
  • Done with ValueAxis.py, which contains
    • class ValueAxisType(Enum):
    • class ValueAxis:
    • class EvolveAxis:
    • Example usage, evaluation and class exercising code using

      if __name__ == '__main__':

• Ran out of space on my primary drive and had to drop everything and fix that

7:00 – 4:30ASRC GOES

• Dissertation
  • Good progress on discussion section
  • I have 222 hours to charge for the rest of the year!
• Evolver
  • Working out index-based calculations in the test case
  • Found a HUGE bug. I was copying EvolveAxis pointers not values
  • Fixed with copy.deepcopy()
  • Need to add a set_value() for crossover
• Several hours with Aaron on vehicle identification
• Nextgen schedule plan – trying to get MSProject
• JuryRoom Meeting
  • Moved time to 6:30
  • Need to write up a peer review use case

7:00 – 4:00 ASRC GOES

• Disseration
  • Looking at editor resources:
    • Cambridge Academic Editors Network
    • Cambridge proofreading and editing services
    • Looked in the Diamondback, but didn’t find anything
  • More discussion organization
• Evolver
  • Work on getting all the functions and Evolver->Evolver stacks putting their arguments and return values in the spreadsheet. then adjust the chromosome so that secondary and tertiary values are permuted correctly. I think everything will have to be listed, but certain parts will need to be frozen.
  • Make sure that genomes don’t repeat. Making progress, but it’s complex and slow going. Right now it doesn’t repeat on the value, but I don’t think that’s quite right