Category Archives: Paper

Phil 6.19.20


12:00 – Sy’s defense at noon!

GPT-2 Agents

  • Fixed the regex in ChessMovesToDb
  • More work on finding closest neighbors.
    • Maybe keep a record of the number and type of pieces that are used?
    • Looks like the basics are working. Here’s the test graph:


    • And here are the results. I made the code so that it only shows each neighbor once, but it may be useful to keep track of the number of times a neighbor shows up in a list. This might not be important in chess, but in less structured text environments (RPGs to Reddit threads), it may be valuable:
      find_closest_neighbors(): nodes = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
      {'node': 'a', 'known_nearest': ['f', 'd']}
      {'node': 'b', 'known_nearest': ['f', 'd']}
      {'node': 'c', 'known_nearest': []}
      {'node': 'd', 'known_nearest': ['f', 'a', 'b', 'g']}
      {'node': 'e', 'known_nearest': []}
      {'node': 'f', 'known_nearest': ['a', 'g', 'd', 'b']}
      {'node': 'g', 'known_nearest': ['f', 'd']}


    • At this point it’s not recursive, but it could be. I’m worried about combinatorial explosion though


  • Submit GVSETS paper – done!
  • Meeting with Vadim and Issac at 11:00
    • Goal is to move all the RW code out of the sim class and into its own and call methods from the sim class

Phil 6.18.20

Hotel reservations!

Sent a ping to Don about a paper to review

GPT-2 Agents

  • Started on common neighbor algorithm. Definitely a good place for recursion
  • Generating larger file



  • If you look at the center of the plot and squint a bit, you can see a bit of the grid:


  • There is an error: The string ‘, White moves pawn from h3 to g4. White takes black pawn. LCZero v0.24-sv-t60-3010 moves black knight from h5 to g7. White moves pawn from g4 to h5. LCZero v0.24‘ is parsing incorrectly due to the truly bizarre name (The little known Grand Master LCZero v0.24-sv-t60-3010). Need to fix the regex. I think I just need to make it so that there has to be a space in front and a space/period after.


  • Readthrough of GVSETS paper
  • 2:00 Meeting


  • Alex had a really good insight in that groups that are working at coming to consensus use terms to discuss their level of agreement that are independent of the points being argued. That’s could really be important in text analysis.

Phil 6.17.20

Listened to a fantastic interview with Nell Irvin Painter (White Supremacy at Home and Abroad):

GPT-2 Agents

  • Working on finding the connections between nodes
  • Now that I know how to add weights to edges, I think I want to add the piece that made the move. It needs to be a list, since multiple types of pieces can connect two squares. Added a dict_array per edge:
    if target not in nlist:
        self.G.add_edge(source, target, weight=0)
        self.G[target]['dict_array'] = []
    self.G[target]['weight'] += 1
    for key, val in data_dict.items():
        a:List = self.G[target]['dict_array']
  • I also realize that moves that repeatedly connect squares are more likely to be close, simply because the available squares of more distant moves increase in a geometric fashion. I added a method that writes out moves to Excel where I can play with them. Here are some moves:


  • In looking at these moves, it does seem to be that the majority of the moves seem to be short (e.g. b6-b7, b6-a7, b6-b5). The only exception is the knight (b6-d7). So I think there is a confidence value that I can calculate for the ‘physical’ adjacency of nodes in a network. This could also apply to belief spaces as well. Most consensus requires coordination and common orientation (pos, heading, speed), so commonly connected topics can be said to be ‘closer’
  • Good chat with Aaron about CVPR and algorithms


  • Finish revisions and send to T and Aaron for review. Last thing is to tie back to ground vehicles in the discussion. Done! I think… Need to read the whole thing and see if it still hangs together
  • 2:00 – Meeting

Phil 6.10.20

Finish ACSOS review

GPT-2 Agents

  • Generate embeddings
    • Trying running much longer sequences (max_length = 1000). The lets games run long enough  that they often conclude (the term “resigns”, “wins”, or “draw occurs in the text)
    • Put together a simple regex ‘[a-h][1-8]’ that pulls out all the squares in sequence from a game
    • Extracting game square sequences to create files that will feed into Word2Vec. The class is started and most of the issues are worked out. I added a check for game endings so beginning and endings are not place together oddly.
    • Here’s the trimmed input text
      The game begins as white uses the Sicilian opening. and black countering with Najdorf, Adams attack. Loek Van Wely moves white pawn from e2 to e4. Black moves pawn from c7 to c5. In move 2, White moves knight from g1 to f3. Black moves pawn from d7 to d6. White moves pawn from d2 to d4. Black moves pawn from c5 to d4. Black takes white pawn. White moves knight from f3 to d4. White takes black pawn. Black moves knight from g8 to f6. In move 5, White moves knight from b1 to c3. Arseniy Nesterov moves black pawn from a7 to a6. Loek Van Wely moves white bishop from c1 to e3. Black moves pawn from e7 to e6. In move 7, White moves pawn from f2 to f4. Black moves knight from b8 to d7. White moves queen from d1 to d2. Black moves pawn from b7 to b5. Loek Van Wely queenside castles. Black moves bishop from f8 to e7. White moves bishop from f1 to d3. Arseniy Nesterov kingside castles. White moves king from c1 to b1. Black moves rook from a8 to b8. White moves pawn from g2 to g3. Black moves queen from d8 to a5. Loek Van Wely moves white king from b1 to a1. Black moves bishop from e7 to d6. Black takes white knight. Loek Van Wely moves white bishop from d3 to e4. White takes black pawn. Black moves rook from b8 to b2. Black takes white pawn. In move 17, White moves bishop from e4 to h7. White takes black pawn. Check. Arseniy Nesterov moves black king from g8 to h8. White moves bishop from h7 to d3. Black moves bishop from d6 to f4. Black takes white pawn. Check. In move 19, White moves bishop from e3 to f4. White takes black bishop. Black moves rook from b2 to f2. White moves rook from h1 to f1. Black moves knight from d7 to e5. White moves queen from d2 to e2. Black moves queen from a5 to d2. In move 22, White moves knight from c3 to e2. White takes black queen. Black moves rook from f2 to e2. Black takes white knight. Loek Van Wely moves white bishop from f4 to e3. Black moves rook from e2 to e3. Black takes white bishop. White moves pawn from f4 to f5. Black moves rook from f8 to d8. White moves pawn from a2 to a4. Arseniy Nesterov moves black bishop from c8 to b7. White moves pawn from a4 to a5. Arseniy Nesterov moves black bishop from b7 to c8. White moves pawn from a5 to b6. White takes. Arseniy Nesterov moves black pawn from a6 to b5. Black takes white pawn. White moves queen from e2 to b5. White takes black pawn. Black moves knight from e5 to c4. White moves pawn from h2 to h3. Black moves knight from c4 to a5. In move 30, Loek Van Wely moves white queen from b5 to a4. Arseniy Nesterov moves black pawn from h7 to h6. White moves bishop from d3 to b1. Black moves rook from d8 to d1. Check. Loek Van Wely
    • And here’s the sequence
      e2 e4 c7 c5 g1 f3 d7 d6 d2 d4 c5 d4 f3 d4 g8 f6 b1 c3 a7 a6 c1 e3 e7 e6 f2 f4 b8 d7 d1 d2 b7 b5 f8 e7 f1 d3 c1 b1 a8 b8 g2 g3 d8 a5 b1 a1 e7 d6 d3 e4 b8 b2 e4 h7 g8 h8 h7 d3 d6 f4 e3 f4 b2 f2 h1 f1 d7 e5 d2 e2 a5 d2 c3 e2 f2 e2 f4 e3 e2 e3 f4 f5 f8 d8 a2 a4 c8 b7 a4 a5 b7 c8 a5 b6 a6 b5 e2 b5 e5 c4 h2 h3 c4 a5 b5 a4 h7 h6 d3 b1 d8 d1


    • I can do other things like split into white and black, but that’s pretty tricky and I don’t think it’s worth it
  • Start building networks. Here are some api possibilities


  • If the devlab is still up, work on pulling down data. Nope, the VPN is working so badly today that I can’t even load my webmail
  • Going to work on the download and transfer using my local Influx – done!


Complete copy of remote data on local server

  • 2:00 Meeting

Phil 6.1.20


It’s all been a bit much recently, so yesterday I took advantage of the wonderful weather and went on a long ride with a few friends.

D20 – Nagged Zach with this image. He responses generally were “It is generally pretty optimistic around here”, and “According to google is is getting better. I wonder where their data comes from”.


GPT-2 Agents

  • Still some debugging. added output of the raw move files to find games better
  • Dates aren’t right either – fixed
  • Added some better triggering of the print_board method
  • WOW! I mean it shouldn’t be that surprising, but the pgn is wrong. Going to add a flag for games with problem moves. Then I think I should be able to generate text.


  • Put paper in the right format (word?)
  • Create the slides. Verify the speaking duration – done. It’s 20 minuts, I think probably 15 for talk and 5 for questions
  • Found the technical paper repo. Looks like I didn’t have to worry about length!
  • Uploaded! Just use the info in the email from GVSETS Tech Session Admin

Google is profiting from dozens of websites that peddle hoaxes and conspiracy theories about Covid-19, according to a Tech Transparency Project (TTP) investigation, revealing a major hole in the company’s claims that it’s fighting misinformation about the pandemic.


Phil 5.29.20


Race/police riots. We’re not even halfway through the year

GPT-2 Agents

  • Wrote my first pgn code! It moves the rooks out, down to the other side of the board and then back
    1. h4 h5 2. Rh3 Rh6 3. Ra3 Ra6 4. Rh3 Rh6 5. 1/2-1/2
  • And it’s working! I did have to adjust the subtraction order to get pieces to move in the right direction. It even generates game text:
    Game.parse_moves(): Move1 = ' h4 h5 '
    Evaluating move [h4 h5]
    piece string = '' (blank is pawn)
    piece string = '' (blank is pawn)
    Game.parse_moves(): Move2 = ' Rh3 Rh6 '
    Evaluating move [Rh3 Rh6]
    piece string = 'R' (blank is pawn)
    piece string = 'R' (blank is pawn)
    Game.parse_moves(): Move3 = ' Ra3 Ra6 '
    Evaluating move [Ra3 Ra6]
    piece string = 'R' (blank is pawn)
    Chessboard.check_if_clear() white rook is blocked by white pawn
    piece string = 'R' (blank is pawn)
    Chessboard.check_if_clear() black rook is blocked by black pawn
    Game.parse_moves(): Move4 = ' Rh3 Rh6 '
    Evaluating move [Rh3 Rh6]
    piece string = 'R' (blank is pawn)
    piece string = 'R' (blank is pawn)
    Game.parse_moves(): Move5 = ' 1/2-1/2'
    Evaluating move [1/2-1/2]
    The game begins as white uses the Polish (Sokolsky) opening opening. 
    Aye Bee moves white pawn from h2 to h4. Black moves pawn from h7 to h5.
    White moves rook from h1 to h3. Cee Dee moves black rook from h8 to h6.
    White moves rook from h3 to a3. Black moves rook from h6 to a6.
    Aye Bee moves white rook from a3 to h3. Black moves rook from a6 to h6.
    In move 5, Aye Bee declares a draw. Cee Dee declares a draw
  • Ok, it’s not quite working. I can’t take pieces any more. Fixed. Here’s the longer sequence where the white rook takes the black rook and retreats back to its start, moving past the other white rook:
    1. h4 h5 2. Rh3 Rh6 3. Ra3 Ra6 4. Rxa6 c6 5. Ra3 d6 6. Rh3 e6 7. Rh1 f6 8. 1/2-1/2
  • And here’s the expanded game:
    The game begins as white uses the Polish (Sokolsky) opening opening. 
    White moves pawn from h2 to h4. Black moves pawn from h7 to h5.
    Aye Bee moves white rook from h1 to h3. Black moves rook from h8 to h6.
    In move 3, White moves rook from h3 to a3. Black moves rook from h6 to a6.
    White moves rook from a3 to a6. White takes black rook. Black moves pawn from c7 to c6.
    In move 5, White moves rook from a6 to a3. Black moves pawn from d7 to d6.
    White moves rook from a3 to h3. Black moves pawn from e7 to e6.
    In move 7, White moves rook from h3 to h1. Black moves pawn from f7 to f6.
    In move 8, Aye Bee declares a draw. Cee Dee declares a draw
  • It’s a little too Friday to try a full run and find new bugs though. Going to work on the paper instead

D20 – Ping Zach. Made contact. Asked for a date to have the maps up or cut bait.


  • Add accuracy/loss diagram and paragraph – done
  • Finish first pass
  • Nope, All hands plus 90 minutes of required cybersecurity training. To be fair, the videos were nicely done, with a light touch and good acting.

Phil 5.28.20

GPT-2 Agents

  • Back to bug hunting today’s job is to figure out why this:
    1. Nf3 Nf6 2. g3 c5 3. Bg2 Nc6 4. O-O e5 5. e4 Nxe4 6. Re1 Nf6 7. Nxe5 Be7 8. c4
    O-O 9. Nc3 Nxe5 10. Rxe5 d6 11. Re1 Be6 12. Bxb7 Rb8 13. Bg2 Bxc4 14. d4 Be6 15.
    b3 Rb4 16. dxc5 dxc5 17. Qxd8 Rxd8 18. Ba3 Rbb8 19. Na4 Rdc8 20. Rac1 Nd7 21.
    Bd5 Bxd5 22. Rxe7 Bc6 23. Nxc5 Nxc5 24. Rxc5 a6 25. f4 h6 26. Kf2 Bb5 27. Ke3
    Rd8 28. Rcc7 Rd3+ 29. Ke4 Rd2 30. Rxf7 Re8+ 31. Kf5 Bd3+ 32. Kg4 Rxh2 33. Rxg7+
    Kh8 34. Bd6 Rf2 35. Bc5 Rd2 36. Bb4 Rc2 37. Rxc2 Kxg7 38. Rc7+ Kg6 39. Rc6+ Kf7
    40. Rxh6 Re2 41. Rd6 Re3 42. Kh4 Be2 43. g4 Rf3 44. Rd4 Rf2 45. Kg5 1-0
  • breaks the system.
  • So I never added logic to see if the path was clear for a move. The game has a move where white rook moves from e1 to e5 and then back. For the move back, the system looks for the closest rook, which is actually at a1, as the search algorithm works. But that way is actually blocked by the white bishop and white queen. It should take the clear path and discard blocked paths. I think this fix is pretty straightforward


  • Wrote the test, but I’m not sure if it’s right. We’ll test tomorrow:
        def check_if_clear(self, loc:Tuple, candidate:Tuple, piece:PIECES) -> bool:
            if piece == PIECES.WHITE_PAWN or piece == PIECES.BLACK_PAWN:
                return True
            if piece == PIECES.WHITE_KNIGHT or piece == PIECES.BLACK_KNIGHT:
                return True
            if piece == PIECES.WHITE_KING or piece == PIECES.BLACK_KING:
                return True
            c_col_i = self.char_index.index(candidate[0])
            c_row_i = self.num_index.index(candidate[1])
            l_col_i = self.char_index.index(loc[0])
            l_row_i = self.num_index.index(loc[1])
            col_dist = l_col_i - c_col_i
            row_dist = l_row_i - c_row_i
            dist = max(abs(col_dist), abs(row_dist))
            col_vec = 0
            row_vec = 0
            if col_dist != 0:
                col_vec = col_dist/dist
            if row_dist != 0:
                row_vec = row_dist/dist
            col_i = l_col_i
            row_i = l_row_i
            for i in range(dist):
                num = self.num_index[row_i]
                char = self.char_index[col_i]
                pos = (char, num)
                p = self.get_piece_at(pos)
                if p != PIECES.NONE:
                    return False
                col_i += col_vec
                row_i += row_vec
            return True



  • More paper writing
    • Finished the first pass of section 2, which describes the whole model.

Phil 5.27.20

Drop off the truck today!

Agents and expensive information

  • Antonio sent a note asking if I’d be interested in contributing to a chapter. Sent him this response:
    • There is something that I’d like to explore that might fit. It’s the idea that in most environments, agents (animal, human, machine, etc.) are incentivized to cheat. I think this is because information is expensive to produce, but essentially free to copy. The problem is that if all the agents cheat, then the system will collapse because the agents become decoupled from reality (what I call a stampede). So the system as a whole is incentivized to somehow restrict cheating.
    • I think this could be very interesting to work through, but I don’t have a model (or even an approach really) developed that would describe it. I think that this might be related to game theory, though I haven’t found much in the literature.

GPT-2 Agents

  • Working on building a text corpora. Going to add a search for “Opening” and “Variation” which I’ll try before using the DB version – done
  • Having some problem that starts after a few games. Found the culprit game. Will work on tomorrow. It might be tied to a linefeed?


  • Working on the GVSETS paper and slide deck

Phil 4.28.20


  • Upload paper to Overleaf – done!


  • Fix bug using this:
    slope, intercept, r_value, p_value, std_err = stats.linregress(xsub, ysub)
    # slope, intercept = np.polyfit(x, y, 1)
    yn = np.polyval([slope, intercept], xsub)
    steps = 0
    if slope < 0:
        steps = abs(y[-1] / slope)
    reg_x = []
    reg_y = []
    start = len(yl) - max_samples
    yval = intercept + slope * start
    for i in range(start, len(yl)-offset):
        yval += slope
  • Anything else?

GPT-2 Agents

  • Install and test GPT-2 Client
  • Failed spectacularly. It depends on a lot of TF1.x items, like There is an issue request in.
  • Checked out the project to see if anything could be done. “Fixed” the contrib library, but that just exposed other things. Uninstalled.
  • Tried using the upgrade tool described here, which did absolutely nothing, as near as I can tell


  • Continue figuring out GANs
  • Here are results using 2 latent dimensions, a matching hint, a line hint, and no hint
  • Here are results using 5 latent dimensions, a matching hint, a line hint, and no hint
  • Meeting at 10:00 with Vadim and Isaac
    • Wound up going over Isaac’s notes for Yaw Flip and learned a lot. He’s going to see if he can get the algorithm used for the maneuver. If so, we can build the control behavior around that. The goal is to minimize energy and indirectly fuel costs


Phil 4.24.20

It is very wet today


Spent far too much time trying to upload a picture to the graduation site. It appears to be broken


  • Changed the CONTROLLED days to < 2, since things are generally looking better


  • Sent the revised draft to Antonio

GPT-2 Agents

  • Found what appears to be just what I’m looking for. Searching on GitHub for GPT-2 tensorflow led me to this project, GPT-2 Client. I’ll give that a try and see how it works. The developer, Rishabh Anand seems to have solid skills so I have some hope that this could work. I do  not have the energy to start this on a Friday and then switch to GANs for the rest of the day. Sunday looks like another wet one, so maybe then.


block_3_conv2More looking at layers. This is Imagenet’s block3_conv3

  • Advanced CNNs
  • Start GANS? Yes!
    • Got this version working. Now I need to step through it. But here are some plots of it learning:
    • I had dreams about this, so I’m going to record the thinking here:
      • An MLP should be able to get from a simple simulation (square wave) to a more accurate(?) simulation sin wave. The data set is various start points and frequency queries into the DB, with matching (“real”/noisy) as the test. My intuition is that the noise will be lost, so that’s the part we’re going to have to get back with the GAN.
      • So I think there is a two-step process
        • Train the initial NN that will produce the generalized solution
        • Use the output of the NN and the “real” data to train the GAN for fine tuning

Phil 4.23.20

Transformer Architecture: The Positional Encoding

  • In this article, I don’t plan to explain its architecture in depth as there are currently several great tutorials on this topic (herehere, and here), but alternatively, I want to discuss one specific part of the transformer’s architecture – the positional encoding.


  • Add centroids for states – done
  • Return the number of neighbors as an argument – done
  • Chatted with Aaron and Zach. More desire to continue than abandon


  • More revisions. Swap steps for discussion and future work


    • IRS proposal went in yesterday
    • Continue with GANs
    • Using the VGG model now with much better results. Also figured out how to loads weights and read the probabilities in the output layer: vgg
    • Same thing using the pre-trained model from Keras:
      from tensorflow.keras.applications.vgg16 import VGG16
      # prebuild model with pre-trained weights on imagenet
      model = VGG16(weights='imagenet', include_top=True)
      model.compile(optimizer='sgd', loss='categorical_crossentropy')


    • Trying to visualize a layer using this code. And using that code as a starting point, I had to explore how to slice up the tensors in the right way. A CNN layer has a set of “filters” that contain a square set of pixels. The data is stored as an array of pixels at each x, y, coordinate, so I had to figure out how to get one image at a time. Here’s my toy:
      import numpy as np
      import matplotlib.pyplot as plt
      n_rows = 4
      n_cols = 8
      depth = 4
      my_list = []
      for r in range(1, n_rows):
          row = []
          for c in range(1, n_cols):
              cell = []
              for d in range(depth):
      nl = np.array(my_list)
      for d in range(depth):
          print("\nlayer {} = \n{}".format(d, nl[:, :, d]))
          plt.imshow(nl[:, :, d], aspect='auto', cmap='plasma')
    • This gets features from a cat image at one of the pooling layers. The color map is completely arbitrary:
      # get the features from this block
      features = model.predict(x)
      farray = np.array(features[0])
      print("{}".format(farray[:, :, 0]))
      for d in range(4):
         plt.imshow(farray[:, :, d], aspect='auto', cmap='plasma')
    • But we get some cool pix!

Phil 4.20.20


  • Reading the Distill article on Gaussian processes (highlighted page here)
  • Copy over neural-tangents code from notebook to IDE
  • Working on regression
  • Ran into a problem with Tensorboard
    Traceback (most recent call last):
      File "d:\program files\python37\lib\", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "d:\program files\python37\lib\", line 85, in _run_code
        exec(code, run_globals)
      File "D:\Program Files\Python37\Scripts\tensorboard.exe\", line 7, in 
      File "d:\program files\python37\lib\site-packages\tensorboard\", line 75, in run_main, flags_parser=tensorboard.configure)
      File "d:\program files\python37\lib\site-packages\absl\", line 299, in run
        _run_main(main, args)
      File "d:\program files\python37\lib\site-packages\absl\", line 250, in _run_main
      File "d:\program files\python37\lib\site-packages\tensorboard\", line 289, in main
        return runner(self.flags) or 0
      File "d:\program files\python37\lib\site-packages\tensorboard\", line 305, in _run_serve_subcommand
        server = self._make_server()
      File "d:\program files\python37\lib\site-packages\tensorboard\", line 409, in _make_server
        self.flags, self.plugin_loaders, self.assets_zip_provider
      File "d:\program files\python37\lib\site-packages\tensorboard\backend\", line 183, in standard_tensorboard_wsgi
        flags, plugin_loaders, data_provider, assets_zip_provider, multiplexer
      File "d:\program files\python37\lib\site-packages\tensorboard\backend\", line 272, in TensorBoardWSGIApp
        tbplugins, flags.path_prefix, data_provider, experimental_plugins
      File "d:\program files\python37\lib\site-packages\tensorboard\backend\", line 345, in __init__
        "Duplicate plugins for name %s" % plugin.plugin_name
    ValueError: Duplicate plugins for name projector
  • After poking around a bit online with the “Duplicate plugins for name %s” % plugin.plugin_name ValueError: Duplicate plugins for name projector, I found this diagnostic, which basically asked me to reinstall everything*. That didn’t work, so I went into the Python37\Lib\site-packages and deleted by hand. Tensorboard now runs, but now I need to upgrade my cuda so that I have cudart64_101.dll
    • Installed the minimum set of items from the Nvidia Package Launcher (cuda_10.1.105_418.96_win10.exe)
    • Installed the cuDNN drivers from here:
    • The regular (e.g. MNIST) demos work byt when I try the distribution code I got this error: tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op ‘NcclAllReduce’. It turns out that there are only two viable MirroredStrategy operations, for windows, and the default is not one of them. These are the valid calls:
      distribution = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.ReductionToOneDevice())
      distribution = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
    • And this call is not
      # distribution = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.NcclAllReduce()) # <-- not valid for Windows
  • Funny thing. After reinstalling and getting everything to work, I tried the diagnostic again. It seems it always says to reinstall everything
  • And Tensorboard is working! Here’s the call that puts data in the directory:
    linear_est = tf.estimator.LinearRegressor(feature_columns=feature_columns, model_dir = 'logs/boston/')
  • And when launched on the command line pointing at the same directory:
    D:\Development\Tutorials\Deep Learning with TensorFlow 2 and Keras\Chapter 3>tensorboard --logdir=.\logs\boston
    2020-04-20 11:36:42.999208: I tensorflow/stream_executor/platform/default/] Successfully opened dynamic library cudart64_101.dll
    W0420 11:36:46.005735 18544] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
    W0420 11:36:46.006743 18544] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
    Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
    TensorBoard 2.1.1 at http://localhost:6006/ (Press CTRL+C to quit)
  • I got this! tensoboard
  • Of course, we’re not done yet. When attempting to use the Keras callback, I get the following error: tensorflow.python.eager.profiler.ProfilerNotRunningError: Cannot stop profiling. No profiler is running. It turns out that you have to specify the log folder like this
      • command line:
        tensorboard --logdir=.\logs
      • in code:
        logpath = '.\\logs'



  • That seems to be working! RunningTBNN
  • Finished regression chapter


  • Submitted RFI response for review


  • Got Antonio’s comments back


  • Need to work on the math to find second bumps
    • If the rate has been < x% (maybe 2.5%), calculate an offset that leaves a value of 100 for each day. When the rate jumps more than y% (e.g. 100 – 120 = 20%), freeze that number until the rate settles down again and repeat the process
    • Change the number of samples to be the last x days
  • Work with Zach to get maps up?

ML seminar

Phil 4.17.20

Can You Beat COVID-19 Without a Lockdown? Sweden Is Trying

I dug into the predictions that we generate of Comparing Finland, Norway, and Sweden, it looks like something that Sweden did could result in about 2,600 people dying that don’t have to:




  • IRS proposal – done!
  • A better snippet: the best way to cheat on taxes is  to deliberately lie to the IRS about what you earned over a year, what you spent over a year, and the ways you would fill out those forms. This is where “time of year” really comes into play. The IRS assumes you worked on April 15 through the 15th of the following year in order to report and pay taxes on your actual income from April 15 through the following year. I’ve put some pictures and thoughts below. There are some really great readers who have put some excellent guides and resources out there on this topic. If you have any additional questions, please feel free to leave a comment below and I will do my best to answer them.
  • Another good snippet: The best way to cheat on taxes is  to set up an LLC or other tax-sheltered company that makes up for your sloth in paying business taxes. By doing this, you can deduct the business expenses and pay your taxes at a much lower tax rate, while also getting a tax refund. So, for example, if your net operating income for 2014 was $5,000 and you think you should owe about $2,000 in taxes for 2015, I suggest you set up a  S-Corporation   for 2015 that only owes $500 in taxes. Then, you can send the IRS a check for the difference between the $2,000 difference you owe them and the $5,000 net operating income for 2015.


  • Finish first pass? Done! And sent to Antonio!


Shortcut Learning in Deep Neural Networks

  • Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today’s machine intelligence. Numerous success stories have rapidly spread all over science, industry and society, but its limitations have only recently come into focus. In this perspective we seek to distil how many of deep learning’s problem can be seen as different symptoms of the same underlying problem: shortcut learning. Shortcuts are decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios. Related issues are known in Comparative Psychology, Education and Linguistics, suggesting that shortcut learning may be a common characteristic of learning systems, biological and artificial alike. Based on these observations, we develop a set of recommendations for model interpretation and benchmarking, highlighting recent advances in machine learning to improve robustness and transferability from the lab to real-world applications.

Phil 4.16.20

Fix siding!

SageMathMore on SageTex here


  • Playing around with something to indicate the linear fit to the data. Trying P value
  • Updated UI code so that the P value will display on the next build
  • Hopefully we try the world map code today?



  • Learning more about multiple inputs to embedding and had to get the keras.utils.plot_model working, which failed with this error: ImportError: Failed to import pydot. You must install pydot and graphviz for `pydotprint` to work. So I pip installed both, and had the same problem.
  • Had problems running the distribution samples. Upgraded tf to version 2.1. No problems and better performance
  • Finished chapter 2


  • Struggled with picture placement. Moving on.
  • Finished first pass. I need to add more ABM text, but I’m down to 10 pages plus references!

Multi-input and multi-output models

  • Here’s a good use case for the functional API: models with multiple inputs and outputs. The functional API makes it easy to manipulate a large number of intertwined datastreams. Let’s consider the following model. We seek to predict how many retweets and likes a news headline will receive on Twitter. The main input to the model will be the headline itself, as a sequence of words, but to spice things up, our model will also have an auxiliary input, receiving extra data such as the time of day when the headline was posted, etc. The model will also be supervised via two loss functions. Using the main loss function earlier in a model is a good regularization mechanism for deep models.