Category Archives: Phil

Phil 5.29.20

WaPo

Race/police riots. We’re not even halfway through the year

GPT-2 Agents

  • Wrote my first pgn code! It moves the rooks out, down to the other side of the board and then back
    1. h4 h5 2. Rh3 Rh6 3. Ra3 Ra6 4. Rh3 Rh6 5. 1/2-1/2
  • And it’s working! I did have to adjust the subtraction order to get pieces to move in the right direction. It even generates game text:
    Game.parse_moves(): Move1 = ' h4 h5 '
    Evaluating move [h4 h5]
    piece string = '' (blank is pawn)
    piece string = '' (blank is pawn)
    Game.parse_moves(): Move2 = ' Rh3 Rh6 '
    Evaluating move [Rh3 Rh6]
    piece string = 'R' (blank is pawn)
    piece string = 'R' (blank is pawn)
    Game.parse_moves(): Move3 = ' Ra3 Ra6 '
    Evaluating move [Ra3 Ra6]
    piece string = 'R' (blank is pawn)
    Chessboard.check_if_clear() white rook is blocked by white pawn
    piece string = 'R' (blank is pawn)
    Chessboard.check_if_clear() black rook is blocked by black pawn
    Game.parse_moves(): Move4 = ' Rh3 Rh6 '
    Evaluating move [Rh3 Rh6]
    piece string = 'R' (blank is pawn)
    piece string = 'R' (blank is pawn)
    Game.parse_moves(): Move5 = ' 1/2-1/2'
    Evaluating move [1/2-1/2]
    
    The game begins as white uses the Polish (Sokolsky) opening opening. 
    Aye Bee moves white pawn from h2 to h4. Black moves pawn from h7 to h5.
    White moves rook from h1 to h3. Cee Dee moves black rook from h8 to h6.
    White moves rook from h3 to a3. Black moves rook from h6 to a6.
    Aye Bee moves white rook from a3 to h3. Black moves rook from a6 to h6.
    In move 5, Aye Bee declares a draw. Cee Dee declares a draw
    
  • Ok, it’s not quite working. I can’t take pieces any more. Fixed. Here’s the longer sequence where the white rook takes the black rook and retreats back to its start, moving past the other white rook:
    1. h4 h5 2. Rh3 Rh6 3. Ra3 Ra6 4. Rxa6 c6 5. Ra3 d6 6. Rh3 e6 7. Rh1 f6 8. 1/2-1/2
  • And here’s the expanded game:
    The game begins as white uses the Polish (Sokolsky) opening opening. 
    White moves pawn from h2 to h4. Black moves pawn from h7 to h5.
    Aye Bee moves white rook from h1 to h3. Black moves rook from h8 to h6.
    In move 3, White moves rook from h3 to a3. Black moves rook from h6 to a6.
    White moves rook from a3 to a6. White takes black rook. Black moves pawn from c7 to c6.
    In move 5, White moves rook from a6 to a3. Black moves pawn from d7 to d6.
    White moves rook from a3 to h3. Black moves pawn from e7 to e6.
    In move 7, White moves rook from h3 to h1. Black moves pawn from f7 to f6.
    In move 8, Aye Bee declares a draw. Cee Dee declares a draw
  • It’s a little too Friday to try a full run and find new bugs though. Going to work on the paper instead

D20 – Ping Zach. Made contact. Asked for a date to have the maps up or cut bait.

GOES

  • Add accuracy/loss diagram and paragraph – done
  • Finish first pass
  • Nope, All hands plus 90 minutes of required cybersecurity training. To be fair, the videos were nicely done, with a light touch and good acting.

Phil 5.28.20

GPT-2 Agents

  • Back to bug hunting today’s job is to figure out why this:
    1. Nf3 Nf6 2. g3 c5 3. Bg2 Nc6 4. O-O e5 5. e4 Nxe4 6. Re1 Nf6 7. Nxe5 Be7 8. c4
    O-O 9. Nc3 Nxe5 10. Rxe5 d6 11. Re1 Be6 12. Bxb7 Rb8 13. Bg2 Bxc4 14. d4 Be6 15.
    b3 Rb4 16. dxc5 dxc5 17. Qxd8 Rxd8 18. Ba3 Rbb8 19. Na4 Rdc8 20. Rac1 Nd7 21.
    Bd5 Bxd5 22. Rxe7 Bc6 23. Nxc5 Nxc5 24. Rxc5 a6 25. f4 h6 26. Kf2 Bb5 27. Ke3
    Rd8 28. Rcc7 Rd3+ 29. Ke4 Rd2 30. Rxf7 Re8+ 31. Kf5 Bd3+ 32. Kg4 Rxh2 33. Rxg7+
    Kh8 34. Bd6 Rf2 35. Bc5 Rd2 36. Bb4 Rc2 37. Rxc2 Kxg7 38. Rc7+ Kg6 39. Rc6+ Kf7
    40. Rxh6 Re2 41. Rd6 Re3 42. Kh4 Be2 43. g4 Rf3 44. Rd4 Rf2 45. Kg5 1-0
  • breaks the system.
  • So I never added logic to see if the path was clear for a move. The game has a move where white rook moves from e1 to e5 and then back. For the move back, the system looks for the closest rook, which is actually at a1, as the search algorithm works. But that way is actually blocked by the white bishop and white queen. It should take the clear path and discard blocked paths. I think this fix is pretty straightforward

chess

  • Wrote the test, but I’m not sure if it’s right. We’ll test tomorrow:
        def check_if_clear(self, loc:Tuple, candidate:Tuple, piece:PIECES) -> bool:
            if piece == PIECES.WHITE_PAWN or piece == PIECES.BLACK_PAWN:
                return True
            if piece == PIECES.WHITE_KNIGHT or piece == PIECES.BLACK_KNIGHT:
                return True
            if piece == PIECES.WHITE_KING or piece == PIECES.BLACK_KING:
                return True
            
            c_col_i = self.char_index.index(candidate[0])
            c_row_i = self.num_index.index(candidate[1])
            l_col_i = self.char_index.index(loc[0])
            l_row_i = self.num_index.index(loc[1])
            col_dist = l_col_i - c_col_i
            row_dist = l_row_i - c_row_i
            dist = max(abs(col_dist), abs(row_dist))
            col_vec = 0
            row_vec = 0
            if col_dist != 0:
                col_vec = col_dist/dist
            if row_dist != 0:
                row_vec = row_dist/dist
    
            col_i = l_col_i
            row_i = l_row_i
            for i in range(dist):
                num = self.num_index[row_i]
                char = self.char_index[col_i]
                pos = (char, num)
                p = self.get_piece_at(pos)
                if p != PIECES.NONE:
                    return False
                col_i += col_vec
                row_i += row_vec
    
            return True

     

GOES

  • More paper writing
    • Finished the first pass of section 2, which describes the whole model.

Phil 5.27.20

Drop off the truck today!

Agents and expensive information

  • Antonio sent a note asking if I’d be interested in contributing to a chapter. Sent him this response:
    • There is something that I’d like to explore that might fit. It’s the idea that in most environments, agents (animal, human, machine, etc.) are incentivized to cheat. I think this is because information is expensive to produce, but essentially free to copy. The problem is that if all the agents cheat, then the system will collapse because the agents become decoupled from reality (what I call a stampede). So the system as a whole is incentivized to somehow restrict cheating.
    • I think this could be very interesting to work through, but I don’t have a model (or even an approach really) developed that would describe it. I think that this might be related to game theory, though I haven’t found much in the literature.

GPT-2 Agents

  • Working on building a text corpora. Going to add a search for “Opening” and “Variation” which I’ll try before using the DB version – done
  • Having some problem that starts after a few games. Found the culprit game. Will work on tomorrow. It might be tied to a linefeed?

GOES

  • Working on the GVSETS paper and slide deck

Phil 5.26.20

Had a good, cathartic ride yesterday:

GPT-2 Agents

  • I’ve been working on the PGNtoEnglish class and was having an odd bug where occasionally a piece would pull a piece from the other side of the board. Since it was intermittent, it required many print statements and searching through the logs for “black knight”

blac knight

  • My problem was in forgetting how Python indexes into arrays. Here’s the code in question:

python

  • When I first wrote this, I had to deal with a lot of potential coordinates that were off the board, with indexes like (-2, -1), or (10, 8) for an 8×8 board. I thought to handle this with a try/except on IndexError (the bottom highlight). In other languages this would have worked, but Python allows negative indexes. Ooops! Adding the test for either index being negative (the top highlight) fixed that bug

D20

  • Ping Zach – done

GOES

  • Write up code review thoughts for Erik -done
  • Add n_critic to base class, along with adjustable false flag value
    • First, making sure that everything still works. Seems to.
    • Here’s the best I can do today, using the OneDGAN2a class with an RMSProp(lr=0.0005)

epochsNoise_trainedacc_loss

  • Assemble all the bits for an example
    • Verified that the InfluxTestTrainBase still works, and it’s using the InfluxDB values
    • Assemble all the bits for an example
      • Created a NoiseGAN2 with the same amount of points as the InfluxTestTrainBase model – done. Looks real good on the noise, too:

epochsNoise_trainedacc_loss

  • How to trim the columns on a 2D Numpy array:
    results = self.ifq.run_query(self.bucket, begin, end, filter_str)
    results = self.ifq.to_nd_array(results)
    results = np.delete(results, slice(clamp, None), 1)
    predict_table = model.predict(results)
  • Here’s all the parts nailed together:
  • Start the paper and the deck

ML Group

  • Need to create a walkthrough of coding practices for next week. I think I’ll use the trajectory of the GAN coding as the basis

 

Phil 5.25.20

GPT-2 Agents

  • Work on openings
  • Maybe create database that contains games as collections of moves. A query could produce the text for the language model
  • Created a database for openings, since there are multiple versions of the same opening and I couldn’t just use the site as an index into a dict. I mean…

openings

  • Chasing down more bugs. Did you know that ‘#’ means checkmate as well as ‘++’? Now you do!

D20

  • Rework the offsets to a y-day linear model rather than an x-y day linear model

Book

  • Semester’s over, so ping Thom – done

Phil 5.22.20

#COVID

  • Got a response back from Huggingface. They can’t replicate. sent more info

GPT-2 Agents

  • Adding better date handling
  • Fixed tied game text
  • Added some randomization to sentence beginnings
  • Added an overall description of the game. Today’s progress:
    On January 21, 2020, F Caruana played Ding Liren. F Caruana was the higer-ranked player, with an Elo rating of 2835. Ding Liren was the lower-ranked player, with an Elo rating of 2791. After 73 moves, the game ended in a draw. The game began with ECO opening B90:
    The game begins as White moves pawn from e2 to e4. Black moves pawn from c7 to c5.
    White moves knight from g1 to f3. Ding Liren moves black pawn from d7 to d6.
    White moves pawn from d2 to d4. Black moves pawn from c5 to d4. Black takes white pawn.
    White moves knight from f3 to d4. White takes black pawn. Black moves knight from g8 to f6.
    In move 5, F Caruana moves white knight from b1 to c3. Black moves pawn from a7 to a6.
    F Caruana moves white queen from d1 to d3. Ding Liren moves black pawn from e7 to e6.

GOES

  • I’m using the OneDGAN2 as a base class to do experiments with. That way, the subclass only needs a generator and/or a discriminator:
    class OneDGAN2a(ODG.OneDGAN2):
        def __init__(self):
            super().__init__()
    
        # create the Keras model to generate fake data (Dense to CNN)
        def define_generator(self) -> Sequential:
            self.g_model = Sequential()
            self.g_model.add(Dense(64, activation='relu', kernel_initializer='he_uniform', input_dim=self.latent_dimension))
            self.g_model.add(BatchNormalization())
            self.g_model.add(Dense(self.vector_size, activation='tanh'))
            self.g_model.add(Reshape((self.vector_size, 1)))
            print("g_model_Dense.output_shape = {}".format(self.g_model.output_shape))
    
            # compile model
            loss_func = tf.keras.losses.BinaryCrossentropy()
            opt_func = tf.keras.optimizers.Adam(0.001)
            self.g_model.compile(loss=loss_func, optimizer=opt_func)
            return self.g_model
    
        # define the standalone discriminator model
        def define_discriminator(self) -> Sequential:
            # activation_func = tf.keras.layers.LeakyReLU(alpha=0.02)
            activation_func = tf.keras.layers.ReLU()
            self.d_model = Sequential()
            self.d_model.add(Conv1D(filters=self.vector_size, kernel_size=20, strides=4, activation=activation_func, batch_input_shape=(self.num_samples, self.vector_size, 1)))
            self.d_model.add(Dropout(.3))
            self.d_model.add(GlobalMaxPool1D())
            self.d_model.add(Flatten())
            self.d_model.add(Dense(self.vector_size/2, activation=activation_func, kernel_initializer='he_uniform', input_dim=self.vector_size))
            self.d_model.add(Dropout(.3))
            self.d_model.add(Dense(1, activation='sigmoid'))
    
            # compile model
            loss_func = tf.keras.losses.BinaryCrossentropy()
            opt_func = tf.keras.optimizers.Adam(0.001)
            self.d_model.compile(loss=loss_func, optimizer=opt_func, metrics=['accuracy'])
            return self.d_model

    Which means that I can keep running instances rather than just notes

  • Continuing with Advanced Deep Learning with Keras
    • Another writeup of Wasserstein Loss cited in the book is here, with the associated TF code here. It’s from 3 years ago, so the TF version is waaaaaay out of date. That being said, the loss function is mostly straight java code:
      if self.model_type == self.WGAN_GP:
          # Wasserstein GAN with gradient penalty
          epsilon = tf.random_uniform([self.batch_size, 1, 1, 1], 0.0, 1.0)
          interpolated = epsilon * inputs + (1 - epsilon) * self.G
          _, self.D_logits_intp_ = self.discriminator(interpolated, self.y, reuse=True)
      
          # tf.gradients returns a list of sum(dy/dx) for each x in xs.
          gradients = tf.gradients(self.D_logits_intp_, [interpolated, ], name="D_logits_intp")[0]
          grad_l2 = tf.sqrt(tf.reduce_sum(tf.square(gradients), axis=[1, 2, 3]))
          grad_penalty = tf.reduce_mean(tf.square(grad_l2 - 1.0))
          self.gp_loss_sum = tf.summary.scalar("grad_penalty", grad_penalty)
          self.grad_norm_sum = tf.summary.scalar("grad_norm", tf.nn.l2_loss(gradients))
          # Add gradient penalty to the discriminator's loss function.
          self.d_loss += self.gp_lambda * grad_penalty
  • Here’s the approach to a WGAN. Pretty much the same, except for the loss function

WGAN

  • Note that the WGAN labels fake data as -1 rather than 0.
  • Starting to implement. Figured out how to do the first part of Wasserstein Loss, which also includes using RMSProp rather than Adam. So I gave that a shot. It’s better! And it might get better still with more epochs:

epochsNoise_trainedacc_loss

  • That is a little better! We’ll keep that:
  • Created a OneDGAN2b to continue
  • Got all the code running. Here’s my first attempt. It’s sloooooow, too

epochs

  • With an RMSProp optimizer of 0.0001 it seems to be better.

Noise_trained

  • I’m going to try a 20,000 epoch run next. A little better? Maybe

epochsNoise_trained

  • Here’s a 50k run. It’s really no better, but now there’s a mechanism to explore more than one layer of detector neurons? I also like to see that the pattern really stabilizes around 8,000 epochs

epochsNoise_trained

  • I think the next thing I want to try is the n-critic approach to the OneGAN2a approach

Phil 5.21.20

GPT-2 Agents

  • Added Queenside castling
  • Found an ECO opening file here! Now I really want to parse it into a json file and use it
  • Started the to_narrative() method. Here’s the first result!
    In 1987, Fred Van der Vliet played Loek Van Wely. Fred Van der Vliet was the higer-ranked player, with an Elo rating of 2330. Loek Van Wely was unrated. 
    The game began with ECO opening E69: 
    In move 1, Fred Van der Vliet moves white pawn from d2 to d4. Black moves knight from g8 to f6.
    this is a comment
    In move 2, White moves pawn from c2 to c4. Black moves pawn from g7 to g6.
    In move 3, Fred Van der Vliet moves white pawn from g2 to g3. Loek Van Wely moves black bishop from f8 to g7.
    In move 4, Fred Van der Vliet moves white bishop from f1 to g2. Loek Van Wely kingside castles.
    In move 5, White moves knight from g1 to f3. Black moves pawn from c7 to c6.
    In move 6, Fred Van der Vliet kingside castles. Loek Van Wely moves black pawn from d7 to d6.
    In move 7, White moves knight from b1 to c3. Black moves knight from b8 to d7.
    In move 8, White moves pawn from e2 to e4. Loek Van Wely moves black pawn from e7 to e5.
    In move 9, White moves pawn from h2 to h3. Loek Van Wely moves black rook from f8 to e8.
    In move 10, White moves rook from f1 to e1. Loek Van Wely moves black queen from d8 to b6.
    In move 11, White moves pawn from d4 to d5. Black moves knight from d7 to c5.
    In move 12, White moves rook from a1 to b1. Black moves pawn from a7 to a5.
    In move 13, White moves bishop from c1 to e3. Black moves queen from b6 to c7.
    In move 14, Fred Van der Vliet moves white knight from f3 to d2. Loek Van Wely moves black knight from f6 to h5.
    In move 15, White moves queen from d1 to e2. Black moves pawn from h7 to h6.
    In move 16, White moves pawn from d5 to c6. White takes black pawn. Black moves queen from c7 to c6. Black takes white pawn.
    In move 17, White moves knight from c3 to d5. Black moves bishop from c8 to e6.
    In move 18, White moves rook from e1 to c1. Black moves king from g8 to h7.
    In move 19, White moves pawn from b2 to b3. Black moves rook from e8 to b8.
    In move 20, White moves pawn from a2 to a3. Loek Van Wely moves black pawn from b7 to b6.
    In move 21, White moves king from g1 to h2. Black moves queen from c6 to c8.
    In move 22, Fred Van der Vliet moves white pawn from f2 to f4. Black moves pawn from e5 to f4. Black takes white pawn.
    In move 23, Fred Van der Vliet moves white pawn from g3 to f4. White takes black pawn. Black moves rook from a8 to a7.
    In move 24, Fred Van der Vliet moves white knight from d5 to c3. Black moves queen from c8 to d8.
    In move 25, White moves knight from c3 to b5. Black moves rook from a7 to d7.
    In move 26, Fred Van der Vliet moves white pawn from b3 to b4. Black moves pawn from a5 to b4. Black takes white pawn.
    In move 27, White moves pawn from a3 to b4. White takes black pawn. Black moves knight from c5 to a6.
    In move 28, White moves knight from d2 to f3. Loek Van Wely moves black queen from d8 to f6.
    In move 29, White moves queen from e2 to d2. Black moves rook from b8 to c8.
    In move 30, White moves knight from b5 to d6. White takes black pawn. Black moves bishop from e6 to c4. Black takes white pawn.
    In move 31, White moves pawn from e4 to e5. Loek Van Wely moves black queen from f6 to d8.
    In move 32, White moves pawn from b4 to b5. Loek Van Wely moves black knight from a6 to b8.
    In move 33, White moves queen from d2 to b4. Black moves rook from d7 to d6. Black takes white knight.
    In move 34, White moves pawn from e5 to d6. White takes black rook. Black moves bishop from c4 to d3.
    In move 35, White moves rook from c1 to c8. White takes black rook. Black moves queen from d8 to c8. Black takes white rook.
    In move 36, White moves rook from b1 to d1. Black moves bishop from d3 to e2.
    In move 37, White moves rook from d1 to c1. Black moves queen from c8 to e8.
    In move 38, Fred Van der Vliet moves white bishop from e3 to b6. White takes black pawn. Black moves bishop from e2 to f3. Black takes white knight.
    In move 39, White moves bishop from g2 to f3. White takes black bishop. Loek Van Wely moves black knight from h5 to f4. Black takes white pawn.
    In move 40, Fred Van der Vliet moves white rook from c1 to e1. Loek Van Wely moves black bishop from g7 to e5.
    In move 41, White moves rook from e1 to e5. White takes black bishop. Loek Van Wely moves black queen from e8 to e5. Black takes white rook.
    In move 42, White moves queen from b4 to d4. Black moves queen from e5 to b5. Black takes white pawn.
    In move 43, White moves queen from d4 to f4. White takes black knight. Black moves queen from b5 to b6. Black takes white bishop.
    In move 44, Fred Van der Vliet moves white queen from f4 to f7. White takes black pawn. Check. Black moves king from h7 to h8.
    In move 45, White moves queen from f7 to f6. Check. Black moves king from h8 to h7.
    In move 46, White moves bishop from f3 to e4. Black moves queen from b6 to a7.
    In move 47, Fred Van der Vliet wins! Loek Van Wely resigns.
    

GOES

  • Adding better plotting of intermediate values
  • Tweaking the margins, which is not adjusted by the plt.margins() method. It’s actually plt.subplots_adjust(). That took entirely too long to find.
  • Got the plot working, and it is super cool:

epochs

Tried a lower latent dimension (8) and fewer neurons in the generator (32). Not really any different. It does take longer to stabilize though.

epochs

  • Upping the strides on the discriminator to 4. Definitely less mode collapse

epochsNoise_trainedacc_loss

  • Trying a stride of 8. That seems even better, though there is this funky artifact on the right. Going to try it with 10,000 epochs:

Noise_trained

  • Nope, no better. Sticking with the stride of 4. And 5,000 epochs seems to be enough to settle. This really looks a lot like the previous stride=4 run above:

epochsNoise_trainedacc_loss

  • Ok, back to Advanced Deep Learning with Keras and GANS
  • The last part of chapter 4 covered conditional GANS, or CGANS. I think that could be very nice to associate a particular type of noise with the state on a sim. For example, daylight vs. eclipse. It looks pretty straightforward, just the concatenation of a state to the input and discriminator vectors:

CGAN

Phil 5.20.20

Graduation today starting at 10:00!

its_official

D20

  • Looks like ASRC actually wanted us to build a completely different marketing app that would be done by 3 people in 3-4 Fridays. So no sponsorship there

GPT2 Agents

  • Adding promotion – done. I have to say that I’m pretty pleased with how the parser handles it. We go from:
    62. a8=Q+ Kg1 63. Qa7 Kg2

    To:

    [62] expanded: 
    	white: F Caruana moves white pawn from a7 to a8. White pawn is promoted to white queen. Check.
    	black: Ding Liren moves black king from g2 to g1.
    
    [63] expanded: 
    	white: F Caruana moves white queen from a8 to a7.
    	black: Ding Liren moves black king from g1 to g2.

     

  • Had a bit of trouble figuring out how to deal with the end of a file with a potentially incomplete game.
  • Everything works!
  • Now I need to write games out to files, do something with the introductions, etc.
  • Also add some variability in the language later
  • Also need to handle queenside castling (O-O-O)

GOES

  • See how the 20,000 epoch run went, and start on Wasserstein Loss
  • The 20,000 epoch really wasn’t any better. rerunning the 10k and saving the model
  • The 10k version seems to be wider and fits better in the bounds. Looking at the two charts, it looks like following the loss/accuracy of the fake data make be best in this case:
  • Reading the GAN chapter in Advanced Deep Learning with Keras (github)
    • Adding a plot of plots to show the appearance of the generator over the epochs. It should help in understanding the evolution of the generator
    • Bless StackOverflow
      side = 4
      fig = plt.figure(1)
      axs = fig.subplots(side, side)
      fig.set_figheight(15)
      fig.set_figwidth(15)
      for i in range(side*side):
          X = gen_data()
          plot_grid(axs, X.T, i, side=side)
      
      fig = plt.figure(2)
      plt.plot([1,2,3,4,3,2,1,2,3], label = "GREEN", color="green")
      plt.plot([1, 2, 3, 4, 5, 6, 7, 8, 9], label = "RED", color="red")
      plt.legend(loc="upper left")
      plt.show()
  • 2:00 Meeting
    • Write up a code walkthrough proposal and send to Erik
    • set up 10:00 meeting with Vadim for tomorrow
  • 3:00 Log file meeting (T & Isaac)
    • Demo’d LMN and RB

#COVID

  • Good discussion about data and paper writing
  • Tried to see if the annotated twitter data is in the DB. Some is, some is not
  • Tried to get the transformers translate example running again, including downloading the files explicitly. Still didn’t work. Submitted a ticket

Phil 5.19.20

Groceries today. In the Before Time, this meant swinging by the store for fresh fruit and veggies while picking up something interesting for dinner that night. Now it means going to two stores (because shortages) standing in separated lines and getting two weeks of food. Very not fun

A Comprehensive guide to Fine-tuning Deep Learning Models in Keras (Part I)

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

The Illustrated Transformer

Collective Intelligence 2020 explores the impact of technology and big data on the ways in which people come together to communicate, combine knowledge and get work done. (Thursday June 18)

Attention

Attention seq2seq example

#COVID

  • Tried to get the HuggingFace translator (colab here) to download and run, but I got this error: ‘Helsinki-NLP/opus-mt-en-ROMANCE’ ‘Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted‘ I’m going to try downloading the model and vocab into a folder within the project to see if something’s missing using these calls:
    # to put the model in a named directory, load the model and tokenizer and then save (as per https://huggingface.co/transformers/quickstart.html):
    tokenizer.save('pretrained/opus-mt-en-ROMANCE')
    model.save('pretrained/opus-mt-en-ROMANCE')

GPT-2 Agents

  • Read in multiple games
    • Handled unconventional names
    • Handling moves split across lines – done
    • Need to handle promotion (a8=Q+), with piece change and added commentary. This is going to be tricky because the ‘Q’ is detected as a piece even though it appears after  the location. Very special case.
      • Rule: A pawn promotion move has =Q =N =R or =B appended to that.
  • Create a short intro
  • Save to txt file

GOES

  • Read the GAN chapter of Generative Deep Learning last night, so now I have a better understanding of Oscillating Loss, Mode Collapse, Wasserstein Loss, the Lipschitz Constraint, and Gradient Penalty Loss. I think I also understand how to set up the callbacks to implement
  • Since the MLP probably wasn’t the problem, go back to using that for the generator and focus on improving the discriminator.
  • That made a big difference!

acc_lossNoise_untrainedNoise_trained

  • The trained version is a pretty good example of mode collapse. I think we can work on improving the discriminator 🙂
  • This approach is already better at finding the amplitude in the noise samples from last week!

Noise_trained

  • Ok, going back to the sin waves to work on mode collapse. I’m going to have lower-amplitude sin waves as well
  • That seems like a good place to start
  • Conv1D(filters=self.vector_size, kernel_size=4, strides=1, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))
  • Conv1D(filters=self.vector_size, kernel_size=6, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1)):
  • Conv1D(filters=self.vector_size, kernel_size=8, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))
  • The input vector size here is only 20 dimensions. So this means that the kernel size is 80% of the vector! Conv1D(filters=self.vector_size, kernel_size=16, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))
  • Upped the vector size from 20 to 32
  • Tried using MaxPool1D but had weird reshape errors. Doing one more pass with two layers before starting to play with Wasserstein Loss, which i think is a better way to go. First though, let’s try longer trainings.
  • 10,000 epochs:
  • 20,000 epochs:

Phil 5.18.20

Call RV repair – done!

D20

  • Aaron started to poke around in his code again and has some thoughts about how to get the sensitivity we need. I realize that the display that we’re working out will be a good dashboard for seeing second waves. We had a good chat on this and the advantage of using multiple sampling windows, possibly of 7, 14, 30 and all days to get a sense of where things are going.
  • Georgia seems to be heading back up?

Georgia

#COVID

  • Public Health and Online Misinformation: Challenges and Recommendations
    • The internet has become a popular resource to learn about health and to investigate one’s own health condition. However, given the large amount of inaccurate information online, people can easily become misinformed. Individuals have always obtained information from outside the formal health care system, so how has the internet changed people’s engagement with health information? This review explores how individuals interact with health misinformation online, whether it be through search, user-generated content, or mobile apps. We discuss whether personal access to information is helping or hindering health outcomes and how the perceived trustworthiness of the institutions communicating health has changed over time. To conclude, we propose several constructive strategies for improving the online information ecosystem. Misinformation concerning health has particularly severe consequences with regard to people’s quality of life and even their risk of mortality; therefore, understanding it within today’s modern context is an extremely important task.

GPT-2 Agents

  • Finish up PGNtoEnglish?
  • Oops, had the black queen in the wrong place
  • Got it working for one game. Need to try out a bunch next!
    Moves:
    [1] expanded: 
    	white:  Fred Van der Vliet moves white pawn from d2 to d4.
    	black:  Loek Van Wely moves black knight from g8 to f6.
    	this is a comment
    
    [2] expanded: 
    	white:  Fred Van der Vliet moves white pawn from c2 to c4.
    	black:  Loek Van Wely moves black pawn from g7 to g6.
    
    [3] expanded: 
    	white:  Fred Van der Vliet moves white pawn from g2 to g3.
    	black:  Loek Van Wely moves black bishop from f8 to g7.
    
    [4] expanded: 
    	white:  Fred Van der Vliet moves white bishop from f1 to g2.
    	black:  Loek Van Wely kingside castles.
    
    [5] expanded: 
    	white:  Fred Van der Vliet moves white knight from g1 to f3.
    	black:  Loek Van Wely moves black pawn from c7 to c6.
    
    [6] expanded: 
    	white:  Fred Van der Vliet kingside castles.
    	black:  Loek Van Wely moves black pawn from d7 to d6.
    
    [7] expanded: 
    	white:  Fred Van der Vliet moves white knight from b1 to c3.
    	black:  Loek Van Wely moves black knight from b8 to d7.
    
    [8] expanded: 
    	white:  Fred Van der Vliet moves white pawn from e2 to e4.
    	black:  Loek Van Wely moves black pawn from e7 to e5.
    
    [9] expanded: 
    	white:  Fred Van der Vliet moves white pawn from h2 to h3.
    	black:  Loek Van Wely moves black rook from f8 to e8.
    
    [10] expanded: 
    	white:  Fred Van der Vliet moves white rook from f1 to e1.
    	black:  Loek Van Wely moves black queen from d8 to b6.
    
    [11] expanded: 
    	white:  Fred Van der Vliet moves white pawn from d4 to d5.
    	black:  Loek Van Wely moves black knight from d7 to c5.
    
    [12] expanded: 
    	white:  Fred Van der Vliet moves white rook from a1 to b1.
    	black:  Loek Van Wely moves black pawn from a7 to a5.
    
    [13] expanded: 
    	white:  Fred Van der Vliet moves white bishop from c1 to e3.
    	black:  Loek Van Wely moves black queen from b6 to c7.
    
    [14] expanded: 
    	white:  Fred Van der Vliet moves white knight from f3 to d2.
    	black:  Loek Van Wely moves black knight from f6 to h5.
    
    [15] expanded: 
    	white:  Fred Van der Vliet moves white queen from d1 to e2.
    	black:  Loek Van Wely moves black pawn from h7 to h6.
    
    [16] expanded: 
    	white:  Fred Van der Vliet moves white pawn from d5 to c6. White takes black pawn.
    	black:  Loek Van Wely moves black queen from c7 to c6. Black takes white pawn.
    
    [17] expanded: 
    	white:  Fred Van der Vliet moves white knight from c3 to d5.
    	black:  Loek Van Wely moves black bishop from c8 to e6.
    
    [18] expanded: 
    	white:  Fred Van der Vliet moves white rook from e1 to c1.
    	black:  Loek Van Wely moves black king from g8 to h7.
    
    [19] expanded: 
    	white:  Fred Van der Vliet moves white pawn from b2 to b3.
    	black:  Loek Van Wely moves black rook from e8 to b8.
    
    [20] expanded: 
    	white:  Fred Van der Vliet moves white pawn from a2 to a3.
    	black:  Loek Van Wely moves black pawn from b7 to b6.
    
    [21] expanded: 
    	white:  Fred Van der Vliet moves white king from g1 to h2.
    	black:  Loek Van Wely moves black queen from c6 to c8.
    
    [22] expanded: 
    	white:  Fred Van der Vliet moves white pawn from f2 to f4.
    	black:  Loek Van Wely moves black pawn from e5 to f4. Black takes white pawn.
    
    [23] expanded: 
    	white:  Fred Van der Vliet moves white pawn from g3 to f4. White takes black pawn.
    	black:  Loek Van Wely moves black rook from a8 to a7.
    
    [24] expanded: 
    	white:  Fred Van der Vliet moves white knight from d5 to c3.
    	black:  Loek Van Wely moves black queen from c8 to d8.
    
    [25] expanded: 
    	white:  Fred Van der Vliet moves white knight from c3 to b5.
    	black:  Loek Van Wely moves black rook from a7 to d7.
    
    [26] expanded: 
    	white:  Fred Van der Vliet moves white pawn from b3 to b4.
    	black:  Loek Van Wely moves black pawn from a5 to b4. Black takes white pawn.
    
    [27] expanded: 
    	white:  Fred Van der Vliet moves white pawn from a3 to b4. White takes black pawn.
    	black:  Loek Van Wely moves black knight from c5 to a6.
    
    [28] expanded: 
    	white:  Fred Van der Vliet moves white knight from d2 to f3.
    	black:  Loek Van Wely moves black queen from d8 to f6.
    
    [29] expanded: 
    	white:  Fred Van der Vliet moves white queen from e2 to d2.
    	black:  Loek Van Wely moves black rook from b8 to c8.
    
    [30] expanded: 
    	white:  Fred Van der Vliet moves white knight from b5 to d6. White takes black pawn.
    	black:  Loek Van Wely moves black bishop from e6 to c4. Black takes white pawn.
    
    [31] expanded: 
    	white:  Fred Van der Vliet moves white pawn from e4 to e5.
    	black:  Loek Van Wely moves black queen from f6 to d8.
    
    [32] expanded: 
    	white:  Fred Van der Vliet moves white pawn from b4 to b5.
    	black:  Loek Van Wely moves black knight from a6 to b8.
    
    [33] expanded: 
    	white:  Fred Van der Vliet moves white queen from d2 to b4.
    	black:  Loek Van Wely moves black rook from d7 to d6. Black takes white knight.
    
    [34] expanded: 
    	white:  Fred Van der Vliet moves white pawn from e5 to d6. White takes black rook.
    	black:  Loek Van Wely moves black bishop from c4 to d3.
    
    [35] expanded: 
    	white:  Fred Van der Vliet moves white rook from c1 to c8. White takes black rook.
    	black:  Loek Van Wely moves black queen from d8 to c8. Black takes white rook.
    
    [36] expanded: 
    	white:  Fred Van der Vliet moves white rook from b1 to d1.
    	black:  Loek Van Wely moves black bishop from d3 to e2.
    
    [37] expanded: 
    	white:  Fred Van der Vliet moves white rook from d1 to c1.
    	black:  Loek Van Wely moves black queen from c8 to e8.
    
    [38] expanded: 
    	white:  Fred Van der Vliet moves white bishop from e3 to b6. White takes black pawn.
    	black:  Loek Van Wely moves black bishop from e2 to f3. Black takes white knight.
    
    [39] expanded: 
    	white:  Fred Van der Vliet moves white bishop from g2 to f3. White takes black bishop.
    	black:  Loek Van Wely moves black knight from h5 to f4. Black takes white pawn.
    
    [40] expanded: 
    	white:  Fred Van der Vliet moves white rook from c1 to e1.
    	black:  Loek Van Wely moves black bishop from g7 to e5.
    
    [41] expanded: 
    	white:  Fred Van der Vliet moves white rook from e1 to e5. White takes black bishop.
    	black:  Loek Van Wely moves black queen from e8 to e5. Black takes white rook.
    
    [42] expanded: 
    	white:  Fred Van der Vliet moves white queen from b4 to d4.
    	black:  Loek Van Wely moves black queen from e5 to b5. Black takes white pawn.
    
    [43] expanded: 
    	white:  Fred Van der Vliet moves white queen from d4 to f4. White takes black knight.
    	black:  Loek Van Wely moves black queen from b5 to b6. Black takes white bishop.
    
    [44] expanded: 
    	white:  Fred Van der Vliet moves white queen from f4 to f7. Check. White takes black pawn.
    	black:  Loek Van Wely moves black king from h7 to h8.
    
    [45] expanded: 
    	white:  Fred Van der Vliet moves white queen from f7 to f6. Check.
    	black:  Loek Van Wely moves black king from h8 to h7.
    
    [46] expanded: 
    	white:  Fred Van der Vliet moves white bishop from f3 to e4.
    	black:  Loek Van Wely moves black queen from b6 to a7.
    
    [47] expanded: 
    	white:  Fred Van der Vliet wins!
    	black:  Loek Van Wely resigns.
    
    
    
    Process finished with exit code 0
    

GOES

  • See if I can get a GAN running with the new parts. I am not convinced that the input vector and times series are set up right. I probably need to set up for testing that as a parameter as well.
  • Still, they seem reasonable. For testing, I have an input vector of 5 and 10 samples, like so:
    [[ 0.99611333]
      [-0.49462109]
      [-0.58444332]
      [ 0.98104957]
      [-0.23207803]]
    
     [[ 0.70418362]
      [ 0.35257367]
      [-0.99762846]
      [ 0.47774618]
      [ 0.60000333]]
  • My latent matrix is 4 wide by 10 samples, arranged like so:
    [  2.46776301  17.36723783  -3.05622169 -15.7807726 ]
     [  4.73444423 -13.03790398  -0.04382812 -17.16272464]
  • And my predicts seem reasonable, and they match the real data exactly, both from the Dense->CNN and the Dense versions
  • Dense (2 steps of 5-feature vector)
    [[-1.        ]
      [ 1.        ]
      [-1.        ]
      [-1.        ]
      [ 1.        ]]
    
     [[-0.845926  ]
      [ 0.98410994]
      [-0.9970485 ]
      [-0.99517673]
      [ 0.910765  ]]
  • Dense->CNN (2 steps of 5-feature vector)
    [[ 0.9974974 ]
      [-0.7852714 ]
      [-0.8944576 ]
      [ 0.99998605]
      [-0.34316158]]
    
     [[-0.9874317 ]
      [-0.90552104]
      [ 0.95992273]
      [ 0.8134167 ]
      [-0.94480175]]
  • And all the predicts from the detector have the same form of (10, 1):
    [[0.5464545 ]
     [0.5419694 ]
     [0.5464503 ]
     [0.5464593 ]
     [0.5464545 ]
     [0.5462216 ]
     [0.45328763]
     [0.541866  ]
     [0.48022035]
     [0.54643464]]
  • So I think I’m just going to try to build a OneDGAN2 class, evaluate that, and then if it’s working well, build a NoiseGAN2 class and use that for the paper/presentation
  • Back to being able to draw plots, which means that the model is being created without blowing up!

Noise_untrained

  • So I made some changes, basically added many more samples, played around with the size of the latent space and so forth. And I got a very unusual result when looking at the untrained model:

Noise_untrained

  • I think this may be related to the use of a dense layer between the latent data and the Conv1D. Need to do some more reading, I guess.

Phil 5.15.20

Fridays are hard. I feel like I need a break from pushing this rock up hill alone. Nice day for a ride tomorrow, so a few of us will probably meet up.

D20

  • Zach seems to be making progress in fits and starts. No word from Aaron
  • One way to make the system more responsive is to see if the rates are above or below the regression. Above can be flagged.

GPT-2 Agents

  • More PGNtoEnglish. Getting close I think.
  • Added pawn attack moves (diagonals)
  • Adding comment regex – done
  • Some problem handling this:
    Evaluating move [Re1 Qb6]
    search at (-6, -6) for black queen out of bounds
    search at (6, -6) for black queen out of bounds
    search at (0, -6) for black queen out of bounds
    search at (7, 7) for black queen out of bounds
    search at (-7, -7) for black queen out of bounds
    search at (7, -7) for black queen out of bounds
    search at (7, 0) for black queen out of bounds
    search at (0, -7) for black queen out of bounds
    raw: white: Re1, black: Qb6
    	expanded: white:  Fred Van der Vliet moves white rook from f1 to e1.
    	black: unset

GOES

  • Need to make the output of the generator work as input to the discriminator.
  • So I need to get to the input vector of latent noise to an output that is the size of the real data. It’s easy to do with Dense, but Dense and Conv1D don’t get along. I think I can get around that by reshaping the dense layer to something that a Conv1D can take. But that probably looses a lot of information, since each neuron will have some of each noise sample in it. But the information is noise in the first place, so it’s just resampled noise? The other option is to upsample, but that requires the latent vector to divide evenly into the input vector for the discriminator.
  • Here’s my code that does the change from a dense to Conv1D:
    self.g_model.add(Dense(self.vector_size*self.num_samples, activation='relu', batch_input_shape=(self.latent_dim, self.num_samples)))
    self.g_model.add(Reshape(target_shape=(self.vector_size, self.num_samples)))
    self.g_model.add(Conv1D(filters=self.vector_size, kernel_size=5, activation='tanh', batch_input_shape=(self.vector_size, self.num_samples, 1)))
  • The code that produces the latent noise is:
    def generate_latent_points(self, span:float=1.0) -> np.array:
        x_input = np.random.randn(self.latent_dim * self.num_samples)*self.span
        # reshape into a batch of inputs for the network
        x_input = x_input.reshape(self.latent_dim, self.num_samples)
        return x_input
  • The “real” values are:
    real_matrix = 
    [[-0.34737792 -0.7081109   0.93673414 -0.071527   -0.87720268]
     [ 0.99876073 -0.46088645 -0.61516785  0.97288676 -0.19455964]
     [ 0.97121222 -0.18755778 -0.81510907  0.8659679   0.09436946]
     [-0.72593361 -0.32328777  0.99500398 -0.50484775 -0.57482239]
     [ 0.72944027 -0.92555418  0.04089262  0.89151951 -0.78289867]
     [ 0.79514567 -0.88231211 -0.06080288  0.93291797 -0.71565884]
     [ 0.78083404 -0.89301473 -0.03758353  0.92429527 -0.73170157]
     [ 0.08266474 -0.94058595  0.70017899  0.3578314  -0.9979998 ]
     [-0.39534886 -0.67069473  0.95356385 -0.12295042 -0.85123299]
     [ 0.73424796  0.31175013 -0.99371562  0.5153131   0.56482379]]
  • The latent values are (note that the matrix is transposed):
    latent_matrix = 
    [[  8.73701754   6.10841293   9.31566343  -2.00751851   0.10715919
        6.94580853  -6.95308374   6.97502697 -11.09777023  -8.79311041]
     [ -3.61789323   0.11091496  10.94717459   3.14579647 -13.23974342
        2.78914476   9.40101397 -17.75756896   2.87461527   6.65877192]
     [  5.77331701   7.71326491   9.9877786   -3.81972802  -5.86490109
       -6.68585542 -13.59478633  -7.66952834 -10.78863284   5.9248856 ]
     [ -3.05226511  -5.36347909   1.3377953   14.87752343  -0.21993387
      -13.47737126   1.39357385  -1.85004465   6.83400948   1.21105276]]
  • The values created by the generator are:
    predict_matrix = 
    [[[-0.9839389   0.18747564 -0.9449842  -0.66334486 -0.9822154 ]]
     [[ 0.9514655  -0.9985579   0.76473945 -0.9985249  -0.9828463 ]]
     [[-0.58794653 -0.9982161   0.9855345  -0.93976855 -0.9999758 ]]
     [[-0.9987122   0.9480774  -0.80395573 -0.999845    0.06755089]]]
  • So now I need to get the number of rows up to the same value as the real data
  • Ok, so here’s how that works. We use tf.Keras.Reshape(), which is pretty simple. You simply put the most of shape you want as the single argument and it. So for these experiments, I had ten rows of 5 features, plus an extra dimension. So you would think that reshape(10,5,1) would be what you want.
  • Au contraire! Keras wants to be able to have flexibility, so one dimension is left to vary. The argument is actually (5, 1). Here are two versions. First is a generator using a Dense network:
    def define_generator_Dense(self) -> Sequential:
        self.g_model_Dense = Sequential()
    
        self.g_model_Dense.add(Dense(4, activation='relu', kernel_initializer='he_uniform', input_dim=self.latent_dim))
        self.g_model_Dense.add(Dropout(0.2))
        self.g_model_Dense.add(Dense(self.vector_size, activation='tanh')) # activation was linear
        self.g_model_Dense.add(Reshape((self.vector_size, 1)))
        print("g_model_Dense.output_shape = {}".format(self.g_model_Dense.output_shape))
    
        # compile model
        loss_func = tf.keras.losses.BinaryCrossentropy()
        opt_func = tf.keras.optimizers.Adam(0.001)
        self.g_model_Dense.compile(loss=loss_func, optimizer=opt_func)
    
        return self.g_model_Dense
  • Second is a network using Conv1D layers
    def define_generator_Dense_to_CNN(self) -> Sequential:
        self.g_model_Dense_CNN = Sequential()
        self.g_model_Dense_CNN.add(Dense(self.num_samples * self.vector_size, activation='relu', batch_input_shape=(self.num_samples, self.latent_dim)))
        self.g_model_Dense_CNN.add(Reshape(target_shape=(self.num_samples, self.vector_size)))
        self.g_model_Dense_CNN.add(Conv1D(filters=self.vector_size, kernel_size=self.num_samples, activation='tanh', batch_input_shape=(self.num_samples, self.vector_size, 1))) # activation was linear
        self.g_model_Dense_CNN.add(Reshape((self.vector_size, 1)))
        #self.g_model.add(UpSampling1D(size=2))
    
        # compile model
        loss_func = tf.keras.losses.BinaryCrossentropy()
        opt_func = tf.keras.optimizers.Adam(0.001)
        self.g_model_Dense_CNN.compile(loss=loss_func, optimizer=opt_func)
        return self.g_model_Dense_CNN

    :

  • Both evaluated correctly against the discriminator, so I should be able to train the whole GAN, once it’s assembled. But that is not something to start at 4:30 on a Friday afternoon!
    real predict = (10, 1)[[0.42996567]
     [0.55048925]
     [0.56003207]
     [0.40951794]
     [0.5600004 ]
     [0.5098837 ]
     [0.4046895 ]
     [0.41493616]
     [0.4196912 ]
     [0.5080263 ]]
    gdense_mat predict = (10, 1)[[0.48928624]
     [0.5       ]
     [0.4949373 ]
     [0.5       ]
     [0.5973854 ]
     [0.61968124]
     [0.49698165]
     [0.5       ]
     [0.5183723 ]
     [0.4212265 ]]
    gdcnn_mat predict = (10, 1)[[0.48057705]
     [0.5026125 ]
     [0.51943815]
     [0.4902147 ]
     [0.5988    ]
     [0.39476413]
     [0.49915075]
     [0.49861506]
     [0.55501187]
     [0.54503495]]

     

Phil 5.14.20

GPT-2 Agents

  • Adding hints and meta information. Still need to handle special pawn moves and pull out comments
    • /[^{\}]+(?=})/g, from https://stackoverflow.com/questions/413071/regex-to-get-string-between-curly-braces
  • Amusingly, my simple parser is now at 560 LOC and counting

GOES

  • Working on creating a discriminator using Conv1D layers.
    • With some help from Aaron, I got the discriminator working. There are some issues. I’m currently using batch_input_shape rather than input_shape, which beans that a pre-sized batch is compiled in. The second issue is that the discriminator requires a 3D vector to be fed in, which can’t be produced naturally with Dense/MLP. That means the Generator also has to use Conv1D at least at the output layer
    • I think this post: How to Develop 1D Convolutional Neural Network Models for Human Activity Recognition should help, but I don’t think I have the cognitive ability at the end of the day. Tomorrow

Phil 5.13.20

Book

D20

  • Zach appears happy with the changes

#COVID

  • The Arabic finetuning didn’t work. Drat.
  • This could, though….

Translate

GPT-2 Agents

  • Need to handle hints, takes, check, and checkmate:
    num_regex = re.compile('[^0-9]')
    alpha_regex = re.compile('[0-9]')
    hints_regex = re.compile('[KQNBRx+]')
    
    def parse_move(m):
        hints = []
        cleaned = hints_regex.sub('', m)
    
        if '++' in m:
            hints.append("checkmate")
        elif '+' in m:
            hints.append("check")
        if 'x' in m:
            hints.append("takes")
        if len(cleaned) > 2:
            hints.append(cleaned[0])
            cleaned = cleaned[1:]
    
        piece = piece_regex.sub('', m)
        num = num_regex.sub('', cleaned)
        letter = alpha_regex.sub('', cleaned)
        return "{}/{}: piece = [{}], square = ({}, {}), hints = {}".format(m, cleaned, piece, letter, num, hints)
  • Roll this in tomorrow. Comments {Anything in curly brackets}? Also, it looks like there are more meta tags (Note player name with no comma!):
    [Black "Ding Liren"]
    [WhiteTitle "GM"]
    [BlackTitle "GM"]
    [Opening "Sicilian"]
    [Variation "Najdorf"]
    [WhiteFideId "2020009"]
    [BlackFideId "8603677"]

GOES

  • More with TF-GAN from this Google course on GANs
    • Mode Collapse is why the GAN keeps generating a single waveform
    • Need to contact Joel shor as Google. Sent a note on LinkedIn and a followup email to his Google account (from D:\Development\External\tf-gan\README)
  • GANSynth: Making music with GANs
    • In this post, we introduce GANSynth, a method for generating high-fidelity audio with Generative Adversarial Networks (GANs).
  • 10 Lessons I Learned Training GANs for one Year
  • Advanced Topics in GANs
  • As I lower the number of neurons in the generator, it starts to look better, but now there are odd artifacts in the untrained data :
    self.g_model.add(Dense(5, activation='relu', kernel_initializer='he_uniform', input_dim=self.latent_dimension))

     

Noise_untrained
Noise_trained

  • Thought I’d try the TF-GAN examples but I get many compatability errors that make me thing that this does not work with TF 2.x. So I decided to try the Google Colab. Aaaaand that doesn’t work either:

ColabError

  • Looking through Generative Deep Learning, it says that CNNs help make the discriminator better:
    • In the original GAN paper, dense layers were used in place of the convolutional layers. However, since then, it has been shown that convolutional layers give greater predictive power to the discriminator. You may see this type of GAN called a DCGAN (deep convolutional generative adversarial network) in the literature, but now essentially all GAN architectures contain convolutional layers, so the “DC” is implied when we talk about GANs. It is also common to see batch normalization layers in the discriminator for vanilla GANs, though we choose not to use them here for simplicity.
  • So, tf.keras.layers.Conv1D
  • 10:00 meeting with Vadim