Had a good, cathartic ride yesterday:

GPT-2 Agents

• I’ve been working on the PGNtoEnglish class and was having an odd bug where occasionally a piece would pull a piece from the other side of the board. Since it was intermittent, it required many print statements and searching through the logs for “black knight”

• My problem was in forgetting how Python indexes into arrays. Here’s the code in question:

• When I first wrote this, I had to deal with a lot of potential coordinates that were off the board, with indexes like (-2, -1), or (10, 8) for an 8×8 board. I thought to handle this with a try/except on IndexError (the bottom highlight). In other languages this would have worked, but Python allows negative indexes. Ooops! Adding the test for either index being negative (the top highlight) fixed that bug

D20

• Ping Zach – done

GOES

• Write up code review thoughts for Erik -done
• Add n_critic to base class, along with adjustable false flag value
  • First, making sure that everything still works. Seems to.
  • Here’s the best I can do today, using the OneDGAN2a class with an RMSProp(lr=0.0005)

• Assemble all the bits for an example
  • Verified that the InfluxTestTrainBase still works, and it’s using the InfluxDB values
  • Assemble all the bits for an example
    • Created a NoiseGAN2 with the same amount of points as the InfluxTestTrainBase model – done. Looks real good on the noise, too:

• How to trim the columns on a 2D Numpy array:

  results = self.ifq.run_query(self.bucket, begin, end, filter_str)
  results = self.ifq.to_nd_array(results)
  results = np.delete(results, slice(clamp, None), 1)
  predict_table = model.predict(results)

• Here’s all the parts nailed together:

• Start the paper and the deck

ML Group

• Need to create a walkthrough of coding practices for next week. I think I’ll use the trajectory of the GAN coding as the basis

Groceries today. In the Before Time, this meant swinging by the store for fresh fruit and veggies while picking up something interesting for dinner that night. Now it means going to two stores (because shortages) standing in separated lines and getting two weeks of food. Very not fun

A Comprehensive guide to Fine-tuning Deep Learning Models in Keras (Part I)

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

The Illustrated Transformer

Collective Intelligence 2020 explores the impact of technology and big data on the ways in which people come together to communicate, combine knowledge and get work done. (Thursday June 18)

Attention seq2seq example

#COVID

• Tried to get the HuggingFace translator (colab here) to download and run, but I got this error: ‘Helsinki-NLP/opus-mt-en-ROMANCE’ ‘Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted‘ I’m going to try downloading the model and vocab into a folder within the project to see if something’s missing using these calls:

  # to put the model in a named directory, load the model and tokenizer and then save (as per https://huggingface.co/transformers/quickstart.html):
  tokenizer.save('pretrained/opus-mt-en-ROMANCE')
  model.save('pretrained/opus-mt-en-ROMANCE')

GPT-2 Agents

• Read in multiple games
  • Handled unconventional names
  • Handling moves split across lines – done
  • Need to handle promotion (a8=Q+), with piece change and added commentary. This is going to be tricky because the ‘Q’ is detected as a piece even though it appears after  the location. Very special case.
    • Rule: A pawn promotion move has =Q =N =R or =B appended to that.
• Create a short intro
• Save to txt file

GOES

• Read the GAN chapter of Generative Deep Learning last night, so now I have a better understanding of Oscillating Loss, Mode Collapse, Wasserstein Loss, the Lipschitz Constraint, and Gradient Penalty Loss. I think I also understand how to set up the callbacks to implement
• Since the MLP probably wasn’t the problem, go back to using that for the generator and focus on improving the discriminator.
• That made a big difference!

• The trained version is a pretty good example of mode collapse. I think we can work on improving the discriminator 🙂
• This approach is already better at finding the amplitude in the noise samples from last week!

• Ok, going back to the sin waves to work on mode collapse. I’m going to have lower-amplitude sin waves as well
• That seems like a good place to start
• Conv1D(filters=self.vector_size, kernel_size=4, strides=1, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))

• Conv1D(filters=self.vector_size, kernel_size=6, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1)):

• Conv1D(filters=self.vector_size, kernel_size=8, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))

• The input vector size here is only 20 dimensions. So this means that the kernel size is 80% of the vector! Conv1D(filters=self.vector_size, kernel_size=16, strides=2, activation=’relu’, batch_input_shape=(self.num_samples, self.vector_size, 1))

• Upped the vector size from 20 to 32
• Tried using MaxPool1D but had weird reshape errors. Doing one more pass with two layers before starting to play with Wasserstein Loss, which i think is a better way to go. First though, let’s try longer trainings.
• 10,000 epochs:

• 20,000 epochs:

Fridays are hard. I feel like I need a break from pushing this rock up hill alone. Nice day for a ride tomorrow, so a few of us will probably meet up.

D20

• Zach seems to be making progress in fits and starts. No word from Aaron
• One way to make the system more responsive is to see if the rates are above or below the regression. Above can be flagged.

GPT-2 Agents

• More PGNtoEnglish. Getting close I think.
• Added pawn attack moves (diagonals)
• Adding comment regex – done
• Some problem handling this:

  Evaluating move [Re1 Qb6]
  search at (-6, -6) for black queen out of bounds
  search at (6, -6) for black queen out of bounds
  search at (0, -6) for black queen out of bounds
  search at (7, 7) for black queen out of bounds
  search at (-7, -7) for black queen out of bounds
  search at (7, -7) for black queen out of bounds
  search at (7, 0) for black queen out of bounds
  search at (0, -7) for black queen out of bounds
  raw: white: Re1, black: Qb6
  	expanded: white:  Fred Van der Vliet moves white rook from f1 to e1.
  	black: unset

GOES

• Need to make the output of the generator work as input to the discriminator.
• So I need to get to the input vector of latent noise to an output that is the size of the real data. It’s easy to do with Dense, but Dense and Conv1D don’t get along. I think I can get around that by reshaping the dense layer to something that a Conv1D can take. But that probably looses a lot of information, since each neuron will have some of each noise sample in it. But the information is noise in the first place, so it’s just resampled noise? The other option is to upsample, but that requires the latent vector to divide evenly into the input vector for the discriminator.
• Here’s my code that does the change from a dense to Conv1D:

  self.g_model.add(Dense(self.vector_size*self.num_samples, activation='relu', batch_input_shape=(self.latent_dim, self.num_samples)))
  self.g_model.add(Reshape(target_shape=(self.vector_size, self.num_samples)))
  self.g_model.add(Conv1D(filters=self.vector_size, kernel_size=5, activation='tanh', batch_input_shape=(self.vector_size, self.num_samples, 1)))

• The code that produces the latent noise is:

    def generate_latent_points(self, span:float=1.0) -> np.array:
        x_input = np.random.randn(self.latent_dim * self.num_samples)*self.span
        # reshape into a batch of inputs for the network
        x_input = x_input.reshape(self.latent_dim, self.num_samples)
        return x_input

• The “real” values are:

  real_matrix = 
  [[-0.34737792 -0.7081109   0.93673414 -0.071527   -0.87720268]
   [ 0.99876073 -0.46088645 -0.61516785  0.97288676 -0.19455964]
   [ 0.97121222 -0.18755778 -0.81510907  0.8659679   0.09436946]
   [-0.72593361 -0.32328777  0.99500398 -0.50484775 -0.57482239]
   [ 0.72944027 -0.92555418  0.04089262  0.89151951 -0.78289867]
   [ 0.79514567 -0.88231211 -0.06080288  0.93291797 -0.71565884]
   [ 0.78083404 -0.89301473 -0.03758353  0.92429527 -0.73170157]
   [ 0.08266474 -0.94058595  0.70017899  0.3578314  -0.9979998 ]
   [-0.39534886 -0.67069473  0.95356385 -0.12295042 -0.85123299]
   [ 0.73424796  0.31175013 -0.99371562  0.5153131   0.56482379]]

• The latent values are (note that the matrix is transposed):

  latent_matrix = 
  [[  8.73701754   6.10841293   9.31566343  -2.00751851   0.10715919
      6.94580853  -6.95308374   6.97502697 -11.09777023  -8.79311041]
   [ -3.61789323   0.11091496  10.94717459   3.14579647 -13.23974342
      2.78914476   9.40101397 -17.75756896   2.87461527   6.65877192]
   [  5.77331701   7.71326491   9.9877786   -3.81972802  -5.86490109
     -6.68585542 -13.59478633  -7.66952834 -10.78863284   5.9248856 ]
   [ -3.05226511  -5.36347909   1.3377953   14.87752343  -0.21993387
    -13.47737126   1.39357385  -1.85004465   6.83400948   1.21105276]]

• The values created by the generator are:

  predict_matrix = 
  [[[-0.9839389   0.18747564 -0.9449842  -0.66334486 -0.9822154 ]]
   [[ 0.9514655  -0.9985579   0.76473945 -0.9985249  -0.9828463 ]]
   [[-0.58794653 -0.9982161   0.9855345  -0.93976855 -0.9999758 ]]
   [[-0.9987122   0.9480774  -0.80395573 -0.999845    0.06755089]]]

• So now I need to get the number of rows up to the same value as the real data
• Ok, so here’s how that works. We use tf.Keras.Reshape(), which is pretty simple. You simply put the most of shape you want as the single argument and it. So for these experiments, I had ten rows of 5 features, plus an extra dimension. So you would think that reshape(10,5,1) would be what you want.
• Au contraire! Keras wants to be able to have flexibility, so one dimension is left to vary. The argument is actually (5, 1). Here are two versions. First is a generator using a Dense network:

  def define_generator_Dense(self) -> Sequential:
      self.g_model_Dense = Sequential()
  
      self.g_model_Dense.add(Dense(4, activation='relu', kernel_initializer='he_uniform', input_dim=self.latent_dim))
      self.g_model_Dense.add(Dropout(0.2))
      self.g_model_Dense.add(Dense(self.vector_size, activation='tanh')) # activation was linear
      self.g_model_Dense.add(Reshape((self.vector_size, 1)))
      print("g_model_Dense.output_shape = {}".format(self.g_model_Dense.output_shape))
  
      # compile model
      loss_func = tf.keras.losses.BinaryCrossentropy()
      opt_func = tf.keras.optimizers.Adam(0.001)
      self.g_model_Dense.compile(loss=loss_func, optimizer=opt_func)
  
      return self.g_model_Dense

• Second is a network using Conv1D layers

  def define_generator_Dense_to_CNN(self) -> Sequential:
      self.g_model_Dense_CNN = Sequential()
      self.g_model_Dense_CNN.add(Dense(self.num_samples * self.vector_size, activation='relu', batch_input_shape=(self.num_samples, self.latent_dim)))
      self.g_model_Dense_CNN.add(Reshape(target_shape=(self.num_samples, self.vector_size)))
      self.g_model_Dense_CNN.add(Conv1D(filters=self.vector_size, kernel_size=self.num_samples, activation='tanh', batch_input_shape=(self.num_samples, self.vector_size, 1))) # activation was linear
      self.g_model_Dense_CNN.add(Reshape((self.vector_size, 1)))
      #self.g_model.add(UpSampling1D(size=2))
  
      # compile model
      loss_func = tf.keras.losses.BinaryCrossentropy()
      opt_func = tf.keras.optimizers.Adam(0.001)
      self.g_model_Dense_CNN.compile(loss=loss_func, optimizer=opt_func)
      return self.g_model_Dense_CNN

  :

• Both evaluated correctly against the discriminator, so I should be able to train the whole GAN, once it’s assembled. But that is not something to start at 4:30 on a Friday afternoon!

  real predict = (10, 1)[[0.42996567]
   [0.55048925]
   [0.56003207]
   [0.40951794]
   [0.5600004 ]
   [0.5098837 ]
   [0.4046895 ]
   [0.41493616]
   [0.4196912 ]
   [0.5080263 ]]
  gdense_mat predict = (10, 1)[[0.48928624]
   [0.5       ]
   [0.4949373 ]
   [0.5       ]
   [0.5973854 ]
   [0.61968124]
   [0.49698165]
   [0.5       ]
   [0.5183723 ]
   [0.4212265 ]]
  gdcnn_mat predict = (10, 1)[[0.48057705]
   [0.5026125 ]
   [0.51943815]
   [0.4902147 ]
   [0.5988    ]
   [0.39476413]
   [0.49915075]
   [0.49861506]
   [0.55501187]
   [0.54503495]]