Phil 5.15.20

Fridays are hard. I feel like I need a break from pushing this rock up hill alone. Nice day for a ride tomorrow, so a few of us will probably meet up.

D20

  • Zach seems to be making progress in fits and starts. No word from Aaron
  • One way to make the system more responsive is to see if the rates are above or below the regression. Above can be flagged.

GPT-2 Agents

  • More PGNtoEnglish. Getting close I think.
  • Added pawn attack moves (diagonals)
  • Adding comment regex – done
  • Some problem handling this:
    Evaluating move [Re1 Qb6]
    search at (-6, -6) for black queen out of bounds
    search at (6, -6) for black queen out of bounds
    search at (0, -6) for black queen out of bounds
    search at (7, 7) for black queen out of bounds
    search at (-7, -7) for black queen out of bounds
    search at (7, -7) for black queen out of bounds
    search at (7, 0) for black queen out of bounds
    search at (0, -7) for black queen out of bounds
    raw: white: Re1, black: Qb6
    	expanded: white:  Fred Van der Vliet moves white rook from f1 to e1.
    	black: unset

GOES

  • Need to make the output of the generator work as input to the discriminator.
  • So I need to get to the input vector of latent noise to an output that is the size of the real data. It’s easy to do with Dense, but Dense and Conv1D don’t get along. I think I can get around that by reshaping the dense layer to something that a Conv1D can take. But that probably looses a lot of information, since each neuron will have some of each noise sample in it. But the information is noise in the first place, so it’s just resampled noise? The other option is to upsample, but that requires the latent vector to divide evenly into the input vector for the discriminator.
  • Here’s my code that does the change from a dense to Conv1D:
    self.g_model.add(Dense(self.vector_size*self.num_samples, activation='relu', batch_input_shape=(self.latent_dim, self.num_samples)))
    self.g_model.add(Reshape(target_shape=(self.vector_size, self.num_samples)))
    self.g_model.add(Conv1D(filters=self.vector_size, kernel_size=5, activation='tanh', batch_input_shape=(self.vector_size, self.num_samples, 1)))
  • The code that produces the latent noise is:
    def generate_latent_points(self, span:float=1.0) -> np.array:
        x_input = np.random.randn(self.latent_dim * self.num_samples)*self.span
        # reshape into a batch of inputs for the network
        x_input = x_input.reshape(self.latent_dim, self.num_samples)
        return x_input
  • The “real” values are:
    real_matrix = 
    [[-0.34737792 -0.7081109   0.93673414 -0.071527   -0.87720268]
     [ 0.99876073 -0.46088645 -0.61516785  0.97288676 -0.19455964]
     [ 0.97121222 -0.18755778 -0.81510907  0.8659679   0.09436946]
     [-0.72593361 -0.32328777  0.99500398 -0.50484775 -0.57482239]
     [ 0.72944027 -0.92555418  0.04089262  0.89151951 -0.78289867]
     [ 0.79514567 -0.88231211 -0.06080288  0.93291797 -0.71565884]
     [ 0.78083404 -0.89301473 -0.03758353  0.92429527 -0.73170157]
     [ 0.08266474 -0.94058595  0.70017899  0.3578314  -0.9979998 ]
     [-0.39534886 -0.67069473  0.95356385 -0.12295042 -0.85123299]
     [ 0.73424796  0.31175013 -0.99371562  0.5153131   0.56482379]]
  • The latent values are (note that the matrix is transposed):
    latent_matrix = 
    [[  8.73701754   6.10841293   9.31566343  -2.00751851   0.10715919
        6.94580853  -6.95308374   6.97502697 -11.09777023  -8.79311041]
     [ -3.61789323   0.11091496  10.94717459   3.14579647 -13.23974342
        2.78914476   9.40101397 -17.75756896   2.87461527   6.65877192]
     [  5.77331701   7.71326491   9.9877786   -3.81972802  -5.86490109
       -6.68585542 -13.59478633  -7.66952834 -10.78863284   5.9248856 ]
     [ -3.05226511  -5.36347909   1.3377953   14.87752343  -0.21993387
      -13.47737126   1.39357385  -1.85004465   6.83400948   1.21105276]]
  • The values created by the generator are:
    predict_matrix = 
    [[[-0.9839389   0.18747564 -0.9449842  -0.66334486 -0.9822154 ]]
     [[ 0.9514655  -0.9985579   0.76473945 -0.9985249  -0.9828463 ]]
     [[-0.58794653 -0.9982161   0.9855345  -0.93976855 -0.9999758 ]]
     [[-0.9987122   0.9480774  -0.80395573 -0.999845    0.06755089]]]
  • So now I need to get the number of rows up to the same value as the real data
  • Ok, so here’s how that works. We use tf.Keras.Reshape(), which is pretty simple. You simply put the most of shape you want as the single argument and it. So for these experiments, I had ten rows of 5 features, plus an extra dimension. So you would think that reshape(10,5,1) would be what you want.
  • Au contraire! Keras wants to be able to have flexibility, so one dimension is left to vary. The argument is actually (5, 1). Here are two versions. First is a generator using a Dense network:
    def define_generator_Dense(self) -> Sequential:
        self.g_model_Dense = Sequential()
    
        self.g_model_Dense.add(Dense(4, activation='relu', kernel_initializer='he_uniform', input_dim=self.latent_dim))
        self.g_model_Dense.add(Dropout(0.2))
        self.g_model_Dense.add(Dense(self.vector_size, activation='tanh')) # activation was linear
        self.g_model_Dense.add(Reshape((self.vector_size, 1)))
        print("g_model_Dense.output_shape = {}".format(self.g_model_Dense.output_shape))
    
        # compile model
        loss_func = tf.keras.losses.BinaryCrossentropy()
        opt_func = tf.keras.optimizers.Adam(0.001)
        self.g_model_Dense.compile(loss=loss_func, optimizer=opt_func)
    
        return self.g_model_Dense
  • Second is a network using Conv1D layers
    def define_generator_Dense_to_CNN(self) -> Sequential:
        self.g_model_Dense_CNN = Sequential()
        self.g_model_Dense_CNN.add(Dense(self.num_samples * self.vector_size, activation='relu', batch_input_shape=(self.num_samples, self.latent_dim)))
        self.g_model_Dense_CNN.add(Reshape(target_shape=(self.num_samples, self.vector_size)))
        self.g_model_Dense_CNN.add(Conv1D(filters=self.vector_size, kernel_size=self.num_samples, activation='tanh', batch_input_shape=(self.num_samples, self.vector_size, 1))) # activation was linear
        self.g_model_Dense_CNN.add(Reshape((self.vector_size, 1)))
        #self.g_model.add(UpSampling1D(size=2))
    
        # compile model
        loss_func = tf.keras.losses.BinaryCrossentropy()
        opt_func = tf.keras.optimizers.Adam(0.001)
        self.g_model_Dense_CNN.compile(loss=loss_func, optimizer=opt_func)
        return self.g_model_Dense_CNN

    :

  • Both evaluated correctly against the discriminator, so I should be able to train the whole GAN, once it’s assembled. But that is not something to start at 4:30 on a Friday afternoon!
    real predict = (10, 1)[[0.42996567]
     [0.55048925]
     [0.56003207]
     [0.40951794]
     [0.5600004 ]
     [0.5098837 ]
     [0.4046895 ]
     [0.41493616]
     [0.4196912 ]
     [0.5080263 ]]
    gdense_mat predict = (10, 1)[[0.48928624]
     [0.5       ]
     [0.4949373 ]
     [0.5       ]
     [0.5973854 ]
     [0.61968124]
     [0.49698165]
     [0.5       ]
     [0.5183723 ]
     [0.4212265 ]]
    gdcnn_mat predict = (10, 1)[[0.48057705]
     [0.5026125 ]
     [0.51943815]
     [0.4902147 ]
     [0.5988    ]
     [0.39476413]
     [0.49915075]
     [0.49861506]
     [0.55501187]
     [0.54503495]]

     

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.