Phil 10.30.20

So yesterday the oven broke, and today it’s the fridge

GPT-2 Agents

  • Try finetuning the large model on the large dev machine. While trying out the gpt2-large model (because the gpt2-xl model didn’t work), I had an odd problem. When I tried to finetune the model using my local file (which I had saved earlier), the system choked on the lack of a pytorch_model.bin, which must not download when you just want to use the model itself. My guess is that if you don’t specify the file, it will probably download and work, but my default drive is a small SSD, and I don’t want to load it up.
  • To see what was going on, I used a script that I had used before to find where transformers stores the model:
from transformers.file_utils import hf_bucket_url, cached_path

pretrained_model_name = f'gpt2-large'
archive_file = hf_bucket_url(
resolved_archive_file = cached_path(archive_file)
  • That printed out the following:


  • After renaming to pytorch_model.bin and moving it to my model directory, the finetuning is now working!
  • At around 10:30 last night, the checkpoints had filled up my 1TB data drive! Tried a bunch of things to use a checkpoint for restarting from chackpoint. Pointing the model to the checkpoint seems to be the right answer, but it was missing the vocab and merges.txt files. Tried to pull that over from the original model, but now I get a:
Traceback (most recent call last):
   File "", line 355, in 
   File "", line 319, in main
   File "D:\Program Files\Python37\lib\site-packages\transformers\", line 621, in train
     train_dataloader = self.get_train_dataloader()
   File "D:\Program Files\Python37\lib\site-packages\transformers\", line 417, in get_train_dataloader
     train_sampler = self._get_train_sampler()
   File "D:\Program Files\Python37\lib\site-packages\transformers\", line 402, in _get_train_sampler
     if self.args.local_rank == -1
   File "D:\Program Files\Python37\lib\site-packages\torch\utils\data\", line 104, in init
     "value, but got num_samples={}".format(self.num_samples))
 ValueError: num_samples should be a positive integer value, but got num_samples=0
  • Not sure what to do next. Going to try restarting and cleaning out the earlier checkpoints as the code runs
  • Another thing that I’m thinking about is the non-narrative nature of the tweets, due to the lack of threading, so I also pulled down the Kagle repository for Trump rally speeches, and am going to see if I can use that. I think that they are particularly interesting because Trump is very attuned to the behavior of the crown during a rally and will “try out” lines to see if they work and adjust what he is talking about. It should reflect what his base is thinking over the time period
  • Need to start thinking about a short presentation for Nov 13


  • Figure out how to taper the beginning and end of the reference frame rotation
  • Add method to adjust the RW contributions. Look at the original spreadsheet and see what the difference is
  • Added the tapering and fooled around a lot exploring how the system is behaving. I think the next step is to see why the vehicle doesn’t recover its pitch