Phil 11.8.19

7:00 – 3:00 ASRC GOES

  • Dissertation
    • Usability study! Done!
    • Discussion. This is going to take some framing. I want to tie it back to earlier navigation, particularly the transition from stories and mappaemundi to isotropic maps of Ptolemy and Mercator.
  • Sent Don and Danilo sql file
  • Start satellite component list
  • Evolver
    • Adding threads to handle the GPU. This looks like what I want (from here):
      import logging
      import concurrent.futures
      import threading
      import time
      
      def thread_function(name):
          logging.info("Task %s: starting on thread %s", name, threading.current_thread().name)
          time.sleep(2)
          logging.info("Task %s: finishing on thread %s", name, threading.current_thread().name)
      
      if __name__ == "__main__":
          num_tasks = 5
          num_gpus = 1
          format = "%(asctime)s: %(message)s"
          logging.basicConfig(format=format, level=logging.INFO,
                              datefmt="%H:%M:%S")
      
          with concurrent.futures.ThreadPoolExecutor(max_workers=num_gpus) as executor:
              result = executor.map(thread_function, range(num_tasks))
      
          logging.info("Main    : all done")

      As you can see, it’s possible to have a thread for each gpu, while having them iterate over a larger set of tasks. Now I need to extract the gpu name from the thread info. In other words,  ThreadPoolExecutor-0_0 needs to map to gpu:1.

    • Ok, this seems to do everything I need, with less cruft:
      import concurrent.futures
      import threading
      import time
      from typing import List
      import re
      
      last_num_in_str_re = '(\d+)(?!.*\d)'
      prog = re.compile(last_num_in_str_re)
      
      def thread_function(args:List):
          num = prog.search(threading.current_thread().name) # get the last number in a string
          gpu_str = "gpu:{}".format(int(num.group(0))+1)
          print("{}: starting on  {}".format(args["name"], gpu_str))
          time.sleep(2)
          print("{}: finishing on  {}".format(args["name"], gpu_str))
      
      if __name__ == "__main__":
          num_tasks = 5
          num_gpus = 5
          task_list = []
          for i in range(num_tasks):
              task = {"name":"task_{}".format(i), "value":2+(i/10)}
              task_list.append(task)
          with concurrent.futures.ThreadPoolExecutor(max_workers=num_gpus) as executor:
              result = executor.map(thread_function, task_list)
      
          print("Finished Main")

      And that gives me:

      task_0: starting on  gpu:1
      task_1: starting on  gpu:2
      task_0: finishing on  gpu:1, after sleeping 2.0 seconds
      task_2: starting on  gpu:1
      task_1: finishing on  gpu:2, after sleeping 2.1 seconds
      task_3: starting on  gpu:2
      task_2: finishing on  gpu:1, after sleeping 2.2 seconds
      task_4: starting on  gpu:1
      task_3: finishing on  gpu:2, after sleeping 2.3 seconds
      task_4: finishing on  gpu:1, after sleeping 2.4 seconds
      Finished Main

      So the only think left is to integrate this into TimeSeriesMl2

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.