Phil 11.28.18

7:00 – 4:00 ASRC PhD

    • Made so much progress yesterday that I’m not sure what to do next. Going to see if I can run queries against the DB in Python for a start, and then look at the Stanford tools.
      • installed pymysql (in lowercase. There is also a CamelCase version PyMySQL, that seems to be the same thing…)
      • Piece of cake! Here’s the test code: 
        import pymysql

class forum_reader:
    connection: pymysql.connections.Connection

    def __init__(self, user_name: str, user_password: str, db_name: str):
        print("initializing")
        self.connection = pymysql.connect(host='localhost', user=user_name, password=user_password, db=db_name)

    def read_data(self, sql_str: str) -> str:
        with self.connection.cursor() as cursor:
            cursor.execute(sql_str)
            result = cursor.fetchall()
            return"{}".format(result)

    def close(self):
        self.connection.close()
if __name__ == '__main__':
    fr = forum_reader("some_user", "some_pswd", "some_db")
    print(fr.read_data("select topic_id, forum_id, topic_title from phpbb_topics"))
      • And here’s the result: 
        initializing
((4, 14, 'SUBJECT: 3 Room Linear Dungeon Test 1'),)
      • Note that this is not an object db, which I prefer, but since this is a pre-existing schema, that’s what I’ll be doing. Going to look for a way to turn a query into an object anyway. But it turns out that you can do this: 
        self.connection = pymysql.connect(
    host='localhost', user=user_name, password=user_password, db=db_name,
    cursorclass=pymysql.cursors.DictCursor)
      • Which returns as an array of JSON objects: 
        [{'topic_id': 4, 'forum_id': 14, 'topic_title': 'SUBJECT: 3 Room Linear Dungeon Test 1'}]
    • Built a MySQL view to get all the data back in one shot: 
      CREATE or REPLACE VIEW post_view AS
SELECT p.post_id, FROM_UNIXTIME(p.post_time) as post_time, p.topic_id, t.topic_title, t.forum_id, f.forum_name, u.username, p.poster_ip, p.post_subject, p.post_text
  FROM phpbb_posts p
  INNER JOIN phpbb.phpbb_forums f ON p.forum_id=f.forum_id
  INNER JOIN phpbb.phpbb_topics t ON p.topic_id=t.topic_id
  INNER JOIN phpbb.phpbb_users u ON p.poster_id=u.user_id;
    • And that works like a charm in the Python code: 
      [{
	'post_id': 4,
	'post_time': datetime.datetime(2018, 11, 27, 16, 0, 27),
	'topic_id': 4,
	'topic_title': 'SUBJECT: 3 Room Linear Dungeon Test 1',
	'forum_id': 14,
	'forum_name': 'DB Test',
	'username': 'dungeon_master1',
	'poster_ip': '71.244.249.217',
	'post_subject': 'SUBJECT: 3 Room Linear Dungeon Test 1',
	'post_text': 'POST: dungeon_master1 says that you are about to take on a 3-room linear dungeon.'
}]

       

  • Tricia Wang thick data <- add some discussion about this with respect to gathering RPG data
  • Spend some time Grokking as well. Need to nail down backpropagation. Not today
  • Long discussions with Aaron about the structure of TimeSeriesML. Including looking at FFTs for the initial analytics.
  • A2P/AIMS meeting
    • Terrabytes of AIMS data?

Progress for today 🙂 ide