Phil 3.30.17

7:00 – 8:00, 4:00 – 6:00

  • Looking more closely at Qt and PyQt. First, the integrated browser is nice. Second, if it’s possible to wireframe UIs in Qt and connect them to Python for matrix calculations and server interactions, then I have some real leverage.
  • Really good overview. Difference between 4 and 5, etc.
  • Lotsa python and machine learning videos. These look promising
  • Meeting with Don

8:30 – 3:30 BRC

  • Cleaning up reading code and adding argument handling
  • Need to add a row-reader to replace the slow_read_pbf and the slow_write_pbf methods. They also need to turn the row into a JSON object, and to have a separate method that will assemble a DataFrame from a set of JSON objects in memory
  • Nope, strike that. We’re now going to read CSV files containing sparse data and assemble them into a DataFrame. The format is:
    "member:123456",CNR:3,CPD:9,TAS:1,AFB:38,FUD:1
    "member:789012",PRK:1,TAS:2
  • The row id is quoted, otherwise key:value pairs are comma separated. Rows are terminated by CR/LF, and there can be multiple rows per file.
  • Python hadoop: Pydoop