Phil 3.29.17

7:00 – 8:30 Research

  • Starting to think seriously about the Research Browser. There are three parts that I want to leverage against each other
    1. I need to get good at Python
    2. I need a level of user-centered design that shows the interaction between analog controls and information. This is hard to do with words, paper design, or Wizard of Oz, so I want to have at least some GUI wrapper
    3. I want to start working with my school server, and get enough up that there is an application that connects to the back end and that has enough of a web wrapper that I can point to that in papers, business cards, etc.
  • If it’s going to be a thick client, then there needs to be a way of building an executable that does not require a Python install. And there is:
    • PyInstaller is a program that freezes (packages) Python programs into stand-alone executables, under Windows, Linux, Mac OS X, FreeBSD, Solaris and AIX. Its main advantages over similar tools are that PyInstaller works with Python 2.7 and 3.3—3.5, it builds smaller executables thanks to transparent compression, it is fully multi-platform, and use the OS support to load the dynamic libraries, thus ensuring full compatibility.
  • Now we have the option of thick or thin client. For thick client we have
    • Kivy – Open source Python library for rapid development of applications
      that make use of innovative user interfaces, such as multi-touch apps
    • PyQT, which is a set of Python wrappers for QT, which is huge UI. Here’s a whitepaper discussing the integration. Seems powerful, but maybe a lot of moving parts
    • PyGUI
      • Develop a GUI API that is designed specifically for Python, taking advantage of Python’s unique language features and working smoothly with Python’s data types.
      • Provide implementations of the API for the three major platforms (Unix, Macintosh and Windows) that are small and lightweight, interposing as little code as possible between the Python application and the platform’s underlying GUI facilities, and not bloating the Python installations or applications which use them.
      • Document the API purely in Python terms, so that the programmer does not need to read the documentation for another GUI library, in terms of another language, and translate into Python.
      • Get the library and its documentation included in the core Python distribution, so that truly cross-platform GUI applications may be written that will run on any Python installation, anywhere.
  • Thin Client:
    • Flexx is a pure Python toolkit for creating graphical user interfaces (GUI’s), that uses web technology for its rendering. Apps are written purely in Python; Flexx’ transpiler generates the necessary JavaScript on the fly.
    • Reahl lets you build a web application purely in Python, and in terms of useful objects that shield you from low-level web implementation issues.
    • Remi is a GUI library for Python applications which transpiles an application’s interface into HTML to be rendered in a web browser. This removes platform-specific dependencies and lets you easily develop cross-platform applications in Python!
      • screenshot
    • WDOM: GUI library for browser-based desktop applications
    • pyjs is a Rich Internet Application (RIA) Development Platform for both Web and Desktop. With pyjs you can write your JavaScript-powered web applications entirely in Python. pyjs contains a Python-to-JavaScript compiler, an AJAX framework and a Widget Set API. pyjs started life as a Python port of Google Web Toolkit, the Java-to-JavaScript compiler.
    • There are more frameworks here. There is some overlap with the above, but the list seems to include more obscure systems. The CEFBrowser embedded browser is pretty interesting and is a needed piece. This note indicates that calling the QT QWebEngineView class from Python (PyQT) is a known pattern.

9:00 – 5:00 BRC

  • Continuing to try and figure out how to assemble and write out a pandas.Dataframe as TFReccords. Now looking at this post
  • Success! Here’s the slow way:
    def slow_write_pbf(self, file_name: str) -> bool:
        df = self._data_frame
        row_label_array = np.array(df.index.values)
        col_label_array = np.array(df.columns.values)
        value_array = df.as_matrix()
        # print(row_label_array)
        # print(col_label_array)
        # print(value_array)
        writer = tf.python_io.TFRecordWriter(file_name)
        rows = value_array.shape[0]
        cols = value_array.shape[1]
        for row in range(rows):
            for col in range(cols):
                val = value_array[row, col]
                unit = {
                    'row_name': self.bytes_feature(str.encode(row_label_array[row])),
                    'col_name': self.bytes_feature(str.encode(col_label_array[col])),
                    'rows': self.int64_feature(rows),
                    'cols': self.int64_feature(cols),
                    'val': self.float_feature(val)
                cur_feature = tf.train.Features(feature=unit)
                example = tf.train.Example(features=cur_feature)
  • Here’s the fast way:
    def write_pbf(self, file_name: str) -> bool:
        df = self._data_frame
        df_str = df.to_csv()
        df_enc = str.encode(df_str)
        writer = tf.python_io.TFRecordWriter(file_name)
        unit = {
            'DataFrame': self.bytes_feature(df_enc)
        cur_feature = tf.train.Features(feature=unit)
        example = tf.train.Example(features=cur_feature)
  • Reading in data.
    def read_pbf(self, file_name: str) -> list[pandas.DataFrame]:
        features = {'DataFrame': tf.FixedLenFeature([1], tf.string)}
        data = []
        for s_example in tf.python_io.tf_record_iterator(file_name):
            example = tf.parse_single_example(s_example, features=features)
            data.append(tf.expand_dims(example['DataFrame'], 0))
        result = tf.concat(data,0)
        return result
  • Parsing the data requires running a TF Session
    # call the function that gets the data from the file
    df_graph_list = pbr.read_pbf(
    # start up a tf.Session to get the data from the graph and parse
    with tf.Session() as sess: #set up a TF Session and initialize
        df_list =[df_graph_list]) # get the data
        for df_element in df_list:
            ptr = df_element[0][0] # dereference. TODO: Understand all the levels
            dfPtr = StringIO(bytes.decode(ptr)) # make a pointer to the string so it can be treated as a file
            df = pandas.read_csv(dfPtr, index_col=0) # create the dataframe. It will have 'float' datatype which imshow chokes on
            mat = df.as_matrix() # get the data matrix
            mat = mat.astype(np.float64) # force it to float64
            df.update(mat) # replace the 'float' mat with the 'float64' mat
            if df.shape[0] < 20:
                if == True: