8:00 – 4:00 (3 hrs FP)

• Deployed VISIBILITY with sideshows and improved user management
• Deployed Dong’s new SWF
• Sat down with JM and filled up a page with notes. Handed off to Dong
• Got glut bitmap printing working (FP)
• Now I need to set up a logging object . Started. (FP)

8:00 – 5:00

• Added a PMP role under S2A
• Added PMP folks to VISIBILITY
• Left a note with JM on progress
• Testing role update
• Also looking at new web hosts, since hostjava.net kinda sucks:
  • viz-n.com (hostjava.net) baseline:
    • ping (min/avg/max/med) – 156/327/442/348
    • 6.7% packet loss
    • tracert hops: 13 hops
  • feale.com (steadfast.net)
    • ping (min/avg/max/med) – 62/176/315/162
    • 11.8% packet loss
    • tracert hops: 9 hops
  • tanori.net (mediatemple)
    • ping (min/avg/max/med) – 84/256/406/260
    • 13.3% packet loss
    • tracert hops: 16 hops
  • philfeldman.com (dreamhost)
    • ping (min/avg/max/med) – 77/171/284/170
    • 15.4% packet loss
    • tracert hops: 21 hops
• A lot of this delay seems to be coming from us. When I ping these sites from a dreamhost server, I get much better numbers. There is no packet loss at all, and the times are better (in the case of mediatemple, they’re phenomenal)
  • tanori.net ping (min/avg/max/med) – 1.8/2.0/2.1/0.09
  • feale.com ping (min/avg/max/med) – 78/78/78/0.078
  • viz-n.com ping (min/avg/max/med) – 87/88/88/0.268
• Struggled with the role code for a while, until I realized that the actionscript object can come across to java with the children Lists being null. All seems to be working.
• Burned a disk of VIS2 and PPM.

7:30 – 3:30

• Deployed new documentation in a test URL for JW to look at
• Walked through the new S14 app with JM. Lots of notes – I called Dong and passed them along.
• Meeting with Anne and a large group from S2A(?). Made the pitch that they pay for the development of a deployable version of VISIBILITY, which they liked
• Need to put in the role editing that I worked out before the move. Done. Need to test

7:00 – 4:30

• Deployed ProjPortfolioMgr. It works! Hosanna!
• Spent 40 min waiting for Jim M to show up
• Meeting with Larry about commercialization. I think we’ll try to package up the charting part of Visibility and use it as the wedge for services. Also, Ed(?) is looking at using Media Temple, so we might be able to piggy back on that effort
• Bike seminar
• And it took 2 hours to get back here…

7:30 – 5:30

• Deployed the new Scripting, which pointed out that I was trying to update the wrong table. Once fixed, the scripts worked fine, so I put them on a schedule to run every day at 7:00 and 7:05
• Couldn’t connect to the S2F endpoint. Somehow, the url is getting changed from https:// (which it is in the settings.xml file) to http://. I printed out relevant stuff and showed to Dong.
• Talked to Jim M
• Meeting/demo with Anne is now set for Monday at 10:30
• useful!
  • find . -name *.mxml -mtime -15 -type f -exec wc -l {} ; | awk ‘{total += $1} END{print total}’
  • This is “find all the mxml files under the current directory that are less than 15 days old. Run a line count on the files and sum them.
• We seem to have fixed the problems with ProjPortfolioMgr. Burning a disk

7:30 – 4:30

• Meeting with Jim M. Came up with a more realistic plan for hitting their goals
• Showed Anne how to ingest data
• Deployed new Vis2 and Scripting
• Tried my new insert. It didn’t work. The reason was that the column name was wrong. Fixed the logging so that the error shows
• Got ProjPortfolioMgr to deploy on S2f. We’ll also make that the first attempt at an installer

9:30 – 4:00 5:00 – 6:00

• Dentist this morning
• Back to Facade insert – done and working!
• Dong added backticking to Ingest Manager
• Built some new sensor rods (paddles?) for Tom

8:00 – 5:00

• Worked with Dong a bit on getting the DB configuration app
• Putting together a precompiled query interface – getting hung up with a generic date parser. Maybe this: http://icecube.wisc.edu/~dglo/software/calparse/index.html
• Dentist tomorrow. I’ll be in a bit late.

7:30 – 4:30

• “Deployed” new details XML for standard and extended employees.
• Need to be able to do a precompiled query through facades
• Unpacked the Feldman Project
• Tom’s building a new set of sensors
• Setting up the testbed next to my desk. Going to set up a clean(er?) code framework
• Installed Phidgets 32 bit drivers from here: http://www.phidgets.com/drivers.php
• Installer instructions: http://www.phidgets.com/documentation/Tutorials/Library_Information.pdf
• Freeglut precompiled binaries: http://www.transmissionzero.co.uk/software/freeglut-devel/
• Freeglut source: http://freeglut.sourceforge.net/index.php#download
• Got OpenGL working nicely
• Can talk to the motors. Going to fix control next

7:30 – 4:30

• Fixed the category problem in employees_std. Talked to Mike C about developing an automated way to do this. Will start doing a daily manual set of the details xml. Later, I’ll do this in a script
• Adding user management to the scripting server – done and checked in.
• Asked Dong to use javaUtils rather than recreate log4j and properties classes

7:30 – 3:30

• Helped John Williams out a bit with generating some unusual tables
• Changed the employees_standard table to reflect the new organization of the data. Had a talk with Mike C. about automatically generating details xml files
• After plugging in my computer, found it running *really* slowly. Deleting my trash (25,025 items) to see if that speeds things up.
• Need to talk to Dong about backticking Ingest Manager – done
• Apparently, I left of 11 employee classification types. Need to fix that tomorrow.
• Adding user management to scripting

7:30 – 3:30

• Fix ServerLog – done. The problem is the changeout of the remote object to the javautils package. Had to generalize a bit.
• Fix PublishServlet – done
• Demo prep (screenshots) – done
• Demo. Went well, I think. One of the lab PMs liked the demonstration enough that he wanted a copy for his lab

7:30 – 4:30

• Prepping for demo
• Got dong’s war file deployed on my machine. That works, rather than building. He’s going to ask Mike to try building and see where the problem lies
• Problems with ie today – it can’t open multiple Flex programs.
• Spent some time getting the projmanager user working in mySql
• better pull for errors: SELECT p.title as project, f.name as funding, o.* FROM obligations_outlays o, fundings f, projects p;
• IngestManager problems:
  • Can’t parse special characters into column names or the values can’t be inserted properly in the column. For example, ‘2/15/60’ won’t read in at all, ‘2-15-60’ won’t have any data in the columns, and ‘2_15_60’ is a date format that Actionscript can’t recognize
  • If the XML SOLR file has CRs in the text file, they are kept, which screws up VISIBILITY. They need to be stripped.

Script to look for decreasing values in outlays:

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Find a month that is less than the previous
def errorsToSolrXml(doc, root, header, table):
    for row in table:
        for i in xrange(len(row)-1):
            if header[i].count("month_") > 0:
                cur = int(float(row[i]))
                next = int(float(row[i+1]))
                #print row[0]+": "+header[i]+" = "+str(cur)
                if(cur > next):
                    #print header[i]+" = "+str(cur)+" > ("+str(next)+")"

                    errorNode = doc.createElement('doc')
                    root.appendChild(errorNode)
                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'type')
                    textNode = doc.createTextNode(str(row[0]))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'cur_month')
                    textNode = doc.createTextNode(header[i])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'next_month')
                    textNode = doc.createTextNode(header[i+1])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'value')
                    textNode = doc.createTextNode(str(cur - next))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

#################### 'main'

impl = getDOMImplementation()

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

errorDoc = impl.createDocument(None, "add", None)
root = errorDoc.documentElement
errorsToSolrXml(errorDoc, root, newTable[:1][0], newTable[1:])
#print errorDoc.toprettyxml()
jf.publishXml("/obligation_errors.xml", errorDoc.toxml())

7:30 – 5:30

• Working on writing a script that does something with Dong’s data

Inserting data for months – no invoice though

jf = ScriptFacades()

table = jf.dbQuery('project_portfolio', 'SELECT uid from obligations_outlays where project_id=1')
tableData = table.getTable()
print tableData[1:]

for uid in tableData[1:]:
    for month in xrange(1,25):
        monthName = 'month_'+str(month)
        query = 'UPDATE obligations_outlays set '+str(monthName)+' = '+str(random.randint(1000,2000)*month)+' where uid = '+str(uid[0])
        print query
        jf.dbQuery('project_portfolio', query)

Getting data and publishing

# This method converts a DbTable (A list of lists) into an Apache SOLR xml document
def solrXmlFromListList(doc, root, headers, table):
    for row in table:
        rowNode = doc.createElement('doc')
        root.appendChild(rowNode)
        #print row
        for i in xrange(len(headers)):
            #print str(headers[i])+" = "+str(row[i])
            fieldNode = doc.createElement('field')
            fieldNode.setAttribute('name', headers[i])
            textNode = doc.createTextNode(str(row[i]))
            fieldNode.appendChild(textNode)
            rowNode.appendChild(fieldNode)

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Change month_xx to a year/date string
def mapDates(row, year):
    for i in xrange(len(row)):
        if row[i].count("month_") > 0:
            splits = row[i].split('_');
            month = (int(splits[1])-1)%12
            addYear = (int(splits[1])-1)/12
            row[i] = str(month+1)+"-1-"+str(year+addYear)
            #print row[i]

#################### 'main'

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

mapDates(newTable[:1][0], 2010)

impl = getDOMImplementation()
myDoc = impl.createDocument(None, "add", None)
rootElement = myDoc.documentElement
solrXmlFromListList(myDoc, rootElement, newTable[:1][0], newTable[1:])
#print myDoc.toprettyxml()
jf.publishXml("/obligation_outlays_proj_1_year_2010.xml", myDoc.toprettyxml())

7:30 -4:30

• Today, in 1862, the Mexican Army defeated the French at the battle of Puebla. In case you were wondering.
• Richard M is getting Tomcat working better on the LDAT server. We will be ready to continue experimenting in a week or so.
• While waiting for Dong to roll in his changes from yesterday, I’m going to try and fix the case where the JVM holding a jython instance dies from some fatal error. Wound up deleting the ScriptEntity instance from the static map of all script instances and loading a new instance. Seems to work, but we need to test on other servers.
• Got Dong’s data, but the project will not build