7:30 – 4:30

• Fixed the category problem in employees_std. Talked to Mike C about developing an automated way to do this. Will start doing a daily manual set of the details xml. Later, I’ll do this in a script
• Adding user management to the scripting server – done and checked in.
• Asked Dong to use javaUtils rather than recreate log4j and properties classes

7:30 – 3:30

• Helped John Williams out a bit with generating some unusual tables
• Changed the employees_standard table to reflect the new organization of the data. Had a talk with Mike C. about automatically generating details xml files
• After plugging in my computer, found it running *really* slowly. Deleting my trash (25,025 items) to see if that speeds things up.
• Need to talk to Dong about backticking Ingest Manager – done
• Apparently, I left of 11 employee classification types. Need to fix that tomorrow.
• Adding user management to scripting

7:30 – 3:30

• Fix ServerLog – done. The problem is the changeout of the remote object to the javautils package. Had to generalize a bit.
• Fix PublishServlet – done
• Demo prep (screenshots) – done
• Demo. Went well, I think. One of the lab PMs liked the demonstration enough that he wanted a copy for his lab

7:30 – 4:30

• Prepping for demo
• Got dong’s war file deployed on my machine. That works, rather than building. He’s going to ask Mike to try building and see where the problem lies
• Problems with ie today – it can’t open multiple Flex programs.
• Spent some time getting the projmanager user working in mySql
• better pull for errors: SELECT p.title as project, f.name as funding, o.* FROM obligations_outlays o, fundings f, projects p;
• IngestManager problems:
  • Can’t parse special characters into column names or the values can’t be inserted properly in the column. For example, ‘2/15/60’ won’t read in at all, ‘2-15-60’ won’t have any data in the columns, and ‘2_15_60’ is a date format that Actionscript can’t recognize
  • If the XML SOLR file has CRs in the text file, they are kept, which screws up VISIBILITY. They need to be stripped.

Script to look for decreasing values in outlays:

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Find a month that is less than the previous
def errorsToSolrXml(doc, root, header, table):
    for row in table:
        for i in xrange(len(row)-1):
            if header[i].count("month_") > 0:
                cur = int(float(row[i]))
                next = int(float(row[i+1]))
                #print row[0]+": "+header[i]+" = "+str(cur)
                if(cur > next):
                    #print header[i]+" = "+str(cur)+" > ("+str(next)+")"

                    errorNode = doc.createElement('doc')
                    root.appendChild(errorNode)
                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'type')
                    textNode = doc.createTextNode(str(row[0]))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'cur_month')
                    textNode = doc.createTextNode(header[i])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'next_month')
                    textNode = doc.createTextNode(header[i+1])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'value')
                    textNode = doc.createTextNode(str(cur - next))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

#################### 'main'

impl = getDOMImplementation()

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

errorDoc = impl.createDocument(None, "add", None)
root = errorDoc.documentElement
errorsToSolrXml(errorDoc, root, newTable[:1][0], newTable[1:])
#print errorDoc.toprettyxml()
jf.publishXml("/obligation_errors.xml", errorDoc.toxml())

7:30 – 5:30

• Working on writing a script that does something with Dong’s data

Inserting data for months – no invoice though

jf = ScriptFacades()

table = jf.dbQuery('project_portfolio', 'SELECT uid from obligations_outlays where project_id=1')
tableData = table.getTable()
print tableData[1:]

for uid in tableData[1:]:
    for month in xrange(1,25):
        monthName = 'month_'+str(month)
        query = 'UPDATE obligations_outlays set '+str(monthName)+' = '+str(random.randint(1000,2000)*month)+' where uid = '+str(uid[0])
        print query
        jf.dbQuery('project_portfolio', query)

Getting data and publishing

# This method converts a DbTable (A list of lists) into an Apache SOLR xml document
def solrXmlFromListList(doc, root, headers, table):
    for row in table:
        rowNode = doc.createElement('doc')
        root.appendChild(rowNode)
        #print row
        for i in xrange(len(headers)):
            #print str(headers[i])+" = "+str(row[i])
            fieldNode = doc.createElement('field')
            fieldNode.setAttribute('name', headers[i])
            textNode = doc.createTextNode(str(row[i]))
            fieldNode.appendChild(textNode)
            rowNode.appendChild(fieldNode)

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Change month_xx to a year/date string
def mapDates(row, year):
    for i in xrange(len(row)):
        if row[i].count("month_") > 0:
            splits = row[i].split('_');
            month = (int(splits[1])-1)%12
            addYear = (int(splits[1])-1)/12
            row[i] = str(month+1)+"-1-"+str(year+addYear)
            #print row[i]

#################### 'main'

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

mapDates(newTable[:1][0], 2010)

impl = getDOMImplementation()
myDoc = impl.createDocument(None, "add", None)
rootElement = myDoc.documentElement
solrXmlFromListList(myDoc, rootElement, newTable[:1][0], newTable[1:])
#print myDoc.toprettyxml()
jf.publishXml("/obligation_outlays_proj_1_year_2010.xml", myDoc.toprettyxml())

7:30 -4:30

• Today, in 1862, the Mexican Army defeated the French at the battle of Puebla. In case you were wondering.
• Richard M is getting Tomcat working better on the LDAT server. We will be ready to continue experimenting in a week or so.
• While waiting for Dong to roll in his changes from yesterday, I’m going to try and fix the case where the JVM holding a jython instance dies from some fatal error. Wound up deleting the ScriptEntity instance from the static map of all script instances and loading a new instance. Seems to work, but we need to test on other servers.
• Got Dong’s data, but the project will not build

7:30 – 3:30

• Some thoughts on Mike’s slideshows
  1. Make sure that the changes/additions to the db happen automatically when the class is first addressed (CREATE TABLE IF NOT EXIST foo()) doesn’t seem to take much overhead. That way, deployment will be easier.
  2. I was thinking that the slide show controls would be another Utils panel that could be hidden as needed.
  3. Selectable transitions (fade, slide vertical, slade horizontal, etc) would be a plus
  4. Maybe the slide controls can live in the top part of the display and be hidden unless needed?
• Brought in EdgeUtils for Clift
• Meeting with Jim M, Trish ? and Dong. Lots of changes, as can be seen is his log entry.

7:30 – 4:30

• Svn is working, but the check in is not working in Eclipse. Attempting to update svn plugin, but things are hanging. May have to say goodby to svn integration. Dong has Subclipse working. Will switch to that later. Tortoise for now
• Call Richard Martin – 443.479.8721
• Got Visibility and VisibilityScripting checked out and running

7:30 – 3:30

• More financial disclosure
• Teleconference with Juan Lopez – 937-255-6565 ext. 4637  at 10:45
• Getting close with Maven – It looks like we have a RAM problem. We need 64 and we have 32. Dong is asking for more

7:30 – 4:00

• Worked on Financial Disclosure
• Organized clutter in VISIBILITY tables
• Waiting for Dong to get the maven repo set up. Without that, we can’t compile, because our jars and swcs come from the repo.
• Meeting at the fort for lunch
• Got the algorithm for role tree updating working. Once maven and subversion are all working, I’ll roll it in and test.

7:30 -3:30

• Tried to move example tables to example_xxx and demo tables to demo_xxx. It looks like I need to some other table manipulation other than just changing their names
• Long meeting with Michelle Griffith of LDAT. Some good progress. She’ll be coming out sometime next week.
• Still moving in and setting up
• Sent Tom some example files so he can get started on the new documentation effort

7:30 -4:00

• Set up some new LDAT accounts
• Got Brian’s data
• Move! – Done, although setting back up has not included the lab
• Starting on RoleTreeUpdate.

7:30 – 3:30

• Set up some new accounts and read in some data. The LDAT folks seem to be liking what we can do.
• Got an idea about adding roles. When a role tree is updated (setRoleTreeRoot), it resets all user roles because we don’t know what the ids are going to be. Instead, make it so that the new role tree is compered against the stored role tree (getRoleRoot) and only changes are added.
• Demo of VISIBILITY with Anne.

7:30 – 3:30

• Made screenshots for Brian
• Got python code to add a column to a table. Pretty easy, actually
• Worked with Dong to debug
• Started to set up viz-n.com