7:30 – 3:30

• Helped John Williams out a bit with generating some unusual tables
• Changed the employees_standard table to reflect the new organization of the data. Had a talk with Mike C. about automatically generating details xml files
• After plugging in my computer, found it running *really* slowly. Deleting my trash (25,025 items) to see if that speeds things up.
• Need to talk to Dong about backticking Ingest Manager – done
• Apparently, I left of 11 employee classification types. Need to fix that tomorrow.
• Adding user management to scripting

7:30 – 3:30

• Fix ServerLog – done. The problem is the changeout of the remote object to the javautils package. Had to generalize a bit.
• Fix PublishServlet – done
• Demo prep (screenshots) – done
• Demo. Went well, I think. One of the lab PMs liked the demonstration enough that he wanted a copy for his lab

7:30 – 4:30

• Prepping for demo
• Got dong’s war file deployed on my machine. That works, rather than building. He’s going to ask Mike to try building and see where the problem lies
• Problems with ie today – it can’t open multiple Flex programs.
• Spent some time getting the projmanager user working in mySql
• better pull for errors: SELECT p.title as project, f.name as funding, o.* FROM obligations_outlays o, fundings f, projects p;
• IngestManager problems:
  • Can’t parse special characters into column names or the values can’t be inserted properly in the column. For example, ‘2/15/60’ won’t read in at all, ‘2-15-60’ won’t have any data in the columns, and ‘2_15_60’ is a date format that Actionscript can’t recognize
  • If the XML SOLR file has CRs in the text file, they are kept, which screws up VISIBILITY. They need to be stripped.

Script to look for decreasing values in outlays:

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Find a month that is less than the previous
def errorsToSolrXml(doc, root, header, table):
    for row in table:
        for i in xrange(len(row)-1):
            if header[i].count("month_") > 0:
                cur = int(float(row[i]))
                next = int(float(row[i+1]))
                #print row[0]+": "+header[i]+" = "+str(cur)
                if(cur > next):
                    #print header[i]+" = "+str(cur)+" > ("+str(next)+")"

                    errorNode = doc.createElement('doc')
                    root.appendChild(errorNode)
                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'type')
                    textNode = doc.createTextNode(str(row[0]))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'cur_month')
                    textNode = doc.createTextNode(header[i])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'next_month')
                    textNode = doc.createTextNode(header[i+1])
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

                    fieldNode = doc.createElement('field')
                    fieldNode.setAttribute('name', 'value')
                    textNode = doc.createTextNode(str(cur - next))
                    fieldNode.appendChild(textNode)
                    errorNode.appendChild(fieldNode)

#################### 'main'

impl = getDOMImplementation()

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

errorDoc = impl.createDocument(None, "add", None)
root = errorDoc.documentElement
errorsToSolrXml(errorDoc, root, newTable[:1][0], newTable[1:])
#print errorDoc.toprettyxml()
jf.publishXml("/obligation_errors.xml", errorDoc.toxml())

7:30 – 5:30

• Working on writing a script that does something with Dong’s data

Inserting data for months – no invoice though

jf = ScriptFacades()

table = jf.dbQuery('project_portfolio', 'SELECT uid from obligations_outlays where project_id=1')
tableData = table.getTable()
print tableData[1:]

for uid in tableData[1:]:
    for month in xrange(1,25):
        monthName = 'month_'+str(month)
        query = 'UPDATE obligations_outlays set '+str(monthName)+' = '+str(random.randint(1000,2000)*month)+' where uid = '+str(uid[0])
        print query
        jf.dbQuery('project_portfolio', query)

Getting data and publishing

# This method converts a DbTable (A list of lists) into an Apache SOLR xml document
def solrXmlFromListList(doc, root, headers, table):
    for row in table:
        rowNode = doc.createElement('doc')
        root.appendChild(rowNode)
        #print row
        for i in xrange(len(headers)):
            #print str(headers[i])+" = "+str(row[i])
            fieldNode = doc.createElement('field')
            fieldNode.setAttribute('name', headers[i])
            textNode = doc.createTextNode(str(row[i]))
            fieldNode.appendChild(textNode)
            rowNode.appendChild(fieldNode)

# Strip off a column based on its name
def stripColumn(headers, table, columnName):
    newRow = []
    for item in headers:
        if item != columnName:
            newRow.append(item)
    newTable = [];
    newTable.append(newRow)
    for row in table:
        newRow = []
        for i in xrange(len(headers)):
            if headers[i] != columnName:
                newRow.append(row[i])
        newTable.append(newRow)
    return newTable

# Change month_xx to a year/date string
def mapDates(row, year):
    for i in xrange(len(row)):
        if row[i].count("month_") > 0:
            splits = row[i].split('_');
            month = (int(splits[1])-1)%12
            addYear = (int(splits[1])-1)/12
            row[i] = str(month+1)+"-1-"+str(year+addYear)
            #print row[i]

#################### 'main'

jf = ScriptFacades()
table = jf.dbQuery('project_portfolio', 'SELECT * from obligations_outlays where project_id=1 and year=2010')
tableData = table.getTable()
newTable = stripColumn(tableData[:1][0], tableData[1:], 'uid')

mapDates(newTable[:1][0], 2010)

impl = getDOMImplementation()
myDoc = impl.createDocument(None, "add", None)
rootElement = myDoc.documentElement
solrXmlFromListList(myDoc, rootElement, newTable[:1][0], newTable[1:])
#print myDoc.toprettyxml()
jf.publishXml("/obligation_outlays_proj_1_year_2010.xml", myDoc.toprettyxml())

7:30 -4:30

• Today, in 1862, the Mexican Army defeated the French at the battle of Puebla. In case you were wondering.
• Richard M is getting Tomcat working better on the LDAT server. We will be ready to continue experimenting in a week or so.
• While waiting for Dong to roll in his changes from yesterday, I’m going to try and fix the case where the JVM holding a jython instance dies from some fatal error. Wound up deleting the ScriptEntity instance from the static map of all script instances and loading a new instance. Seems to work, but we need to test on other servers.
• Got Dong’s data, but the project will not build