Monthly Archives: September 2009

Phil 9.30.2009

7:30 – 3:30

  • Deployed the new version – there are a few bugs:
    • Although remote files validate, they don’t read in – looks like the problem was that the remote files don’t terminate in .xml
    • Blanks are counted in the UniqueCounts view – fixed
    • Need to clear out the “where clauses” if the last item is unchecked – fixed
    • Need to keep the where clauses in the unique count query construction – fixed
  • Had a good discussion with Mike C about data
  • Scheduled a meeting with Anne to go over the new features
  • Burned a new disc and am ready to deploy tomorrow.

Dong Shin Visibility Status – Sep '09

Week of 09/28

  • IE chokes when URL (endpoint) is not set up with full path; ***.***.com or IP address
  • Fixed tables returning from server not role/user based
  • Added publish status, changed mode to Overwrite to IngestConfirm window
  • Changed date in the table names to return YYYY-MM-DD format
  • Bug found in AutoIngestor
    • found that files with no ext (.xml, .xls, etc.) fails to return columns
    • input validation added to RemoteFileWindow and RoleSelectinCanvas
  • Added dataTips to AutoIngest DataGrid
  • Rearranged New and Remote File buttons in AutoIngestConfig
  • Extending File class to HTTPFile to use remote files
  • Adding HTTPS and HTTP capabilities to other types of files
  • All changes uploaded and downloaded.

Week of 09/21

  • Administrators can now delete any tables from the IngestManager
    • isUserAdmin() methods added to both client and server
  • Changed displaying ID of the user to Names of the Users in IngestPanel and IngestedDataConfiguration
    • Database Query to join TABLE_USERS and TABLE_DATA tables
    • StyleFunction for the AdvancedDataGrid
  • AutoIngest now checks for existing data (file name and table name) on the client
  • Remote File Ingest to handle http and https
    • Server – and under com.edgeti.utils
    • Flex – added Remote File button in IngestVBox, RemoteFileWindow.mxml to handle remote file ingest
    • remote file added to Files Available after validation
    • change handleFileSelected() in IngestVBox to handle http and https and related events/services
    • columns returned from http and https
    • removed old remote file routines from AutoIngestConfigVBox and put RemoteFileWindow to use http and https autoingest…..
    • changed server codes to use full filename for remote (http) and uploadDir for local files.
    • ingest works!!!! seems nothing is broken….

Week of 09/14

  • disabled Use Remote location for Auto Ingest for now….
  • fixed auto ingest to run specified time, hourly, daily, weekly.
  • creating RoleColumns/RoleTree component so it can be used by IngestVBox and AutoIngest.

Week of 09/07

  • Resolved most of the bugs/fixes/enhancements Phil asked.
  • Reworking Ingestor, AutoIngest Config, and Ingested Data Config to use Roles and Role Columns. This requires changes to the Servlet, too.
  • Ingestor problem resolved
    • moved connect/disconnect for each insert SQL to east ingest. It seems ingestion is a lot faster now. XLS ingestor should be changed
  • Continued troubleshooting ingestor problem
    • Ingestor running too fast?
    • file too large?
  • Remote File ingestion
    • running Tomcat under SYSTEM prevents the Ingestor to read remote files
    • running different user (me) works fine.

Week of 08/31

  • Test Deploy to EdgeDemo – successful
    • change endpoint in assets/setupModel.xml
    • MakeFakeData in ManagementUtilsTest.html fails. It seems there is a problem in populateFakeUsers in
  • Having problems with ingestor. MySQL reports socket not available error
  • Continue looking into remote file ingesttion

Phil 9.28.2009

7:30 – 3:30

  • Gotta install sp2…. Well that was 20 minutes well spent! Wheee.
  • Working on multiple select = done. Leaving the single select code in place for a while out of, well, superstition and fear, actually.
  • I’m thinking about adding another chart that has the unique counts for each column – that way it would be possible to easily find out how many people we in a query, even if they had multiple skills. – done

Mike 9.27.09

  • Search now automatically scrolls to the selected item in the datagrid if it finds one
  • Saving / loading now once again works with the AdvQueryWidget
  • DynamicPanel may now go directly from minimized to maximized and vice versa
  • Updated all Layout Managers to handle the new panel feature

Phil 9.25.2009

7:30 – 3:30

  • Another meeting with John, Anne, Brian, and Mike this time
  • Based on getting new data (roster2) John is much happier with the tool. Mind you, there have been no changes to the tool.
  • Working on getting an “insert column” capability, *then* we might get to multiple selects – done

Phil 9.24.2009

7:30 – 4:00

  • Long and painful meeting with John and Anne. My notes follow:
    • Data names are not clear from source
    • Should be able to check a box and get a list of names
    • Eliminate redundant columns
    • Fix column names
    • Either the basic tree or the question tree needs to have some checkboxes that represent “where clauses” that would be OR-ed together
    • Collapse redundant entries in columns on Flex side – i.e. how many unique names there are. This may be a new chart view
  • Working on Question interface – first pass is done. Rolling up to bring in.
  • Tomorrow, work on AdvancedQueryWidget.onCreateChartClicked() so that multiple bars can be selected.

Phil 9.23.2001

7:30 – 5:00

  • Ingested data at the fort. Talked to John, who tried to find info with Pat yesterday. I said that the display is only as good as the data, but I don’t think it stuck.
  • Working on “Questions” interface
    • Modify – done
    • Add – done
    • Add as Child – done
    • Delete – done – turned out to be tricky. You have to use the delete operator.
    • Have “Set Query Builder” populate the _questionQuery.text field
    • Add a setTableQuestions(String questions,  String tableName) method in DbRemoteObject that a) Creates a table_questions table if there is none, and b) loads the xml as a blob in the table row (note: only one set of questions per table) See the dashboard entry for how to handle xml – done
    • Add a getTableQuestions(String tableName) method in DbRemoteObject that a) Creates a table_questions table if there is none, and b) either returns the xml if it exists or returns and empty <questions/> xml string. – done

Phil 9.22.2009

7:30 – 3:30

  • Deployed the new version with Mike’s fixes. Everything seems to be working fine
  • Talked to Anne about building special-purpose tables from the master table to make it easier to explore the data
  • Floated the question idea
  • Realized that we could also have all the available strings in a column listed in the (a?) hierachy view, so some level of drilldown could be available…
  • Changed all query double quotes to single quotes
  • Phone call from Anne – Meeting with here and John on Thursday. Might have a version by then.
  • Started on the question tab, mostly because it’s an easier problem to solve that might give us what we want. Go the part where you can click on a tree component “question” that runs a query.
    • Need to be able to interactively build the xml
    • Need to save the xml in a table-specific entry (only one entry per table holding the xml)

Phil 9.21.2009

7:30 – 4:00

  • Hey! It’s Fall! No wonder it’s dark when I get up these days
  • So I come in this morning and find that Mike has fixed half the issues that Anne and John listed. It’s like having pixies do the dishes at night!
  • VIS2 Meeting at 10:30
  • Writing up support for S2Dash
  • Need to make the system more Pat-freindly. Not sure what that means, but I’m leaning towards “Questions” that point to queries

Mike 9/20/09

  • Allow users to delete their dashboards (from the save menu)
  • Start up in ‘Builder’ layout
  • If nothing is selected, have a popup that says “No items selected for query” (done in adv and basic query widget, querybuilder defaults to select * so I didn’t change that)
  • Don’t allow components (particularly minimized windows) to go behind the top “VISIBILITY2” UIComponent
  • Resize events on the builder window should re-trigger the layout (Only occurs when in builder layout)

Phil 9.18.2009

7:30 – 5:30

  • Deployed the latest attempt at the fort and it seems to be working well. Uploaded new data
  • Trained up Anne and John on the new features. Based on their suggestions, here’s what needs to be added
    • Allow users to delete their dashboards
    • Start up in ‘Builder’ layout
    • Update tips
    • Add tips to the IngestManager
    • If nothing is selected, have a popup that says “No items selected for query”
    • Provide direct transition from minimize and maximize
    • Don’t allow components (particularly minimized windows) to go behind the top “VISIBILITY2” UIComponent
    • Resize events on the builder window should re-trigger the layout
    • Automatically scroll to the selected item in a search on a datagrid
    • Add a “Suggestion Box” under Utils that’s like the old bug reporter on TUMMS
  • Architecture (get aquainted session?) in Fairfax. Ran a little long. Note. If you leave VA at 3:15 on a Friday, it can take ’till 5:00 to get back to Laurel.

Phil 9.17.2009

7:30 – 5:00

  • Uploaded data
  • Tried another version of VIS2 – this time it got confused with lines that have the same words in them – i.e. “Position_Person” vs “Person_Position
  • Fixed the ordering problem and discovered a case where I got runaway recursion.
  • Fixed that, but then had a problem where one of the leaf nodes of a branch had the same name as the branch. Fixed that, too.
  • Added timers for how long pull and collapse hapens on the server
  • Removed redundant data from QueryInfoObject
  • Removed some logging from the XML ingestor
  • Added tooltips to titlebar and query entry field. Tried to get datatips on the widget finder to work, but no luck
  • Ftp’d new version home so I can burn a disk
  • John Williams is having trouble with the system running in IE. Related to the debugger version?

Phil 9.16.2009

7:30 – 4:30

  • Uploaded data
  • Tried deploying a new VIS2, but the hierachical display didn’t work on some of the tables. Rewrote so that the system uses recursion
  • Meetings
  • I had more to add, but my computer sized up yesterday and rebooted on me.

Phil 9.15.2009

7:30 -4:00

  • Ingested data
  • Yesterday I got most of the hierarchical decomposition of column headers into a tree view. Need to make that work with BaseQueryWidget. Done.
  • Integrating with Dong’s code – done