Category Archives: 3D Charting

Phil 4.25.16

5:30 – 4:00 VTX

  • Saw this on Twitter about visualizing networks with D3
  • Working my way through the JavaFX tutorial. It is a lot like a blend of Flex and a rethought Swing. Nice, actually…
  • Here is the list of stock components
  • Starting with the ope file dialog – done.
  • Yep, there’s a spinner. And here’s dials and knobs
  • And here’s how to do a word cloud.
  • Here’s a TF-IDF implementation in JAVA. Need to build some code that reads in from our ‘negative match’ ‘positive match’ results and start to get some data driven terms
  • Tregex is a utility for matching patterns in trees, based on tree relationships and regular expression matches on nodes (the name is short for “tree regular expressions”). Tregex comes with Tsurgeon, a tree transformation language. Also included from version 2.0 on is a similar package which operates on dependency graphs (class SemanticGraph, calledsemgrex).
  • Semgrex
  • Sprint review
    • Google CSEs
      • Switched over from my personal CSEs to Vistronix CSEs
      • Added VCS rep for CSEs
      • Figured out how to save out and load CSE from XML
      • Added a few more CSEs ONLY_NET, MOBY_DICK
      • Wrote up care and feeding document for Confluence
      • Added blacklists
    • Rating App
      • Re-rigged the JPA classes to be Ontology-agnostic Version 2 of nearly everything)
      • Upped my JQL game to handle SELECT IN WHERE precompiled queries
      • Reading in VA and PA data now
      • Added the creation of a text JSON object that formalizes the rating of a flag
      • Got hooked up to the Talend DB!!!
      • Deployed initial version(s)
      • Added backlink logging using SemRush
    • Future work
      • Developed Excel ingest
      • Still working on PDF and Word ingest

Phil 10.15.15

8:00 – 5:00 SR

  • Still don’t have my access back
  • But, was able to update the needed table for FR using the scripting system. So that’s pretty cool.
  • Looking to migrate over to the new servers next Friday.
  • Adding behaviors to support the changes I made to the GUI.
    • Link to wampeter
      • GUI – Done
      • DB – Done. Changed the way that linkSelected worked so that if a sourceID was passed along a wampeter was not created.
    • Update wampter value change DB (means updating association weights)– Done
    • Update Rating value change DB – Done
  • Taadaa! ItAllWorks

Phil 10.12.15

1:30 – 5:30 SR

  • Adding data objects to ITarget – done
  • Adding test in parseDataObject and addNetItems to see if links aren’t loading sometimes. Done
  • Adding a check to see prevent (non-URL, non-Query) items that only have one link aren’t added to pagerank calculation and listing. Done.
  • Adding an ‘add selected’ to the Wampeter feed directive – button added TODO: add functionality
  • Adding unlink from: <wampeter name> to any icon that is, well, linked to a wampeter. Not that this may need to be an ng-repeat. – button added TODO: add functionality
  • May switch over to keywords only. Concepts and entities overlap a lot. Also thinking that Adding the url could do that without exploring. Just open and make sure that it’s not 404? And just running with that thought, Explore could return a ‘Cliff Notes’ Dialog that could have the capability to link to other items? Or just select what gets attached? And merged?…

Phil 9.10.15

8:00 – 4:00 SR

  • Lost my accounts on all the servers. Asked Ronda to look into it.
  • And since I can’t get on the server, asked Ronda to fix the setupModel.xml files to fix login issues.
  • Sent Dong a note to see what’s going on. He’ll take a look this weekend.
  • Some of my demos that use the WebGl3dCharts module were broken. The reason is that I was looking for items in the optional data objects and throwing a null. Two lessons:
    • FF console does not log this error
    • Just because you have static type checking in TypeScript doesn’t mean that you shouldn’t use hasOwnProperty() to check if an item is present. Here’s the fix: 
      var forceVisible:boolean = false;
if(itemData.hasOwnProperty('data') && itemData.data.hasOwnProperty('type')) {
    forceVisible = this.getForceVisible(itemData.data.type);
}

      So now it’s much more robust.

    • Adding ability to modify text. Done. Changing the title also ripples through the display. I also don’t (re)calculate page rank if the weight hasn’t changed.  Also added 
      findModelFromDataObject()

      for WebGlCanvas.

  • Found these guys (http://www.intensityanalytics.com/), who do keyboard biometrics. Trying to contact them.

Phil 10.8.15

8:00 – 4:00 SR

  • Noticed yesterday that the time it takes for the map to center is just too long. When I tried increasing the speed I got a lot of jitter because I was testing for proximity to goal. I wound up changing so that the goal is reached when the distance to the goal is increasing.
  • Add editing of Rating and linkSelected
  • Make sure that items with only one link that are not ‘force visible’ are not shown in lists
  • Bold selected items in lists
  • Change the query list to be the ‘visible’ list
  • Trim the length of the list items so the containing box sizes don’t go crazy.
  • Look for floating panel CSS?

Phil 10.7.15

8:00 – 6:00 SR

  • Gina’s still having problems logging on
  • Write up the DB and programming test for Lenny, since the next interview(?) won’t have me along
  • Banged away at PageRank and finally have it working. The matricies were evaporating, so I wound up layering inbound and outbound links over an identity matrix. That worked great. I need to think about why though.
  • Meeting with Dr. Pan and Dr. Lutters at 4:00(?). Went well.
    • One thing that came up in the discussion was how to feed in enough information to make the analytics work. Dr. Pan suggested using Politifact, since it’s well formatted ground truth that references a source. In essence, I could build a small java program that could iterate over all the PF reports and build a network for each of the Entities (or concepts) referenced. Once nice side effect is that this could be a ‘seed set’ of vetted data that could be used to inform other searches.
    • The other item discussed is whether this is what kind of view this is. Dr. Lutters suggested that this might be the analyst’s (development) view, while a consumer’s view would be more like a traditional  news feed, like inkl.com

Phil 9.10.15

8:00 – 2:30 SR

  • Taking tomorrow off
  • No progress on dev machine admin
  • Added page rank function to PhilsUtils
  • Discovered that I still needed to pull fresh meta
  • Make it so that a shift-click on a BaseComponent selects all the items that the item is directly connected to. Done.  Bonus points for each shift-click selects an additional hop. TODO
  • Got my first linkage!

DriverF1

  • Next is some automated culling, and then adding rating/pagerank. TODO

Phil 9.8.15

8:00 – 4:00 SR

  • Off yesterday for Labor Day.
  • More dev justification
  • Google Cloud platform java – https://cloud.google.com/appengine/docs/java/. The goal of this would be to calculate overall page ranks
  • Added Save for networks. In reality, this only Creates a network that is then ‘saved live’. TODO: Need to make that more clear
  • Work on correlation of concepts, keywords, etc. Start with ‘horse’ and ‘rider’?  TODO
  • Work on support/dispute 
  • PageRank in PHP 

Phil 8.31.15

8:00 – 4:30 SR

  • Status report – Done
  • Timesheets! – Done
  • Get selected items showing up in the feed panel. Done! looks good. Commented out the default addition to the feed. It did make me realize that I need to add a ‘filter network’. Selections should highlight in the GUI and list in the feed.
    • By text search – searching title, description and query
    • By Date range
    • Other?
  • Got the selection working. Query-ish items go in the query box where they are combined into a single query, URL-ish items are listed in the feed section.
  • Get parsing of webpages working so so that they fit into the googleNewsFeed object. This should be just using the Alchemy object with some additional parsing.
  • Deployed to the server, and verified that the page works with Chrome, FF, Edge and IE11

Phil 8.28.15

8:00 – 5:00 SR

  • Commuting in the reverse order. Don’t forget to pick up energy drink mix too!
  • 3:00 meeting with Ronda to work on the servers
  • Adding isAssociation() and isItem() methods to the rssFeed directive. Done. Will probably add isRating() and isTouch() once they start getting used.
  • Deleting ComponentNetwork class from rssController – Done
  • Deleting pendingQueryArray and queryHistory from rSSController. The history can be pulled from the items in the DataProvider. Which works.
    • Add some test QUERY types to validate – Done – replaced the ENTITIES in tn_items. Worked with no other changes.
  • Setting items if their associated queries are clicked and setting the query if a query item in the chart is clicked. Done!
  • Deploying new version to the server.
  • And I am done for the day. Next week needs the following
    • Add items, which will (should?) return the whole data provider. Only differences (additions and deletions) should cause an effect.
    • Have selected items (only?) show up in the list.
    • Have the ‘query’ task change it’s behavior based on what’s sitting in the field. If it’s an url, open it in a new tab.If it’s text (can be multiple queries) combine all the text into a new query and run it.
    • Continue to flesh out the types.
    • Save, Save As, reload (clear and load) models. Verify! Test!

Phil 8.24.15

8:00 – 3:00 SR

  • Still working the Zeno’s Administrator problem.
  • Had a good discussion with Lenny about Low Threshold/High Ceiling coding and also about functional vs declarative programming and how ExtJS and AngularJS relate to those concepts.
  • Adding items to the live network and then bring down and parse the data object. This means pulling out the code that loads before the DB call.
  • Added queries and associations to any query that is not “__topNews__”

Phil 8.21.15

8:00 – 4:30 SR

  • Still no admin, but we’re at the last step. I think.
  • Meeting with the GMO folks. They have an ExtJS framework (YUI-based!?) for all their user IO.
  • We will *not* be abandoning Angular: xtJsVsAngularJs
  • Long discussion with Mark and Lenny about next steps. Maybe integrate all the efforts? I made the point that what we have is an information navigation system, but the info is questionable. Wheat we need is a data/inference system
  • More housekeeping for network logging in. Cleaned up the add network code on the client and server. Everything seems to be working as it’s supposed to. Next step will be to add items to the live network and then bring down and parse the data object.
  • I also realized that the best way to show the items is to have the master list populated by sublists. So the status item would be loaded first, then the selected items, then the search results. Then the various sublists can be managed separately and the other lists won’t care.

Phil 8.19.15

8:00 – 5:00 SR

  • Admin has been approved. Waiting for my logins to re-activate
  • Friday, 9:00-ish meeting with GAO-ish folks to talk about ingest.
  • Continue working on storing items and associations.
  • These people integrate annotations into the article, which is deeper. But in the reading of an article, this seems overwhelming
  • Bayes Server – worth getting? John G. Sandiford’s thesis.
  • Adding a UID to associations, since they need to be unique and findable. Done, and it’s for within a network.
  • Added a HOST type so that the root of the domain can be stored as an attribute as well. Of course, there was a PHP call to do this.
  • Tested creation of items and associations against this Miami Herald article.
  • Need to make sure that NLP items don’t get stored if they don’t return right.

Phil 8.18.15

8:00 – 5:00 SR

  • Still no admin
  • I forgot to save the new SQL for the truancy reports. Fixed now.
  • There is something that’s not quite right in the network physics. When an object is added, sometimes the forces get asymmetric. My guess is that the velocities have to be made non-zero each time an item is added.
  • Was getting a problem where additional information was getting stuck on the end of the image url, which was contaminating the name. Now, rather than assuming that the image name ends with the extension, I look within the image url for a reasonable extension and return that.
  • Need a query that returns the associations shared between two items. Done
  • Working on adding Alchemy NLP results as items with associations

Phil 8.17.15

8:00 – 3:00 SR

  • Still no admin
  • Given that I’m using FizzBuzz for the coding side of the SWE interview, I though I should put together a test for the database portion of the interview:
    • Create three tables:
      • One for names and salaries,
      • One for roles,
      • One that assigns a particular person to one or more roles.
    • Create a query that prints the name strings with their role strings. Extra points if it’s a view.
    • Create a query that sums the salaries.
    • Create a query that sums the salaries for a given role. 
      DROP TABLE IF EXISTS test_people;
CREATE TABLE test_people (
	id_index 		INT AUTO_INCREMENT PRIMARY KEY NOT NULL,
	name			varchar(255),
	salary			BIGINT
);

insert into test_people (name, salary) values
('Sophia',10000000000),
('Emma',1000000000),
('Olivia',100000000),
('Isabella', 10000000),
('Jackson',1000000),
('Aiden',100000),
('Liam',10000),
('Noah', 1000),
('Charlotte', 100),
('Caden', 10),
('Harper', 1);

DROP TABLE IF EXISTS test_roles;
CREATE TABLE test_roles (
	id_index 		INT AUTO_INCREMENT PRIMARY KEY NOT NULL,
	role			varchar(255)
);

insert into test_roles (role) values
('Minion'),
('Worker'),
('Manager'),
('CEO'),
('Evil Overlord');

DROP TABLE IF EXISTS test_assignments;
CREATE TABLE test_assignments (
	person_id 	INT,
	role_id		INT
);
insert into test_assignments (person_id, role_id) values
(1, 5),
(2, 4),
(3, 3),
(4, 2),
(5, 1),
(6, 1),
(7, 1),
(8, 1);

drop view IF EXISTS test_join;
create view test_join as
select p.name, r.role, p.salary
	from test_assignments a
	inner join test_people p on a.person_id = p.id_index
	inner join test_roles r on a.role_id = r.id_index;

select * from test_join;
select sum(salary) from test_join where role = "Minion";

DROP TABLE IF EXISTS test_people;
DROP TABLE IF EXISTS test_roles;
DROP TABLE IF EXISTS test_assignments;
drop view IF EXISTS test_join;
  • Add the ability to save a network and then start saving one!
  • Since the mysqldump file doesn’t work on my server, spit the current schema file into three sections:
    • Table creation (incorporate DROP TABLE IF EXISTS table_name) – done
    • View creation – done
    • Tests – done
  • Using the linkFn() in ngFlyoutPanel to call getNetworks() on the load of the directive. Wasn’t sure that it would work, but it does. In Chrome, FF, and IE yet..
  • Uploaded to the main server. Works there, too!