Phil 12.7.15

8:00 – 5:00 VTX

  • Got my laptop from John and got it set up. Incredibly slow network performance, which I figured was the wifi. Hooked up the hard line and disabled the wifi, which doesn’t see the network at all, and won’t let me reconnect the wifi. Working from home for the rest of the day.
  • At seminar, had a really interesting discussion about how taxonomies intersecting with knowledge graphs essentially result in a kind of pro-forma synthesis. Hmm.

Phil 12.4.15

8:00 – VTX

  • Scrum
  • Found an interesting tidbit on the WaPo this morning. It implies that if there is a pattern of statement followed by a search for confirming information followed by a public citation of confirming information could be the basic unit of an information bubble. For this to be a bubble, I think the pertinent information extracted from the relevant search results would have to be somehow identifiable as a minority view. This could be done by comparing the Jaccard index of the adjusted results with the raw returns of a search? In other words, if the world (relevant search)  has an overall vector in one direction and the individual preferences produce a pertinent result that is pointing in the opposite direction (large dot product), then the likelihood of those results being the result of echo-chamber processes are higher?
  • If the Derived DB depends on analyst examination of the data, this could be a way of flagging analyst bias.
  • Researching WebScaleSQL, I stumbled on another db from Facebook. This one,  RocksDB, is more focused on speed. From the splash page:
    • RocksDB can be used by applications that need low latency database accesses. A user-facing application that stores the viewing history and state of users of a website can potentially store this content on RocksDB. A spam detection application that needs fast access to big data sets can use RocksDB. A graph-search query that needs to scan a data set in realtime can use RocksDB. RocksDB can be used to cache data from Hadoop, thereby allowing applications to query Hadoop data in realtime. A message-queue that supports a high number of inserts and deletes can use RocksDB.
  • Interestingly, RocksDB appears to have integration with MongoDB and is working on MySQL integration. Cassandra appears to be implementing similar optimizations.
  • Just discovered reported.ly, which is a social medial sourced, reporter curated news stream. Could be a good source of data to compare against things like news feeds from Google or major news venues.
  • Control System Meeting
    • Send RCS and Search Competition to Bob
    • Seems like this whole system is a lot like what Databricks is doing?

Phil 12.3.15

7:00 – 5:00 VTX

  • Learning: Genetic Algorithms
    • Rank space (probability is based on unsorted values??)
    • Simulated annealing – reducing step size.
    • Diversity rank (from the previous generation) plus fitness rank
  • Some more timing results. The view test (select count(*) from tn_view_network_items where network_id = 1) for the small network_1 is about the same as the pull for the large network_8, about .75 sec. The pull from the association table without the view is very fast – 0.01 for network_1 and 0.02 for network_8. So this should mean that a 1,000,000 item pull would take 1-2 seconds.
  • mysql> select count(*) from tn_associations where network_id = 1;
 11 
1 row in set (0.01 sec)

mysql> select count(*) from tn_associations where network_id = 8;
 10000 
1 row in set (0.01 sec)

mysql> select count(*) from tn_view_network_items where network_id = 8;
 10000 
1 row in set (0.88 sec)

mysql> select count(*) from tn_view_network_items where network_id = 1;
 11 
1 row in set (0.71 sec)
  • Field trip to Wall NJ
    • Learned more about the project, started to put faces to names
    • Continued to look at DB engines for the derived DB. Discovered WebScaleSQL, which is a collaboration between Alibaba, Facebook, Google, LinkedIn, and Twitter to produce a big(!!) version of MySql.
    • More discussions with Aaron D. about control systems, which means I’m going to be leaning on my NIST work again.

Phil 12.2.15

7:00 –

  • Learning: Neural Nets, Back Propagation
    • Synaptic weights are higher for some synapses than others
    • Cumulative stimulus
    • All-or-none threshold for propagation.
    • Once we have a model, we can ask what we can do with it.
    • Now I’m curious about the MIT approach to calculus. It’s online too: MIT 18.01 Single Variable Calculus
    • Back-propagation algorithm. Starts from the end and works forward so that each new calculation depends only on its local information plus values that have already been calculated.
    • Overfitting and under/over damping issues are also considerations.
  • Scrum meeting
  • Remember to bring a keyboard tomorrow!!!!
  • Checking that my home dev code is the same as what I pulled down from the repository
    • No change in definitelytyped
    • No change in the other files either, so those were real bugs. Don’t know why they didn’t get caught. But that means the repo is good and the bugs are fixed.
  • Validate that PHP runs and debugs in the new dev env. Done
  • Add a new test that inputs large (thousands -> millions) of unique ENTITY entries with small-ish star networks of partially shared URL entries. Time view retrieval times for SELECT COUNT(*) from tn_view_network_items WHERE network_id = 8;
    • Computer: 2008 Dell Precision M6300
    • System: Processor Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz, 2201 Mhz, 2 Core(s), 2 Logical Processor(s), Available Physical Memory 611 MB
    • 100 is 0.09 sec
    • 1000 is 0.14 sec
    • 10,000 is 0.84 sec
    • Using Open Office’s linear regression function, I get the equation t = 0.00007657x + 0.733 with an R squared of 0.99948.
    • That means 1,000,000 view entries can be processed in 75 seconds or so as long as things don’t get IO bound
  • Got the PHP interpreter and debugger working. In this case, it was just refreshing in settings->languages->php

Phil 12.1.15

7:30 – 5:00

  • Learning: Identification Trees, Disorder
    • Trees of tests
    • Identification Tree (Not a decision tree!)
    • Measuring Disorder – lowest disorder is best test
      • Disorder(set of binaries) = -(positive/total*log2(positive/total)) – (neg/total*log2(neg/total))
        • is the base log related to the base of the set?
        • Add up the disorder of each result in the test to determine the disorder of the test normalized by the number of samples. Lowest disorder is winner
  • Bringing in my machine learning, pattern recognition and stats books.
  • Bringing in my big laptop
  • Setting up dev environment.
    • Using the new IDEA 15.x, which seems to be OK for the typescript, will check PHP tomorrow.
    • Installed grunt (grunt-global, then grunt-local from the makefiles)
    • installed typescript (npm i -g typescript)
    • Installed gnuWin32 , which has makefile and touch support, along with all the important DLLs. It turns out that there is also gnuWin64. Will use that next time
    • Fixed bugs that didn’t get caught before. Older compiler?
      • commented out the waa.d.ts from the three.d.ts definitelytyped file
      • deleted the { antialias: boolean; alpha: boolean; } args from the CanvasRenderer call in classes/WebGlCanvasClasses
      • added title?:string and assoc_name?:string to IPostObject in RssController
      • had to add the experiments/wglcharts2 folder to the xampp Apache htdocs
      • added word?:string to IPostObj in RssAppDirectives
      • added word_type_name?:string to IPostObj in RssAppDirectives
      • fixed the font calls in WebGl3dCharts IComponentConfig.
    • Since these issues really shouldn’t have happened, I’m going to verify that they are not in my home dev environment before checking in.
  • And the new computer arrived, so I get to do some of the install tomorrow.

Phil 11.30.15

7:00 – 2:30: ???

  • Introduction to Learning, Nearest Neighbors
    • Learning based on observations of regularity (Bulldozer Computing)
      • Nearest Neighbor
        • Pattern Recognition
      • Neural Networks
      • Boosting
    • Learning based on constraint (Human-Like)
      • One Shot Learning
      • Explanation-based learning
    • Pattern Recognition
      • Feature detector produces a vector of values.
      • Fed into a Comparator which tests the new vector against a library of other vectors
      • Can use decision boundaries
      • If something is similar in some respects, it is likely to be similar in other respects.
      • Robotic motion is a search problem these days??
  • Work
    • Standard first-day stuff
    • Discussions with Aaron about design
    • And the interesting thought for the day:
      • Do we need a sort of crowd-sourced weighting determination of machine ethics? Right now, the person that writes the code for the first self-driving car that decides the runaway trolley problem could reasonably be thought of as having committed premeditated murder. But what if we all together set those outcomes, in a way that reflected our current culture and local values?

Phil 11.26.15

7:00 – Leave

  • Constraints: Visual Object Recognition
    • to see if to signals match, a maximising function that integrates the area under the signal with respect to offsets (translation and rotation) is very good, even with noise.
  • Dictionary
    • Add ‘Help Choose Doctor’, ‘Help Choose Investments’, ‘Help Choose Healthcare Plan’, ‘Navigate News’ and ‘Help Find CHI Paper’ dictionaries. At this point they can be empty. We’ll talk about them in the paper.
    • Added ‘archive’ to dictionary, because we’ll need temporary dicts associated with users like networks.
    • Deploy new system. Done!
      • Reloaded the DB
      • Copied over the server code
      • Ran the simpleTests() for AlchemyDictText. That adds network[5] with tests against the words that are in my manual resume dictionary. Then network[2] is added with no dictionary.
      • Commented out simpleTests for AlchemyDictText
      • copied over all the new client code
      • Ran the client and verified that all the networks and dictionaries were there as they were supposed to be.
      • Loaded network[2] ‘Using extracted dict’
      • Selected the empty dictionary[2] ‘Phil’s extracted resume dict’
      • Ran Extract from Network, which is faster on Dreamhost! That populated the dictionary.
      • Deleted the entry for ‘3’
      • Ran Attach to Network. Also fast 🙂
  • And now time for ThanksGiving. On a really good note!

AllWorking

Phil 11.25.15

7:00 – 1:00 Leave

  • Constraints: Search, Domain Reduction
    • Order from most constrained to least.
    • For a constrained problem, check over and under allocations to see where the gap between fast failure and fast completion lie.
    • Only recurse through neighbors where domain (choices) have been reduced to 1.
  • Dictionary
    • Add an optional ‘source_text’ field to the tn_dictionaries table so that user added words can be compared to the text. Done. There is the issue that the dictionary could be used against a different corpus, at which point this would be little more than a creation artifact
    • Add a ‘source_count’ to the tn_dictionary_entries table that is shown in the directive. Defaults to zero? Done. Same issue as above, when compared to a new corpus, do we recompute the counts?
    • Wire up Attach Dictionary to Network
      • Working on AlchemyDictReflect that will place keywords in the tn_items table and connect them in the tn_associations table.
      • Had to add a few helper methods in networkDbIo.php to handle the modifying of the network tables, since alchemyNLPbase doesn’t extend baseBdIo. Not the cleanest thing I’ve ever done, but not *horrible*.
      • Done and working! Need to deploy.

Phil 11.24.15

7:00 – Leave

  • Constraints: Interpreting Line Drawings
    • Successful research:
      • Finds a problem
      • Finds a method that solves the problem
      • Using some principal (That can be generalized)
  • Gave Aaron M. A subversion account and sent him a description of the structure of the project
  • Back to dictionary creation
    • Wire up Extract into Dictionary
      • I think I’m going to do most of this on the server. If I do a select text from tn_view_network_items where network = X, then I can run that text that is already in the DB through the term extractor, which should be the fastest thing I can do.
      • The next fastest thing would be to pull the text from the url (if it exists) and add that to the text pull.
      • Added a getTextFromNetwork() method to NetworkDbObject.
      • The html was getting extracted badly, so I had to add a call to alchemy to return the cleaned text. TODO: in the future add a ‘clean_text’ column to tn_items so this is done on ingestion. I also added
      • Added all the pieces to the rssPull.php file and tested. And integrated with the client. Looks like it takes about 8 seconds to go through my resume, so some offline processing will probably be needed for ACM papers, for example.
    • Wire up Attach Dictionary to Network
      • The current setup is set so that a new item that is read in will associate with the current network dictionary. Need to add a way to have the items that are already in the network to check themselves against the new dictionary.
      • Added class AlchemyDictReflect that will place keywords in the DB. Still need to debug. And don’t forget that the controller will have to reload the network after all thechanges are made.

 

Phil 11.23.15

7:00 – Leave

  • Search: Games, Minimax, and Alpha-Beta
    • Branching factor (B)
    • Search depth (D)
    • Combining the two gives the number of leaf nodes or B^D
    • Branching factor of chess is approximately 14?
  • Dictionaries
    • Wire up Create New Dictionary – done
    • Wire up Extract into Dictionary
      • I think I’m going to do most of this on the server. If I do a select text from tn_view_network_items where network = X, then I can run that text that is already in the DB through the term extractor, which should be the fastest thing I can do.
      • The next fastest thing would be to pull the text from the url (if it exists) and add that to the text pull.
    • Wire up Attach Dictionary to Network
      • The current setup is set so that a new item that is read in will associate with the current network dictionary. Need to add a way to have the items that are already in the network to check themselves against the new dictionary.

Phil 11.19.15

7:00 – 5:00 Leave

  • Reasoning: Goal Trees and Rule-Based Expert Systems
    • There, now I’m back in order.
    • H. Simon – The complexity of the behavior is max(cplx(prgm), cplx(env))
    • More Genesis – Elaboration graphs
    • Genesis judges similarity in multiple ways: (this presentation, page 25)
      • Using word vectors
      • Using concept vectors: seeing similarities not evident in the words.
    • Genesis aligns similar stories for analogical reasoning (Needleman-Wunch algorithm, which is a way of comparing string similarity using matrices)
  • IRB renewal – ask Wayne
  • In a fit of orderliness, created shortcuts to the cmd windows that the makefiles run in
  • Dictionaries
    • Don’t forget to move the text-extraction calls to the methods that need it and see if that speeds up the others. Done. Much faster. PHP is dumb. Or needs a preloader/compiler
    • Cleaned up the loading and display code a bit. Need to add a tree view (which looks like it can be done in pure CSS), but that can wait.
    • Adding manual entry
      • Finished the form
      • Clear for entry and separate parent is done
      • addition
      • deletion
      • modification (adding parents in particular)
    • Adding text extraction

Phil 11.18.15

7:00 – 5:00 Leave

  • Search: Depth-First, Hill Climbing, Beam
    • Looks like I missed one. Will go back tomorrow.
    • Patrick Winston is the instructor, and he showed a version of Genesis (around 42 minutes in), which maps stories into graphs and then searches through them at different levels of abstraction. It is capable of providing relevant answers to questions. Here’s a white paper. What I think is interesting here is the following:
      • How they describe their hierarchy (revenge, tit-for-tat, etc). This is built, not computed based on what looks to be a rules engine.
      • How text in a story can be looked at as a graph.
      • And that’s without reading the paper!
  • Working on dictionary directive
    • Ramifying the dictionary choice through the controller. It should be set initially when the network loads (done), but should be changed if a new one is selected.
    • There does need to be a second step where the DictPull is run and the text of the items is compared against the selected dictionary and KEYWORDS items are added to the network. This is when the network’s dictionary index should be changed.
    • Downloading the dictionary.
    • Need to upload a new word
  • 12:00 workshop on Grant Writing in ITE 459
  • 4:00 meeting with Dr. Pan.

Phil 11.17.15

7:00 – 4:00 leave

  • Reasoning: Goal Trees and Problem Solving (why is this pertinent to me?)
    • Apply all safe transforms
    • Apply heuristics
    • AND nodes and OR nodes (AND/OR, problem reduction, or goal tree)
  • Realized that I have a redundant user_id in tn_dictionary_entries. This could be used to allow non-owners to add words to the dictionary. Which means that dictionaries could be shared. On the whole, I think that’s a good idea. Adding the change to the dictionary code.
  • Ok, back to text extraction (Is this a safe transform?)
  • Added a _buildExtractor() private method and imported all the extractor parts.
  • It does take a long time. Just to load? I’d like to try profiling…
  • Wow. Extracting, loading into the database, getting the JSON output and deleting the dictionary all work!
  • I think the next step is to either get some definitions or start building the directive. After my lunchtime ride, I decided that the directive is probably the best thing to do next. Adding user functionality is a good way of ensuring that the server functionality makes sense.
  • Added ‘get_user_dictionaries‘ to rssPull.php. We’ll start with that.
  • Got the skeleton of a directive up and retrieving the dictionary list.

 

Phil 11.16.15

7:00 – 6:00 Leave

  • Found a new programmer resource that looks good – I Programmer. They pointed me to an article about Babel, which compiles JavaScript to… other things. It might even be able to monkeypatch modern JS to run on old browsers. Need to test one of these years. It’s based on plugins which really means that it can map from one thing to anything else. My only issue is that it could break debugging unless there is a mapping file like typescript has.
  • Discovered another communication app – Telegram. ISIS used it to announce Paris?
  • Noon – Thad Starner in ITE 459. Very interesting. Met Aaron Massey, who might be good on the Committee.
  • I’ve been reading Tefko Saracevic‘s paper RELEVANCE: A Review of and a Framework for the thinking on the Notion in Information Science. It’s full of really nice stuff, from a time when you couldn’t just throw processing power at problems and brute force an answer. It’s clarified my thinking about the client word-based network:
    • Search engines are pragmatic relevance engines (i.e topic-relatedness, quality, novelty, importance, credibility, etc.). The networks that they produce try to correlate knowledge at the ‘source’ – basically ‘in the world’
    • We, as individuals are pertinence/situational relevance machines (Wilson’s concerns, preferences and stock of knowledge). Our internal knowledge graph represents our view of the world. We are the ‘destination’ for information.
      • “Situational Relevance is relevance to a particular individual’s situation – but to the situation as he sees it, not as others see it, nor as it really is.”
      • The ‘shape’ of our internal knowledge graph, the sources of information that we lean more heavily on, the weights that we give to certain words (or possibly concepts) may be able to determine whether we are dependably credible or dependably counter-credible.
    • By enabling client-side weighting, we let users adjust components of a relevant search so that it becomes pertinent to us.
    • The information that we produce in this process (dictionaries, weights, etc) can be stored so that a well-structured record of what is pertinent to individuals (and more importantly, groups of individuals) becomes part of the world knowledge. Correlations with respect to internal credibility may then in turn be able to infer the credibility (or lack of) of information in the world.
  • Getting back to dictionary integration.
    • Re-upped my IntelliJ subscription for another two years
    • Updated files and DB. All seems to work
    • DbDictionary.removeDictionary returns a fail JSON message. Fixing. Fixed!
    • Adding ability to update an entry – done.
    • Finishing CreateDictionary. Finished and tested
    • Adding DeleteDictionary. Finished and tested
    • Adding ModifyDictionary. Finished and tested
    • Adding term extraction. Started poking, but that’s it. More tomorrow.