Monthly Archives: December 2015

Phil 12.31.15

Phil 7:00 – 4:00 VTX

  • Decided to get a copy (hopefully with student discount) of Atlas. It does taxonomic analysis and outputs a matrix to Excel that I should be able to use to produce PageRank???
  • Timesheets! Done
  • Seeing if I can add a bit of reflection to addXXX(YYY) that will invoke add/setYYY(XXX). Since the target is a map, it shouldn’t care, but I do need to worry about recursion…
  • Added addSourceToTarget() and testSourceInTarget() to GuidBase. So Now addM2M looks like 
    public void addM2M(GuidBase obj) throws Exception {
    System.out.println("person.addM2M");
    addSourceToTarget(this, obj);
    addSourceToTarget(obj, this);
}

    and the example of Showroom.addCar() looks like

    public void addCar(Car car){
    if(cars == null){
        cars = new HashSet<>();
    }
    cars.add(car);

    try {
        if(!testSourceInTarget(this, car)){
            addSourceToTarget(this, car);
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

    Which now means that the two way mapping is automatic. And in case you’re wondering, testSourceInTarget looks first for a method that returns the source type, and then looks for a method that returns a Set<source type>. If it find the source in either one of those, it returns true.

  • Got queries running. Simple queries are easy, but the more complex ones can be pretty ugly. Here’s an example that pulls a Showroom Object based on a nested Person Object (instanced as ‘customer’): 
    // do a query based on a nested item's value. Kinda yucky...
Criteria criteria = session.createCriteria(Showroom.class, "sh");
criteria.createAlias("sh.customers", "c");
List result = criteria.add(Restrictions.like("c.name", "%Aaron%")).list();
for(Object o : result){
    System.out.println(o.toString());
}

Phil 12.30.15

7:00 – 5:00 VTX

  • Finished up notes on the Gezi paper
  • Back to hibernate
    • finishing up M2M method
    • Have both items pointing at each other. Currently throwing a plain Exception. Will probably need to change that later, if the message turns out not to be enough.
    • Moved M2M to the GuidBase class and tested with the showroom example I’ve been playing with. Success! Here’s the use case from the header: 
      Class that sets up the mutual relationships between two classes that have to have access to each other's
Sets in the context of a 'ManyToMany' Hibernate configuration. To set up the classes, do the following:
1) Add the annotations that define the mapping table in one class - e.g:
For the class 'Kitten'
   @ManyToMany(cascade = CascadeType.ALL)
   @JoinTable(name = "kittens_puppies",
      joinColumns = @JoinColumn(name = "kitten_id"),
      inverseJoinColumns = @JoinColumn(name = "puppy_id")
   )
   private Set<Puppy> puppies;
Similarly, for the class 'Puppy'
@ManyToMany(cascade = javax.persistence.CascadeType.ALL, mappedBy = "puppies" )
 private Set<Kitten> kittens;
Each class will need an 'addXXX(XXX xxx) class that adds a single element of XXX to the Set. This is the
template that M2M is looking for. There needs to be one and only one method for each mapping.
An example from the class 'Kitten' is shown below:
   public class addPuppy(Puppy puppy){
       if(puppies == null){
           puppies = new HashSet()
       }
       puppies.add(puppy)
   }
Lastly, the code that handles the session needs to call xxx.M2M once for each of the relationships:
session.beginTransaction();
Kitten k1 = new Kitten();
Kitten k2 = new Kitten();
Kitten k3 = new Kitten();
Kitten k4 = new Kitten();
Puppy p1 = new Puppy();
Puppy p2 = new Puppy();
Puppy p3 = new Puppy();
Puppy p4 = new Puppy();
k1.M2M(p1);
k1.M2M(p2);
k1.M2M(p2);
k1.M2M(p4);
k2.M2M(p2);
k2.M2M(p4);
k3.M2M(p1);
k3.M2M(p3);
session.save(k1);
session.save(k2);
session.save(k3);
session.save(k4);
session.getTransaction().commit();
    • And that seems to be working! I got a little confused as to which item should be mapped to, but now understand that the collection to be mapped to has to (should?) be the Set<XXX> of items that are referenced in the YYY class that contains the @JoinTable annotations.
    • Need to add a relation between networks and associations and items. Done.
    • Time to figure out queries. Get all the networks names for a user, then get a network, that sort of thing.

Phil 12.29.15

7:00 – 5:30 VTX

  • Finished Social Media and Trust during the Gezi Protests in Turkey.
  • More Hibernate
    • After stepping through the ‘Showroom’ example on pg 54 (Using a Join Table), of Just Hibernate I think I see my problem. In my case, the corpus has already been created and exists as an entry in the corpora table. I need to add a relationship when a word is run against a new corpus. Which come to think about it, should have the word count in it. Maybe?
    • Ok, I don’t like the way that ManyToMany is implemented. Hibernate should be smart enough to figure out how to make a default mapping table. Sigh.
    • And you have to call each object with the other object or it doesn’t load properly.
    • After being annoyed for a while, I decided to try reflection as a making fewer calls. The following still needs the get/set member element calls, but I like the direction it’s going. Will work on it some more tomorrow (need to add the call on the this class): 
      public void addM2M(Object obj){
    System.out.println("person.addM2M");
    Class thisClass = this.getClass();
    String thisName = thisClass.getName();
    Class thatClass = obj.getClass();
    Method[] thatMethods = thatClass.getMethods();
    Method thatMethod = null;
    for(int i = 0; i < thatMethods.length; ++i){
        Method m = thatMethods[i];
        Type[] types = m.getGenericParameterTypes();
        if (types.length == 1) { // looking for one arg setters
            for (int j = 0; j < types.length; ++j) {
                Type t = types[j];
                if ((t.getTypeName() == thisName)) {
                    thatMethod = m;
                    break;
                }
            }
        }
        if(thatMethod != null){
            break;
        }
    }
    if(thatMethod != null){
        try {
            thatMethod.setAccessible(true);
            thatMethod.invoke(obj, this);
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        } catch (InvocationTargetException e) {
            System.out.println("addM2M failed: "+e.getCause().getMessage());
            e.printStackTrace();
        }
    }
}

Phil 12.28.15

7:00 – 5:00 VTX

  • Oliver, J. Eric, and Thomas J. Wood. “Conspiracy theories and the paranoid style (s) of mass opinion.” American Journal of Political Science 58.4 (2014): 952-966., and the Google Scholar page of papers that cite this. Looking for insight as to the items that make (a type of?) person believe false information.
  • This follows up on an On the Media show called To Your Health, that had two interesting stories: An interview with John Bohannon, who published the intentionally bad study on chocolate, and an interview with Taryn Harper Wright, a blogger who chases down cases of Munchausen by Internet, and says that excessive drama is a strong indicator of this kind of hoax.
  • Reading Social Media and Trust during the Gezi Protests in Turkey.
    • Qualitative study that proposes Social Trust and System Trust
      • Social Trust
      • System Trust
  • Hibernating Moderately
    • Working on the dictionary
    • Working on the Corpus
      • Name
      • Date created
      • Source URL
      • Raw Content
      • Cleaned Content
      • Importer
      • Word count
      • guid
    • I think I’ll need a table that has the word id that points to a corpus and gives the count of that word in that corpus. The table gets updated whenever a dictionary is run against a corpus. Since words are not shared between dictionaries (Java != Java), getting the corpus to dictionary relationship is straightforward if needed.
    • Created a GuidBase that handles the name, id, and guid code that’s shared across most of the items.
    • Discovered Jsoup, which has some nice (fast?) html parsing.
    • Finished most of Corpus. Need to add a join to users. Done
    • Added BaseDictionary.
    • Added BaseDictionaryEntry.
    • Working on getting a join table working that maps words to corpora and am getting a “WARN: SQL Error: 1062, SQLState: 23000”. I was thinking that I could create a new HashMap, but I think I may have to point to the list in a different way. Here’s the example from JustHibernate: 
              session.beginTransaction();
        Showroom showroom = new Showroom();
        showroom.setLocation("East Croydon, Greater London");
        showroom.setManager("Barry Larry");
        Set cars = new HashSet();
        
        cars.add(new Car("Toyota", "Racing Green"));
        cars.add(new Car("Nissan", "White"));
        cars.add(new Car("BMW", "Black"));
        cars.add(new Car("Mercedes", "Silver"));

        showroom.setCars(cars);
        
        session.save(showroom);
        
        session.getTransaction().commit();
    • Where the Showroom class has the Cars Set annotation as follows: 
       @OneToMany
    @JoinTable
    (name="SHOWROOM_CAR_SET_ANN_JOINTABLE",
     joinColumns = @JoinColumn(name="SHOWROOM_ID")
     )
    @Cascade(CascadeType.ALL)
    private Set cars = null;
    • Anyway, more tomorrow…
    • Start on queries that:
      • List networks for users
      • List dictionaries for users
      • List Corpora

Phil 12.24.15

7:00 – 4:00 VTX

Phil 12.23.15

7:00 – 3:00 VTX

  • Model Merging, Cross-Modal Coupling, Course Summary
    • Bayesian story merging – Mark Finlayson
    • Cross-modal coupling and the Zebra Finch – Coen
      • If items are close in one modality, maybe they should be associated in other modalities. CrossModalCoupling1
      • Good for dealing with unlabeled data that we need to make sense of
    • How You do it (Just AI?)
      • Define or describe a competence
      • Select or invent a representation
      • Understand constraints and regularities – without this, you can’t make models.
      • Select methods
      • Implement and experiment
    • Next Steps
      • 6.868 Society of Mind – Minsky
      • 6.863, 6.048 Language,  Evolution – Berwick
      • 6.945 Large Scale Symbolic Systems – Sussman
      • 6.xxx Human Intellegence Enterprise – Winston
      • Richards
      • Tenenbaum
      • Sinha
      • MIT underground guide?
  • Hibernate
    • So the way we get around joins is to explicitly differentiate the primary key columns. So where I had ‘id_index’ as a common element which I would change in the creation of the view, in hibernate we have to have the differences to begin with (or we change the attribute column?) regardless, the column names appear to have to be different in the table…
    • Here’s a good example of one-table-per-subclass that worked for me.
    • And here’s my version. First, the cfg.xml: 
      <hibernate-configuration>

    <session-factory>
        <property name="connection.url">jdbc:mysql://localhost:3306/jh</property>
        <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="connection.username">root</property>
        <property name="connection.password">edge</property>
        <property name="dialect">org.hibernate.dialect.MySQL5Dialect</property>
        <property name="hibernate.show_sql">true</property>

        <!-- Drop and re-create the database schema on startup -->
        <property name="hbm2ddl.auto">create-drop</property>

        <mapping class="com.philfeldman.mappings.Employee"/>
        <mapping class="com.philfeldman.mappings.Person"/>
    </session-factory>

</hibernate-configuration>
    • Next, the base Person Class: 
      package com.viztronix.mappings;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.Inheritance;
import javax.persistence.InheritanceType;
import javax.persistence.Table;
import java.util.UUID;


@Entity
@Table(name = "person")
@Inheritance(strategy=InheritanceType.JOINED)
public class Person {

    @Id
    @GeneratedValue
    @Column(name = "person_ID")
    private Long personId;

    @Column(name = "first_name")
    private String firstname;

    @Column(name = "last_name")
    private String lastname;

    @Column(name = "uuid")
    private String uuid;

    // Constructors and Getter/Setter methods,
    public Person(){
        UUID uuid = UUID.randomUUID();
        this.uuid = uuid.toString();
    }

    public Long getPersonId() {
        return personId;
    }

    // getters and setters...

    @Override
    public String toString(){
        return "["+personId+"/"+uuid+"]: "+firstname+" "+lastname;
    }
}
    • The inheriting Employee class: 
      package com.viztronix.mappings;

import java.util.Date;

import javax.persistence.*;

@Entity
@Table(name="employee")
@PrimaryKeyJoinColumn(name="person_ID")
public class Employee extends Person {

    @Column(name="joining_date")
    private Date joiningDate;

    @Column(name="department_name")
    private String departmentName;

    // getters and setters...

    @Override
    public String toString() {
        return super.toString()+ " "+departmentName+" hired "+joiningDate.toString();
    }
}
    • The ‘main’ program that calls the base class and subclass: 
      package com.philfeldman.mains;

import com.viztronix.mappings.Employee;
import com.viztronix.mappings.Person;
import org.hibernate.HibernateException;

import java.util.Date;


public class EmployeeTest extends BaseTest{

    public void addRandomPerson(){
        try {
            session.beginTransaction();
            Person person = new Person();
            person.setFirstname("firstname_" + this.rand.nextInt(100));
            person.setLastname("lastname_" + this.rand.nextInt(100));
            session.save(person);
            session.getTransaction().commit();
        }catch (HibernateException e){
            session.getTransaction().rollback();
        }
    }

    public void addRandomEmployee(){
        try {
            session.beginTransaction();
            Employee employee = new Employee();
            employee.setFirstname("firstname_" + this.rand.nextInt(100));
            employee.setLastname("lastname_" + this.rand.nextInt(100));
            employee.setDepartmentName("dept_" + this.rand.nextInt(100));
            employee.setJoiningDate(new Date());
            session.save(employee);
            session.getTransaction().commit();
        }catch (HibernateException e){
            session.getTransaction().rollback();
        }
    }

    public static void main(String[] args){
        try {
            boolean setupTables = false;
            EmployeeTest et = new EmployeeTest();
            et.setup("hibernateSetupTables.cfg.xml");
            //et.setup("hibernate.cfg.xml");


            for(int i = 0; i < 10; ++i) {
                et.addRandomEmployee();
                et.addRandomPerson();
            }

            et.printAllRows();

            et.closeSession();

        }catch (Exception e){
            e.printStackTrace();
        }

    }
}
    • And some output. First, from the Java code with the Hibernate SQL statements included. It’s nice to see that the same strategy that I was using for my direction db interaction is being used by Hibernate:: 
      Hibernate: alter table employee drop foreign key FK_apfulk355h3oc786vhg2jg09w
Hibernate: drop table if exists employee
Hibernate: drop table if exists person
Hibernate: create table employee (department_name varchar(255), joining_date datetime, person_ID bigint not null, primary key (person_ID))
Hibernate: create table person (person_ID bigint not null auto_increment, first_name varchar(255), last_name varchar(255), uuid varchar(255), primary key (person_ID))
Hibernate: alter table employee add index FK_apfulk355h3oc786vhg2jg09w (person_ID), add constraint FK_apfulk355h3oc786vhg2jg09w foreign key (person_ID) references person (person_ID)
Dec 23, 2015 10:40:26 AM org.hibernate.tool.hbm2ddl.SchemaExport execute
INFO: HHH000230: Schema export complete
Hibernate: insert into person (first_name, last_name, uuid) values (?, ?, ?)
... lots more inserts ...
Hibernate: insert into person (first_name, last_name, uuid) values (?, ?, ?)
There are [2] members in the set
key = [com.philfeldman.mappings.Employee]
executing: from com.philfeldman.mappings.Employee
Hibernate: select employee0_.person_ID as person1_1_, employee0_1_.first_name as first2_1_, employee0_1_.last_name as last3_1_, employee0_1_.uuid as uuid4_1_, employee0_.department_name as departme1_0_, employee0_.joining_date as joining2_0_ from employee employee0_ inner join person employee0_1_ on employee0_.person_ID=employee0_1_.person_ID
  [1/17bc0f66-da60-4935-a4d2-5d11e93e2419]: firstname_15 lastname_96 dept_7 hired Wed Dec 23 10:40:26 EST 2015
  [3/6c15103a-49b2-4b63-8ef9-0c8ab3f84eab]: firstname_30 lastname_88 dept_75 hired Wed Dec 23 10:40:26 EST 2015

key = [com.philfeldman.mappings.Person]
executing: from com.philfeldman.mappings.Person
Hibernate: select person0_.person_ID as person1_1_, person0_.first_name as first2_1_, person0_.last_name as last3_1_, person0_.uuid as uuid4_1_, person0_1_.department_name as departme1_0_, person0_1_.joining_date as joining2_0_, case when person0_1_.person_ID is not null then 1 when person0_.person_ID is not null then 0 end as clazz_ from person person0_ left outer join employee person0_1_ on person0_.person_ID=person0_1_.person_ID
  [1/17bc0f66-da60-4935-a4d2-5d11e93e2419]: firstname_15 lastname_96 dept_7 hired Wed Dec 23 10:40:26 EST 2015
  [2/3edf8d12-dbd9-42d3-893f-c740714a2461]: firstname_6 lastname_99
  [3/6c15103a-49b2-4b63-8ef9-0c8ab3f84eab]: firstname_30 lastname_88 dept_75 hired Wed Dec 23 10:40:26 EST 2015
  [4/f5bba5c6-77a7-438b-bd73-5e12288d3b2c]: firstname_91 lastname_43
  [5/75db23a9-3be3-44f5-80bf-547ab8c7f12f]: firstname_7 lastname_84 dept_36 hired Wed Dec 23 10:40:26 EST 2015
  [6/45520bb5-8d3d-4577-b487-3e45d506bf50]: firstname_22 lastname_35
  [7/c0bb18e6-6114-4e8a-a7ce-e580ddfb9108]: firstname_1 lastname_22
    • Last, here’s what was produced in the db: dbResults
    • Starting on the network data model
    • Added NetworkType(class) network_types(table)
    • Added BaseNode(class) network_nodes(table)
      • The mapping for the types in the BaseNode class looks like this (working from this tutorial): 
        @Entity
@Table(name="network_nodes")
public class BaseNode {
    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    @Column(name="node_id")
    private int id;
    private String name;
    private String guid;

    @ManyToOne
    @JoinColumn(name = "type_id")
    private NetworkType type;

    public BaseNode(){
        UUID uuid = UUID.randomUUID();
        guid = uuid.toString();
    }

    public BaseNode(String name, NetworkType type) {
        this();
        this.name = name;
        this.type = type;
    }

    //...

    @Override
    public String toString() {
        return "["+id+"]: name = "+name+", type = "+type.getName()+", guid = "+guid;
    }
}
      • No changes needed for the NetworkType class, so it’s a one-way relationship, which is what I wanted: 
        @Entity
@Table(name="network_types")
public class NetworkType {
    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    @Column(name="type_id")
    private int id;
    private String name = null;

    public NetworkType(){}

    public NetworkType(String name) {
        this.name = name;
    }

    // ...

    @Override
    public String toString() {
        return "["+id+"]: "+name;
    }
}

 

Phil 12.22.15

VTX 7:00 – 6:00

  • Probabilistic Inference II
    • Assertion – Any variable in a graph is said by me to be independent of any other non-descendant, given its parents. All the causality flows through the parents.
    • A belief net or Bays net is *always* acyclic and directed.
    • Traverse the graph from the bottom up, so that no node depends on a node to its left in a list.
    • Generating the list:BayesNetFromData
    • When using the list, work from the top down in the list
    • Naive Bayesian inference
      • P(a|b)P(b) = P(a,b) = P(b|a)P(a)
      • P(a|b) = (P(b|a)P(a))/P(b) BayesChain
      • Can use Bayes to decide between models – Naive Bayesian Classification
      • Use the sum of the logs of the probabilities rather than the products because otherwise we run out of bits of precision
    • The right thing to do when you don’t know anything (just have symptoms)
  • Hibernate
    • Adding config.setProperty(“hbm2ddl.auto”, “update”); to the setup, so that tables can be rebuilt on demand. Nope, that didn’t work. Maybe I can’t split configuration between the config file and programmatic variables?
    • The only way that I was able to get this to work as an argument was to have a setupTables flag indicate which config to read. That works well though.
    • Got simple collections running, which means that I should be able to get networks built. Basically modified the example from Just Hibernate that starts on page 53.
    • Next, we work on getting inheritance to work. I think this will help.
  • Initial Java class network thoughts, just to try storing and retrieving items
    • BaseItem
      • guid
    •  BaseNode extends BaseItem
      • node_id
      • name
    • BaseEdge extends BaseItem
      • edge_id
      • source
      • target
      • weight
    • BaseNetwork extends BaseItem
      • network_id
      • name
      • owner
      • edgeList
      • nodeList (we need this because we may have orphans in the network)
    • BaseOwner extends BaseItem
      • owner_id
      • name
      • password?

Phil 12.21.15

8:00 – 6:00 VTX

  • No MIT video today. Went out and saw Star Wars. Fun! Need to see it again when the crowds thin out in an IMAX theater.
  • Copied some stunt data into the hibernate_test db.
  • Ran the code that set up the session and connected to the (empty) db. No exceptions, so I think it’s working this time…
  • IDE is tracking annotations. The names in the annotation class need to be the same as the table and element names or there is an error IntelliJ Hibernate Setup
  • Ok, reading and writing into the db. Now to clean it up and separate elements;
  • Here’s the current cleaned up version. Still need to create the table more properly. 
    package com.philfeldman.mains;

import com.philfeldman.mappings.Employee;
import org.hibernate.HibernateException;
import org.hibernate.Query;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.cfg.Configuration;
import org.hibernate.metadata.ClassMetadata;
import org.hibernate.service.ServiceRegistry;
import org.hibernate.service.ServiceRegistryBuilder;

import java.util.Map;
import java.util.Random;

/**
 * Created by philip.feldman on 12/21/2015.
 *
 * A simple test program that will read and write from a table in a database. In MySql, the
 * table is in the form:
         CREATE TABLE employee (
         id int(11) NOT NULL AUTO_INCREMENT,
         name varchar(50) DEFAULT NULL,
         PRIMARY KEY (id)
 ) ;
 */
public class EmployeeTest {
    private SessionFactory sessionFactory;
    private ServiceRegistry serviceRegistry;
    private Session session;
    private Random rand;

    public EmployeeTest()throws ExceptionInInitializerError{
        this.rand = new Random();
        try {
            Configuration config = new Configuration();
            config.configure("hibernate.cfg.xml");

            this.serviceRegistry = new ServiceRegistryBuilder().applySettings(config.getProperties()).buildServiceRegistry();
            this.sessionFactory = config.buildSessionFactory(serviceRegistry);
            this.session = this.sessionFactory.openSession();
        } catch (Throwable ex) {
            throw new ExceptionInInitializerError(ex);
        }
    }

    public void closeSession(){
        this.session.close();
    }

    public void printAllEntityNames(){
        System.out.println("querying all the managed entities...");
        final Map metadataMap = this.session.getSessionFactory().getAllClassMetadata();
        System.out.println("There are [" + metadataMap.keySet().size() + "] members in the set");
        for (Object key : metadataMap.keySet()) {
            System.out.println("key = ["+key.toString()+"]");
        }
    }

    public void printAllEmployees(){
        String key = Employee.class.getCanonicalName();
        final Map metadataMap = this.session.getSessionFactory().getAllClassMetadata();
        final ClassMetadata classMetadata = (ClassMetadata) metadataMap.get(key);
        final String entityName = classMetadata.getEntityName();
        final Query query = session.createQuery("from " + entityName);
        System.out.println("executing: " + query.getQueryString());
        for (Object o : query.list()) {
            Employee e = (Employee) o;
            System.out.println("  " + e.toString());
        }
    }

    public void addRandomEmployee(){
        try {
            session.beginTransaction();
            Employee employee = new Employee();
            employee.setName("rand(" + this.rand.nextInt(100) + ")");
            session.save(employee);
            session.getTransaction().commit();
        }catch (HibernateException e){
            session.getTransaction().rollback();
        }
    }

    public static void main(String[] args){
        try {
            //System.out.println("Employee.class.getCanonicalName: "+Employee.class.getCanonicalName());
            /***/
            EmployeeTest et = new EmployeeTest();
            et.printAllEntityNames();
            et.printAllEmployees();
            et.addRandomEmployee();
            et.closeSession();
             /***/
        }catch (Exception e){
            e.printStackTrace();
        }

    }
}

Phil 12.18.15

7:00 – 5:00 VTX

  • Was listening to the Planet Money podcast on A/B testing last night and they mentioned how they were using the ‘skip’ button to determine how to shape their podcast. So this is a feedback device that people use that has at most a very indirect effect on the relevance of the provided item, but it does provide the system with a value judgement from the consumer. The benefit to the user is the ability to skip content, and that appears to be enough. The benefit to the producer is the aggregate responses of the users (40k in this place, so lots of statistical power). Somewhat related:
  • And I thought of a title that describes the focus of this effort: Using Value-Decorated Semantic Nets to Infer Credibility
  • Probabilistic Inference I
    • Joint probability tables are the ideal, but impractical
    • Basic probability (intuition at 21:00)
      • probability 0 … 1
      • True = 1
      • False = 0
      • P(a) + P(b) – P(a, b) = P(a or b)
    • Conditional probability
      • Definitions
      • P(a|b) = P(a,b)/P(b)
        • Probability of a given be is the probability of a AND b over the probability of b (24:00)
      • P(a, b, c) = ??
        • y = b, c
        • P(a, b, c) = P(a, y) = P(a|y)P(y)
        • = P(a|b,c)P(b,c)
        • = P(a|b,c)P(b|c)P(c) note that as we go from left to right, there is less elements to depend on.
      • Generalized
        • (Px1, …, xn) = chain rule (starts at 28:31)chainedConditionalProbability
    • Independence
      • Definitions
      • P(a|b) = P(a) if a independent of b – video at 32:30
        • The probability of a in the universe is the same as the probability of a and b in b. The two rations are the same. Why is this definition needed?
      • Conditional independence
      • P(a|b,z) = P(z)
      • P(a,b|z) = P(a|z)P((b|z)
    • Belief Nets
      • Causal relationships. The dog barks because of the Racoon
      • Every node is dependent only on its parent(s) and possibly its children (descendents)
      • BeliefNets If this were a joint probability table there would be 2^5 (32) as opposed to the number here, which is 10.
      • P(p,d,b,t,r) = P(p|d,b,t,r)…P(r), which we can reduce the combinations. (See 46:30 or so)
  • Hibernating slowly
    • The ./basic/ as described in the hibernate 5 quickstart doesn’t seem to exist in either the 4.3 or the 5.5 bubndle. It does look like IntelliJ has a JPA and Hibernate section. Trying that.
    • Importing the current pg db, which did get found since I had already set up that relationship with database in yesterday’s post.
    • In the Import Database Schema wizard, I had to create a package for the files to be created in. In this case, since I’ve already had to create a new module under HibernateTest1 (HibernateTest1Module1), I called the package com.philfeldman.ht1m1, which is similar to the Entity prefix of ht1m1_ that I decided to add.
    • Got a ‘Basic’ attribute type should not be ‘Object’ error. When opening up the ‘weather’ element in the dialog (see below) ImportDatabaseSchema I could see that the tempHi and tempLo items are being mapped as Objects. Typing java.Lang.Integer corrects the problem. The thing to remember here is that the error doesn’t ripple up. When ‘weather’ is closed, there are no red items.
    • That worked, but there were some significant compiler errors. Fixed by letting the IDE download java EE6 libraries. It still looks like we’re using java 1.8, but now have a bunch of External libraries that appear to be redundant? LotsaLibs.jpg
    • Anyway, using the persistence view, created a ht1m1_UserEntity class. Now I need to make it persist and add values to it. for that matter, I need to query the weather table…
    • Haven’t gotten to accessing data yet, but you can set up relationships graphically in IntelliJ, which is pretty cool.
    • And now I’m kind of stuck. The console interface with the hibernate/db keeps on asking for a persistence provider which seems to be in the classpath but doesn’t seem to be helping.
  • Starting over
    • Spent a few bucks and got Just Hibernate. Let’s see if that works better.
    • Need to install Git – Done. Yay!
    • Created JustHibernate1 as a JavaEE project with Hibernate and the default download libraries (4.2.2). Also created a corresponding hibernate_test database in MySql. Nothing in it yet.
    • Opened up the database view and connected to my MySql database. This gives me the opportunity to (a) test the connection and (b) get the URL for the hibernate.cfg.xml file (jdbc:mysql://localhost:3306/hibernate_test)
    • Still needed to get the jdbc driver, so I used the Project Structure pane (F4) to import the mysql:mysql-connector-java:5.1.38 from maven. IntelliJ downloaded and stuck it in the lib directory. Here’s the module structureProjectStructureModules And here’s the library structure for the mysql driver. Note that it’s actually pointing at my m2 repo…ProjectStructureLibraries
    • So now I’m about where I was at lunch, but everything is cleaner. Afraid to actually try connecting at 5:00 on a Friday, so we’ll try this on Monday <fingers  crossed>

Phil 12.17.15

7:00 – 4:00 VTX

  • Architectures: GPS, SOAR, Subsumption, Society of Mind
    • GPS – General Problem Solver Newell & Simon
    • SOAR – State Operator and Result. (RCS for problem solving + GOMS?)
    • Emotion Machine – Minsky  Multiple Levels
      • Instinctive reaction
      • Learned reaction
      • Deliberative thinking
      • Reflective thinking – Memory
      • Self-reflecting (planning?)
      • Self-conscious thinking (social interaction)
    • Based on the Common Sense Hypothesis
      • Open Mind Concept
      • Henry Lieberman
      • Media Lab
    • Alternative Ideas
      • Rodney Brooks – Subsumption architecture
        • Creature Hypothesis – once you can get a machine to be as smart as an insect, the rest will be easy. (Very RCS!)
        •  Layers of abstraction, each with its own Vision, Reasoning and Action layers.
          • Avoiding Object Layer
          • Wandering Layer
          • Explore Layer
          • Seek Layer
          • Etc.
        • Rules
          • No representation (no world model)
          • Use the world instead of a model. Everything is reactive.
          • Finite State Machines
        • Roomba is an example.
      • Genisys System
        • Strong Story Hypothesis
          • White room experiment (described in video here)
            • Children begin to orient correctly after they start using the words ‘left’ and ‘right’ when they describe the world.
            • Adults doing ‘english to english translation’ they fail the test.
            • Also in a radiolab show: Words
        • Based on language
          • Perception (Real and imagined [running with a bucket of water])
          • Description of events
            • Stories
            • Culture
              • Macro
              • Micro
  • Did a little poking around with hibernate, since Jeremy says that Hibernate plus annotations are the standard here. It does look like 4.3.8 final is the version that’s being used (4.3.11 is close enough?) with jpa annotations. Jeremy’s also been using Spring Data JPA, which I guess needs to be on the list as well.
  • Debating on whether I should set up a Hibernate sandbox with Gradle, but I think that’s a bridge too far.
  • Oh yeah, when you check out a project in subversion, check it out at its trunk node. Otherwise Gradle doesn’t know what to do. It also seems to be downloading everything again as I import the project. I wonder if this will take 41 minutes again?
    • You can then run by clicking on src/main/java/com.philfeldman.nlpservice/web/Application.
    • Verified that everything works by sending json object to localhost:8870/nlpservice/analyze in Postman: Postman
  • Ok, back to setting up a sandbox for schema development
    • Downloading and installing Postgresql, version 9.4.5
    • The install kind of broke and didn’t create the data files. I wound up doing the Short Version from the command line, which is working just fine.
    • To start the db server – C:\Program Files\PostgreSQL\9.4\bin>postgres.exe -D \Development\PostGresSQL\Data
    • To run the client – C:\Program Files\PostgreSQL\9.4\bin>psql test
    • Set up shortcuts that launch the server and the test db following these instructions.
  • Starting the Hibernate sandbox project.
    • Had to enable the the hibernate IntelliJ plugin
    • connected IntelliJ to the postgres db using the Database View. . I thought the superuser name was ‘postgres’, but \du says it’s ‘philip.feldman’. It must have pulled that from the OS. Password was what I thought I set it to though.
    • In a fit of unrealistic expectation, decided to start with the latest hibernate Version 5.5.1.Final. The jar structure is really different from 4.3.11.Final, but we’ll see how that goes. Using the Hibernate 5.0 quickstart

Phil 12.16.15

7:00 – 9:00, 10:30 – 4:30 VTX

  • Since I’ll be missing the scrum today, sent Aaron an email with status. Which is basically until I know if we’re going to have a semantic network for our derived data, I don’t know how to do a taxonomy.
  • Got RabbitMQ running, following the Local RabbitMQ Setup in Confluence. To open a command prompt as full admin, you have to run it from the ‘start’ input field with Ctrl-shift-enter
  • Running the NLPService with errors. Doesn’t seem to be a permissions issue. Sent Balaji an email, but here are the errors for future reference: 
    2015-12-16 08:25:24.449 ERROR 3588 --- [pool-8-thread-1] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_NLPSERVICE/PFELDMAN-NCS - was unable to sen
 heartbeat!

com.sun.jersey.api.client.ClientHandlerException: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184)
        at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:120)
        at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:28)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.put(WebResource.java:211)
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1097)
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060)
        at com.netflix.discovery.DiscoveryClient.access$500(DiscoveryClient.java:105)
        at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1583)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170)
        ... 14 common frames omitted
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 21 common frames omitted


2015-12-16 08:25:26,620 ERROR [pool-9-thread-1] com.netflix.discovery.DiscoveryClient [nlp-service-local] Can't get a response from http://localhost:8761/eurek
/apps/
Can't contact any eureka nodes - possibly a security group issue?
com.sun.jersey.api.client.ClientHandlerException: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184) ~[jersey-apache-client4-1.11.jar!/:1.11]
        at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:120) ~[jersey-client-1.11.jar!/:1.11]
        at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:28) ~[eureka-client-1.1.147.jar!/:1.1.147]
        at com.sun.jersey.api.client.Client.handle(Client.java:648) ~[jersey-client-1.11.jar!/:1.11]
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) ~[jersey-client-1.11.jar!/:1.11]
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.11.jar!/:1.11]
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503) ~[jersey-client-1.11.jar!/:1.11]
        at com.netflix.discovery.DiscoveryClient.getUrl(DiscoveryClient.java:1567) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1113) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient.getAndStoreFullRegistry(DiscoveryClient.java:835) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:746) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient.access$1400(DiscoveryClient.java:105) [eureka-client-1.1.147.jar!/:1.1.147]
        at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1723) [eureka-client-1.1.147.jar!/:1.1.147]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_66]
        at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_66]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_66]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_66]
        at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827) ~[httpclient-4.2.1.jar!/:4.2.1]
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170) ~[jersey-apache-client4-1.11.jar!/:1.11]
        ... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_66]
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[na:1.8.0_66]
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[na:1.8.0_66]
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[na:1.8.0_66]
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[na:1.8.0_66]
        at java.net.PlainSocketImpl.connect(Unknown Source) ~[na:1.8.0_66]
        at java.net.SocksSocketImpl.connect(Unknown Source) ~[na:1.8.0_66]
        at java.net.Socket.connect(Unknown Source) ~[na:1.8.0_66]
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) ~[httpclient-4.2.1.jar!/:4.2.1]
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) ~[httpclient-4.2.1.jar!/:4.2.1]
        ... 25 common frames omitted

2015-12-16 08:25:26.652 ERROR 3588 --- [pool-9-thread-1] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_NLPSERVICE/PFELDMAN-NCS - was unable to ref
esh its cache! status = org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused

com.sun.jersey.api.client.ClientHandlerException: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184)
        at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:120)
        at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:28)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
        at com.netflix.discovery.DiscoveryClient.getUrl(DiscoveryClient.java:1567)
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1113)
        at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060)
        at com.netflix.discovery.DiscoveryClient.getAndStoreFullRegistry(DiscoveryClient.java:835)
        at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:746)
        at com.netflix.discovery.DiscoveryClient.access$1400(DiscoveryClient.java:105)
        at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1723)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8761 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
        at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170)
        ... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 25 common frames omitted
  • It turns out that these errors are related to service registering with the discovery process. In local environment you don’t have a service registry running. You can run it if you want and that’s a different process(project). Irrespective of those service trying to register errors, the REST calls will still work.
    • The way to remove the service registration (remove the noise errors) in local env is, in src/main/resources/nlpservice-config.xml, change this
    • <serviceRegistry>http://localhost:8761/eureka/</serviceRegistry&gt;
    • to
    • <serviceRegistry>none</serviceRegistry>
    • Do the gradle build again and run it.
    • And a useful thread: 
      Patakula, Balaji 11:44a 
Hello Phil

Me 11:44a 
Hiya

Patakula, Balaji 11:45a 
the url should be localhost:8870/nlpservice/ner

Patakula, Balaji 11:45a 
with no double slash after the port

Me 11:46a 
localhost:8870/nlpservice/ner gives the same error in my setup

Patakula, Balaji 11:46a 
also the body should be like { "text": "my name is Phil Feldman."}

Patakula, Balaji 11:46a 
or any text that u want

Me 11:47a 
So it's not the JSON object on the NLPService page?


Patakula, Balaji 11:48a 
if u import the nlp.json into the postman

Patakula, Balaji 11:48a 
all the requests will be already there

Me 11:49a 
Import how?
Patakula, Balaji 11:49a 
there is an import menu on postman

Me 11:50a 
Looking for it...

Patakula, Balaji 11:50a 
the middle panel on the top black menu last item

Me 11:50a 
Got it.

Patakula, Balaji 11:51a 
just import that json downloaded from the wiki

Patakula, Balaji 11:51a 
and u should have the collection now in ostman 

Patakula, Balaji 11:51a 
postman 



Patakula, Balaji 11:51a 
and u can just click to send

Me 11:52a 
Added the file. Now what.

Patakula, Balaji 11:52a 
can u share the screen

Me 11:52a 
using what?

Patakula, Balaji 11:52a 
u have the nlp service running?

Patakula, Balaji 11:53a 
just in IM 

Patakula, Balaji 11:53a 
there is a present screen on the bottom of this chat

Me 11:53a 
nlp service is running. It extracted my entity as well. Now I'm curious about that bigger json file

Patakula, Balaji 11:54a 
that json file is just the REST calls that are supported by the service

Patakula, Balaji 11:54a 
it is just a way of documenting the REST 

Patakula, Balaji 11:54a 
so some one can just import the file and execute the commands

Me 11:54a 
So how does it get ingested?

Patakula, Balaji 11:55a 
which one?

Me 11:55a 
nlp.json

Patakula, Balaji 11:55a 
its not ingested. NLP is service. It gets the requests through Rabbit queue from the Crawler ( another service)

Patakula, Balaji 11:56a 
if u need to test the functionality of NLP, the way u can test and see the results is using the REST interface that we are doing now

Me 11:56a 
so nlp.json is a configuration file for postman?

Patakula, Balaji 11:56a 
thats right

Me 11:57a 
Ah. Not obvious.

Patakula, Balaji 11:57a 
Is Aaron sit next to you?

Me 11:58a 
No, he stepped out for a moment. He should be back in 30 minutes or so.

Patakula, Balaji 11:58a 
may be u can get the data flow from him and he knows how to work with all these tools

Me 11:58a 
Yeah, he introduced me to Postman.

Me 11:58a 
But he thought nlp.json was something to send to the NLPService.

Patakula, Balaji 11:58a 
may be he can give a brain dump of the stuff and how services interact, how data flows etc.,

Me 11:59a 
I'm starting to see how it works. Was not expecting to see Erlang.

Me 12:00p 
Can RabbitMQ coordinate services under development on my machine with services stood up on a test environment, such as AWS?

Patakula, Balaji 12:01p 
u can document all the REST calls that a service exposes by hand writing all those ...or just export the REST calls from postman and every one who wants to use the service can just import that json and work with the REST interface

Me 12:01p 
Got it.

Patakula, Balaji 12:01p 
RabbitMq is written in Erlang and we interface with it for messaging

Patakula, Balaji 12:02p 
yes, u can configure the routes to work that way

Patakula, Balaji 12:02p 
meaning mismatch services between different environments

Me 12:02p 
Yeah, I see that. Not that surprising that a communications manager would be written in Erlang. But still a rare thing to see.

Me 12:03p 
Is there a collection of services stood up that way for development?

Patakula, Balaji 12:04p 
u installed rabbit yesterday locally on ur machine

Me 12:04p 
Yes, otherwise none of this would be working?

Patakula, Balaji 12:04p 
so u can run various services now orchestrated through ur local rabbit



Me 12:05p 
Understood. Are there currently stood-up services that can be accessed on an ad-hoc basis, or would I need to do that?

Patakula, Balaji 12:05p 
Rabbit is only for streaming messages. Every service exposes both streaming ( Rabbitmq messages) and REST interfaces

Patakula, Balaji 12:06p 
there are no services stood up in adhoc env currently. There is a CI ,QA and Demo env

Patakula, Balaji 12:06p 
all those envs have all the services running

Me 12:07p 
What's Cl?

Me 12:07p 
I'd guess continuous integration, but it's ambiguous.

Patakula, Balaji 12:07p 
continuous integration. Every code checkin automatically builds the system, runs the tests, creates docker images and deploys those services and starts them 

Me 12:08p 
Can these CI services be pinged directly?

Patakula, Balaji 12:08p 
ye

Patakula, Balaji 12:08p 
yes

Me 12:09p 
Do you need to be on the VPN?

Patakula, Balaji 12:09p 
http://dockerapps.philfeldman.com:8763/ <http://dockerapps.philfeldman.com:8763/>  

Patakula, Balaji 12:09p 
those are the services running

Patakula, Balaji 12:09p 
and dockerapps is the host machine for CI

Me 12:09p 
And how do I access the NLPService on dockerapps?

Patakula, Balaji 12:10p 
access meaning? u want t send the REST requests to CI service?

Me 12:10p 
Yeah. Bad form?

Patakula, Balaji 12:11p 
just in the REST, change the localhost to dockerapps.philfeldman.com

Me 12:12p 
I get a 'Could not get any response'

Me 12:12p 
dockerapps.philfeldman.com:8870/nlpservice/ner

Patakula, Balaji 12:12p 
sorry, NLP is running on a different host 10.18.7.177
  • Learning about RabbitMQ
  • Installing the google chrome Postman plugin
    • Set the POST option
    • Set RAW
    • Header to Content-Type
    • Value to application/json
    • URL is localhost:8870//nlpservice/ner
    • place the JSON in the ‘Body’ tag

Phil 12.15.15

7:00 – 3:30 VTX

  • Representations: Classes, Trajectories, Transitions
    • Inner language, the language with which we think
    • Semantic nets
      • parasitic semantics – where we project knowing to the machine. We contain the meaning, not the machine.
    • Combinators = edge
    • Reification – linking links?
    • Sequence
    • Minsky – Frames or templates add a localization layer.
    • Classification
    • Transition
      • Vocabulary of change, not state
      • (!)Increase, (!)decrease, (!)change, (!)appear, (!)disappear
    • Trajectory
      • Objects moving along trajectories
      • Trajectory frame (prepositions help refine – by, with, from, for, etc)
        • Starts at a source
        • Arranged by agent, possibly with collaborator
        • assisted by instrument
        • can have a conveyance
        • Arrives at destination
        • Beneficiary
      • Wall Street Journal Corpus
        • 25% transitions or trajectories.
      • Pat comforted Chris
        • Role Frame
          • Agent: Pat
          • Action: ??
          • Object: Chris
          • Result: Transition Frame
            • Object: Chris
            • Mood: Improved (increased)
    • Story Libraries
      • Event Frames – adds time and place
        • Disaster -adds fatalities, cost
          • Earthquake – adds name, category
          • Hurricane – – adds magnitude, fault
        • Party
          • Birthday
          • Wedding – adds bride and groom
  • Scrum
  • Working on downloading and running the NLP code
    • Downloaded Java EE 7u2
    • Downloaded Gradle 2.9
    • Installed and compiled. Took 41 minutes!
    • Working on running it now, which looks like I need Tomcat. To run Tomcat on port 80, I had to finally chase down what was blocking port 80. I finally found it by running NET stop HTTP, (from here) which gave me a list that I could check against the services. I monitored this with Xampp’s nifty Netstat tool. The offending process was BranchCache, which I disabled. Now we’ll see what that breaks…
    • Tomcat up and running
    • NLPService blew up. More secret knowledge: 
      Local RabbitMQ Setup

Install Erlang 

# http://www.erlang.org/download/otp_win64_17.5.exe

# Set *ERLANG_HOME* in system variables. (e.g. C:\Program Files\erl6.4)

Install RabbitMQ 

# http://www.rabbitmq.com/releases/rabbitmq-server/v3.5.3/rabbitmq-server-3.5.3.exe

#* If you get Windows Security Alert(s) for *epmd.exe* and/or *erl.exe*, check "Domain networks..." and uncheck "Private networks" and "Public networks"

# Open the command prompt as *administrator*

# Go to C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.5.3\sbin.

# Run the following commands:             

rabbitmq-plugins.bat enable rabbitmq_web_stomp rabbitmq_stomp rabbitmq_management

rabbitmq-service.bat stop                                                        
rabbitmq-service.bat install                                                     
rabbitmq-service.bat start                                                      

RabbitMQ Admin Console
http://localhost:15672/mgmt

guest/guest
    • Installed Erlang and RabbitMQ. We’ll try running tomorrow.

Phil 12.14.15

7:00 – 3:30 VTX

  • Learning: Boosting
    • Binary classifications
    • Weak Classifier = one that is barely better than chance.
    • Adaboost for credibility analysis? Politifact is the test. Speakers, subjects, etc are classifiers. What mix of classifiers produces the most accurate news? Something like this (check citations in the paper)
    • Which means that we can keep track of those items that are always moved to the top of the pertinence list and score them as true(?). This means that we can then use that result to weight the sources that appear to be credible so that they in turn become more relevant (we can also look at the taxonomy terms that get maximized and minimized) the next query.
  • Discussion with Jeremy about the RDB schemas
  • Scrum – really short
  • RDB design meeting. Lots of discussion about data sources but nothing clear. Jeremy didn’t like the unoptimized storage of the general model
  • Followon discussions with Jeremy. I showed him how unions can fix his concerns. He adjusted the schema, but I can’t get on the VPN at home for some reason. Will see tomorrow.

Phil 12.11.15

8:00 – 5:00 VTX

  • No AI course this morning, had to drop off the car.
  • Some preliminary discussions about sprint planning with Aaron yesterday. Aside from the getting the two ‘Derived’ database structures reconciled, I need to think about a few things:
    • who the network ‘users’ are. I think it could be VTX, or the system customers, like Aetna.
    • What kinds of networks exist?
      • Each individual doctor is a network of doctors, keywords, entities, sources, threats and ratings. That can certainly run on the browser
      • Then there is the larger network of ‘relevant’ doctors. That’s a larger network, certainly in the 10s – 100s range. On the lower end of the scale that could be done directly in the browser. For larger networks, we might have to use the GPU? Which seems very doable, via Steve Sanderson.
      • Then there is the master ranking, which should be something like most threatening to least threatening, probably. Queries with additional parameters pull a subset of the ordered data (SELECT foo, bar from ?? ORDER BY eigenvalue). Interestingly, according to this IEEE article from 2010, GPU processing  was handling 10 million nodes in about 30 seconds using optimized sparse matrix (SpMV) calculations. So it’s conceivable that calculations could be done in real time.
  • More documentation
  • More discussions wit Aaron about where data lives and how it’s structured.
  • Sprint planning

Phil 12.10.15

7:00 – 3:30 VTX

  • Sandy Spring Bank!
  • Honda!
  • Learning: Support Vector Machines
    • More sophisticated decision bounding, with fewer ad hoc choices than GAs and NNs
    • A positive sample must have a dot product with the ‘nomal vector’ that is >= 1.0. Similarly, a negative sample mus be <= -1.0.
    • Gotta minimize with constraints: Lagrange Multipliers from Multivariable Calculus
    • Guaranteed no local maxima
  • System Description (putting it up here)