Monthly Archives: February 2016

Phil 2.4.16

7:00 – 4:00 VTX

The way to handle multidimensional (human) ranking of documents (i.e. web pages) is to take the dimensions and and webpages and put them on a matrix? Each page has a greater or lesser score on that dimension. Then apply page rank. Tweak weights until pages order the way we think they should
Does “authority” mean quality? predicting expert quality ratings of Web documents
LandScan (Oak Ridge Labs)
Uppsala Conflict Data Program Geo-referenced Event Dataset
Nils Weidmann Dataverse (University of Konstanz)
Continuing On the Accuracy of Media-based Conflict Event Data. Done. Wow. And look at all the databases ^^^ !
Microsoft bot API
- Creating a generic bot
Back to GoogleHacking
- Added ‘CredEngine1’ as BASELINE search engine
- Looks like we blew through our limits. Using my key. Verified that the BASELINE search runs. That does mean that the current 4 queries factor out to 24 searches (6 search engines * 4 queries)
- Building search persistent object
- Building result item object. Actually, building a JasonLoadable base class since this trick is going to be used for the query items and info object
- Need a result info object that stores the meta information.
- Just stumbled across a GCS twitter search. Neat.
- Hitting the CSE and getting results. Tomorrow I’ll finish of the classes that will persist the search results. I’ve got a buffered search result to use instead of hitting google. Although it will still need to pull down the document referenced in the result. I wonder how Jsoup handles pdf and Word documents?

Phil 2.3.16

7:00 – 3:00 VTX

Just discovered Publius – a Web publishing system that is highly resistant to censorship and provides publishers with a high degree of anonymity. No longer active, but produced a paper.
Continuing On the Accuracy of Media-based Conflict Event Data. Currently starting Matching Media-based Conflict Reports with Military Records

Back to Googlehacking

Since I’ve got the provider JSON, setting up objects that I can use for more in-depth parsing. Thinking that this could be an example of ‘code’ in the dictionary. A work can be an object that knows how to look through a section of text to see if it can find itself.
I think running several dictionaries over a document could be interesting. For example, using a medical and a legal dictionary on a document would let the system infer malpractice as opposed to a document on foreign aid.

Generating the right queries and they work in the browser:

"Ram Singh"
	ALL_GOV(sanctions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+sanctions
	ALL_GOV(criminal): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+criminal
	ALL_GOV(malpractice): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+malpractice
	ALL_GOV(board actions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+board+actions
	ALL_US(sanctions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:9qwxkhnqoi0&q=%22Ram+Singh%22+VA+sanctions
	ALL_US(criminal): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:9qwxkhnqoi0&q=%22Ram+Singh%22+VA+criminal
	ALL_US(malpractice): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:9qwxkhnqoi0&q=%22Ram+Singh%22+VA+malpractice
	ALL_US(board actions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:9qwxkhnqoi0&q=%22Ram+Singh%22+VA+board+actions
	ALL_ORG(sanctions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:ux1lfnmx3ou&q=%22Ram+Singh%22+VA+sanctions
	ALL_ORG(criminal): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:ux1lfnmx3ou&q=%22Ram+Singh%22+VA+criminal
	ALL_ORG(malpractice): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:ux1lfnmx3ou&q=%22Ram+Singh%22+VA+malpractice
	ALL_ORG(board actions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:ux1lfnmx3ou&q=%22Ram+Singh%22+VA+board+actions
	RESTRICTED_COM(sanctions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:swl1wknfxia&q=%22Ram+Singh%22+VA+sanctions
	RESTRICTED_COM(criminal): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:swl1wknfxia&q=%22Ram+Singh%22+VA+criminal
	RESTRICTED_COM(malpractice): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:swl1wknfxia&q=%22Ram+Singh%22+VA+malpractice
	RESTRICTED_COM(board actions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:swl1wknfxia&q=%22Ram+Singh%22+VA+board+actions
	ALL_EDU(sanctions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+sanctions
	ALL_EDU(criminal): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+criminal
	ALL_EDU(malpractice): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+malpractice
	ALL_EDU(board actions): https://www.googleapis.com/customsearch/v1?key=AIzaSyAj6wa-zWuNWXrjeJ4FteuBMKj92mRP4vo&cx=017379340413921634422:lqt7ih7tgci&q=%22Ram+Singh%22+VA+board+actions

So the next thing is to start running these queries and looking at the results to see if there are patterns. And I would be further along, but IntelliJ choked when I tried to add JPA. After flailing for a while I just gave up, created a new project, copied all the lib src and persistence directories over, updated the structure, and it all works. Grumble grumble.

Phil 2/2/16

7:00 –

My Google Developer Dashboard. I keep forgetting how to find this thing.
Starting On the Accuracy of Media-based Conflict Event Data. One quick note – these datasets could easily go into GLOBE. Wouldn’t that be interesting…
Jacob N. Shapiro Associate Professor of Politics and International Affairs
Ok, back to entity extraction. Trying the Google CSE
- You have to include the www for healthgrades.com. Huh.
- added linkedin and top npi to exclude
- It definitely works better than the ‘-site’ version of google search
- Trying it with a rest interface. Google’s documentation
- My key was wrong. Using the pro key
  - All us: cx=017379340413921634422:9qwxkhnqoi0 – verified
  - All com (no healthgrades) : cx=017379340413921634422:swl1wknfxia – verified
  - All gov : cx=017379340413921634422:lqt7ih7tgci – verified
- Going to build a small test app that produces a list of providers and domains that we can look through in more detail and test out different analytics
  - Get back links?
  - Jsoup + NLP
  - Change over time
- Was just getting started on loading all the providers from JSON, when my IntelliJ stopped inserting breakpoints. Installing a new version

Phil 2.1.16

9:00 – 4:00VTX

Seminar today from 10:00 – 12:00. Need to send Coleen an email asking how to charge that. Done
Need to write a brief description of each coding term. How convenient! Atlas allows you to edit code description and then changes the icon. I am liking this package…
Back to reading about interrogation. Done. Not directly related to what I’m doing but still interesting was the section on the Scharff technique
Adding the Armed Conflict & Event Data Project User’s Manual. A nice example of good coding and definitions, I think. Also pointed to On the Accuracy of Media-based Conflict Event Data, which looks like a must-read.
Ok, Let’s get back to better searches
Looked at common crawl and the common crawl index some more. I’m worried that it misses smaller targets, as philfeldman.com doesn’t show, and that’s been up for years. We’ll come back to that later if I can’t make Google place nicer?
Playing with the google search API(s)
- This lovely example (from Google yet) seems to provide everything you need in JSON. Even without a key…
- Their example: https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Paris%20Hilton&userip=USERS-IP-ADDRESS
- A version that excludes all .com sites and irs.gov: https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=1040+-site%3A*.com+-site%3Airs.gov
- Wow – you can get back links
- Many options, including start and num. Num doesn’t seem to work in JSON, but start does (the first value is zero). So you seem to be limited to 4 returns at a time?
- Same query starting at the 20th result
- Looks like a complete list of operators
So now I’m going to try getting better provider queries
- https://www.google.com/search?safe=off&q=ramh+singh+malpractice+-site%3A+healthgrades.com
- This kinda works. It seems to exclude a lot more than I was expecting. healthgrades is gone, but so is a bunch of other sites like doctorwiki.com
- Regions work though: https://www.google.com/search?q=%22ram+singh%22+malpractice+site%3A.org&cr=countryUS
When Aaron is in tomorrow, I’ll ask him how the CSE/JSON integration works, and where to get ids. I got one from Google, but is sure doesn’t look right.

viztales

Dimension reduction, State, Orientation, and Speed

Monthly Archives: February 2016

Phil 2.4.16

Phil 2.3.16

Phil 2/2/16

Phil 2.1.16