Phil 3.16.16

7:30 – 5:00 VTX

  • Write up some thoughts on Comps. What synthesis is, how study design doesn’t stop with the design, matrix of themes and papers, nuggets more than notes, etc.
  • Andy Field has a new stats book out. It looks nice: Discovering Statistics – The Reality Enigma
  • Starting on Presenting Diverse Political Opinions: How and How Much
  • Conference
  • Keynote – Pia Borlund Reflections on interactive IR evaluation issues.
    • What data is required to answer the research question
    • Which methods can provide the required data
    • Types of users to test
    • Data collection to use
    • Number of participants
    • Tasks to use in testing
    • Time constraints
    • Tailoring the work task situations according to requirements and inclusion of genuine information need that they bring with them.
    • Rotation and counterbalancing of search tasks and systems (remove confounding variables) e.g. neutralize
      • System knowledge
      • topical knowledge
      • fatigue
    • Protocols
    • Tutorials,
    • Pilot testing
    • Test Design
      • Define the purpose of the study
      • formulate the questions
      • Design the study accordingly
        • Pre search questionnaire
        • Simulated work task
        • Transaction logging
        • Post-search interview
        • Observation
    • A protocol (can be 50 steps including things like turning on the test computer, cleaning the keyboard, etc)
      • Is a step-by step description of the overall study procedure
      • Is a checklist to the investigator
      • Ensures consistency in the conduction of the study
      • Ensures that all participants get the same information
      • and….
      • Before testing section
      • Testing section
      • After testing section
    • Tutorials
      • Introduction
      • Tasks expectation
      • Have the user explain what they are supposed to do
    • Pilot testing
      • What questions the participants ask
      • Does the protocol work? Missing steps, wrong order
      • How long the study takes
      • Technical problems
      • that the required data is collected
      • To practice the control of pleasing effect and human nature
      • Continue doing the pilot test until the the protocol is sufficient for the study. This means being consistent in explaining the procedure. Maybe use a video instead?
      • Don’t do the study too close to the pilot, since lessons learned need to be incorporated.
    • Everyday life information seeking: Approaching information seeking in the context of “way of life”
    • Discussion
      • The amount of dedication in the search makes a difference. How does this manifest in the data?
      • Are we studying Great White Sharks in aquariums?
      • Does there need to be a research-based ISP/telecom? Spyware? Securityware? Studyware? How does the user get to review/edit set anonymization levels?
  • A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services
    • Good paper on logging WRT new IIR tools to determine usefulness.
  • Assessing Learning Outcomes in Web Search:  A Comparison of Tasks and Query Strategies
    • The search system was hosted on Amazon EC2 and used an architecture derived from uFindIt [1] that logs user events such as queries and clicks to a MySQL database. The baseline ranked document lists for the single and multiple query conditions were provided by the Google Custom Search API.
    • Intrinsically diverse presentation – rolls subtopics into ’10 blue link’ list.
    • Coding by looking at the presence or absence of a predetermined list of items, this allows for sorting (n/N). Questions require examples of different levels of learning.
    • Two groups – lots of clicking and exploring vs. fewer results longer reading.
    • Need to do follow up studies to see the difference between learning and robust learning, where the knowledge is retained.
    • There is something here. Have to ponder the kind if inferences that can be made.
  • A Comparison of Primary and Secondary Relevance Judgments for Real-Life Topics
    • Shows how to make a corpus of variable relevance with respect to an information need. Very nice. Could be used for the person task but inverted? In other words we know what query returned a piece of text, but we give the reviewer the goal (bad doctor) and they judge relevance on those grounds, which lets us determine the quality of the query.
    • Relevance – binary
    • Confidence – 1 – 7
    • Time, etc
    • Experts have scanning behavior and can look for synonyms. Secondary assessors had to read in depth
      • Closed topics – what date did x happen? More agreement
      • Open topics – what caused the 2008 recession? Less agreement
    • There is an altruistic(?) component in looking for “information that might be useful”.
  • Interactive Topic Modeling for aiding Qualitative Content Analysis
    • Peter Bruza <- check out more. Semantic spaces and such
    • Identification and interpretation of themes within text
    • Types of content analysis
      • Summative – keywords
      • Conventional (inductive) Observation – NMF, LDA
      • Directed (Deductive)
      • Logic-LDA – Steerable using rules, and open source NMF – Linear Algebra underpinnings. LDA is probabilistic.
      • Semantic Validity – important for analyst confidence.Also known as topic coherence
      • Email for github repo.- done
    • The Information Network: Exploiting Causal Dependencies in Online Information Seeking
      • Network of informational elements
      • Granger causality modeling? (prediction of causality model) not used for prediction in information before
      • They use wikipedia page views as a marker topic newsworthiness!!!
      • And that means that you can look at one page becoming popular that it’s popularity in time exists with in the context described by a Granger Causality
      • Auto regressive model is good for time series modeling
      • jaspreet singh <– Call about Wikipedia!!!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.