7:30 – 5:00 VTX
- Write up some thoughts on Comps. What synthesis is, how study design doesn’t stop with the design, matrix of themes and papers, nuggets more than notes, etc.
- Andy Field has a new stats book out. It looks nice: Discovering Statistics – The Reality Enigma
- Starting on Presenting Diverse Political Opinions: How and How Much
- Sean A. Munson
- Paul Resnick – has worked on some very interesting things
- New aggregator site for political news Memeorandum
- New papers to add to the list….
- The Methods section is particularly good. I should probably use something similar.
- Conference
- Keynote – Pia Borlund Reflections on interactive IR evaluation issues.
- What data is required to answer the research question
- Which methods can provide the required data
- Types of users to test
- Data collection to use
- Number of participants
- Tasks to use in testing
- Time constraints
- Tailoring the work task situations according to requirements and inclusion of genuine information need that they bring with them.
- Rotation and counterbalancing of search tasks and systems (remove confounding variables) e.g. neutralize
- System knowledge
- topical knowledge
- fatigue
- Protocols
- Tutorials,
- Pilot testing
- Test Design
- Define the purpose of the study
- formulate the questions
- Design the study accordingly
- Pre search questionnaire
- Simulated work task
- Transaction logging
- Post-search interview
- Observation
- A protocol (can be 50 steps including things like turning on the test computer, cleaning the keyboard, etc)
- Is a step-by step description of the overall study procedure
- Is a checklist to the investigator
- Ensures consistency in the conduction of the study
- Ensures that all participants get the same information
- and….
- Before testing section
- Testing section
- After testing section
- Tutorials
- Introduction
- Tasks expectation
- Have the user explain what they are supposed to do
- Pilot testing
- What questions the participants ask
- Does the protocol work? Missing steps, wrong order
- How long the study takes
- Technical problems
- that the required data is collected
- To practice the control of pleasing effect and human nature
- Continue doing the pilot test until the the protocol is sufficient for the study. This means being consistent in explaining the procedure. Maybe use a video instead?
- Don’t do the study too close to the pilot, since lessons learned need to be incorporated.
- Everyday life information seeking: Approaching information seeking in the context of “way of life”
- Discussion
- The amount of dedication in the search makes a difference. How does this manifest in the data?
- Are we studying Great White Sharks in aquariums?
- Does there need to be a research-based ISP/telecom? Spyware? Securityware? Studyware? How does the user get to review/edit set anonymization levels?
- A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services
- Good paper on logging WRT new IIR tools to determine usefulness.
- Assessing Learning Outcomes in Web Search: A Comparison of Tasks and Query Strategies
- The search system was hosted on Amazon EC2 and used an architecture derived from uFindIt [1] that logs user events such as queries and clicks to a MySQL database. The baseline ranked document lists for the single and multiple query conditions were provided by the Google Custom Search API.
- Intrinsically diverse presentation – rolls subtopics into ’10 blue link’ list.
- Coding by looking at the presence or absence of a predetermined list of items, this allows for sorting (n/N). Questions require examples of different levels of learning.
- Two groups – lots of clicking and exploring vs. fewer results longer reading.
- Need to do follow up studies to see the difference between learning and robust learning, where the knowledge is retained.
- There is something here. Have to ponder the kind if inferences that can be made.
- A Comparison of Primary and Secondary Relevance Judgments for Real-Life Topics
- Shows how to make a corpus of variable relevance with respect to an information need. Very nice. Could be used for the person task but inverted? In other words we know what query returned a piece of text, but we give the reviewer the goal (bad doctor) and they judge relevance on those grounds, which lets us determine the quality of the query.
- Relevance – binary
- Confidence – 1 – 7
- Time, etc
- Experts have scanning behavior and can look for synonyms. Secondary assessors had to read in depth
- Closed topics – what date did x happen? More agreement
- Open topics – what caused the 2008 recession? Less agreement
- There is an altruistic(?) component in looking for “information that might be useful”.
- Interactive Topic Modeling for aiding Qualitative Content Analysis
- Peter Bruza <- check out more. Semantic spaces and such
- Identification and interpretation of themes within text
- Types of content analysis
- Summative – keywords
- Conventional (inductive) Observation – NMF, LDA
- Directed (Deductive)
- Logic-LDA – Steerable using rules, and open source NMF – Linear Algebra underpinnings. LDA is probabilistic.
- Semantic Validity – important for analyst confidence.Also known as topic coherence
- Email for github repo.- done
- The Information Network: Exploiting Causal Dependencies in Online Information Seeking
- Network of informational elements
- Granger causality modeling? (prediction of causality model) not used for prediction in information before
- They use wikipedia page views as a marker topic newsworthiness!!!
- And that means that you can look at one page becoming popular that it’s popularity in time exists with in the context described by a Granger Causality
- Auto regressive model is good for time series modeling
- jaspreet singh singh@l3s.de <– Call about Wikipedia!!!
viztales 