PERSPECTIVES ON

ADAPTIVITY IN INFORMATION RETRIEVAL INTERACTION

an ASIS&T 2010 annual meeting panel

PAIRI

PAIRI

Agenda/Outline

Adaptivity in IR Interaction

Four Dimensions

Pecha-Kucha Presentation and Timeline

Audience Interaction!

Wrapping Up

PAIRI

PRESENTERS

Birger Larsen, Royal School of LIS, Copenhagen, Denmark

Marianne Lykke, Aalborg University, Denmark

Diane Kelly, University of North Carolina at Chapel Hill, USA

Peiling Wang, University of Tennessee, Knoxville, USA

ModeratorPeter Ingwersen, Royal Schoolof LIS, Copenhagen, Denmark

PAIRI

ADAPTIVITY in IR INTERACTION

… requires that IR systems adapt to users’ situations, and

… the users adapt to the systems.

PAIRI

SYSTEM ADAPTION to USERS

… entails dynamic user modeling;

effective information architecture founded in practice, and

enhanced search features, such as

search integration

relevance feedback

PAIRI

SEARCHER ADAPTION to SYSTEMs through INTERACTION

… entails mental model building of systems and modification; through

Learning – leading to

Knowledge change, such as from an anomalous state of knowledge (ASK) towards

a coherent state of knowledge (CSK);

PAIRI

Adaption - Central DIMENSIONS:

throughintegration of

information objects

of the Information Retrieval system to the current searcher

to context and practice through

information architecture

of searchers to the Information Retrieval systems

PAIRI

PRESENTATION – PECHA-KUCHA

Present Introduction – 7 minutes

First Dimension: 20 slides x 20 seconds (7 min) Questions – interaction (10 minutes)

Second Dimension: 20 slides x 20 sec. (7 min)

Questions – interaction (10 minutes)

Third Dimension: 20 slides x 20 sec. (7 min) Questions – interaction (10 minutes)

Fourth Dimension: 20 slides x 20 sec. (7 min) Questions – interaction (10 minutes)

10 minutes vivid interaction on additional audience questions

Wrapping up (5 min)

PAIRI

QUESTIONS ON PAPER SLIPS

During presentations you may write questions to the panel (aside from oral questions!!)

’Collectors’ will collect the slips and hand over to Moderator

Moderator selects which questions to be posed during last 10 minutes of interaction

Moderator knows which questions will be dealt with by ensuing panellists.

PAIRI

Adaption through…

…integration of information objects

PAIRI

Adaptivity example: me!

Wrap it up –this is not

your topic…

PAIRI

Me – again!

Work web page

Images

Bibliography

PAIRI

Different ‘verticals’

PAIRI

How do we handle ‘verticals’ in libraries?

As separate silos!User

Catalogue .

Repositories Databases eJournals

JSTOR

Publisher 1

Publisher 2

eBooks

PAIRI

How to access the silos?

Information literacy! = educate users how to use each system:

Content, fields, indexing, search operators, interfaces…

But…

‘Why do we want to teach our users to be librarians?’

(Dave Pattern, Library Systems Manager)United Kingdom Serials Group 2009 Annual conference

PAIRI

Users don’t want silos!

‘Why is Google so easy and the library so hard?’

(Claire Duddy - student)

United Kingdom Serials Group 2009 Annual conference

PAIRI

Challenge

Each silo has its own set of metadata and standards developed for different purposes Very low common denominator…

PAIRI

Solution: Federated search?

(William H. Mischo, 2005)

PAIRI

Solution: Federated search?

“The jury is still out on federated search systems, even though more libraries now have them. There are murmurings that federated search has lower-than-expected use and may not be the magic search bullet we were led to believe” (Tenopir, 2007)

PAIRI

Solution: Integrated search?

Harvest all of the relevant data sources, normalize them into a single metadata schema, and index all of them together in one large union index

Add Google-like search box and ranking

(…a kind of federated search 2.0)

PAIRI

Solution: Integrated search?

Is being implemented in university and research libraries as we speak University of Huddersfield , UK Queens University, Canada State and Uni library, Denmark …

Several commercial products Summon by Serial Solutions EBSCO Discovery Service WorldCat Local ExLibris' Primo Central III Encore Synergy …

PAIRI

Solution: Integrated search?

But…

Development handed over to large commercial vendors

or giant web search services (Google; Amazon)

Where does that leave us as a field, as practitioners and researchers?

PAIRI

Research opportunities

There is sooo much we don’t know: How to optimise each type of information

object?

How to integrate information objects?

How to do relevance feedback in integrated search?

How to best involve users in testing?

etc etc…

PAIRI

Research opportunities

So, we can study various elements and details…

But how to generate real knowledge of these colossal integrated systems? Larger, more complex than ever

How to evaluate these scientifically, and not just rely on vendors?

PAIRI

Information Retrieval (IR) test collection

Purpose: to facilitate studies of integrating different document types

iSearch test collection

PAIRI

Three types of documents from physics

18,841 book records

291,244 metadata records, incl. abstract

143,569 full text articles

iSearch test collection

PAIRI

65 thoroughly described Information tasks1. Information need - What are you looking for?

2. Work task context - Why are you looking for this?

3. Knowledge state - Background knowledge of topic?

4. Ideal information - To solve your problem or task?

5. Search terms - Which search term would you use?

+ graded relevance assessments

iSearch test collection

PAIRI

What can we do with this? Study these tasks

Optimise each document type

Optimise the integration

Simulate relevance feedback and interfaces?

What we can’t do

Study another domain

Study real user interaction

iSearch test collection

PAIRI

Adaption through integrationof information objects

Pertinent issues How to design and evaluate systems that

successfully integrate genres, media and document types?

Even with shallow data?

How to generate real knowledge and evaluate these scientifically?

Without relying on what vendors offer?

PAIRI

Adaptation of …

information retrieval (IR) system to the user

PAIRI

TECHNIQUE EXPLICIT-O-SAURUS

PAIRI

USER-MODEL-O-SAUR

PAIRI

TECHNIQUE IMPLICIT-O-SAURUS

PAIRI

PAIRI

Schultz, C. K. (1968). H.P. Luhn: Pioneer of Information Science (p.32). London, UK: American Documentation Institute.

1950s: Luhn’s Selective Dissemination of Information

PAIRI

1960s: Salton, Lesk, Rocchio, Ide and Others

Ide, E. (1967). User interaction with an automated information retrieval system. In G. Salton (Ed.) Information Storage and Retrieval: Scientific Report No. ISR-12.

PAIRI

Oddy, R. N. (1977). Information retrieval through man-machine dialogue. Journal of Documentation, 33(1), 1-14.

1970s: Oddy’s Thomas

PAIRI

1980s: USER-MODEL-O-SAUR

Allen, R. B. (1990). User models: Theory, method, and practice. International Journal of Man-Machine Studies, 32, 511-543.

Rich, E. (1983). Users are individuals: Individualizing user models. International Journal of Human-Computer Studies, 51, 323-338.

“While the term ‘user model’ emphasizes the information about the person, it is obvious that a great deal of situational, task, or environmental information may be encoded in the model.”

PAIRI

Croft, W. B., & Thompson, R. H. (1987). I3R: A new approach to the design of document retrieval systems. Journal of the American Society for Information Science, 38, 389-404.

1980s: Intelligent IR

PAIRI

Maes, P. (1994). Agents that reduce work and information overload. CACM, 37(7), 30-40.

1990s: Agents

PAIRI

Maes, P. (1994). Agents that reduce work and information overload. CACM, 37(7), 30-40.

SYSTEM

USER

PAIRI

Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434.

And more relevance feedback

PAIRI

What Caused Explicit-O-Saurus’ Demise?

Users are unwilling to put forth the effort required to provide feedback

Users don’t have the additional cognitive resources to engage in explicit feedback

PAIRI

2000s: Implicit-O-Saurus

Click-through Dwell time Scrolling Query Behavior

PAIRI

Users are unwilling to put forth the effort required to provide feedback.

http://www.pewinternet.org/http://www.visualeconomics.com/how-the-world-spends-its-time-online_2010-06-16/

REALLY?

PAIRI

http://www.movielens.org/rateMorehttp://www.grouplens.org/

2010

In My Own Words

Interests

PAIRI

Users don’t have the additional cognitive resources to engage in explicit feedback.

Well, maybe back then …

PAIRI

We need to figure out better ways of eliciting feedback from users:

Better questions

Better measures

More creative

More engaging

More adaptive

Saving Explicit-O-Saurus

PAIRI

http://hunch.com/

PAIRI

Saving Explicit-O-Saurus

Questions

How can we create better questions and measures?

How can we make the process creative, engaging and adaptive?

Objection

Users are creatures of habit. Let’s reintroduce TECHNIQUE

EXPLICIT-O-SAURUS into the information ecology

Users will adapt …

PAIRI

Users’ Adaptability in IT

Environment

Adaptivity is a much needed system functionality, yes but

System alone cannot solve all the interaction problems

Humans are evolutionary learners

IT users must / can adapt to the system

PAIRI

Biological View

PAIRI

Psychological View

Mooers’ Law?

Satisficing principle

Today, majority IT users

Do not really have a choice at one level

Do have choices at another level

PAIRI

Cognitive View

Individual differences

Context/situation-based

Knowledge

Skills

Mental models

Tacit

PAIRI

Constructivist View

Problem-based learning

Social learning

• Observational learning specially relevant to building IT skills

• Knowledge sharing

• Collective intelligence

PAIRI

Adaptability — What is it?

AdaptAbility is a key

metacompetency

which enables

individuals quickly,

effectively respond

to unexpected

environmental

changes

PAIRI

Adaptability in Contexts

In workplace: job adaptability

In multicultural society: cross-cultural adaptability

In information environment: ITadaptability

PAIRI

Measuring Adaptability

Job Adaptability Inventory (JAI)

Cross-cultural Adaptability Inventory(CCAI)

Boyatzis-Kolb Adaptive Style Inventory

IT Adaptability – we need instruments

Adaptability – contributing factors?

PAIRI

Job Adaptability

Creative problem solving (new …)

Dealing Uncertainty/unpredictable work situations

Learning new tasks, tech, procedures …

Adapting culturally

Adapting interpersonally

Physical factors

PAIRI

Cross-Cultural Adaptability Inventory (CCAI)

A survey of 50 questions to uncover individuals’ current strengths and weaknesses within four critical skill areas proven necessary for effective cross-cultural communication:

Emotional Resilience

Flexibility/Openness

Perceptual Acuity

Personal Autonomy

PAIRI

Adaptive Style Inventory

• Acting situation• new• on time

• Deciding Situation• two alternatives• an opportunity

• Thinking• an idea• analyzing something

• Valuing• my feelings• see the world as the another

person sees it

--Kolb Learning Style Inventory?

PAIRI

IT Adaptive Behaviour

We need systematic research to understand factors underlying the IT adaptive behavior

We need instruments to measure IT adaptability (either adopt existing or develop new relevant one)

PAIRI

Where do we start?

Adaptive situations

IT system being updated

Need a function never used

New IT system

“I have used it before, but how do I get to it now”?

“The output doesn’t look right”!

“Where are my files I saw all of them just a moment ago?! Or, no, I cannot lose them ...”

PAIRI

Adaptive Performance

User’s context

IT Environment

User’s IT competency

User’s mental model

The task situation

PAIRI

AdaptAbility vs. LearnAbility

User adaptability calls for the system’s learnAbility

IT system learnability support user’s adaptAbility

learnability is an aspect of usability –rarely studied

PAIRI

ALPHA-IT

A (Adaptability)with

L (Learnability)for

P (Personalizing)&

H (Humanizing)

A (Adaptivity)in

IT (Information Technology)

PAIRI

Personalizing & Humanizing

Personalizing: Beyond user profiling

IT knowledge state

IT competency

Adaptive style

Humanizing : Beyond affordances

Demonstrate/encourage empathy

Affective responses

PAIRI

IT Affordance

Specifies the range of possible actions about an object (physical or digital)

Must be visible to the users to be usable

Individual differences in perceived affordances

When affordances fail users: an action does not result in an expected result --?

PAIRI

Building IT Adaptability as Social Learning

Sharing learning experience

Transferring knowledge

Collecting user problems

Banking user strategies

PAIRI

Study of User Adaptability & IT Design

How do we study users adaptive behavior

in adopting a new system, new applications, new functions, new …?

How can personal adaptive behaviors help

system design?

PAIRI

Adaption to context and practice

Focus on adapting information architecture (IA)

Focus on models and methods for developing tailored information architecture (IA)

PAIRI

Quick definition of IA (1)

ORGANISATION SYSTEM

LABELSYSTEM

Structural design

Combination of organisation, labelling,search and navigation

PAIRI

Quick definition of IA (2)

NAVIGATION SYSTEM

SEARCHSYSTEM

PAIRI

Adapting IA

Organization system, e.g.

Categories– tailored to user needs

Organization– organized to user perspective

Labels, e.g.

Terminology– jargon, language of youth, basic level

PAIRI

Adapting IA

Navigation system, e.g.

Tailored short cuts

Tailored links

Tailored recommending,

Search system, e.g.

Tailored filters

Tailored ranking

PAIRI

Adapting IA

PAIRI

System development process

(Rosenfeld & Morville, 2007, 232)

PAIRI

Research process

Context

UsersContent

Methods for data collection: e.g. interview, workshops, questionnaires, content analysis, log files

Mission, information behavior, policies, culture, technology, resources (time, economy, competences)

Content types, number, genre, usage, relation to work tasks

Target groups, information behavior, work tasks, search tasks, discourses, terminology

(Rosenfeld & Morville, 2007, 233)

PAIRI

Strategy process

(Rosenfeld & Morville, 2007, 269)

PAIRI

System development models

Plenty of models and approaches:

User-oriented

Domain-oriented

Work-task oriented

Interaction-oriented

Participation-oriented

…

All believe that we can grasp the context

PAIRI

System development models

Can we grasp, understand, model the context and practice – and adapt?

Are we“heroes” between

“victims” and “tyrants”?

(Spinuzzi, 2004)

PAIRI

Borger.dk – an example

Development of organisation system for Danish Government portal Borger.dk

PAIRI

Borger.dk – an example

Timeline

2008 Analysis, development, implementation

2009 Evaluation: usability tests, online survey and search log analysis

2009 Redesign and implementation of updated version and new features

PAIRI

Borger.dk – design methodology

Development phrase Survey of domain of public digital

communication

Personas – user needs and behaviour

Bench marking – related portals and organisations systems

Content analysis

Expert evaluation

Usability tests - 12 citizens completing 4 tasks

PAIRI

Borger.dk – design methodology

Evaluation phrase

Usability tests - 40 citizens completing 4 tasks

Online survey - 453 self-selected respondents

Search log analysis - Google Analytics over 11 month

PAIRI

Borger.dk - redesign

Information needs• 75% of needs met by categories

• Miss personal and factual information

Categories, organization, navigation• Unclear labels

• Short cuts - to frequent categories and forms

• Cross references

• Prioritized ordering

• Search • Synonym rings

• Best bets

Marianne Lykke & Brian Kirkegaard Lunn

PAIRI

Explanation of difficult concepts

PAIRI

More and tailored cross references

PAIRI

Best bets for important topics

PAIRI

??? Adaption to context ???

Do “heroes” exist within system design and adaption?

What models are feasible to adaption to context and practice?

What tools should be developed?

PAIRI

Last round of questions & Interaction!!

PAPER SLIP QUESTIONS FROM

YOU

PAIRI

WRAPPING UP the PANEL

Interesting discussions on:

Design and evaluation of integrated systems?

How to avoid re-engineering of vendor offerings?

Feasibility of explicit relevance feedback for systems adaptivity?

PAIRI

WRAPPING UP the PANEL - 2

Interesting discussions on:

How does personal adaption behaviorhelp systems (design)?

Research methdology and tools studying adaptive behavior to new ...?

How to adapt to context and work practice?

PAIRI

THANK

YOU