Evaluating Visual and Statistical Exploration of Scientific Literature Networks

Robert Gove1,3, Cody Dunne1,3, Ben Shneiderman1,3, Judith Klavans2, and Bonnie Dorr2,3

1HCIL, 2CLIP, 3Computer ScienceUniversity of Maryland, College Park, USA

How do we help users explore scientific literature domains?

Commonly available capabilities

• Document statistics• Keyword summary• Citation visualization• Corpus statistics• Full-text search

Three types of systems

• Search engines & digital libraries• Reference managers• Summarization & recommender systems

Taxonomy of System Functionalities

Action Science Explorer

Visual and statistical exploration

Evaluation

• Corpus of 147 papers on Dependency Parsing• Exploration-based usability testing– 4 domain experts, two hours each

1. Training

2. Predefined tasks

3. Exploration questions

4. Participant-defined tasks

1. Training

• 15-30 minutes• Showed videos demonstrating Action Science

Explorer• Answered participant questions• Gave think-aloud instructions

2. Predefined tasks

Number of participants

Capability

Task 1: Identifying important authors and

papers

Task 2: Collecting evidence

Attribute Ranking ███ 3 ██ 2

Find Communities █ 1

Plot Nodes

In-Cite Text █ 1

In-Cite Summary

Search █ 1

Sorting █ 1

Groups ██ 2 █ 1

3. Participants’ questions

• How has the field evolved?• What are subtopics within the field?• What are the most accessible papers?• What are the relationships between authors?• What are the methods and applications?

4. Participant-defined tasks

Number of participants

CapabilityAll participant-defined

tasks

Attribute Ranking ██ 2

Find Communities █ 1

Plot Nodes

In-Cite Text █ 1

In-Cite Summary

Search █ 1

Sorting █ 1

Groups █ 1

Answering participants’ questions

• 2/4 questions were answered– 1 unanswered question was uncompletable– 1 unanswered question not well supported

Design recommendations for exploration

• Confirmed design decisions– Provide metrics for ranking documents– Support user-created groups and document

annotation– Give overview of corpus

• Suggested new capabilities– Allow control over corpus refinement– Incorporate undo/redo functionality

Conclusion

• A taxonomy of capabilities in current tools• An evaluation of Action Science Explorer• A list of literature exploration questions• A list of system design recommendations

Robert Gove - rpgove@gmail.com

Acknowledgements

• Supported by NSF grant IIS 0705832and Booz Allen Hamilton

• Dragomir Radev and Vahed Qazvinian, University of Michigan

• David Zajic and Michael Whidby,University of Maryland