learning-based evaluation of visual analytic systems
DESCRIPTION
Presenter: Remco ChangBELIV 2010 Workshophttp://www.beliv.org/beliv2010/TRANSCRIPT
Learning-Based Evaluation of Visual Analytics Systems
Remco Chang, Caroline Ziemkiewicz, Roman Pyzh, Joseph Kielman*, William Ribarsky
UNC CharlotteCharlotte Visualization Center
*Department of Homeland Security
Why Another Evaluation Method?
• Based on a discussion with Joe Kielman (DHS)– Why is it difficult for agencies like the DHS to adopt and
use visual analytics systems?
• Most existing metrics are not indicative of success of adoption– Task completion time– Errors– Subjective preferences– Etc.
Current Methods
• Methods for evaluating visual analytics systems have been proposed. Each has its unique perspective and goal. For example:
– Insight-based Evaluation (North et al.)– Productivity-based Evaluation (Scholtz)– MILC -- Multi-dimensional in-depth long-term case
studies (Schneiderman, Plaisant)– Grounded Evaluation (Isenberg et al.)
Our Goal for Evaluation
• What Joe wants is:– Proof that the user of the visual analytics system can gain
proficiency in solving a problem using the system
– By using the VA system, show that a user can gradually change from being a “novice” to becoming an “expert”
• In other words, Joe wants proof that by using the VA system, the user is gaining knowledge…– The goal of visualization is to gain insight and knowledge
(ViSC report, 1987) (Illuminating the Path)
Learning-Based Evaluation
• In light of this goal, we propose a “learning-based evaluation” that attempts to directly test the amount of knowledge gained by its user.
• The idea is try to determine how much the user has learned after spending time using a VA system by:– Giving a user a similar but different task.– Directly testing if the user has gained proficiency in the
subject matter.
Current Method
Our Proposed Method
Types of Learning
• In designing either a new task or the questionnaire, it is important to differentiate and isolate what is being tested:
– Knowledge gained about the Interface– Knowledge gained about the data– Knowledge gained about the task (domain)
iPCA Example
• iPCA stands for “interactive Principle Component Analysis”. By using it, the user can learn about:– The interface – The dataset
• relationships within the data
– The task• What is principle
component analysis, and• How can I use principle
component analysis to solve other problems?
Application to the VAST Challenge
• Current method: – Give participants a dataset and a problem– Ask participants to develop VA systems to solve
the problem– Ask participants to describe their systems and
analytical methods– Judges score each submission based on the
developed systems and their applicability to the problem
Application to the VAST Challenge
• Proposed method: – Give participants a dataset and a problem– Ask participants to develop VA system to solve the
problem– Ask participants to bring their systems to VisWeek– Give participants a similar, but different dataset and
problem– Ask participants to solve the new problem using their
VA systems– Judges score each participant based on the
effectiveness of each system in solving the new task.
Types of Learning
• In designing either a new task or the questionnaire, it is important to differentiate and isolate what is being tested:
– Knowledge gained about the Interface– Knowledge gained about the data– Knowledge gained about the task (domain)
Discussion/Conclusion
• This learning-based method seems simple and obvious because it really is. Teachers have been doing this for ages.
• The method is not unique. There are many aspects of this proposed method that are similar to existing methods. In spirit, we are all looking to address the same problem.
• The difference is the perspective. If we think about the problem from the perspective of a client (e.g., Joe at DHS), what they look for in evaluation results currently are not the same as what we as researchers give them.
Future Work
• Integrate the proposed learning-based method to:– Grounded Evaluation– Long term effects (MILC)
Thank you!
[email protected]://www.viscenter.uncc.edu/~rchang
The Classroom Analogy
• Say you’re a math teacher in middle school, and you’re trying to decide which text book to use, the blue one or the red one. You can:– Ask your friends which book is better
• Analogous to an “expert-based evaluation”. Problem is that the sample size is typically small, and the results difficult to replicate.
– Ask your students which book they like• Analogous to subjective preferences. The issue here is that
the students can prefer the blue text book because its blue.
– Test which text book is more effective by giving the students tests.