interaction techniques for ambiguity resolution in recognition-based interfaces jennifer mankoff coc...
TRANSCRIPT
Interaction Techniques for
Ambiguity Resolution in Recognition-based
Interfaces
Jennifer MankoffCoC & GVU Center
Georgia Tech
Acknowledgements
Gregory Abowd & Scott Hudson FCE Group & GVU NSF
Outline
Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusion & Future Work
Ambiguity
am·big·u·ous1 a : doubtful or uncertain especially from obscurity or indistinctness <eyes of an ambiguous color> b : INEXPLICABLE2 : capable of being understood in two or more possible senses or ways
An anathema to computers Normal for humans
Where does ambiguity arise? Web search
user doesn’t know correct answer multiple correct answers
Implicit input user not involved with application
Multiple users System states may not agree
Recognition
Focus: Recognition
Recognition is becoming ubiquitous
Recognition is difficult to use
A range of interface problems result
OOPS toolkit helps solve them
Research Methodology
Motivated by real world, non-CS problems
Evaluators Study individual problems/solutions Design space of possible solutions
Builders Facilitate solutions to sets of problems
(Architectural / toolkit solutions) Design space exploration
Outline
Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work
Definitions Mediation
dialogue between user and computer used for resolving ambiguity
Recognizer interprets user input creates ambiguity
Error mistake from user’s perspective represented with ambiguity
SILK (Landay, 1996)
Burlap
Outline
Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work
OOPS Toolkit (CHI’00)
Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support
Based on subArctic
Library of mediators
Design space based on survey Generic and re-usable Three major classes
Repetition Choice Automatic
Library of mediators
Design space based on survey Generic and re-usable Three major classes
Repetition Choice Automatic
Library of mediators
Design space based on survey Generic and re-usable Three major classes
Repetition Choice Automatic
if (result is “W.”)reject it
elsedo nothing
Architectural Support
INDEPENDENT of any specific toolkit
Separation of mediators, recognizers, and application
Communication by a common internal model (ambiguous hierarchical events)
Maintains ambiguity indefinitely
Three key pieces
Ambiguous hierarchical events Changes to event dispatch Mediation subsystem
down drag up• • • • • •
s
Ambiguous Hierarchical Events
strokestroke
down drag up• • • • • •
s c
med1med2med3med4medn
rec1rec2rec3rec4recn
Event Dispatch
1. A sensed event arrives
eventInput
handler
2. It is dispatched to all recognizers3. It is mediated
Mediation Subsystem
Ambiguity is identified automatically Presence of multiple interactor leaf nodes
Hierarchy is passed to mediators Recognizers, recipients informed of
accept/reject decisions Accept/reject modifies hierarchy
Application selects mediators from library
Outline
Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work
Problem Areas
Errors & Ambiguity rejection errors target ambiguity
Mediation adding alternatives occlusion
Rejection Errors
Problem: The user’s input is completely missed
Rejection Errors
Problem: The user’s input is completely missed
Solution: Allow the user to tell the system
Rejection Errors
Problem: The user’s input is completely missed
Solution: Allow the user to tell the system
Other applications: Substitution errors
Rejection Errors
Problem: The user’s input is completely missed
Solution: Allow the user to tell the system
Other applications: Substitution errors Any spatial
recognition Requires extended
recognizer API
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
Other applications: Any interface
involving mouse press/release
Requires separation of concerns
Works with all interactors
Occlusion
Problem: A mediator may obscure important information
Occlusion
Problem: A mediator may obscure important information
Solution: Move that information into a more visible location
Occlusion
Problem: A mediator may obscure important information
Solution: Move that information into a more visible location
Other applications: Any crowded
interface that uses an n-best list
Requires extensible mediators
Requires separation of concerns
Adding alternatives
Problem: The correct choice isn’t always present
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
Other applications: Closely related
choices (e.g. URL prediction)
Requires extensible mediators
Benefits from recognizer API
Outline
Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work
Conclusions
Resolution of ambiguity in recognition through mediation
General toolkit architecture (CHI 00) flexible, re-usable support for mediation
separates recognition, mediation, and
applications allows exploration of design space
Next Steps
Implicit input Sensed information about
environment Ambiguous output
Ambient displays Brain-computer interface
Ambiguous, limited, error-prone input
500,000 people worldwide
Locked-In Syndrome
Disease, stroke, accident survivors Completely paraylzed Unable to speak Cognitively intact
Brain-computer interface
Problem: locked-in syndrome No alternative modalities available Need for efficient error handling
Challenge: interpret brain signal in as rich a form as possible
A new hope
Brain signals can be intercepted Implanted electrodes (Schwartz, Chapin et
al.) External sensors (Junker, Wolpaw,
Middendorf, Birbaumer et al., Spencer et al.)
Signals can be produced and controlled by imagined movements
Signals can be interpreted by a computer
A Neurotrophic Electrode
Cone electrode invented in 1987 Animal studies showed it to be stable FDA permission given for human implantation in 1996
Project Goals
High level: Recreate movement Restore communication Turn disability into ability
Low level: Intelligent Interpretation
Recreating movement
Haptic feedbackMuscle stimulation Eventually re-connect nerves
Restore Communication
A basic need
From virtual keyboards to word-prediction
Turning disability into ability:
Supporting Creativity
Converting a the brain signal to music
Example
Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can
reproduce
Intelligent Interpretation
Signals are difficult to control Daily variability in signal Patient Endurance
Adaption necessary
Intelligent Interpretation
Raw signal-> mouse Logical control Neural gestures
Alternative Forms of Input
Cirrin (UIST 98) Locked-In Syndrome
(Brain-UI; Current) Cerebral Palsy
(Cursor Activity Recognition; Current)
How general is “general”?
Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas
(UIST 00)
Further generalization
Testing
Further generalization
Testing Implicit input
(CT-OOPS; Current)
Further generalization
Testing Implicit input Arbitrary input devices
Further generalization
Testing Implicit input Arbitrary input devices Ambiguity
Related Themes
Input Output
Ambient Displays (Ten Inch Pixels; 1999)
Is subArctic doing the work here?
No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers
events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just
a library!)
Recognizer
Definition: something that interprets user input generally has a domain (of input) and a
range (of output)
Examples: DragonDictate (speech to text) GDT (strokes to gestures)
Problem areas: Support for correction of errors
Error Definition:
a mistaken interpretation (from the user’s perspective)
Examples: substitution rejection insertion
Problem areas: rejection
Mediation
Definition: a dialogue
between the user and application used to determine the correct interpretation
Problem areas Occlusion Wrong choices
Examples:
Ambiguity Definition
A case where there is more than one potentially correct interpretation of the user’s input
Problem areas target ambiguity
Examples target ambiguity segmentation
ambiguity recognition
ambiguity
Future Work
Testing Implicit input Arbitrary input devices Ambiguity
Conclusions
The problem areas are not intractable
Toolkit-level support allows us to explore them
OOPS allows us to build general, re-usable solutions
Other thesis results
Survey of mediation techniques found in existing interfaces to recognition systems
Two implementations of our architecture
Comparing SILK and Burlap
SI LK BURLAP
Modes Four One Combined
Only Draw Mode ThroughoutMediationSeparate Window I ntegrated
Comparing SILK and Burlap
SI LK BURLAP
Modes Four One Combined
Only Draw Mode ThroughoutMediationSeparate Window I ntegrated
Editing Yes No
Storyboarding Yes No
Architectural support
architecture
interactors
interface
application
Input handler
inters
OOPS Architecture
architecture
interactors
interface
application
recognizers
Input handler
meds
application
interface
stroke
downdrag up• • • • • •
sc
stroke
downdrag up• • • • • •
s
Big Picture
1/5 People have disabilities Everyone has temporary
disabilities Computers can help to overcome
disabilities… … or make them worse
Modified Event Dispatch
1. A sensed event arrives 2a. It is dispatched to all
unambiguous components2b. It is dispatched to all recognizers3. It is mediated4. Any accepted leaf nodes are
dispatched to all unambiguous components
Comparison with GUI
event event
interactors
interface
application
Input handler
recognizers meds
application
interface
Input handler
inter
recognizers mediators