usigesture: an environment for integrating pen-based interaction in user interface development
DESCRIPTION
— Several algorithms have been developed for pen-based gesture recognition. Yet, their integration in streamline engineering of interactive systems is bound to several shortcomings: they are hard to compare to each other, determining which one is the most suitable in which situation is a research problem, their performance largely vary depending on contextual parameters that are hard to predict, their fine-tuning in a real interactive application is a challenge. In order to address these shortcomings, we developed UsiGesture, an engineering method and a software support platform that accommodates multiple algorithms for pen-based gesture recognition in order to integrate them in a straightforward way into interactive computing systems. The method is aimed at providing designers and developers with support for the following steps: defining a dataset that is appropriate for an interactive system (e.g., made of com¬mands, symbols, characters), determining the most suitable gesture recognition algorithm depending on contextual variables (e.g., user, platform, environment), fine-tuning the parameters of this algorithm by multi-criteria optimization (e.g., system speed and recognition rate vs. human distinguishability and perception), and incorporation of the fine-tuned algorithm in an integrated development environment for engineering interactive systems.TRANSCRIPT
![Page 1: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/1.jpg)
1
UsiGesture: an Environment for Integrating Pen-based Interaction in User
Interfaces
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
François Beuvens and Jean Vanderdonckt [email protected]
Researcher at LiLab, http://www.lilab.beUniversité catholique de Louvain (Belgium)
![Page 2: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/2.jpg)
2
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 3: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/3.jpg)
3
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 4: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/4.jpg)
4
Context- Scope: 2D pen-based gestures- Needs of pen-based gesture recognition
- PDAs – Graffiti- Mouse gestures plug-in for Firefox- Games- …
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 5: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/5.jpg)
5
Motivations- No « One fits all » algorithm- Needs of algorithms comparison
- Same data and conditions- Data acquisition
- Enhancements of existing algorithms
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 6: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/6.jpg)
6
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 7: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/7.jpg)
7
Gesture database acquisition
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 8: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/8.jpg)
8
Gesture database acquisition- 30 participants
- 16 men and 14 women with average age of 28- 60 gestures to reproduce
- 26 lower case letters, 10 digits, 16 action commands, 8 geometrical shapes
- 10 reproductions of each gesture by participant- More information:
https://sites.google.com/site/comparativeevaluation/data-collection
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 9: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/9.jpg)
9
Comparison study: analysis criteria- Training types
- User dependent, user independent (2 methods)
- Gesture types- Letters, digits, actions, geometrical shapes
- Algorithms- Rubine, One Dollar, Levenshtein, stochastic Levenshtein
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 10: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/10.jpg)
10
Rubine- Training
- Compute mean vector of features and weights for learning classes
- Recognition- Compute vector of features- Scalar product of weights
and vector- Best score is chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 11: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/11.jpg)
11
One Dollar- Training
- Preprocessing: resampling, rotation, rescaling, translation
- Recognition- Preprocessing- Comparison to all training examples
- Finding optimal angle- Evaluating point-to-point Euclidean distance- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 12: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/12.jpg)
12
Levenshtein- Training
- Transformation into strings
- Recognition- Comparison to all training examples
- Computing Levenshtein distance- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 13: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/13.jpg)
13
Stochastic Levenshtein- Training
- Transformation into strings- Creation of a string pairs set- Optimization of the primitive edition costs
- Expectation-maximization algorithm
- Recognition- Comparison to all training examples
- Evaluating distance based on probabilities- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 14: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/14.jpg)
14
Results – initial version- Training types for Levenshtein on digits
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 15: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/15.jpg)
15
Results – initial version- Gesture types for Rubine with user dependent
training and user independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 16: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/16.jpg)
16
Results – initial version- Algorithms on digits and actions with user
independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 17: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/17.jpg)
17
Results – initial version- Summary
- More Information: https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 18: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/18.jpg)
18
Extensions- Rubine
- Preprocessings
- One Dollar- kNN
- Levenshtein- Preprocessings, kNN, normalisations, multi-strokes gestures,
other features handling, costs matrix
- Stochastic Levenshtein- Idem except costs matrix
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 19: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/19.jpg)
19
Results - Rubine’s extensions- Preprocessings: resampling, rotation, rescaling
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 20: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/20.jpg)
20
Results - One Dollar’s extensions- kNN
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 21: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/21.jpg)
21
Results - Levenshtein’s extensions- Normalizations, pre-processings, kNN, multi-
strokes, pressure inclusion, costs
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 22: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/22.jpg)
22
Results - Stochastic Levenshtein’s extensions- Normalizations, preprocessings, kNN, multi-
strokes, pressure inclusion
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 23: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/23.jpg)
23
Results – extensions- Summary
- More Information: https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 24: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/24.jpg)
24
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 25: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/25.jpg)
25
Woke example (before)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 26: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/26.jpg)
26
Woke example (after)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 27: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/27.jpg)
27
WindowBuilder integration (demo)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 28: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/28.jpg)
28
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 29: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/29.jpg)
29
Ongoing and future works- Theoretical
- Conceptual modeling of gesture-based UIs
- Methodological- Integration into UsiXML- Definition of development lifecycle (e.g. SPEM)
- Empirical- User study on user satisfaction with gesture integration - Development benefits estimation
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![Page 30: UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development](https://reader036.vdocuments.site/reader036/viewer/2022070302/547b2983b37959822b8b4c3c/html5/thumbnails/30.jpg)
30
Thank you for your attention!
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science