1 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
A Methodological Frameworkfor Multi-Fidelity Sketching
of User InterfacesAdrien Coyette
Université catholique de Louvain (UCL)Louvain School of Management (IAG-LSM)
Information Systems Unit (ISYS)Belgian Lab. of Computer-Human Interaction (BCHI)
http://www.isys.ucl.ac.be/bchi
2 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
3 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Motivations: UI importance
User interface determines how easily a user may control underlying functions of a computer program
A program equipped with powerful functionalities and low quality user interfaces may be under-exploited or misused
In an interactive application, the UI is probably the portion which affects the most the general acceptability of the system by end usersOn the average, 48% of the code written for the 69 applications studied was devoted to the user interface portion
[Nielsen 93; Myers & Rosson 92]
4 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Motivations: UI development life cycle
Software plans and requirements
Validation
Validation
Product design
Verification
Detailed design
Verification
Code
Unit test
Integration
Product verification
Operations and maintenance
Revalidation
Implementation
System test
System feasibility
[Boehm88]
UI Design occupied 45% of the totalamount of time devoted to each activity
50%
37%
Mean = 44%
5 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Motivations: methods
As in software engineering, HCI community wants to develop a well-structured and step-wise method for developing user interfaces.
The systematization and the reproducibility found in software engineering methods however cannot be transferred straightforwardly in HCI
The development life cycle remains inherently:
Open, Ill-defined, Highly iterative
Interaction is of higher complexity than algorithms
[Sumner et al.Sumner et al., 1997; Wegner, 1997]
6 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Motivations: iterative design
Designers often consider that informal techniques based on mock up is the best alternative for the early design phase.
It allows to cope with the fact the such process is highly iterative in 3 stages
[Norman, 1996Norman, 1996]
7 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Motivations: empirical findings
UI Drawing or sketching increases participatory design and subjective satisfaction with respect to UI editors
[Landay & Myers, 1995, Plimmer & Apperley, 2004, Buxton, 2005]
Equal usability problems found for a sketched UI than with an edited one[Virzi et al., 1996]
Explanatory power of a sketched UI does not decrease with respect to edited one; tool support is preferred over paper prototyping
[Walker et al., 2002 ]
A sketched UI delivers same quantity and quality of outputs than an edited one[Sefelin et al., 2006]
UI sketching fosters alternative designs and communication [Tohidi et al., 2006]
Supports throw-away prototyping due to low cost[Schneider & Petrie, 2006, Bailey & Konstan, 2003]
8 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
9 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Prototyping: when?
Early design
Late design
Low fidelity paper prototypes
Content: Mainly Presentation
Use: Exploration, Communication
Medium fidelity prototypes
Content: Presentation, content, basic navigation
Use: Simulation, refinement, user testing…
High fidelity prototypes
Content: Presentation, navigation, functionalities, content
Use: Final specifications, marketing, documentation
A paper based prototype
A PowerPoint prototype
The final user interface with few functionalities
10 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Prototyping: sketches
• Paper prototypes– Familiar and unconstrained approach – Fast to learn and quick to produce – Focus on basic structural issues instead of unimportant
details – Support collaboration between designers and end-users – …
• Purpose– Early design phase– Get a sense of the user’s needs and goals– Supports brainstorm competing representations– …
11 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Prototyping:Limiting prototype functionality
• Vertical prototypes– Includes in-depth functionality for only a few selected
features– Common design ideas can be tested in depth
• Horizontal prototypes– The entire surface interface with no underlying
functionality– A simulation; no real work can be performed
• Qcenario– Scripts of particular fixed uses of the system; no deviation allowed
Vertical prototype
Scenario
Horizontal prototype
Full interface
[Nielsen 1993]
12 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Prototyping:Integrating prototypes and products
• Throw-away– Prototype only serves to elicit user reaction– Creating prototype must be rapid, otherwise too expensive
• Incremental– Product built as separate components (modules)– Each component prototyped & tested, then added to the
final system
• Evolutionary– Prototype altered to incorporate design changes– Sometimes becomes the final product
13 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
14 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
State of the Art:Computer Assisted Prototyping
Several levels of granularity
Support for scenario-based design
Good documentation
No shape recognition and interpretation
No code generationNo preview mode
DENIM
[Lin,2000]
15 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
State of the Art:Computer Assisted Prototyping
Performance (speed and accuracy)Multi-stroke
gesturesRecognizes rotated shapes
Interpretation in Java
Only generates Java
No scenario editor
Limited widget setMono-window
JavaSketchIt
[Caetano,2002]
16 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
State of the Art:Eliciting requirements
17 different software and approaches surveyedSources: scientific literature and commercial
softwareComparative analysis relying on consistent
analysis gridTool identification (7)Install and first use (4)Tool functions (8)Shape recognition (4)Shape interpretation (7)UI editors (9)
Applied to the same case study
(eCommerce app.)
17 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
State of the Art:List of final requirements
Avoidance of Effort loss
Well defined editing functionalities
Language neutrality
Robust recognition
Large conceptual coverage
Recognition and process flexibility
Design history
Expressive scenario editor
Ease of use (naturalness)
Preview (Run-mode)
18 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
19 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Demonstation
20 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Shape Recognition
21 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Shape Recognition
Recognition engine: Cali libraryRecognition engine: Cali library
A fast, simple and compactA fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes)
Combines temporal adjacency, Fuzzy Logic and geometric temporal adjacency, Fuzzy Logic and geometric featuresfeatures to classify scribbles.
Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap.
Shapes are recognized independently of changes in rotation, recognized independently of changes in rotation, size or number of individual strokessize or number of individual strokes
The recognizer has a recognition rate of 94%94% and a fast response
22 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Shape Recognition
Trainable Gesture RecognizerTrainable Gesture Recognizer Handdrawn input (sketch) is
superimposed with a grid Each sampling point is mapped to
its closest grid node For adjacent nodes, a stroke
direction (out of 8 different directions) is taken
A string is generated from direction coding of input
Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances
23 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Shape Recognition
Distance is shortest sequence of edit commands that transform s to t Simplest set of operations:
• Copy character from s over to t • Delete a character in s (cost 1) • Insert a character in t (cost 1) • Substitute one character for another (cost 1)
Levenstein Levenstein distance distance
24 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Shape Interpretation
25 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
SketchiXML: Grammar
<widget type="ListBox"> <representation id="0">
<constraint id="0" shape1="Triangle_3" shape2="Rectangle_0" condition="isInsideInUpperRightCorner" /> <constraint id="1" shape1="Triangle_4" shape2="Rectangle_0" condition="isInsideInLowerRightCorner" />
<shape id="Rectangle_0" type="Rectangle" /> <shape id="Triangle_3" type="Triangle" /> <shape id="Triangle_4" type="Triangle" /> </representation><representation id="1"><constraint id="0" shape1="Triangle_0" shape2="Rectangle_2" condition="isInsideInTop"/>
<constraint id="1" shape1="Triangle_1" shape2="Rectangle_2" condition="isInsideInBottom" /> <constraint id="2" shape1="Rectangle_2" shape2="Rectangle_3" condition="isInsideOnTheRight" />
<shape id="Triangle_0" type="Triangle" /> <shape id="Triangle_1" type="Triangle" /> <shape id="Rectangle_2" type="Rectangle" /> <shape id="Rectangle_3" type="Rectangle" /> </representation> <representation id="2"><constraint id="0" shape1="Line_0" shape2="ListBox_1" condition="isInside" /> <constraint id="1" shape1="Line_0" shape2="-" condition="isHorizontal" /> <shape id="Line_0" type="Line" /> <shape id="ListBox_1" type="ListBox" /> </representation></widget>
26 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
27 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Methodologies: framework
• Prototyping types
User interface
Control layer
Functional core
Completeinteractiveapplication
Horizontal prototypeVertical prototype
User interface
Control layer
Functional core
Completeinteractiveapplication
Diagonal prototype
[Nielsen93]
28 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Methodologies: framework
• Prototyping types
User interface
Control layer
Functional core
Completeinteractiveapplication
PresentationGlobal
navigationLocal
navigation
29 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Methodologies: framework
• Prototyping types
Control layer
Functional core
PrésentationGlobal
navigationLocal
navigation
Presentational prototype first
Control layer
Functional layer
PresentationNavigation
globaleLocal
navigation
Global navigation prototype first
Control layer
Functional core
PrésentationNavigation
globaleNavigation
locale
Local navigation prototype first
30 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Methodologies: Support by fidelity
High fidelity prototype : GrafiXML
Medium fidelity prototype : VisiXML
Low fidelity prototype: SketchiXML
31 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Methodologies: Application to development methodologies
Analyst claimed in the past that prototyping should be considered as an alternative to the systems development life cycle…
More recent analysis advocate the use of prototyping as a part of the traditional systems development life cycle
Existing methodologies tend to integrate prototyping as part of the approach
Based on the framework we are thus able to support effectively the current development methodologies. SketchiXML permit to involve end users without pre-requisite
[Kendall 97]
32 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Best practices:
• Collect information
• Develop simple prototype
• Discuss with the end-users
• Test interaction between end user and prototype
• Adapt the fidelity level
• Avoid polished prototype
• Iterate
Applied to the latest methodologiesApplied to the latest methodologies
[Meyer 2005]
Methodologies: Application to development methodologies
33 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
34 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: Widget catalogue
Building the widget catalogue Building the widget catalogue
35 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: widget catalogue
36 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: User testing of fidelity
Method:
• 12 participants
• Wacom graphic tablet
• 12 widgets randomly selected
• 4 fidelity levels
• Repeated twice
Factors observed:
• Time required
• Number of errors
• Number of delete operations
37 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: User testing of fidelity
Results:
• Fidelity has no impact on the sketching of individual widget (time or error)
• User profile influence the widget sketching time (non UI designers performed even better !)
• Proved learning effect over time
38 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: Performances
Widget
Recognition rateSketching Time
Q1 Q2 Q3 Q4 total Q1 Q2 Q3 Q4 total
Button 0.88 0.94 1.00 1.00 0.95 6654 2646 1952 2036 3322
Checkbox 1.00 0.94 0.88 1.00 0.95 1221 2040 2030 1486 1694
Combox 0.75 0.88 0.94 0.88 0.86 4501 2445 2859 3178 3246
Label 0.94 0.94 1.00 1.00 0.97 2513 353 35 37 734
Listbox 0.88 0.94 0.88 0.94 0.91 7325 7250 8535 5852 7241
Picture 1.00 1.00 1.00 0.87 0.97 1858 1541 1378 2836 1903
Progress 0.69 0.81 0.94 0.94 0.84 11429 5697 4118 4547 6448
Radio 1.00 0.94 0.88 0.94 0.94 1085 2030 1913 2029 1764
Slider 0.94 0.88 0.94 0.94 0.92 3313 2789 2474 1851 2607
Textarea 0.94 1.00 0.81 1.00 0.94 3545 2342 4838 2312 3259
Textfield 0.94 1.00 1.00 1.00 0.98 3379 1080 1089 1204 1688
Toggle 0.69 0.69 0.88 0.94 0.80 16041 6641 7445 4283 8603
Total 0.89 0.91 0.93 0.95 0.92 5238,62 3071,19 3222,15 2637,61 3542,39
39 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Surveys: Representations
Group Widget Gesture repres
Recognition rate
Spe
cific
or
ienta
tion
Sim
ple
incl
usi
on
Com
plex
in
clusi
on
Juxt
apos
itio
n
Inte
rsec
tion
sequ
ence
of
com
ponen
ts
Siz
e
1
Text field 0,98 X X
1 Label 0,97 X
1 Picture 0,97 X
2 Button 0,95 X
2 Checkbox 0,95 X X X
2 Radio button 0,94 X X X
2 Text area 0,94 X X X
2 Slider 0,92 X X X
3 List box 0,91 X X X X
3 Combobox 0,86 X X
3 Progress bar 0,84 X X X X
3 Toggle button 0,80 X X
40 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Presentation plan
MotivationsMotivations
PrototypingPrototyping
State of the ArtState of the Art
SketchiXML SketchiXML
MethodologiesMethodologies
SurveysSurveys
ConclusionConclusion
41 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
1) Fundamental
• Notion of fidelity level (low, medium, high) in multi-fidelity prototyping
• Graphical definition of widgets in terms of representations• Formal definition of representations in a grammar• Gesture recognition algorithm based on Levenstein distance
2) Methodological
• Multiple prototype types (diagonal=vertical+horizontal) and paths• Intertwining of presentational and navigational prototyping• Integration within various development life cycles• Transition between the fidelity levels (within SketchiXML and
other tools based on UsiXML) at Concrete User Interface level
Conclusion: contributions
42 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
3) Empirical
• User and designer acceptance of widget representations• User testing of sketching tool• Complexity of widget representations
4) Implementation
• Implementation of SketchiXML (50000 LOC)• Multi-agent shape recognizer
– 8 basic shapes (i.e., triangle, rectangle, cross, line, wavy line, arrow, ellipse, and circle)
– 6 basic commands (i.e., undo, redo, copy, paste, cut, new window)– 27 widgets
• Trainable gesture recognition engine• Multi-fidelity support by smooth transition between (none, low, medium, high)• Multi-agent interpreter• All 10 requirements addressed (partially or completely)
Conclusion: contributions
43 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Conclusion: future work
• Extending the coverage of sketching artifacts• Improving the Text Divider• Tuning the Recognition Engine more extensively• Support for Multi-windowing Design• Augmenting the Support for Design Memory• Extending to other domains than Computer Science
• General Improvement of sketching facilities
44 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Conclusion: example of extension
• Sketching Physical User Interfaces with transition between digital and physical worlds
45 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
For more information and downloading,http://www.isys.ucl.ac.be/bchi
http://www.usixml.orgUser Interface eXtensible Markup Language
http://www.similar.ccEuropean network on Multimodal UIs
Special thanks to all members of the team!
This work has been conducted in the context of the following projects
Projet WIST 1 ReQuestRapid prototyping of e-commerce applications
46 Private Ph.D. defense – Louvain-la-Neuve, June 25th, 2007
Thank you very much for your attention!Questions?