model-driven engineering of user interfaces
DESCRIPTION
This presentation contains the slides of the Doctoral Course given at University of Valencia (Spain) regarding model-driven engineering of user interfaces based on UsiXML (User Interface eXtensible Markup-Language, www.usixml.org), November 2006.TRANSCRIPT
1 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Model-Driven Engineeringof User Interfaces
Jean Vanderdonckt
Université catholique de Louvain (UCL)Louvain School of Management (LSM)
Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)
http://www.isys.ucl.ac.be/bchi
2 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Course outline
• Day 1: Why do we need to model the user interface?– H1: Motivations and introduction to usability evaluation of user
interfaces• Case studies and evaluation
– H2: Concepts used for usability evaluation for every UI• Usability guideline, ergonomic criteria, IFIP properties
• Day 2: What do we need to model to cover UI aspects? (Part 1)– H1: The Cameleon reference framework for multi-target UIs
• Underlying metamodel• Terminology and revisitation of IEEE Taxonomy of RE• Task modelling, domain modelling in IdealXML
– H2: Abstract UI• Mapping model• Model-to-model transformation in IdealXML
3 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Course outline
• Day 3: What do we need to model to cover UI aspects? (Part 2)– H1: Concrete UI
• Model-to-model transformation in TransformiXML• Model-to-code generation in GrafiXML• Graph grammars and other techniques
– H2: Final UI• UI rendering: by interpretation, by compilation• UI rendering in multiple computing platforms
• Day 4: When do we need to model what to cover UI aspects?– H1: Multi-path development of UIs
• Forward, reverse, lateral engineering• UI adaptation: graceful degradation
– H2: Multiple targets• Multiple users, platforms, environments
4 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Course outline
• Day 5: How do we assess the UI modelled?– H1: Automated evaluation of guidelines
• Evolution of code static analysis• Evaluation for web and non-web applications
– H2: Towards genuine model-checking of UIs• Usability advisor: natural language evaluation• Evaluation of usability guidelines, design rules, properties of
interest• Final conclusion: evolution of MDA for UIs
– Low threshold– High ceiling– Wide walls
5 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What is the situation today?
• Technological aspects of user interfaces progress significantly faster than– Software engineering aspects
• It takes time to develop a user interface with a new device, a new interaction technique
• It takes more time to develop a toolkit• It takes even more time to rely on a model-driven approach
– Usability engineering aspects• New user interfaces are shipped with usability problems
because– Little or no experience– No past, no empirical evidence
• Empirical experiments require a lot of resource if done carefully
Day 1: Why?
[Dragicevic et al.,2004]
6 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006Target Applications,
Domains Notations and tools User Interface Interaction
Techniques
2006
TOD
AY
Adapted from[Palanque,2002]
No Interaction TechniqueAutomated, batch systems Nothing
Character UIsBusiness applications Screen definitions
Graphical UserInterfaces
Information systems Entity-relationshipAttribute modelState-transition diagrams
7 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What’s the situation of today?
• Interactive Software evolution: context of use =(U,P,E)
time
Platform
User
Environment
Language
Day 1: Why?
8 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Software evolution
WML 2.0WML 1.0
XML
XHTML
1986 1989 1997 1998 20031996
HTML 4.0HTMLSGML
VoiceXML2000
time
Environment
User
Platform
Day 1: Why?
9 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Diversity of users
• Traditional users
• People with disabilities– Visual, motor, cognitive
Day 1: Why?
10 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Variety of tasks
• Users want to determine their path on each platform
Day 1: Why?
[Forrester Research,2003]
11 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Heterogeneousness of platformsDay 1: Why?
12 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multiplicity of contexts of use
Location Role Device Experience
Sporting Multimedia Travel programme
Working Travel booking sitePowerful interface for complex operations
Travelling Booking notificationEverywhere connectivity for simple data exchange
Family TV is multi-media family device #1
Day 1: Why?
13 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UI #12UI #11UI #10UI #9Application 3
UI #8UI #7UI #6UI #5Application 2
UI #4UI #3UI #2UI #1Application 1
Platform #4Platform #3Platform #2Platform #1
Application 1
Application 2
Application 3
UI model #1
UI model #2
UI model #3
Platform #1
Platform #2
Platform #3
Platform #4
Platform model #1
Platform model #2
Platform model #3
Platform model #4
Consequence
• Proliferation of developments
Day 1: Why?
[Abrams et al.,2001]
14 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Is this web site usable?Day 1: Why?
15 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Ergonomic criteria
• Criteria for assessing the usability of any UI– A priori and/or a posteriori– At design and/or evaluation time
• Structured into 8 first-level criteria– Compatibility– Consistency– Work load– Adaptation– Dialog control– Guidance– Error management
• Implicit order: sorted by importance
Day 1: Why?
[Scapin & Bastien,1997]
[Vanderdonckt,1995]
16 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Navigation
• Use a reactive image as a navigation map
No map without any relation
Metaphoricmap
Day 1: Why?
17 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Navigation
• Include navigation clues
Navigationbar
Landmark
Global vslocaldiagram
Day 1: Why?
18 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Navigation
• Navigation clues should be consistent
Inconsistencies between the menu and navigation bar
Use the same clues at the same location for the same purpose
Day 1: Why?
19 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Navigation
• No more "Clic here"
Clic here to go to the Publications pageClic here to go to the LSM pageClic here to go to the UCL page
Go to : UCL / LSM / Publications
Day 1: Why?
20 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Navigation
• Do not use "Return to..." links
Aller à :...
ButtonAvoid : come back,return, redo,cancel, undo,redirect
Homepage
Page A Page B Page C Page D Page E
Firstsite
Page 1 Page 2 Page 3 Page 4 Page 5
Second site
Day 1: Why?
21 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Presentation
• Information wich are semantically related should be presented together
Related label
Related label
Day 1: Why?
22 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Presentation
• Example of a web page format (before)
Day 1: Why?
Display zone of browser navigation buttons
Browser status bar
Area for accessing to main commands, langages, map, about, contact
User categorypicture
Site menuszone Informational contents Display zone
for external linksand externalapplications
Organisation logo area
Area for accessing to local commands
Structure indicator
23 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Presentation
• How to format? Position of home page link, internal links
Day 1: Why?
[Shaikh & Lenz,2006]
24 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Presentation
• How to format? Position of search engine, advertisements, about us
Day 1: Why?
[Shaikh & Lenz,2006]
25 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Presentation
• Example of a web page format (after)
Day 1: Why?
Logo to home page
Site menuszone Informational contents External links
You are here: structure indictor
NL-FR-DLTitle, sub-title
Contact - Mentions légales – Vie privée – Médiateur - Accessibilité
Log in – Site map - Help
Search
Update – Printer-friendly v.
26 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Simple choice
Mixeddomain:
Unknown domain :
Amount ofpossible values [2,3]
Amount ofpossible values [8,50]
Amount ofpossible values[4,7]
Amount ofpossible values]50,+[
Knowdomain:
Amount ofpossible values[2,3]
Amount ofpossible values[4,7]
Amount ofpossible values [8,50]
Amount ofpossible values
]50,+[
Day 1: Why?
[Vanderdonckt & Faulkner,2002]
27 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Choix multipleDomaineinconnu :
Mixed domain:Known domain:
Amount ofpossible values[4,7]
Amount ofpossible values [8,50]
Amount ofpossible values]50,+ [
Amount ofpossible values[2,3]
Amount ofpossible values[4,7]
Amount ofpossible values [8,50]
Amount ofpossible values]50,+ [
Amount ofpossible values[2,3]
Day 1: Why?
[Vanderdonckt & Faulkner,2002]
28 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Graphics
• Use graphics for headers
Use moderately artistic fonts for headersKeep hear in a separate layerUse anti-aliasing
Day 1: Why?
29 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• The background should be appropriate for the task
Horizon line Example Meaning
Use various horizon lines depending on the context
Low Brings attention to the sky, the abstract, high values
High Stresses immediate results, the concrete
Median Suggests balance, equilibrium, peace
Diagonal Creates some instability, a feeling of moving
SharpSuggests dynamic aspects, changing, aggressivity
Day 1: Why?
30 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Low horizon line
Day 1: Why?
31 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• High horizon line
Day 1: Why?
32 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Median horizon line
Day 1: Why?
33 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Diagonal horizon line
Day 1: Why?
34 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Sharp horizon line
Day 1: Why?
35 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Do prefer low-intensity, light backgrounds
No whiteon black
Day 1: Why?
36 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Do not use backgrounds with textures
Pale backgrounds, submit and test
Pages with automatically built background
Day 1: Why?
37 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Keep the good color combinations
Multimedia elementsDay 1: Why?
[Murch,1987]
Blue (94%) Black (63%) Red (25%)
White (75%) Yellow (63%)
Yellow (75%) White (56%) Black (44%)
Black (100%) Blue (56%) Red (25%)
White (75%) Yellow (63%) Cyan (25%)
Blue (69%) Black (56%) Red (37%)
Black (63%) White (56%) Blue (44%)
Background Thin lines and text
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Background Thin lines and text
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow Red (63%) Blue (63%) Black (56%)
38 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Keep the good color combinations
Multimedia elementsDay 1: Why?
[Murch,1987]
Black (69%) Blue (63%) Red (31%)
Yellow (69%) White (50%) Green (25%)
Black (50%) Yellow (44%) White (44%)
Black (69%) Red (63%) Blue (31%)
Yellow (38%) Magenta (31%) Black (31%)
Red (56%) Blue (50%) Black (44%)
Blue (50%) Black (44%) Yellow (25%)
Red (75%) Blue (63%) Black (50%)
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Black (69%) Blue (63%) Red (31%)
Yellow (69%) White (50%) Green (25%)
Black (50%) Yellow (44%) White (44%)
Black (69%) Red (63%) Blue (31%)
Yellow (38%) Magenta (31%) Black (31%)
Red (56%) Blue (50%) Black (44%)
Blue (50%) Black (44%) Yellow (25%)
Red (75%) Blue (63%) Black (50%)
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
39 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Avoid the bad color combinations
Multimedia elementsDay 1: Why?
[Murch,1987]
Yellow (100%) Cyan (94%)
Blue (87%) Red (37%) Magenta (25%)
Magenta (81%) Blue (44%) Green (25%)
Cyan (81%) Magenta (50%) Yellow (37%)
Green (62%) Red (37%) Black (37%)
Green (81%) Yellow (75%) White (31%)
Green (75%) Red (56%) Cyan (44%)
White (81%) Cyan (81%)
Background Thin lines and text
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Yellow (100%) Cyan (94%)
Blue (87%) Red (37%) Magenta (25%)
Magenta (81%) Blue (44%) Green (25%)
Cyan (81%) Magenta (50%) Yellow (37%)
Green (62%) Red (37%) Black (37%)
Green (81%) Yellow (75%) White (31%)
Green (75%) Red (56%) Cyan (44%)
White (81%) Cyan (81%)
Yellow (100%) Cyan (94%)
Blue (87%) Red (37%) Magenta (25%)
Magenta (81%) Blue (44%) Green (25%)
Cyan (81%) Magenta (50%) Yellow (37%)
Green (62%) Red (37%) Black (37%)
Green (81%) Yellow (75%) White (31%)
Green (75%) Red (56%) Cyan (44%)
White (81%) Cyan (81%)
Background Thin lines and text
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Background Thin lines and text
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
40 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Avoid the bad color combinations
Multimedia elementsDay 1: Why?
[Murch,1987]
Yellow (95%) Cyan (75%)
Blue (81%) Magenta (31%)
Magenta (69%) Blue (50%) Green (37%)
Cyan (81%) Magenta (44%) Yellow (44%)
Green (44%) Red (31%) Black (31%)
Yellow (69%) Green (62%) White (56%)
Cyan (81%) Green (69%) Red (44%)
White (81%) Cyan (56%) Green (25%)
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Yellow (95%) Cyan (75%)
Blue (81%) Magenta (31%)
Magenta (69%) Blue (50%) Green (37%)
Cyan (81%) Magenta (44%) Yellow (44%)
Green (44%) Red (31%) Black (31%)
Yellow (69%) Green (62%) White (56%)
Cyan (81%) Green (69%) Red (44%)
White (81%) Cyan (56%) Green (25%)
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
Background Bold lines and panels
White
Black
Red
Green
Blue
Cyan
Magenta
Yellow
41 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multimedia elements
• Consider font variations induced by different platforms
Fixed size fontLabel on top offield
Same fontsTwo browsers
Day 1: Why?
42 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
IFIP Quality Properties for UIs
• Ten properties in 3 categories• Category 1: information presentation
– Multiplicity of devices: keyboard, mouse, tablet,…– Multiplicity of representations: alternate representations both in input and output,
honesty– Input/output reusability (use output produced by one action as input for another)
• Category 2: ordering of task planning– Multiplicity of user roles– Multiplicity of execution paths: users should be able to be engaged in different tasks
simultaneously– Non-preemptiveness: degree of freedom for the user to decide what’s next– Reachability: possibility to navigate in the system (undo, redo)– Observability vs Browsability
• Category 3: dialog adaptation– Reconfigurability: system ability to support user personalization– Adaptivity: system ability to support automated adaptation– Migrability: system ability to transfer responsibility from one user to another, among
users, among users and systems– Plasticity: system ability to adapt to the context of use while preserving predefined
usability properties• Dix’s Principle of Effort Maximility
– A complex action should be easy to do, but complex to undo
Day 1: Why?
[Gram & Cockton,1986]
43 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
IFIP Quality Properties for UIsDay 1: Why?
• Multiplicity of devices and representations
[Stephanidis,2001]
44 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Derivation panel with preview
CTTEOperator
Operatorparameters
Design comment resulting
fromApplying
guidelines
DesignPreview
that dynamically changes
according to parameters
Legends
Day 1: Why?
[Vanderdonckt et al.,2003]
45 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Plastic UI
• Example: the Virtual keyboard
Day 1: Why?
[Grolaux et al.,2001]
46 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Plastic UI
• Plastic UI: adaptable bounded value
Day 1: Why?
[Vanderdonckt et al.,2005]
47 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Plastic User interface
• Property of plasticity = adaptation to the context of use while satisfying predefined usability properties of interest
Day 1: Why?
[Grolaux et al.,2002]
48 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
How to address this issue?
• Capture essence of concepts through models– Separation of concerns, Correlability– Parsability, editability– If possible, human readability
• Typical models Task & Concepts
Abstract UI
Concrete UI
Final UI
Task & Concepts
Abstract UI
Concrete UI
Final UI
Source platform Target platform
Task & Concepts
Abstract UI
Concrete UI
Final UI
Task & Concepts
Abstract UI
Concrete UI
Final UI
Source platform Target platform
Day 2: What?
[Calvary et al.,2003]
49 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Typical models
Task & Concepts
Abstract UI
Concrete UI
Final UI
Task & Concepts
Abstract UI
Concrete UI
Final UI
Source platform Target platform
Task & Concepts
Abstract UI
Concrete UI
Final UI
Task & Concepts
Abstract UI
Concrete UI
Final UI
Source platform Target platform
textInput button button
Window
AICfacet=control
AICfacet=control
AICfacet=control
AbstractIndividualContainer
textInput button button
Window
AICfacet=control
AICfacet=control
AICfacet=control
AbstractIndividualContainer
50 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Environment T
What do we want to get?
Final userInterface T
Concrete userInterface T
Task and Domain T
Abstract userInterface T
T=Target context of use
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
Abstraction
Reflexion
Translation
http://www.plasticity.org
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S Platform TUser T
Day 2: What?
[Calvary et al.,2003]
51 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Software engineering interpretation
• Different types of engineering– Forward engineering– Reverse engineering– Lateral engineering– Diagonal engineering (with or without shortcuts)
• Different approaches– Top-down– Bottom-up– Middle-out
• Different development paths– Example: Round-trip engineering = composition of
• Reification CUI -> FUI• Reflexion FUI -> FUI• Abstraction FUI -> CUI• Reflexion CUI -> CUI
52 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
IEEE Reverse Engineering Taxonomy
Reference: Elliot J. Chikofsky , James H. Cross II, Reverse Engineering and Design Recovery: A Taxonomy, IEEE Software, v.7 n.1, p.13-17, January 1990: http://labs.cs.utt.ro/labs/acs/html/resources/ReengineeringTaxonomy.pdf
Reengineering-"the examination of a subject system to reconstitute it in a new form and the subsequent implementation of the new form.“
Restructuring- "a transformation from one form of representation to another at the same relative level of abstraction." The new representation is meant to preserve the semantics and external behavior of the original.
Redocumentation- "a form of restructuring where the resulting semantically-equivalent representation is an alternate view intended for a human audience.“
Design recovery- "a subset of reverse engineering in which domain knowledge, external information, and deduction or fuzzy reasoning are added to the observations of the subject system." The objective of design recovery is to identify meaningful higher-level abstractions beyond those obtained directly by examining the system itself
[Chikofsky & Cross,1990]
53 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Revisitation
Task &Concepts
Abstract UI
ConcreteUI
Final UI Reformating
TranscodingRecoding
Respecification
Retasking
Restructuration
Programunderstanding
Redocumentation
Reverse Engineering
Design recovery
Reengineering
Revamping
[Bouillon,2006]
54 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multi-Path Development of UI
• Multi-path development qualifies a methodology that support the realization of multiples types of development paths withing a single framework
Task and DomainTask and Domain
Abstract User Interface
Abstract User Interface
Concrete User Interface
Concrete User Interface
User InterfaceCode
User InterfaceCode
Task and DomainStage A
Abstract User InterfaceStage B
Concrete User Interface Stage C
User InterfaceCode Stage D
e.g., DetailedSpecification or Implementation
e.g., High Level Specification
Task and DomainTask and Domain
Abstract User Interface
Abstract User Interface
Concrete User Interface
Concrete User Interface
User InterfaceCode
User InterfaceCode
Task and DomainStage A
Abstract User InterfaceStage B
Concrete User Interface Stage C
User InterfaceCode Stage D
Task and DomainTask and Domain
Abstract User Interface
Abstract User Interface
Concrete User Interface
Concrete User Interface
User InterfaceCode
User InterfaceCode
Task and DomainStage A
Abstract User InterfaceStage B
Concrete User Interface Stage C
User InterfaceCode Stage D
e.g., DetailedSpecification or Implementation
e.g., High Level Specification
Forward engineeringReverse EngineeringAdaptationAny Relevant Composition
Development step
Development step
Development path
Development path *
1
Development Scenario
Development Scenario
*
*
Development step
Development step
Development path
Development path *
1
Development Scenario
Development Scenario
*
*
[Limbourg,2004]
55 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Our goals
• Objectives– Description of interactive systems
• Using pre existing domain specific meta-models• Used both at design and run time
– Capitalization• Properties• Transformations
Interactive system
Model of the IS
Model of the IS ‘
Interactive system ‘
Checks of properties
Transformation
Models, instances of Meta-Models described in MOF (even for properties and transformations…)
56 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UsiXML
• UsiXML =– USer Interface exTensible Markup Language– XML-compliant specification language for user interfaces suitable
for any interface• Web• Java• Windows-based applications• Multimodal applications, 3D applications• Virtual, mixed reality applications
– http://www.usixml.org– Join the UsiXML Consortium by registering on line– Download the CD image from
http://www.usixml.org/index.php?download=UsiXML RelOne.iso
57 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UsiXML = User Interface eXtensible Markup Language
• What is UsiXML?– It is a XML-compliant User Interface Description Language– Publicly available from http://www.usixml.org– Free to use, open for access, easy to expand– Definition of the language
UML Class DiagramsUsiXML Reference manual
XSD XML Schema Descriptions UsiXML Models
58 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006Target Applications,
Domains Notations and tools User Interface Interaction
Technique
2006
TOD
AY
Adapted from[Palanque,2002]
2007
No Interaction TechniqueAutomated, batch systems Nothing
Character UIsBusiness applications Screen definitions
Graphical UserInterfaces
Information systems Entity-relationshipAttribute modelState-transition diagrams
Multi-platform User Interfaces
Web, Java applications
Task model, contextmodel, UML,…
59 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
The problem
• To develop user interfaces (UIs) simultaneously for multiple contexts of use
• A context of use = triple– User– Computing platform– Surrounding environment
• Organisation• Socio-psychological factors
[Calvary et al.,2003]
60 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
The problem
• Why is it difficult?– For the designer:
• Multiplicity of contexts of use• No systematic design method
– For the user:• Poor usability of resulting UI• UI not adapted to the genuine context of use
– For the developer:• Increase of development and maintenance costs• Increasing development complexity• Little of no factoring out of common elements
61 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
The problem
• Why should it be systematic?– Some consider it noble to have a method– Other consider it noble not to have a method– Not to have a method is bad– But to stop entirely at any method is worse still– One should at first observe rules severely, then change them in an
intelligent way– The aim of possessing method is to seem finally as if one had no method
[Lao Tch’ai Tche, many many years ago]
62 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Traditional approach– Visual
development
63 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Advantages of traditional approach– UI is graphical by nature– A UI prototype can be
• Rapidly produced• Easily modified• Visually demonstrated
64 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Shortcomings of traditional approach– No structured method for developing a UI – All the knowledge remain implicit– Modification can lead to unstructured design– Little or no tool support for iterative design
• Selection of widgets can be inappropriate• UI Layout can be tedious, repetitive • Problem of the spaghetti of callbacks
– Early prototyping is hard to achieve and time consuming– Limited reusability (throw away)
65 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Shortcomings of traditional approach– Limited use of UI builder
Table with dynamic data Gantt chart Direct manipulation
Desired interface Obtained interface
Menu bar and pull-down menus Toolbar Scrollbars
[Szekely,1996]
66 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Interface builders cannot build their own UI
[Szekely,1996]
67 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Any development method (or methodology) is decomposed into 3 axes:– Models: explicitly capture knowledge about UI and Interactive
Applications with appropriate abstractions– Method: structures the definition and use of underlying models in
a stage-wise approach– Supporting tools: support the use of the method by providing
tools for models and their related operations. Ideally, one model should be supported by at least one tool
[Bodart,1989]
68 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mono-platform UI
• Goal: to integrate all three facets
Method
Model 1Model …
Model n
Models
Model
ModelModel
Interface1
Tools
69 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
MDE based on UsiXML
Model to Model
PlatformIndependentModel (PIM)
PlatformSpecific
Model (PSM)Model to Code Source code
MDA Components
Techniques proposed based on UsiXML
ComputingIndependentModel (CIM)
Model to Model
UsiXML model:Abstract user
interface
UsiXML model:Concrete user
interfaceRendering
Final userinterface
UsiXMLmodels: task,
domain
Graphtransformations
Graphtransformations
70 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
CIM Step 1: Task model
71 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
New Abstraction: the user’s task
• Task = set of actions carried out by a user in a given context to reach a goal
• Logical decomposition of task into sub-tasks• Temporal ordering: LOTOS operators (in CTTE)
– T1 >> T2 Enabling – T1[ ]>>T2 Enabling + information passing– T1 |> T2 Suspend/resume– T1 [ ] T2 Non-deterministic choice– T1 T2 Deterministic choice– T1 [> T2 Disabling (e.g. Form submit)– T1 |=| T2 Independence (any order, but finished)– T1* Iteration– T1{n} Finite iteration– T1 ||| T2Concurrency– T1 |[x]| T2 Concurrency + information passing– [T] Optional– T Recursion
[Markopoulos,1992]
72 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
New Abstraction: the user’s task
• Task definition = action + object– Action types
• CRUD pattern: create, read, update, delete• Select, control,…• Acquire, render, modify, publish, compute, derive,…
– Object types:• Element, list, table, collection, compound,…
73 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
New Abstraction: the task meta-model
[Limbourg,2004]
74 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
CIM Step 2: domain model
[Limbourg,2004]
75 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
CIM Step 3: Task-domain mappings
IdealXML
[Limbourg,2004]
76 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
New Abstraction: the abstract UI
• Different CIOs can be used for the same purpose, but with different interaction modalities
• Definition– Abstract Container = set of Abstract Individual Component– AIC = abstraction of CIOs of the same type, but independently of any interaction
modality– Abstract User Interface (AUI) = decomposition into AC+AIC
[Vanderdonckt & Bodart,1993]
77 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Abstraction: the abstract UI
78 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Abstraction: the abstract UI
• Notation: based on L. Constantine’s notation for canonical abstract prototypes
Abs.container
Abs. component
Input
Output
Navigation
Control
Select
IdealXML
[Montero et al.,2005]
[Constantine,2003]
79 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
IdealXML
[Montero,2005]
80 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example of AUI produced
81 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
triggers (tg): { , } x
updates (up): x
observes (ob): x
isExecutedIn (ex): x
manipulates (ma): { , } x
These mappings can be established:
Mapping the models
[Montero,2005]
82 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mapping the models
• Mapping the models with a mapping model (!!)
83 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Your turn now!
• Virtual Polling System– Characterization of a participant
• Age• Gender• Region• Country
– Polling• Series of questions & answers• Each answer may have
– Simple choice– Multiple choices
• More capabilities…
84 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example solution (first variant)
85 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example solution (first variant)
86 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Task and domain models
[Montero,2005]
87 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Typed Model-to-Model Transformation
Uses language
Meta-Meta-ModelGraph Structure
is instance of
Meta-ModelOur Meta-Model
Meta-Model Subset 1e.g., Task+Domain Model
is instance of
Meta-Model Subset 2e.g., Concrete UI Model
is instance of
Initial UI Modele.g., MyTaskAndDomainModel
Resultant UI Modele.g., MyConcreteUIModel
Transformation RuleOur transformation catalog
[Limbourg,2004]
88 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Expression of models as graphs
• All transformations are in UsiXML– Each model = instance of meta-model– Each model = graph as instance of graph type– Each model transformation =
• graph transformation• Set of productions
89 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Definition of a production
• Find an occurrence of LHS in G (this occurrence is called a match). If several occurrences exist, choose one non-deterministically.
• Check preconditions of both type PAC and NAC. If not verified, then skip.
• Remove the part of G which corresponds to (LHS – K), where K is the morphism specified between LHS and RHS.
• Add RHS – K into G – (LHS – K) as it is given by the corresponding relation between RHS – K and K
• Check postconditions of both type PAC (and notably that the resulting graph is properly typed) and NAC. If not verified, then undo the transformation rule.
LHS
G
RHS
G’
r
m m*
r*
LHS
G
RHS
G’
r
m m*
r*
90 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GHost USIXM specification
G’Resultant USIXM specification
LHS RHS
Matches -Co-Matches
Is Transformed Into
Is Transformed Into
Transformation Rule 1
Transformation Rule 2…
Transformation Rule N
Tra
nsfo
rma
tion
Sys
tem
NAC
Not Matches
+
Transformation system
[Limbourg,2004]
91 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM step: task+domain to AUI
• Abstract UI (AUI) = UI independent of any interaction modality• Definition of AUI structure in terms of Abtract Containers (AC)
– Which tasks should be logically grouped?
• Definition of Abstract Individual Components (AIC) types – Which « functionnality » should assume AICs and what data do
they manipulate ?
• Definition of spatio-temporal arrangement– How should AIC be arranged in space and time ?
• Definition of dialog control– What is the valid flow of action on AICs ?
UsiXML model:Abstract user
interface
UsiXMLmodels: task,
domain
Graphtransformations
92 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM step: task+domain to AUI
Identification of AUI structure
Spatio-Temporal Arrangement of AIOs
Selection of AIC
Definition of Abstract Dialog Control
STEP : From Task & Domain to AUI
SU
B-S
TE
PS
Derivation of AUI to Domain Relationships
93 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
How to read a graph transformation?
Node typeNode
(Attribute,value)
Edge typeEdge
(Attribute,value)
Node
Edge
94 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 1
LHS RHS
::=Ø
95 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 2
LHS RHS
::=
96 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 3
LHS RHS
::=
97 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 4
LHS RHS
::=
NAC
98 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 5
LHS RHS
::=
NAC1 NAC2
99 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 6
LHS RHS
::=
100 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 7
LHS RHS
::=
PAC
“X < 3000”
101 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 8
LHS RHS
::=
PAC
“X > 3000”
NAC
102 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 9
LHS RHS
::=
103 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rule 10
PAC LHS RHS
::=“X > Y”
104 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM sub-step 1: Definition of AUI structure (AC)
NAC LHS RHS
::=
NAC LHS RHS
::=
NAC LHS RHS
::=
NAC LHS RHS
::=
105 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM sub-step 1: Definition of AUI structure (AC)
NAC LHS RHS
::=
NAC LHS RHS
::=
106 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Facet type
NAC LHS RHS
::=
NAC LHS RHS
::=
107 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
108 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
ACParticipate to OpinionPoll
AC Answer Poll
AC Answer Questionnaire
AC Answer Question
AIC (Input/Single Selection)Select Proposition
AIC (Output)See Statistics
AIC (Input/Single Selection )Chose Poll
AIC (Control)Validate Question
AIC (Output)Provide Personal Data
AIC (Output)Read Question
AIC (Input/Single Selection)Chose Questionnaire
109 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM sub-step 2: definition of AIC types
NAC LHS RHS
::=
NAC LHS RHS
::=
AC = Abstract ContainerAIC = Abstract Individual ComponentCIC = Concrete Interaction Component
110 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PIM sub-step 3: Definition of spatio-temporal arrangement
NAC LHS RHS
::=
NAC LHS RHS
::=
111 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PSM Step: AUI to CUI
• Concrete UI (CUI) = UI independent of toolkit• Definition of CUI structure
– Which AIC is a window? • Definition of Concrete Interaction Component (CIC) type
– Which « widget » should represent which AIC ?• Definition of placement
– What layout can be specified between CICs,…• Definition of navigation
– Which container can be started or closed from which container? • Definition of dialog control
– What is the valid flow of action on AIOs
UsiXML model:Abstract user
interface
UsiXML model:Concrete user
interface
UsiXMLmodels: task,
domain
Graphtransformations
Graphtransformations
112 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PSM Step: AUI to CUI
Reification of AC into CC
Arrangement of CICs
Selection of CIC
Concrete Dialog Control Definition
STEP : From AUI to CUIS
UB
-ST
EP
S
Definition of Navigation
Derivation of CUI to Domain Relationships
113 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PSM sub-step 3: definition of navigationAn example of a complex rule
NAC LHS RHS
::=
NAC LHS RHS
::=
114 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
PSM: Concrete User Interface
A rrange F lightA rrange F light
D eterm ine P re fs
Airpor t N am e
C ity
C ountry
D e te rm ine Or ig in
D e te rm ine V ia
D e te rm ine D estina tion
Airpor t N am e
C ity
C ountry
Airpor t N am e
C ity
C ountry
D e te rm ine T im e
At D ate day m onth year
D epar t T im e
Arr iva l T im e
D eterm ine R eturn-T im e
At D a te day m onth year
D epar t T im e
Arr iva l T im e
Launch Sea rch
Proceed Paym en tSearch F ligh t
D eterm ine Budge t
C ur rency
U pper
Low er
F ligh t
Input C ard D eta ils
Se lect C ard T ype
Input C ard H o lder
Inpu t C a rd N um ber
Exp ira tion D a te
V isa
Am ex
M aster C ard
C on firm
Content to determine at run-time
Feedback of Check CardFeedback of Check Card
OK
Feedback of CheckCard
A rrange F lightA rrange F light
D eterm ine P re fs
Airpor t N am e
C ity
C ountry
D e te rm ine Or ig in
D e te rm ine V ia
D e te rm ine D estina tion
Airpor t N am e
C ity
C ountry
Airpor t N am e
C ity
C ountry
D e te rm ine T im e
At D ate day m onth year
D epar t T im e
Arr iva l T im e
D eterm ine R eturn-T im e
At D a te day m onth year
D epar t T im e
Arr iva l T im e
Launch Sea rch
Proceed Paym en tSearch F ligh t
D eterm ine Budge t
C ur rency
U pper
Low er
F ligh t
Input C ard D eta ils
Se lect C ard T ype
Input C ard H o lder
Inpu t C a rd N um ber
Exp ira tion D a te
V isa
Am ex
M aster C ard
C on firm
Content to determine at run-time
A rrange F lightA rrange F light
D eterm ine P re fs
Airpor t N am e
C ity
C ountry
D e te rm ine Or ig in
D e te rm ine V ia
D e te rm ine D estina tion
Airpor t N am e
C ity
C ountry
Airpor t N am e
C ity
C ountry
D e te rm ine T im e
At D ate day m onth year
D epar t T im e
Arr iva l T im e
D eterm ine R eturn-T im e
At D a te day m onth year
D epar t T im e
Arr iva l T im e
Launch Sea rch
Proceed Paym en tSearch F ligh t
D eterm ine Budge t
C ur rency
U pper
Low er
F ligh t
Input C ard D eta ils
Se lect C ard T ype
Input C ard H o lder
Inpu t C a rd N um ber
Exp ira tion D a te
V isa
Am ex
M aster C ard
C on firm
Content to determine at run-time
Feedback of Check CardFeedback of Check Card
OK
Feedback of CheckCard
Arrange FlightArrange Flight
Select Flight
Proceed Payment
Proceed PaymentProceed Payment
Input Card Details
Confirm Payment
Back to Arrange Flight
Search FlightSearch Flight
Determine Prefs
Launch Search
Flight
Back to Arrange Flight
Feedback of Check CardFeedback of Check Card
OK
Feedback of CheckCard
Determine PrefsDetermine Prefs
Determine Origin
Determine Destination
Determin Via
Determine Time
Determine Budget
Back to Search Flight
Launch Search
Determine OriginDetermine Origin
Back to determine prefs
Airport Name
City
Country
Input Card DetailsInput Card Details
Select Card Type
Input Card Holder
Input Card Number
Expiration Date
Visa
Amex
Master Card
Back to Proceed Payment
Determine DestinationDetermine Destination
Airport Name
City
Country
Back to determine prefs
Determine Return-TimeDetermine Return-Time
Back to determine prefs
At Date day month year
Depart Time
Arrival Time
Determine BudgetDetermine Budget
Back to determine prefs
Currency
Upper
Lower
Determine ViaDetermine Via
Back to determine prefs
Airport Name
City
Country
Determine TimeDetermine Time
Back to determine prefs
At Date day month year
Depart Time
Arrival Time
115 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: Platform adaptationwidget substitution (1)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><cuiModel name="MyModel"> <version modifDate="2004-03-24T17:09:17.402+01:00" xmlns="">7</version> <authorName xmlns="">Youri</authorName> <window height="500" width="600" name="Formulaire (2/5)" id="window_1"> <box relativeHeight="100" name="box1_0" id="box1_0"> <box type="vert" name="boxTodo" id="boxTodo"> ... <box type="horiz" name="box_2_2_2_1" id="box_2_2_2_1">
<textComponent defaultContent="Sexe" isBold="true" id="label_2"/><radioButton groupName=“grupo01" defaultContent="Femme"
defaultState="false" id="radiobutton_0"/><radioButton groupName="grupo01" defaultContent="Homme"
defaultState="true" id="radiobutton_1"/> </box> ... </box> </box> </window></cuiModel>
Excerpt for a UsiXML CUI specification
116 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (2)
117 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (3)
The UsiXML graph before applying any rule
118 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (4)
LHSRHS
NAC
Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons.
119 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (5)
Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons.
The UsiXML graph after applying the first rule
120 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (6)
LHS RHS
::=
Rule 2: Convert every radio button within the group “x” into an item for the comboBox “x” that we have just created thanks to Rule 1
121 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (7)
Rule 2: Convert every radioButton within the group “x” into an item for the comboBox “x”, we have just created.
The UsiXML graph after applying the second rule: radio buttons disappeared
122 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><cuiModel name="MyModel"> <version modifDate="2004-03-24T17:09:17.402+01:00" xmlns="">7</version> <authorName xmlns="">Youri</authorName> <window height="500" width="600" name="Formulaire (2/5)" id="window_1"> <box relativeHeight="100" name="box1_0" id="box1_0"> <box type="vert" name="boxTodo" id="boxTodo"> ... <box type="horiz" name="box_2_2_2_1" id="box_2_2_2_1">
<textComponent defaultContent="Sexe" isBold="true" id="label_2"/><comboBox id="comboBox001" name="label_3" isDropDown="true"> <item id="radiobutton_0" name="radiobutton_0" defaultContent="Femme"/> <item id="radiobutton_1" name="radiobutton_1" defaultContent="Homme"/></comboBox>
... </box> </box> </window></cuiModel>
Example: widget substitution (8)
Excerpt from the final transformated UsiXML specification
123 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example: widget substitution (9)
124 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What do we have so far?
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S
125 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Task and Domain
Abstract User Interface
Concrete User Interface
FinalUser Interface
Abstract User Interface
T1 RenderingT2
T3
Task and Domain
Abstract User Interface
Concrete User Interface
FinalUser Interface
Abstract User Interface
T1 RenderingT2
T3
Task and Domain
Abstract User Interface
Concrete User Interface 1
(2-D Desktop)
FinalUser Interface
T1
Rendering
T2
T3 Concrete User Interface 2(2-D small display)
Concrete User Interface 3
(auditory)
FinalUser Interface
FinalUser Interface
FinalUser Interface
Concrete User Interface
Task and Domain
Abstract User Interface
T4
Rendering
Rendering
Rendering
T5
T6 T7
Task and Domain
Abstract User Interface
Concrete User Interface 1
(2-D Desktop)
FinalUser Interface
T1
Rendering
T2
T3 Concrete User Interface 2(2-D small display)
Concrete User Interface 3
(auditory)
FinalUser Interface
FinalUser Interface
FinalUser Interface
Concrete User Interface
Task and Domain
Abstract User Interface
T4
Rendering
Rendering
Rendering
T5
T6 T7
Multiple development paths
126 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
A development library stores (in usiXML textual format) paths, steps and sub-steps definition and their associated transformation systems and transformation rules
Mapping the Methodological World onto the Transformation World
Development step
Development step
Development sub-step
Development sub-step
Development path
Development path
Transformation System
Transformation System
TransformationRule
TransformationRule
isComposedOf
isRealizedBy
isComposedOf
isComposedOf
*
1
*
1
1
*
1
0..1
Development step
Development step
Development sub-step
Development sub-step
Development path
Development path
Transformation System
Transformation System
TransformationRule
TransformationRule
isComposedOf
isRealizedBy
isComposedOf
isComposedOf
*
1
*
1
1
*
1
0..1
Methodological World
Development step
Development step
Development sub-step
Development sub-step
Development path
Development path
Transformation System
Transformation System
TransformationRule
TransformationRule
isComposedOf
isRealizedBy
isComposedOf
isComposedOf
*
1
*
1
1
*
1
0..1Transformation World
[Limbourg,2004]
127 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multiple development paths
Rule n
Transformation System 1
Rule 1
Rule 2
…
Rule n
Transformation System 2
Rule 1
Rule 2
…
Rule n
Transformation System ...
Rule 1
Rule 2
…
Rule n
Transformation System n
Rule 1
Rule 2
…
: when source terminates apply target
: execute development step
Development Step α
128 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
TransformiXML API/GUI
•API = set of transformations
<window><button>....
<window>
USIXML specification(initial)
::=
Transformation rules expressed in USIXML
<window><button>....
<window>
USIXML specification(resultant)
Transformation API
rules applied
<window><button>....
<window>
USIXML specification(initial)
::=
Transformation rules expressed in USIXML
::=
Transformation rules expressed in USIXML
<window><button>....
<window>
USIXML specification(resultant)
Transformation API
rules applied
129 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
From T&D to AUI
• TransformiXML TransformiXML
[Bouillon et al.,2005]
130 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Final user interface
• Two forms of UI rendering– Interpretation
• By run-time static analysis and direct rendering (InterpiXML & FormiXML)
– Code generation• By program synthesis (GrafiXML)• By generative programming (Angie)
– Feature model– Components assembling
UsiXML model:Abstract user
interface
UsiXML model:Concrete user
interface
RenderingFinal userinterface
UsiXMLmodels: task,
domain
Graphtransformations
Graphtransformations
Generativeprogramming
131 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
CC
F1F1
CC
F1F1 F2F2
CC
F1F1 F2F2
DependenciesDependencies
1.1.
2.2.
What is a Feature Model?
OptionalOptional ExclusiveExclusive
AlternateAlternate
[Schlee & Vanderdonckt,2004]
132 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example of a Feature Model
Alternate
[Schlee & Vanderdonckt,2004]
133 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Interpretation of a feature model
CC
F1F1 F2F2 F3F3
F2aF2a F2bF2b F3aF3a F3bF3b
CC
F1F1 F2F2 F3F3
F2aF2a F2bF2b F3aF3a F3bF3b
0 1 1
1 10 0
UsiXML SpecificationsUsiXML Specifications
<C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3></C>
<C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3></C>
ABAANGIE-Part
ABAANGIE-Part
[Schlee & Vanderdonckt,2004]
134 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
V
oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT();
virtual char* getClassName() const; virtual char getClassID () const;
ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);
ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);
ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);
V
oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT();
virtual char* getClassName() const; virtual char getClassID () const;
ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);
ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&);
ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL);
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
US
IXM
L S
peci
f ica
t ion
sU
SIX
ML
Spe
cif i
cat i
ons
ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL);
ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL)
ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL);
ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL)
Components void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl );
void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl );
void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& );
void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const;
void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT();
virtual char* getClassName() const; virtual char getClassID () const;
ImgProcessT<T>& operator+=(const ImgProcessT<T>&);I
void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl );
void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl );
void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& );
void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const;
void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const;
ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public: ImgProcessT(const char* const psz = "", Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT();
virtual char* getClassName() const; virtual char getClassID () const;
ImgProcessT<T>& operator+=(const ImgProcessT<T>&);I
Final codeImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL);
ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public:
ImgProcessT(const char* const psz = "", Pl* ppl = NULL);
ImgProcessT(const int&,
const int& nWidth = 20,
const int& nHeight = 20,
const int& nFillColor = 0,
ProgressLine* ppl = NULL
);
ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL);
ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL);
public:
ImgProcessT(const char* const psz = "", Pl* ppl = NULL);
ImgProcessT(const int&,
const int& nWidth = 20,
const int& nHeight = 20,
const int& nFillColor = 0,
ProgressLine* ppl = NULL
);
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL);
void channelRange (T*, Pl* ppl = NULL );
Generative Programming
[Schlee & Vanderdonckt,2004]
135 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Developed by Benjamin Michotte
GrafiXML consists of a user interface builder for designing a graphical user interface (GUI) independently of the platform and save it in UsiXML format language.
Features• Exports GUI in
– Java source (using Swing)– XHTML 1.0 Strict– XUL
• Works on Windows, Linux and MacOs X• Available in English, French and Spanish• Based on plugins
– Export– Import– Project management– Graceful degradation of user interfaces
• Allows creating context models
[Michotte,2004]
136 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Design Tab
Design windowDesign window
Components toolbar
Components toolbar
Components options
137 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Localisation of UIs
GrafiXML allows the user to create multi-language GUI
Support for mnemonics and
shortcuts
Support for mnemonics and
shortcuts
138 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Preview
At any time, you can preview the UI in the language you want
139 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
XML Editor
GrafiXML contains a XML editor which shows the UsiXML specification of your work
• You can edit yourself some part of the XML
Edit the UsiXML
Edit the UsiXML
Show attributes
Show attributes
A click on the tree highlights the
corresponding UsiXML
A click on the tree highlights the
corresponding UsiXML
140 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Plugins
GrafiXML works with plugins– Install / remove using the plug-in manager– Updated if needed using one click
Click on « add » to open the
downloader
Choose the plugins you want install
And install themDouble-click on
a plugin
And look at the plugin
informations
141 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What do we have so far?
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S
142 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Reverse engineering
• From FUI to CUI• From CUI to AUI, task & domain
143 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
From FUI to CUI
• Do you have the source code?– Markup languages (e.g., HTML): static code analysis (ReversiXML)
• Do you have the compiled code?– Programming languages (e.g., compiled C): resource file extraction and
static code analysis• Do you have the running application?
– Video analysis: computer vision• Do you have only the documentation?
– Image analysis: image processing
[Bouillon & Vanderdonckt,2004]
144 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
From compiled code
[Marion,2005]
145 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
The calculator
Begin VB.Form Calculator BorderStyle = 1 'Fixed Single Caption = "Calculator" ClientHeight = 2970 ClientLeft = 2580 ClientTop = 1485 ClientWidth = 3270 ClipControls = 0 'False BeginProperty Font name = "System" charset = 1 weight = 700 size = 9.75 underline = 0 'False italic = 0 'False strikethrough = 0 'False EndProperty Height = 3375 Icon = "CALC.frx":0000 Left = 2520 LinkMode = 1 'Source LinkTopic = "Form1" MaxButton = 0 'False ScaleHeight = 2970 ScaleWidth = 3270 Top = 1140 Width = 3390 Begin VB.CommandButton Number Caption = "7" Height = 480 Index = 7 Left = 120
<Window id=“1” name=“1”isVisible=“true” isEnabled=“true”defaultBorderTitle=“Calculator” borderWidth=“1”height=“358” width=“309”> <Box id=“2” name=“2” type=“verticall” isVisible=“true” isEnabled=“true”><button isEnabled=“true”…
USIXML
[Marion,2005]
146 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
From CUI to AUI, task & domain
• Re-engineering = reverse + forward
• Possible re-interpretation by QtkiXML
147 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example of Derivation Rules
• Examples:– For a textbox in HTML/WML
x Ts : x = input ٨ (x.type=“text” ٧ x.type=“password” ٧ x.type=NULL) Addnode (“textComponent”, idtext)
where idtext= NodeAmount(Tt)– For a table in HTML/WML
x Ts : x = table ٨ x.border>0 Addnode (“table”, idtable) where idtable = NodeAmount(Tt)
x Ts : x = table ٨ x.border=0 Addnode (“box”, idbox) where idbox = NodeAmount(Tt)
AddAttribute (idbox, “type”, “vertical”)
Input
Name=in1Maxlength=15Value=loginType=text
Element B
TextComponent
Name=in1Maxlength=15defaultContent=loginId=in1
Element Y
[Bouillon,2006]
148 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Derivation rules categories
• Rules can be classified into different categories, outlining the common issues in a RE process for various source UIs.– Element recovery– Attribute detection– Layout / temporal ordering analysis– Dialog recuperation– Hierarchy recovery– Multi-model transformations– Retargeting operations
Element 1 Element 2
Element 3
Horizontal box
Horizontal box
Ver
tical
box
Element 1 Element 2 Line Break Element 3
x Tt: x=bix ٨ ((rigthSibling(x)!=table ٧ rigthSibling(x)!=bix ٧ rigthSibling(x)!=cell ٧ rigthSibling(x)!=box) ٨ rigthSibling(x)!=NULL) CloneNode (rightSibling(x).id, idnew) ٨ RemoveNode (rightSibling(x), rightSibling(x).id) ٨ RemoveArc (ParentNode(rightSibling(x)), rightSibling(x)) ٨ AddArc(x.id, idnew) where idnew =∑ node Tt
Layout Analysis x to Tti ,y to Tt0: x = window ٨ (y=box ٧ y=window) ٨x.filename =y.targetfile
CloneNode(x.id, idnew, Tt0) where idnew =∑ node Tt0 ٨ RemoveNode(x, x.id) ٨ RemoveArc(ParentNode(x).id, x.id) ٨ z=root(Tti) ٨
Remove Node(z,z.id) ٨ AddArc(y.id, idnew) x to Tti ,y to Tt0: x = vocalGroup ٨ y=VocalGroup ٨x.filename =y.insertFile
CloneNode(x.id, idnew, Tt0) where idnew =∑ node Tt0 ٨ RemoveNode(x, x.id) ٨ RemoveArc(ParentNode(x).id, x.id) ٨
z=root(Tti) ٨ Remove Node(z,z.id) ٨AddArc(y.id, idnew)
Multi-model transformations
149 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
RE of Web Pages: ReversiXML
• Written in PHP5• On-line RE• About 12,000 LOC• Set of libraries
150 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
RE of Phone Interfaces
• Major working hypotheses– Static RE of WML 1.1 and voiceXML 2.0 up to the CUI in USIXML
1.4.6– Recovery of the layout and hierarchy/temporal ordering.– Dialog-Navigation analyzed (but not scripts)– No retargeting operations
• Similar method to HTML applied for WML & VoiceXML reverse engineering– Description of languages meta-models and derivation rules.
151 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
RE of WML Phone Interfaces
• Example– Prototype using XSLT + XPATH– Local process allowing no design alternatives
WML<p> Yahoo! ID:<input name="login" value="" format="*m" />Password:<input type="password" name= "passwd" value="" format="*m" /><anchor title="OK">Submit<go method="post" href="/raw?">
USIXML<textComponent id="textComponent_1" glueHorizontal="left“ isVisible="true" isEnabled="true" defaultContent="Yahoo! ID:<textComponent id="textComponent_2" glueHorizontal="left" defaultContent="" isEditable="true" isVisible="true"/>
[Cui,2005]
152 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Results of the Evaluation
• Case studies illustrating various HTML RE techniques
• Study of reengineering of the Sedan-Bouillon Website thanks to two FE tools: Teresa and QtkXML
153 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Results of the Evaluation
• With Teresa– RE applied without retargeting– USIXML CUI model used as input in Teresa that performs
translations for another context of use– Produces the Web site designed for Pocket PC (XHTML)
154 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Results of the Evaluation
• With QtkiXML
Original UIReengineered UI
155 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What do we have so far?
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
Abstraction
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S
Let’s go for adaptation
[Calvary et al.,2003]
156 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Environment T
What do we want to get?
Final userInterface T
Concrete userInterface T
Task and Domain T
Abstract userInterface T
T=Target context of use
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
Abstraction
Reflexion
Translation
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S Platform TUser T
[Calvary et al.,2003]
157 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rules for Graceful Degradation
• Constitution of an original corpus of rules• Typology of rules, following the CAMELEON framework:
– (Rules at the Final User Interface level)– Rules at the Concrete User Interface level– Rules at the Abstract User Interface level– Rules at the Tasks & Concepts level
• Structured description of these rules
[Florins,2006]
158 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rules at the Concrete UI level
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
[Florins,2006]
159 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
Rules at the Concrete UI level
[Florins,2006]
160 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
Rules at the Concrete UI level
[Florins,2006]
161 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
Rules at the Concrete UI level
[Florins,2006]
162 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
Rules at the Concrete UI level
[Florins,2006]
163 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Transformation of graphical objects– Resizing rules– Modification rules– Substitution rules– Removal rules
• Transformation of graphical relationships– Reorientation rules– Moving rules
Rules at the Concrete UI level
[Florins,2006]
164 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Spitting rules• Consist in breaking the initial UI into chunks
+ adding transitions
Rules at the Abstract UI level
[Florins,2006]
165 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Important because:– Difficult and significant step: generates important
changes into the very structure of the UI– Appreciated by users
• Supporting algorithms developed during the thesis.
• Originality: involve UI description at several abstraction levels– Can be rely on the sole CUI level – Can exploit information from the AUI / task models.
Rules at the Abstract UI level
[Florins,2006]
166 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Source interface (in the graphical editor GrafiXML)
(b)
Execution of the splitting rule
(a)
box box
box
• Application of the rule using CUI level information
Rules at the Abstract UI level
[Florins,2006]
167 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
• Application of the rule using task level information
Rules at the Abstract UI level
[Florins,2006]
168 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Rules at the Tasks&Concepts level
• Task deletion• Information summarization• …
169 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GD Plug-in (4)
•Sections of rules
[Florins et al.,2006]
170 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GD Plug-in (4)
•Sections of rules
171 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GD Plug-in (4)
•Sections of rules
172 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GD Plug-in (5)
•Rules selection / parameters
173 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
GD Plug-in (6)
•Results
174 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multi-platform for Emergency
[Amouh et al.,2005]
175 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multi-platform for Emergency
• Three platforms– Pocket PC– Desktop PC– Wall Screens
176 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multi-platform for Emergency
• Model and method– Design the reference screen first– Refine the others screens later
• By applying graceful degradation• By applying transformation techniques
177 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Graceful degradation
[Florins & Vanderdonckt,2004]
178 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Transformation rules
179 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Transformation rules
Add all >>
Add >
<< Remove all
< Remove
>>
>
<<
<
>
<
Group box
Item 1Item 2Item 3Item 4
Item 5Item 6Item 7Item 8
Item 1Item 2Item 3Item 4Item 5Item 6Item 7
Item 1
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Item 7
180 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Transformation rules
181 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Considering the context of use in designing user interfaces
• Context of use = triple– User– Platform– Environment
• Each time one of these facets change, the context changes
Generation for multiple platforms
[Plomp & Keranen,2004]
182 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Context model
183 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006Target Applications,
Domains Notations and tools User Interface Interaction
Technique
2006
TOD
AY
Adapted from[Palanque,2002]
2007
No Interaction TechniqueAutomated, batch systems Nothing
Character UIsBusiness applications Screen definitions
Graphical UserInterfaces
Information systems Entity-relationshipAttribute modelState-transition diagrams
Multi-platform User Interfaces
Web, Java applications
Task model, contextmodel, UML,…
Virtual Reality User Interfaces3D Applications Scene model
184 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Virtualisation of UIs
• Going from 2D to 3D– Mapping widgets in 2D to 3D
[Vanderdonckt et al., 2004]
185 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Developed by Donatien Grolaux
• Problem: how to design a UI that takes care of multiple displays?
• Solution: Migration = DEMIPLAT• DistriXML = software architecture for migrating UIs from one
platform to another at run-time
Pencil
Palette
Painting
Paintingtool
[Grolaux & Vanderdonckt,2005]
186 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Why take care of multiple displays?
Effects of Display Size on Task Times
0
20
40
60
80
100
120
140
160
DISPLAY
Avera
ge T
ask
Tim
e (
Seco
nds)
Small
Large
[Czerwinsky,2005]
187 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Why take care of multiple displays?
The tasks were easy to perform
0
1
2
3
4
5
Small Large
Display Size
Avera
ge R
ati
ng (
1=
Dis
agre
e,
5=
Agre
e)
[Czerwinsky,2005]
188 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Why take care of multiple displays?
I was satisfied with the ease of windows layout
012345
Display Size
Ave
rage
Rat
ing
(1=D
isag
ree,
5=Agr
ee)
Small
Large
[Czerwinsky,2005]
189 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Demonstration
[Grolaux & Vanderdonckt,2005]
190 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Demonstration using two displays from two different computers
191 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example using a Pocket PC
192 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UI Migration: DEMIPLAT
• Detach
193 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UI Migration: DEMIPLAT
• Detach - Migrate
194 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UI Migration: DEMIPLAT
• Detach - Migrate - Plastify
195 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
UI Migration: DEMIPLAT
• Detach - Migrate - Plastify - Attach
196 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
This is not a floating toolbar
Process
197 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Computer B
Process
This is migration !
Process
Computer A
198 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
User model
• User population = hierarchy of user stereotypes• User stereotype = set of parameters
– Context independent • Age, gender, motor skills, general preference,…
– Context dependent• Preference for certain contents• Interaction history• Can be initiated by the corresponding context independent parameter• Can overwrite this parameter
199 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Why user preference is important?
• End users tend to manipulate more efficiently the user interface they prefer– When end users notice significant differences in their own performance and when
they consider performance as important, then preference and performance are highly correlated.
– Preference is more important than performance
[Johnsgard,1995]
If information density is low If information density is high Simple choice Multiple choice
Input with exte n-sion
Predefined sele c-tion
200 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example of preference-based design
• Goal = to input the temperature of a human body– Solution n°1: edit box– Solution n°2: list box– Solution n°3: drop down list– Solution n°4: field with scroll bar– Solution n°5: thermometer
201 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Users
• Properties of the user also constrain interaction.– Expertise
• Does the user understand this interaction technique?– Privileges
• What tasks is the user allowed to perform?– Preferences
• What tasks is the user likely to perform?
202 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Modeling a range of designs
• UI modeling languages must model a range of design possibilities to account for different contexts
• This could be accomplished by modeling preferences instead of decisions
203 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
A Spectrum of Preference Relations
• We describe a spectrum of preference relations– Less flexible relations are easier to implement but less powerful– More flexible relations can model highly adaptive UIs, but are
difficult to implement
[Eisenstein,2001]
204 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
The Spectrum
• Concrete Preferences– Bindings– Simple Preference– Ordered Preference
• Abstract Preference• Design guideline
205 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Modeling preferences in UsiXML
• UsiXML = User Interface eXtensible Markup Language (http://www.usixml.org)
User1
Domain2
Domain3
Presen-tation4
Presen-tation5
User2
206 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Bindings
• Bindings are simple one-to-one mappings– Task A is implemented by interactor X– A is the condition– X is the target
• Bindings are our point of departure,– Most current modeling formalisms support only bindings
207 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
<PREFERENCE ID="G1"><CONDITIONS>
<RELATION_STATEMENT DEFINITION="condition" REFERENCE="U1"/></CONDITIONS><TARGETS>
<RELATION_STATEMENT DEFINITION="target" REFERENCE="P1"/></TARGETS>
</PREFERENCE>
Binding example
• User U1 prefers presentation element P1
208 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Simple Preferences
• Simple Preferences are many-to-one mappings– For user U, task A is implemented using interactor X– Conditions: U, A– Target: X
209 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
<PREFERENCE ID="G1"><CONDITIONS>
<RELATION_STATEMENT DEFINITION="condition" REFERENCE="U1"/><RELATION_STATEMENT DEFINITION="condition" REFERENCE="DM1"/>
</CONDITIONS><TARGETS>
<RELATION_STATEMENT DEFINITION="target" REFERENCE="P1"/></TARGETS>
</PREFERENCE>
Simple preference example
• User U1 prefers presentation element P1 for representing domain model DM1
210 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Bindings
• Bindings are simple one-to-one mappings– Task A is implemented by interactor X– A is the condition– X is the target
• Bindings are our point of departure,– Most current modeling formalisms support only bindings
211 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
<PREFERENCE ID="G1"><CONDITIONS>
<RELATION_STATEMENT DEFINITION="condition" REFERENCE="U1"/></CONDITIONS><TARGETS>
<RELATION_STATEMENT DEFINITION="target" REFERENCE="P1"/></TARGETS>
</PREFERENCE>
Binding example
• User U1 prefers presentation element P1
212 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Ordered Preferences
• An ordered preference is a type of many-to-many mapping– For user U, task A should be implemented by one of the following
interactors: X1, X2, or X3• Multiple targets, in order of preference: X1, X2, X3• Targets can be assigned “priorities”
– This specifies the “level of preference”– E.g.: X1 = 100, X2 = 95, X3 = 20
» Means choose either X1 or X2, it doesn’t matter much– E.g.: X1 = 100, X2 = 35, X3 = 35
» Try hard to choose X1
213 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Ordered preference example
• User U1 prefers presentation element P1 to presentation element P2, for representing domain model DM1<PREFERENCE ID="G1">
<CONDITIONS><RELATION_STATEMENT DEFINITION="condition" REFERENCE="U1"/><RELATION_STATEMENT DEFINITION="condition" REFERENCE="DM1"/>
</CONDITIONS><TARGETS>
<RELATION_STATEMENT DEFINITION="target" REFERENCE="P1">< ATTRIBUTE_STATEMENT DEFINITION="priority">
100 </ATTRIBUTE_STATEMENT>
</RELATION_STATEMENT><RELATION_STATEMENT DEFINITION="target" REFERENCE="P2">
<ATTRIBUTE_STATEMENT DEFINITION="priority">50
</ATTRIBUTE_STATEMENT></RELATION_STATEMENT>
</TARGETS></PREFERENCE>
214 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Summary of Concrete Preferences
• All of the previous are concrete preferences– They model the target directly– Can be combined with a model of the contextual constraints:
• Example: For task A, – interactors X1, X2, and X3 are preferred, in order– But X1 cannot be implemented on current device– Therefore, X2 is used
215 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Abstract Preferences
• Abstract preferences do not model the target directly• They model characteristics of the target• Abstract preferences have criteria for making mappings
– Preferential, stating which is preferred– Logical, stating which is allowed
216 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Example of Abstract Preference
• User U prefers the interactor– Occupying the least screen space
• This is a preferential criteria• The behavior is to minimize the value of the criteria
– But not less than 20 pixels wide • This is a logical criteria
217 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Abstract preference example
• User U1 prefers the presentation element that occupies the least screen space<PREFERENCE ID="G1">
<CONDITIONS><RELATION_STATEMENT DEFINITION="condition" REFERENCE="U1"/>
</CONDITIONS><TARGETS>
<RELATION_STATEMENT DEFINITION="target" REFERENCE="P1"/><RELATION_STATEMENT DEFINITION="target" REFERENCE="P2"/>
</TARGETS><CRITERIA>
<RELATION_STATEMENT DEFINITION="criteria" REFERENCE="screen space"><ATTRIBUTE_STATEMENT DEFINITION="criteria type">
preferential</ATTRIBUTE_STATEMENT><ATTRIBUTE_STATEMENT DEFINITION="criteria behavior">
minimum</ATTRIBUTE_S TATEMENT>
</RELATION_STATEMENT></CRITERIA>
</PREFERENCE>
218 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Multiple Preferential Criteria
• Preferential criteria may have “priorities”– This specifies their relative importance– E.g.: “Prefer presentation elements that require few clicks and are
small, but the criteria of having few clicks is most important”– Here, the “number of clicks” criteria has a higher priority than the
“size” criteria
219 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Design Guidelines
• Abstract preferences consider criteria that relate to the targets
• Design guidelines consider criteria of any object in the UI model
• Frequently take the form of If-Then statements• Example: “If the experience level of User U is less than
“expert”, then map to interactor X1”
220 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Chaining Abstract Preferences
• Design guidelines can have other preference relations as their targets:– “If User U is not an expert, then select the navigational structure
with the least cognitive complexity.”– In this case, the target of the design guideline is an abstract
preference• The abstract preference has a single, preferential criteria:
– Minimize cognitive complexity
221 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Chaining Design Guidelines
• Design guidelines can also target other design guidelines
• This can create complex logical formulae:– “If User U is an expert, and the platform is a PDA, then
choose one of X1, X2, or X3, whichever is smallest”
• Such formulae can model highly adaptive design phenomena
222 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
How to argue for preference?
• Use the QOC notation
Question?
Criteria 1
Criteria 2
Criteria j
Criteria m
Option 1
Option 2
Option i
Option n
= negatively affects= positively affects
[McLean & Belotti,1998]
223 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Preference example
Problem
Solution 1
Solution 2
(RE1)
(RE2)
(A1)
(A2)
(A3)
(A4)
(A5)
(A6)
Standard compatibility (e.g. Windows)
Cognitive respect
Minimal actions
Dialog representativity
Prompting
Clarity
suggests
suggests
avoids
contradicts
contradicts
respects
= for= against
(A1) = drop down list requires less space than radio buttons(A2) = some possible values become obsolete when they are infrequently used(A3) = drop down list shows better then current value than the possible values(A4) = radio buttons are faster and easier to manipulate than drop down list(A5) = radio buttons are recommended in Microsoft Windows style guide(A6) = radio buttons do not allow users to change the values
drop down list
radio buttons
(A4) is contradicted by (A2) and (A3): the widget should be more used for output than input.(A6) is contradicted by (A3): it is better to present all possible values than only one at a time(A5) is an autority argument, and can fall in front of (A1)+(A2)+(A3)
[Vanderdonckt,1997]
224 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Theory or argumentation
Design problem Design option parameter(parameter, value)
Is described by
Is solved by
Accepted solution Set of admissiblesolutions
Solution 1 Solution 2 Solution n
Counter-argument(rebuttal, counter-claim)
Argument(ground, typically data)
can be rejectedbecause of
Can be acceptedthanks to
Is justified byDesign claim Warrant
Is reinforced by
Qualifiersatisfied
unsatisfied
Is related to
corresponds to
Is justified by
Is qualified by
[Toulmin,1975][Perelman, 1978]
225 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mixed reality
•Mixed reality = real world + digital world
226 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Mixed reality for games
[Alterface,2006]
227 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Adaptive IS composition based on focus of interest
Mixed reality in surgery
• No representation for– Tasks in real or virtual world– Tasks independent of user interaction e.g. tracking object of
interest– Various contexts of use in Mixed Reality Systems i.e. changing
point-of-view
[Trevisan et al.,2004]
228 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Composing the MIS
Real CIOPatient head
Virtual CIOsAnatomic organs
Virtual CIOs Annotations
Primary object of interest
229 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Annotation placement strategy
• No overlapping annotations for readability• Highest priority of virtual object (3)• Medium priority of real object (2)• Low priority of background (1)• Annotations placed from peripheral to central zones• Other objects as part of Background
230 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Context-aware MultimodalAnnotation Rendering
(a) Background context (b) Background/Real Object context
(c) Virtual Object context
- what information to display- annotation positions- modality switch
User focus controls adaptive presentation management
231 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Context model in mixed reality
• A context model describes all the entities that may influence carrying out the interactive task of user with the intended UI – Environment model - such attributes may be physical (e.g.,
lighting conditions), psychological (e.g., level of stress), and organization (e.g., location and role definition in the organization chart).
– Platform model – couple of software/hardware, such atributes may be screen resolution/size
– User model – stereotypes, sub-stereotypes (profiles)
232 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Plasticity Domains in the MIS
Plasticity thresholds and context boundary valuesInter-context continuity
Perceptive, cognitive, functional intra-context continuity and inter-context continuityIntra-context / inter-context frequency and frequency switch
Refined Metrics for Plastic MR systems
Plasticity analysis and usability evaluation
233 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Environment T
What do we have now?
Final userInterface T
Concrete userInterface T
Task and Domain T
Abstract userInterface T
T=Target context of use
Concrete userInterface S
Final userInterface S
Task and Domain S
Abstract userInterface S
S=Source context of use
Reification
Abstraction
Reflexion
Translation
UsiXMLunsupported
model
UsiXMLsupported
model
User S Platform S Environment S Platform TUser T
234 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Conclusion
• User model is described by parameters– Context independent
• Age, gender, motor skills, general preference,…– Context dependent
• Preference for certain contents• Interaction history• Can be initiated by the corresponding context independent parameter• Can overwrite this parameter
• Preference specification– Is stored in user model– Can be processed by « critiquing tools » based on argumentation, QOC
notation
235 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Evaluation
0
2000
4000
6000
8000
10000
12000
14000
16000W
ord
KB
KB
Pa
rse
r
Qu
ery
Ne
two
rk o
ps.
UI
sta
tes
Pre
sen
tatio
n
Act
ion
s
Up
da
te
Inte
ract
ion
s
Application logic User interface
Generated code
Models
[Skezely,1996]
236 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Coverage
• How to improve our MDA-based method?– High ceiling– Low threshold– Wide walls
Ca
pa
bili
ties
Resources (time, experience,…)
100%
50%
Ceiling
Threshold
Firs
t gen
erat
ion
Sec
ond
gene
ratio
n
MD
A C
AS
E to
ols
Inte
rfac
e bu
ilder
s an
din
tegr
ated
env
ironm
ents
Resource win for applicationssupported by MDA-compliant tools
Resource win for applicationssupported by first-generation
237 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
What’s the User Interface language?
• Today– Programming languages: Visual Basic, Java, C++,…– Markup languages: HTML, XUL,…
• Tomorrow– Multimodal interfaces: XHTML + VoiceXML = X+V– Vectorial interfaces: SVG, LZX– Virtual reality interfaces: VRML97, XVRML– 3D user interfaces: X3D, MEL– Tabletop interface: OpenGL– Many new modalities are arriving
• Conclusion– It is impossible to evolve/survive without any modelling approach,
even for user interfaces– It is required to progress also on the design process
238 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Conclusion: the big picture of MDA
UsiXML model:Abstract user
interface
UsiXML model:Concrete user
interface
Rendering
Final userinterface
UsiXMLmodels: task,
domainGenerative
programming
Graphtransformations
Graphtransformations
Derivation rules
IdealXML
ReversiXML
FlashiXMLQtkXMLJaviXML
VisualiXML
TransformiXML
GrafiXMLVisiXML
SketchiXMLFormiXML
KnowiXML
MethodiXML
239 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006Target Applications,
Domains Notations and tools User Interface Interaction
Technique
The Invisible UI
No more programming: only models
All Applications 2020
2006
TOD
AY
2008
Adapted from[Palanque,2002]
2007
No Interaction TechniqueAutomated, batch systems Nothing
Character UIsBusiness applications Screen definitions
Graphical UserInterfaces
Information systems Entity-relationshipAttribute modelState-transition diagrams
Multi-platform User Interfaces
Web, Java applications
Task model, contextmodel, UML,…
Virtual Reality User Interfaces3D Applications Scene model
Mixed RealityUser Interfaces
Command andcontrol systems, games
World models
Tangible UIsEmbodied UIsPhysical
modelsEmbarked syst
240 Polytechnic University of Valencia – Doctoral Course, Valencia, November 2006
Thank you very much for your attention
For more information and downloading,http://www.isys.ucl.ac.be/bchi
http://www.usixml.orgUser Interface eXtensible Markup Language
http://www.similar.ccEuropean network on Multimodal UIs
Special thanks to all members of the team!