Download - Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB Paderborn, Germany
Cooperative Computing &Communication Laboratory
Steffen Bleul, Wolfgang Mueller, Robbie Schäfer Paderborn University/C-LAB
Paderborn, Germany
Multimodal Dialog Description for Mobile Devices
© Siemens AG and Paderborn University 2004
Motivating Example
Control and monitoring of electronic devices
WinampMediaplayer
PC User Interface
Problem: UI does not scale for other platforms / devices / modalities
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
MIRS - Multimodal Interaction and Rendering System
Dialog and Interface Specification Language - DISL• Structural model
Definition of generic widgets (no specific modality)(inherited from UIML with ideas from Paterno, Plomp...)
• Control/Dialog modelVariables, Rules, Actions, Events(inherited from Dialog Specification Notation – DSN)
UI Spec(DISL)
S-DISL
XSLT
HW &User
Profile
S-DISL Interpreter
abstract to concretemapping
RendererRendererRendererRendererRenderer
Apps
Server
(Mobile)Client
httptcp
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Dialog and Interaction Specification Language - DISL
Modification of UIML Interface Section
• structure - widget organization
• Generic widget (modality independent widget)• New generic widget elements
• style - properties
• defines properties of generic widgets
• behavior - user interaction
• defines behavior for generic widgets• extension for advanced state-oriented dialog specification• states• rules• events• communication
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
1. Container Elements- variablefield- textfield- genericfield- widgetlist
- command- confirmation- genericcommand
2. Interaction Elementsb. Misc
- variablebox- textbox- genericbox- choice
a. Input
DISL – Structure Elements
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
DISL – Structure/Style/Behavior
<structure> <widget id="Play" generic-widget="command"/> ...</structure><style> <part generic-widget id="Play" >
<property id="title">Title</property><property id="description">
This is a Remote Control for Audio Playing Devices</property><property id=“help“>Press Button to Start Playing</property><property id=“visible“ >true</property><property id=“selected“ >false</property><property id=“activated“>false</property>
</part></style><behavior> ...</behavior>
<interface ...>
</interface>
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
DISL - Behavior<interface id=“MediaPlayer" state=“init">… <behavior>
<variable ...> ... </variable> ...
<rule ...> <condition>...</condition>
<event...> ...</event></rule> ...<transition ...>
<if-true .../><action>
<statement ...> </statement>...
</action></transition>
... </behavior></interface>
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Events:TogglePowerSetVolumeToggleMutePlayTitleStopTitleTogglePauseNextTitle...
States:Power {#isOn,#isOff}Volume {#0,#1,…,#100}Mute {#MuteOn,#MuteOff}Playing {#PlayingOn,#PlayingOff}Paused {#PauseOn,#PauseOff}...
Dialog Specification Notation (DSN)Basic Concepts
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Rules:
Power 1. #isOn TogglePower #isOffPower 2. #isOff TogglePower #isOn
Mute 1. #isOn #MuteOn ToggleMute #MuteOffMute 2. #isOn #MuteOff ToggleMute #MuteOn
Playing 1. #isOn PlayTitle #PlayingOn #PauseOffPlaying 2. #isOn NextTitle #PlayingOn #PauseOffPlaying 3. #isOn PreviousTitle #PlayingOn #PauseOffPlaying 4. #isOn #PlayingOn StopTitle #PlayingOff #PauseOff
Paused 1. #isOn #PausedOn #PausedOff #PlayingOnPaused 2. #isOn #PausedOff #PlayingOn #PausedOn #PlayingOff
Dialog Specification Notation (DSN)Basic Concepts (cont‘d)
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
States: Power {#isOn,#isOff}Events: TogglePower
DSN to DISL
<rule id="isOn"> <condition> <equal> <variable id=“Power">true</variable> </equal> </condition></rule>
<variable id="Power" type="boolean">false</variable>
<rule id="TogglePower"> <condition> <equal> <property generic-widget="TogglePower" id="selected">true</property> </equal> </condition></rule>
<rule id="isOff"> <condition> <equal> <variable id=“Power">false</variable> </equal> </condition></rule>
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Rules:
Power 1. #isOn TogglePower #isOff
DSN to DISL
<transition> <if-true rule-id="TogglePower" expression="and"> <if-true rule-id="isOn"/> </if-true> <action> <call source="http://localhost:8080/mediaplayer/servlet" id="tryoff" synchronized="yes" timeout="5s"> <parameter id="request"> <value-of>turnoff</value-of> </parameter> </call> <statement> <variable-content id=“isOn“><const>false</const></variable> </statement> </action></transition>
2 conditions to be true
2 actions to execute
- http request with 1 parameter
- 1 variable assignment
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
<transition> <if-true ...> ... </if-true> <action> <interface id="VolumeControl"/> </action></transition>
DISL – Additional Features
Change of Interface at Runtime
More Complex Expressions<condition> <op operator=“and“> <op-expression expression=“greaterthan“> ... </op-expression> <op-expression expression=“lessthan“> ... </op-expression> </op></condition>
<transition> <event id=“checkstatus“ repeat=“yes“ time=“5s“> <action> <call source=„...“ synchronized="yes" timeout="5s"> </action> </event></transition>
Periodic Events
Content of virtually any element can be modified
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Current Implementation
UI Spec(DISL)
S-DISL
XSLT/Saxon ProcessorS-DISL Interpreter
J2ME/MIDP1.0
AWT-Renderer(J2ME/
MIDP1.0)
PCTomcat
http-server
http
kXML Parser
Siemens M55
Now with Bluetooth support!!
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion
© Siemens AG and Paderborn University 2004
Summary & Future WorkSummary
• XML-based specification language for general UIs• advanced state-oriented dialog specification
by variables, rules, actions, events(basic concepts inherited from Dialog Specification Notation – DSN)
• definition of generic widgets NOT for a specific modality, e.g., text, graphics, speech(basic concepts inherited from UIML, Paterno, Plomp...)
• Implementation for Java enabled phones
Future Work• use shorthand vocabulary• Better structuring through object orientation (step from DSN to
ODSN)• Cooperation with high level modelling techniques (bridging the gap
between e.g. task models and rendered UI) • Proof of concept with more modalities (Speech and Gesture)
Motivation
MIRS
DISL
Implemen-tation
Summary &Conclusion