Are Sensory EffectsReady for the World Wide Web?

Christian Timmerer, Markus Waltl, and Hermann Hellwagner

Klagenfurt University (UNIKLU) Faculty of Technical Sciences (TEWI)

Department of Information Technology (ITEC) Multimedia Communication (MMC)

http://research.timmerer.com http://blog.timmerer.com mailto:christian.timmerer@itec.uni-klu.ac.at

20 May 2010

Acknowledgments. This work was supported in part by the European Commission in the context of the NoEINTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), and the ALICANTE project (FP7-ICT-248652).

Outline

• Background / Introduction

– MPEG-V Media Context and Control– Concept of MPEG-V Sensory Information– Sensory Effect Description Language (SEDL)– Software/Hardware components: SEVino, SESim, SEMP, and

amBX+SDK

• Are Sensory Effects Ready for the World Wide Web?

– How to embed SEM into Web documents?– How to interpret SEM within a browser?– How to synchronize with audio/video content of the Web document?– How to extract color information automatically?

• Conclusions and Future Work

2010/05/20 2Christian Timmerer, Klagenfurt University, Austria

MPEG-V: Media Context and Control (MCC)

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 3

System Architecture

Pt. 1: Architecture

Pt. 3: Sensory Information

Pt. 4: Virtual World Object Characteristics

Pt. 2: Control Information

Pt. 6: Common Types and Tools

Pt. 7: Conformance and Reference Software

Pt. 5: Data Formats for Interaction Devices

http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V

Concept of MPEG-V Sensory Information

• Consumption of multimedia content may stimulate also other senses– Vision or audition– Olfaction, mechanoreception, equilibrioception, thermoception, …

• Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 4

… giving her/him the sensation of being part of the particular media➪ worthwhile, informative user experience

Sensory Effect Description Language (SEDL)

• XML Schema-based language for describing sensory effects– Basic building blocks to describe, e.g., light, wind, fog, vibration, scent– MPEG-V Part 3, Sensory Information– Adopted MPEG-21 DIA tools for adding time information (synchronization)

• Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)– Extensibility: additional effects can be added easily w/o affecting SEDL– Flexibility: each application domain may define its own sensory effects

• Description conforming to SEDL :== Sensory Effect Metadata (SEM)– May be associated to any kind of multimedia content (e.g., movies, music,

Web sites, games)– Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate

mediation device

➪ Increase the experience of the user➪ Worthwhile, informative user experience

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 5

Sensory Effect Description Language (cont’d)

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 6

SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|

Effect|ReferenceEffect)+

Declarations ::= (GroupOfEffects|Effect|Parameter)+

GroupOfEffects ::= timestamp EffectDefinition

EffectDefinition (EffectDefinition)*

Effect ::= timestamp EffectDefinition

EffectDefinition ::= [activate][duration][fade][alt]

[priority][intensity][position]

[adaptability]

SEVino, SESim, SEMP, and amBX

amBX (Ambient Experience) system + SDK• Two fan devices, a wrist rumbler, two sound

speakers, a subwoofer, two lights, and a wall washer

• Everything controlled by SEM descriptionsexcept light effect

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 7

Annotation Tool: SEVino Simulator: SESimPlayer: SEMP

Are Sensory Effects Ready for the World Wide Web?

• How to embed SEM into Web documents?

• How to interpret/parse SEM within a browser?– Directly via JavaScript

– Browser plug-in

• How to synchronize with audio/video content of the Web document?– HTML5’s video and audio elements and corresponding

DOM attributes

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 8

<link rel="alternate" type="text/xml”

href="http://server.org/sem.xml”

media="light, wind, vibration"/>

Are Sensory Effects Ready for the World Wide Web? (cont’d)

• Synchronization with audio/video (cont’d)– currentTimeDOM attribute: current playback

position in seconds– timeupdateevent: indicates that the current

playback position changed as part of normal playback

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 9

Are Sensory Effects Ready for the World Wide Web? (cont’d)

• How to extract color information automatically?

– Used for controlling ambient lights

– currentFramedoes neither exist nor is foreseen in HTML5

– canvas element allows for manipulating the display’s content

(+) Pro: extract the pixel information from the display

(-) Con: needs to be added explicitly during authoring

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 10

Conclusions

• Introduction to MPEG-V Sensory Effects

• Are Sensory Effects Ready for the World Wide Web? – Yes, they are!

• Future work

– Implementation is ongoing

– Subjective quality assessments

2010/05/20 Christian Timmerer, Klagenfurt University, Austria 11

Thank you for your attention

... questions, comments, etc. are welcome …

Ass.-Prof. Dipl.-Ing. Dr. Christian TimmererKlagenfurt University, Department of Information Technology (ITEC)

Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIAchristian.timmerer@itec.uni-klu.ac.at

http://research.timmerer.com/Tel: +43/463/2700 3621 Fax: +43/463/2700 3699

© Copyright: Christian Timmerer

122010/05/20 Christian Timmerer, Klagenfurt University, Austria