synchronizationcs778/ralf/10c-sync.pdf · 10c-sync.fm215.february.01 scope contents ©r....
TRANSCRIPT
u-darmstadt.de Fax. +49 6151 16-3052
, Dept. of Computer Science51 166151,M.tu-darmstadt.de Fax. +49 6151 166152
ogy
10C-sync.fm 1 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser Multimedia-Systems:
Synchronization
Prof. Dr.-Ing. Ralf SteinmetzProf. Dr. Max MühlhäuserMM: TU Darmstadt - Darmstadt University of Technology,
Dept. of of Computer ScienceTK - Telecooperation, Tel.+49 6151 16-3709,Alexanderstr. 6, D-64283 Darmstadt, Germany, [email protected]
RS: TU Darmstadt - Darmstadt University of Technology,Dept. of Electrical Engineering and Information Technology
KOM - Industrial Process and System Communications, Tel.+49 61Merckstr. 25, D-64283 Darmstadt, Germany, Ralf.Steinmetz@KOGMD -German National Research Center for Information Technolhttc - Hessian Telemedia Technology Competence-Center e.V
User Interfaces
.Synchro-nization
GroupCommuni-
cations
Programming
ms Communications
ice Networks
ession
Video Audio
10C-sync.fm 2 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Scope
Usa
ge Applications
Learning & Teaching Design
Ser
vice
s ContentProcess-
ing
Docu-ments
Security ..S
yste
ms Databases
Media-Server Operating Syste
Opt. Memories Quality of Serv
Bas
ics Computer
Archi-tectures
Compr
Image &Graphics
Animation
zation?
zation
nt
10C-sync.fm 3 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Contents
1. Motivation - What is (Multimedia) Synchroni
2. Synchronization and Multimedia
3. Reference Model for Multimedia - Synchroni
4. Synchronization in a Distributed Environme
5. Synchronization Techniques
6. Case Studies
Synchronization?
ion 4
ntent
10C-sync.fm 4 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
1. Motivation - What is (Multimedia)
We concentrate on:• Temporal relationship
• New flavor in the context of multimedia• Essential for communications
Applicat
Application 2
Application 3
Application 1
Space
Time
Co
heetame data
d update
e related views.
the use of
10C-sync.fm 5 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Content Relations
Dependence of media objects on data
Examples:• A graphic that visualizes data from a spreads• Two graphics showing different views of the s
Explicit definition of dependences for automate• Only the data are edited• All views of the data are
• generated automatically• cannot be edited directly
• An update of the data triggers an update of th
Implementation of content relations is based on• common data structures or• common object interfaces
tation of a media object
entation
gned to this frament may be
cts:d in a windowon a stereo output device
vatars are positioned inith talking avatar!)
10C-sync.fm 6 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Spatial Relations
Usually known as layout relationships• Define the space which is used for the presen
• on an output device• at a certain point of time in a multimedia pres
Typically expressed in layout frames:• A layout frame is placed and a content is assi• The positioning of a layout frame in a docume
• fixed to a position in a document• fixed to a position on a page• relative to the positioning of other frames
Spatial relations for time-dependent media obje• A frame or a group of frames may be presente• Note: an audio presentation can be positioned
(imagine, e.g., a virtual meeting room where aa VRML space: audio should be positioned w
existed before)
tandardization
st correspond to the temporal
10C-sync.fm 7 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Temporal Relations
Importance for time-dependent media objects• new in context of Multimedia (content, spatial• essential for communications, scheduling• non-trivial yet "general" issue --> subject of s• --> to be emphasized in this chapter
Example for temporal relations:• Temporal relation between
• video object• audio object
recorded during a concertAt presentation time:
• temporal relation of the two media objects murelation at the recording moment
Relations:• at the same time• independently• in sequel
e relation betweenndent media object
ideo sequence.d each of the frames has to
ws a jumping ball
t
10C-sync.fm 8 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Intra-object Synchronization
Intra-object Synchronization
Intraobject synchronization defines the timvarious presentation units of one time-depe
Example:• Time relation between the single frames of a v
For a video with a rate of 25 frames per seconbe displayed for 40 ms
Frames of a video sequence that sho
40ms
nchronization between
n
dio sequence
o2
tion
t
10C-sync.fm 9 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Inter-object Synchronization
Inter-object Synchronization
Inter-object synchronization defines the symedia objects
Example:• Time relations of a multimedia synchronizatio
that starts with an audio/video sequence,followed by several picturesand an animation that is commented by an au
Audio1
Video
P1 P2 P3
Audi
Anima
ence of information
bjects
phony
10C-sync.fm 10 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Logical Data Units
Logical Data Unit
An information unit - usually part of a sequunits - of a time-dependent media object
Granularity levels of LDUs• Application dependent• Imply a hierarchical decomposition of media o• Example:
The LDU hierarchy of a Sym
Symphony
Movements
Notes
Samples
f continuous media like
ation execution
or a microphone, or
10C-sync.fm 11 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of LDUs
Closed LDUs• have a predictable duration
Example:• LDUs that are parts of stored media objects o
• audio and video or• stored media objects with a fixed duration.
Open LDUs• duration is not predictable before the present
Example:• Open LDUs typically represent
• input from a life source, for example a camera• media objects that include a user interaction
ce as LDUs
s except lowest)
equence as LDUs
Us, open block-LDU
t
Pic.n
t
Audio n
cy (e.g., 1/8000 s)
(e.g., = 0.064 s)
t
10C-sync.fm 12 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of LDUs (cont.)
Physical units as LDUs:
Physical units of a video sequen
Blocks of samples as LDUs: (all hierarchy level
Blocked physical units of an audio s
note: e.g., life-stream video has closed phys-LD
Pic. 1 Pic. 2 Pic. 3 Pic. 4
1/30 s
Audio 1 Audio 2 Audio 3 Audio 4
physical frame duration = 1/sample frequen
duration of a Logical Data Unit of 512 Bytes
n
generated media objects.
ser selected size
rded user interactions.
em with varying durations
Frame n
e (= e.g.1/15 s)t
tion 4
t
10C-sync.fm 13 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of LDUs (cont.)
User-defined vs. capturing-defined LDU duratio
Fixed vs. varying LDU duration
• e.g., fixed&user-defined: typical for computer
LDUs of an animation sequence of u
• e.g., fixed&capturing-defined: typical for reco
LDUs of an user interaction in a window syst
Frame1 Frame 2 Frame 3 Frame 4
duration of Logical Data Units of selected siz
Action 1 Action 2 AcA.3
LDUs of a recorded user interaction
rationknown duration
er interaction
DUs
l processingt
t
10C-sync.fm 14 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of LDUs (cont.)
Note: open LDUs always with unpredictable du• Typically a user interaction with in advance un
An open LDU representing an us
Note: Timers represented as stream of empty L
LDUs of a timer
LDU duration not known before the actua
User selection
LDUs of 10 ms duration
ed by
User
nimation,Timer
Userteraction
10C-sync.fm 15 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of LDUs - Overview
• open vs. closed• open: always unknown duration• closed: fixed or variable duration
• fixed vs. variable duration• user-defined vs. capturing-related• physical --> blocked
Duration defin
Capturing
Fixed Duration of LDU Audio, Video A
Variable or unknownDuration of LDU
Recordedinteraction in
eo streams withs
zation
ow
Frame n
Audio n
ms advance
e 3 Slide 4
10C-sync.fm 16 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Examples
Lip synchronization• demands for a tight coupling of audio and vid• a limited skew between the two media stream
LDU view of lip synchroni
Slide show with audio comment:
LDU view of a slide sh
Frame 1 Frame 2 Frame 3 Frame 4
Audio 1 Audio 2
Audio max. 80 ms advance Video max. 80
Audio sequence
Slide 1 Slide 2 Slid
udio1 and Video) is follo-ion (RI), a slidetion) which is partiallyo2). Starting the anima-n is presented to thection, a final picture
ple
Interaction
Audio2
Animation
P4
t
10C-sync.fm 17 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Application ExampleExample used further in the course:
A lip synchronized audio video sequence (Awed by a replay of a recorded user interactsequence (P1 - P3) and an animation (Animacommented using an audio sequence (Audition presentation, a multiple choice questiouser (Interaction). If the user has made a sele(P4) is shown.
Synchronization exam
Audio1
Video
RI
P1 P2 P3
ser Interface
Sync. QoSProvision
10C-sync.fm 18 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
2. Synchronization and Multimedia
Typical Requirements:• hard vs. soft
Strong Interaction with QoS• acceptable, too bad ...
Variety of Mechanisms• Live Synchronisation• Live Synchronisation with Storage• Synthetical Synchronisation
MMOperatingSystemEnvironment
ProgrammingEnvironment
Application
MM U
MM
MM Sync. QoSRequirements
ed by Skew (QoS)?????????
r Interface
Sync.QosProvision
10C-sync.fm 19 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Why Multimedia-Synchronization?
Media Influenc
Video
Animation
?
AudioImageText
Audio
AnimationAudioImageText
Pointer
MMOperatingSystemEnvironment
ProgrammingEnvironment
Application
MM Use
MM
MMSync.
QoSRequirements
iments
ation requirements for:
equipment
10C-sync.fm 20 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Multimedia Synchronization - Exper
Example:• at IBM ENC Heidelberg to quantify synchroniz
• Audio/video synchronization• Audio/pointer synchronization
Selection of material:• Duration
• 30s in experiments• 5s would have been sufficient
• Reuse of same material for all tests
Introduction of artificial skew:• By media composition with professional video• With frame based granularity
Test conditions:• Huge set of test candidates
• Professional: cutter at TV studios• Casual: every day “user”
• Awareness of the synchronization issues• Set of tests with different skews lasted 45 min
any artifact or
ords. (-)
f or behind the
or.o watch all your(.)
10C-sync.fm 21 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Feedback - Questionnaire
1 While watching this video clip, did you detectstrange effect?
If so, please try to describe it in a few w
....
2 Are you able to identify if audio was ahead omoving pictures? (.)
a) Yes, I identify audio to be played aheadof video
b) Yes, I identify audio to be played behindvideo
c)No, I notice that audio is out of sync with
respect to video but, I am not sure ifaudio is played ahead of or behind video.
3 You noticed a synchronization errHow would you qualify this error if you have t
TV programs with such an error?
a) I would not mind, the error is acceptable
b) I dislike it, the error is annoying
c) I am not sure if I would accept such anerror or if I would really dislike it
encing Factors
(hammer and nails)
iew)
10C-sync.fm 22 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Lip Synchronization: Major Influ
Video:• Content
• Continuous (talking head) vs. discrete events• Background (no distraction)
• Resolution and quality• View mode (head view, shoulder view, body v
Audio:• Content• Background noise or music• Language and articulation
tection
ahead of video
10C-sync.fm 23 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Lip Synchronization: Level of De
Areas:• In sync QoS: +/- 80 ms• Transient• Out of sync
audio behind video audio
noyance
o ahead of video
10C-sync.fm 24 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Lip Synch.: Level of Accuracy/An
Some observations:• Asymmetry• Additional tests with long movie
• +/- 80 ms: no distraction• -240 ms, +160 ms: disturbing
audio behind video audi
etch”)n map”)
10C-sync.fm 25 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Pointer Synchronization
Fundamental CSCW shared workspace issue
Analysis of CSCW scenarios:• Discrete pointer movement (e.g. “technical sk• Continuous pointer movements (e.g. “route o
Most challenging probes:• Short audio• Fast pointer movement
etection
ahead of pointer
10C-sync.fm 26 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Pointer Synchronization: Level of D
Observations:• Difficult to detect “out of sync”:
• i.e., other magnitude than lip sync• Asymmetry:
• According to every day experience
audio behind pointer audio
nnoyance
ahead of pointer
10C-sync.fm 27 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Pointer Synchronization: Level of A
Areas:• In sync: QoS -500 ms, +750 ms• Transient• Out of sync
audio behind pointer audio
n QoS+/- 120 ms
+/- 80 ms
+/- 240 ms
+/-500 ms
+/- 240 ms
+/-500 ms
+/- 80 ms
+/- 11 µs
with various +/- 120 ms
d music) +/- 500 ms
otes) +/- 5 ms
) +/- 500 ms
+/- 240 ms
-500ms+750 ms
10C-sync.fm 28 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization QoS: Skew Values
Media Mode, Applicatio
Video
Animation correlated
Audio lip synchronization
Imageoverlay
non-overlay
Textoverlay
non-overlay
Audio
Animation event correlation (e.g., dancing)
Audio
tightly coupled (stereo)
loosely coupled (dialogue modeparticipants)
loosely coupled (e.g., backgroun
Imagetightly coupled (e.g., music with n
loosely coupled (e.g., slide show
Text Anmerkungen zu Text
Pointer Audio Related to the Itlem
Synchronization
isms
ynchronization systems
10C-sync.fm 29 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
3. Reference Model for Multimedia -
A reference model is needed
• to understand the various requirements formultimedia synchronization
• to identify and structure runtime mechanisms
• to identify interfaces between runtime mechan
• to compare system solutions for multimedia s
t
Timecompo-sition
Deriva-
Inter-pretation
tion
10C-sync.fm 30 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Existing Synchronization Models
Classification methods:• Physical, system, and human level• Intra-stream (fine-grain) synchronization and
inter-stream (coarse-grain) synchronization• Live and synthetical synchronization• Mapping
Uninterpreted
Media Objects
Bytestream
DerivedMedia Objects
MultimediaObject
chniques
on
10C-sync.fm 31 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Classification of Synchronization Te
Inter-object synchronization techniques:• used to control jitter between media streams• classification according to the type and locati
of the synchronization control
Distributed synchronization control:• distributed control based on protocols• distribution based on servers• distribution on nodes without server structure
Local synchronization control:• control on several layers• use of local servers
Ab
stractio
n
low
highlayer
10C-sync.fm 32 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Four Layer Model
Media layer
Stream layer
Object layer
Multimedia application
Specification
cluded objects
ct
ject synchronizations
ject synchronizations
ynchronization
10C-sync.fm 33 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Specification
Synchronization specification• describes all temporal dependencies of the in
in a multimedia object• is part of the description of a multimedia obje• should comprise:
• Intra-object synchronization specificationsfor the media objects of the presentation
• Quality of service descriptions for the intra-ob• Inter-object synchronization specifications
between the media objects of the presentation• Quality of service descriptions for the inter-ob
• may comprise:• alternative presentation forms
• must be explicitly described in a synthetical s• is implicitly defined in a live synchronization
Tasksto edit
to format
map the quality of service values tothe object layer
Plan and coordinate presentationexecution
tiate presentation of time-dependentmedia objects
itiate presentation of time-indepen-dent media objects
source Reservation and schedulingof LDU processing
File and device access
10C-sync.fm 34 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Layered Model - Summary
Layer Interface AbstractionSpecification The tools performing the tasks of this
layer have interfaces, the layer itself hasno upper interface
to
Object Objects that hide types of enclosedmedia
Synchronization Specification
Media-oriented quality of service (Interms like allowed skew, allowed jitter)
Ini
In
Stream Streams and groups of streams
Guarantees for intra-streamsynchronization
Guarantees for inter-stream synchroni-zation of streams in a group
Re
Media Device independent access to LDUs.
Guarantees for single LDU processing
Environment
ation
10C-sync.fm 35 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
4. Synchronization in a Distributed
More complex than in a local environment:
• Distributed storage of synchronization inform
• Distributed storage of media objects
• Communication delays and jitter
• Multi-party communication patterns
fication
ation
Sink
ation
Sink
10C-sync.fm 36 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Transport of Synchronization Speci
Delivery of the complete synchronization informbefore the start of the presentation
Use of an additional synchronization channel:
Use of a multiplexed channel:
Source
Video
Audio
Synchronization
Source
Video
Audio
Synchroniz
multiplexed
ions
ink
Sinkidth
and
10C-sync.fm 37 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Location of Synchronization Operat
Source SBandwidthdemand
Synchronization at the sink node
Synchronization at the source node
Source Bandwdem
ing of the presentation:
Ta=Tav - Nla - SaTv=Tav - Nlb - Sv.skew
Sink AVTav
10C-sync.fm 38 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Clock Synchronization
Problem: Skew between the clocks
• Clock skew has to be considered in the plann• Tav: Start of presentation• Nla: Known net delay• Sa,Sv as Skews• Start time of audio transmission of Source A:• Start time of video transmission of Source V:
• Restricted buffer size demands for restricted
Source A
Source V
Audio
Video
Ta, Sa
Tv, Sv
Nla
Nlv
Sinks
10C-sync.fm 39 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Multiple Communication Relations
1:1
n:1
1:n
n:m
Sources
Sources : Sinks
o the network
10C-sync.fm 40 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Multi Step Synchronization
• Synchronization during object acquirement
• Synchronization of retrieval
• Synchronization during delivery of the LDUs t
• Synchronization during the transport
• Synchronization at the sink
• Synchronization within the output device
en two objects
A B
AB
AB
A meets B
A during B
A finishes B
10C-sync.fm 41 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
5. Synchronization Techniques
Specification of great importance
• Simple specification method:
Types of temporal relations betwe
B
AB
A
BA
AB
A before B
A overlaps B
A starts B
A equals B
ts
nance of specificationsnit in the specificationbe easily describable
ia objectsribableation
10C-sync.fm 42 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Specification Methods - Requiremen
Complex specification comprise:• multiple object synchronization• user interactions
Requirements for a specification method:• Support of object consistency and for mainte• Media objects should be kept as one logical u• All types of synchronization relations should• Support for the integration
of time-dependent and time-independent med• Quality of service requirements must be desc• Support of hierarchical levels of a synchroniz
either a single-media object
ronization operators
Video
onous audio/videoesentation
10C-sync.fm 43 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Basic Hierarchical Specification
Two main synchronization operations:• Serial synchronization of actions• Parallel synchronization of actions
Action:• an atomic action handles the presentation of
or a user input or a delay.• a compound action is a combination of synch
and atomic actions.
Serial and parallel presentations
Pic. 1 Pic. 2 Pic. 3
Slide sequence
Audio
Lip synchrpr
synchronization.
(RI = Recorded Interaction, Pic.teraction)
Pic.3
Ani.3 UI Pic.4
10C-sync.fm 44 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Basic Hierarchical Specification
Delay as a possible action allows for• delays in serial presentations and• delayed presentations of objects in a parallel
Application example:
Hierarchical specification of the application example= Picture, Aud. = Audio, Ani. = Animation, UI = User In
Audio1 Video
RI Pic.1 Pic.2
Ani.1 Ani.2 Aud.2
not be described
of skewessary
ependent mediamust be added
ia objects foroses
r media object
A3
B3
C3
10C-sync.fm 45 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Basic Hierarchical Specification
Not all relations can be described.
Example of a synchronization that can
Advantages Disadvantages
Easy tounderstand
Additional descriptionquality of service nec
Natural supportof
hierarchies
For the presentation of time-indobjects presentation durations
Integration ofinteractive
objects is easy
Need for the splitting of medsynchronization purp
No adequate abstractions focontents
A1 A2
B1 B2
C1 C2
rios can not be
10C-sync.fm 46 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Some synchronization scenadescribed
Advantages Disadvantages
Timer
xis
the application example
teraction
io2
ation P4
?t
10C-sync.fm 47 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Based on a Global
All single-media objects are attached to a time a
Application example
Time axis synchronization specification of
Audio1
Video
Record. UI In
P1 P2 P3
Aud
Anim
Timer
vantages
nknown durationgrated, extensionsel are required
f service has to beirectly by using the
axis or additionalvice specificationsto be given
d environment thebal unique time hasbe kept
10C-sync.fm 48 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Based on a Global
Advantages Disad
Easy to understandObjects of u
cannot be inteto the mod
Support of hierarchies easy torealize
Skew quality ospecified indcommon timequality of ser
have
Easy to maintain because of themutual independence of objects
In a distributenotion of a glo
to
Good abstraction for mediacontents
Integration of time-independentobjects is easy
xes
ement unitsate space
xes
at
10C-sync.fm 49 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Based on Virtual A
Specification using• coordinate systems with user defined measur• several virtual axes to create a virtual coordin
Example:
Music notes as example for virtual axes
At runtime the virtual axes are mapped to real a
Virtual axis with measurement unit be
Virtual axes with measurement unit pitch
xes
ple
teraction
io2
ation
P4
t
t
10C-sync.fm 50 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Based on Virtual A
Application example:.
Virtual time axis specification of the application exam
Audio1
Video
Record. UI
In
P1 P2 P3
Aud
Anim
Time axis
Time axis
Interaction axis
xes
vantages
ty of service onlyor by additionaltions definable
on may becomeby many axes
xes at runtime maynd time consuming
10C-sync.fm 51 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization Based on Virtual A
Advantages Disad
Easy to understandSkew quali
indirectlyspecifica
Often specification according tothe problem space possible
Specificaticomplex
Good possibility for buildinghierarchies
Mapping of abe complex a
Easy maintainable, becauseobjects are kept as units andmutual independent objects
Good abstraction for mediacontent
Easy integration of timeindependent media objects
Interactive objects can beincluded using specialized axes
e Points
rded as sequences of
ation of a media objectent media objects
it is reachedlicit reference to time.
10C-sync.fm 52 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization based on Referenc
Synchronization via reference points:• Time-dependent single-media objects are rega
LDUs• Reference points:
• the start time and the stop time of the present• the start-times of the subunits of time-depend
• Synchronization point:• A set of connected reference points• Defines the synchronization between objects• LDUs have to be started resp. stopped when
• Description of temporal relations without exp
e Points
ation model
eference point model
3 Slide 4
10C-sync.fm 53 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization based on Referenc
Lip synchronization example
Lip synchronization in the reference point synchroniz
Slide show example.
Example of a slide show with audio sequence in the r
Video
Audio
Slide 1 Slide 2 Slide
Audio
e Points
e-independent media objectssynchronization specification
Interaction
dio
mation
P4
10C-sync.fm 54 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization based on Referenc
Application example
Example of the integration of time-dependent and timas well as closed and open LDUs in a reference point
Audio
Video
Record. UI
P1 P2 P3 Au
Ani
e Points
vantages
to change becausect description ofndencies
omplex hierarchies
10C-sync.fm 55 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization based on Referenc.
Advantages Disad
Easy to understandMore difficult
of the diredepe
Natural description oftemporal relations Sometimes c
Easy integration of open LDUs
Integrated description of skewquality of service
Easy integration of time-independent objects
Time axis are special case of thereference point synchronization
model
ns
hat reaches a specific LDU
10C-sync.fm 56 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Event-based Synchronization
Event-based synchronization presentation actioare initiated by the synchronization events
Typical presentation actions are:• to start a presentation• to stop a presentation• to prepare a presentation
Events• may be external,
e.g. generated by a timer, or• internal to the presentation
generated by a time-dependent media object t
r1.dy ...
p
rt
10C-sync.fm 57 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Event-based Synchronization
Application example:
ActionEvent
Start Audio.stop Timerea
Audio1 start
Video start
Pic.1 start sto
Timer1 start(3)
Pic.2 sta
...
vantages
sy to handle
specification
to maintain
f time-independentng additional timers
riptions of quality ofnecessary
e of hierarchies
10C-sync.fm 58 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Event-based Synchronization
Advantages Disad
Easy integration of interactiveobjects Not ea
Easy extensible by new events Complex
Flexible because any eventcan be specified Hard
Integration oobjects by usi
Separate descservice
Difficult us
on blocking token.n is removed and to each
that is assigned to its new
ion
F4
A1
ms 11 ms
33ms
F5
10C-sync.fm 59 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Petri net-based synchronization
The rules for a timed petri net are:• A transition fires if all input places contain a n• If a transition fires, on each input place a toke
output place a token is added• After firing, a token is blocked for the duration
place
Lip synchronization example.
Petri net based synchronization of a lip synchronizat
F2 F3
A1
11 ms 11 ms 11 ms
33ms
11
F1
F3
11 ms D
0 ms
10C-sync.fm 60 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Petri-Net based synchronization
Example for a hierarchy:
F1 F2
A1
11 ms
33ms
11 msD
0 ms
S
33 ms
A3nimationubnet 3
10C-sync.fm 61 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Petri-Net based synchronization
Application example:
Petri net specification of the application example
S
60 s
Audio-Videosubnet
R
20 s
Record.UI
P
15 s
Picturesubnet
PI
A1
Au
A2Animation-subnet 1
As
Animation subnet 2and Audio subnet
P4
User interaction Closing picture
vantages
sy to handle
specification
f media objects
bstraction for thehe media objects
10C-sync.fm 62 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Petri-Net based synchronization.
Advantages Disad
Hierarchies can be created Not ea
Easy integration of time-independent objects Complex
Easy integration of interactiveobjects Splitting o
Integrated skew quality of service Insufficient acontent of t
ariots
y the timing operations
10C-sync.fm 63 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Scripts
Script• Textual description of a synchronization scen• Elements of scripts are activities and subscrip• Three main operations in a script are
• serial presentation• parallel presentation• repeated presentation of a media object
• Often full programming languages extended b
;
.rec”);eg”);eg”);eg”);eg”);;on.au”);on.ani”);
;
ple {(
}
10C-sync.fm 64 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Scriptsactivity DigAudioAudio(“video.au”)activity SMPVideo(“video.smp”);activity XRecorderRecorder(“windowactivity PicturePicture1(“bild1.jpactivity PicturePicture2(“bild2.jpactivity PicturePicture3(“bild3.jpactivity PicturePicture4(“bild4.jpactivity StartIntercationSelectionactivity DigAudioAniAudio(“animatiactivity RTAnimaAnimation(“animatiscript Picture_sequence 3Pictures=
Picture1.Duration(5) >>Picture2.Duration(5) >>Picture3.Duration(5);
script Lipsynch AV = Audio & Videoscript AniComment AA = Animation &AniAudio.Translate(2);script Multimedia Application_exam
AV >> Record. UI >> 3Pictures >>( (Selection >> Picture4) & AA )
vantages
sy to handle
specification
age of commonnecessary
structs for skewervice necessary
10C-sync.fm 65 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Scripts
Advantages Disad
Easy hierarchies Not ea
Logical structure of objectscan be kept Complex
Easy integration of time-independent objects
Implicit ustimers
Easy integration of interactiveobjects
Special conquality of s
Skew quality of service canbe specified
Easy extensible by newsynchronization constructs
Flexible because programmable
reated further here:
vironment)
AND HYPERMEDIA OBJECTS
tion objects to bed services
at are part of the
p a set of MHEG objectsctions to be performed
oral and conditional
10C-sync.fm 66 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
6. Case Studies
1. Some "historical" or academic systems not t• HyTime• FireFly• MODE (Multimedia Objects in a Distributed En• Multimedia Tele-Orchestra• ACME (Abstractions for Continuous MEdia)• Apple QuickTime
2. MHEG: CODED REPRESENTATION OF MULTIMEDIA
• representation and encoding of• final form multimedia and hypermedia informa
interchanged within or across applications an
MHEG objects• Content objects encapsulate media objects th
presentation• Composite objects act as a ‘container’ to grou• Action objects are used to exchange sets of a
between objects• Link objects are used to specify spatial, temp
relations between MHEG objects
tent objectsneric units of time=altitude) of finite length
inate space
are sent to an object
ables the object
10C-sync.fm 67 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
Synchronization in MHEG
Generic space:• provides a virtual coordinate system• used to specify the layout and relation of con
• one time axis of infinite length measured in ge• three spatial axes (X=latitude, Y=longtitude, Z
in an interval of [-32768,+32767]• MHEG maps from the virtual to the real coord
Presentation of content objects:• Based on the exchange of action objects that• Examples for actions are:
• prepare for setting the object in a state that ento present itself
• run to start the presentation• stop to stop the presentation
elayed sequential actions.
...
10C-sync.fm 68 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
MHEG action lists
Action list:• Combination of actions to action lists• Parallel action lists are executed in parallel• Each list is composed of a delay followed by d
Links:• used to create event-based synchronizations• can trigger events
delay delay
action action
Parallel lists of actions
Del
ayed
sequ
entia
l
actio
ns
Gen
eric
Pre
sent
atio
nS
ervi
ces AV-Sub-
UserInterface
system
Services
10C-sync.fm 69 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
MHEG Engine
Application
MHEG Engine
Object Manager
Link Processor Interpreter
Operating System
ultimedia object
Engine
User Interface
bsystem
Service
10C-sync.fm 70 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
MHEG Relation to Reference Model
Media layer
Stream layer
Object layer
Specification layer
Object layer interface
Stream layer interface
Media layer interface
MHEG coded m
MHEGInterpreter
AV Su
Object ManagerLink Processor
guage
)
ia Synchronization
ay (cf. Flash, Shockwave)
ontents not accessible
b?
10C-sync.fm 71 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
SMIL
SMIL: Synchronized Multimedia Integration Lan
part of W3C standards, part of XML&friends
treated in lectures about Web Engineering
based on "Amsterdam Model" (CWI Amsterdam
major contribution of recent years
goal: integrate Hypertext concept and Multimed
remember: multimedia possible on the Web tod
but:• requires plug-in• MM presentation is "one node" in hypertext, c• no links into MM presentation, no links out• same text used within flash and as part of We
--> duplicate --> two different elements!
SMIL introduces MM synchronization in XML• similar to hierarchical specification• plus support for spatial arrangements
MM node B
TML
t
MM node B
TML
t
ave been, e.g.:
" duration="0.5" target=2>e" in <img> tag)
left=... top.. height.. width../>
10C-sync.fm 72 15.February.01
Scope
Contents
htt
p:/
/ww
w.k
om
.e-t
ech
nik
.tu
-da
rmst
adt
.de
htt
p:/
/ww
w.t
k.in
form
atik
.tu
-da
rmst
adt
.de
©R
.S
tein
me
tz,
M.
Mü
hlh
äu
ser
SMIL
Example for temporal spec:<par>
<img src="logo.rp" region="logo"/><text src="info.rt" region="info"/><video src="pk.rm" region="video"/>
</par>
<seq><audio src="music1.ra"/><audio src="music2.ra"/>
</seq>
link
in H
MM node A
single mediumwithin A
t
link
in H
MM node A
single mediumwithin A
t
prior definitions, might h
<fadein start="0.2(target: -> "handl
<region id="logo"