nus.soc.cs5248 ooi wei tsang 1 intermedia synchronization ooi wei tsang

nus.soc.cs5248Ooi Wei Tsang

1

Intermedia Synchronization

Ooi Wei Tsang

2

Overview

• Why we need to sync?• How close must we sync?• How to sync?• How to measure sync?• Applications

3

Synchronization

Intrastream Interstream

Local playback Networked

4

Problem

• Ensure that audio/video captured at the same instance at the source, are displayed at the same instance at the receivers.

nus.soc.cs5248Ooi Wei Tsang

5

Why we need Synchronization?

6

Why not Pre-sync?

V A V A

Sender Receiver

7

Seperating A/V Flows

Sender Receiver

V

A

V

ASender Receiver

• Can receive audio without video• Can prioritize audio over video

8

May Come from Different Sources

• UNLV Access Grid

Sender Receiver

V

A

V

ASender Receiver

9

or Displayed at Different Sources

Sender Receiver

V

ASender Receiver

A

V

10

More Complications

• Different decoding time for video and audio

• Network delay and jitter• Different clocks

11

Bottom Line

• Must synchronize at the receiver

nus.soc.cs5248Ooi Wei Tsang

12

Synchronization Requirements

13

Reference

• Human Perception of Jitter and Media Synchronization

by Ralf Steinmetz IEEE JSAC 14(1) 1996

14

Results “Badness”

A/V offset-80ms +80ms

15

What matters

• Video ahead of audio vs. audio ahead of video

• Size of head• Occupancy

16

What does not matter

• Language• Speaking or Singing• Hobby• Speed

17

Conclusion

• -80ms to +80ms acceptable• -160ms to +240ms noticable

but not annoying

• - audio > video• + video > audio

nus.soc.cs5248Ooi Wei Tsang

18

How to Synchronize?

19

Receiver needs

• Exact time a sample is generated

• Each packet needs a timestamp

20

RTP Protocol

Media TimestampRTPHdr

21

But..

• Media timestamp has a random offset

22

RTCP Protocol

:

Media Timestamp

NTP Timestamp

RTCPHdr

:

23

NTP

• Network Time Protocol• Synchronize clock up to few

milliseconds

24

Sender Audio

Sender Video

Receiver Audio

Receiver Video

Audio RTCP

Video RTCP

Audio RTP

Video RTP

25

Sender Audio

Sender Video

Receiver Audio

Receiver Video

Audio RTCP

Video RTCP

Audio RTP

Video RTP

Synchronization Delay

26

Multiple Receivers

• Everyone must have the same synchronization delay

27

How to set Sync Delay?• Receivers tell each other their

network delay• Synchronization Delay =

max {Network Delays} + Buffer

28

Buffer Control

29

No Delay No Jitter

30

Delay, No Jitter

31

Delay, Jitter

32

If Buffer too Full/Empty• Send message “Play faster!” or

“Play slower!” to other receivers

33

Must Synchronize!

• Send message “Play faster at time T!” or “Play slower at time T!” to other receivers

nus.soc.cs5248Ooi Wei Tsang

34

How do we Measure Synchronization?

35

Reference

• A Method and Apparatus for Measuring Media Synchronization

• by Brian Smhmidt etc.

• NOSSDAV 1995

36

System under Test

Measurement System

Generator

nus.soc.cs5248Ooi Wei Tsang

37

Beyond Video Conferencing

38

HQ Video and Audio

• Multiple Channels• Multiple Displays

39

Network Musical Performance

BOS

BOS

SF

BOS

SF

ACM Multimedia ‘94

40

NASDAQ

• Synchronize display of stock quotes