school of informatics cg087 time-based multimedia assets timecodes & syncp. vickers/j....

Timecodes & Sync P. Vickers/J. Edwards/A. Watson 1

School of Informatics

CG087 Time-based Multimedia Assets

Time codes and Synchronisation

Synchronising video and audio with time codes such as SMPTE

and SMIL


CG087 Time-based Multimedia AssetsSchool of Informatics

Time codes

• A time code allows each frame of video to be numbered.

• Numbering frames allows precise identification when editing.

• It also facilitates synchronisation of audio with video

• An 80-bit word is stored on each frame to represent hours, minutes, seconds, frames and other synchronizing information. There is also room for ‘user bit’



Time code example

03:10:45:18

Hours

Minutes

Seconds

Frames



Starting time codes

• 2 ways of starting a time code– Record run– Free run



Record run

• Only stamps a time code when camera is recording.

• Time code set to zero at the start of the day’s operation and a continuous record is produced on each tape covering all takes

• Customary to record the tape number in place of the hour section on the time code– E.g. the first cassette of the day would start

01.00.00.00, and the second cassette would start 02.00.00.00.



Free run

• Time code is set to the actual time of day and when synchronized is set to run continuously.

• Whether the camera is recording or not, the internal clock will continue to operate.

• When the camera is recording, the actual time of day will be recorded on each frame.– Useful for editing day-long events (e.g. sport)

Action can be logged by time as it occurs and can be quickly found by reference to the time code on the recording



Free run: gaps

Shot 1(8 seconds)

Shot 2(5 seconds)

Shot 3(7 seconds)

200 frames 125 frames 175 frames

04:12:45:01

14:44:22:05

15:02:44:03



Free run: gaps

• A change in shot produces a gap in time code corresponding to the elapsed time between recordings

Shot 1(8 seconds)

Shot 2(5 seconds)

Shot 3(7 seconds)

200 frames 125 frames 175 frames

04:12:45:01

14:44:22:05

15:02:44:03



Time code problems

• Missing time code numbers can cause problems with edit controller when it rolls back from intended edit point and is unable to find time code number it expects there (i.e. the time code of the frame to cut on, minus the pre-roll time).



Synchronisation

• To fit a sound track to a video sequence you need to syncrhonise the audio with the video

• When a system is synchronized, all the video and audio decks, computer sequencers, hard disks and so forth will play back exactly together. This allows you to compose layers of sound, secure in the knowledge that once you get something in place, it will stay there. (see http://arts.ucsc.edu/ems/music/equipment/video/smpte/SMPTE.html)



SMPTE

• This section shamelessly plagiarised from http://arts.ucsc.edu/ems/music/equipment/video/smpte/SMPTE.html

• The Society for Motion Picture and Television Engineers (SMPTE) has defined its own time code

• SMPTE format allows different devices to be synchronised in a master/slave arrangement– Master device must be a video deck– Other devices follow the master– Multiple video decks are ‘genlocked’ to a clock device

• Genlocking is a line-by-line (rather than frame-by-frame) synchronisation system



SMPTE overview

• SMPTE code recorded on one audio track• Actually a modulated tone that is decoded

into data• 80-bit word for each frame to store

hh:mm:ss:ff + subcode information



SMPTE gadgets

• SMPTE generator– Creates the time code

• SMPTE reader– Displays the time

• Regenerator– Reads the code and recreates it for recording on

another device



SMPTE gadgets continued

• MTC generator– Converts SMPTE into MIDI time code which can

be sent over MIDI cables

• SPS converter– Turns SMPTE into song pointer and MIDI clock

data to control sequencers

• FSK– System for recording MIDI song pointer and MIDI

clock data directly on tape. Simpler than SMPTE but less flexible



SMPTE gadgets continued

• Controller– Tape deck remote control– Uses SMPTE to monitor tape position and can cue

tape to any desired location– More accurate than a mechanical tape counter

• Synchroniser– Controls >=2 tape decks



SMPTE frame rates

• 24 fps– Film

• 25 fps– European video (PAL, SECAM)

• 30 fps– American & Japanese B&W video

• 29.97 fps– American & Japanese colour video. Can lock 30fps

audio deck to this, but pitch will drop from A=440 to 335.6 Hz



SMPTE numbering schemes

• Each frame contains hh:mm:ss:ff• SMPTE time code can only increment in

whole frame numbers• How to deal with odd division of frames in

US/Jap colour format (29.97 fps)?• At each new frame (now lasting a little more

than 1/30th of a sec.), the Time Code still increments by a whole 1/30th and drags behind real time



SMPTE DF and NDF

• In Non Drop Frame mode we slow down the code generator to match the frame rate– This makes a 1-hour programme actually last

59:56.4!– 29.97/30 * 3600 = 3596.4s

• In Drop Frame mode we skip two frame numbers a the start of each minute except those ending in zero (00, 10, 20, 30, 40 ,50)– 54 mins that have 2 frame no.s added = 108– 3600s *29.97 fps = 107892 frame no.s + 108 =

108000 = 3600 * 30



SMPTE and MIDI

• MIDI time code is a specialised SMPTE code that can be transmitted over MIDI cables

• Most sequencers now support SMPTE control

• Look at DOP’s Options|Sync Settings



SMIL

• Synchronized Multimedia Integration Language.

• Pronounce it "smile" !• An XML abstraction for Synchronisation of

Multimedia• Originally proposed around 1998• At present v2.0• Can be used in IE6 and RealPlayer• GRINS and SOJA also example implementations• Synchronisation reflects position and time

A simple SMIL document …… looks familiar ???

<smil> <head> <meta name="copyright" content="Your Name" /> <layout>  </layout> </head> <body>  </body> </smil>

SIMILAR STRUCTURE TO HTML



Positioning - 101

• Not really very exciting (link region to data)

<smil>

<head>

<layout>

<root-layout width="300" height="200" background-color="white" />

<region id="vim_icon" left="75" top="50" width="32" height="32" /> </layout>

</head>

<body>

<img src=“myimage.gif" alt="The vim icon" region="vim_icon" /> </body> </smil>



Positioning - Relative

<smil> <head> <layout> <root-layout width="300" height="200" background-

color="white" /><region id="vim_icon" left="50%" top="40%" width="32"

height="32" /> </layout> </head> <body> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" /> </body></smil>



OTHER MM TAGS AVAILABLE

Media Tag G2 GRiNS Soja

GIF img OK OK OK

JPEG img OK OK OK

Microsoft Wav audio OK OK -

Sun Audio audio OK OK OK

Sun Audio Zipped audio - - OK

MP3 audio OK - -

Plain text text OK OK OK

Real text textstream OK - -

Real movie video OK - -

AVI video OK OK -

MPEG video OK OK -

MOV video OK - -



The meat of SMIL is timings!!

• This is really what sets it apart from DHTML• Allows the specification of

– Duration – Dur = “3s”– Sequence - <seq> </seq> – Synchronisation – id referencing



Duration

<smil> <head> <layout> <root-layout width="300" height="200" background-

color="white"/> <region id="vim_icon" left="75" top="50" width="32"

height="32"/> </layout> </head> <body> <img src="vim32x32.gif" alt="The vim icon"

region="vim_icon" dur="6s" /> </body> </smil>

Sequence

<smil> <head> <layout> <root-layout width="300" height="200" background-color="white" /> <region id="vim_icon" left="75" top="50" width="32" height="32" /> <region id="soja_icon" left="150" top="50" width="100" height="30" /> </layout> </head> <body>  <seq> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" dur="6s" /> <img src="madewithsoja.gif" alt="Made with SOJA" region="soja_icon"

dur="4s“ begin="1s" /> </seq> </body> </smil>



Sychronisation

<smil> <head> <layout> <root-layout width="300" height="200" background-color="white" /> <region id="vim_icon" left="75" top="50" width="32" height="32" /><region id="soja_icon" left="150" top="50" width="100" height="30" /></layout> </head> <body> <par> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" dur="6s" /> <img src="madewithsoja.gif" alt="Made with SOJA" region="soja_icon“

dur="6s" /> </par> </body> </smil>

Synchronisation

<smil>  <body> <par> <img src="vim32x32.gif" region="vim_icon" id="vim" begin="4s" /> <img src="madewithsoja.gif" region="soja_icon" begin="id(vim)

(2s)" /></par> </body> </smil>

• synchronisation specified us id(previous sources)(time duration)



SMIL Summary

• FLASH for the open source world !!!• SMIL 2.0 augments the above framework to

incorporate more media handling– More animation– More event handling– Multiple Windows

• Reference : All examples from– http://www.helio.org/products/smil/tutorial/chapter1/index.

html

• Reference: SMIL 2.0– http://www.w3.org/Talks/2001/06CSMIL/slide1-0.html

school of informatics cg087 time-based multimedia assets timecodes & syncp. vickers/j....

Documents