Delivering Audio and Video

Over the Web

Norman WhiteStern School of Business

Agenda

• Audio/video formats• Rates of transmission• Hardware & software advances• Network requirements• Streaming audio & video

Business Case

• Multimedia is a powerful way to deliver a message

• As a listener/viewer uses more senses to get a message, retention rate goes up.

• Much information is better understood visually

• Question? What are the problems of delivering audio and video over the internet?

Audio Formats

• WAV – direct digitization of audio– Size and quality depend on

“sampling” rate– The higher the sample rate, the larger

the file size for the same length, but the better the quality

• Au• MPEG• Many others

Video Formats

• AVI – Windows video format

• Supports variety of CODECs– CODEC – Compression-Decompression

• Quicktime (MOV)– Apple supported format

• MPEG – Motion Pictures Experts Group– Hard to manipulate, high quality

Digital Audio – ISDN Integrated Service, Digital

Network

• To digitize speech, the telephone standard is to take 8000 samples a second, each sample having a value (0-255, I.e. 8bits)

• Hence 1/2 of an ISDN line is 64kb (64000 bits per second) (allows simultaneous voice and data transmission)

• Note, this is uncompressed. We will talk about compression later)

Bit rates for different audio streams

• Telephone– G.711 PCM– G.721 ADCMP– G.722– G.728

• CD – CD - DA– MPEG consumer– MPEG studio

• Bit rate Kbps– 64– 32– 48,56,64– 16

– 1411– 192– 384

Motion Video Bit Rates (Mbps)

• HDTV– Compressed

• Digital TV (studio)– MPEG2

• Broadcast TV (mpeg2)

• VCR (MPEG-1)• Videoconferencin

g

• 2,000 – 34

• 166– 3-6

• 2-4

• 1.2• .1 – 1.5

Some typical network rates

• 56k modem - 56K bits per second• ISDN - 64kbps (2 channels 128)• Wireless - 2MB - 11Mb/s, 54Mbs• Cable Modem - 10 Mb/s• ethernet/token ring 10mb/s, now

100mbs, 1GB• T3 - 45 mb/s• OC3 - 154MB/s

Video Tips

• Very important to capture video with the highest quality you can

• Consumer Camcorders lose quality quickly as light level drops

• Remember, audio quality is just as important, consumer camcorders have poor microphones, need separate high quality microphone for good results.

Digital Camcorders

• The new digital Prosumer cameras have much higher quality, but still some problems

• Recording time limits– Usually 1 hour for consumer cameras, OK

for some applications, but unacceptable if there is no break to change media

• Blair Witch Project – First major movie to be shot with Digital Camera

Cost Dropping quicklyApplications expand

• Until recently, needed a > $10K Sony beta max for decent video

• Now $1,000 digital cameras give comparable quality (and MUCH smaller and lighter)

• Inexpensive disk drives now offer capacity to deal with “raw” video (100MB per minute, 6 gigabytes for a one hour avi file)

• Need fast processors to encode, decode, transfer data

More hardware advances

• IEEE 1394 – “Firewire” standard– Allows direct connection of digital

devices without a separate card– 400 Mbit/s– Also supplies power to camera

• USB – Universal Serial Bus– USB 2 will be 480 Mbit

Networking Advances

• Gigabit ethernet – Allows streaming of MPEG Video

• “Switched” Ethernet – allows a separate stream to each individual PC without collisions

• Multicasting router capabilities allow listeners to share a stream

• Cable Modems, ADSL, Satellite Dishes much higher bandwidth to home, small business

• TCP/IP v6 – Quality of Service support• Internet 2- 600Mbit connections• Gigabit switches to handle traffic

Software Advances

• Rapidly improving streaming and compression techniques to take advantage of more processing power, new computer capabilities (special instructions for A/D)

• Inexpensive, powerful video manipulation software

Demand Explosion

• E-Commerce and B2B now can use on-line video training, sales clips etc.

• Person to Person IP Video Conferencing

• On-line TV? Movies? Not yet…

Editing Video

• Adobe Premiere– Powerful Mac and PC based editor

• Supports “non-linear” editing– Used to have to edit by splicing tape (linear)

• Copy tape from one machine to another as you edit

– Now can hold digital video content on disk and edit frame by frame

• AVID – Commercial quality edit system

Adobe Premiere Functionality

• Cut / Splice Video clips• Dissolve between clips• Multiple audio tracks• Supports many CODECS• Background colors• Titles• Filters (blur, pan, zoom,sharpen distort

etc.)

Premiere Basics

• Video clips are shot• Clips are brought into Premiere• Clips are dropped on one of the video

tracks in constructor window• Audio clips added, sound adjusted• Titles, transitions, filters added• “Make Movie” produces a new video clip

in AVI , QT, Real Media etc format.• AVI format can be edited again

Conclusion

• Video is now capable of being edited on consumer class computers

• Need LOTS of disk space– 100MBS / minute of video– Need 3 times as much space– Hard drive defragmented– Lot’s of memory– Fast processor (Pentium minimum)– Digital capture card, Firewire or USB to

bring video in

Conclusion

• All indications are that Streaming Media applications will continue to have rapid growth

• Recent Dot-com problems have shut down some of the early entrants in the consumer market.

BUT!!!

• Audio and video file sizes very large and need transfer rates much faster than available bandwidths.

• Many difficult problems need to be solved in order to use audio and video over the internet.

Network Requirements of audio and motion video

Problems for live usage

• Error rate– Network may not deliver bits reliably– Shouldn’t happen that often in a

digital network due to error control

• Transit delay– Data is not delivered at a constant

speed, or at the right speed

Audio Characteristics

• Ear is a differentiator– Has high powers of differentiation– Means that small audio problems are

easily recognized

Video Characteristics

• Eye is an integrator– It puts signals together– Hard to recognize multiple intermixed

drawings and interpret them– Missing information is less noticeable

• Means video can tolerate more errors and higher delays

Implications for multimedia

• In a transmission of mixed audio and video, audio should have a higher priority– Audio more sensitive to delays

• Note - this is a problem on the internet– Internet currently has no priority

mechanism

Compression

• Can we reduce the amount of information we need to send?

• We can compress it• two types of compression

– Lossy - Can’t exactly recreate original– Lossless - Can exactly recreate

original file, bit for bit.

Compressing different types of info

• Images/ Graphics – Gif (lossless) – JPEG - Can be lossy

• Sound (first digitize at some sampling rate)– Then compress

• Video (usually lossy)– Intraframe compression– interframe compression

Video compression techniques

• Intraframe compression– Compress the pixels within one frame

• Interframe compression– Since video frames often change very little

between frames, we can increase compression by only recording the changed bits from one frame to the next.

– This works extremely well for many situations

• Video - 1 frame = 640*480*8 bits = 300K bytes ( 10MB/sec 30 fps)

Compression Problems

• Sending compressed data means that problems in one block (packet) of data may impact subsequent data blocks that have related information which is based on the content of an earlier data block (I.e. keyframes)

Partial solution to network delay

• Add an offset delay to reception of message

• The receiving side will collect N seconds of information before it starts playing the audio or video clip

• How big should ‘N’ be?– Depends on effective speed of

communications channel– higher variance means larger N

Other Problems

• Intermedia Synchronization– Need to synchronize audio and video

streams• Compression and burstiness

– Compression can introduce burstiness into a non-bursty transmission

– Example. Audio uncompressed has a fixed number of bits per second, compressed it can go down to zero (periods of silence)

Conclusion

• Network multimedia demands many complex tradeoffs between bandwidth, errors, delays and cost

• Compression makes the problem even more complex– Error rates have to be lower to use

compression

• What are some reasonable levels for audio and video?

Streaming Audio and Video

Problems with audio/video over the web

• File sizes large, perhaps infinite for continuous live webcasts– Can’t wait for file to completely download

• WEB bandwidth unreliable - high variance• Bandwidth varies from minute to minute• Need to have complex compression and

streaming protocols to compensate– Not supported by HTTP protocol

• http is basically a download protocol, not sophisticated

– Demands much more complex protocol to try to compensate for problems

Solution

• Move audio and video files to a separate server which uses a different (more sophisticated) communications protocol

• Have client application communicate directly with the media server.

Current Solutions

• Real Networks– RTSP - Real Time Streaming Protocol– Needs a separate server– Protocol supports multiple, simultaneous

streams (audio, video, animation, text)– Helper App to play the streams (REAL PLAYER )– SMIL language to describe relationship between

streams– Automatic, dynamic bandwidth negotiation

• Server and client continue negotiating transfer rate in real time.

RTSP features

• Determines average bandwidth between video server and client and variance

• Preloads part of file before it starts playing• Dynamically adjusts needed bandwidth by

varying quality of stream as necessary– Audio switches codecs when bandwidth drops

• CD -> HiFi-> mono – Voice

– Video increases compression and drops frames

• Objective is to keep a “reasonable” output at all times.

Real Player Encoding

• User specifies bit rates to encode in the file at encoding time.

• Real media file (.rm) is generated that contains data necessary to generate any of the supported bit streams

• More bit streams larger files on server

• More bit streams better possibility of playback

Example

• User is connected to server with a connection that is varying between 30 and 45 K bits per second

• User plays a 60 second video clip that was encoded at 30, 40, 50, 100 K bits per second

• Server and Player will negotiate a 40 K bit per second stream

• Player will buffer (predownload part of file) enough to ensure the stream will play continuosly even if throughput is only 30K bits per second

How long does Real Player wait before starting to

play?

• Stream is 60*40K bits/second = 2,400,000 bits• At 30K bits per second it will take

2,400,000/30,000 seconds = 80 seconds• Need to wait 20 seconds before starting to play

stream, i.e load up 20 seconds of material (20*40,000 = 800,000 bits or 100,000 characters

• Note the many design limitations here.– Is there enough memory– How long can user wait before giving up– What happens if bandwidth changes and they

need to renegotiate …

Other Solutions

• Microsoft ASF - Advanced Streaming Format– Only supports one stream, need 3rd party

tools to combine multiple streams into ASF format

– Works with Microsoft NETSHOW – Supports multiple bitrates in one stream– includes high bandwidth intranet option

Publishing Streaming Media with Real Networks

• Need 3 components– WEB Server

• Create links to .RAM, or .SMI files• RAM files point to REAL Networks server

– Real Networks Server• Files are housed here• Special “Streaming” support• Multicast support• Handles Live Streams

– Browser Helper APP• Real Player G2 - can run standalone

Example ..

• First, need to capture and encode Video/audio in Real Media (.rm) Format

• Run Encoder software– Specify audio and video “codecs” to use

• Each bitrate has a different codec• Can use multiple bitrates

– Move Encoded file to Real Networks Server– Create Links on WEB server to point to file

Video file TEST.RM

• create link to .ram file on WEB server– <a href=“TEST.ram”> Click here to

see video</a>

• create TEST.ram file– rtsp://videos.stern.nyu.edu:554/

TEST.rm

What Happens• User clicks on link

– Link is a .ram file, so helper app is invoked to process the file

– Helper App (REAL PLAYER ) starts and is passed the URL of the 1 line ram file by the web server

– REAL PLAYER reads (HTTP GET request) ram file and then issues RTSP connection to video server.

• Video Server and Real Player now have a separate TCP/IP connection (port 554) to communicate over using the Real Time Streaming Protocol.

• Why so complicated?– Because web servers only support http protocol

Synchronized Media

• What happens if I want to synchronize several different streams. Need to use SMIL– Synchronized Media Interpretation Language?

• The link needs to point to a .SMI file instead of a RAM file.

• Simple example, put an intro in front of the video

• I.e. play intro file followed by TEST.rm

SMIL Example

• <smil>• <root> </root>• <body>• <seq>• <rtsp://videos.stern.nyu.edu:554/file1.rm

….> first file to play• <rtsp://videos.stern.nyu.edu:554/file2.rm

…> second file to play• </seq>• </body>• </smil>

More Advanced SMIL

• Smil can also play material in parallel in different windows

• Example would be a REAL TEXT presentation in one window, with synchronized Real pics in another window, and synchronized audio playing as the slides (and text) changes.

• Real Presenter – Real Networks add-on ($29.95) to Powerpoint to allow streaming Powerpoint narration.

Some links

• WWW.REAL.COM• WWW.ABC.COM

Real Networks Tools Beta versions available for free

• Real Player – Helper App to play Real player audio and

video

• Real Publisher – Set of tools to encode audio/video to support

a variety of speeds (network and computer)– Supports SURESTREAM ™

• Automatically switches speeds

• REAL SERVER– Host content

Other Features

• REAL TOOLS– REAL PIX - Stream images– REAL TEXT - Stream Text– REAL FLASH - Stream Macromedia

Flash– SMIL - Synchronize all of the above

Final Project Ideas

• Stern TV Station - Multiple Channels– Live events– Archived events– News?

• Video classes with power point slides coordinated with talk.