e0262 -multimedia information systems anandi giridharan electrical communication engineering, indian...

E0262 -Multimedia Information Systems

Anandi GiridharanElectrical Communication

Engineering,Indian Institute of Science, Bangalore – 560012, India

MULTIMEDIA DATA


Multimedia Applications?

An application that deals with one of more of the following data types: Text Images Audio Video

Example: Media player, web browser


Multimedia applications

Streaming stored audio and video. Streaming live audio and video. Real-time interactive audio and video.· All the above have common characteristics: Delay sensitivity:: End-to-end packet delay.

Delay jitter :: variability of packet delay within the same packet stream.


Multimedia Applications can tolerate packet losses. Occasional packet losses cause minor disturbances during playback.

· Requirement is just the reverse as compared to normal data transmission. Cannot tolerate losses. Can tolerate delay variations.


Multimedia

A multimedia signal is one that integrates signals from several media sources, such as video, audio, graphics, animation, text in a meaningful way to convey some information.

The integration of different media must be done with some definite objective.

we need a system which can acquire the separate media streams, process them together to make it an integrated multimedia stream.


Elements involved in a multimedia transmitter


• Elements of Multimedia Transmitter, Devices like cameras, microphones, keyboards, mouse, touch-screens, storage medium etc. are required to feed inputs from different sources.

• All further processing till the transmission is done by the computer.

• The data acquisition from multiple media is followed by data compression to eliminate inherent redundancies present in the media streams.

• This is followed by inter-media synchronization by insertion of time-stamps, integration of individual media streams and finally the transmission of integrated multimedia stream through a communication channel, which can be a wired or a wireless medium.


• The destination end should have a corresponding interface to receive the integrated multimedia stream through the communication channel.

• At the receiver, a reversal of the processes involved during transmission is required.


• Elements of Multimedia Receiver


Challenges involved with multimedia communication

•Bandwidth limitations of communication channels.•Real-time processing requirements.•Inter-media synchronization.•Intra-media continuity.•End-to-end delays and delay jitters.•Multimedia indexing and retrieval.


Bandwidth limitations

• Limited bandwidth of communication channels poses the most serious Challenge in multimedia communication. •Some of the media streams, such as video sequences, large-sized still images, even stereo quality audio require large volumes of data to be transmitted in a short-time, considering real-time requirements.•Suppose, we want to transmit a color image of 1024 x 768 picture elements through a telephone channel, supported with modem, having a speed of 14.4 Kbits/second.•Assuming 24-bits for each pixel, i.e., 8-bits for each color components (Red, Green and Blue), the total number of bits to be transmitted for the entire image is given by Total number of bits, B = 1024 x 768 x 24 = 18.8 x 106

Therefore, the total time required to transmit this image is (18.8 x 103 /14.4), i.e., approximately 22 minutes, which is excessively high. The total time will be halved, if we double the modem speed, but 11 minutes of transmission time is still too high.


Bandwidth requirement of Video


• Let us now consider the example of a video sequence to be used for video conferencing. We consider a fairly small frame size of 352 x 288 pixels and a colored video sequence having 24-bits / pixel, acquired at a frame rate of 30 frames per second, which is to be transmitted through a leased line of bandwidth 384 Kbits/second. From the given data, the raw video bit rate is (352 x 288 x 24 x 30), i.e., 72.9 Mbits/second. Hence, we cannot transmit the video through the given channel bandwidth, unless the data is significantly compressed.


Spatial Redundancy• Elements that are duplicated within a structure,

such as pixels in a still image and bit patterns in a file. Exploiting spatial redundancy is how compression is performed.

The white spots/pixel on right shoulder and black spots/pixels on his face. Use the redundant information provided by the "good" pixels in the neighborhood to fix the "bad" pixels.


Temporal redundancy• Pixels in two video frames that have the same values in

the same location. Exploiting temporal redundancy is one of the primary techniques in video compression.


• In case of CD-quality stereo audio sampled at 44.1 Khz, the raw data rate is 192 Kbps, which is also somewhat high, if we consider leased line bandwidth or wireless channel bandwidth. Hence, audio compression is also a requirement.

• Other media streams (text, graphics, animation etc) have relatively less data content in most of the applications and may not require compression. These therefore lead to loss of data and quality degradation. There is always a trade-off that is present between compression,i.e, bandwidth reduction and quality.


1.4.1 Real-time processing requirements:Whichever techniques we may adopt to exploit the redundancies and achieve data compression, significant amount of processing will be involved. If the processing time is high, the advantage of data compression may be lost. Challenges are much more in the case of video. Video frames are captured at a rate of 30 frames per second and this leaves a time of 33 milliseconds between successive frames. Hence, video compression and whatever additional processing are required, needs to be completed within one frame time. Although today's processors, operating at GHz rate are extraordinarily fast, quite often even such high speed processors are unable to perform real-time processing and high-speed, dedicated parallel processing hardware may be required. Development of multimedia processing hardware with real-time capability is a highly challenging research topic of today.


Inter-media synchronization

The media streams are available from different and independent sources and are asynchronous with respect to each other. Lack of lip-synchronization is a commonly observed problem in multimedia systems involving audio and video. Multimedia standards have addressed this problem and adopted "time-stamping" to ensure synchronization. Time-stamps, with respect to a system clock reference (SCR) are appended with the different media (audio,video etc) packets before integrating the individual streams into a multimedia one. At the receiver end, the individual media streams are decoded andpresented to the playback units in accordance with the time-stamps obtained from the pack and packet headers.


Traffic Characterization Parameter• Variability of bit rates:

• Constant bit rate (CBR) applications: example uncompressed digitized voice transmission

• Variable bit rate (VBR) applications: example video transmission using compression.

• Different parts of video/audio will not be same.

• Most multimedia applications generate VBR traffic.

• Burstiness ratio=Mean bit rate/Peak bit rate


Intra-media continuity:

Each frame will undergo different extent of compression and this in turn will vary the bit rate (i.e., the number of bits to be transmitted per second) from one frame to the other. If we use a channel that supports constant bit rate, accommodating a variable bit rate source will be a challenging task. This is achieved by providing a buffer before the bit-stream generation. The buffer may be filled up at variable rate, but emptied during transmission at a constant rate. One must therefore ensure that at no instant should the buffer be completely emptied and the channel starves for data. On the other hand, at no instant should the buffer be completely filled up and further incoming data is lost (this is known as buffer overflow problem.. the continuity is lost during presentation.).


End-to-end delays and delay jitters:

In a multimedia broadcast or multimedia conferencing, if the users receive the multimedia contents after considerable delays or different users receive the same contents at different times, the interactivity is lost.


Synchronization Accuracy Specifications (SAS)

Synchronization Accuracy Specifications: Goodness of SynchronizationDelay- Acceptable time between transmitter and receptionDelay jitter- Difference between the desired presentation time and actualPresentation timeDelay skew- Any difference between desired and actual presentationError rate- Level of error is specified by BER (Bit Error Rate)BER=No. Of bits in error/total no. of bits.


Multimedia Indexing and retrieval:

. Unless MM files are properly indexed, retrieval of desired multimedia file becomes a tough task, in view of the search complexities.

If the multimedia files are based on their contents and then a content-based query system is developed, efficient retrieval can be obtained.

Often, for quick browsing of multimedia files, video summaries are needed and automated generation of video summaries is a very challenging task.

All these issues are being addressed in the upcoming MPEG-7 multimedia standards.


Multimedia transmission requirements

• Response of human ear-20Hz to 20KHz

• Ear is sensitive to changes to signal level not absolute value.

• Eye has retention capability for few m-sec before image decays.

• Tolerance to error-Higher error tolerance for uncompressed signals)

• Tolerance to delay and jitter (small for live applications)

• Lip synchronization(most critical)

• Voice and video (recorded and compressed and transmitted separately)


SAS for audio Delay-100 to 500 msec (one way delay)

Should not be greator than for echo concellation.

Delay jitter-10 times better than delay

eg. dely=100 msec, jitter< 10 msec.Lip synchronization: Better than 80ms. Time gap between Audio and Video should be 80msBER=<0.01 for telephoneBER<0.001 UC CDBER<0.0001 for C- CD


Video (SAS) Delay and Jitter

< 50 ms for HDTV

<100 ms Broadcast TV

<500 ms video conferencing

Error rate- <0.000001 for HDTV

<0.00001 for BCTV

<0.0001 for Video conf.


Network Performance parameter(NPP)

Throughput- Effective rate of transmission of information bits.

For Ethernet data rate is 10 Mbps, throughput <3 Mbps due to collision, re-transmission etc.

Delay- End to End delay (250 msec max) based on NW type

Delay variance- 10 msec max

(sending pkts, transmission, propagation time)

Error rate- BER, PER(packet error rate), PLR(packet loss rate), CLR(cell loss rate).

QoS depends on NPP and SAS


Image

Images are an important component of Multimedia. These are generated by the computer in two ways, as bitmap or raster images and as vector images.

Raster or Bitmap Images-Bit map is a simple matrix of the tiny dots called pixel that forms a raster or bitmap image.

Each pixel has two or more colors

Vector maps- based on drawing elements or objects such as lines, rectangles, circles to create an image.


Monochrome/Bit-Map Images- An example 1 bit monochrome image is illustrated in Fig. Sample Monochrome Bit-Map Image: Each pixel is stored as a single bit (0 or 1)A 640 x 480 monochrome image requires 37.5 KB of storage.


Gray-scale Images- An example gray-scale image is illustrated in Fig. Each pixel is usually stored as a byte (value between 0 to 255.

* A 640 x 480 greyscale image requires over 300 KB of storage.


• 8-bit Colour Images

• One byte for each pixel

• * Requires Colour Look-Up Tables (LUTs)

• * A 640 x 480 8-bit colour image requires 307.2 KB of storage


• 24-bit Colour Images

• An example 24-bit colour image is illustrated

• in Fig. where:

• Each pixel is represented by

• three bytes

• (e.g., RGB)

• * Supports 256 x 256 x 256 possible combined

• colours (16,777,216)

• * A 640 x 480 24-bit colour image would

• require 921.6 KB of storage

• * Most 24-bit images are 32-bit images, the extra byte of data for each pixel is used to store an alpha value representing special effect information


Common File FormatsBitmap FormatsGIF: graphics interchange format

JPEG: joint photographic experts group

PNG: portable network graphic

BMP: Windows bitmap

TIFF: tagged image file format

Vector FormatsSVG : scalable vector graphics

EPS: encapsulated postscript

CMX: Corel meta exchange

PICT: Macintosh Picture

WMF: Windows metafile


• Basically, there are two types of compression, lossless and lossy.

• When we say that a compression algorithm is "lossless," that means we can undo the algorithm and completely recover all of the original file. No information whatsoever is lost. GIF (Graphics Interchange Format) and PNG (Portable Network Graphics) are examples of lossless file formats.

• A compression algorithm is "lossy" means that we cannot completely recover the original file after it's been compressed. Some information has been lost. JPEG (Joint Photographic Experts Group) is an example of a lossy file format.


IMAGE

JPEG GIF

Graphics

Computer graphics refers to using a computer to create or manipulate any kind of picture, image, or diagram.”


Animation

Animation consists of still images displayed so quickly that they givethe impression of continuous movement. To give the impression of smoothness the frame rate has to be at least 16 frames per second, and for natural looking motion it should be at least 25 frames per second. Animations may be two or three dimensional. Path Animation Path animations involve moving an object on a screen that has aconstant background e.g. a cartoon character may move across the screen regardless any change in the background or the character. Frame Animation In frame animations, several objects are allowed to move simultaneously and the objects or the background can also change.


Animation

e0262 -multimedia information systems anandi giridharan electrical communication engineering, indian...

Documents

multimedia signal

multimedia transmitter

india multimedia data

media sources

media player

multiple media

separate media streams

integration of different