An Introduction to MPEG

Based on B. Girod and M. van der Schaar’s lecture notes

Motion JPEG

JPEG system for compressing static images could be applied to a sequence of images, compressing each individually, this is called motion JPEG

Motion JPEG takes no advantage of any correlation between successive images

In a typical scene there will be a great deal of similarity between nearby images of the same sequence.

In a typical scene there will be a great deal of similaritybetween nearby images of the same sequence.

Motion Compensation ApproachBasic idea of Motion Compensation:

Many “moving” images or image sequences consist of a static background with one or more moving foreground objects. We can get coding advantage from this.

we code the first frame by baseline JPEG and use this frame as reference image.

Treat the second image block by block and compare each block with the same block in the reference image.

For blocks that have identical block in reference image, we only send a special code instead of whole code.

For other blocks, we just encode them as usual.

Motion Compensation Approach(cont.)

Motion Vectorsstatic background is a very special case, we should consider the displacement of the block.Motion vector is used to inform decoder exactly where in the previous image to get the data.Motion vector would be zero for a static background.

Motion Compensation Approach(cont.)

Block Matching--how to find the matching block?Matching criteria:

In practice we couldn’t expect to find the exactly identical matching block, instead we look for close match.Most motion estimation schemes look for minimum mean square error(MMSE) between block.

Matching block size:How large the matching block will affect coding efficiencyblock size MPEG used: 16×16

∑=

−=N

nnn yxIyxI

NMSE

1

2' )),(),((1

Motion Compensation Approach(cont.)

Search range:

It’s reasonable to consider an displacement of 360 pixels/s or about 60pixels/image in standard-definition television.

In real-world scenes there is usually more or faster motion horizontally than vertically, generally the width of search area should be twice the height.

Suggested search range: ±60 pixles × 30 pixles

Motion Compensation Approach(cont.)

ResidualsThe differences between the block being coded and it’s best match are known as residuals.

The residuals maybe encoded and transmitted along with the motion vector, so the decoder will be able to reconstruct the block.

We should compare the bits of transmitting the motion vector plus the residuals with the bits of transmitting the block itself and use the most efficient mechanism.

MPEG-1 Introduction

MPEG: Moving Pictures Experts Group.MPEG-video is addressing the compression of video signals at about 1.5Mbits/sMPEG-1 is asymmetric system, the complexity of the encoder is much higher than that of the decoder.

Table 1: MPEG-1 Constraints

Horizontal picture size ≤768 pixelsVertical picture size ≤ 576 linesNumber of macroblocks ≤ 396Number of macroblocks× picture rate

≤ 396 × 25 = 9900

Picture rate ≤ 30 pictures/sVBV buffer size ≥ 2,621,440 bitsBit rate ≤ 1,856,000 bits/s

MPEG Hierarchy

The six layers of MPEG video bit streamSequence Layer: video clip, complete program item.Group of Pictures Layer(GOP): include three different coding ways.Frame LayerSlice Layer: in case the data is lost or corrupted.Macroblock Layer: 16×16 luminance block.Block Layer(DCT unit)

MPEG: Structure of the Coded Bit-Stream

Sequence layer: picture dimensions, pixel aspect ratio, picture rate, minimum buffer size, DCT quantization matricesGOP layer: will have one I picture, start with I or B picture, end with I or P picture, has closed GOP flag, timing info, user dataPicture layer: temporal ref number, picture type, synchronization info, resolution, range of motion vectorsSlices: position of slice in picture, quantization scale factorMacroblock: position, H and V motion vectors, which blocks are coded and transmitted

GOP-1 GOP-2 GOP-n

I B B B P B B..

Slice-1

Slice-2

…

Slice-N

mb-1 mb-2 mb-n

0 12 3 4 5

Sequence layer

Picture layer

Slice layer

Macroblock layer

8x8 bloc

GOP layer

k

Frame Types in MPEG

Intra frames (I-frames)A I-frame is encoded using only information from within that frame(intra coded) -- no temporal compression(inter coded).

Non-intra frames (P-frames and B-frames)motion compensated information will be used for coding.P frame (predicted frame) use preceding frame as reference imageB frame (bidirectional frame) use both preceding frame and following frame as reference images

Motion estimation for different frames

X Z

Available from earlier frame (X)

Available from later frame (Z)

Y

Coding I Frame

Framebeing coded

DCT QuantizeVariablelengthcode

Referenceframe

InverseDCT

De-quantize

Totransmitbuffer

A typical group of pictures in coding order1 5 2 3 4 9 6 7 8 13 10 11 12

I P B B B P B B B P B B B

I B B B P B B B P B B B P

A typical group of pictures in display order

Coding of Macroblock (cont.)

Intra coding of macroblocks– just as what JPEG does– MPEG has two default quantization tables, one for intra coding, another one for non-intra coding of residuals

8 16 19 22 26 27 29 3416 16 22 24 27 29 34 3719 22 26 27 29 34 34 3822 22 26 27 29 34 37 4022 26 27 29 32 35 40 4826 27 29 32 35 40 48 5826 27 29 34 38 46 56 6927 29 35 38 46 56 69 83

16 11 10 16 24 40 51 6112 12 14 19 26 58 60 5514 13 16 24 40 57 69 5614 17 22 29 51 87 80 6218 22 37 56 68 109 103 7724 35 55 64 81 104 113 9249 64 78 87 103 121 120 10172 92 95 98 112 100 103 99

JPEG quantization table(luminance)MPEG quantization table(for intra coding)

Coding of Macroblock (cont.)

Non-intra coding of macroblocks

The first step is to intra code the macroblock--just in case if we fail to find a reasonable match in motion estimation.

Then we use motion estimation to find the nearest match and get the motion vector. Only luminance samples are used in motion estimation.

Then each DCT block in macroblock will be treated separately. The residuals will be encode by DCT and quantization (use flat table) as in intra coding. DC along with AC.

Motion vectors are coded predictively

Coding of Macroblock (cont.)

P-framesIf the block can be skipped, we just send a “skip” codeotherwise, we compare the number of total bits of inter and intra coding, choose the more efficient one. Mark this block accordingly.

B-framesComparison among three methods of encoding

A Simplified MPEG encoder

Framerecorder DCT Quantize

Variable-lengthcoder

Transmitbuffer

Predictionencoder

De-quantize

InverseDCT

Motionpredictor

Referenceframe

Ratecontroller

IN OUT

Scalefactor

Bufferfullness

Prediction

Motion vectors

DC

MPEG-2

Why another standard?Support higher bit rates e.g., 80-100 Mbits/s for HDTV instead of the 1.15 Mbits/s for SIFSupport a larger number of applicationsThe encoding standard should be a toolkit rather than a flat procedure

Interlaced and non-interlaced frameDifferent color subsampling modes e.g., 4:2:2, 4:2:0, 4:4:4Flexible quantization schemes – can be changed at picture levelScalable bit-streamsProfiles and levels

MPEG-2 Applications

Digital Betacam: 90 Mbits/s videoMPEG-2

Main Profile, Main Level, 4:2:0: 15 Mbits/sHigh Profile, High Level, 4:2:0: adequate, expensiveImage quality preserved across generations of processingMultiview Profile

Stereoscopic view – disparity predictionVirtual walk-throughs composed from multiple viewpoints

MPEG-4Support highly interactive multimedia applications as well as traditional applicationsAdvanced functionalities: interactivity, scalability, error resilience, …Coding of natural and synthetic audio and video, as well as graphicsEnable the multiplexing of audiovisual objects and composition in a scene.

TV/film

Computer Interactivity Telecom

Wireless

•Video on LANs, Internet video

•Wireless video

•Video database

•Interactive home shopping

•Video e-mail, home video

•Virtual reality games, multi-viewpoint training

MPEG-4

MPEG-4 Video CodingBasic video coding

Definition of Video Object (VO), Video Object Layer (VOL), Video Object Plane (VOP)Improved coding efficiency vs. MPEG-1 and MPEG-2Based on H.263 baselineGlobal motion compenstationSpriteQuarter pixel motion compensation

MPEG-4: Video codingObject-based video coding

Binary shape codingGreyscale shape codingPadding for block-based DCT of textureShape-adaptive DCT

DWT for still texture codingMesh animation, face and body animation

MPEG-7Semantic Annotationfor Motion Pictures

Application ScenariosImage Understanding (surveillance)Intelligent VisionSmart Camera/VCRsInformation RetrievalInformation Filtering

Computer-based training

MPEG-7

MPEG-7

Think of MPEG-7 as a meta structure to allow the decoder to access to the semantic the videos.