thomas wiegand technische universität berlin€¦ · · 2017-10-27thomas wiegand technische...

Image and Video Coding I:Fundamentals

Thomas Wiegand

Technische Universität Berlin

Organization

Vorlesung: Donnerstag 10:15-11:45Raum EN-368

Material: http://www.ic.tu-berlin.de/menue/studium_und_lehre/

Literatur:Cover, T. M. and Thomas, J. A. (2006), “Elements of Information Theory,”John Wiley & Sons, New York.Gersho, A. and Gray, R. M. (1992), “Vector Quantization and SignalCompression,” Kluwer Academic Publishers, Boston, Dordrecht, London.Jayant, N. S. and Noll, P. (1994), “Digital Coding of Waveforms,”Prentice-Hall, Englewood Cliffs, NJ, USA.Wiegand, T. and Schwarz, H. (2010), “Source Coding: Part I ofFundamentals of Source and Video Coding,” Foundations and Trends inSignal Processing, vol. 4, no. 1-2. (available on web page)

T. Wiegand (TU Berlin) — Image and Video Coding

http://www.ic.tu-berlin.de/menue/studium_und_lehre/

Introduction

encoder decoderbitstream

Motivation

Motivation for Source Coding

Source Coding / Data Compression

Efficient transmission or storage of dataUse less throughput for given dataTransmit more data for given throughput

Source Coding: Enabling TechnologyEnables new applicationsMakes applications economically feasible

ExamplesDistribution of digital imagesDistribution of digital audio (first mobile audio players)Digitial television (DVB-T, DVB-T2, DVB-S, DVB-S2)Internet video streaming (YouTube, Netflix, Amazon, ...)

T. Wiegand (TU Berlin) — Image and Video Coding: Introduction 2 / 20

Motivation

Practical Source Coding Examples

File CompressionCompress large document archivesGZIP, WinZIP, WinRar, ...

Audio CompressionCompress audio for storage on mobile deviceFLAC, MP3, AAC, ...

Image CompressionCompress images for storage and distributionRaw camera formats, JPEG, JPEG-2000, JPEG-XR, ...

Video CompressionCompress video for streaming, broadcast, or storageMPEG-2, H.264 | AVC, VP9, H.265 | HEVC, ...


Motivation

Types of Compression

Lossless Compression

Invertible / reversible form of data compressionOriginal input data can be completely recoveredRequired for document compression

Examples: Lempel-Ziv coding (GZIP), FLAC, JPEG-LS

Lossy CompressionNot invertibleOnly approximation of original input data can be recoveredAchieves much higher compression ratiosDominant form of compression for media data (audio, images, video)

Examples: MP3, AAC for audioJPEG, JPEG-2000 for imagesMPEG-2, H.264 | AVC, H.265 | HEVC for video


Source Data Sampling and Quantization

Source Data

Analog Signals

Continuous-time/space and continuous-amplitude signalsTypically resulting from physical measurementsMost signals in reality (audio and visual signals)

Digital Signals

Discrete-time/space and discrete-amplitude signalsOften generated from analog signalSometimes directly measured

Input Data for Source CodingRequire digital signalsNeed to be stored and processed with a computerCompression methods are computer programs



Analog-to-Digital Conversion

Sampling

Maps continuous-time/space signal into discrete-time/space signal

QuantizationMaps continuous-amplitude signal into discrete-amplitude signalSimplest case: Rounding



Impact of Sampling (Spatial Resolution)

400×300 samples 200×150 samples

100×75 samples 50×38 samples



Impact of Quantization (Bit Depth)

8 bits per component 4 bits per component

3 bits per component 2 bits per component



Raw Images and Videos

Single-Component ImageMatrix of integer samples

s[x , y ]

Characterized byNumber of samples in horizontaland vertical direction W×H

Sample bit depth B

x

y

Color ImagesThree color components (typically RGB or YCbCr)Additionally characterized by color sampling format

VideosSequence of imagesAdditionally characterized by frame rate



Color Sampling Formats

RGB YCbCr 4:4:4 YCbCr 4:2:2 YCbCr 4:2:0

mostcommonformat



Raw Data Rate

Raw Data RateBit rate R of raw data format

R = (samples per time unit) · (bit depth per sample) (1)

Example: Full High Definition (HD) Video

1920×1080 luma samples, 50 frames per second (Europe)4:2:0 chroma format (chroma components with 1/4 luma resolution)8 bits per sampleraw data rate = 50Hz · 1920 · 1080 · (1+ 2/4) · 8 bits ≈ 1.24Gbit/s

Example: Ultra High Defintion (UHD) Video

3840×2160 luma samples, 60 frames per second (USA, Japan)4:2:0 chroma format, 10 bits per sampleraw data rate = 60Hz · 3840 · 2160 · (1+ 2/4) · 10 bits ≈ 7.5Gbit/s


The Source Coding Problem

Typical Video Communication Scenario

capture

raw inputvideo samples

raw inputvideo samples

pre-processing

videoencodervideo

encoder

transmission channel (can be replaced by storage)

channelencodermodulatorchanneldemodulatorchannel

decoder

videodecodervideo

decoderpost-

processing

raw outputvideo samplesraw output

video samples

display andperception

encodedbitstream

receivedbitstream

no transmission errorsbitstream



Application Examples

HD movie UHD broadcaston Blu-ray Disc over DVS-S2

raw video format

1920×1080 luma samples 3840×2160 luma samples4:2:0 chroma format 4:2:0 chroma format8 bits per sample 10 bits per sample24 frames per second 60 frames per second

raw data rate ca. 600 Mbit/s ca. 7.5 Gbit/s

channel bit rate 36 Mbit/s (read speed) 58 Mbit/s (8PSK 2/3)

video bit rate ca. 20 Mbit/s ca. 25 Mbit/s

required compression ca. 30:1 ca. 300:1



The Basic Communication / Source Coding Problem

Basic Source Coding ProblemTwo equivalent formulations:

Representing source data with highest fidelity possiblewithin an available bit rate

and

Representing source data using lowest bit rate possiblewhile maintaining a specified reproduction quality

Source CodecSource codec: System of encoder and decoder

encoder decoderbitstream



Practical Communication / Source Coding Problem

Characteristics of Source Codecs / Overall Communication

Bit rate: Throughput of the communication channelQuality: Fidelity of the reconstructed signalDelay: Start-up latency, end-to-end delayComplexity: Computation, memory, memory access

Practical Source Coding Problem

Given a maximum allowed complexity and a maximum delay,achieve an optimal trade-off between bit rate and reconstructionquality for the transmission problem in the targeted application

In this course:Will concentrate on source codec

Ignore aspects of transmission channel (e.g., transmission errors)


Coding Efficiency

Image Compression Example

100%

Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%


Coding Efficiency

Trade-Off between Quality and Compression Ratio

Coding EfficiencyAbility to trade-off bit rate and reconstruction qualityWant best reconstruction quality for a given bitrate (or vice versa)

codec A

bettercodec B

bit rate

quality


Coding Efficiency

Coding Efficiency Example

1:100 Compression: JPEG1:100 Compression: H.265 | HEVC


Coding Efficiency

Measuring Coding Efficiency

Average Bit Rate

R =number of bits in bitstream for a video sequence

nominal duration of the video sequence(2)

Quality / Distortion

Typically: Mean square error (MSE) and peak-signal-to-noise ratio (PSNR)For a W×H array s[x , y ] of original samples, the array s ′[x , y ] ofreconstructed samples, and a sample bit depth B

MSE =1

W · H∑∀(x,y)

(s[x , y ]− s ′[x , y ]

)2 (3)

PSNR = 10 · log10

((2B − 1)2

MSE

)(4)

For videos: Average PSNR over entire video sequence


Outline

Outline of Course

Image and Video Coding I: FundamentalsProbability, Random Variables, and Random ProcessesLossless CodingRate-Distortion TheoryQuantizationPredictive CodingTransform Coding

Image and Video Coding II: Algorithms and ApplicationsAcquisition, Representation, Display, and Perception of ImagesVideo Coding OverviewVideo Encoder ControlIntra-Picture CodingInter-Picture CodingVideo Coding Standards


thomas wiegand technische universität berlin€¦ · · 2017-10-27thomas wiegand technische...

Documents