towards practical distributed coding

Towards Practical Distributed Coding

Bernd GirodBernd Girod

Information Systems Information Systems LaboratoryLaboratory

Stanford UniversityStanford University

B. Girod: Towards Practical Distributed Coding 2

Outline

Distributed lossless compression– Simple examples– Slepian-Wolf Theorem– Slepian-Wolf coding vs. systematic channel coding– Turbo codes for compression

Lossy compression with side information– Wyner-Ziv Theorem– Optimal quantizer design with Lloyd algorithm

Selected applications– Sensor networks– Low complexity video coding– Error-resilient video transmission


Simple Example

1212

Xx p xabcd

101001000

001011000

Source A

1212

Xx p xabcd

Source B

Source statistics exploited in the encoder.Different statistics Different code.


Simple Example - Revisited

1212

Xx p xabcd

0101

Source A

1212

Xx p xabcd

Source B

0101

Different statistics Same code. Source statistics exploited in the decoder.“Lossless” compression with residual error rate.


X

| |121 12 2

12

| |0

00 0

X Y X Yx p x A p x Babcd

Compression with Side Information

SourceA/B

Encoder Decoder

Y

141214

0

Xx p xabcd

X

1.5 bitR H X

X

Y Y | 1 bitR H X Y ??R


Distributed Compression of Dependent Sources

Source X

Source Y

Encoder X

Encoder Y

JointDecoder

1 8 1 81 8 1 8

1 8 1 81 8 1 8

XYp x a b c dy a

bcd

X

Y

??XR

??YR


Achievable Rates for Distributed Coding Example

[bits]XR

[bits]YR

1

1

2

2

Separate encodingand decoding of X and Y

Separate encodingand joint decoding of X and Y


General Dependent i.i.d. Sequences

[bits]XR

[bits]YR

H X

H Y

|H Y X

|H X Y

,X YR R H X Y

Vanishing error probabilityfor long sequences

No errors

[Slepian, Wolf, 1973]


Distributed Compression and Channel Coding

Idea Interpret Y as a “noisy” version of X with

“channel errors” Encoder generates “parity bits” P to

protect against errors Decoder concatenates Y and P and

performs error-correcting decoding

SourceX|Y

Encoder Decoder

Y

X

YX Y

010011000101010100110101010100000001000000

XY

P


Practical Slepian-Wolf Encoding

Coset codes [Pradhan and Ramchandran, 1999] Trellis codes [Wang and Orchard, 2001] Turbo codes

[Garcia-Frias and Zhao, 2001] [Bajcsy and Mitran, 2001] [Aaron and Girod, 2002]

LDPC codes [Liveris, Xiong, and Georghiades, 2002]


12

n

RX

Interleaver length L

1PX

XL bits in

L bitsSystematic Convolutional Encoder Rate n

n 1

1nL

bits

2PX

Systematic Convolutional EncoderRate n

n 1

1nL

bits

L bits

Compression by Turbo Coding

[Aaron, Girod, DCC 2002]



1PX

L bits out

“Channel” probabilities calculations

1nL

bits in 2PX “Channel”

probabilities calculations

Y

1nL

bits in

)|( yxP

SISO Decoder

Pchannel

PextrinsicPa priori


Deinterleaver length L

SISO Decoder

Pchannel

Pextrinsic Pa prioriDeinterleaver

length L

Decision X

Pa posteriori

Pa posteriori


Turbo Decoder


0.15 bit

Results for Compression of Binary Sequences

X,Y dependent binary sequences with symmetric cross-over probabilities

Rate 4/5 constituent convolutional codes;

RX=0.5 bit per input bit

[Aaron, Girod, DCC 2002] |H X Y


)|( YXHRX

),()(

YXHYHRX

1.3511.445

1.033 1.0551.114

1.404

0.000

0.200

0.400

0.600

0.800

1.000

1.200

1.400

1.600

1 2 3

Results with Puncturing

0.5 bitsxR 0.25 bitsxR 0.125 bitsxR



Channel Coding vs. Slepian-Wolf Coding

Systematic Channel Coding

Systematic bits and parity bits subject to bit-errors

Mostly memoryless BSC channel

Low bit-error rate of channel

Rate savings relative to systematic bits

Slepian-Wolf Coding

No bit errors in parity bits

General statistics incl. memory

Whole range of “error” probabilities

Rate savings relative to parity bits

Might have to compete with conventional compression (e.g., arithmetic coding)


Distributed Lossy Compression of Dependent Sources

Source X

Source Y

Encoder X

Encoder Y

JointDecoder

X’

Y’

XR , 'XD E d X X

, 'YD E d Y YYR

, , , ??X Y X Yf R R D D Achievable rate region


X

Lossy Compression with Side Information

'X

X XR R D

Source Encoder Decoder

Y

, 'XD E d X X

Y Y

|X X Y XR R D | |wz

X X Y X X Y XR R D R D [Wyner, Ziv, 1976]

| |

, i.i.d. jointly Gaussian, mse distortion

wzX Y X X Y X

X Y

R D R D

| | |

General statistics, mse distortion

0.5 bits wzX Y X X Y X X Y XR D R D R D

[Zamir,1996]


Practical Wyner-Ziv Encoder and Decoder

Wyner-Ziv Decoder

QuantizerSlepian-

Wolf Encoder

Wyner-Ziv Encoder

Slepian-

WolfDecode

r

Minimum Distortion

Reconstruction

Y Y

X 'XQ Q


Non-Connected Quantization Regions

Example: Non-connected intervals for scalar quantization

Decoder: Minimum mean-squared error reconstruction with side information

x

1q 2q 3q

x

| |X Yf x y 2

ˆ

conditional centroid

ˆ ˆ | ,arg min x

x E X x y q


Quantizer Reconstruction Function

Side information y

Reconstructionx̂

ˆ 1x

ˆ 2x

ˆ 3x

ˆ 1,x y

ˆ 2,x y

ˆ 3,x y


Finding Quantization Regions

1 2 3 4 1 2q =

1

2

3 4

x

Lagrangian cost 2

| ˆ, , , |Y Xj x q E x x q Y r q Y X x

,j x q


Lloyd Algorithm for Wyner-Ziv Quantizers

EndY

Find best reconstruction functions for current quantizers ),(ˆ )( yqx k

(1) ( )q xChoose initial quantizers 1k

Update rate measure for current quantizers ),()( yqr k

Lagrangian cost for current quantizers, reconstructor and rate measure

)(kJ

NFind best quantizers for current reconstruction and rate measure

( 1) ( )kq x

Convergence1kk

[Fleming, Zhao, Effros, unpublished]

[Rebollo-Monedero, Girod, DCC 2003]


Which Rate Measure?

X

Wyner-Ziv Decoder

QuantizerSlepian-

Wolf Encoder

Wyner-Ziv Encoder

Slepian-

WolfDecode

r

Minimum Distortion

Reconstruction

Y Y

'XQ

Conditional entropy coder H(Q|Y)

2 |

2 |

, | log |

hence, use , log |Q Y

Q Y

E r Q Y H Q Y E p Q Y

r Q Y p Q Y

[Rebollo-Monedero, Girod, DCC 2003]


Example

Data set: Video sequence Carphone, 100 luminance frames, QCIF

X: pixel values in even frames

Y: motion-compensated interpolation from two adjacent odd frames

0 50 100 150 200 2500

0.2

0.4

0.6

0.8

1

x

p X|Y(x|y)

0 50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

x

p X(x)


Example (cont.)

Quantizer w/ rate constraint H(Q) Quantizer w/ rate constraint H(Q|Y)

PSNR=39 dBH(Q)=3.05 bit H(Q|Y)=0.54 bit

PSNR=37.4 dBH(Q)=1.87 bit H(Q|Y)=0.54 bit


Example (cont.)

0 1 2 3 4 5 632

34

36

38

40

42

44

46

48

50

Rate [bit]

PS

NR

[dB

]

DistributedUniformConditionalDistributed at R=H(Q|Y)

0 0.5 1 1.5 2 2.532

34

36

38

40

42

44

46

48

50

Rate [bit]

PS

NR

[dB

]DistributedUniformConditional

Quantizer w/ rate constraint H(Q) Quantizer w/ rate constraint H(Q|Y)

Rate [bit] Rate [bit]

PS

NR

[dB

]

PS

NR

[dB

]


Wyner-Ziv Quantizers: Lessons Learnt

Typically no quantizer index reuse for rate constraint H(Q|Y) and high rates: Slepian-Wolf code provides more efficient many-to-one mapping in very high dimensional space.

Uniform quantizers close to minimum m.s.e., when combined with efficient Slepian-Wolf code

Quantizer index reuse required for rate constraint H(Q) and for fixed-length coding

Important to decouple dimension of quantizer (i.e. scalar) and Slepian-Wolf code (very large)


Sensor Networks

Central Unit Local Sensor

Remote Sensor

Side Information

Remote Sensor

Remote Sensor

Remote Sensor [Pradhan, Ramchandran, DCC 2000]

[Kusuma, Doherty, Ramchandran, ICIP 2001][Pradhan, Kusuma, Ramchandran, SP Mag., 2002][Chou, Perovic, Ramchandran, Asilomar 2002]


Video Compression with Simple Encoder

Reconstruction X’

Y

Interframe Decoder

Scalar Quantizer

Turbo Encoder Buffer

Video frameX

Intraframe Encoder

Turbo Decode

rRequest bits

Slepian-Wolf Codec

InterpolationKey frames

previous

next[Aaron, Zhang, Girod, Asilomar 2002][Aaron, Rane, Zhang, Girod, DCC 2003]

http://www.polarisusa.com/cgi-bin/view_product_detail.pl?product_category_id=17&template_id=17


Decoder Side information After Wyner-Ziv Decoding


16-level quantization (~1 bpp)


Decoder Side information After Wyner-Ziv Decoding16-level quantization (~1 bpp)



Performance of Simple Wyner-Ziv Video Coder

7 dB


Digitally Enhanced Analog Transmission

Forward error protection of the signal waveform Information-theoretic bounds [Shamai, Verdu, Zamir,1998] “Systematic lossy source-channel coding”

AnalogChannel

Wyner-Ziv

Encoder

DigitalChannel

Wyner-Ziv

Decoder

Sideinfo


Forward Error Protection for MPEG Video Broadcasting

MPEG Encoder

MPEG Decoder with Error

Concealment

Err

or-P

rone

cha

nnel

S S’

Wyner-Ziv Decoder A S*

Wyner-Ziv Encoder A

Wyner-Ziv Decoder B S**Wyner-Ziv

Encoder B

Graceful degradation without a layered signal representation


Error-resilient Video Transmissionwith Embedded Wyner-Ziv Codec

Carphone CIF, 50 frames @ 30fps, 1 Mbps,1% Random Macroblock loss

[Aaron, Rane, Rebollo-Monedero, Girod, ICIP 2003]


Towards Practical Distributed Coding:Why Should We Care?

Chance to reinvent compression from scratch– Entropy coding– Quantization– Signal transforms– Adaptive coding– Rate control– . . .

Enables new compression applications– Sensor networks– Very low complexity encoders– Error-resilient transmission of signal waveforms– Digitally enhanced analog transmission– Unequal error protection without layered coding– . . .

The EndThe End

towards practical distributed coding

Documents

compressionlossy compression

parity bits p

parity bits subject

channel codingideainterpret

channel errors dencoder

interleaver length lbitsbitsb

turbo codes garciafrias

residual error rate