Scalable Joint Source-Network Coding of Video

Yufeng Shan

Supervisors: Prof. Shivkumar Kalyanaraman Prof. John W. Woods

April 2007

2

Outline Problem statement and motivations Our proposed techniques

Fine grain adaptive FEC (FGA-FEC) Generalized FGA-FEC for wireless networks Overly multi-hop FEC scheme Distributed FGA-FEC over multihop network

Conclusion and future work

3

Problem statement and challenges

Problem:Simultaneously streaming video to diverse users, such as

powerful PCs, laptops and handset devices, over heterogeneous networks.

Challenges:Different users may have different video frame rate /resolution

/quality preferences. May be in different networks, such as high speed wired network,

multi-hop ad hoc wireless network or cellular network.Neither the network nor the video application can provide

quality assurances working independent of each other.

4

Conventional Approaches

Windows media, Real player Send separate copies of the same bitstream to different users,

bandwidth utilization is inefficient (unicast to each user).

Layered multicast (by Steven McCanne) Send different video layer to different multicast groups,

limited by number of video layers, no efficient error correction approach.

Proxy-based streaming Caches video content at local proxy disk and transcodes it for

different users, delay and computation inefficiency.

5

Our Techniques

Router Router

Router

Router

Router

Router

Video server

DSN

DSN DSN

DSN

DSN DSN

Encoding

Scalable joint source-network coding:

Scalable video + Scalable FEC + in network adaptation

FGA-FEC

Generalized FGA-FEC

OM-FEC

Distributed FGA-FEC

Header CRC/FEC

Cross-layer FEC

Adaptation

Adaptation

VideoPerforman

ce

Video Performan

ce

G-FGA-FEC

G-FGA-FEC

G-FGA-FEC

FGA-FECDe/re-code

FGA-FECDe/re-code

6

Outline Problem statement and motivations Our proposed techniques

Fine grain adaptive FEC (FGA-FEC); Generalized FGA-FEC for wireless networks; Overly multi-hop FEC scheme; Distributed FGA-FEC over multihop network

Conclusion and future work

7

Background -- scalable video

We treat three types of scalability:

SNR (aka quality, bitrate) scalability Spatial (resolution) scalability Temporal (frame-rate) scalability

8

SNR (quality) scalability

Decoding more data corresponds to better quality

Embedded compressed file

9

Resolution scalability

Decoding more data corresponds to larger picture size

Embedded compressed file

10

Frame-rate scalability

One GOP - 16 frames

Original video

Half frame rate – 8 frames

¼ frame rate – 4 frames

1/8 frame rate – 2 frames

Decoding more data corresponds to displaying frames at higher rate

Embedded compressed fileBegin End

11

A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)

A(0,0,0) A(0,1,0) A(0,2,0) A(0,3,0) A(0,4,0)

A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)

A(1,0,0) A(1,1,0) A(1,2,0) A(1,3,0) A(1,4,0)

A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)

A(2,0,0) A(2,1,0) A(2,2,0) A(2,3,0) A(2,4,0)

A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)

A(3,0,0) A(3,1,0) A(3,2,0) A(3,3,0) A(3,4,0)

A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0)

A(4,0,0) A(4,1,0) A(4,2,0) A(4,3,0) A(4,4,0)

A(4,0.1) A(4,1,1) A(4,2,1) A(4,3,1) A(4,4,1)A(4,0,2) A(4,1,2) A(4,2,2) A(4,3,2) A(4,4,2)

Resolution

Quality

Frame Rate

Digital items in view of three forms of scalability (quality / resolution / frame-rate).

3-D representation of bitstream

Adaptation of the bitstream is achieved by choosing a subset of these items along user’s preferred adaptation order:

For example: SNR -> Temporal -> Spatial

12

Next, we need scalable FEC

To protect the scalable video bitstream, we have these basic requirements for channel coding:

High adaptivity: If part of the video bitstream is actively dropped (scaled), FEC protecting that part of data should also be removed

Accuracy: near that of robust non-scalable video Efficiency: Low computational cost

13

Fine Grain Adaptive FEC

Goal: To encode a video to facilitate efficient and

precise adaptation of the encoded bitstream at intermediate overlay nodes for diverse users.

Main idea: Extend existing approaches (PET, MD-FEC) to work with

scalable video and perform the adaptation in the network.

14

Multiple Description FEC (MD-FEC)

If any i out of N packets are received, the decoder can decode up to Ri.

Our work extends this concept to provide fine-grained scalability of FEC, applied to scalable video

by Puri and RamachantranMSB LSB

15

FGA-FEC overview

Server

DSN DSNDSN

30/C/3M

A

B30/C/3M

30/C/2M

C 30/C/1M

D 15/Q/1M

E

F

30/C/1M

G

15/C/384k

15/Q/384k

A-G : users and their requirements (frame rate/resolution/bitrate)

DSN : Data service node, which aggregates video requirements to server and performs data adaptation for users, without transcoding.

FGA-FEC uses overlay infrastructure to assist video streaming to heterogeneous users simultaneously by providing light weight support at intermediate overlay nodes.

Encoding

Adaptation

Adaptation

Adaptation

16

FGA-FEC Encoding

Bitstream is divided into N sections from MSB to LSB Each section is further split into small blocks RS(N,i) codes are applied at block level to section Si, vertically Each block column is independently accessible one description = one network packet

MSB one GOP video bitstream LSBS1 ... Si ... SN

R1 Ri-1 Ri RN-1 RNR0

B1 ... ...Bi ... ...B2i Bmi

Enlarged view of Section S i

B1

B2

B3

Bi

...

Bi+1

Bi+2

Bi+3

...

B2i

B2i+1

...

...

...

B3i

...

...

...

...

Bmi

FEC

FEC

...FEC

FEC

...FEC

FEC

...FEC

FEC

...

...

...

...

...

...

...

...

...

Si(1)

Si(2)

Si(3)

Si(i)

RS

en

cod

ing

... ...S1(1) S2(1) Si(1) ... SN(1)

FEC S2(2) Si(2) ... SN(2)

FEC FEC Si(3) ... SN(3)

FEC FEC Si(i) ... SN(i)

FEC FEC FEC ... SN(i+1)

FEC FEC FEC ... SN(N)

...

...

Sect. 1 Sect. 2 Sect. i

Description 1

Description 2

Description 3

Description i

Description i+1

Description N

...

...

Sect. N

...

...

...

...

...

...

...

...

... ... ... ...

17

FGA-FEC – optimal rate allocation

N

iii RDqRDE

0

)()]([

FEC assignment is optimized by minimizing the expected distortion over the channel with packet loss probability p, available bandwidth B.

Subject to:

],1[,

0

1

21

NiirRR

BR

RRR

iii

total

N

where qi is the probability of receiving i out of N packets, Rtotal is the total bandwidth occupied by both FEC and video data. Also ri is the source rate of section Si. N is the number of descriptions encoded for each GOP

The result of the optimization is the video source-rate breakpoints Ri

18

FGA-FEC adaptation at intermediate nodes

DSNf

One description

OneGOP

Adapted GOP

Direct truncation: only shorten each packet in a GOP by removing unwanted blocks. FGA-FEC adaptation: adapt the encoded GOP by a combined shortening and/or

dropping packets. An algorithm is proposed to near optimally adapt the encoded GOP for available

bandwidth and user preference No video or FEC transcoding, only a packet shorting/dropping

Two adaptation methods are used in FGA-FEC :

At DSN Adapted description One description

one GOP

DSNf

One description

OneGOP

Adapted GOP

Shorten& drop

19

Simulation-- FGA-FEC adaptation vs. optimal decoding/recoding

S D

2

8

9

1

500 K

600 K

1000 K

1100kDSN

User 1

Video Server

User 2

User 6

User7

Comparison of FGA-FEC, optimal decoding/recoding solution, and direct truncation at different available bandwidths; In FGA-FEC, MC-EZBC encoded GOP #7, Foreman CIF is adapted from 1100 Kbps to satisfy different users; FGA-FEC optimization is based on 15% loss rate at source, N=64.

6

7

20

ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding

Source1

13

12

10

11

9

8

7

65

43

2

15

14

1.64

0.73

0.60

0.66

0.66

1.481.52

1.44

1.32

1.381.451.25

1.32 1.51

1.77

Group4

Group3

Group2

Group1

MD-FEC transcoding:

FEC decode/recode at each intermediate node

FGA-FEC adaptation:

Encode at server, adaptation in the intermediate nodes

Want to see the received video quality at User 4 and User 12,

Link loss probability is 0.01, N=64, Foreman, CIF

21

ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding

Receiver 4

FGA-FEC is 0.01dB lower

The quality loss is due to the adaptation precision.

Receiver 12

FGA-FEC is 0.4 dB lower

Vidview.exe

22

ns-2 Simulation-- FGA-FEC vs. Layered multicast

Layered multicast:

Base layer bit rate 0.6, and enhancement layers 0.32, 0.42, 0.32 Mbps, respectively

FGA-FEC adaptation:

Encode at server, adaptation in the intermediate nodes

Want to see the received video quality at User 5 and User 7, Link loss probability is 0.01, N=64, Foreman, CIF

Source1

13

12

10

11

9

8

7

65

43

2

15

14

1.64

0.73

0.60

0.66

0.66

1.481.52

1.44

1.32

1.381.451.25

1.32 1.51

1.77

Group4

Group3

Group2

Group1

23

ns-2 Simulation-- FGA-FEC vs. Layered multicast

Receiver 5

Layered multicast can subscribe up to two layers

The quality loss of layered multicast is due to coarser available layers.

Receiver 7

Layered multicast can subscribe up to three layers

Vidview.exe

24

Outline Problem statement and motivations Our proposed techniques

Fine grain adaptive FEC (FGA-FEC); Generalized FGA-FEC for wireless networks; Overly multi-hop FEC scheme; Distributed FGA-FEC over multihop network

Conclusion and future work

25

Generalized FGA-FEC for wireless

channel bit errors due to channel fading and noise; large bandwidth fluctuations, and intermediate node computational capability constraint limited maximum transmission unit (MTU) size.

In addition to congestion related packet losses, in wireless must deal with:

26

Generalized FGA-FEC for wireless

A kind of Product codes (extend Sherwood and Zeger’s codes):

Column codes: Reed Solomon for packet loss

Row codes: CRC + BCH for bit errors

CRC

CRC

CRC

CRC

CRC

BCH

BCH

BCH

BCH

BCH

FGA-FEC + BCH/CRC Encoded Packet 1

FGA-FEC + BCH/CRC Encoded Packet 2

FGA-FEC + BCH/CRC Encoded Packet 3

FGA-FEC + BCH/CRC Encoded Packet k

FGA-FEC + BCH/CRC Encode Packet k+1

…

…CRC

BCHFGA-FEC + BCH/CRC Encoded Packet N

S1(1) S2(1) ... Sk(1) ... SN(1)

FEC S2(2) ... Sk(2) ... SN(2)

FEC FEC ... Sk(3) ... SN(3)

FEC FEC ... Sk(k) ... SN(k)

FEC FEC ... FEC ... SN(k+1)

FEC FEC ... FEC ... SN(N)

... ...

... ...

Sect. 1 Sect. 2 Sect. k... ...Description 1

Description 2

Description 3

Description k

Description k+1

Description N

...

...

S1 ... Sk ... SN

R1 Rk-1 Rk RN-1 RNR0

Sect. N

RS

en

code

d in

blo

ck le

vel

Embedded bitstream

Generalized FGA-FEC = FGA-FEC Bit level protection+

A BCH code is represented as: BCH(n,k,t): n=2m-1, n-k<=mt

27

Optimal product code assignment

Find a concatenated column RS code assignment cc and row BCH code assignment cr from a set of RS codes CRS and BCH codes CBCH BCH(n,k,t), such that the end-to-end video distortion over a lossy channel is minimized.

],,|[, minarg,

CRCBCHRS

CcCc

rc CCCDEccBCHrRSc

BRRRR BCHCRCRSS Subject to channel rate constraint:

where Rs , RRS , RCRC and RBCH are rates allocated to the video source, RS parity bits, CRC and BCH check bits, respectively. Here B denotes the maximum available channel bandwidth

28

Fast optimization method

jnb

jb

t

jBCH pp

j

nCP

)1()(

0

Given a BCH code BCH(n,k,t)

Exhaustive search results

Probability of correctly decoding BCH code

Near optimal points, search starts here,

it can find the optimal solution within a few iterations

GOP #7 of Foreman, CIF, N= 64, pdrop=0.05, BER pb varies

n=8191, m=13

29

Multi-cluster descriptions coding

Multi-cluster descriptions coding: one GOP Multiple of N descriptions due to:

Limited network packet size; Easier adaptation, such as encode each temporal layer as one cluster as below.

Example of encoding one GOP into 3 clusters, each cluster is coded with product codes

Single-cluster description coding: one GOP N descriptions

One GOP of scalable bitstream

Low frame rate

Mid frame rate

High frame rate

30

Simulation – FGA-FEC vs. MD-FEC in wireless

Channel changes over time between node 2 and 3

Vidview.exe

Consider two adaptation orders:

SNR->Temporal, SNR-> Spatial

Frame rate scaling Resolution scaling

pdrop=0.05 at node2

31

Outline Problem statement and motivations Our proposed techniques

Fine grain adaptive FEC (FGA-FEC); Generalized FGA-FEC for wireless networks; Overly multi-hop FEC scheme; Distributed FGA-FEC over multihop network

Conclusion and future work

32

Motivations -- an example

Hop 1 2 3 4 5 6

B(i) 660K 625K 615K 700K 900K 1100K

P(i) 2.5% 3% 3.5% 2.5% 1.5% 1%

Sender ReceiverN1 N5N2 N3

(B6 , P6 , RTT6 )(B1, P1, RTT1 )

Given an overlay path:

The parameter of each hop

FEC method

Endto end

Hop-by-hop

OM-FEC

data rate(goodput)

529K 594 K 594K

Path loss rate

14% 14% 14%

OM-FEC vs. Traditional FEC

Segment 2 Segment 3Segment 1

N4

33

OM-FEC algorithm

][][ hopbyhopFECOM DEDE

i L M

N1 NnNh NjNi……… …

OM-FEC partitions a given overlay path into segments:

By solving the following: minimize(Nsegement)

min{Bstart , …, Bi} (Bdata + BFEC(start>i)) min{Bstart, …, Bi+1}< (Bdata +BFEC(start>(i+1))

FEC that should be added to the Segment is BFEC(start ->i),

Subject to:

The i-th node is boundary node, if the two inequalities hold:

34

Video Simulations

Topology: loss rate on each hop (L1–L10) random within [0.4%-1.2%]

H.263+ encoded Foreman CIF, at bitrate 1530 Kbps;

Available bandwidth for each hop: 1656 Kbps

Average number of segments partitioned is only 2 vs. 10 in hop-by-hop FEC

35

PlanetLab Experiments

Video: Foreman QCIF, H.263+, 512 Kbps, 30 fps, I frame every second Packet-loss rate from Utah to CMU is set to 5%, other links are set to 1%. The available bandwidth from Utah to CMU is also upper bounded to 550 Kbps. Two paths: 1): with nodes shown at map 2): Add Node 4: planet1.ecse.rpi.edu at last hop with 1% loss rate

planetlab1.flux.utah.edu

planetlab-1.cmcl.cs.cmu.edu

planetlab1.cs.cornell.edu

video.testbed.ecse.rpi.edu

nima.eecs.berkeley.edu

36

OM-FEC experimental results

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250 300 350Frame Number

PS

NR

(db)

our OM-FEC Scheme End-to-End Scheme

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250 300 350Frame Number

PS

NR

(db

)

our OM-FEC Scheme End-to-End Scheme

Video PSNR of OM-FEC scheme vs.

e2e scheme (5 virtual links)

Video PSNR of OM-FEC scheme vs.

e2e scheme (4 virtual links)

Results show: • OM-FEC outperforms e2e FEC scheme• As more congested links are encountered, gain due to OM-FEC increases

Large PSNR loss of the e2e scheme is due to the dependency of bitstream, one packet drop could cause later packets of the same GOP useless .

Vidview.exe

37

Outline Problem statement and motivations Our proposed techniques

Fine grain adaptive FEC (FGA-FEC); Generalized FGA-FEC for wireless networks; Overly multi-hop FEC scheme; Distributed FGA-FEC over multihop network

Conclusion and future work

38

Problem statement and our solution

In FGA-FEC

We assume no congestion between DSNs

Remaining problem:– What if there is congestion between DSNs? For

example, in a rate constrained multihop ad hoc wireless network.

Our solution – distributed FGA-FEC: - FEC coding and optimization algorithm will be

run at DSNs in a distributed way to serve the diverse users.

39

The optimization algorithm

Minimize the e2e distortion for each user:

N

iiii RDqRDE

0

)()]([

],1[,0

0

1

21

NirirRR

BR

RRR

iiii

total

N

Subject to:

iNii pp

i

Nq

)1(

where

N

iiii

N

itotal RR

ii

NR

11 )1(

Problem: Solution:

Use Lagrange multiplier method:

)()(),,...,,(1 1

21 BRRDqRRRFN

i

N

iiiiiN

N

iii

N

N

N

N

i

i

i

i

BR

qR

RD

qR

RD

qR

RD

0

1

1

1

1

)(

)(

)(

Given one value of λ,

i

i

i

i

qR

RD

)(

corresponds to the point on the D(R) curve with slope equal to

i

i

q

Solution can be found by searching over λ

40

The Distributed FGA-FEC Algorithm

To reduce the overall computation while maintain the best possible video quality, we propose two approaches:

A coordination method between optimization processes running at adjacent nodes to reduce the optimization computation. Full search, search with previous GOP, search with neighbor

Applied the idea of OM-FEC to reduce the number of FGA-FEC decode/recode nodes, do FEC computation at selected DSNs

41

Simulation - reduce optimization computation

Test number of λ iterations need to reach optimization stop points, Foreman, CIF.

The proposed coordination between adjacent nodes can reduce the optimization computation requirement.

42

Simulation– distributed FGA-FEC vs. Hop-by-hop FEC decode/recode

Hop-by-hop FGA-FEC decode/recode:

FGA-FEC decode/recode at each intermediate node

Distributed FGA-FEC:

Identify the congested links and select appropriate nodes to do FGA-FEC decode/recode.

Want to see the received video quality at User 5 and User 12

Link loss probability is 0.01, N=64, Foreman, CIF

Source1

13

12

10

11

9

8

7

65

43

2

15

14

1.15

0.73

0.60

0.66

0.66

1.481.32

1.44

1.32

1.381.151.25

1.32 1.51

1.25

Group4

Group3

Group2

Group1

43

Simulation– distributed FGA-FEC vs. Hop-by-hop FGA-FEC decode/recode

Receiver 5

The two schemes deliver similar video quality, but distributed FGA-FEC uses fewer FEC computation nodes 3 vs. 6 at FEC decode/recode

Receiver 12

Vidview.exe

44

Measured CPU time

We measure the CPU time using FGA-FEC adaptation and FGA-FEC decode/recode in a Dell PC with 1.6 GHZ CPU, 256 M memory, running

Red Hat linux 8.2, N=64.

Scheme performed at DSN CPU time (ms)

FGA-FEC decode/recode 52.6

FGA-FEC adaptation 2.9

FGA-FEC direct truncation 10-2

Distributed FGA-FEC can greatly reduce the computation burden, while can deliver near optimal video quality

45

Conclusions Fine grain adaptive FEC (FGA-FEC)

to encode and adapt a scalable video Generalized FGA-FEC for wireless networks

to do encoding and adaptation over wireless networks Overly multi-hop FEC scheme

to efficiently utilize one congested path Distributed FGA-FEC over multihop network

to stream video over a congested heterogeneous networks Header error correction

To improve the effective throughput of wireless networks Cross-layer FEC

To joint optimize protocol with application layer FEC

46

Future work

Evaluate FGA-FEC over DCT-based standard coders such as H.264/AVC and SVC

Extend the distributed FGA-FEC to work with Spatial and Temporal scalability

Extend FGA-FEC idea to multi-point video conferencing

47

[1] Yufeng Shan, Ivan Bajic, John W. Woods and Shivkumar Kalyanaranman “ Scalable video streaming with fine grain adaptive forward error correction” submitted to IEEE trans. CSVT, 2006

[2] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Video streaming over 802.11 Ad Hoc wireless ntworks with header error protection", submitted to Ad Hoc Networks, 2006

[3] Yufeng Shan, Ivan V. Bajic, Shivkumar Kalyanaraman and John W. Woods, “Overlay multi-hop FEC scheme for video streaming,” Signal Processing: Image Communications Vol. 20/8, 2005

[4] Yufeng Shan, John. W. Woods and Shivkumar Kalyanaraman “Fine grain adaptive FEC over wireless networks”, submitted to ICIP 2007

[5] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Header error protection for multimedia data transmission in wireless AdHoc networks", ICIP, 2006

[6] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Joint source-network error control coding for ccalable overlay streaming," ICIP, 2005

[7] Yufeng Shan, Su Yi, Shivkumar Kalyanaraman and John.W. Woods, "Two-Stage FEC scheme for scalable video transmission over wireless networks" SPIE Communications/ITCom, Multimedia Systems and Applications, Oct. 2005

[8] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Overlay multi-hop FEC scheme for video streaming over peer-to-peer networks," ICIP, 2004

[9] Yufeng Shan and Shivkumar Kalyanaraman “Hybrid video downloading/streaming over peer-to-peer networks,” ICME, 2003 

Publication and submissions

48

Thanks