Adaptive Multi-path Prediction for Error Resilient H.264 Coding

Xiaosong Zhou, C.-C. Jay KuoUniversity of Southern California

Multimedia Signal Processing 2006

Outline Introduction Adaptive Multi-path Prediction for H.264

Observation and Research Motivation Computation of Expected Decoder Distortion

Adaptive Reference Selection (ARS) Scheme Experimental Results Conclusion and Future Work

Introduction Video communication problem

Source

Encode

Output Decode

Erroneous

Channel

Introduction Error Resilient tools

Error Resilient Entropy Coding (EREC)

Unequal Error Protection by Layered Coding

… Mismatch exists,

error propagation can’t be stopped properly

Not compatible with H.264

Introduction One way for reducing error propagation

Intra Refreshing : insert intra macroblocks in temporally coded (P or B) video frames Intra macroblocks in H.264 are coded by intra prediction

based on neighbors Intra macroblock have much lower coding efficiency

than inter macroblocks Not suitable for H.264

Introduction Multiple reference (long-term reference)

motion compensation predictive coding (LTMCP) Enhance coding efficiency Proposed for error resilience

T. Wiegand et. al, “Error-resilient video transmission using long-term memory motion-compensated prediction” Select the best reference frame by evaluating the

expected reconstruction calculated based on the error feedback and an error propagation model

Observation and Research Motivation Sequential prediction

In H.264, LTMCP allows encoder to choose the best prediction from a number of reference frames The best reference of some blocks may exist in a

long-term reference frame But, sequential prediction is still common in H.264

Observation and Research Motivation Utilize long term reference frames for error resilience

Error resilience performance is improved using alternative prediction patterns

Different predictive patterns

Observation and Research Motivation If a video stream is encoded into these fixed

prediction patterns As most video frames are forced to use a distant

reference frame, coding efficiency is likely to be sacrificed

Performance vary for different macroblocks since it’s largely dependent on video content

It’s difficult to design a fixed prediction pattern at the frame level

In H.264, reference frame selection is done at macroblock level Incorporate the idea of multi-path predictive

coding at the macrokblock level

Computation of Expected Decoder Distortion Error map

Created and maintained for each allowed reference frame in buffer

Store the absolute value of the expected error e of every pixel in the frame

ffe~ˆ

Encoder Erroneous Channel

Decoderf f~

Encoder Error-free Channel

Decoderf f̂

Computation of Expected Decoder Distortion can not be obtained directed For a pixel in the nth frame, update its value of e by

pe : channel error rate ep,n : expected error from error propagation ec,n : expected error from error concealment when

pixel is lost

f~

ncenpen epepe ,,)1(

Computation of Expected Decoder Distortion To calculate ec,n, the mismatch caused by

reconstruction of error concealment scheme Consider a simple error concealment method

Intra block : copy pixel from the boundary of correctly reconstructed block above the target block

Inter block : copy the block from the same position in the previous frame

Should also consider de-blocking operation which attenuate the error generated by error concealment

d en-1 : error value of the pixel where MV points in reference

frame αn-1 : attenuation factor of the propagating error from n-1st

to nth frame Expected decoder distortion

11, nnnp ee

2, nne eD

Adaptive Reference Selection (ARS) in LTMCP, multiple predictions can be created

to encode block M

The predictions generated from multi-reference frames are evaluated based on both coding and error resilience performance

Adaptive Reference Selection (ARS) Three used vectors

xN = (X1, … , XN) : the set of all N MBs in a GOP mN = (M1, … , MN) : all modes selected by each MB qN = (Q1, … , QN) : quantization parameters used to

encode these MBs

The overall mode decision problem is :

gNNNNNNqm

RqmxRqmxDNN

),,( subject to ),,(min,

Adaptive Reference Selection (ARS) Convert to an unconstraint optimization

problem using Lagrange Multiplier method

),,( ),,(),,(

where

, ),,(min 1

,

NNiNNiNNi

N

iNNi

qm

qmXRqmXDqmXJ

qmXJNN

Adaptive Reference Selection (ARS) Overall expected distortion through erroneous

channel

: expected error that has mean zero : decoder error and uncorrelated to

Therefore

)~ˆ)(ˆ(2)

~ˆ()ˆ(

})]~ˆ()ˆ{[(

])~

[()(

22

2

2

ffffEffEffE

ffffE

ffEDE

eDDffEffEDE 22 )~ˆ()ˆ( )(

ff~ˆ ff ˆ ff

~ˆ

Adaptive Reference Selection (ARS) Rewrite mode decision problem equation

Assumption: rate and distortion of MB i have no impact to other MBs

: expected overall distortion of the GOP due to error propagation

),,( ),(),,(),,(

where, ),,(min1

,

NNiNieNNiNNi

N

iNNi

qm

qmXRmXDqmXDqmXJ

qmXJNN

)],,(

),()1(),,([min),,(min,,

iii

iieiiiQM

iiiQM

QMXR

MXDQMXDQMXJiiii

),,( iii QMXR ),,( iii QMXD

),( iie MXD

Adaptive Reference Selection (ARS) d α : attenuation factor M : expected number of frames in the future

prediction path of the pixel The method to calculate M

M

j

j

1

2

)( bNb

aM

Experimental Results

Experimental Results

Experimental Results Averaged Frame Index

Experimental Results R-D performance comparison under test condition

T2

Experimental Results R-D performance comparison under test condition

T7

Conclusion and Future Work An adaptive prediction selection scheme was

proposed in this work to create multiple prediction paths in the compressed video stream

The proposed scheme is able to maintain good coding efficiency of the compressed stream while serve as an effective error resilience tool in visual communication applications

In the future, we plan to develop a new model to simplify the calculation of the expected decoder distortion to reduce the complexity of the proposed scheme