Download - Lecture 5: Error Resilience
![Page 1: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/1.jpg)
Lecture 5: Error Resilience & Scalability
Dr Reji MathewA/Prof. Jian Zhang
NICTA & CSE UNSWCOMP9519 Multimedia Systems
![Page 2: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/2.jpg)
Outline
� Error Resilience
� Scalability� Including slides from Lecture 3
� Tutorial Question Sheet� Introduction
![Page 3: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/3.jpg)
5.1 Error Resilience
� Most digital storage media and communication channels are not error free.
� Channel coding is beyond the scope of the MPEG specification
� Therefore the compression scheme needs to be robust to errors or loss of data
� Error resilience schemes can be applied at the Encoder & Decoder
![Page 4: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/4.jpg)
5.1 Error Resilience
� Due to temporal prediction � there is dependency between frames� hence errors can propagate from frame to frame� Example below
� an error in the first “I” frame can propagate to all other predicted frames (i.e the last “I” frame is not affected)
� An error in the first “P” frame can propagate to other “P” and “B” frames
![Page 5: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/5.jpg)
5.1 Error Resilience
� Due to spatial prediction schemes� there is dependency within a frame� For example: spatial prediction of Motion Vectors (MV)
� Median prediction – covered in the previous lecture� Pred = MEDIAN (A,B,C)
� An error in the MV of the neighbour can cause error in the MV ofthe current block.
![Page 6: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/6.jpg)
5.1 Error Resilience
� Require techniques to limit propagation of error in coded video streams.
� Errors can be due to, for example, lost packets when streaming video
� How to be resilient to packet losses or loss of data ?� Error resilience techniques
![Page 7: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/7.jpg)
5.1 Error Resilience
� At the Encoder: Temporal/Spatial Localization� Prevents error propagation by inserting regular intra coded
pictures, slices or macro-blocks
� Two type of error propagations: � temporal error propagation – errors are propagated into next frames � spatial error propagation – errors are propagated within a video
picture
� At the Decoder: Concealment� The impact of any errors can be concealed using correctly
decoded information from the current or previous pictures
![Page 8: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/8.jpg)
5.1 Error Resilience
� The impact of errors within decoded picture
Original picture Error damaged picture
![Page 9: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/9.jpg)
5.1 Error Resilience
� Temporal Localization� Cyclic intra-coded pictures
� Extra intra-coded I-pictures can be inserted. Error propagation can be reduced at the cost of extra overhead in the bit stream
� This scheme will reduce coding efficiency
II
![Page 10: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/10.jpg)
5.1 Error Resilience
� Temporal Localization� Cyclic intra-coded slices
![Page 11: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/11.jpg)
5.1 Error Resilience
� Temporal Localization� Cyclic intra-coded slices
� Extra intra-coded I-slices can be used to periodically refresh the frame from the top to the bottom over a number of frames
� The disadvantage is that the partial updating of the screen in a frame period will produce a noticeable “windscreen wiper”effect
![Page 12: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/12.jpg)
5.1 Error Resilience
� Temporal Localization� Cyclic intra-coded marcoblock (MB) refreshment
…….
A fixed number
of frames
…….
A fixed number
of frames
…….
A fixed number
of frames
![Page 13: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/13.jpg)
5.1 Error Resilience
� Temporal Localization� Cyclic intra-coded marcoblock (MB) refreshment
� Extra intra-coded MBs can be inserted to refresh the frame periodically over a fixed number of frames
� This scheme has been widely used in current MPEG-4 codec
� Example : choose N random blocks in a frame to be coded as INTRA
![Page 14: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/14.jpg)
5.1 Error Resilience
� Spatial Localization
� Slice Mode (used in MPEG-4 & H.263)
� The 11MBs or 22MBs per slice can reduce the damage to decoded picture after a corrupted MB is detected.
� Each slice coded independently of other slices in the picture
� So there is no dependency from one slice to another.
� For example: no MV prediction across slice boundaries
![Page 15: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/15.jpg)
5.1 Error Resilience
� Spatial Localization� Slices Mode (used in MPEG-4 or H.263)
![Page 16: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/16.jpg)
5.1 Error Resilience
� Spatial Localization� Slice Mode : useful for IP streaming
� Example: one slice per packet
� The loss of one packet is contained within one slice – only a small part of the frame is lost
![Page 17: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/17.jpg)
� Concealment Techniques� Use available information to conceal errors
� Spatial concealment – use nearby blocks
� Temporal concealment – use blocks in previous frame
� Performed at the decoder
� Need to conceal errors to limit perceptual quality
5.1 Error Resilience
![Page 18: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/18.jpg)
5.1 Error Resilience
� Spatial concealment:
� This technique is best suited to little spatial activity but is far less successful in areas where there is significant spatial details.
Simple interpolation with above/below macro-blocks
![Page 19: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/19.jpg)
5.1 Error Resilience
� Spatial concealment – at the decoder
Error damaged picture Spatial concealed picture
![Page 20: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/20.jpg)
5.1 Error Resilience
� Simple temporal concealment
� This technique is effective in a relatively stationary area but much less effective in the fast moving regions
Previous frame Current frame
![Page 21: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/21.jpg)
5.1 Error Resilience
� Temporal concealment – at the decoder
Error damaged picture Temporal concealed picture
![Page 22: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/22.jpg)
5.1 Error Resilience
� Motion compensated concealment� Combines both temporal replacement and motion
compensation (MC)
� High correlation among nearby MVs in a picture
� Assumes linear changes in MVs from below to above MBs.
� This scheme can significantly improve error concealment in moving areas of the picture
� Refer to example on next slide
![Page 23: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/23.jpg)
5.1 Error Resilience
� Motion compensated (MC) concealment� Calculate a MV for lost MB
� Use MV of MB’s above and below
� Assume linear variation
� Obtain concealment block by performing MC
Lost MBLost MB
![Page 24: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/24.jpg)
5.1 Error Resilience
� Concealment Techniques� Motion compensated concealment
Motion comp concealment Spatial concealed picture
![Page 25: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/25.jpg)
5.1 Error Resilience
� Layered Coding for error concealment� Base layer – important data/information
� High layer – less important
� Spatial Scalability, SNR Scalability and Temporal Scalability
� Scalability explored in next section
Layered
Coder
Low Priority
Data
High Priority
DataLow Loss Channel
High Loss Channel X X
Layered
Coder
Low Priority
Data
High Priority
DataLow Loss Channel
High Loss Channel X X
Layered
Coder
Low Priority
Data
High Priority
DataLow Loss Channel
High Loss Channel X X
![Page 26: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/26.jpg)
5.2 Scalability
� Scalable video coding means the ability to achieve more than one video resolution or quality simultaneously.
Scalable
Encoder
2-Layer
Scalable
Decoder
Single
Layer
Decoder
Enhanced Layer
Base Layer
Full (scale)decodedsequence
Base-linedecodedsequence
![Page 27: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/27.jpg)
5.2 Scalability
� Useful for � inter-working between different clients and networks.
� unequal error protection (protect the base layer more than the enhancement layer)
Scalable
Encoder
2-Layer
Scalable
Decoder
Single
Layer
Decoder
Enhanced Layer
Base Layer
Full (scale)decodedsequence
Base-linedecodedsequence
![Page 28: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/28.jpg)
5.2 Scalability
� Temporal Scalability
� Spatial Scalability
� Quality (SNR) Scalability
Scalable
Encoder
2-Layer
Scalable
Decoder
Single
Layer
Decoder
Enhanced Layer
Base Layer
Full (scale)decodedsequence
Base-linedecodedsequence
![Page 29: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/29.jpg)
5.2 Scalability
� Temporal Scalability� Base layer: low temporal resolution
� Enhancement layer: higher resolution
� Spatial Scalability
� Quality (SNR) Scalability
![Page 30: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/30.jpg)
5.2 Scalability
� Temporal Scalability� Naturally achieved with I,P and B frames
� Can drop the B frames to reduce frame rate
I B B P B B P B B P
I B B P B B P B B P
Enhancement
Layer
Base
Layer
![Page 31: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/31.jpg)
5.2 Scalability
� Spatial Scalability� A spatially scalable coder operates by filtering and
decimating a video sequence to a smaller size prior to coding.
� An up-sampled version of this coded base layer representation is then available as a predicator for the enhanced layer
![Page 32: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/32.jpg)
5.2 Scalability
� Spatial Scalability� A general architecture of a spatially scalable encoder
and decoder is shown on the next slide.
� The low resolution frame from the base layer is available to the high resolution encoder as a predictor option
� Trying to make use of information already contained in the base layer
![Page 33: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/33.jpg)
5.2 Scalability
� Spatial Scalability
![Page 34: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/34.jpg)
5.2 Scalability
� Spatial Scalability� Example:
� Full resolution HD TV� Base layer + enhancement layer = HDTV service
� Base layer – standard TV resolution� Base layer = Standard TV
� Spatial scalability as defined by MPEG-2 is shown on the next slide� Base layer frame is up-sampled and provided to the enhancement
layer coder
![Page 35: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/35.jpg)
5.2 Scalability
2 layer spatially scalable coder
16x16
8x8
16x16
![Page 36: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/36.jpg)
5.2 Scalability
� Sub-band coding� Slides from Lecture 3
![Page 37: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/37.jpg)
3.1 Subband Coding
� Input signal can be decomposed into frequency sub-bands.� Two-channel filter bank example
� Sub-bands representing low and high frequency components.
� Recursive application possible giving a tree structured transform� Refer to diagram on next slide
COMP9519 Multimedia Systems – Lecture 3 – Slide 37 – J Zhang
![Page 38: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/38.jpg)
3.1 Subband Coding
COMP9519 Multimedia Systems – Lecture 3 – Slide 38 – J Zhang
x[k] FL
FH
2
2
FL
FH
2
2
L LL
LH
FL
FH
2
2
H HL
HH
GL
GH
2
2
GL
GH
2
2
+ 2 GL +
+ 2 GH
x[k]
f
LL LH HL HH
![Page 39: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/39.jpg)
3.1 Subband Coding
� Sub-sampling after filtering to produce each sub-band.� Sub-sampling by a factor of 2 in the previous example
� The combined sampling rate (of all sub-bands) is the same as the input signal.
� By choosing appropriate analysis and synthesis filters it is possible to achieve perfect reconstruction.� Analysis Filters (FH, FL)
� Synthesis Filters (GH, GL)
COMP9519 Multimedia Systems – Lecture 3 – Slide 39 – J Zhang
![Page 40: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/40.jpg)
3.1 Subband Coding
� Consider recursive application of the sub-band transform for only the low pass channel
� Provides a logarithmically spaces pass bands� Unlike the uniform pass-bands seen in the earlier
example.
� Allows the signal information to be decoded incrementally; starting from the lowest frequency sub-band.
� Useful for scalable representations of the original signal� Refer to diagram on the next slide
COMP9519 Multimedia Systems – Lecture 3 – Slide 40 – J Zhang
![Page 41: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/41.jpg)
3.1 Subband Coding
COMP9519 Multimedia Systems – Lecture 3 – Slide 41 – J Zhang
x[k] FL
FH
2
2
FL
FH
2
2
L LL
LH
H
GL
GH
2
2
+ 2 GL +
2 GH
x[k]
f
LL LH H
![Page 42: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/42.jpg)
3.1 Subband Coding
� Discrete Wavelet Transforms (DWT) are tree structured subband transforms with non-uniform (logarithamic) spaced pass-bands� Decomposing the signal/image to base resolution
� And accompanying detailed components
� For image coding DWT applied in both horizontal and vertical directions (2D DWT)� Diagram in future slide provides example of filter
application.
COMP9519 Multimedia Systems – Lecture 3 – Slide 42 – J Zhang
![Page 43: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/43.jpg)
COMP9519 Multimedia Systems – Lecture 3 – Slide 43 – J Zhang
3.3.1 Analysis/Synthesis Stages� Analysis
� A signal is first filtered to create a set of signals, each of which contains
a limited range of frequencies. These signals are called subbands.
� Since each subband has a reduced bandwidth compared to the original
fullband signal, they may be downsampled. That is, a reduced number
of samples may be taken of the signal without causing aliasing
� Synthesis-- Reconstruction is achieved by:
� Upsampling the decoded subbands.
� Applying appropriate filters to reverse the subbanding process.
� Adding the reconstructed subbands together.
![Page 44: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/44.jpg)
COMP9519 Multimedia Systems – Lecture 3 – Slide 44 – J Zhang
3.1 Analysis/Synthesis Stage
� 2D dimensional decomposition structure
![Page 45: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/45.jpg)
COMP9519 Multimedia Systems – Lecture 3 – Slide 45 – J Zhang
3.1 Analysis/Synthesis Stage
Ref: H.Wu
![Page 46: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/46.jpg)
3.1 Analysis/Synthesis Stage
� Recursive application of the DWT� First level and second level decomposition shown
COMP9519 Multimedia Systems – Lecture 3 – Slide 46 – J Zhang
LL1 HL1
LH1 HH1
LL2 HL2
LH2 HH2
HL1
LH1 HH1
![Page 47: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/47.jpg)
COMP9519 Multimedia Systems – Lecture 3 – Slide 47 – J Zhang
3.1 Analysis/Synthesis Stage
� The formation of subbands does not create any compression in itself.
� The same total number of samples is required to represent the subbands as is required to represent the original signal.
� The subbands can be encoded efficiently:
� The significance of the different spatial frequencies are not uniform, and this fact may be exploited by different bit allocations to the various subbands
� The subbands are then encoded using one or more coders. Different bit rates or even different coding techniques may be used for each subband.
![Page 48: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/48.jpg)
3.1 Subband Coding
COMP9519 Multimedia Systems – Lecture 3 – Slide 48 – J Zhang
2D-DWT
Image
decomposition
Quantization
of
transform
coefficients
Coding
of
Quantized
coefficients
image Bit stream
Refer to recommended text for various sub-band coding techniques.
DWT employed in JPEG-2000 standard for image compression.
![Page 49: Lecture 5: Error Resilience](https://reader030.vdocuments.site/reader030/viewer/2022012509/618608a95df43124267fcc88/html5/thumbnails/49.jpg)
COMP9519 Multimedia Systems – Lecture 3 – Slide 49 – J Zhang
3.1 Subband Coding : Summary
� The fundamental concept behind Subband Coding is to split up the frequency band of a signal and then to code each subband using a coder that accurately matches the statistics of the bands.
� At the receiver, the sub-bands are re-sampled, fed through interpolation filters and added to reconstruct the image
� With an appropriate choice of filters, perfect reconstructions can be achieved