1

A Parallelism Encoding Framework for The Temporal Scalability of H.264/AVC Scalable Extension

Shu-Sian Yang, Sung-Wen Wang, Hong-Ming Chen, and Ja-Ling Wu

Department of Computer Science and Information Engineering

Graduate Institute of Networking and MultimediaNational Taiwan University, Taipei, Taiwan

E-mail:{pigyoung, song, blacksmith, wjl}@cmlab.csie.ntu.edu.tw

2

Outline

1. Problem Statement2. Overview of H.264/AVC Scalable Extension

I. Temporal ScalabilityII. Spatial ScalabilityIII. Complexity Reduction

3. Previous Parallel Encoding Scheme for Video Coding

1. MB-Level (Wave-front) Parallelism2. Frame-Level Parallelism

4. Parallel Encoding Based on Hierarchical B-Picture Structure

I. Frame-Level Parallel Scheme5. Conclusions and Future Work

3

SVC Encoder Structure Overview Combined scalability. H.264 based, layered video coding.

4

Higher Complexity

Base Layer (BL) is identical to the standard H.264 Enhancement Layers (EL) have “inter-layer”

predictions in additional:• H.264:

– Inter 16x16– Inter 8x16– Inter 16x8– Inter 8x8

• Inter 8x8• Inter 4x8• Inter 8x4• Inter 4x4

– Intra 16x16 (4 modes)– Intra 4x4 (9 modes)

• SVC additional:– BL Inter 16x16– BL Inter 8x16– BL Inter 16x8– BL Inter 8x8

• BL Inter 8x8• BL Inter 4x8• BL Inter 8x4• BL Inter 4x4

– BL Intra 16x16– BL Intra 4x4

– BL Inter 16x16 w. residue pred.

– BL Inter 8x16 w. residue pred.

– BL Inter 16x8 w. residue pred.

– BL Inter 8x8 w. residue pred..

• BL Inter 8x8 w. residue pred.

• BL Inter 4x8 w. residue pred.

• BL Inter 8x4 w. residue pred.

• BL Inter 4x4 w. residue pred.

– BL Intra 16x16 w. residue pred.

– BL Intra 4x4 w. residue pred.

5

Scalabilities

Three kinds of scalabilities: Quality (SNR) scalability▪ Fine-Grain-Scalability (FGS)▪ Bit-plane coding

Spatial scalability▪ Decimation▪ Wavelet transform

Temporal scalability▪ Hierarchical B-picture

30 fps15 fps7.5 fps

4CIF

CIFQCIF

6

Temporal Scalability

Hierarchical B-picture H.264 allows B pictures may or may not be used

as references. Hierarchical prediction. Temporal scalability can be achieved by

hierarchical truncating B pictures.

8

4

2

1 3

6

5 7

10

9 11

14

13 15

12

16Key Picture

Group of Pictures (GOP size = 16)

16Key Picture

Level 1

Level 2

Level 3

Level 4

7

• Higher temporal level, larger distance between current and reference frames.

• Frames at higher temporal level are the references frames of subsequent lower temporal level frames.

Temporal Scalability- Motion Characteristics of Different Temporal Levels

Level 1

Level 2

Level 3

Level 4

8

4

2

1 3

6

5 7

10

911

14

13

15

12

4 pictures away

8 pictures away

16

8 pictures away

8

• Statistical distribution of optimal MVs• Obtained from full search.• Total 7 test sequences.

• MVs are scattered sparsely at higher temporal levels.

Temporal Scalability- Motion Characteristics of Different Temporal Levels

(%)

Level 1 Level 2 Level 3 Level 4

Origin 18.74 19.67 21.18 26.05

Within 9x9 44.12 46.20 66.08 82.96

Within 15x15 56.35 66.04 84.38 91.34

Level 1Level 2Level 3Level 4

8

1

9

Level 3Series1 Series2 Series3 Series4Series5 Series6 Series7 Series8Series9 Series10 Series11 Series12Series13 Series14 Series15 Series16Series17 Series18 Series19 Series20Series21 Series22 Series23 Series24Series25 Series26 Series27 Series28Series29 Series30 Series31 Series32Series33

MV distribution

Perc

enta

ge

Level 4Series1 Series2 Series3 Series4Series5 Series6 Series7 Series8Series9 Series10 Series11 Series12Series13 Series14 Series15 Series16Series17 Series18 Series19 Series20Series21 Series22 Series23 Series24Series25 Series26 Series27 Series28Series29 Series30 Series31 Series32Series33

MV distribution

Perc

enta

ge

Level 2Series1 Series2 Series3 Series4Series5 Series6 Series7 Series8Series9 Series10 Series11 Series12Series13 Series14 Series15 Series16Series17 Series18 Series19 Series20Series21 Series22 Series23 Series24Series25 Series26 Series27 Series28Series29 Series30 Series31 Series32Series33

MV distribution

Perc

enta

ge

Temporal Scalability- Motion Characteristics of Different Temporal Levels

Level 1Series1 Series2 Series3 Series4Series5 Series6 Series7 Series8Series9 Series10 Series11 Series12Series13 Series14 Series15 Series16Series17 Series18 Series19 Series20Series21 Series22 Series23 Series24Series25 Series26 Series27 Series28Series29 Series30 Series31 Series32Series33

MV distribution

Perc

enta

ge

1616

-16

0-16 016

16

-16

0-16 0

1616

-16

0-16 0 16

16

-16

0-16 0

10

Parallel Encoding Based on Hierarchical B-Picture Structure

1. Data-Level Parallelism• GOP, Slice, Picture, Macroblock

• GOP: Extensive memory usage limits its scalability.• Picture: Difficult to identify independent pictures.• Slice: Coding efficiency degrades due to slice

boundaries.• MB: Extensive requirement of synchronizations.

• Applicable to all encoders

2. Function-Level Parallelism• Asymmetric workload• Depends on encoder implementations

11

Parallel Encoding Based on Hierarchical B-Picture Structure

• MB-Level (Wave-front) Parallelism:• Only MB-Level parallelism can be achieved in traditional

codecs.• Extensive controls and synchronizations required.

• Frame-Level Parallelism:• Using IBBPBBP pattern, set B pictures as non-reference

pictures.

12

• Proposed Picture Decomposition Based on Hierarchical B-Picture:• Utilizing the hierarchical B-Picture structure, picture-level

parallelism is allowed in SVC

Parallel Encoding Based on Hierarchical B-Picture Structure

Level 4

Level 3

Level 2

Level 1

13

• Experimental results: execution time of motion estimation

Bus Stefan Conoa Garden Dancer Toys and Cal. Vintage Car Average

Sequen-tial

100 100 100 100 100 100 100 100

MB-Level 64.915215072876

66.8130943672275

64.0276532137518

64.9564379336929

65.785350781534

62.0400888512799

62.5428296438884

64.4400956948931

Proposed 58.1437125748503

59.1463414634145

59.0262582056893

57.7388149939541

60.0809170600134

59.1362697866482

56.4955270322832

58.5382630166932

5152535455565758595

ME Module Performance

Exec

ution

Tim

e (%

)

Parallel Encoding Based on Hierarchical B-Picture Structure

14

Parallel Encoding Based on Hierarchical B-Picture Structure

Experimental results: coding efficiency comparison

15

Future Work

• For parallel video encoding, modules like motion compensation and up-sampling are good candidates for data level parallel processing. Along with data level parallelism, the function level one can also be integrated into a hybrid scheme.

• Platform dependent issues such as power consumption and load balancing on asymmetric architectures are also important research issues

16

Thank for your attendance!

Any Comment is Welcome