video transcoding in h.264 prof. maurizio bonuccelli francesca martelli francesca lonetti pisatel
TRANSCRIPT
![Page 1: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/1.jpg)
Video Transcoding in H.264
Prof. Maurizio Bonuccelli
Francesca Martelli
Francesca Lonetti
PISATEL
![Page 2: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/2.jpg)
2
PISATEL
Outline
H.264 coding and transcoding
Motion Vector Composition algorithms
Overview of our temporal transcoding results (MPEG4, H.263)
PISATEL
![Page 3: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/3.jpg)
3
PISATEL
H.264 coding
H.264 mean features:
Variable and small block sizes (4×4)
PISATEL
Average bit rate reduction!
High complexity!
Quarter-pixel resolutionNAL unitsNew entropic coding algorithms (CABAC)Deblocking filterMoto compensation with multiple reference pictures
![Page 4: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/4.jpg)
4
PISATEL
H.264 temporal transcoding
PISATEL
TranscoderInput bit-rate 64 Kb Output bit-rate 32 Kb
123 13123
Skipping frames to reduce the output bit-rate
Three main issues:– Motion Vector Computation(MVC)– Prediction Error Computation– Frame Skipping Policy
![Page 5: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/5.jpg)
5
PISATEL
Motion Vector CompositionThe motion vectors are computed by
– Motion Vector Composition Algorithms (BI, TVC,
FDVS, ADVS)
– Restricted Motion Estimation (RME)
MVC
skipped
F(n-1) F(n)F(n-2)
mvn-1mvn
![Page 6: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/6.jpg)
6
PISATEL
Telescopic Vector Composition
F(n-1)
(skipped)
MBn
MB4n-1
MB1n-1
MB2n-1
MVn
MB3n-1
MV1n-1
MV2n-1
MV4n-1
MV3n-1
F(n)
MVn=MVn+MVTVC
New reference area
F(n-2)
MBnMVTVC=MV1
n-1
![Page 7: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/7.jpg)
7
PISATEL
Bilinear InterpolationMVBI=(1-α)(1-β)MV1
n-1+(α)(1-β)MV2n-1+(1-α)(β)MV3
n-1+(α)(β)MV4n-1
F(n-2)
F(n-1)
(skipped)
MBn
MB4n-1
MB1n-1
MB2n-1
MVn
MB3n-1
MBn
MV1n-1
MV2n-1
MV4n-1
MV3n-1
F(n)
MVn=MVn+MVBI
New reference area
MVBI
αβ
![Page 8: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/8.jpg)
8
PISATEL
H.264 features New in H.264: variable macroblock partition (16 motion vectors for each macroblock)
Transcoder keeps the same partitions of the remote encoder (most efficient solution)
How to apply MVC in H.264?
– BI and TVC adaptation
– New MVC algorithm
?
H.264
?
Previous standards
![Page 9: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/9.jpg)
9
PISATEL
MVC in H.264MB1 MB2
MB3 MB4
LIV.0
Mv_MVC=
f(mv_MB1,mv_B2, mv_B3, mv_B4)
MB1
B1 B2
MB3
B1 B2
B3 B4
MB4
B1
B2
LIV.1
MB2
B1
B3 B4
B2
Mv_B4
LIV.3
Mv_B2 Mv_B2 Mv_B2Mv_B1 Mv_B1=
f(sb1,sb2)Mv_B2
Mv_MB4=
f(mv_B1,mv_B2)
Mv_MB3=
f(mv_B2,mv_B4)Mv_MB2=
f(mv_B1,mv_B2)
Mv_MB1=
mv_B2
B1
sb1
B2B2
B2 B4 B2
LIV.2
B1
sb2
![Page 10: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/10.jpg)
10
PISATEL
A new MVC algorithmBasic Idea: looking for a reference area, the most similar to the current macroblock
Two steps:– For each motion vector of a macroblock (or block) of
the reference area in the skipped frame, compute the difference between the macroblock and an area pointed by the motion vector with the same size of the current macroblock
– Choose the motion vector of the reference area that minimizes this difference
![Page 11: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/11.jpg)
11
PISATEL
New MVC algorithm: example
F(n)
A
MB
N
M
Vf=argmin i Є S MSE(A, Ai) = argmin i Є S (1/NxM |A-Ai|2)
F(n-2)
mv
F(n-1)
MB4MB3
MB1 MB2b1 b2
b3 b4
b2b2
b1
b1
(skipped)
mv2
A2
A4
mv4
A3
mv3
A6
mv6
mv8
A8
mv7
A7
mv5
A5
mv1
A1
new_mv
![Page 12: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/12.jpg)
12
PISATEL
MVC performance (time)
PISATEL
TVC BI New ME
![Page 13: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/13.jpg)
13
PISATEL
MVC performance (quality)P
SN
R(d
B)
Frames
Akiyo
Motion Estimation
New MVC Algorithm
![Page 14: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/14.jpg)
14
PISATEL
MVC performance (quality)Akiyo
PS
NR
(dB
)
New MVC Algorithm
BI Algorithm TVC Algorithm
Frames
![Page 15: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/15.jpg)
15
PISATEL
MVC performance (quality)
Motion Estimation
New MVC Algorithm
PS
NR
(dB
)
Frames
Coastguard
![Page 16: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/16.jpg)
16
PISATEL
MVC performance (quality)P
SN
R(d
B)
Frames
Coastguard
New MVC Algorithm
BI Algorithm TVC Algorithm
![Page 17: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/17.jpg)
17
PISATEL
Our activities in video transcodingWe started in studying video features in MPEG4
With this codec, we developed two temporal transcoders:– One based on an architecture known as FSC (Frame
Skipping Control)– One based on an architecture known as DFS
(Dynamic Frame Skipping)
By simulation, we realized that the second architecture results better in terms of video quality, while the first one results better in terms of processing times
![Page 18: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/18.jpg)
18
PISATEL
MPEG4 transcoder architecturesAfter a skipped frame:– In the FSC architecture, motion vectors and
prediction errors are computed without executing motion estimation worse quality, re-encoding errors amassment
– In the DFS architecture, motion vectors and prediction errors are computed with a restricted motion estimation better quality, but worse time performance
![Page 19: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/19.jpg)
19
PISATEL
Motion Vector CompositionIn literature we found 4 MVC algorithms:– Bilinear interpolation– Telescopic Vector Composition– Forward Dominant Vector Selection– Activity Dominant Vector Selection
By using the DFS architecture (that performs motion estimation) we realized by simulation that these algorithms are equivalent
![Page 20: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/20.jpg)
20
PISATEL
Architectures and MVC algorithms
M. A. Bonuccelli, F. Lonetti, F. Martelli. Video Transcoding Architectures for Multimedia Real Time Services, ERCIM News No. 62, pp. 39-40, July 2005.
![Page 21: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/21.jpg)
21
PISATEL
Skipping policiesWe developed 4 skipping policies, all of them based on the transcoder buffer occupancy:– Buffer based– Motion activity– Consecutive skipping– Random
M. A. Bonuccelli, F. Lonetti, F. Martelli. Temporal Transcoding for Mobile Video Communication. In Proceedings of 2nd Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services (Mobiquitous 2005), pp.502-506, July 17-21, 2005, San Diego, CA.
![Page 22: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/22.jpg)
22
PISATEL
H.263 Video TranscoderThen, we studied the H.263 video codec, and we developed a temporal transcoder, based on DFS architecture.By observing the test results, we realize that video quality is influenced by the rate control of the front encoderThen, we tested and implemented several rate control algorithms for the front encoder:– TMN5– TMN8 domain– Perceptual rate control– Multiple zone (Activity)
![Page 23: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/23.jpg)
23
PISATEL
H.263 Video TranscoderIn our transcoder architecture, every frame is first transcoded, and then transmitted or dropped.To avoid the computation of frames that will be discarded, we developed a new frame skipping policy that predicts the frames to be transcodedM. A. Bonuccelli, F. Lonetti, F. Martelli. A Fast Skipping Policy for H.263 Video Transcoder. In Proceedings of 12th International Workshop on Systems, Signals and Image Processing (IWSSIP'05). September 22-24, 2005, Chalkida, Greece.
![Page 24: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/24.jpg)
24
PISATEL
Finally, H.264This codec is the product of the union between VCEG (H.263) and MPEG, forming the Joint VideoTeam (JVT)
It includes all benefits of previous standards in order to achieve good quality performance even at low bit-rate
We worked in two directions:– Optimizing the codec, to speed up it– Implementing a transcoder
![Page 25: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/25.jpg)
25
PISATEL
Optimizing H.264 encoderWe operated some modifications to the reference software in order to obtain acceptable encoding times:– instead of computing all half and quarter pixels in two
rounds, we compute them in only one round– fast way for choosing the optimal partitioning: instead of
using the SAD (Sum of Absolute Differences) measure as decision parameter, we use other metrics:
• the number of differences in terms of pixels • the maximum difference value• the average difference value• the most popular difference value
compared with proper self-adjusting thresholds.
![Page 26: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/26.jpg)
26
PISATEL
H.264 transcoderWe first implemented the cascade pixel domain transcoder by simply concatenating a decoder with an encoder, in order to develop the motion vector composition algorithms
Then we are implementing the frame skipping to obtain a transcoder that decides the frames to be dropped
![Page 27: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/27.jpg)
27
PISATEL
H.264 rate controlFinally, we are implementing the TMN8 rate control algorithm to be used in the front encoder
We think that, with a rate control algorithm able to skip frame in encoding phase, the transcoding process may be improved both in terms of quality and computation time
![Page 28: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/28.jpg)
28
PISATEL
Master ThesesLuigi D’Amaro. Algoritmi per la transcodifica video.Gianni Rosa. Transcodifica video per comunicazione mobile: studio di rate control.Luca Leonardi. Transcodifica video temporale: politiche di frame skipping.Marina Paletta. Realizzazione di un transcodificatore video temporale H.264 per video comunicazione mobile.Riccardo Vagli. Implementazione di un transcoder video basato sullo standard H.264/AVC.Alsona Dema. Rate Control in H.264.
![Page 29: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/29.jpg)
29
PISATEL
ConclusionsWe studied the video transcoding problem in real-time communications
We developed temporal transcoders with MPEG4, H.263 and H.264 codecs
We developed some skipping policies to be used in each transcoder
We developed three MVC algorithms to be used in the H.264 transcoder
![Page 30: Video Transcoding in H.264 Prof. Maurizio Bonuccelli Francesca Martelli Francesca Lonetti PISATEL](https://reader030.vdocuments.site/reader030/viewer/2022032705/56649db65503460f94aa79ba/html5/thumbnails/30.jpg)
30
PISATEL
Acknowledgements
We thank all ERI people who introduced us in this research area, for the helpful discussions and advices
We thank PisaTel Lab people
We thank all students who worked with us in this project