-
Video Compression with Film Grain
Aesthetics Preservation
Mickael RAULETCTO @ ATEME
-
2
By considering dedicated features
• Grain• Legacy• Artistic intent
• Like it or not, people want grain• But grain is uncompressible• AV1 has a very unique feature:
• Film Grain Synthesis (FGS)
• Not exactly new, but, • Normative and MANDATORY for the first time
• Highly regarded by Netflix (proponent)• Gain is huge• FGS was previously available in AVC and
HEVC through SEIs (partially supported by decoders)
-
3
AV1: Royalty-Free Codec (October 2018)
• AV1 (AOM Video codec 1)• Alliance for Open Media (AOM)
• Interoperable and open
• Optimized for OTT delivery
• High Video Quality and Real-time delivery modes
• Compression gains up to 20% w.r.t state-of-the-art
• From low to high resolutions (including UHD, HDR, WCG)
• Classical structure + • new coding tools
• additional features (Film grain synthesis, scalability, SCC)
-
4
Coding Structure: HEVC vs AV1HEVC AV1
Multiples sizes/forms blocks: 128x128 to 4x4 (mixing intra and inter)Rectangular separable TUs: 64x64 to 4x4 (4 kernel types)
Recursive tree coding structure
56 Intra prediction directions + 11 non-directional (DC + Paeth + 3 Smooth + 5 Filter + Chroma from Luma)
Coding block128x128 to 4x4
Multiples sizes/forms PUs: 64x64 to 4x4 (intra or inter for all PUs)Square non separable TUs: 32x32 to 4x4 (2 kernel types)
Quadtree coding structure
33 Intra prediction directions + 2 non-directional (DC + PLANAR)
17 1
6 1
5 1
4 1
3 1
2 1
1 1
0 9
8 7
6 5
4 3
2
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
0 : Intra_Planar
1 : Intra_DC
35: Intra_FromLuma
Coding Unit64x64 to 8x8
Efficient spatio-temporal mv prediction (Merge, AMVP) Efficient spatio-temporal mv prediction (Refmv, Newmv)
-
5
Film Grain Synthesis process
• Straightforward idea
• Challenging implementation
What we have
What we encode
What we synthesize
-
6
Denoising and modeling
• 2 steps denoising• Rough + fine denoising
• Or pre-analysis + fine denoising
• Grain parameters computation
L = 2
L = 2
-
7
For all the flat blocks indexed from 1 to m, we look for 𝜑 minimizing the quantity:
control on the AR coefficients + unicity of the solution guaranteed
andwith
𝜑 is solution of the linear system:
.
Computation of the AR coefficients 𝜑1,…, 𝜑𝑝
Ridge regression
Advantages : - Taking λ hi h enough, the scheme is guaranteed to be stationary- Taking λ hi h enough, the coefficients are < 2
-
8
About Unpleasing Patterns
• It can happen by design
• Detecting and avoiding them
Pattern selectedPattern not selected
-
9
Compression Gains
Grain Encoding (no film grain synthesis) Film Grain Synthesis
Tag Bitrate(Mbps)
Comments Bitrate(Mbps)
Gain(in %)
Chimera-DinnerSceneUHD 10bits
112 33.90 Grain preserved 5.23 84,57
118 10.25 Grain washed out 3.33 67,49
ZombieLongPS_cut2UHD 10bits
112 29.16 Grain preserved 15.02 48,48
124 10.71 Ugly unstable grain patterns 7.78 27,33
ebu_farytale1HD 8bits
85 19.26 Grain preserved 9.52 50,61
90 9.94 Grain already washed out, even though bitrate is quite high
5.99
39,79
-
10
Parts of the Reconstructed Frames
Coded at 10 709 kbps
Coded at 9 944 kbps
Coded at 7 782 kbps
Coded at 5 987 kbps
Grain Management
ZombieLongPS_cut2UHD 10bits
ebu_fairytale1HD 8bits
-
Source #1 version #2 version
-
Source #1 version #2 version
-
13
Considering dedicated features: FGS
• Still ongoing effort• Patents
• Research papers
• Practical tests / interoperability• TV LG
• STB broadcom
• NAB DEMO AV1 8k HDR + grain, with compatible TV set !
• Differentiating feature for VOD operators
-
14
MERCI, Danke, gracias , obrigado, tack, Dank, grazie, Thank you,dankie, faleminderit, Barak Allahu fiik, chnorakaloutioun, çox sag olun, a ni kié,bedankt, waita, eskerrik, dhanyabaad, dziakuju, thint ko, kyay tzu tin pa te, a nikié, trugéré, blagodaria, gràcies, salamat, kam sah hamnida, grazie, mèsi, tak,dankon, akpé, kiitos, multumesc, gracie, a dank, tesekkur ederim, tapadh leat,go raibh maith agat, diolch, efharisto, aguyjé, meherbani, mahalo, toda, tau,dhanyavad, köszönöm, terima kasih, takk, arigatô, tanemirt, hvala,dhanyavadagalu, akun, murakoze, sobodi, tenki, spas, mercé, khob chai, gratiasago, paldies, choukrane, aciu, blagodaram, terima kasih, misaotra, nizzik hajr,aabhari aahe, bayarlalaa, dank u wel, mercé, shukriya, danki, motashakkeram,mamnun, dzikuj, dakujem, multumesc, spacibo, marci, faafetai lava, gratzias,ahsante, salamat po, mauruuru, nanedri, rahmat, dkuji, kop khun krap,yekeniele, sagolun, diakuiu, cám Ön, grces, djiere dieuf, enkosi, ngiyabonga