5g 寬頻應用 future video codec 視訊規格標準化進程
Post on 09-Feb-2022
6 Views
Preview:
TRANSCRIPT
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
5G寬頻應用Future Video Codec
視訊規格標準化進程
工業技術研究院資訊與通訊研究所視訊多媒體通訊技術組林俊隆
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 2
Biography
• Ph.D. degree in Computer Science from
National Tsinghua University, 2010
• Information and Communications
Research Laboratories, ITRI − 2010/11~
• Leader of MPEG Standard team– Over 100+ MPEG standard contributions
– Over 80+ pending or granted patents
• Leader of tech. team on emerging
VR/AR/MR technology
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 3
Outline
• MPEG Roadmap
• JVET activities– Overview of Call for Proposal(CfP)
• Versatile Video Coding (VVC)/H.266
– Results of CfP responses
– WD and TM status
• MPEG activities– Point Cloud Compression(PCC)
– Coded Representation of
Neural Networks (NNR)
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
� 行動多媒體影音傳輸與串流是行動通訊市場與寬頻網路市場的Killer
Application
� Mobile video (ex: video streaming,
video conferencing) 預計將占行動網路總流量的75%以上
� 4k8k UHD、3D Video、HDR/WCG
video and VR/AR等將大幅增加未來通訊頻寬的需求
� 高效能視訊編碼技術� 大幅降低多媒體影音資訊的資料傳輸量� 提升下世代行動通訊視訊應用的滲透率
� MPEG/ITU-T標準組織� 針對各種多媒體視訊應用需求制定編碼
及傳輸標準� MPEG-2, MPEG-4, MPEG-H
� H.261, H.263, H.264, H.265
Importance of Video Codec
Source :ERICSSON MOBILITY REPORT JUNE 2017
Source : Cisco Visual Networking Index: Global Mobile Data
Traffic Forecast Update, 2016–2021
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
� Video codec and 3GPP
� ETSI 3GPP (H.263)
� 3GPP release 6 (H.264/AVC)
� 3GPP release 12 (H.265/HEVC)
� 5G (??? H.266/H.265/AV2/EQ-AVC)
� Versatile Video Coding (VVC)/H.266
� 為配合5G通訊標準制定，MPEG/ITU-T 2018年開始下一代 Video codec標準制定(暫名VVC/H.266)，並預計在2020年完成H.266 v1的標準制定
3GPP SA4 and Video Codec
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
MPEG/ITU-T Standard Activities
6
MPEG
Moving Picture Experts Group
VCEG
Video Coding Experts Group
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 7
MPEG Roadmap
7
2018 20202017 2019 2021 2022 Jan 2023
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
2018年開始制定年開始制定年開始制定年開始制定 H.266 第一第一第一第一版標準版標準版標準版標準，，，，預計預計預計預計2020完成完成完成完成
MPEG視訊標準的演進
AVC/H.264 HEVC/H.265VVC/H.266
Immersive Media
HDR
VR360
Light Field
(Sparse)
Point Cloud
Full-HD Mobile
UHD Broadcasting
Blu-ray
HDTV
Internet Video
第一版標準第一版標準第一版標準第一版標準於於於於2013完成完成完成完成第一版標準於第一版標準於第一版標準於第一版標準於2003完成完成完成完成
2003 2013 2020
FVC(Future Video Coding)
HDR(High Dynamic Range)
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 9
JVET Call for Proposals (CfP)
• San Diego, USA
• Date: 10 ~ 20 April, 2018
• Approximately 350+ participants
• 23 CfP proposals
• Approx. 80 input documents– Including 23 CfP proposals
• New project launched– Versatile Video Coding(VVC)
– Versatile Test Model(VTM)
9
*WD : Working Draft
*TM : Test Model
*CD : Committee Draft
*FDIS: Final Draft International Standard
Timeline
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 10
JVET Call for Proposal• Test categories:
– Standard Dynamic Range(SDR)
– High Dynamic Range(HDR)
– 360° Video
• 46 category-specific submissions to be tested
(not counting the anchors)– SDR:22 submissions (8 of which are registered only in this
category)
– HDR: 12 submissions (4 of which are registered only in this
category)
– 360:12 submissions (4 of which are registered only in this
category)
10
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 11
CfP Performance
• Measured by objective performance,
– >40% bit rate reduction compared to HEVC
– >10% compared to JEM (for SDR case)
– More elements show better performance
– Some proposals show similar performance as JEM with
significant run time reduction
– Similar ranges for HDR and 360°
• Results of subjective tests generally show similar (or
even better) tendency
– Benefit over HEVC very clear
– Benefit over JEM visible at various points
11
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 12
Performance of SDR
12
Y U V Enc Time Dec Time Y U V Enc Time Dec Time
Peking Univ. CN N
DJI CN N
Ericsson SE Y
Nokia FI Y
ETRI KR Y
Sejong Univ. KR Y
-7.55% -6.94% -5.96% 126% 102% -38.06% -46.88% -46.53% 1046% 780%
6.88% 6.10% 6.65% 126% 101% -28.64% -40.98% -41.19% 1047% 775%
InterDigital US N
Dolby US N
J0016 KDDI JP N JEM 7.0 -0.57% -0.52% -1.30% 108% 1886% -33.50% -43.57% -44.18% 858% 18614% Y
J0017 LG KR Y JEM -2.52% -5.29% -6.19% 191% 84% -34.75% -45.89% -46.73% 1523% 644%
-16.06% -6.75% -10.43% 152% 227% -43.81% -45.61% -47.41% 1190% 1302%
-14.40% -5.13% -8.82% 77% 232% -42.38% -44.64% -46.37% 606% 1330%
-2.28% -3.44% -3.88% 107% 56% -34.63% -45.06% -45.52% 817% 384%
-0.06% 0.91% 0.52% 60% 55% -33.18% -42.83% -43.15% 456% 372%
Qualcomm US Y -15.53% -3.66% -5.97% 148% 84% -43.08% -44.38% -46.05% 1180% 639%
Technicolor FR Y -10.26% 0.05% -1.65% 46% 85% -39.72% -42.80% -43.94% 370% 646%
Qualcomm US Y
Technicolor FR Y
J0023 RWTH Aachen Univ. DE Y JEM 7.0 -0.79% -1.52% -1.52% 440% 122% -33.68% -44.16% -44.37% 3507% 927%
Samsung KR Y
Huawei CN Y
GoPro US N
HiSilicon CN Y
Huawei CN Y
GoPro US N
HiSilicon CN Y
Samsung KR Y
Sharp JP Y
Foxconn TW N
NHK JP N -2.14% -5.55% -5.61% 237% 214% -34.57% -45.96% -46.32% 1890% 1630%
SHARP JP Y -3.26% -6.48% -6.57% 273% 257% -35.28% -46.42% -46.86% 2175% 1955%
J0028 Sony JP Y JEM -8.15% -8.66% -8.80% 644% 223% -38.41% -47.54% -48.07% 5133% 1830%
J0029 Tencent CN N NextSoftware -4.70% -8.34% -8.91% 242% 125% -36.17% -47.49% -48.15% 1928% 954%
J0031 Bristol Univ. UK N JEM 7.0 -4.54% 20.19% 18.68% 90% 262% -36.09% -19.30% -21.72% 767% 1678% Y
USTC CN Y
Peking Univ. CN N
HIT CN N
Wuhan Univ. CN N
Y
CNN
Y
CS1(Over HM16.16)Organizations Country code baseResponse HEVC
-1.57% -0.71%
Doc #CS1(Over JEM7.0)
-1.72% 100% 381%JEM7.0
J0012 JEM 7.0
-34.19% -43.75% -44.37% 765% 2911%
-33.73% -43.92% -44.10% 777% 777%
J0011
J0013 JEM 7.0
-0.90% -1.14% -1.15% 103% 98%
0.64% -0.39% -0.89% 105% 53% -32.74% -43.48% -43.89% 841% 404%
J0015 JEM
NextSoftwareJ0014 Fraunhofer HHI DE Y
-3.98% -3.28% -3.16% 205% 33% -35.72% -44.75% -44.95% 1710% 263%
J0021
J0018 Media Tek TW Y JEM
J0020
-13.60%
JEM7.0
JEM
Panasonic JP Y
-41.86% -44.77% -46.13% 728% 582%
-37.00% -35.96% -37.39% 902%
J0022
NEWJ0024
-3.80% -5.63% 90% 82%JEM
-6.01%
-4.24%
10.34% 8.53%
10.71% 9.23%
120%
68%
36%
39% -35.68% -36.17% -37.33% 513% 296%
274%
-38.76% -38.74% -40.30% 1058% 281%
-37.20%
J0025
-6.31%
NEW
-8.78% 5.89% 3.83% 141% 45%
-36.02% -37.55% 1043% 283%
-8.15% -8.66% -8.80% 644% 223%
10.11% 7.59% 139% 45%
J0027
JEM
J0026
JEM
-47.54% -48.07% 5133% 1830%JEM
J0032 281824%-10.11% -9.59% -9.97% 527% -39.63% -48.10% -48.69% 2184%79868%
-38.41%
Y
Y
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 13
Performance of SDR
• JVET-J0080: Report of subjective evaluation
contains 28 plots as below, one per sequence
HM
JEM
Rate 1...4
Proposals ranked by MOS (per rate)
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 14
WD1 / VTM1• SW code base: Next Software (HHI)
• Block structure– QTBTTT
– Unified tree (coding block unites prediction and transform)
– CTU size: 128x128, Maximum transform size 64x64
– Smallest luma block size 4x4
• Some removed elements of HEVC: – Mode dependent transform (DST-VII), mode dependent scan
– Strong intra smoothing
– Sign data hiding in transform coding
– High-level syntax (e.g. VPS)
– Tiles and wavefront
– Quantization weighting
14
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 15
Current Performance of VVC
• PSNR-based Common Test Conditions BD-Rate savings
relative to HEVC reference software (10 bit)
15
vs HM AI RA
gain Enc. Dec. gain Enc. Dec.
VTM 1.0 4% 9.6X 1.1X 8% 2.2X 0.8X
BMS 1.0 15% 98X 2.2X 23% 9.3X 2.3X
VTM 2.0 18% 18X 1.6X 23% 3.7X 1.3X
AI RA
gain Enc. Dec. gain Enc. Dec.
VTM 2.0 vs.
VTM 1.015% 1.9X 1.5X 16% 1.7X 1.5X
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 16
Point Cloud Compression
(PCC)
16
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
Point Cloud
Static Objects and Scenes
(Category 1)
Dynamic Objects
(Category 2)
Dynamic Acquisition
(Category 3)
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
Point Cloud
• A set of 3D points
– Not ordered,
– Without relations
between them
• Each point is
defined by
– (X, Y, Z)
– Attribute
• (R, G, B) or (Y, U, V)
• Reflectance,
transparency
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 19
PCC Timeline
19
PCC
Extension
2017
CfP
2018
Review CfP results
Develop PCC video standard
01 04 07 10 01 04 07 10
2019 2020
01 04 07 10 01 04 07 10
FDIS
CD
TM established
2014
WD establishedIssue CfP
Initial PCC
…
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 20
MPEG 123 meeting of PCC
• 4ndth F2F Meeting of PCC after CfP
• Date: 15 ~ 20 July, 2018
• Approximately 60+ participants
• 132 technical contributions
33%
67%
Cat.13 Cat2.
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 21
PCC Participants
21
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 22
Num. of contributions in PCC
22
40
15
0
15
57
67
3640
36
120 121 123
Percentage of each category
TMC1 TMC2 TMC3&13
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 23
Coded Representation of
Neural Networks (NNR)
23
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 24
Coded Representation of
Neural Networks (NNR)
• 3rd AHG meeting
• April 15 2018
• 20+ participants– ETRI, Fujitsu, Hanyang Univ., Huawei, Mitsubishi,
NEC, Nokia, Peking Univ.
• 9 contributions
24
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
Overview of the evaluation process
25
• Image classification
• Feature extraction for compact video descriptors (CDVA)
• NN based components for video compression
• Classification of health care records
• (Re)training for machine reading comprehension (MRC)
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.
Coded Representation of Neural
Networks• Call for test data
– Visual analysis, image coding, text
understanding
– Test data, training data, network, compressed
network
– Audio data
• Call for Evidence
– Submission 10/2018
All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 27
Thank You
27
top related