mpeg for the past, present and future of television

50
MPEG for the past, present and future of television Leonardo Chiariglione Keynote Speech at ATSC Annual Meeting Washington, DC – 2012/05/08

Upload: malo

Post on 09-Feb-2016

41 views

Category:

Documents


2 download

DESCRIPTION

MPEG for the past, present and future of television. Leonardo Chiariglione Keynote Speech at ATSC Annual Meeting Washington, DC – 2012/05/08. MPEG for the past of television. System Transport Stream DSM-CC Timed metadata Video MPEG-2 High Profile Audio MPEG-1 Audio Layer II - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: MPEG for the  past, present and future  of television

MPEG for the past, present and

future of television

Leonardo ChiariglioneKeynote Speech at ATSC Annual

MeetingWashington, DC – 2012/05/08

Page 2: MPEG for the  past, present and future  of television

MPEG for the past of television

SystemTransport StreamDSM-CCTimed metadata

VideoMPEG-2 High Profile

AudioMPEG-1 Audio Layer IIMPEG-2 AAC

2012/05/02 2MPEG for the past, present and future of TV

Page 3: MPEG for the  past, present and future  of television

An assessmentMPEG has provided the means for the television

business on air, cable and satellite to migrate from the analogue to the digital age

It has been a collective effortsRepresentatives from the entire spectrum of the

broadcasting world provided their requirementsPackages of patent licences were created (outside of

MPEG)Today there is virtually no broadcasting system that

is not based, at least partially, on MPEG standards

MPEG for the past, present and future of TV 32012/05/08

Page 4: MPEG for the  past, present and future  of television

MPEG for the present of television

SystemsCarriage of MPEG-4 on MPEG-2 TSDynamic Adaptive Streaming over HTTP (DASH)

VideoMPEG-4 Advanced Video Coding (AVC)3D extensions

AudioVarious forms of Advanced Audio Coding (AAC)Universal Speech and Audio (USAC)

2012/05/08 4MPEG for the past, present and future of TV

Page 5: MPEG for the  past, present and future  of television

An assessmentThe deployment of 2nd generation digital

broadcasting systems continues successfullyUse of the AVC on MPEG-2 TS allows packing twice

as many programs as MPEG-2 Video in an analogue channel

With IPTV the notion of broadcasting is expandingMPEG is helping the broadcasting industry

To enrich the broadcasted user experience Through standard technologies that serve multiple

fields

MPEG for the past, present and future of TV 52012/05/08

Page 6: MPEG for the  past, present and future  of television

MPEG for the future of television

Well, that’s what I am supposed to talk about…

2012/05/08 6MPEG for the past, present and future of TV

Page 7: MPEG for the  past, present and future  of television

Broadcasting is a great business to be in…

Broadcasting is a proven business with a constant revenue flow

Many technologies appear thatCan help new competing businesses but Can also be used to extend the broadcasting

businessMPEG is a neutral body devoted to the

development of standard technologies for user communities to exploit

MPEG for the past, present and future of TV 72012/05/08

Page 8: MPEG for the  past, present and future  of television

MPEG-H: a video standard for

future broadcastingISO/IEC 23008 High Efficiency Coding and Media

Delivery in Heterogeneous EnvironmentsA suite of standards for media coding and

deliveryCurrently scheduled to appear in 3 parts

MPEG Media TransportHigh Efficiency Video Coding3D Audio

Timeline

MPEG for the past, present and future of TV 82012/05/08

Part CD DIS FDISMMT 12/07 13/01 13/07HEVC 12/02 12/07 13/01

3D Audio

14/04 14/10 15/01

Page 9: MPEG for the  past, present and future  of television

Service composition in MMT

3D Earth

UD contents

Bird is flying…

Bird is flying…

<subtitle> composition information friendly with html5

legacy broadcasting future broadcasting

MMT Package

9MPEG for the past, present and future of TV 2012/05/02

Page 10: MPEG for the  past, present and future  of television

CloudsMMT Encapsulator

- Service Component List- Component location- Composition relationship- QoE requirement- How to deliver- How to consume

Webvideo

file

- Configuration- Adaptive delivery- Sync, Delay, Loss control

MMT Frame

pull pull

Smart delivery in Clouds

102012/05/08

Page 11: MPEG for the  past, present and future  of television

Mash-up service

Service 1 (MMT Package1) Service 2 (MMT Package2)

MMT assets

Audio Video1 widget Web pageVideo2

112012/05/08MPEG for the past, present and future of TV

Page 12: MPEG for the  past, present and future  of television

Second screen management

Additional Information with Live Content

(HTML)

Video content on 2nd device

composition information <device 1>…<device2>…

MMT Package

122012/05/08MPEG for the past, present and future of TV

Page 13: MPEG for the  past, present and future  of television

MPEG-H part 2 High Efficiency Video

CodingPrimary goal: ≥ 50% better compression than AVC

For large displays (e.g. 4kx2k)For mobile (e.g. WVGA)

Developed jointly with ITU-T by Joint Collaborative Team (JCT) on Video Coding

Technically in the same stream of past MPEG Video Coding standards (AVC and prior standards)

Currently at Committee Draft level

MPEG for the past, present and future of TV 132012/05/08

Page 14: MPEG for the  past, present and future  of television

HEVC vs AVC Subjective Performance

Measurements Preliminary subjective tests carried out

for HEVC Committee Draft Reasonably small and non-overlapping

confidence intervals Average rate savings at same

(interpolated) MOS points HEVC with 67% lower rate in class B (Full HD) HEVC with 49% lower rate in class C (WVGA) HEVC with 58% lower rate overall

2012/05/08MPEG for the past, present and future of TV 14

Page 15: MPEG for the  past, present and future  of television

A performance sample

2012/05/08MPEG for the past, present and future of TV 15

Page 16: MPEG for the  past, present and future  of television

For those technology aware…

Block-based/Variable block sizesBlock motion compensationFractional-pel motion vectorsSpatial intra predictionSpatial transform of residual differenceInteger-based transform designsArithmetic or VLC-based entropy codingIn-loop filtering to form final decoded picture

2012/05/08MPEG for the past, present and future of TV 16

Page 17: MPEG for the  past, present and future  of television

Where we are (May 2012)Activity: level of 1,000 documents per meeting crossed Very diverse participation from industry/academiaSignificant technical advance over prior standardsChallenge: computational/implementation complexityDeliverables:

SpecReference softwareConformance

Currently only one profile

2012/05/08MPEG for the past, present and future of TV 17

Page 18: MPEG for the  past, present and future  of television

HEVC Scalability Extensions/1

Serve the needs of heterogeneous environment of future digital TV distributionMultiple devices: SHDTVs, set-top boxes, tablets,

smart phones, PCs etc.Wide range of processing powers, display sizes and

power consumption needsMultiple networks: Cable, Satellite, Terrestrial, CDN,

Cable Modem, xDSL, WiFi, 4GWide range of channel bandwidths and QoS

Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDPMultiple services: Broadcasting, On demand,

Streaming, Over The Top

2012/05/08MPEG for the past, present and future of TV 18

Page 19: MPEG for the  past, present and future  of television

HEVC Scalability Extensions/2

Options for backward compatible migration pathsFrom 720p60 to 1080p60From1080p60 to 4kx2kx60From AVC to HEVC

Time LineMay 2012 Preliminary Call for Proposals (public) July 2012 Final Call for Proposals (public)October 2012

Evaluation of proposals & start of collaborative design phaseDevelopment of the time line associated with standardisation

phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD)

2012/05/08MPEG for the past, present and future of TV 19

Page 20: MPEG for the  past, present and future  of television

Service Transitions1950s: Color TV

Analog, backward compatible1990s: Digital TV

New infrastructure requiredTransitions from SD to HD

2010s: 3DMixed results from services introduction isNot a single format across all services

2012/05/08MPEG for the past, present and future of TV 20

Page 21: MPEG for the  past, present and future  of television

Current Status of 3D VideoIncreasing investments in 3D production and

servicesIncreasingly capable 3D displays in the market

(many competing and emerging technologies)Market is primarily stereo

(mix of different formats being deployed right now)Focus of current MPEG work: development of a new

3D format and associated compression techniques that could facilitate generation of multiview output

2012/05/08MPEG for the past, present and future of TV 21

Page 22: MPEG for the  past, present and future  of television

Existing 3DV Coding Formats

Pack pixels from left and right views into a single frame (loss of resolution)

Leverage existing infrastructure and equipment, only require additional signaling to de-interleave frame

Embraced by broadcasters for first phase of 3D broadcast services

Full-resolution coding of stereo and MV video as extension of AVC

High coding efficiency achieved via inter-view prediction techniques

2D compatibility supported Adopted as format for 3D Blu-ray

Disc Being considered for second phase

of broadcast standards

2012/05/08MPEG for the past, present and future of TV 22

Left Right

Left View Right View

Effective prediction between views for high compression performance

Left View Right View

Effective prediction between views for high compression performance

Page 23: MPEG for the  past, present and future  of television

Targets of Future 3DV Format

2012/05/08MPEG for the past, present and future of TV 23

Data Format

Data Format

Constrained Rate(based on distribution)

LimitedCamera Inputs

• Wide viewing angle• Large number of

output views

Left

Right

Auto-stereoscopicN-view displays

Stereoscopic displays• Variable stereo baseline• Adjust depth perception

Page 24: MPEG for the  past, present and future  of television

3DV Reference FrameworkLimited # Video

Inputs(e.g., 2 or 3 views)

DepthEstimation

Video/DepthEncoder

Decoder and View Synthesis

Larger #Output Views

Binary Representation& Reconstruction

Process

+

1010001010001

242012/05/08MPEG for the past, present and future of TV

Page 25: MPEG for the  past, present and future  of television

Overview of Call For Proposals

Test Material & Conditions8 test sequences (1920x1080 & 1024x768)4 target bit rates per sequence2-view and 3-view test scenariosAVC and HEVC compatible test categories

23 responses (12 AVC + 11 HEVC)Evaluation in Nov 2011

Objective quality (PSNR of synthesized views)Subjective assessment (stereo and auto-stereo)

2012/05/08MPEG for the past, present and future of TV 25

Page 26: MPEG for the  past, present and future  of television

3DV Testing Scenarios

2012/05/08MPEG for the past, present and future of TV 26

Left

Right

Center

Depth Estimation

3D Video Codec

Multiview Rendering

virtual views

Output toAuto-Stereo

N-view Display

Left

Right

Center

Depth Estimation

3D Video Codec

Multiview Rendering

virtual views

Output toAuto-Stereo

N-view Display

Left

Right

Depth Estimation

3D Video Codec

Multiview Rendering

virtual view

Output toStereo Display

Left

Right

Depth Estimation

3D Video Codec

Multiview Rendering

virtual view

Output toStereo Display

2-view Configuration

3-view Configuration

Page 27: MPEG for the  past, present and future  of television

Tool CategoriesTexture coding

Independent of depth E.g., inter-view prediction of color view, inter-view prediction

of motion parameters and residual dataUsing depth data

E.g., view synthesis prediction, motion prediction

Depth coding Independent of texture

E.g., depth modeling modes, weighted prediction, reduced resolution coding

Using texture dataE.g., motion parameter inheritance, intra prediction

2012/05/08MPEG for the past, present and future of TV 27

Page 28: MPEG for the  past, present and future  of television

3DV Current Status Extensive subjective evaluation

of 20+ proposals to the 3DV CfP Considered both AVC and HEVC

compatible

Many new coding tools proposed for MV texture and depth coding Substantial rate savings

compared to capabilities of existing standards

AVC and HEVC extensions standardisation under wayd Add support for depth New AVC-based coding tools MV extensions of HEVC Hybrid architectures

2012/05/08MPEG for the past, present and future of TV 28

Page 29: MPEG for the  past, present and future  of television

Hybrid ArchitecturesFrom a pure compression efficiency point of view, it

is always best to use the most advanced codecHowever, when introducing new services, providers

must also consider capabilities of existing receivers and transition plan

Use Case: Many terrestrial broadcast systems based on MPEG-2US cable systems based on mix of MPEG-2 and AVCMay not be easy to simply switch codecs in near term

2012/05/08MPEG for the past, present and future of TV 29

Page 30: MPEG for the  past, present and future  of television

Multiview Extensions of HEVC

HEVC Stereo/MV extensions being consideredExtensions expected to be completed

approximately 12 months after the base specification, i.e., Jan 2014

Reportedly simple extensions of HEVC can achieve 30-40% bit rate reduction compared to HEVC simulcast

Further gains expected if block-level changes to codec are considered

2012/05/08MPEG for the past, present and future of TV 30

Page 31: MPEG for the  past, present and future  of television

MPEG-H part 3: 3D AudioWell, there is no Systems and Video without

Audio…

2012/05/08MPEG for the past, present and future of TV 31

Page 32: MPEG for the  past, present and future  of television

3D Audio requirements/1High quality: quality of decoded sound perceptually

transparentLocalization and Envelopment: Accurate sound

localization and very high sense of sound envelopment within a targeted listening area (“sweet spot”)

Flexible Loudspeaker Placement: the transmitted audio program to be mapped to a rendering setup in which loudspeakers are in alternate locations

Interactivity: Interactive modification of the sound scene rendered from the coded representation

MPEG for the past, present and future of TV 322012/05/08

Page 33: MPEG for the  past, present and future  of television

3D Audio requirements/2Rendering on setups with fewer

loudspeakers (including headphones): ability to derive a signal from the transmitted program material for reproduction with fewer loudspeakers

Audio/visual alignment & consistency: ability to adapt the rendered acoustic scene to be consistent with the visual

Efficiency for decoding on different setups: ability to be rendered on all reproduction loudspeaker setups or headphone

MPEG for the past, present and future of TV 332012/05/08

Page 34: MPEG for the  past, present and future  of television

3D Audio requirements/3Appropriate computational complexity and

system latency for the target application (Broadcasting, Spatial two-(n-)way communication / telepresence

Transcoding for low bandwidth devices: A lower bandwidth signal can be extracted from the original program material

Backward compatibility: (e.g. to 5.1 channel programs and decoders or transmission of HRTF encoded signals)

MPEG for the past, present and future of TV 342012/05/08

Page 35: MPEG for the  past, present and future  of television

Envisioned Architecture

Inputs Audio Channels

“normal” content Audio Objects

“helicopter overflight” Audio Scene

Ambisonics

35

3D Audio Encoder

500 kb/sbitstream

3D Audio Encoder

Transcoder 64 kb/sbitstream

22.2 Channel Audio Program

Audio Objects

Audio Scene

SmartPhone TV

Home Theatre

Tablet TV

22.2 Loudspeakers

Spatialized on Headphones

In-Frame Speakers

Spatialized on Headphones

Spatialized on Headphones

• Outputs– Loudspeakers– Headphones

• Spatialized sound– In-frame

loudspeakers• “Sound Bar”

• Channel– 500 kb/s primary

channel– 64 kb/s cellular

channel

MPEG for the past, present and future of TV 2012/05/08

Page 36: MPEG for the  past, present and future  of television

Audio, part of an Audio-visual

ExperienceWhat is new in video?

Larger, high-resolution displaysGreater sense of envelopmentPossibly closer viewing distance

More efficient compressionWireless transmission to portable devices

Immersive and enveloping audio experience

362012/05/08MPEG for the past, present and future of TV

Page 37: MPEG for the  past, present and future  of television

Use Case 1Home Theatre

Many loudspeakers10.122.2

“3D” ExperienceHeight loudspeakers for greater envelopmentSense that audio objects are near or distantWhen near, they are consistent with video image

372012/05/08MPEG for the past, present and future of TV

Page 38: MPEG for the  past, present and future  of television

22.2 multichannel sound system (NHK)

2012/05/08MPEG for the past, present and future of TV 38

TV Screen

Top layer9 channels

Middle layer10 channels

Bottom layer3 channels

LFE2 channels

TpFL TpFC TpFR

TpSiLTpC

TpSiR

TpBLTpBC

TpBR

BLBC

BR

SiL

FLFLc FC FRc

FR

SiR

BtFLBtFC

BtFRLFE1 LFE2

TV Screen

Top layer9 channels

Middle layer10 channels

Bottom layer3 channels

LFE2 channels

TpFL TpFC TpFR

TpSiLTpC

TpSiR

TpBLTpBC

TpBR

BLBC

BR

SiL

FLFLc FC FRc

FR

SiR

BtFLBtFC

BtFRLFE1 LFE2

Page 39: MPEG for the  past, present and future  of television

Home Theater – IssuesMaintain compatibility with legacy systems: 5.1,

stereo Will content providers really step up to 22.2?Will consumers adopt 22.2?: 2 front highHow to compress 22.2 for transmission?

MPEG AAC 64 kb/s/channel: -> 1.5 mb/sHow to render for legacy setups?Relatively slow-growth market

2012/05/08MPEG for the past, present and future of TV 39

Page 40: MPEG for the  past, present and future  of television

Use Case 2Personal TV

Small but high-resolution display“super-tablet”

Local wireless communicationsTo cable or fiber home hub

Possible audio presentationHeadphonesLoudspeakers around perimeter of display

Issue: To what extent can the user get an enveloping experience from only “front” loudspeakers?

40MPEG for the past, present and future of TV 2012/05/08

Page 41: MPEG for the  past, present and future  of television

Use Case 3Mobile TV

Hand-held display: SmartphoneHeadphone listening:

Stereo, perhaps with binauralisationFastest growing market: Quick rollout and adoption

of standardWhat is new?

USAC for stereo; MPEG Surround for binauralisationWhat is the “Wow” factor?

To spur adoption of new technology

41MPEG for the past, present and future of TV 2012/05/08

Page 42: MPEG for the  past, present and future  of television

User Experience

42

Home Theatre

Tablet TV

Spatialized on headphones

2012/05/08MPEG for the past, present and future of TV

Page 43: MPEG for the  past, present and future  of television

Flexible RenderingContent providers create 22.2 programConsumed on stereo, 5.1, 10.1, 22.2 layoutsConsumed on “wrong” layouts

Mis-positioned surroundsMissing surroundsNon-standard layouts

2 front high, 5.1 mid

43MPEG for the past, present and future of TV 2012/05/08

Page 44: MPEG for the  past, present and future  of television

3D Audio Test PlatformNHK “Loudspeaker

Array Frame” (LAF)Suggestions for

Alternative platform for assessing “Home Theatre” use case

Model for assessing “Personal TV” use case

44MPEG for the past, present and future of TV 2012/05/02

Page 45: MPEG for the  past, present and future  of television

SummaryProvide compelling immersive audio experience

for audio/visual programsCreate content once, present on many different

loudspeaker layouts or on headphonesProvide high compression and high-quality audio

presentation

45MPEG for the past, present and future of TV 2012/05/08

Page 46: MPEG for the  past, present and future  of television

MPEG-V: Humans have more than two senses

2012/05/08MPEG for the past, present and future of TV 46

Page 47: MPEG for the  past, present and future  of television

MARBle

MediaServers

ServiceServers

User

LocalSensors

& Actuators

RemoteSensors & Actuators

ARAF file or

stream

Local Real World

Environment

Remote Real World

Environment

Augmented Reality Application Format (ARAF)

MPEG for the past, present and future of TV 2012/05/08 47

Page 48: MPEG for the  past, present and future  of television

Augmentation Region in ARAF

Augmentation Region

Broadcaster

AR service provider A

AR service provider B

User A

User B

MPEG for the past, present and future of TV 2012/05/02 48

Page 49: MPEG for the  past, present and future  of television

The past, the present and the future, again

25 years ago MPEG selected the digital media way bringing innovation and interoperability

…and thrivedThis year the broadcasting industry has started

a move in future broadcast services with interoperability at its core

…and it is bound to thriveMPEG looks forward to continuing a cooperation

with the broadcasting industry providing the necessary standard technologies

2012/05/08MPEG for the past, present and future of TV 49

Page 50: MPEG for the  past, present and future  of television

502012/05/08MPEG for the past, present and future of TV

http://mpeg.chiariglione.org/