ultra-high-definition quality of experience with mpeg-dash
TRANSCRIPT
Ultra-High-Definition Quality of Experience with MPEG-DASH
Priv.-Doz. Dr. Christian TimmererDaniel Weinberger, Christopher Mueller and Stefan Lederer
Chief Innovation Officer (CIO) at bitmovin GmbHhttp://www.bitmovin.com [email protected]
Alpen-Adria-Universität Klagenfurt (AAU) Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC) Multimedia Communication (MMC) Sensory Experience Lab (SELab)
http://blog.timmerer.com http://selab.itec.aau.at/ http://dash.itec.aau.at [email protected]
http://www.slideshare.net/christian.timmerer
Outline• Introduction• Quality, Quality of Experience, and DASH• Evaluation strategies• Results• Conclusions
• Acknowledgment [some slides]: Ali C. Begen, CISCOApril 16, 2015 NAB2015 - BEC - QoE-DASH 2
Introduction• Real-time entertainment
– Streaming video and audio– > 60% of Internet traffic
• All delivered over-the-top (OTT)• MPEG Dynamic Adaptive Streaming
over HTTP (DASH)– Coding format agnostic
• DASH Industry Forum– Interoperability Points (IOPs) for
common codecs and others (v3.0)– E.g., AVC/H.264, HEVC/H.265, and AAC
April 16, 2015 NAB2015 - BEC - QoE-DASH 3
Over-The-Top – Adaptive Media Streaming
April 16, 2015 NAB2015 - BEC - QoE-DASH 4
Adaptation logic is within the client, not normatively
specified by the standard, subject to research and
development
Open Digital Media Value Chain
April 16, 2015 NAB2015 - BEC - QoE-DASH 5
Create Content
Aggregate
Monetize
Distribute
Content
Consume Content
Any Content Any Storefront
Any Network Any Device
CDNsMedia Protocol
s
Internet Transpor
t
DRMEncoding
Encapsulation
DynamicAds
Clients
Simplified Example Workflow: bitcodin/bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 6
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/
Internet TV vs. Traditional TV in 2010• Areas most important to overall TV
experience are– Content– Timing control– Quality– Ease of use
• While traditional TV surpasses Internet TV only in quality, it delivers better “overall experience”
April 16, 2015 NAB2015 - BEC - QoE-DASH 7
When comparing traditional and Internet TV, which option is better?
Traditional Internet
Content 7% 79%
Timing / Control 7% 83%
Quality 80% 16%
Ease of Use 23% 52%
Control (FF, etc.) 9% 77%
Portability 4% 92%
Interactivity 31% 52%
Sharing 33% 56%
Overall Experience 53% 33%
Source: Cisco IBSG Youth Survey, Cisco IBSG Youth Focus Group Sessions, 2010
Quality (of Experience)• QoE as evolution of QoS [ITU-T
P.10/G.100]• QoS: totality of characteristics of a
telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service
• QoE: the overall acceptability of an application or service, as perceived subjectively by the end-user
April 16, 2015 NAB2015 - BEC - QoE-DASH 8
Many definitions but in general, it’s like an elephant
Quality of Experience• COST Action IC1003 – QUALINET (http://www.qualinet.eu/)
“the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of
the application or service in the light of the user’s personality and current state”
• QoE influence factors– Any characteristic of a user, system, service, application, or context– Grouped into human, system, and context
• QoE features– Perceivable, recognized and namable characteristic of the individual’s experience– Depends on the level of direct perception, interaction, the usage situation
April 16, 2015 NAB2015 - BEC - QoE-DASH 9
QoE for DASH• Different application domains have different QoE requirements
– Need to provide specializations of the general QoE definition– Take into account requirements formulated by means of influence
factors and features of QoE• QoE influence factors for DASH
– Initial/start-up delay (low)– Buffer underruns, stalls, freezes (zero)– Quality switches (low)– Media throughput (high)– …
April 16, 2015 NAB2015 - BEC - QoE-DASH 10
QoE Evaluation for DASH-based Services• Test sequence
– Many datasets available– Adopted Big Buck Bunny & DASHed it with bitcodin
• Players– bitdash– …and compare it with ten different adaptation algorithms
• Objective evaluation– Test setup– Predefined bandwidth trajectory (or real network traces)
• Subjective evaluation– Lab vs. crowdsourcing
April 16, 2015 NAB2015 - BEC - QoE-DASH 11
http://www.bitcodin.com/
http://www.dash-player.com/
Objective Evaluations
April 16, 2015 NAB2015 - BEC - QoE-DASH 12
Stalls (lower is better)Average Bitrate (higher is better)
Stalls are really bad…
April 16, 2015 NAB2015 - BEC - QoE-DASH 13
Conviva: Viewer Experience Report. 2014
DASH-JS vs. bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 14
Subjective Evaluation• Microworker platform
– Limited to Europe, USA/Canada, India• DASH clients
– DASH-JS (dash.itec.aau.at)– dash.js (DASH-IF)– YouTube
• Tears of Steal trailer according to YouTube configuration
• Screening techniques– Browser fingerprinting– Presentation time– QoE ratings and Pre-QuestionnaireApril 16, 2015 NAB2015 - BEC - QoE-DASH 15
What about 4K and 8K?
April 16, 2015 NAB2015 - BEC - QoE-DASH 16
• Why? – because we can!• Supported on the Web
– HTML5, MSE– AVC/H.264– [HEVC/H.265 needed
to lower bitrate]• See demo @
http://www.dash-player.com/
• UHD-QoE evaluation
Conclusions• QoE for DASH-based services (a rule of thumb)
– Startup delay (low [but live vs. on-demand & short vs. long-tail content])– Buffer underrun / stalls (zero)– Quality switches (low) and media throughput (high)– Energy- and cost-awareness (data plan)
• No general applicable QoE model for DASH– (Too) many factors influencing / features of QoE for DASH-based services– Methodology for reproducible research is in place and well established– Ample research opportunities
April 16, 2015 NAB2015 - BEC - QoE-DASH 17
Main QoE factors for DASH
Thank you!
April 16, 2015 NAB2015 - BEC - QoE-DASH 18
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/