Building media streaming apps and sites without plug-ins using MPEG-DASHDaniel SchneiderSenior Lead Program Manager, Microsoft

Anthony ParkVice President, Engineering - Netflix 3-089

Key questions this talk aims to answer

• What is MPEG-DASH and how does it help you to deliver a better video experience to your customers at a lower cost?

• How does Windows enable video services to build MPEG-DASH clients?

• Which tools are available to quickly get started developing an MPEG-DASH service on Windows?

Agenda

• Introduction• The MPEG-DASH Standard• W3C Media Source Extensions• W3C Encrypted Media Extensions• Implementing an MPEG-DASH service in IE or a

Windows Store app using JavaScript and HTML• Summary

Why standardize video APIs in browsers?• Customers expect to be able to watch Netflix on

any operating system or browser• Common playback APIs reduce the burden of

developers to have different implementations on different platforms

• Custom adaptive streaming algorithms can be written once in JavaScript and leveraged everywhere

Demo: Netflix in IE11 Previewon Windows 8.1 Preview

The Future of Netflix on Windows• HTML5 Video in the Web Browser

• Supports multi-tasking scenarios, multi-tab, etc.

• Windows Store App• Touch-first design, great for tablet and touch-screen hardware• Provides deep integration with platform features (share contracts, search

integration, etc.)• Rich app experience (ex: semantic zoom)• Dolby Digital 5.1 audio support

Moving from Plugins to HTML5 video• Improved battery/CPU efficiency

• GPU-acceleration means less work for the CPU• Full-quality 1080p now possible within the browser on low to mid-range PCs

• New browsers shying away from plug-in support• Internet Explorer in the new Windows UI, as an example, doesn’t support plug-ins

• No more installing plug-ins!• Removes a hurdle for new customers• HTML5 video just works, no plug-ins necessary

0306

09

Relevant standards and proposalsISO/IEC 23009-1 “Dynamic Adaptive Streaming over HTTP (DASH)”- Specifies manifest and media segment formats ISO/IEC 14496-12 “Part 12: ISO Base Media File Format, Edition 4” (aka MP4 container)- Specifies data structures for packaging media ISO/IEC 23001-7 “Common Encryption in ISO Base Media File Format”- Specifies how to package protected content for

consumption by multiple DRM systemsW3C Proposal “Media Source Extensions”- Specifies an HTML extension API enabling the

implementation of adaptive streaming in the browserW3C Proposal “Encrypted Media Extensions”- Specifies an HTML extension API enabling access to

DRM implementations on the system from the browser

MSE and EME

Trident / IE

Web app or Windows Store app

using JavaScript and HTML

MFDRM

Manifest

- 480,- 720- 1080

Formats

Client

The MPEG-DASH standard

• DASH stands for “Dynamic Adaptive Streaming over HTTP”

• Specifies an adaptive media-streaming model• Control of the presentation lies exclusively with the client• Uses HTTP and standard web servers

• Specifies formats for • Media Segments (media files)• Media Presentation Description (manifest)

• Set of restrictions on the offered Media Presentation (MPD & Segments)

• Windows MSE implementation supports implementation of ISOBMFF based profiles:

• On-Demand Profile• Segments are addressed as byte-range

requests• Suitable for VOD only

• Live Profile• Segments can be addressed by index number

or time stamp• Can be used for Live and VOD delivery

DASH Profiles

Full Profile

ISO Base media file format Main (MP4)

MPEG-2 TS

Main

ISO BMFF

On Demand

ISO BMFF Live

MPEG-2 TS Simpl

e

Typical DASH ISOBMFF (MP4) file

Mime Type (ftyp)

Movie Metad

ata (moov)

Segment

Index (sidx)

Movie Frag #1 (moof)

Media Data #1

(mdat)

Movie Frag #2

(moof)

Media Data #2

(mdat)....

Initialization Segment

IndexSegment

(not for live)

Media (or Data)Segment

Media (or Data)Segment

Typical DASH MP4 file

Mime Type (ftyp)

Movie Metad

ata (moov)

Segment

Index (sidx)

Movie Frag #1 (moof)

Media Data #1

(mdat)

Movie Frag #2

(moof)

Media Data #2

(mdat)....

PT: 0PT: 3PT: 6

IndexSegment

(not for live)

Typical DASH MP4 file

Mime Type (ftyp)

Movie Metad

ata (moov)

Segment

Index (sidx)

Movie Frag #1 (moof)

Media Data #1

(mdat)

Movie Frag #2

(moof)

Media Data #2

(mdat)....

Initialization Segment

IndexSegment

(not for live)

Media (or Data)Segment

Media (or Data)Segment

DASH Manifest (MPD) high-level structure (live profile)

<MPD profiles="urn:mpeg:dash:profile:isoff-live:2011“><Period>

<ContentProtection schemeIdUri="urn:mpeg:dash…" value="cenc"/><AdaptationSet mimeType=“video/mp4”>

<SegmentTemplatemedia=“Level($Bandwidth$)/Fragment($Time$)initialization="Level($Bandwidth$)/Fragments(init)><SegmentTimeline>

<S d=3 r=430 /></SegmentTimeline>

</SegmentTemplate><Representation bandwidth="6000000" width="1920" height="1080“/><Representation bandwidth="2962000" width="1280" height="720“/><Representation bandwidth="1427000" width="856" height="480“/>

</AdaptationSet><AdaptationSet mimeType=“audio/mp4”>

<SegmentTemplate …<Representation bandwidth=“160000”/><Representation bandwidth=“64000”/>

Manifest

- 480,- 720- 108

0

Media Presentation Description Live.XML

DASH Manifest (MPD) high-level structure (on-demand profile)

Manifest

- 480,- 720- 108

0

<MPD profiles="urn:mpeg:dash:profile:isoff-on-demand:2011“/><Period>

<ContentProtection schemeIdUri="urn:mpeg:da…" value="cenc"/><AdaptationSet mimeType=“video/mp4”>

<Representation bandwidth="6000000" width="1920" height="1080“>

<BaseURL>movie-high.mp4</BaseURL><SegmentBase indexRange=“804-1123”>

<Initialization range=“0-803”></SegmentBase>

</Representation><Representation bandwidth="2962000" width="1280" height="720“>

<BaseURL>movie-medium.mp4</BaseURL><SegmentBase …

</Representation><Representation bandwidth="1427000" width=“856" height="480“>

<BaseURL>movie-low.mp4</BaseURL><SegmentBase …

</Representation></AdaptationSet>

…

Media Presentation Description OD.XML

W3C Media Source Extensions

- W3C proposal extending the HTML standard

- Defines an API to control media getting to the media element

- Allows a web app to push init and media fragments into the video/audio tag

- The main objects are MediaSource and SourceBuffer

W3C Media Source Extensions

<video> MediaSource

SourceBufferaudio

SourceBuffervideo

src sourceBuffers

JavaScript AppJavaScript mediaSource <video>

sourceBuffer Video

new MediaSource()

video.src = URL.creatObjectURL(mediaSource)

sourceopen!event

addSourceBuffer($video/mp4')

'updateend' event

'loadedmetadata' event

appendBuffer(initSegment_video_quality1)

HTTP Server

XHR to get initSegment

Setup

1. Create MediaSource and set it as src to the video tag

2. Add source buffers for audio and video

3. Download and append init segments to each of them

JavaScript AppJavaScript mediaSource <video> sourceBuffer Video

appendBuffer(dataSegment1_video_quality1)

updateend !event

canplay !event

play()

appendBuffer(dataSegment2_video_quality1)

updateend !event

remove(sourceBufferVideo.buffered.start(0), video.currentTime - MAX_BUFFER)

updateend !event

HTTP Server

XHR to get dataSegment

XHR to get dataSegment

Playback

1. Download and append data segment to each source buffer

2. Keep downloading and appending as content plays

3. Also keep removing to control your buffer

JavaScript AppJavaScript mediaSource <video> sourceBuffer Video

appendBuffer(dataSegment3_video_quality2)

updateend !event

'updateend' event

appendBuffer(initSegment_video_quality2)

HTTP Server

XHR to get initSegment

XHR to get dataSegment

Switch streams

1. Download and append init segment for new quality level

2. Download and append data segments at the new quality level

JavaScript AppJavaScript mediaSource <video> sourceBuffer Video

appendBuffer(dataSegment10_video_quality2)

updateend !event

'seeking' event

video.currentTime = a new time position (mapped to dataSegment10)

map video.currentTime to dataSegment10

'seeked' event

HTTP Server

XHR to get dataSegment

Seek

1. currentTime on video tag being set to new position

2. Map currentTime to data segments and download these

3. Append data segments for the new position

Demo: Simple streaming client using Media Source Extensionshttp://ie.microsoft.com/testdrive

W3C Encrypted Media Extensions

- W3C proposal extending the HTML standard

- Defines an API that allows the use of a DRM system in a web app entirely in JavaScript

- The main objects are MediaKeys and MediaKeySession

- The entry points to the API are prefixed with MS

W3C Encrypted Media Extensions

<video> MSMediaKeys MediaKeySession

msKeys

CloudLicense Server JavaScript mediaKeys mediaKeySession <video>

new MSMediaKeys("com.microsoft.playready")

'msneedKey' event w/ initData

'mskeymessage' event w/ key request + default URL + session ID

License Challenge

License Response

update(licenseResponse)

'mskeyadded' event

msSetMediaKeys(mediaKeys)

createSession(type, initData,...)

Content Playback

'canplay' event

play()

Protected Content1. In response to

msneedKey event, create MSMediaKeys object and set it on the video tag

2. Create a MediaKeySession using init data from msneedkey event

3. In resonse to mskeymessage event acquire license from server and update mediaKeySession with response

Implementation of MSE and EME in Windows• MSE and EME can be used in IE and “Windows Store app using

JavaScript and HTML”• Through MSE and EME, Windows can process/play DASH

segments• Only ISO Base Media File Format (no MPEG2-TS)• Supported codecs: – In IE: H264, AAC– In HTML store apps: also Dolby Digital+

• PlayReady key system is pre-installed on all Windows versions with media capabilities and can be invoked via: • new MSMediaKeys(‘com.microsoft.playready’)

Implementing your MPEG-DASH streaming service in Internet Explorer or in a Windows Store app using JavaScript and HTML

Getting started: what you need

1. DASH sample app • (e.g. DASH.js)

2. A tool to inspect MP4 files • (e.g. MP4 Explorer)

3. Packaging tool/packaging pipeline • (e.g. MP4 box, Windows Azure Media Services)

4. Delivery Server • (e.g. IIS, Windows Azure Media Services)

• The DASH Industry Forum’s reference player.

• A JavaScript implementation of a DASH client in the browser using MSE and EME

Player: http://dashif.org/reference/players/javascript/index.html Source code: https://github.com/Dash-Industry-Forum/dash.js

DASH.JS

What are Media Services?

http://www.windowsazure.com/en-us/pricing/free-trial/

http://www.windowsazure.com/en-us/pricing/member-offers/msdn-benefits/

Begin today!

Things to note• When fetching segments / acquiring license via

XHR• Observe Cross Origin Resource Sharing (CORS)• Host the player page on an http (not https) URL

• For power efficiency• To avoid unnecessary disk caching, set

xhr.responseType = ‘ms-stream’ or xhr.msCaching = ‘disabled’

• make use of the HTML Fullscreen APIdocument.getElementById(“video”).msRequestFullscreen();

• Monitor download speed and playback quality to determine when to change bitrate

partial interface HTMLVideoElement {

readonly attribute VideoPlaybackQuality videoPlaybackQuality;

};

interface VideoPlaybackQuality { readonly attribute DOMHighResTimeStamp creationTime; readonly attribute unsigned long totalVideoFrames; readonly attribute unsigned long droppedVideoFrames; readonly attribute unsigned long corruptedVideoFrames; readonly attribute double playbackJitter;};

Monitoring playback quality

Attributes likely to change

Summary

• MPEG-DASH provides standardization in the area of adaptive streaming• One format going to multiple clients

• In IE and Windows Store apps using JavaScript and HTML, the MSE and EME APIs can be leveraged• Use the same code across app and site• Take full advantage of the hardware• Plugin-free in the browser

• End-to-end tools are available today to kick-start development of a DASH service

ResourcesDASH Specification (ISO 23009-1): • http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html Media Source Extensions Specification: • https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html Encrypted Media Extensions Specification: • https://dvcs.w3.org/hg/html-media/raw-file/tip/encrypted-media/encrypted-media.html ISO Base Media File Format Specification (ISO 14496-12:2008/FDAM-3):• http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html Common Encryption Specification (ISO 23001-7): • http://

www.3gpp.org/ftp/Inbox/LSs_from_external_bodies/ISO_IEC_JTC1_SG29_WG11/29n12313.zip

DASH.js: Reference Implementation of DASH in the browser by the DASH-IF.• https://github.com/Dash-Industry-Forum/dash.js/wiki• http://dashif.org/reference/players/javascript/0.2.3/index.htmlAdaptive Streaming SDK + Player Framework• http://playerframework.codeplex.com/

Evaluate this session

Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize!

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.