web du mobile meow

112
Mobile Meow By YouTube's Greg Schechter and Matt Ward Bring YouTube to the Mobile Web

Upload: greg-schechter

Post on 17-May-2015

809 views

Category:

Technology


1 download

DESCRIPTION

This is the story of the glory and struggle of bringing a high quality YouTube experience to the mobile web. Once upon a time there was a web developer who wanted to play videos on the web. So he filmed a cat and wrote a Flash and a HTML5 player. He spent many hours making it work on his favorite desktop browsers and even the one his grandfather still used. People could watch his cat video, he smiled, and the world was good. Then one day someone put a browser in a phone and soon there were many phones with many different browsers. This new set of environments were even harder to develop for and had a slew of new terrifying bugs. The web developer was miserable knowing people couldn’t watch his cat video. With much time and effort he figured out many of the secrets needed to combat the evils of the different mobile platforms. Once again people could watch his cat video, he smiled, and the world was good. The end.

TRANSCRIPT

Page 1: Web DU Mobile Meow

Mobile Meow

By YouTube's Greg Schechter and Matt Ward

Bring YouTube to the Mobile Web

Page 2: Web DU Mobile Meow

Greg SchechterThe Web Warrior

[email protected]

Matt WardSeek Bar Engineer

[email protected]

Page 3: Web DU Mobile Meow

Video

Page 4: Web DU Mobile Meow

Video

Page 5: Web DU Mobile Meow
Page 6: Web DU Mobile Meow
Page 8: Web DU Mobile Meow

Video

Page 9: Web DU Mobile Meow

Flash

Page 10: Web DU Mobile Meow

<video>

Page 11: Web DU Mobile Meow

<video>

IE

<video>

Chrome

<video>

Firefox

<video>

Opera

<video>

Safari

Page 12: Web DU Mobile Meow

Video

Page 13: Web DU Mobile Meow

Playback Stats

Page 14: Web DU Mobile Meow

Flash Flash

Page 15: Web DU Mobile Meow

Flash Native <video>

Page 16: Web DU Mobile Meow

Flash Flash==Flash

Page 17: Web DU Mobile Meow

Native

Page 18: Web DU Mobile Meow

○ For fallbacks, use rtsp:// protocol (serving .3pg) if the device won't support HTML5

○ Use custom protocol / URL scheme to launch your own Android, iOS native app (no Windows Phone yet)

■ youtube://video_id

Native

Page 19: Web DU Mobile Meow

<video>

Page 20: Web DU Mobile Meow

The Different Browsers

Safari Android

Page 21: Web DU Mobile Meow

Safari Android Chrome Opera Mini Silk

IE

The Different Browsers

Firefox Opera

Page 22: Web DU Mobile Meow

Safari Android

Firefox

Chrome

Opera

Opera Mini Silk

IE

The Different Browsers

SAMPLE

Page 23: Web DU Mobile Meow

Fragmentation

Page 24: Web DU Mobile Meow

By brownpau at http://www.flickr.com/photos/brownpau/533267369/

Page 25: Web DU Mobile Meow

<video>

How do we start

Page 26: Web DU Mobile Meow

<video src="funny_cat_video">

The Source

Page 27: Web DU Mobile Meow

<video src="funny_cat_video.webm/mp4/ogg">

The Source

Page 28: Web DU Mobile Meow

Mobile Formats Support

Chrome Safari Firefox Opera IE Android

H.264

WebM Android 4+ Android 4+ Android 4+

HLS Android 3+

Platform Versions and Distribution

Page 29: Web DU Mobile Meow

The Source

<video> <source src="funny_cat_video.mp4"> <source src="funny_cat_video.webm"></video>

Page 30: Web DU Mobile Meow

HLS = HTTP Live Streaming

<video src="playlist.m3u8">

Page 31: Web DU Mobile Meow

HLS

index.m3u8

format-0.m3u8

format-0.m3u8

format-0.m3u8

video-0-1.tsvideo-0-2.ts

...

video-1-1.tsvideo-1-2.ts

...

video-2-1.tsvideo-2-2.ts

...

Page 32: Web DU Mobile Meow

HLS

Big Features○ Live and Adaptive Streaming○ Built in CDN failover○ Metadata + Cue Points

Apple App Requirements○ > 10 minutes in duration or 5 mb in 5 minutes○ at least one 64 kbps or lower stream required

Tools and more information on Apple's website:Apple's Using HTTP Live Streaming

Page 33: Web DU Mobile Meow

Power

Page 34: Web DU Mobile Meow

Power Consumption: H264 vs WebM

**fullscreen flash**windowed flash

Research done by Yossi Oren For more information visit http://iss.oy.ne.ro/

Page 35: Web DU Mobile Meow

Stick it in a page

Page 36: Web DU Mobile Meow

Basic video tag

Safari○ Renders a thumbnail poster and play button (flaky)○ Background is set to black and cannot be overwritten○ If a parent element has display:none set at any point the video fails to

playback properly

Opera○ Gives video dimentions black background if the information is

available○ Has issues with stretching WebM

IE and Android○ Play button/film icon for all video tags regardless if browser can play

the format○ Background is set to black/grey and cannot be overwritten

Page 37: Web DU Mobile Meow

Give it a poster and click it

Page 38: Web DU Mobile Meow

Poster Attribute

Use the poster attribute to get a thumbnail Safari, Chrome, Android, and Opera

○ Preserve aspect ratio of poster image

IE and Firefox○ Stretch image to size of video tag

IE○ Keeps poster as paused state

All Others○ Replace video with the paused frame

Page 39: Web DU Mobile Meow

Chrome○ Default click to toggle play/pause○ Remove with an empty onclick handler

Safari○ Will launch any supported video with or without an onclick handler

Android and IE○ Shows click interactions but does not play with empty onclick

Basic video tag

Page 40: Web DU Mobile Meow

Pro tips:○ Create your own cued state to convey you can play the video○ Set the background color to black for continuity across platforms

Basic video tag

Page 41: Web DU Mobile Meow

Controls

<video controls>

Page 42: Web DU Mobile Meow

Lets get some controls

Page 43: Web DU Mobile Meow

Controls Chrome Safari Firefox Opera IE Android

Play/pause Button

Volume Controls

Only in Fullscreen

Mute toggle only

Don't work in Android 4

Seek bar

Fullscreen button

Button Gesture Button

Playback Type

Both Fullscreen only

Inline only Determined by hardware

Fullscreen only

Both

Page 44: Web DU Mobile Meow

Fullscreen

Page 45: Web DU Mobile Meow

requestFullScreen

Page 46: Web DU Mobile Meow

requestFullScreen

● Desktop Only○ Firefox, Safari, and Chrome○ Vendor Prefixed

Page 47: Web DU Mobile Meow

Fullscreenvar elem = document.getElementById("my-element");document.onwebkitfullscreenchange = function() { // The fullscreen element: // document.webkitFullScreenElement; console.log ("We went fullscreen!");};elem.webkitRequestFullScreen();

Page 48: Web DU Mobile Meow

What about Mobile?

Page 49: Web DU Mobile Meow

Isn't it already fullscreen?

Page 50: Web DU Mobile Meow

Yes (sort of)

Page 51: Web DU Mobile Meow

Fullscreen

● Desktop○ requestFullScreen

● Mobile

Page 52: Web DU Mobile Meow

Fullscreen

● Desktop○ requestFullScreen

● Mobile○ Open New Tab

Page 53: Web DU Mobile Meow

Fullscreen

● Desktop○ requestFullScreen

● Mobile○ Open New Tab○ webkitEnterFullscreen

● Webkit only

● Video element only

● metadata must be loaded

Page 54: Web DU Mobile Meow

Fullscreen

● Desktop○ requestFullScreen

● Mobile○ Open New Tab○ webkitEnterFullscreen

● Webkit only

● Video element only

● metadata must be loaded

● Pro Tip: Remember size context changes so use viewport to scale icons and controls

Page 55: Web DU Mobile Meow

@viewport { width: device-width; }

Page 56: Web DU Mobile Meow

Custom Controls

Lets get some custom controls

Page 57: Web DU Mobile Meow

Custom Controls

● Preserve your brand● Unified experience across platforms and

browsers● Allows us to expand the set of controls and add our own

○ annotations○ playlist○ captions○ more

Page 58: Web DU Mobile Meow

Custom Controls Pro Tips● User expect to be able to drag the progress

bar○ Need to remender to prevent scroll on touchstart

● Volume can't be set everywhere and users are accustomed to using device controls○ So don't build controls for it

Page 59: Web DU Mobile Meow

Custom Controls Pro Tips● Fullscreen

○ The browsing context is always fullscreen so fake it● Don't overload clicking for both play/pause

and show/hiding controls● Don't trigger content with hover

Page 60: Web DU Mobile Meow

Custom Controls Pro Tips● Fingers are fat

○ Average finger is 11mm so make targets at least 40px with 10px padding

○ Use SVG's so icons can be scaled and shared with desktop application

○ Do what you can in css

Page 61: Web DU Mobile Meow

autoplay

<video autoplay>

Page 62: Web DU Mobile Meow
Page 63: Web DU Mobile Meow

autoplay

In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.

Page 64: Web DU Mobile Meow

autoplay

In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.

Page 65: Web DU Mobile Meow

What about everyone else?

Page 66: Web DU Mobile Meow

Autoplay

Chrome Safari Firefox Opera IE Android

Attribute

Scripted Buggy

Page 67: Web DU Mobile Meow

autoplay

Page 68: Web DU Mobile Meow

autoplay

<video onclick="this.play()">

Page 69: Web DU Mobile Meow
Page 70: Web DU Mobile Meow
Page 71: Web DU Mobile Meow

autoplayfunction someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call.} function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); myVideoElement.play();}

Page 72: Web DU Mobile Meow

autoplayfunction someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call.} function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); // For Android window.setTimeout(function() { myVideoElement.play(); }, 0);}

Page 73: Web DU Mobile Meow

autoplay

setMediaPlaybackRequiresUserAction

Page 75: Web DU Mobile Meow

Embeds

<script>

Page 76: Web DU Mobile Meow

Embeds

<script>○ We need our content to be sandboxed○ More than just a video tag

Page 77: Web DU Mobile Meow

Embeds

<script>○ We need our content to be sandboxed○ More than just a video tag

<object>

Page 78: Web DU Mobile Meow

Embeds

<script>○ We need our content to be sandboxed○ More than just a video tag

<object>○ Can load content with the data

attribute○ But no way to interact with it via

JavaScript

Page 79: Web DU Mobile Meow

Embeds

<iframe>

Page 80: Web DU Mobile Meow

Embeds

<iframe>○ Allows our content to be sandboxed○ JavaScript API communication

Page 81: Web DU Mobile Meow

<iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen></iframe>

Embeds

Page 82: Web DU Mobile Meow

Daily impressions of Embed

April 2011 April 2012

iframe

object

Total

Page 83: Web DU Mobile Meow

Pro tip: Plan for the future (if you can) <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen></iframe>

Embeds

Page 84: Web DU Mobile Meow

Embeds Pro Tips:

html { /** Hack to fix iPhone resizing. */ overflow: hidden;}

Page 85: Web DU Mobile Meow

Embeds Pro Tips:

body { /** Dymanic Resizing **/ background-color: #000; height: 100%; width: 100%; /** Remove highlight when use clicks **/ -webkit-tap-highlight-color: rgba(0, 0, 0, 0);}

Page 86: Web DU Mobile Meow

API

Page 87: Web DU Mobile Meow

API Communication

Page 88: Web DU Mobile Meow

API Communication● Poll the URL fragment?

○ http://youtube.com/embed/video_id#fragment

Page 89: Web DU Mobile Meow

API Communication● Poll the URL fragment?

○ http://youtube.com/embed/video_id#fragment● Messages are one dimensional.● Polling eats up CPU and is not instant.● Both directions of communication use the same

fragment.

Page 90: Web DU Mobile Meow

API Communication● Better idea: PostMessage API

○ someWindow.postMessage(message, targetOrigin);

Page 91: Web DU Mobile Meow

API Communication● Better idea: PostMessage API.

○ someWindow.postMessage(message, targetOrigin);● Uses JSON for native encoding and decoding of data.● No polling.● Native event listeners.● Communications are sandboxed per-window.● Calls are asynchronous.● Great for Mobile, but desktop support is missing in IE7

Page 92: Web DU Mobile Meow

Remember all communication is asynchronous

Page 93: Web DU Mobile Meow

Testing

Page 94: Web DU Mobile Meow

● Device lab○ accurate○ costly○ space inefficient○ boring

Testing: The Old Fashioned Way

Page 95: Web DU Mobile Meow

Testing: With Software● Hardware Emulators / Simulators

○ available for major systems○ approximation varies○ still boring

● Android Emulator○ bulky, slow, no video codecs

● iOs Simulator○ both tablet and phone

Page 96: Web DU Mobile Meow

Testing: With Software● Browser Simulators

○ Poorer approximation● Opera Mobile Simulator

○ missing video tag support● Fennec (FF) Simulator

○ poor touch control mapping○ sends desktop user agent

Page 97: Web DU Mobile Meow

Testing: Automation● Selenium!

○ use with simulated or real devices○ not boring

Page 98: Web DU Mobile Meow

Testing: Automation● Android Webdriver

○ Still no video support ● iOS Webdriver

○ Need to register as ios dev○ Intermittent issues with playback

● IE○ No webdriver APIs for mobile

Page 99: Web DU Mobile Meow

Testing: Automation● Chrome, Opera

○ Driver APIs built-in○ Remote debugging

● FFMobile○ No webdriver APIs

Page 100: Web DU Mobile Meow

Testing: Automation Strategy● Test API methods in mobile context first

○ cheap○ can approximate video playback by video.

currentTime

Page 101: Web DU Mobile Meow

Testing: Automation Strategy● More sophisticated tests to follow

○ screenreader● Screen cap processing

○ check for distortion○ playback accuracy

Page 102: Web DU Mobile Meow

Testing: Automation Strategy● Screen cap processing

Page 103: Web DU Mobile Meow

Testing: Hardware Assistance

Page 105: Web DU Mobile Meow
Page 106: Web DU Mobile Meow

Or

Page 107: Web DU Mobile Meow

PointerEventshttp://msdn.microsoft.com/en-us/library/ie/hh673557(v=vs.85).aspx

Page 108: Web DU Mobile Meow

http://imgs.xkcd.com/comics/in_ur_reality.png

Page 109: Web DU Mobile Meow

Track

<video> <track src="cats_meow" kind="subtitles"></video>

Page 112: Web DU Mobile Meow

Questions?

By cloudzilla http://www.flickr.com/photos/cloudzilla/378829651/

[email protected]@google.com

can haz question?