WebKit, HTML5 mediaand GStreamer onmultiple platforms

Spreading GStreamerawesome in WebKit

Philippe Normand

Talk outlineAbout WebKitHTML5 Media playback in WebKit withGStreamerWebKit/GStreamer: opening the door to moreplatformsEnd goal: spread the usage of GStreameron more and more WebKit-basedapplications!

About WebKitA content rendering/editing engineNot a browser!Highly standards-compliant

WebKit EcosystemApple, Nokia, Google, RIM, Adobe, Sony-EriccsonSamsung, Palm, Motorola, Electronic Arts, ...=> lots of devices using WebKit in the market=> wide range of applications using thevarious WebKit ports

High level WebKitArchitecture

JavaScriptCore: Javascript engine andplatform-specific bitsWebCore: platform-independant blocks andtheir counter part platform implementationsfor:

parsing, layoutingnetwork, paintingmedia playback...

WebKit: platform-specific APIs => used byapplication developersPorts: GTK+, Qt, EFL, Chromium, Mac,WinCairo, Apple Windows, ...

HTML5 Mediaplayback in WebKit

3 layers: DOM/HTML level(HTMLMediaElement), WebCore/graphics(MediaPlayer), platform-dependantimplementation (MediaPlayerPrivate)Some MediaPlayerPrivate implementations:

GStreamer: GTK+ and EFL portsQt/Phonon: Qt portQuicktime: Apple Mac/Win ports

GStreamer mediaplayer

playbin2 of course!Custom video sink and source elementsBasic trick-modes supportOn-disk bufferingFullscreen video display

The GStreamer playerin other WebKit ports?

GStreamer cross-platform already :)

the MediaPlayer layer allows multiple Privateinstances

choice based on media mime-type andcodecs:

<video src=test.ogv type="video/ogg; codecs=&quot;theora,vorbis&qu

Targets: Mac and WinCairo ports

Porting check-listGStreamer platform packagesGLib main context integrationwebkit src elementwebkit video sinkFullscreen supportWebKit build :)

GStreamer platformpackages

on Windows: OSSBuildon Mac OSX: OSSBuild too! soon!For now on Mac: scripts developed based ongtk-osx-build

GStreamer build onMac

jhbuild to the rescueadd -headerpad_max_install_names toLDFLAGSuse install_name_tool to change theshared library paths in the various .dylib filesabsolute paths -> relative paths to@loader_path

GLib main contextintegration

GstBus signal watch used in the player =>need periodic calls ofg_main_context_iteration()on Mac: NSRunLoop observer triggering theglib main context iterationson Windows: integration in the message loop

WebKit src elementBased on appsrc, work by Sebastian Dröge!using WebCore Resource loader (=> HTTPsession data: cookies, referer)Contains some platform-specific code topause/resume data downloadFor now: #ifdefs :( Better approach: useWebCore resource defersLoading API.

Video paintingvideo sink exports buffers to the player("repaint-requested" signal)ImageGStreamer: GstBuffer ->WebCore::Imageactual painting deferred to theImageGStreamer implementations:

ImageGStreamerCairo: used by GTK+ andWinCairo portsImageGStreamerCG: CoreGraphicsimplementation for the Mac port

Fullscreen support(Overview)

webkitEnterFullscreen() VideoElement DOM API=> WebKit WebView private API=> => FullscreenVideoController

enterFullscreen() hooks intoMediaPlayer::platformMedia()MediaPlayerPrivate-specifics: to create awidget where the video is paintedController gets the widget, adds controls,goes fullscreen

Fullscreen support(GStreamer side)

Our playbin2 video-sink is a bin!

fullscreen off:

tee name=t t. ! queue ! webkitvideosink

fullscreen on:

tee name=t t. ! queue ! webkitvideosink t. ! queue !ffmpegcolorspace ! videoscale ! autovideosink

Fullscreen support(Video overlay)

GstXOverlayPlatformVideoWindow : window on whichautovideosink overlays the video

PlatformVideoWindowGtkPlatformVideoWindowEFLPlatformVideoWindowCocoa...

WebKit buildmodifications

(note: WTF here means Web TemplateFramework, obviously!)add JavaScriptCore/wtf/gobject/ inyour buildaddWebCore/platform/graphics/gstreamer/in your buildenable WTF_USE_GSTREAMER andENABLE_GLIB_SUPPORT definesport-specific FullscreenVideoControlleradaptations

Implementationstatus : Mac OSX

Most of the patches merged upstreamalready, excepted:

XCode config additionsgst webkit src patchPlatformVideoWindowCocoaimplementationFullscreenVideoController patch

Packaging: WebKit-nightly DMG includingGStreamer and its dependencies.

Implementationstatus : WinCairo

HTML5 media implemented from scratch forWinCairo:

WinCairo still using Visual Studio 5 => bigblocker :(Patches in a separate public git repository

Basic media controlsFullscreen video

Recap!1. GStreamer build2. Periodic calls to

g_main_context_iteration()3. Patching gst webkit src element and/or your

ResourceHandle implementation4. ImageGStreamerMyPort implementation5. PlatformVideoWindowMyPort

implementation6. Patching your WebKit

FullscreenVideoController7. Build WebKit!

Some bit of adviceUse playbin2 :-)Facade design-pattern for your app platform-specific bitsSingle build-system == less maintenanceburden (gyp, cmake, ...)Continuous integration on multiple platforms:BuildBot

DemoWebM playback, in Safari! With GStreamer! :-)

Future plansFinish upstream merge of the patchesWebKit/GStreamer build slavesFullscreen video improvementsWebKitGTK+ API to allow fine-grained controlon the player like:

per-webview volume controlcodec installer

Luke, may the sourcebe with yougst-mac and win-gst WebKit branches:

http://git.igalia.com/cgi-bin/gitweb.cgi?p=webkit.githttp://trac.webkit.org/wiki/GStreamerOnMac

That would be all,folks

Work sponsored by:IgaliaR&D funds from Galicia governmenta private Igalia client

Contact:http://igalia.compnormand@igalia.comIRC: philn-tp on #gstreamer, #webkit and#webkit-gtk

Questions?