developing applications with kurento

Open APIs for Open Minds

Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento

Mul$media infrastructures for the Future Internet

2

Media is here Media got

there

Media got there

Analyze Transform

Store

Transport

Enrich Augment Adapt

Sensors Context

Events Media is here

Enrich, augment, adapt, analyze, transform, store: what’s the problem?

3

Complexity

Future Internet

Mul$media Infrastructure

Simple Development

APIs

The FI-‐WARE Stream-‐oriented Generic Enabler Implementa$on

4

5

• Interoperable media exchange (mul$plaNorm/mul$protocol) • WebRTC, RTP, HTTP (video tag), etc.

• Process media (Computer vision, augmented reality, media indexing, etc.) • Media and metadata recording and recovery • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) • Media rou$ng and mixing • Etc.

Provides mul$media capabili$es to the FI-‐WARE infrastructure

• REST API •  JavaScript API •  Java API

Exposes those capabili$es through a simple to use APIs

•  LGPL 2.1

Is distributed through a flexible FOSS license

Kurento

Kurento Media Server (KMS): the nucleus of Kurento

6

•  KMS is a middleware for media streams – Receives the stream – Process the stream –  Issues the stream

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

KMS

The Media API: The API for accessing KMS capabili$es

7

Send Receive Analyze Augment Enrich

Transform Transcode Record Process Replicate

Media Source

Media Sink

KMS

Java Media API

JavaScript Media API

Other languages

Applica$ons define the processing of streams geang through KMS

Media API: Media Elements and Media Pipelines

8

Sink%

SRC% Sink%

SRC%

SRC%

Sink%

Sink%

§ Media Element • Provides a specific media

functionality › Send/receive media › Process media › Transform media

• Exchange media through ›  Sources ›  Sinks

§ Media pipeline • Chain of media elements

implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

Developers create applica$ons just connec$ng Media Elements

9

Protocols(&(Codecs(

Computer((Vision(

Augmented((Reality(

Mul9sensory((Mul9media(

Media((Repository(

IPTV(Integra9on(

Group(Communica9ons(

Sink%

H>pEndpoint(

Sink%

SRC%

RtpEndpoint(

SRC$

DataChannelEndpoint(

Sink%

SRC%

WebRtcEndpoint(

Sink%

SRC%

BarCodeReader(

Sink%

SRC%

PointerTracker(

Sink%

SRC%

FaceDetector(

Sink%

SRC%

FaceOverlay(

Sink%

SRC%

ChromaFilter(

Sink%

SRC%

HeartRateBlender(

Sink%

RecorderEndpoint(

SRC$

PlayerEndpoint(

Sink%

CdnUpload(

Sink%

IPTVConnector( Mixer(

Sink%

SRC% Sink%

SRC%

Sink%

SRC%

Sink%

Sink%

Applica$on 2

Sink%

SRC%

Sink%

SRC%

Sink%Sink%

SRC%

Sink%

Applica$on 3

Sink%

SRC%

SRC%

Sink%

SRC% Sink%

Sink%SRC%

Applica$on 1

Toolbox of media elements

Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build(); H2pGetEndpoint hepEndpoint = mp.newHepGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(hepEndpoint); hepEndpoint.getUrl(); //URL where the media is made available

10

Media Pipeline

HepGetEndpoint

Media from file or URI

HTTP media streaming Si

nk

SRC

PlayerEndpoint

Is that enough? Think about the WWW development model

11

Process WWW request -‐  DDBB access -‐  Authen$ca$on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

•  Intui$on behind the WWW model – Client asks what it wants – Server side APIs execute the associated processing

We need an equivalent model: the role of the Signaling Plane

•  The API must provide nego$a$on capabili$es –  I want “this media” …

•  Iden$fica$on of the media to exchange –  File in hard-‐drive, IP camera, user, etc.

–  in “this way” … •  Iden$fica$on of the processing of media

–  Augmented, analyzed, etc.

–  with “this format” … •  Quality

–  Codec, screen-‐size, frame-‐rate, etc.

–  at “this moment” •  Stream control

–  Play, stop, start, pause, etc.

12

Don’t get it? think about WWW development again …

13

Process WWW request -‐  DDBB access -‐  Authen$ca$on -‐  XML processing -‐  Etc.

HTTP request: I want this resource

HTTP response: The resource

Process media request -‐  Media API -‐  DDBB access -‐  Authen$ca$on -‐  XML processing -‐  Etc.

Signaling request: I want this media

Signaling response: The media is here

Intui$on behind tradi$onal WWW

Applica$ons (Servlets, ASP, PHP,

Rails, etc.)

Intui$on behind Kurento

development APIs: Mul$media RTC is just another feature of your

applica$on

Dealing with the signaling: The Content Handler

14

KMS

Sink

SRC

Sink

SRC

Sink

SRC

Sink

Media API

REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script -‐  When receiving “this request”… -‐  execute “this logic” Developer can use the media API

Code building the media pipeline and execu$ng the applica$on logic the

developer wants

Kurento Architecture

15

Kurento Media Server (KMS)

Receive Video

Augmented Reality

Send Video

Computer Vision

Video Playing and Recording

Java EE compaJble container

HTTP Servlet

SIP Servlet

Web services

Kurento REST API

Specific handler implementa$ons

Signaling and WWW traffic

Med

ia

Med

ia

Signaling and WWW traffic

Media API DD.BB.

Kurento ApplicaJon Server (KAS)

Other java APIs.

Applica$on execu$on flow Client Code

Applica$on Server (KAS)

Media Server (KMS)

I want this media in this way … (JSON)

Commands reques$ng the crea$on of a pipeline

What you want is here … (JSON)

Media negoJaJon

phase

Media exchange phase

1

2

Specific applica$on logic at the server-‐side

(Content Handler)

Media pipeline creaJon

Media exchange between client and server

Content Handler: trivial example @H2pPlayerService(path = "/player”) public class MyPlayerHandler extends HepPlayerHandler { @Override public void onContentRequest(HepPlayerSession contentSession) {

//Create the pipeline for providing media through HTTP } @Override public void onContentStarted(HepPlayerSession contentSession) {

//Media started flowing, you can execute addi$onal ac$ons } @Override Public void onSessionTerminated(HepPlayerSession contentSenssion){

//Media exchange termianted, you can collect your resources }

Let’s develop with Kurento •  What you need

–  A Kurento instance •  You can install your own Kurento instance •  You can launch a Kurento instance at the FI-‐LAB

–  hep://lab.fi-‐ware.org

•  Geang help –  FI-‐WARE catalog entry

•  hep://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento –  Installa$on guide

•  heps://forge.fi-‐ware.org/plugins/mediawiki/wiki/fiware/index.php/StreamOriented_-‐_Installa$on_and_Administra$on_Guide

–  Developer guide •  heps://forge.fi-‐ware.org/plugins/mediawiki/wiki/fiware/index.php/

StreamOriented_-‐_User_and_Programmers_Guide –  Kurento web site

•  hep://www.kurento.org

18

Kurento Hello World: Playing a file

19

Media Pipeline

HepGetEndpoint

Media from file or URI


nk

SRC

PlayerEndpoint

Media API

REST API (Open API protocol)

Create the pipeline connec$ng: HepGetEndpoint PlayerEndpoint

I want “this media”

Playing a file: Handler code @HepPlayerService(path = "/player”) public class MyPlayerHandler extends HepPlayerHandler {

@Override public void onContentRequest(HepPlayerSession contentSession) throws Excep$on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAeribute("player", playerEndpoint); H2pGetEndpoint h2pEndpoint = mp.newH2pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h2pEndpoint); contentSession.start(hepEndpoint);

} @Override public void onContentStarted(HepPlayerSession contentSession) {

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAeribute("player"); playerEndpoint.play();

} }

20 Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerHandler.java

Playing a file: client code

21

<!DOCTYPE html> <html>

<head> <script src="./js/kws-‐content-‐api.js"></script> <script>

var conn; func$on start() {

var op$ons = { remoteVideoTag : "remoteVideo"

}; conn = new kwsContentApi.KwsContentPlayer("./player", op$ons);

} func$on terminate() {

conn.terminate(); }

</script> </head> <body>

<bueon onclick="start();">Start</bueon> <bueon onclick="terminate();">Terminate</bueon> <br /> <video id="remoteVideo" autoplay></video>

</body> </html>

Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/player.html

Media Pipeline

Adding Computer Vision

22

HepGetEndpoint Media from file or URI


nk

SRC

PlayerEndpoint

SRC

Sink

JackVaderFilter

Media API

REST API (Open API protocol)

Create the pipeline connec$ng: HepGetEndpoint, Filter and

PlayerEndpoint

I want “this media”

Adding Computer Vision: Handler code

23

@HepPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends HepPlayerHandler {

@Override public void onContentRequest(HepPlayerSession contentSession) throws Excep$on {

MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAeribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); H2pGetEndpoint hepEndpoint = mp.newHepGetEndpoint().terminateOnEOS().build(); filter.connect(hepEndpoint); playerEndpoint.connect(filter); contentSession.start(hepEndpoint); }

@Override public void onContentStarted(HepPlayerSession contentSession) {

PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAeribute("player"); playerEndpoint.play();

} }

Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java

Adding Computer Vision: Client code

24





}; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op$ons);


conn.terminate(); }



</body> </html>

Source: heps://github.com/Kurento/kmf-‐tutorial/blob/master/src/main/webapp/playerFilter.html

Adding Computer Vision: Client code

25





}; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op$ons);


conn.terminate(); }



</body> </html>


26


Thanks ! •  hep://fi-‐ppp.eu •  hep://fi-‐ware.org •  hep://ww.kurento.org •  Follow @Fiware on Twieer ! •  Follow @Kurentoms on Twieer !