timeshift everything, miss nothing - mashup your pvr with kamaelia

Timeshift Everything, Miss Nothing

Mash up your PVR with Kamaelia

Euro OSCON 2006, Brussels

Michael Sparks,
Senior Research Engineer, BBC Research

Prologue

Time shifting...

Enshrined as a right in UK Law

17A. - (1) The making in domestic premises for private and domestic use of a recording of a broadcast solely for the purpose of enabling it to be viewed or listened to at a more convenient time does not infringe any right conferred by Part 2 in relation to a performance or recording included in the broadcast.

Time shifting...

Enshrined as a right in UK Law

17A. - (1) The making in domestic premises for private and domestic use of a recording of a broadcast solely for the purpose of enabling it to be viewed or listened to at a more convenient time does not infringe any right conferred by Part 2 in relation to a performance or recording included in the broadcast.

TiVO*, Sky+, Goodmans, Grundig, Humax, Sagem, Topfield, Sharp, Sony, Thomson, Technosonic, etc...

* not sold in UK anymore

Kamaelia Macro

Kamaelia Macro

It records and transcodes what is broadcast over DTT for future viewing.

(video)

How do you build this?

DVB MultiplexKamaelia Macro: Target

outbox Channel Transcoderinbox DVB Demuxer

BBC_ONEinbox Channel Transcoderinbox Channel Transcoderinbox Channel Transcoder Channel Transcoder Channel TranscoderinboxinboxinboxBBC_TWOBBC_THREEBBC_FOURCBEEBIESCBBC

DVB MultiplexKamaelia Macro: Status



DVB MultiplexKamaelia Macro: Status



Scaling up simplyawaiting hardware

DVB MultiplexTop Level Schematic


BBC_ONEinbox

Channel Transcoder EITDemuxChannel Transcoder Schematic

inbox_eit_inbox EITParsingoutboxinbox Carousel

transcoder_factoryinboxnextoutbox

EIT ParsingPSI PacketReconstructorProgramme Identification Schematic

inboxinbox EIT PacketParseroutboxinbox

Now Nextservice filter Now NextChangesoutboxinboxoutboxinboxoutboxoutbox

Programme TranscoderProgramme Transcoding Schematic

(a pipeline)

Pipethrough(mencoder -o file options...)

inboxcontrolsignalinboxcontrol_transcodingcomplete_stopsignal

But code isn't pretty web 2.0 esque boxes gradiented , it's never like that...

End of Prologue

Aims of Kamaelia

Aims of this Talk

To stave off doom from buggy software on parallel machines

Aims of this Talk

ahem

Aims of this Talk

(nb, that's something that concerns me,Kamaelia is partly aimed at staving that off,I'm not claiming it does that but it is oneapproach we're finding useful)

Aims of this Talk

Understand what Kamaelia is

Aims of this Talk


Take away our research results, and assimilate our code or approach in your systems

Aims of this Talk



To make it clear concurrency can be much simpler if you have the right tools

Aims of this Talk




Can be simpler than single threaded code

Aims of this Talk




To encourage you to think about using Kamaelia, and collaborating with us

Learning More: Kamaelia Open Space

What: kinda like a project's Foo/Bar Camp

Why: To help you get started, share knowledge etc,

When: This Friday 22nd Sept, 11am-5pm

Where: Here, Brussels, The FoAM Offices

Koolmijnenkaai 30-34, 1080 Brussels

Details: http://tinyurl.com/f8szz

Map: http://tinyurl.com/hsarl

Aims of this Talk

To share an approach we find fun and useful

What ?

What ?

(demo)

What you've just seen.

ComponentWhat you've just seen

outboxinbox


outboxinbox Componentoutboxinbox





Componentoutboxinbox




What you've just seen

Componentoutboxinbox Componentoutboxinbox












Componentinbox Component Componentoutboxoutbox | inboxoutbox | inbox


Componentinbox Component Componentoutboxoutbox | inboxoutbox | inbox


Componentinbox Component Componentoutboxoutbox | inboxoutbox | inboxWhat's inside the box?

Python Generators

Python Generators

Fibonacci Demo

Component Basics

Component Basics

Rotating Angle Demo

turning a generator into a component

Component Basics

What use is this?

class RotatingAngle(Axon.Component.component): def main(self): while 1: self.send(x, "outbox") x = x + 1 if x > 359: x = 0 yield 1

Component Basics

Consider these components:

class loopingCounter(Axon.Component.component) class cartesianPingPong(Axon.Component.component) class bouncingfloat(Axon.Component.component) class continuousIdentity(Axon.Component.component) class continuousZero(Axon.Component.component) class continuousOne(Axon.Component.component) (from Kamaelia.Automata.Behaviours...)

Component Basics

Bouncing Cats

Component Basics

Graphline( CAT = BasicSprite(image=cat), ROTATOR = loopingCounter(rotation_speed), MOVER = cartesianPingPong(position, 800, 600, 40), SCALER = bouncingFloat(scale_speed), IMAGER = continuousIdentity(cat), linkages = { ("ROTATOR","outbox" ) : ("CAT", "rotator"), ("MOVER","outbox" ) : ("CAT", "translation"), ("SCALER","outbox" ) : ("CAT", "scaler"), ("IMAGER","outbox" ) : ("CAT", "imaging"), } ).activate()

From Examples/SimpleGraphicalApps/BouncingCatGame/Simplegame.py(shutdown fanout removed for space)

What's it good for?

Some stuff we've done

Networked Audio Mixer MatrixDirac/Vorbis distribution/decode/playersSubtitling distribution server (mobiles, PCs)Presentation ToolsCollaborative Whiteboards * with audioSimple gamesVisual programming toolsVisual Introspection toolsKamaelia Macro...

Stuff trainees have done

pre-university trainee:

Previewing PVR content on a mobile phone Shakespeare script parsing and doing something fun with events & data


pre-university trainee:

Previewing PVR content on a mobile phone Shakespeare script parsing and doing something fun with events & data

Later adapted simply to have the characters talk
the play on an IRC channel


Summer trainee: (2nd year student)

Tools for simple reliable multicastMulticast island joining

Google Summer of Code Students

Ryan Lothian

Thomas Flanitzer

Devendra Laulkar

Anagha Mudigonda

Google Summer of Code

For us SoC was an opportunity to have components implemented that we've wanted for some time

Students implementing them appear to have had fun.

Also they've been guinea pigs to test our ease of use hypothesis

Large code chunks in our 0.5.0 release


For us SoC was an opportunity to have components implemented that we've wanted for some time

Students implementing them appear to have had fun.

Also they've been guinea pigs to test our ease of use hypothesis

Large code chunks in our 0.5.0 release

Hopefully uploaded to sourceforge today :-)


Components for:

Bit Torrent integration, a web server, web clients, IRC clients, icecast clients tone generation

3D systems widgets, pygame component wrapping, 3D scribbling, simple image viewer

secure communications including a nascent (choppy) secure phone.

prototype trusted communications tools

eg reporter trusting his comms to the BBC won't be changed without detection

Ease of Use

Now have 6 data points for ease of use

The youngest, most nave students have produced the most complex and interesting systems

The older ones have generally had more baggage, and unless used to thinking about X talking to Y, to Z, have had more problems.

Even our weakest SoC student has produced interesting data

Stuff other parts of the BBC have done

Radio & Music Interactive:

Prototyped a system for recording all BBC radio output, transcoding it and making podcasts from the resulting data.




Bootstrapped the final system rapidly




Bootstrapped the final system rapidly

Created the system without any involvement from us




This directly led on to the Macro workas a result, including optimisations

Kamaelia Macro: Revisited

What Macro looks like inside

(yes, it really is boxes)

DVB MultiplexTop Level Schematic


BBC_ONEinbox

Top Level Code

Graphline( SOURCE=DVB_Multiplex(freq, pids["BBC ONE"]+pids["EIT"], feparams), DEMUX=DVB_Demuxer({ 610: ["BBCONE"], 611: ["BBCONE"], 18: ["BBCONE"], }), BBCONE_HI = ChannelTranscoder(service_ids["BBC ONE"], **params["HI"]), linkages={ ("SOURCE", "outbox"):("DEMUX","inbox"), ("DEMUX", "BBCONE"): ("BBCONE_HI", "inbox"), }).run()

Channel Transcoder EITDemuxChannel Transcoder Schematic

inbox_eit_inbox EITParsingoutboxinbox Carousel

transcoder_factoryinboxnextoutbox

Channel Transcoder Code

def ChannelTranscoder(service_id, mencoder_options, dir_prefix):

def transcoder_factory(eit): return ProgrammeTranscoder(eit, mencoder_options, dir_prefix)

return Graphline( PROG_CODER = Carousel( transcoder_factory ), EIT_PARSE = EITParsing(service_id), DEMUX = EITDemux(), linkages = { ("self","inbox") : ("DEMUX","inbox"), ("DEMUX","outbox") : ("PROG_CODER","inbox"), ("DEMUX","_eit_") : ("EIT_PARSE", "inbox"), ("EIT_PARSE", "outbox") : ("PROG_CODER", "next"), } )

EIT ParsingPSI PacketReconstructorProgramme Identification Schematic

inboxinbox EIT PacketParseroutboxinbox

Now Nextservice filter Now NextChangesoutboxinboxoutboxinboxoutboxoutbox

Programme Identification Code

def EITParsing(*service_ids): return pipeline( PSIPacketReconstructor(), EITPacketParser(), NowNextServiceFilter(*service_ids), NowNextChanges(), )

Programme TranscoderProgramme Transcoding Schematic

(a pipeline)

Pipethrough(mencoder -o file options...)

inboxcontrolsignalinboxcontrol_transcodingcomplete_stopsignal

Programme Transcoding Code

class ProgrammeTranscoder(Axon.Component.component): Inboxes = { "inbox" : "TS packets containing audio and video", "control" : "the usual", "_transcodingcomplete" : "signal from the transcoder", } Outboxes = { "outbox" : "", "_stop" : "stop the transcoder", "signal" : "the usual", } def __init__(self, eitdata, mencoder_options, dir_prefix): super(ProgrammeTranscoder,self).__init__() self.eitdata = eitdata self.mencoder_options =mencoder_options self.dir_prefix = dir_prefix def main(self): # first, generate unique filename uid = str(time.time()) encodingfile = "/data/encoding"+self.dir_prefix+"/"+uid+".avi" waitingEIT = "/data/encoding"+self.dir_prefix+"/"+uid+".eit" finishedfile = "/data/finished"+self.dir_prefix+"/"+uid+".avi" finishedEIT = "/data/finished"+self.dir_prefix+"/"+uid+".eit" print uid,"Starting transcoding into: "+encodingfile transcoder = Pipethrough("mencoder -o "+encodingfile+" "+self.mencoder_options) print uid,"Transcoder pipethough =",transcoder.name data_linkage = self.link( (self,"inbox"), (transcoder,"inbox"), passthrough=1 ) ctrl_linkage = self.link( (self,"_stop"), (transcoder,"control")) done_linkage = self.link( (transcoder,"signal"), (self,"_transcodingcomplete") )

Kamaelia Macro

Code is in the distribution is you want to play

(only works under Linux due to Linux having a uniform DVB-T API)

But Why Mashup?

A web mashup often takes data from multiple sources (websites) transforms them and republishes them somehow, allowing recomposition.

Kamaelia Components are essentially the same, except you're not mashupping websites but systems, including potentially your PVR.

It's just data.

Kamaelia: Ideas

Things you might want to do (to Macro and beyond)

3D user interface to the PVR

IRC based control (your PVR sitting on

an IRC channel)

Web integration (beyond blogs/rss)

Integrate common sense engine

An open alternative to Second Life

Video Conferencing

Build an open technology for BBC iPlayer?

Kamaelia Futures

We want to replace the need to write code when writing components.

Further optimisations

More concurrency mechanisms

More tutorials

(AKA Specfic stuff we want to do)

Kamaelia Futures

Collaborative Storyboarding/programme production tools

Open source iPlayer

Integrate PGU

Better Tk Components

PyQT, wxWindows integration

(AKA Specfic stuff we want to do)

Thank you!

Thank you!

Kamaelia Open Space

This Friday 22nd Sept, 11am-5pm

FoAM's Offices:

Koolmijnenkaai 30-34, 1080 Brussels

Map: http://tinyurl.com/hsarl

Possible discussions: Get started; integrate your favourite libraries; apply to teaching (gesture recognition); flesh out our system to be more relevant to your problem domain (as we do for the BBC)? Re-implement in C++, Ruby?

http://tinyurl.com/f8szz

Click to edit the title text format

Click to edit the outline text format

Second Outline Level

Third Outline Level

Fourth Outline Level

Fifth Outline Level

Sixth Outline Level

Seventh Outline Level

Eighth Outline Level

Ninth Outline Level

Michael Sparks, [email protected], http://kamaelia.sf.net/

Kamaelia

timeshift everything, miss nothing - mashup your pvr with kamaelia

Technology