GStreamer base classes… or how to make your life easier

Introduction

Who Am I?

Sebastian Dröge <sebastian.droege@collabora.co.uk>• GStreamer developer since 2006• Working on GStreamer, GNOME and related projects• Working for Collabora Ltd.

About this talk

Why base classes?

✔ Solves common problems once

✔ Coherent behaviour of elements

✔ Faster results, less code

✔ Easier API✘ Different API (but we try to keep it coherent)

Why talk about this at all?!

Introduce all the important base classes

Explain their basic concepts

What is simplified and taken care of?

Also …

Developers still don't use them as oftenas they should ☹

Why? Possible reasons include

Still in gst-plugins-bad

(this is not a reason)

Developers don't know of their existence

Developers don't know if they suit their needs

Missing feature → do everything yourself ☹

Let's start

What's common in all base classes?

• open(), start(), stop(), close() vfuncs• set_caps() or set_format() vfunc• negotiation, state, event & query handling• Overriding of default behaviour with vfuncs

Now the specifics but let's stay media agnostic

First the (hopefully) well-known ones

GstBaseSrc + GstPushSrc• For single source pad sources• create() or alloc() and fill() vfuncs for providing buffers• Multi-source pad sources as GstBins or container format

GstBaseSink• For single sink pad sinks• render() vfunc• Handles synchronization, events, …• Multi-sink pad sinks as GstBins

GstBaseTransform• For 1:1 buffer transformations• transform_caps() and transform() / transform_ip() vfuncs• Simplifies (re-)negotiation and buffer handling a lot• GstAudioFilter and GstVideoFilter subclasses

GstCollectPads* (* not a base class)• Sink pad controller for N:1 elements• collected() function called whenever all pads have data• Simplifies event, state & data (waiting) handling

Now to the slightly newer, less-known ones

GstBaseParse• For parser elements• handle_frame() vfunc for parsing/skipping• finish_frame() function when frame complete• Handles push/pull mode, seeking, reverse playback, … ☺

GstTagDemux• For demuxers of tags at the start/end of a file• identify_tag() vfunc for detecting a tag• parse_tag() for parsing it• Transparent for downstream elements, can even pull

GstTagMux• Basically inverse to GstTagDemux• render_{start,end}_tag() vfuncs converting a GstTagList to a GstBuffer• Transparent for upstream elements

… but media agnostic is boring.Let's work with audio ♫

GstAudioSink & GstAudioSrc• write() or read() vfuncs• Implement a threaded ringbuffer (GstAudioRingBuffer) for storing audio• Up/downsampling for trick modes

And finally the brand new newcomers

GstAudioEncoder & GstAudioDecoder• handle_frame() vfunc + finish_frame() function• Simplify negotiation & raw caps handling a lot (GstAudioInfo)• Timestamp & segment handling• Decoder: minimal parsing functionality, reverse playback, …• Porting to base classes reduced LOC by 40%-60%

… and now something for the eyes. Video!

GstVideoEncoder & GstVideoDecoder• handle_frame() vfunc + finish_frame() function• Simplify negotiation & raw caps handling a lot (GstVideoInfo)• Codec & frame state tracking (GstVideoCodecFrame, GstVideoCodecState)• Handles frame reordering, timestamp & segments, …• Decoder: minimal parsing functionality, reverse playback, …• Porting to base classes reduced LOC by 40%-60%

Look into the future, what's missing?What's on the horizon?

First some useful base classes that alreadyexist in gst-plugins-bad

GstAudioVisualizer• For audio visualizers• Helpers for drawing and shading• Takes care of negotiation and everything

GstBaseCameraSrc• For camera sources inside camerabin• Allows usage of zoom, snapshot & preview features• camerabin wraps around this and provides a nice API

… and the following ones are just ideas so far

GstBaseDemuxer• Similar to GstBaseParse on the input side• Tracking of multiple streams with a stream object• Should reduce demuxer LOC a lot• Segment handling, seeking, reverse playback, …• Lots of copy & pasted code could be removed• Many small behaviour differences disappear

GstBaseMuxer• On top of GstCollectPads• Tracking multiple streams and their information• Handling of rewriting of headers, bitrate/duration estimations, …

GstVideoSubOverlay• For elements rendering sparse streams on top of video• Synchronizing/waiting for sparse stream buffers• Segment handling• Complex code currently duplicated in many elements

Want to help?Need one of these?

Need help with existing base classes?Just talk to us!

¿Questions?

Feel free to contact meMail: sebastian.droege@collabora.co.ukIRC: slomo on Freenode and others

Thanks to Vivia Nikolaidou for suggestions andideas, Reynaldo Verdejo for the pinpoint template,memegenerator.net, memebase, etc. for the pictures