development of hardware-based elements for gstreamer 1.0: a case study (gstreamer conference 2012)
TRANSCRIPT
![Page 1: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/1.jpg)
Development of hardware-based elements
for GStreamer v1.0
Víctor Manuel Jáquez Leal
2012-08-27 Mon
![Page 2: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/2.jpg)
Once upon a time. . .
![Page 3: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/3.jpg)
Once upon a time. . .
![Page 4: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/4.jpg)
Yeah. . .
![Page 5: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/5.jpg)
Yeah. . .
![Page 6: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/6.jpg)
What I learned
Everything should be made as simple as possible,but not simpler.
� Einstein / Zukofsky / Sessions / Ockham / . . .
![Page 7: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/7.jpg)
Meanwhile in GUADEC 2012
Drop abstraction, make our layers thinDon't stack things anymore� Lennart Poettering
![Page 8: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/8.jpg)
Then I moved to Spain
![Page 9: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/9.jpg)
The love is still in the air. . .
![Page 10: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/10.jpg)
GStreamer v0.10
I Workarounds
I Reinventing the wheel
![Page 11: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/11.jpg)
What's AMP?
Asymmetric MultiProcessing
![Page 12: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/12.jpg)
What's IPC?
Inter-Processors Communication
![Page 13: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/13.jpg)
What do I want?
![Page 14: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/14.jpg)
Wat!?
I A simple,
I bare bones,
I home brew
video player
![Page 15: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/15.jpg)
What do I need?
I A simple video sink
I A video decoder which uses IVAHD
![Page 16: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/16.jpg)
Let's go
![Page 17: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/17.jpg)
Video Sink
No frame bu�erBut there is DRM/KMS
![Page 18: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/18.jpg)
Direct Rendering Management
![Page 19: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/19.jpg)
Kernel Mode Settings
I framebu�er
I CRTC
I Overlay
I Encoder
I Connector
![Page 20: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/20.jpg)
Kernel Mode Settings
![Page 21: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/21.jpg)
Video Decoders
SysLink 2 is deprecatedomapdcee
![Page 22: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/22.jpg)
Distributed Codec Engine (DCE)
I It provides access to the Codec Engine interface on theco-processor (ducati/M3) from the host.
I It invokes the hardware accelerated codecs on IVA-HD viarpmsg/remoteproc
![Page 23: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/23.jpg)
Ducati
![Page 24: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/24.jpg)
omapdce
![Page 25: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/25.jpg)
History of the TI's IPCs?
I dsp-gateway - OMAP1/OMAP2
I DSPLink - OMAP2/OMAP3/DaVinci
I DSP/Bridge - OMAP3
I SysLink2 - OMAP4
I RPMsg - OMAP4/OMAP5
![Page 26: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/26.jpg)
RPMsg
I remoteproc
I rpmsg
![Page 27: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/27.jpg)
remoteproc
I Loading & Bootup
I Power Management
I Exception Management
I Error Recovery
![Page 28: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/28.jpg)
rpmsg
I virtio-based messaging bus that allows kernel drivers tocommunicate with remote processors
![Page 29: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/29.jpg)
GStreamer v1.0
HIC SVNT DRACONES
![Page 30: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/30.jpg)
GstBu�er
I gst_buffer_map()
I gst_buffer_unmap()
I Bu�ers can contain a list of GstMemory objects
![Page 31: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/31.jpg)
GstMemory
I It is a lightweight & refcounted object.
I It wraps a region of memory.
I Memory is usually created by allocators.
![Page 32: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/32.jpg)
GstKMSMemory
struct _GstKMSMemory {
GstMemory mem;
uint32_t fb_id;
uint32_t handle[4];
void *bo[4];
};
![Page 33: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/33.jpg)
GstAllocator
I Allocate memory blocks
I Virtual methods
I allocI free
I Callbacks
I mapI unmapI copyI share
I Parameters
I alignI pre�xI padding
![Page 34: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/34.jpg)
GstKMSAllocator (alloc)
static GstMemory *
gst_kms_allcator_alloc (GstAllocator * allocator,
gsize size,
GstAllocationParams * params)
{
GstKMSAllocator *self = GST_KMS_ALLOCATOR (allocator);
GstKMSMemory *buf = g_slice_alloc (sizeof (GstKMSMemory));
buf->fb_id = -1;
buf->bo[0] = omap_bo_new (self->dev, size,
OMAP_BO_SCANOUT | OMAP_BO_WC);
buf->handle[0] = omap_bo_handle (buf->bo[0]);
gst_memory_init (GST_MEMORY_CAST (buf),
GST_MEMORY_FLAG_NO_SHARE, allocator, NULL, size,
0, 0, size);
return GST_MEMORY_CAST (buf);
}
![Page 35: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/35.jpg)
GstKMSAllocator (free)
static void
gst_kms_allocator_free (GstAllocator * self,
GstMemory * mem)
{
GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem);
omap_bo_del (buf->bo[0]);
g_slice_free1 (sizeof (GstKMSMemory), mem);
}
![Page 36: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/36.jpg)
GstKMSAllocator (map / unmap)
static void *
gst_kms_allocator_map (GstMemory * mem,
gsize maxsize,
GstMapFlags flags)
{
GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem);
omap_bo_cpu_prep (buf->bo[0], OMAP_GEM_WRITE);
return omap_bo_map (buf->bo[0]);
}
static void
gst_kms_allcator_unmap (GstMemory * mem)
{
GstKMSMemory *buf = GST_MEMORY_CAST_KMS (mem);
omap_bo_cpu_fini (buf->bo[0], OMAP_GEM_WRITE);
}
![Page 37: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/37.jpg)
GstMeta
I Describes the low-level properties of the bu�er content.
I It contain variables that can change between each bu�er
I timestamp, duration, o�set, interlacingI video alignment, cropping, panning informationI extra container informationI global bu�er properties
![Page 38: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/38.jpg)
GstBu�erPool
I Pre-allocate and recycle bu�ers with the same properties.
I After the bu�er pool is created, it needs to be con�gured.
I Bu�er pools increase performance
I reduces allocation overheadI add more possibilities to implement zero-copy memory
![Page 39: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/39.jpg)
GstVideoBu�erPool
I GstBu�erPool + Video Meta + Video Alignment
![Page 40: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/40.jpg)
GstVideoContext
I It shares video context between neighbour elements andthe application.
I gst-plugins-bad / GST_USE_UNSTABLE_API
I Sharing the DRM �le descriptor between the decoder andthe sink.
![Page 41: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/41.jpg)
gst_kms_sink_query (GstBaseSink * bsink,
GstQuery * query){
GstKMSSink *sink = GST_KMS_SINK (bsink);
const char **types =
gst_video_context_query_get_supported_types (query);
if (!types)
return GST_BASE_SINK_CLASS (parent_class)->query (bsink, query);
for (i = 0; types[i]; i++) {
if (!strcmp (types[i], "drm-fd")) {
GstStructure *structure =
gst_query_writable_structure (query);
gst_structure_set (structure, "video-context-type",
G_TYPE_STRING, "drm-fd", "video-context",
G_TYPE_INT, sink->fd, NULL);
}
}
return TRUE;
}
![Page 42: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/42.jpg)
GstVideoDecoder
I Base class fore converters of encoded data into raw videoframes.
I Con�guration.
I _start ()I _set_format ()
I Data processing.
I get or _parse () framesI _handle_frame ()I call gst_video_decoder_finish_frame () or
gst_video_decoder_drop_frame ()
I Shutdown phase.
I _stop ()
![Page 43: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/43.jpg)
@TODO
I kmssink should support more DRM backends
I dcevdec should support more decoders
I Implement a DRM bu�er metadata
I Push kmssink and dcevdec (??) to bugzilla
I Watch a movie with my video player
![Page 44: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/44.jpg)
Lessons I learned
I I suck at communication
I Communicate your development openly
I GStreamer v1.0 looks great!
I omapdce/rpmsg + remoteproc too!
![Page 45: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/45.jpg)
Thanks to
I Rob Clark
I Ohad Ben-Cohen
I Nicolas Dechesne
I Alessandro Decina
I Sebastian Dröge
I GStreamer community
![Page 46: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/46.jpg)
Credits
Clear Inner Vision - CC BY-NC-ND 2.0The 1265 Psalter world mapRage Comics
![Page 47: Development of hardware-based Elements for GStreamer 1.0: A case study (GStreamer Conference 2012)](https://reader038.vdocuments.site/reader038/viewer/2022102618/55a28de81a28ab1b588b4667/html5/thumbnails/47.jpg)
Questions?