linux stateless video decoder support · 2020. 6. 30. · acc. 5 v4l2 m2m output vpu capture...
TRANSCRIPT
![Page 1: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/1.jpg)
Open FirstOpen First
Linux StatelessVideo Decoder Support Nicolas DufresnePrincipal Software Engineer
![Page 2: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/2.jpg)
2
About me
● Over 10 years at Collabora
● Core GStreamer developer
● Contribute to Linux Media
![Page 3: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/3.jpg)
3
The beginning of Linux CODECs
● Google partners with Samsung and Asus
● Produce the first ARM Chromebook
● Based on Exynos 5 SoC
● Includes Samsung MFC Decoder
● MFC V4L2 M2M driver landed mainline
2011
![Page 4: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/4.jpg)
4
State-FullProcessor
ACC
ACC
ACC
![Page 5: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/5.jpg)
5
V4L2 M2M
Output CaptureVPU
/dev/videoX
● A V4L2 output queue is used for the bitstream
● A V4L2 capture queue is used for the decoded pictures
● Additional control flow are added to support draining, flushing
● Inter-queue configuration flow is needed
![Page 6: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/6.jpg)
6
● Minimal per CODEC code needed
● Requires a firmware
● Harder to multiplex
Pros Cons
![Page 7: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/7.jpg)
7
● CODA driver was added
– Design from Chips&Media
– CODA Hx4 and 960 support
● Enabling i.MX51 and i.MX6
● Reversed engineering
2014
CODA Driver
![Page 8: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/8.jpg)
8
The beginning of Linux State-Less CODECs
● Google partners with Rockchip
● 2nd gen of ARM Chromebook
● New type of CODEC hardware
● Rockchip VDPU ?
2015
![Page 9: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/9.jpg)
9
State-Less
ACC
ACC
ACCReference(s)
Bitstream
Parameters
Picture
![Page 10: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/10.jpg)
10
GPU
● Registers are replace with a command as part of the command stream
● Crafting command stream is HW specific and is only implement in userspace drivers (Mesa)
● Commands are scheduled by the GPU driver
● Exposing VPU in Mesa is done through standards API (VA API, VDPAU, DXVA2, NVDEC), but without a GPU, using these APIs can be cumbersome and overkill
● Using multiple GPU hardware in the same application remains tedious
![Page 11: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/11.jpg)
11
V4L2 M2M+
Request
Output CaptureVPU
/dev/videoX
● Per-frame (or slice) controls are associated with bitstream buffers using requests
● References are signaled using user defined timestamps
● Better identification can be done using the Media Controller topology
/dev/mediaX
Request
CODEC Controls
Topology
![Page 12: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/12.jpg)
12
H.264
SPS PPS IDR Slice P Slice B Slice
NALU Sequence
Start-Code
(0x000001)
HDR Payload
Annex B NALU
Size HDR Payload
AVCc NALU
![Page 13: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/13.jpg)
13
Decoding process
● Locate and parse NAL headers● Parse non-VLC and VCL NAL (SPS, PPS, Slice Headers)● Calculate frame_num (and handle gaps)● Calculate POC and pic_num● (Sliced Base) prepare reference lists● ….
![Page 14: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/14.jpg)
14
Decoding process (continued)
● Fill SPS / PPS, Decode parameters, Slice params V4L2 structures
● (Slice Based) Modify reference lists● Decode the slices/frame● Do DPB management as per spec● Output frames that could be re-ordered
![Page 15: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/15.jpg)
15
V4L2 Specific Process
● Allocate a Request (an FD)● Set per-frame/slice params for this request● Queue a v4l2_buffer for the request● Queue the request● Poll the request FD for completion
![Page 16: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/16.jpg)
16
2016
MediaTek VPU
● State-full H.264, VP8 and VP9 decode
● Tiled output only (requires HW converter)
![Page 17: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/17.jpg)
17
2017
Qualcomm Venus
● State-full MPEG4, MPEG2, H.264, VC1, H.264, VP8, VP9 and HEVC decode
● MPEG4, H.263, H.264, VP8, HEVC
![Page 18: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/18.jpg)
18
Upstreaming Stalled
● Could not settle on the Request / Job API
● Low knowledge of CODEC decoding process by the linux-media maintainers
● Only one hardware to test the API design
● No formal specification (not that state-full CODEC had any either)
![Page 19: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/19.jpg)
19
2018
Allwinner VPU support Kickstarter by Bootlin
● Request API is finalized
● MPEG2 Support landed in Staging
● H.264 support was progressing (but only sliced based)
● Reversed engineered from binary userspace blob
● VAAPI userspace drivers
![Page 20: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/20.jpg)
20
2019
● The crowd funding had gain good momentum
● A formal specification was merged
● H264, VP8, HEVC uAPI added as staging control API
● Cedrus gained H264 and HEVC support
● RK3288 driver was mainlined (MPEG2, H264, VP8)
● RK3288 driver was renamed !?!
![Page 21: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/21.jpg)
21
The HantroDriver
● I.MX8M Quad, using Hantro G1/G2
● Registry compatible with the RK3288
![Page 22: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/22.jpg)
22
Hantro Company
![Page 23: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/23.jpg)
23
STMMAC
● `stmmac` is an ethernet driver, that was thought to be an STM design
● It was later found to be DesignWare design, shared across numerous SoC
● Still called `stmmac` to maintain kernel API stability
![Page 24: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/24.jpg)
24
Testing andFixing
● FFMPEG support
● Bug fixing
● Interlaced Content Support
![Page 25: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/25.jpg)
25
2020
● RK3399 JPEG, MPEG2, H.264 and VP9 support landed
● GStreamer gained base classes for state-less CODEC, with already merge DXVA2 and NVDEC support
● GStreamer H.264 and VP8 V4L2 support landed
● Embedded World Conference 2020 canceled
● VA V4L2 Request driver was abandoned
● And a lot more coming …
● (at 2m distance of course)
![Page 26: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/26.jpg)
26
![Page 27: Linux Stateless Video Decoder Support · 2020. 6. 30. · ACC. 5 V4L2 M2M Output VPU Capture /dev/videoX ... ACC Reference(s) Bitstream Parameters Pictur e. 10 GPU](https://reader036.vdocuments.site/reader036/viewer/2022071405/60f9f13113a39b13166c110d/html5/thumbnails/27.jpg)
27