developing an embedded video application on dual linux + fpga architecture

11 Avenue Marigny 13014 Marseille www.ciose.fr [email protected]

Developing an embedded video application on dual Linux + FPGA architecture 109/26/17

Developing an embedded video Developing an embedded video application on dual Linux + FPGA application on dual Linux + FPGA

architecturearchitecture

C. CharreyreC. [email protected]@ciose.fr

http://www.ciose.frhttp://www.ciose.frhttps://twitter.com/CIO_SysEmbhttps://twitter.com/CIO_SysEmbhttp://fr.slideshare.net/charreyrehttp://fr.slideshare.net/charreyre

http://www.ciose.fr/

https://twitter.com/CIO_SysEmb

License


Attribution-Noncommercial-Share Alike 4.0 International● You are free:

to Share - copy and redistribute the material in any medium or format

to Adapt - remix, transform, and build upon the material

The licensor cannot revoke these freedoms as long as you follow the license terms.

● Under the following conditions:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

NonCommercial — You may not use the material for commercial purposes.

ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

● No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

● License text : http://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

http://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

Small company dedicated to embedded solutions (16 persons)

Electronics and software developments

Embedded Linux expertise since 15 years

Open Embedded and Yocto expertise since 7 years

Headquarters in Saint-Etienne

Agency in Marseilles

Embedded Linux expert for Cap’tronic program


Presentation of CIO Systèmes Embarqués

Developing an embedded video application on dual Linux + FPGA architecture 4

Responsible of Embedded Linux technologies in CIO Systèmes Embarqués

Embedded Linux trainer

30 years in embedded and Unix / Linux world

Developer and maintainer of kernel, u-boot ports on Arm boards

Developer and maintainer of Yocto layers

09/26/17

Presentation of the speaker


The embedded video application

● High performance video acquisition, real time treatments and display

● Merge and synchronization of 2 images issued by 2 cameras● Deployed in a device with safety constraints related to video● Latency between real world and display < 200 ms● Video must be available 7 s after Power On● Device developed on an ARM architecture (CIO’s skills) →

embedded Linux software environment


Why use a FPGA ?

● Previous video applications developed on iMX.6 processor with gstreamer to manage video pipeline

● This architecture can’t be used here :– Safety certification problem on Linux– Real time performances (latency, jitter) ???

● High performance IP for FPGA dedicated to video applications available● We decided to share the application between Embedded Linux on ARM

processor and FPGA– Video processing is done by the FPGA– Configuration, errors monitoring and utilities done by Embedded

Linux distribution


The hardware platform

● The choosen hardware platform is a PicoZed board mounted on a custom base board developed by CIO

● PicoZed = System On Module based on the Xilinx Zynq®-7000 All Programmable (AP) SoC● Zynq 7000 offers a dual core ARM processor + FPGA in the same chip● ARM processor is called Processing System (PS)● FPGA is called Programmable Logic (PL)


Linux software environment

● PicoZed Linux environment proposed with 2 options :– Petalinux is the Linux environment proposed by Xilinx– Wind River Pulsar Linux, a Yocto based environment

● CIO has great experience of OpenEmbedded and Yocto tools :– Petalinux quickly eliminated as Yocto is a de facto standard in

Embedded Linux– Wind River Pulsar Linux evaluated :

● Rejected due to the presence of containerization in the solution. We did not want to loose time with this technology

– We finally selected meta-xilinx layer and assembled it with other layers to build our Embedded Linux environment


Linux software environment

● All necessary layers selected from OpenEmbedded Layer Index● Layer coherency assured by repo tool● Finally we have made a Yocto based build system tailored for the

embedded distribution● The embedded distribution is built by customizing Yocto recipes

and creating new ones for in house applications● A more complex Board Support Package due to interactions

between PS and PL (detailed later)


FPGA developments

● FPGA developments made with Xilinx Vivado tool (Eclipse based)


FPGA developments

● FPGA engineer designs its IPs inside Vivado● When the design is finished, the final VHDL code for the FPGA is

generated● But we don’t want to have 2 isolated parts : Embedded Linux

distribution & logic design. AXI interface allow us to communicate between PS and PL.– AXI stream : dedicated to stream data– AXI ACP : allow copy data to cache memory (close to the processor)– AXI light : allow exchange of some data– AXI HP : allow access to the DDR and store data like images without

bandwidth issues.


How can PS and PL interact ?

● It is possible to design hardware components in the PL that can be seen by the PS

● Potential use cases :– Add additional hardware resources for the embedded Linux

distribution, normally not available on PS → extend PS capabilities● Create an additional serial line

– Use dedicated hardware to communicate between PL and PS● Create GPIOs in the PL and make Embedded Linux use them to read / write

logic data from / to the VHDL application

– Allow dynamic customization of IPs● Perform a specific crop on an image which can be changed from Linux

(position, size).


Technically, how does it work ?

● We have seen– The application constraints– The selected hardware and software architecture– The tools

● But of does it work ?


Boot Mechanism

● PicoZed needs a First Stage Boot Loader (FSBL) before u-boot● FSBL code is generated by Vivado according to the design

– This code can be compiled inside Vivado or externally with and ARM toolchain

● FSBL then starts u-boot which in turn starts Linux● The FPGA code (bitstream) must be loaded by FSBL or by u-boot

– The FPGA can’t start alone– The VHDL code is reloaded at each boot → impact on boot

time


Interactions between PS and PL

● The Xilinx Linux kernel (founder specific) has dedicated drivers for hardware resources implemented by PL and shared with PS

● These resources are appended to the device tree, thus making them available to Linux kernel

● Vivado automatically generates a pl.dtsi file that is included in the board device tree

● pl.dtsi reflects selection and design of hardware resources of the PL accessible to the PS



/* * CAUTION: This file is automatically generated by Xilinx. * Version: HSI 2016.4 * Today is: Fri Aug 4 16:29:54 2017*/

/ {amba_pl: amba_pl {

#addresscells = <1>;#sizecells = <1>;compatible = "simplebus";Ranges ;RS485_PL: serial@42c00000 {

clocknames = "ref_clk";clocks = <&clkc 0>;compatible = "xlnx,xpsuartlite1.00.a";currentspeed = <115200>;device_type = "serial";interruptparent = <&intc>;interrupts = <0 35 1>;portnumber = <0>;reg = <0x42c00000 0x10000>;xlnx,baudrate = <0x2580>;xlnx,databits = <0x8>;xlnx,oddparity = <0x0>;xlnx,saxiaclkfreqhzd = "50.0";xlnx,useparity = <0x0>;

};

Additional serial line

Link with Xilinx driver



…

video_in_0_Status: gpio@41230000 {#gpiocells = <2>;compatible = "xlnx,xpsgpio1.00.a";gpiocontroller ;reg = <0x41230000 0x10000>;xlnx,allinputs = <0x1>;xlnx,allinputs2 = <0x0>;xlnx,alloutputs = <0x0>;xlnx,alloutputs2 = <0x0>;xlnx,doutdefault = <0x00000000>;xlnx,doutdefault2 = <0x00000000>;xlnx,gpiowidth = <0x5>;xlnx,gpio2width = <0x20>;xlnx,interruptpresent = <0x0>;xlnx,isdual = <0x0>;xlnx,tridefault = <0xFFFFFFFF>;xlnx,tridefault2 = <0xFFFFFFFF>;

};};

};

Information towards PS

Link with Xilinx driver


Device tree generation

● In fact, Vivado generates the whole device tree :

– PS part through zynq7000.dtsi

– PL part through pl.dtsi

– And the final system.dts that includes PS and PL part, + nodes customizations according to Vivado design

● Vivado covers PL but also parameters impacting PS

● Device tree must be rebuilt after each change in Vivado → automation of the workflow welcome– Development of dedicated Yocto recipes to manage generation and

dispatch of all these files in Board Support Package dedicated recipes● U-boot and device tree recipes impacted


Boot time optimization

● The requirement is to have the image completely stable at much 7 s after Power ON

● Tasks necessary before image :– FSBL– U-boot– Bitstream loading and FPGA programming – Kernel initialization– Userland startup (system V init)– Cameras initialization through spi communication– FPGA video IP configuration


Boot time optimization

● Misc techniques used to minimize image arrival time :– Activation of bootstage report feature of u-boot– Use of bootchart to identify where boot time is consumed during Linux start-up– U-boot optimization to reduce peripherals inits– Linux kernel optimization to reduce kernel size (media access time) and

peripherals init– System console removal– Completely reorder init scripts to start video application as early as possible– Use of read only root file system to avoid file system checks and corrections

(power off without halt)– Maximize spi speed on spi bus with camera

● Finally image is here and stable 5.6 s after Power On.


Consequences of safety requirements

● The design has been done so that the safety certification will impact only the FPGA– Linux developments not directly impacted by certification

process● But the software must be updated through IP communication over

Ethernet● As FPGA software is updated through Linux IP features, the

installation & upgrade process must be secured to avoid potential hacking

● Installer and updater utilities use asymmetric cryptography mechanism to authenticate all software elements before installation / update


Conclusion

● At the beginning of the project, 2 major decisions :– Use a mixed design based on Embedded Linux + FPGA– Use meta-xilinx layer and assemble layers by ourself instead

of standard solutions (Petalinux or Wind River Pulsar Linux)● What are the results ?


Conclusion

● The main concerns regarding the dual architecture was about :– Splitted design– Communication between PS and PL– Kernel management of peripherals built in PL part

● Splitted design is a constraint but the advantages of using a FPGA are more important than the drawbacks

● AXI interface offers many solutions to implement communications between PS and PL

● Drivers available in Xilinx kernel + automatic generation of device tree make the peripherals designed in PL well supported by the kernel

● PL design make the final platform easily adaptive to new requirements– Ex : debug GPIOs useful only during development phase synthesized in the PL

part then removed


Conclusion

● Using meta-xilinx and assembling them by ourselves with other layers from OE layers repository was a good choice– Good quality of meta-xilinx layer– We select exactly the layers we need– Finally we have a very well adapted distribution

● This was also possible because we have good skills on Yocto build tool and layers architecture and features.


Questions ?

● Thank you for your attention● Time for questions

developing an embedded video application on dual linux + fpga architecture

Technology