Development of Robot Vision System with FPGA and Real-time Linux

Kenichi Asami Yoshinobu Ooka Hayato Hagiwara Mochimitsu Komori Department of Applied Science for Integrated System Engineering

Kyushu Institute of Technology 1-1, Sensui, Tobata, Kitakyushu 804-8550, Japan

ABSTRACT The robot vision system using FPGA and real-time Linux is presented. The system mechanism uses two stepper motors for setting line of vision of the CMOS camera. The image sensor and the driver ICs are connected to external I/O ports of the FPGA. The image processing and motor drive circuits are implemented into the reconfigurable device as original logic. The image capture circuit applies state machine and FIFO memory buffer for adjusting timing of pixel data transmission. The motor drive circuit generates clock signal for steps according to the directive value from processor in the FPGA. The real-time device driver has been developed for visual feedback and object tracking applications. Cooperation between flexible hardware circuits and real-time software drivers enhances performance of robot vision applications.

Keywords: Robot Vision System, Autonomous Mobile Robots, FPGA, Real-time Linux, Device Driver.

1. INTRODUCTION

In recent years various advanced autonomous mobile robots have been developed in academic and industrial fields. The purpose for life support would acquire greater importance in order to adapt quickly and effectively to rapid changes in future social environment, for example in Japan, the aging of society and declining number of children. The autonomous mobile robots need to recognize the surrounding for achieving tasks by using image processing. The robot vision systems require real-time response and low electric power consumption for the accurate and long time movement. Therefore, common image processing for application software such as outline and edge detection should be implemented on the hardware [2], [3].

This paper presents the development of robot vision system using FPGA (Field Programmable Gate Array) device for image capturing and preprocessing and Xenomai hard real-time extension of GNU/Linux for the fast and deterministic time response to device I/O. A CMOS image sensor is connected to I/O ports of the FPGA device on the pan-tilt mechanism for tracking an object. The digital pixel data from the image sensor is gained into the internal circuits synchronously with the pixel clock signal, and is written to FIFO (First-In First-Out) memory buffer that is read from processor by device driver software under real-time Linux. The periodically-scheduled image data

gets available to application software for object recognition, pan-tilt motors control, data communication, and so on. The integration of hardware and software for image processing that can be utilized from robot vision application would be provided.

2. SYSTEM CONFIGURATION

The robot vision system consists of four main parts: a CMOS camera module, a FPGA board, two stepper motors and motor drivers. The CMOS camera module has a size of 12mm x 12mm x 11mm with a 1/4 inch receiving area in VGA resolution as digital output. For the electrical power consumption, the camera module requires only 80mW at 2.8V in 12MHz. The standard video data output is served in the form of ITU_R RT.656 in YUV 4:2:2 pixel format, which compresses data size into 16 bits for a pixel element so that the application software can save the processing and communication costs. The FPGA board equips Xilinx Virtex-4 FX with PowerPC 405 processor at 350MHz as a hard core in the 12,312 logic cells. The platform of embedded systems has 64MB SDRAM, 8MB flash ROM, 86 external I/O ports, and UART serial communication interface, which realizes the running environment of GNU/Linux operating system and a variety of useful open source software.

The stepper motors convert electrical pulses into precisely movements for turning the CMOS camera toward an object. The two-phase unipolar stepper motor has a size of 28mm x 28mm x 32mm and a weight of 110g with the stepping angle of 1.8 degree, and the maximum stationary torque produces 0.05Nm. The stepper motor drivers are designed for simple requirement and made with easily available parts. The motor driver IC includes four power FETs to amplify electrical current, two PWM controllers to synchronize two phases, and a logic sequencer to generate excitation patterns for a full or half step. The driving voltage is 10 to 44V and the maximum output current is 1A.

The mechanism of the robot vision system is constructed from two shafting structure as pan-tilt rotations with two frames attaching the main parts. The FPGA board is fixed on the upper frame, and the stepper motors and the drivers are fixed on the lower frame. The two frames and main parts are assembled by shafts, bearings and gears, so that the robot vision system can change the visual field by itself.

3. IMAGE PROCESSING CIRCUITS

For real-time robot vision, there is a problem of unpredictable time response in PC-based systems where the image processing is achieved by software using uniprocessor in many cases. Moreover, video data transmission to processor by USB interface could become a bottleneck for the image processing speed. In the other cases, dedicated hardware systems based on DSP could fall into inflexible image processing structure and result in a lot of useless computing resources. In order to overcome the problem, robot vision systems based on FPGA devices could make advantages of reconciling reconfigurability with efficiency for large-scale and high-speed data processing.

Xilinx EDK (Embedded Development Kit) tool was used to build the FPGA configuration file for capturing video data from the CMOS camera module. EDK supports to create embedded processor systems and integrate them with peripheral IP cores. In the robot vision system, an image capture circuit, a state machine circuit and a FIFO memory buffer were built by using Verilog HDL (Hardware Description Language). Additionally,

the stepper motors control circuit is included in the FPGA configuration file, which commands stepping speed and rotating directions to the motor drivers.

The image capture circuit supplies the operating power with a ground connection to the CMOS camera module, and is wired to the reset and standby terminals. The master clock is generated from a digital clock manager that combines a frequency synthesizer, a phase shifter and a clock delay locked loop, and is given to the camera module as output of the image capture circuit. Once the CMOS camera module starts the operation, the image capture circuit receives input signals: pixel data in 8 bits, pixel clock for data synchronization, horizontal synchronization for a line and vertical synchronization for a frame. According to the width of data bus of 64 bits between the processor and the peripheral IPs, the image capture circuit collects four pixel data that includes two luma components (Y, brightness) and two chrominance components (UV, color difference of blue and red) for two pixel elements, and then transfers the quadruplet to the state machine circuit.

(b) FPGA board(a) CMOS camera module (c) Motor drivers

Figure 1 Main parts of system

fixed stand

stepper motorstepper motor

FPGA board

CMOS camera module

pan shaft

tilt shaft

motor driver IC

(a) Structure of robot vision system (b) Overview of robot vision system

83mm

47mm

Figure 2 Structure of system

In the state machine circuit, there are three finite states: wait, write and write-acknowledge states, which represent sequential behavior associated with input and output. When the pixel data of 64 bits from the image capture circuit changes the value as the next sequential content at a wait state, the state machine moves to a write state by sending a write requirement to FIFO memory buffer. If the FIFO is not full up with data, the write operation is completed and the state machine moves to a write-acknowledge state. If the FIFO sends back the write-acknowledge signal, the state machine moves to a wait state, and the cycle of state transition is repeated.

The FIFO memory queue plays a role as buffer circuit for adapting timing between two circuits or data transmission. The proposed robot vision system finally uses the pixel data on the processor with Linux OS and application software. Therefore, a FIFO memory buffer is constructed in the block RAM on the Virtex-4 FX device, where the captured pixel data can be accessed from the PowerPC 405 hard core processor. In the Virtex-4 FX device, a user IP core is connected on Xilinx PLB (Peripheral Local Bus) local bus to the embedded processor through IPIF interface, which can be mapped on the memory I/O addresses.

Figure 3 Structure of image processing circuit

(a) Block diagram of image processing circuit

external I/O

PowerPC 405processor

MCLKVDDGND

CMOScamera module

user logic

imagecapturecircuit

statemachinecircuit

block RAM

FIFOmemorybuffer

PDATAPCLKHSYNCVSYNC

PLB local bus

Xilinx Virtex-4 FX

(b) State transition for FIFO memory buffer

wait

writeacknowledge

pixel dataupdated

FIFOnot full

writecompleted

write

Figure 4 Implementation of motor drive circuit

(a) Verilog HDL description of motor drive circuit (b) State transition for FIFO for motor drive

FIFOnot empty

readcompleted

drivecompleted

drivestartedread

drivereadacknowledge

driveacknowledge

module step1_drive (CLK1, DIR1, SCLK, VAL1, ... );

output CLK1; // clock signal to driveroutput DIR1; // direction of driveinput SCLK; // clock signal for stepsinput [7:0] VAL1; // value requiredreg [9:0] STEPS1; // number of steps

assign CLK1 = (STEPS1 == 0) ? 0 : SCLK;assign DIR1 = ~VAL1[7];

always @(posedge SCLK) beginif (STEPS1 != 0)

STEPS1 <= STEPS1 − 1;end

endmodule

In the same way as the image processing circuit, the motor drive circuit has been built in the reconfigurable device as original logic. The motor drive circuit accesses the insertion FIFO for operation data from processor through peripheral bus. In the state machine circuit, there are four finite states: read, read-acknowledge, drive, and drive-acknowledge. The FIFO receives an operation value in 4 bytes for motor drive from application software at the states of read and read-acknowledge. The FIFO transmits the value to the motor drive circuit at the states of drive and drive-acknowledge. According to the directive value, the sequential logic generates clock signal for steps to the driver ICs, and then the stepper motors start a drive. When the number of steps becomes zero, drive-end signal is returned to the state machine circuit in order to permit the next drive operation. Small-grained operations can be successively managed in the state transition.

4. REAL-TIME DEVICE DRIVER

In embedded or control systems, real-time means that a target processing can be guaranteed to start and finish within a fixed time. Technically an application requires a program to respond to stimuli less than small upper limit of milliseconds or microseconds. In the robot vision system, a part of image processing requires real-time response with respect to the strong relation to the physical motion. On the other hand, various image processing such as object recognition, distance detection, motion detection and visual navigation by a map would be required for the application software, where it is not realistic that the entire image processing is implemented on a FPGA device because of size limitation of integration and inefficiency of development compared with software. Real-time operating systems based on GNU/Linux have many advantages as a real-time solution consisting with various processing for the robot vision system.

GNU/Linux is an open source implementation of the Unix operating system which has been published under GPL. The embedded Linux system provides a benefit for applications with large numbers of programmers that are familiar with the programming API. There are a rich set of development tools, network protocol stack and other application software available for GNU/Linux on major hardware platforms. The Linux kernel is not designed for real-time performance, but for general-purpose by fairly scheduling to share computing resources among tasks. There are some real-time extensions for the Linux kernel by replacing the interrupt handler, where a tiny scheduler manages interrupt propagation to the Linux kernel, so that real-time tasks always have a higher priority. In order to achieve the robot vision system, Xenomai [4] real-time extension was ported to the embedded system on the Virtex-4 FX device by configuring and building the target kernel for the PowerPC architecture. In this case, Linux kernel source 2.6.18, Xenomai 2.4.10 and GNU cross toolchain 4.1.1 for PowerPC were used, and the target kernel and the user space libraries were compiled. The binary image in romfs

format that includes the Linux kernel with Xenomai extension, libraries and applications was made and downloaded to the flash ROM on the FPGA board. After boot messages, Xenomai real-time environment was successfully established.

A real-time device driver under Xenomai was written to get image data from the image capture circuit through the FIFO memory buffer. RTDM (Real-Time Driver Model) under Xenomai is an approach to unify the interfaces for developing device drivers and associated applications under real-time Linux [1]. The RTDM device structure was defined to be registered with the real-time subsystem, and the memory region was mapped to the peripheral IP circuit with ioremap() function. The stream-oriented read() procedure that repeats dequeuing image data for every frame into double-buffered memories was described, and the device became possible to be accessed from application software.

5. CONCLUSION

This paper introduced the development of robot vision system using FPGA and real-time Linux. Application software for tracking an object by detecting color was developed, and the robot vision system achieved smooth tracking motion toward the object. The developed vision system could be loaded on small-sized mobile robots with respect to the size, the weight and the electric power consumption. The system would be useful for applications of watching an aged person and detecting a suspicious person by indoor patrol camera robot. The integration of hardware and software platform for real-time image processing could be available for various applications.

There are three remaining subjects in this system. Common image processing algorithm needs to be implemented on a FPGA device. A mobile robot uses motion detection, edge detection, and registered template matching. A stereo vision system would be desired with distance measurement by trigonometry. A remote vision function using RTnet Ethernet controller driver would be needed for transferring image data to a distant terminal.

6. REFERENCES

[1] J. Kiszka and B. Wagner, “Domain and Type Enforcement for Real-Time Operating Systems,” Proc. IEEE Int. Conf. on.Emerging Technologies and Factory Automation, Vol.2 pp.439-446, 2003.

[2] K. Kudo, Y. Myokan, W.C. Than, S. Akimoto, T. Kanamaru, and M. Sekine, “Hardware Object Model and Its Application to the Image Processing,” IEICE Trans., Vol.E87-A, No.3, pp.547-558, 2004.

[3] K. Shimuzu and S. Hirai, “CMOS+FPGA Vision System for Visual Feedback of Mechanical Systems,” Proc. IEEE Int. Conf. on Robotics and Automation, pp.2060-2065, 2006.

[4] Xenomai Project, “http://www.xenomai.org.” [5] Xilinx Inc., “Virtex-4 FPGA User Guide,” 2008.