research article...

Hindawi Publishing CorporationInternational Journal of Reconfigurable ComputingVolume 2008, Article ID 636145, 8 pagesdoi:10.1155/2008/636145

Research Article

FPGA-Based Embedded Motion Estimation Sensor

Zhaoyi Wei, Dah-Jye Lee, Brent E. Nelson, James K. Archibald, and Barrett B. Edwards

Electrical and Computer Engineering Department, Brigham Young University, Provo, UT 84602, USA

Correspondence should be addressed to Dah-Jye Lee, [email protected]

Received 27 March 2008; Accepted 24 June 2008

Recommended by Fernando Pardo

Accurate real-time motion estimation is very critical to many computer vision tasks. However, because of its computational powerand processing speed requirements, it is rarely used for real-time applications, especially for micro unmanned vehicles. In ourprevious work, a FPGA system was built to process optical flow vectors of 64 frames of 640 × 480 image per second. Comparedto software-based algorithms, this system achieved much higher frame rate but marginal accuracy. In this paper, a more accurateoptical flow algorithm is proposed. Temporal smoothing is incorporated in the hardware structure which significantly improvesthe algorithm accuracy. To accommodate temporal smoothing, the hardware structure is composed of two parts: the derivative(DER) module produces intermediate results and the optical flow computation (OFC) module calculates the final optical flowvectors. Software running on a built-in processor on the FPGA chip is used in the design to direct the data flow and managehardware components. This new design has been implemented on a compact, low power, high performance hardware platform formicro UV applications. It is able to process 15 frames of 640 × 480 image per second and with much improved accuracy. Higherframe rate can be achieved with further optimization and additional memory space.

Copyright © 2008 Zhaoyi Wei et al. This is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

Optical flow aims to measure motion field from the apparentmotion of the brightness pattern. Optical flow is one ofthe most important descriptions for an image sequenceand is widely used in 3D vision tasks such as motionestimation, structure from motion (SfM), time-to-impact,and so forth. An accurate hardware-friendly optical flowalgorithm is proposed and its hardware design is presentedin this paper. This design can be used for unmanned airvehicle (UAV) navigation tasks, moving object detection, andother applications which require fast and accurate motionestimation.

The basic assumption of optical flow algorithms is thebrightness constancy constraint, which assumes that imagebrightness changes are due only to motion. In other words,if motion between frames is small, other effects (such aschanges in lighting conditions) causing brightness changescan be neglected. However, one limitation of the use ofoptical flow is its computational requirement. The processingtime of existing optical flow algorithms is usually on theorder of seconds or longer per frame. This long processingtime thus prevents optical flow algorithms from being

used for most real-time applications such as autonomousnavigation for unmanned vehicles.

In recent years, a number of different schemes have beenproposed to implement optical flow algorithms for real-timeapplications. The basic idea behind them is to use pipeline orparallel processing architectures to speed up computations.For example, graphics processing unit (GPU) devices haverecently been used for optical flow implementation withgood results [1–3]. Alternatively, Correia and Campilho [4]proposed a design which can process the Yosemite sequencein 47.8 milliseconds using a pipeline image processor.Compared to GPUs or processors, FPGAs are more suitablefor small UAV operation because of their compact sizeand low power consumption. FPGAs have been used tocalculate optical flow [5–13] in the past few years becauseof their configuration flexibility and high data processingspeed. For example an iterative algorithm proposed by Hornand Schunck [14] was implemented in an FPGA in [5–7]. The classical Lucas and Kanade approach [15] was alsoimplemented in [7, 8] and provided a tradeoff betweenaccuracy and processing efficiency.

To improve accuracy and provide high-performancesolutions, many different optical flow algorithms have been

mailto:[email protected]

2 International Journal of Reconfigurable Computing

developed in the last two decades. During this time, 3Dtensor techniques have shown their superiority in producingdense and accurate optical flow fields [16–19]. The 3D tensorprovides a powerful and closed form representation of thelocal brightness structure. In our previous work, a tensor-based optical flow algorithm was implemented on an FPGAto process 64 frames of 640 × 480 image per second.

Despite the fact that hardware-based optical flow designcan achieve much higher processing rates than software,hardware-based designs are usually less accurate. Usingthe design in [12] as an example, its accuracy measuredin angular error is 12.7◦ on the Yosemite sequence whilestate-of-art optical flow algorithms using software and withsufficient memory space can achieve accuracy close to 1.0◦

on the same sequence. Two main reasons affecting accuracyare the following.

(1) Algorithm limitation: many software-based algo-rithms use iteration and apply various optimizationtechniques to find optimal values. However, hardwareis best for algorithms which can be pipelined andprocessed in parallel. Therefore, only a very limitedsubset of the available software-based algorithms isa good fit for hardware. Sacrificing accuracy forprocessing speed is inevitable for hardware imple-mentation.

(2) Hardware resource limitations: for optical flowalgorithms, smoothing is a very important opera-tion for suppressing noise and extracting accuratemotion information. However, current smoothingalgorithms for software are not suitable for hardwareimplementation where a tradeoff between accuracyand feasibility needs to be made. Moreover, software-based algorithms mostly use floating point compu-tations. When implementing software optical calcu-lations in hardware, data truncation, rounding, andsaturation operations lower the algorithm’s accuracy.

Our goal in this paper is to improve the accuracy ofprevious hardware optical flow designs under the prereq-uisite of speed and feasibility. The design described hereinuses temporal smoothing in addition to spatial smoothing inthe calculation to improve the accuracy and stability of thealgorithm. To accommodate temporal smoothing (which isnot trivial in hardware), a new hardware organization is used.The hardware pipeline is decomposed into two modules.The first module calculates derivative frames for each imageframe. The second module calculates optical flow from thederivative frames. From the experimental analysis done, itcan be seen that this design achieves an error rate of 6.7◦

on the Yosemite sequence which doubles the accuracy ofprevious work [12]. The temporal smoothing used preventsthe creation of a fully pipelined design. Further, it requiresadditional memory bandwidth, leading to a processing rateof 15 frames per second for 640 × 480 images. Higherframe rates could be achieved with further optimizationsand additional memory space. However, this frame rateis sufficient for many unmanned vehicle applications thatrequire higher accuracy.

The proposed algorithm is implemented on a newlydeveloped standalone hardware platform. Images capturedby the on-board CMOS camera are processed by hardwareand can be transferred to PC for debugging and evaluationpurposes.

This paper is organized as follows. In Section 2, the algo-rithm is formulated and our modifications are introduced. InSection 3, the hardware structure and tradeoffs made in thedesign are discussed. In Section 4, the hardware platform isintroduced. Performance analysis of the proposed design onsynthetic and real sequences is also shown. Conclusions andfuture work are discussed in Section 5.

2. Optical Flow Algorithm

2.1. Algorithm Description

An image sequence g(X) can be treated as volume data whereX = (x, y, t)T , x and y are the spatial components, andt is the temporal component. According to the brightnessconstancy constraint, object movement in the spatiotempo-ral domain will generate brightness patterns with certainorientations. The 3D tensor is a compact representation oflocal orientation and there are different types of 3D tensorsbased on different formulations [20, 21]. The gradient tensoris used in this design because it is easier to implement withpipelines in hardware.

The outer product O of the averaged gradient ∇g(x) isdefined as

O = ∇g(x)∇g(x)T =

⎛⎜⎜⎝

o1 o4 o5

o4 o2 o6

o5 o6 o3

⎞⎟⎟⎠ , (1)

where

∇g(x) =∑

i

wi∇g(

xi) =

(gx(x) g y(x) gt(x)

)T, (2)

and wi are weights for averaging the gradients. The gradientcomponent is calculated using a simple mask shown in (3)which is the same as in [12], which was chosen as a tradeoffbetween accuracy and efficiency,

d =(

1 −8 0 8 −1). (3)

Gradient tensor T is a 3× 3 positive semidefinite matrix thatis constructed by weighting O in a small neighborhood as

T =∑

i

ciOi =

⎛⎜⎜⎝

t1 t4 t5

t4 t2 t6

t5 t6 t3

⎞⎟⎟⎠ . (4)

Optical flow (vx, vy)T is measured in pixels per frame

and can be extended to a 3D spatiotemporal vector, V =(vx, vy , 1)T . For an object with only translational movementand without noise in the neighborhood, VTTV = 0. Inthe presence of noise and rotation, VTTV will not be zero.

International Journal of Reconfigurable Computing 3

DER

PLB IPIF

MultiportSRAM arbiter

SRAM

OFC

PLB IPIF

PLB bus

PLB IPIF PLB IPIF PLB IPIF

SDRAMUSB CameraCTRL

PLB2OPBbridge

GPIO

OPB IPIF

PC OPB bus

CameraOPB IPIF OPB IPIF OPB IPIF

CAMserial UART INTC

Figure 1: System hardware diagram.

Instead, V can be determined by minimizing VTTV. A costfunction can be defined for this purpose as

e(V) = VTTV. (5)

The velocity vector V is the 3D spatiotemporal vectorwhich minimizes the cost function at each pixel. The initialoptical flow can be solved as

vx =(t6t4 − t5t2

)(t1t2 − t24

) ,

vy =(t5t4 − t6t1

)(t1t2 − t24

) .

(6)

The initial optical flow vector is smoothed in a localneighborhood to suppress noise further. The final opticalflow vector is formulated as

vi =(vx

vy

)=∑

i

mi

(vxi

vyi

). (7)

The algorithm proposed in this paper is similar to theone in [17] which assumes a constant motion model. Affinemotion model is often used to incorporate tensors in asmall neighborhood [17] where pixels in a neighborhood areassumed to belong to the same motion model. To conservehardware resources, the constant model is used in this design.The constant model performs almost as well as affine motionmodel when operating in a small neighborhood.

2.2. Smoothing Masks

As mentioned in Section 2.1, smoothing is very importantto the algorithm performance. Smoothing is performedthrough three smoothing masks wi, ci, and mi as shown in(2), (4), and (7) where wi is for gradient averaging, ci is fortensor construction, and mi is for velocity smoothing. Thefirst mask is a spatiotemporal smoothing mask, and the othertwo are spatial smoothing masks.

To reach an optimal tradeoff between algorithm per-formance and hardware resources, the configurations of

these masks were evaluated carefully by simulation beforeimplementation. Smoothing mask parameters were deter-mined by three factors: mask shape, mask size, and maskkernel components. In software, large smoothing masks (e.g.,19 × 19 or larger) are often used. In hardware, smallermasks must be used because of resource limitations (e.g.,7 × 7 or smaller). As for mask shape, a square maskis usually used for the sake of simplicity and efficiency.Spatial and temporal smoothings can use different size masksto improve performance. While spatial smoothing can bereadily performed in either software or hardware, spatiotem-poral smoothing is much more challenging in hardwarethan in than software. Temporal smoothing is significantlymore complicated than spatial smoothing because it involvesmultiple image frames. However, temporal smoothing isimportant for estimating motion field consistency over ashort period of time. Incorporating temporal smoothingsubstantially improves algorithm performance as can be seenin the experimental analysis section. In this design, the sizeof the first smoothing mask wi in (2) is 5 × 5 × 3 of which3 is the temporal smoothing size (3 frames). The number offrames used for temporal smoothing is largely determined byhardware resources and processing speed requirement. Thesizes of the other two spatial smoothing masks ci, and mi are3× 3 and 7× 7, respectively. Parameters of all the smoothingmasks are in a shape of Gaussian function. To save hardwareresources, a 2D Gaussian mask is decomposed into two 1DGaussian masks which are cascaded and convolved along xand y directions separately [12]. Different settings of thesemasks are simulated by software at bit-level accuracy andevaluated on synthetic sequence with ground truth to obtainan optimal combination in practice.

3. Hardware Structure

The diagram of the hardware platform using a XilinxVirtex-4 FX series FPGA for implementing our optical flowalgorithm discussed in Section 2 is shown in Figure 1. Mosthardware components are either connected to the processorlocal bus (PLB) or on-chip peripheral bus (OPB). The PLB


Rea

din

glo

gic

frame(t)

frame(t)

frame(t − 1)

frame(t − 3)

frame(t − 4)

frame(t − 2)

g tca

lcu

lati

ong x

,gy

calc

ula

tion

gt(t)

gt(t)

gx(t)

gx(t)

gy(t)

gy(t)

Data stored in SRAM

Data stored in SDRAM

Figure 2: DER module diagram.

is used for connecting hardware components that requirehigh bandwidth such as camera and USB ports, and so forth.The OPB is used for connecting relatively slower units suchas UART, interrupt controller, and so on. PLB and OPB areconnected through PLB2OPB bridge. Two main componentsof this design: derivative (DER) module and optical flowcomputation (OFC) module are connected to the PLB busthrough a bus interface. The DER and OFC modules sharethe SRAM through a multiport SRAM arbiter connectedto the SRAM to obtain optimal accessing bandwidth. Theslower SDRAM is accessed through the PLB bus.

The main difference between this design and our previ-ous design [10] is that it incorporates temporal smoothing inthe pipeline. In this design, three sets of derivative frames, gx,gy , and gt shown in (2) are temporally smoothed.

With temporal smoothing, multiple sets (3 in this design)of derivative frames must be stored as they are calculated andthen reloaded during the smoothing process. It is impossibleto store multiple sets of derivative frames on-chip usingthe hardware resources that are available. Therefore, theoptical flow calculation pipeline has to be divided into twosteps. The first part (called DER module) generates derivativeframes and the second part (called OFC module) handles therest of calculations. These two hardware modules must bemanaged to synchronize their computation tasks and handleexceptions such as dropped frames. Software running onthe built-in on-chip powerPC processors is used for thismanagement task.

Figure 2 shows the diagram of the DER module. Everycycle when a new image frame(t) is captured directly intothe SDRAM through the PLB bus, reading logic readsthe captured image from the SDRAM into a pipeline andstores it in the SRAM. Whenever there are five consecutive

image frames stored in the SRAM, they are all read out forcomputing the derivative frames gx, gy , and gt using (3). gxand gy are calculated from the current incoming frame(t)and gt is calculated from frame(t−4), frame(t−3), frame(t−2), frame(t − 1), and the current incoming frame. Theresulting derivative frames are stored in the SRAM as wellas the SDRAM for future usage. The duplicate copy storedin the SDRAM is needed for temporal smoothing for futureframes. This storing and retrieving of derivative frames fromSDRAM consumes PLB bus bandwidth and hence slowsdown the processing speed. As shown in Figure 1, if thehardware platform used had sufficient SRAM (currently only4 Mb), then all 9 derivative frames (3 sets of gx, gy , and gt)could be stored in the SRAM and take the advantage of ahigh-speed multiport memory interface.

Figure 3 shows the dataflow of the OFC module. Oncea new set of derivative frames is calculated, software willtrigger the OFC module to start the calculation of opticalflow. Derivative frames for the current frame in the SRAM(gx(t), gy(t), and gt(t)) and the derivative frames alreadystored in the SDRAM (gx(t − 1), gy(t − 1), and gt(t − 1))and (gx(t − 2), gy(t − 2), and gt(t − 2)) are first readinto the pipeline for temporal smoothing. Because the sizeof the temporal smoothing mask is 3, derivative framesat time t, t − 1, and t − 2 are averaged to obtain thesmoothed derivative frames for the current frame at timet (gx t, g y t, and gt t, in Figure 3). These frames are thenspatially smoothed. The spatially smoothed derivative frames(gx t, g y t, and gt t, in Figure 3) are then used to constructthe tensors as shown in (4). Six tensor components t1,t2, t3, t4, t5, and t6 are obtained after this step. These sixcomponents are spatially smoothed using a 3 × 3 smoothingmask. Then, vx and vy can be calculated from these smoothedcomponents as shown in (6). Two motion components vxand vy are spatially smoothed with mi to get the final opticalflow vectors as shown in (7).

Figure 4 shows the system software diagram. There arethree types of frames existing in the system:

(1) image frames captured by the camera;

(2) derivative frames calculated by the DER module;

(3) optical flow fields calculated by the OFC module.

The DER module uses the raw images as input and the OFCmodule uses the output from the DER module (derivativeframes) as the input. Three linked lists are used to storethese frames and maintain their temporal correspondence.A frame table entry (FTE) is used to store image frames, aDER FTE is used to store derivative frames and an OFC FTEis used to store optical flow frames. As shown in Figure 4,there are 5 corresponding pairs of FTE and DER FTE and 3pairs of DER FTE and OFC FTE.

It is noted that these modules execute asynchronously.When a new raw image is captured (FTE7 in this case), thecamera core invokes an interrupt. This interrupt is sensedby the FTE interrupt handler in software and a triggersignal is generated and sent to the DER module to initiatea derivative computation. When a new set of derivativeframes is calculated (DER FTE4 in this case), the DER


gx(t)

gy(t)

gt(t)

gx(t − 1)

gy(t − 1)

gt(t − 1)

gx(t − 2)

gy(t − 2)

gt(t − 2)

g xte

mp

oral

smoo

thin

g

g yte

mp

oral

smoo

thin

g

g tte

mp

oral

smoo

thin

g

gx t gx t

gy t gy t

gt t gt tTe

nso

rca

lcu

lati

on

g xsp

atia

lsm

ooth

ing

g ysp

atia

lsm

ooth

ing

g tsp

atia

lsm

ooth

ing

t1spatial

smoothingt1 t1

t2t2

t3 t3

t4 t4

t5 t5

t6t6

t2spatial

smoothing

t3spatial

smoothing

t4spatial

smoothing

t5spatial

smoothing

t6spatial

smoothing

Opt

ical

flow

calc

ula

tion

Opt

ical

flow

smoo

thin

gvx

vy

vx

vy

Data stored in SRAM

Data stored in SDRAM

Intermediate data

Figure 3: OFC module diagram.

module invokes an interrupt. This interrupt is sensed by theDER FTE interrupt handler in software and a trigger signal isgenerated and sent to the OFC module to initiate an opticalflow computation.

4. Results

4.1. Hardware Platform

This design was implemented on the BYU Helios RoboticVision board [22, 23]. This embedded vision sensor boardwas developed in the Robotic Vision Laboratory at BrighamYoung University for real-time visual computing. The Heliosboard, shown in Figure 5(a), is compatible with XilinxVirtex-4 FX series FPGAs, up to the FX60. The Virtex-4FX60 has 25 280 slices and two built-in 400 MHz powerPCprocessors, allowing for both custom hardware and soft-ware development. The Autonomous Vehicle Toolkit (AVT)daughterboard (Figure 5(c)) which is capable of supportingup to two CMOS cameras (Figure 5(b)) was designed toenhance the functionalities of the Helios board. The CMOScamera (capable of acquiring 60 frames per second) was setto capture 30 frames of 640 × 480 8-bit color image data persecond in this design.

For computer vision tasks, high-speed memory is critical.There are currently 32 Mb SDRAM and 4 Mb SRAM onthe Helios platform. The SRAM is faster and easier toaccess than the SDRAM. In this design, we tried to storeas much intermediate data as possible (e.g., image frames

FTE linklist

FTE1

FTE2

FTE3

FTE4

FTE5

FTE6

FTE7

FTE8

DER FTElinklist

DER FTE1

DER FTE2

DER FTE3

DER FTE4

DER FTE5

OFC FTElinklist

OFC FTE1

OFC FTE2

OFC FTE3

Figure 4: Software diagram.

and derivative frames) in the SRAM. The rest of the datawas stored in the SDRAM and accessed through the PLBBus. There are other hardware resources and features on theHelios that are useful for other real-time vision tasks butwhich were not used in this design. This hardware platform


(a) (b) (c)

Figure 5: Hardware components: (a) Helios FPGA board, (b) CMOS imager, and (c) AVT daughter board.

(a) (b)

Figure 6: Result of synthetic sequences: (a) Yosemite, and (b) Flower garden.

is suitable for small UGV or UAV which requires low power,real-time computation capability. Detailed information canbe found in [22, 23].

The whole system executes at a clock speed of 100 MHzand the system is using one clock domain. At this frequency,it is able to calculate 15 frames of optical flow field persecond. All computations used fixed point representationto save hardware resources. Temporal smoothing requiresmuch more data transfer than spatial smoothing, and thePLB bandwidth is the limiting factor to increased processingspeed. Our estimation is that higher frame rate can beachieved with further optimization work and additionalmemory space (specifically on-board SRAM). These process-ing speed improvements are discussed in Section 5. From theperformance analysis in the following sections, it can be seenthat with temporal smoothing the accuracy of the estimatedoptical flow is improved substantially on a synthetic sequencethat is commonly used for benchmarking.

The whole design utilized 21481 slices (84% of the total25280 slices on a Virtex-4 FX60 FPGA). The DER moduleused 2196 slices (8% of the total) and the OFC module used6324 slices (25% of the total). The remainder was used for thecamera core, I/O, and other interface circuitry. A graphicaluser interface (GUI) was developed to transfer the status ofthe Helios board and the real-time video to PC for displaythrough the USB interface.

4.2. Synthetic Sequences

This design was tested on two synthetic sequences to showits effectiveness. One of these sequences (Yosemite) hasan established ground truth and is commonly used for

benchmarking. A bit level simulation coded in MATLABwas programmed to evaluate the algorithm’s accuracy. Theerror is measured in angular error which is widely used forevaluating optical flow algorithm.

Figure 6(a) shows one frame of the Yosemite sequencewith the calculated optical flow vectors superimposed forvisual feedback. The angular error of the Yosemite sequencewas 6.7◦. To the best of our knowledge, this is the mostaccurate result generated to date using hardware, and hashalf the error rate of our previous design [12] whichachieved an error rate of 12.7◦. There are two major reasonsfor this improvement. The first is that a better hardwareplatform (the Helios platform) was used and supported theimplementation of a more complex algorithm on the hard-ware. The second reason is the incorporation of temporalsmoothing in the algorithm. Without temporal smoothing,the accuracy would deteriorate to approximately 10.5◦. Thisexperiment demonstrates that temporal smoothing has asignificant impact on the algorithm performance. Usingtemporal smoothing is an acceptable tradeoff of processingspeed for accuracy for applications that require higheraccuracy.

Figure 6(b) shows the result of the other syntheticsequence (flower garden). While the proposed algorithm didnot perform well in regions without much texture, this isa shortcoming of all optical flow and motion estimationalgorithms. In regions with noticeable texture, the algorithmperformed very well.

4.3. Real Sequences

Two real sequences were also used to test the performance ofthe proposed algorithm. Figure 7(a) shows the result of the


(a) (b)

Figure 7: Result of real sequences: (a) SRI trees and (b) corridor.

(a) (b)

Figure 8: Real-time result from hardware: (a) original frame and (b) coded optical flow.

SRI tree sequence, another standard benchmark sequence fortesting optical flow algorithms based on captured image data.It can be seen that the proposed algorithm performed verywell on this sequence. Optical flow vectors were generallyvery smooth except for some few noisy vectors along themotion boundary. This was caused by the violation of theassumption that the motion should be constant in a localneighborhood which is another shortcoming of all opticalflow and motion estimation algorithms.

Figure 7(b) shows the result of the corridor sequence, animage sequence consisting of around 500 frames captured inour lab. As can be seen, there are noisy motion vectors in thebackground area which has less texture than other regions. Inthis case, even stronger smoothing would help suppress thenoise and “infer” the velocity from the local neighborhoods.

The data width transferred between the Helios board andGUI was 8 bits per pixel. To transfer the optical flow vectorvalues, we defined the data as a 4.4 format signed fixed pointnumber which means it had 1 sign bit, 3 integer bits, and 4fraction bits. Therefore, small optical flow vectors are codedwith gray intensity values. Large positive optical flow vectors(object moves down or to the right) are coded with brightintensity values. Similarly, large negative optical flow vectors(object moves up or to the left) are coded with dark intensityvalues. This coding mechanism was for the display of themotion field in real time for debugging purpose.

Figure 8(a) shows the screenshot of an original videoframe that was captured using our hardware platform,transferred to the PC through the USB interface, anddisplayed in the GUI. The palm moved downwards from the

top of the screen at a steady speed. Figure 8(b) shows themotion field (mostly in y direction). Since the palm movingdownwards, the resulting optical flow vectors are coded withlow intensity as explained previously. The light gray intensityvalues in the background represent very small motion thatwas caused by the flickering of the fluorescent lights in thelab and the digitization noise from the camera.

5. Conclusion

High computation and low power consumption require-ments make it difficult to use general purpose processorsor GPU to implement optical flow algorithms for real-time small unmanned vehicle applications. A hardwareaccelerated design of a tensor-based optical flow algorithmhas been developed and implemented in an FPGA platform.Two synthetic sequences and two real sequences were used totest its effectiveness. The accuracy on the Yosemite sequencewas shown to be substantially better than our previous designand other hardware implementation found in the literature.

The accuracy improvement in this design is due to tworeasons.

(1) A better hardware platform is used in this design tobe able to compute a more complex algorithm.

(2) Temporal smoothing is incorporated into the cal-culation. Temporal smoothing increases the tem-poral coherency, provides stronger smoothing, andimproves performance.


To accommodate temporal smoothing, the pipelinedhardware structure is divided into two parts: DER and OFCmodules. Accordingly, the amount of image data that mustbe stored and transferred increases dramatically. Currently,this design is running slower (15 fps of 640× 480 image) thanour previous design. The limiting factor of the processingspeed as well as the accuracy is the PLB bus bandwidth usage.This limiting factor can be alleviated by reducing the imagesize or increasing the SRAM size. Processing speed couldreach 60 fps if the image size is reduced to 320× 240. Increas-ing the SRAM size could avoid the storing and retrieving ofderivative frames from the slower SDRAM through the PLBbus. Also, increasing the SRAM size would allow the use of alarger temporal smoothing kernel which would significantlyimprove the optical flow computation accuracy.

Our future work includes improving the hardwarestructure so that the design uses less bus bandwidth andthereby is able to achieve a higher frame rate. Another goalis to develop navigation algorithms for obstacle avoidanceand implement the whole design on a UGV. Once this goalhas been achieved, we will look into the hallway or canyonnavigation applications for UAV.

Acknowledgment

This work was supported in part by David and DeborahHuber.

References

[1] Y. Mizukami and K. Tadamura, “Optical flow computationon compute unified device architecture,” in Proceedings of the14th International Conference on Image Analysis and Processing(ICIAP ’07), pp. 179–184, Modena, Italy, September 2007.

[2] R. Strzodka and C. Garbe, “Real-time motion estimation andvisualization on graphics cards,” in Proceedings of the 15thIEEE Visualization Conference (VIS ’04), pp. 545–552, Austin,Tex, USA, October 2004.

[3] J. Chase, B. Nelson, J. Bodily, Z. Wei, and D.-J. Lee, “FPGAand GPU architectures for real-time optical flow calculations,”in Proceedings of the IEEE Symposium on Field-ProgrammableCustom Computing Machines, Palo Alto, Calif, USA, April2008.

[4] M. V. Correia and A. C. Campilho, “Real-time implementationof an optical flow algorithm,” in Proceedings of the 16thInternational Conference on Pattern Recognition (ICPR ’02),vol. 4, pp. 247–250, Quebec City, Canada, August 2002.

[5] A. Zuloaga, J. L. Martın, and J. Ezquerra, “Hardware architec-tural for optical flow estimation in real time,” in Proceedingsof the IEEE International Conference on Image Processing (ICIP’98), vol. 3, pp. 972–976, Chicago, Ill, USA, October 1998.

[6] J. L. Martın, A. Zuloaga, C. Cuadrado, J. Lazaro, and U.Bidarte, “Hardware implementation of optical flow constraintequation using FPGAs,” Computer Vision and Image Under-standing, vol. 98, no. 3, pp. 462–490, 2005.

[7] J. C. Sosa, J. A. Boluda, F. Pardo, and R. Gomez-Fabela,“Change-driven data flow image processing architecture foroptical flow computation,” Journal of Real-Time Image Process-ing, vol. 2, no. 4, pp. 259–270, 2007.

[8] J. Dıaz, E. Ros, S. Mota, and R. Rodriguez-Gomez, “FPGA-based architecture for motion sequence extraction,” Interna-

tional Journal of Electronics, vol. 94, no. 5, pp. 435–450, 2007.[9] J. Dıaz, E. Ros, F. Pelayo, E. M. Ortigosa, and S. Mota, “FPGA-

based real-time optical-flow system,” IEEE Transactions onCircuits and Systems for Video Technology, vol. 16, no. 2, pp.274–279, 2006.

[10] P. C. Arribas and F. M. H. Macia, “FPGA implementation ofcamus correlation optical flow algorithm for real time images,”in Proceedings of the 14th International Conference on VisionInterface (VI ’01), pp. 32–38, Ottawa, Canada, June 2001.

[11] H. Niitsuma and T. Maruyama, “High speed computationof the optical flow,” in Proceedings of the 13th InternationalConference on Image Analysis and Processing (ICIAP ’05),vol. 3617 of Lecture Notes in Computer Science, pp. 287–295,Cagliari, Italy, September 2005.

[12] Z. Wei, D.-J. Lee, B. Nelson, and M. Martineau, “A fast andaccurate tensor-based optical flow algorithm implemented inFPGA,” in Proceedings of the IEEE Workshop on Applicationsof Computer Vision (WACV ’07), p. 18, Austin, Tex, USA,February 2007.

[13] Z. Wei, D.-J. Lee, and B. E. Nelson, “FPGA-based real-timeoptical flow algorithm design and implementation,” Journal ofMultimedia, vol. 2, no. 5, pp. 38–45, 2007.

[14] B. K. P. Horn and B. G. Schunck, “Determining optical flow,”Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981.

[15] B. D. Lucas and T. Kanade, “An iterative image registrationtechnique with an application to stereo vision,” in Proceedingsof the DARPA Image Understanding Workshop, pp. 121–130,Washington, DC, USA, April 1981.

[16] G. Farneback, “Very high accuracy velocity estimation usingorientation tensors, parametric motion, and simultaneoussegmentation of the motion field,” in Proceedings of the 8thIEEE International Conference on Computer Vision (ICCV ’01),vol. 1, pp. 171–177, Vancouver, Canada, July 2001.

[17] G. Farneback, “Fast and accurate motion estimation usingorientation tensors andparametric motion models,” in Pro-ceedings of the 15th IEEE International Conference on PatternRecognition (ICPR ’00), vol. 1, pp. 135–139, Barcelona, Spain,September 2000.

[18] B. Jahne, H. Haussecker, H. Scharr, H. Spies, D. Schmundt,and U. Schur, “Study of dynamical processes with tensor-basedspatiotemporal image processing techniques,” in Proceedings ofthe 5th European Conference on Computer Vision (ECCV ’98),vol. 2, pp. 322–336, Freiburg, Germany, June 1998.

[19] G. Farneback, “Orientation estimation based on weightedprojection onto quadratic polynomials,” in Proceedings of theConference on Vision, Modeling, and Visualization, pp. 89–96,Saarbrucken, Germany, November 2000.

[20] B. Johansson and G. Farneback, “A theoretical comparisonof different orientation tensors,” in Proceedings of the SSAB02Symposium on Image Analysis, pp. 69–73, Lund, Sweden,March 2002.

[21] H. Haussecker and H. Spies, Handbook of Computer Vision andApplication. Vol 2, chapter 13, Academic Press, New York, NY,USA, 1999.

[22] Robotic Vision Lab, Brigham Young University,http://www.ee.byu.edu/roboticvision/helios.

[23] W. S. Fife and J. K. Archibald, “Reconfigurable on-boardvision processing for small autonomous vehicles,” EURASIPJournal of Embedded Systems, vol. 2007, Article ID 80141, 14pages, 2007.

http://www.ee.byu.edu/roboticvision/helios

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttp://www.hindawi.com Volume 2010

RoboticsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation http://www.hindawi.com

Journal ofEngineeringVolume 2014

Submit your manuscripts athttp://www.hindawi.com

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

SensorsJournal of


Modelling & Simulation in EngineeringHindawi Publishing Corporation http://www.hindawi.com Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks


research article...

Documents