integrating a computational model and a run time system for image

Abstract

Recently substantial research has been devoted toUnmanned Aerial Vehicles (UAVs). One of a UAV’s mostdemanding subsystem is vision. The vision subsystem mustdynamically combine different algorithms as the UAV’sgoal and surrounding chang. To fully utilize the availablehardware, a run time system must be able to vary the qual-ity and the size of regions the algorithms are applied to, asthe number of image processing tasks changes. To allowthis the run time system and the underlying computationalmodel must be integrated. In this paper we present a com-putational model suitable for integration with a run timesystem. The computational model is called Image Process-ing Data Flow Graph (IP-DFG). IP-DFG has beendeveloped for modeling of complex image processing algo-rithms. IP-DFG is based on data ßow graphs, but has beenextended with hierarchy and new rules for token consump-tion, which makes the computational model more ß exibleand more suitable for human interaction. In this paper wealso show that IP-DFGs are suitable for modelling expres-sions, including data dependent decisions and iterations,which are common in complex image processingalgorithms.

1. Introduction

Substantial research has recently been devoted to develop-ment of Unmanned Aerial Vehicles (UAVs) [1, 2, 3, 4]. AUAV is a complex and challenging system to develop. Itoperates autonomously in unknown and dynamicallychanging environment. This requires different types of sub-systems to cooperate. For example, the subsystemresponsible for planning and plan execution base its deci-sions on information derived in the vision subsystem. Thevision system, on the other hand, decides which image pro-

cessing algorithms to run based on expectations of thesurrounding, information which the planning and plan exe-cution subsystem derives. To simplify this cooperation it isimportant that the subsystems are ß exible.

We currently develop a UAV in the WITAS UAV project[1]. This is a long term research project covering manyareas relevant for UAV development. Within the projectboth basic research and applied research is done. As part ofthe project a prototype UAV is developed. The UAV plat-form is a mini helicopter and we are looking at scenariosinvolving trafÞc supervision and surveillance missions.The helicopter currently used in the project for experimen-tation is depicted in Þgure 1. On the side of the helicopterthe onboard computer system is mounted and beneath thehelicopter the camera gimbal is mounted. The project is acooperation between several universities in Europe, theUSA, and South America. The research groups participat-ing in the WITAS UAV project are actively researchingtopics including, but not limited to, knowledge representa-tion, planning, reasoning, computer vision, sensor fusion,helicopter modeling, helicopter control, human interactionby dialog. See [1] for a more complete description of theactivities within the WITAS UAV project.

One of the most important information sources for theWITAS UAV is vision. In this subsystem symbolic infor-mation is derived from a stream of images. There exists a

This work is part of the WITAS UAV project and is Þnanced by the Wal-lenberg Foundation.

Figure 1. The mini-helicopter used for experimentsin the WITAS UAV project.

Integrating a Computational Model and a Run Time System for Image Processing on a UAV

Per Andersson and Krzysztof KuchcinskiLunds University

P.O. Box 118SE-221 00 Lund

Sweden{Per.Andersson, Krzysztof.Kuchcinski}@cs.lth.se

Klas Nordberg and Patrick DohertyLinköping University

S-581 83 LINKÖPING Sweden

[email protected] and [email protected]

Proceedings of the Euromicro Symposium on Digital System Design (DSD’02) 0-7695-1790-0/02 $17.00 © 2002 IEEE

Authorized licensed use limited to: Linkoping Universitetsbibliotek. Downloaded on March 17, 2009 at 05:58 from IEEE Xplore. Restrictions apply.

large set of different image processing algorithms that aresuited for different purposes and different conditions. It isdesirable for a UAV to be able to combine different algo-rithms as its surrounding and goal changes. To allow this aß exible model for the image processing algorithmstogether with a run time system that manages the combina-tion and execution of the algorithms is needed. Furthermore a run time system needs to represent image process-ing algorithms in a model with a clear semantic deÞ nition.This would make it possible to do both static (of-line) anddynamic (run-time) optimizations. Optimizations makes itpossible to utilize the available processing power optimallyby varying the quality and the size of the regions an algo-rithm is applied to, as the number of image processing tasksvaries.

One such model commonly used for image processingalgorithms is Synchronous Data Flow Graphs (Synchro-nous DFGs). However, this model can only represent staticbehavior. Another computational model is boolean DFG.Boolean DFGs can model dynamic behavior, but are hardto use. To overcome the mentioned problems we introducein this paper a new model of computation aimed at highlevel descriptions of image processing algorithms.

2. Related work

Several systems for modeling signal processing or systemsdedicated for image processing algorithms have beendeveloped over the years [11, 12, 13]. Early work laid thefoundation by developing data ßow models of computa-tion. Based on the token ßow model several systems formodeling or designing signal processing in general orimage processing is particular have been developed.

Data ßow graphs are a special case of a Kahn processnetworks [5]. A DFG is a directed graph where nodes rep-resent processes and arcs represent communicationchannels. The communication channels are directedunbounded FIFO queues, where reads are blocking andwrites are non-blocking. A DFG is executed by repeatedÞring of actors (nodes in the graph). The Þrings are datadriven. Each actor has a set of Þring rules, each specifyinga pattern of tokens on the input channels. A pattern specifythe number of tokens needed on each channel for the actorto Þre. The tokens are removed from the input channels asthe actor Þres. A pattern can also specify a value for a spe-ciÞc token. For example an actor with three inputs can havetwo Þring rules with the patterns {[true], [*],

⊥

} and{[false],

⊥

, [*]}.

⊥

is called bottom and represent the emptysequence of tokens, hence is always matched. * is a wild-card and is matched by any token. This actor will consumeone token on either its second or third input, depending onthe value of the token on its Þrst input. The actor is a deter-ministic merge.

Synchronous DFGs are a special case of DFGs, wherean actor consumes and produces a Þ xed number of tokenseach time it Þres, i.e. it has one Þring rule. To avoid tokenaccumulation all tokens produced must be consumed. Thisleads to linear system of equations called the balancingequation. From a solution of the balancing equation a staticschedule and bound on the buffer size needed for the com-munication channels can easily be derived. See [5, 6] for adetailed description of synchronous DFGs.

Synchronous DFGs can only model static behavior. Torelax this while still preserving some analyzability booleandata ßow graphs was introduced [7]. A boolean DFG actorhas one input called the control input. This input receivesboolean tokens. When a boolean DFG actor Þres the num-ber of tokens consumed and produces is a function of thevalue of the control token. This is equivalent to two Þ ringrules, one with the pattern

true

and one with the pattern

false

for the control input and a Þ xed number of wildcardson the other inputs. A systematic approach for consistencyanalysis of boolean DFGs is presented in [8].

DFGs are powerful and can be used to model compleximage processing algorithms. However their structure arecomplex and working directly with DFGs is error prone.Synchronous DFGs are much nicer to work with, but theycan not model dynamic behavior which is needed in morecomplex image processing algorithms. Boolean DFGsextends synchronous DFGs with dynamic behavior, butthey are cumbersome to use for representing recursivealgorithms. Hence they are not suitable for modeling ofalgorithms containing iterative approximations.

3. Integrating a computational model and a run time system

By integrating the run time system and the underlying com-putational model it is possible to consider different aspectsat different levels of granularity. Consider the general ori-entation algorithm shown in Þgure 2. For an algorithm likethis the application typically implies a timing requirementfor the whole algorithm, hence it is preferable to specifytiming constraints at the algorithm level of granularity.Other aspects are better to consider at a Þner level of gran-ularity. If the algorithm is to be executed in a parallelarchitecture the distribution should be done at node leveland the distribution algorithm should balance the amountof computation on each processing unit and the communi-cation between them. Similarly a scheduler shouldschedule the nodes individually which would make it pos-sible to minimize the amount of live data. This couldreduce the memory requirements considerably [14, 15].The information needed for this type of optimization couldbe derived automatically if the computational model wascarefully designed and integrated with the run time system.



This approach would be more ß exible and offer more con-trol over execution than using a traditional thread modeland implement algorithm using concurrent threads.

When deciding on a computational model it is not sufÞ -cient only to consider the interaction with the run timesystem. One also have to consider what expression poweris needed for the intended applications. For the vision sub-system of the WITAS UAV project the applications areimage processing. Many image processing algorithms aresimple. They are composed of well known operationsapplied to the whole image. The output of one operation isthe input to another. To illustrate consider the orientationalgorithm shown in Þgure 2. The input image is convolvedwith four different kernels resulting in four temporaryimages. The features extracted in the temporary images arethen combined on pixel level using a binary tree structure.

All algorithms are not as simple as the orientation algo-rithms described above. Consider a camera controlalgorithm for a UAV. The camera it to be pointed at an areaof interest. The camera should point to this area indepen-dent of the helicopter movement. This can be done bytracking a number of Þxpoints in the image [16]. A Þ xpointis a point that is easy to track. If one Þxpoint is out of sight,i.e. hidden, the camera control algorithm should Þ ndanother Þxpoint. To describe this behavior a model of com-putation that allows data dependent decisions is needed.

Another type of image processing algorithms whichneed to be considered are recursive algorithms, commonlyiterative approximation algorithms. These algorithms itera-tively improve their result until some conditions aresatisÞed. The conditions could be that the result is sufÞ -ciently accurate, i.e. a value exceeds a threshold, or that amaximum number of iterations has been done.

4. IPAPI - a run time system

Early in the WITAS UAV project it was clear that the plan-ning and plan execution subsystem would need to accessand guide the image processing subsystem. For this pur-pose an Image Processing Application Program Interface(IPAPI) was deÞned. IPAPI has evolved from a simple runtime system with static behavior to a run time system withdynamic behavior, where the planning and plan executionsubsystem conÞgures the set of algorithms to be executedbased on what symbolic information it needs to extractfrom its surrounding.

IPAPI has a library with implementations of differentalgorithms. An algorithm is executed as long as its resultsare needed. The creation, execution and removal of algo-rithms is managed by IPAPI. IPAPI also updates the VisualObject data Base (VOB) with the result of the algorithms.The VOB contains one entry for each dynamic object theUAV keeps track of in its surrounding, The VOB is theUAV’s view of the world. Other subsystem access imageprocessing result from the VOB.

Internally the image processing algorithms are repre-sented using a DFG based model. This representation isdescribed later in this paper. IPAPI has functionality fordynamic creation and removal of graphs. It dynamicallymanages the size of temporary data in the algorithms i.e.buffers for images. The planning and plan execution sub-system sets the size of the region of either the input or theoutput of a graph. IPAPI then propagates the size throughthe graph and allocates memory for buffers needed duringexecution. When the size of the regions propagates throughthe graph operations that affect the size, i.e. the result is theintersection of the input regions, are accounted for. IPAPIalso manages the execution of the graphs.

IPAPI has a type system. For each input and output of anactor a type is speciÞed. The types are checked when inputsare connected to outputs during the creation of graphs. Thetyping of inputs and outputs of actors makes it possible forthe run time system to automatically add conversionbetween graphs, for example an image with pixels encodedas integers can be converted to an image with pixelsencoded as ß oating points.

5. IP-DFG - a computational model

We had two goals when developing the computationalmodel for IPAPI. First the model should be simple forhumans to work with yet it should have sufÞcient expres-sion power to allow seamless modeling of complex imageprocessing algorithms. The second goal was to Þnd a modelthat is suitable for integration with a ß exible run time sys-tem such as IPAPI. The model should simplify dynamiccombinations of different algorithms. It should allow

conv(k1)

conv(k3)

conv(k2)

conv(k4)

op1

op1

op2

Figure 2. Orientation, a simple image processingalgorithm



dynamic change of region sizes and the inherited parallel-ism of an algorithm should be preserved. Since there is noneed to describe non determinism in image processingalgorithms the model should only allow deterministicbehavior.

DFGs are suitable for integrating with a run time sys-tem. Using data dependent Þring rules one can createcomplex control structures. However DFGs are cumber-some to use for these functions. There are two majorobstacles, deadlock and token accumulation. The problemof deadlocks can be reduced by using a tagged token ßowmodel. In a tagged token ßow models tokens are taggedwhich makes it possible to Þre actors out of order whilepreserving the implicit synchronization of tokens [9, 10].Token accumulation is illustrated in Þgure 3. If the streamsof the control input (horizontal arc) of

select

and

merge

areidentical the graph in Þgure 3a represents two mutuallyexclusive calculations (an

if then else

statement). If the con-trol input to

merge

is the negation of the control input to

switch

, as in Þgure 3b, the behavior is quite different. Con-sider the input sequences for select to be {true, true, true,false} and {x

0

, x

1

, x

2

, x

3

}. First switch Þres and output x

0

on the

f

branch. The actor

f

Þres and outputs f(x

0

).

Merge

receives false (the negation of

selects

input) on its controlinput. It can not Þre because the matching Þring rule needsone token on the

g

branch. This will not happen until

switch

has Þred four times. At this time x

3

will be sent on the

g

branch. The output sequence of the graph will be {g(x

3

),f(x

0

)} and several tokens are still remaining in the graph,but no Þ ring rules are matched.

Boolean DFGs can express data dependent conditionsand iterations needed by complex algorithms. However it iscumbersome to express some complex algorithms usingboolean DFGs. Therefore we have developed a new variantof DFGs targeted for complex image processing algorithmsand more suitable for humans to work with. We call thiscomputational model Image Processing Data Flow Graph

(IP-DFG) and we have used it for integration with IPAPI.IP-DFG is based on boolean DFG, but has been

extended with hierarchy and new rules for token consump-tion when an actor Þres. This makes the model moreß exible. IP-DFG has the same concept, as boolean DFG, ofone control input and two Þring rules for each actor. Thevalue of the control token decides which Þring rule is usedand the Þring rule determines the number of tokens con-sumed and produced.

As in DFGs tokens are stored on the arcs in FIFO order.However in IP-DFG an actor do not need to remove alltokens matching the Þring rule from the input. A Þring ruleconsists of a token pattern and a number for each input tothe actor. The token pattern indicates which tokens need tobe present for the actor to Þre and the number indicate howmany tokens should be removed from the inputs when itÞres. If an actor has a pattern with

n

tokens and

m

tokensare removed when it Þres the actor is said to read

n

tokensand consume

m

. This simpliÞes the implementation offunctions that use the same token in several Þrings, forexample sliding average. Sliding average over

n

tokensreads

n

tokens but consume only 1.

n

-1 tokens remains inthe FIFO-queue of the input so they can be read the nexttime the actor Þres. This behavior is common for imageprocessing algorithms that extract dynamic features in animage sequence, for example optical ßow.

5.1. Hierarchy

An IP-DFG is composed of boolean DFGs in a hierarchicalway. An actor can encapsulate an acyclic boolean DFG.Such actor is called a hierarchical actor. Since the internalgraph of a hierarchal actor is acyclic it can not model iter-ations. Instead in IP-DFG iterations are explicitly deÞ ned.This makes the model more suitable for human interaction,see section 5.2. In an IP-DFG a hierarchical actor is no dif-ferent from any other boolean DFG actor. It has two Þ ringrules and when it Þres it consumes and produces tokensaccording to the matching Þring rule. Internally the Þring isdivided into three steps. The three steps perform

initializa-tion

,

execution

and

result transmission

. For initializationand result transmission a

token mapping function

is used. Atoken mapping function maps a set of token sequences toanother set of token sequences, .A token mapping function is usually simple, i.e. a resultsequence is identical to a source sequence. However it ispossible to perform more complex operations in a tokenmapping function, for example split one sequence or mergeor concatenate two sequences. The token sequences in atoken mapping function always have a Þnite length in con-trast to DFGs which are working on inÞnite tokensequences. Token mapping functions are described in sec-tion 5.3.

Figure 3. A DFG example

select

merge

gf not

a) b)

select

merge

gf

S0 … Sn, ,{ } S0 … Sm, ,{ }→



A hierarchical actor is stateless. To guarantee this theÞrst step of the Þring of a hierarchical actor is to generatethe initial state of all internal arcs. The contents of the arcsare created from the tokens read by the hierarchical actor asit is Þred. This is done according to a token mapping func-tion. When a hierarchical actor with

n

inputs and

m

internalarcs Þres a sequence of tokens, , are read from eachinput,

i

j

. The token mapping function maps the set of allinput sequences, , to a set of sequences

. The sequence is used as the initial state ofthe FIFO queue on arc . A token mapping function canalso create new tokens from constants. There is one tokenmapping function for each Þ ring rule.

The second step is the execution of the internal booleanDFG. This is done according to the Þring rules of the actorsin the internal boolean DFG. When no internal Þring rulesare matched the boolean DFG is blocked and the secondstep ends.

In the third and Þnal step the result tokens are mappedto the outputs of the hierarchical actor. This is done accord-ing to a token mapping function. When the internal booleanDFG is blocked there is a sequence of tokens, , on eacharc . The token mapping function maps the set of all thesequences of the internal arcs, , to a set ofsequences . Each sequence is transmittedon the output . After the token mapping function is per-formed all tokens remaining in the internal boolean DFGare dead. The initialization mapping guarantees that theywill not be used in the next Þring of the hierarchical actorand the run time system can reclaim the memory used bythese tokens. This also prevents token accumulation.

To illustrate consider the camera control algorithm out-lined in section 3. The algorithm tracks a constant numberof Þxpoints in the images and based on this aims the cam-era to the area of interest. If a Þxpoint is no longer visibleit is replaced by a new visible Þxpoint. Figure 4 shows asub-function in the algorithm. The sub-function is called

pointLocator

and Þnds one Þxpoint with a given signaturein a given image, or if the Þxpoint is not found selects a newone. The result of

pointLocator

is an updated or new Þ x-point signature and corresponding parameters used in thecamera control loop.

PointLocator

consume/produce onetoken on each input/output when it Þres. The internal actorshas the following Þring rules (inputs are ordered counterclockwise starting with the the upper-most left-most one):

•

Þ nd

: {[*], [*]}•

select new:

{[*], [true]}, {

⊥

, [false]}•

merge:

{[*], [true],

⊥

}, {

⊥

, [false], [*]}The Þring of

pointLocator

starts with the initializationstep. There are two token sequences,

S

Þ xpoint

and

S

image

.Both contain one token. According to the token mapping

function the sequence

S

Þ xpoint

is placed in the FIFO queueof the internal arc

f

and the sequence

S

image

is placed in theFIFO queue of

i

. After this the internal boolean DFG is exe-cuted. The Actor

Þ nd

Þ res Þrst, since it is the only actorwith a matched Þring rule. It generates one token on eachof its outputs. The gray arc is connected to the control inputof

select new

and

merge

. The generated control token is

true

if the Þxpoint was found or else

false

. If the token istrue both

select new

and

merge

can Þre. In this case

selectnew

do not generate any token and

merge

forwards the Þ x-point token from

Þ nd

. If the Þxpoint was not found only

select new

can Þ re.

Select new

now selects a new Þ xpointand send it to its output.

Merge

forwards this token to the

f ’

arc. Now the boolean DFG is blocked and the result map-ping starts. The tokens on the internal arcs

f ’

and

p

aretransferred to the outputs

Þ xpoint’

and

pos

. In this examplethe actor

select new

Þred independent of whether a new Þ x-point needs to be selected. It is important to understand thatit is only in one of these Þrings that the actor do any work.The purpose of the second Þring rule is only to removetokens from the control input. If the run time system knowsthat the actor is stateless it can chose not to Þre an actor thatdo not produce any output. Instead the tokens matching theÞring rule are discarded. Also note that if the Þxpoint wasnot found the Þxpoint token from

Þ nd

to

merge

will not beconsumed. This is not a problem because the next time

pointLocator

Þres it will be removed during initialization.In fact a run time system should reclaim this token directlywhen

pointLocator

is blocked.From this example it seems unnecessary to have a token

mapping function for initialization and transferring theresult to the outputs. It would be simpler to directly connectthe internal arcs with the FIFO queues of the surroundingarcs. However this separation allows an actor to create con-stant tokens during the initialization step and it simpliÞ es

Si j

Si1… Sin

,{ , }Sa1

… Sam,{ , } Sal

al

Sal

alSa1

… Sam,{ , }

So1… Som

,{ , } Sol

Ol

Figure 4. A hierarchical actor for tracking a Þxpoint. Ifthe Þxpoint is not visible a new Þxpoint isselected.

find

select new

fixpoint fixpoint’

merge

f

i

f’

init mapping:f = fixpointi = image

result mapping:fixpoint’ = f’pos = p

image

pointLocator

p

pos



modeling of iterative functions as described below.

5.2. Iteration

There are two types of iterations of interest when dealingwith image processing, iteration over data and tail recur-sion. In iteration over data the same function is applied onindependent data. In a computational model based on thetoken ßow model this is achieved by placing several tokens,each containing one quantum of data, in the graph. Theactors will Þre appropriate number of times. To use the

pointLocator

shown in Þgure 4 for tracking ten Þ xpointsone simply place ten tokens containing the signature on the

Þ xpoint

arc and ten images on the

image

arc,

pointLocator

will then Þre ten times. This implementation tracks differ-ent Þxpoints in different images. For tracking severalÞxpoint in the same image a more suitable approach is tochange the Þring rules. By setting the Þring rule of

point-Locator

to consume ten tokens on the

Þ xpoint

input and oneon the

image

input the proper tokens are placed on the

f

and

i

internal arcs during the initialization mapping. The Þ ringrule of the internal actors

Þ nd

and

select new

must also bechanged. They should read one token and consume zerotokens from the

i

arc. The image token on the internal arc

i

will be removed after the result mapping, so

pointLocator

will behave correct the next time it Þ res.Tail recursion is an important type of iteration com-

monly used for iterative approximations. In tail recursionthe result of one invocation of a function is used as argu-ment for the next invocation of the same function. Tailrecursion is equivalent to iteration and in the rest of thispaper we will use the term iteration. In IP-DFGs iterationis implemented using hierarchical actors. When a hierar-chical actor fires the internal graph can be executed severaltimes. For modeling of iteration we have extend hierarchi-cal actors with a

termination arc

and an

iteration mappingfunction

. Such actors are called iterative actors. When an iterative actor Þres it is initialized and the

internal boolean DFG is executed once as a hierarchicalactor. As part of the execution a termination condition isevaluated, resulting in a boolean token on the

terminatearc

. The

terminate arc

is a special arc in the internal bool-ean DFG of an iterative actor. If the last token on the

terminate arc

is false when the internal boolean DFG isblocked the internal boolean DFG is to be executed again.Before the repeated execution starts the internal booleanDFG must be transformed from its blocked state, where noÞring rules are satisÞed to the initial state of the next itera-tion. This is done according to the

iteration mappingfunction

. The

iteration mapping function

generates a tokensequence for each arc in the internal graph from the tokensequences in the blocked graph. This mapping is deÞned bya token mapping function. Tokens not explicitly declared to

remain in the boolean DFG will be removed. This is toavoid token accumulation. The internal boolean DFG isexecuted until the termination condition is satisÞed. Whenthis happens result tokens are generated according to theoutput mapping as described in the section on hierarchicalactors.

The separation of the iteration body and token mappingfunctions allows a cleaner implementation of iterativefunctions. Consider the camera control algorithm from sec-tion 3. and 5.1. The core of the algorithm is to Þnd thecameras position and heading in the 3D world. This can bedone by using the Geographical Position System (GPS) andanchoring objects in an image from the camera with a geo-graphical information system. This is very computationalintensive, so it is preferable not to do this often. A less com-putational intensive approach is to track the camerasmovement in the 3D world. This can be done by tracking aset of points in the 2D image from the camera [16]. The cal-culations assume that all points are located on the groundplane. However this is not the case for all points in theworld, i.e. points at the top of a building. An algorithmbased on this approach should ignore points not behavingas if they was on the ground plane. An iterative actor whichdo this is shown in Þgure 5. All actors consume/produceone token on each input/output, except the

plane

input ofthe

new plane

actor, which reads one token, but the token isnot consumed. In each iteration one estimate of the groundplane is calculated. Also the distance between the estimatedplane and the Þxpoints are calculated. If there is one Þ x-point with a distance larger than a threshold then theÞxpoint with the largest distance is removed and a new iter-ation is to be done. The actor

new plane

calculates a newestimate of the ground plane from an earlier ground planeestimate and a set of current and earlier Þxpoints positions.The set of Þxpoints is stored in one token. The actor

resid-

Figure 5. A recursive algorithm for estimating theground plane relative to the camera.

init mapping:plan = oldPlaneFPPair = Þ xpoints

plane FPPair

iteration mapping:plane = planeFPPair = FPPair’

result mapping:newPlane = last(plane’, 0)

fixpoints

new

residual

plane

terminateFPPair’

oldPlane

plane’

newPlane



ual

Þnds the Þxpoint with the largest distance from theestimated ground plane. If this distance is larger than athreshold then this point is removed and a token with theremaining Þxpoints are sent to the

FPPair’

arc and a falsetoken is sent to the terminate arc, otherwise true are sent tothe terminate arc. The Þring of the iterative actor is simpleto follow. First in the initialization mapping function anearlier ground plane and a set of tracked Þxpoints aremapped to the internal boolean DFG. Next the internalboolean DFG is executed. The actor new plane Þ res Þ rstfollowed by residual, resulting in a blocked boolean DFG.If the estimate of the ground plane was based on fixpointnot on the plane then the last token generated on the termi-nate arc is false and the set of Þxpoints now believed to beon the ground plane is in the token on the FPPair’ arc. Theinternal boolean DFG is to be executed again and the iter-ation mapping function generates the initial state for thenext iteration. The token on the plane arc is to stay on thearc and the token with the set of Þxpoints on the PFPair’arc is mapped to the FPPair arc. Now the internal booleanDFG is executed again. This repeats until all Þxpoints issufÞciently close to the estimate of the ground plane,resulting in a true token on the terminate arc. The resultmapping function then maps the last estimated plane to thenewPlane output arc.

5.3. Token mapping functions

Token mapping functions are used in hierarchical and iter-ative actors to transfer tokens between the internal arcs andthe surrounding. They are also used in iterative actors tochange internal state between iterations. This can be seenas a mapping from one set of token sequences of Þ nallength to another set of token sequences of Þnite length,

. Each sequence is associatedwith one arc and the token sequence is the contents of thearcs FIFO queue. In a token mapping function a resultsequence is created from concatenations of sequences. Thesequences can be original sequences, new sequences or anempty sequence (⊥). New sequences are created from indi-vidual tokens from the source sequences and new tokenscreated from constants.

In a token mapping function a new token can also be cre-ated by wrapping a token sequence. The original tokensequence can be recreated by unwrapping token created bya wrapping. This makes it possible to treat some elementsas a set, but still have the possibility to apply functions tothe individual elements. Consider the plane estimate actorfor the camera control algorithm in Þgure 5. In the iterativeactor the Þxpoints are treated as a set, encapsulated in onetoken. However as part of the algorithm the distance foreach Þxpoint to a plane is to be calculated. This is done bythe hierarchical actor in Þgure 6. During the initialization

mapping the set of Þxpoints is unwrapped and placed as asequence of tokens on the fp arc. The distance actor will Þ reonce for each Þxpoint and the result mapping will laterwrap the sequence of distances to one token.

To preserve properties of a graph a token mapping func-tion is said to be monotonic if it preserves the relativeprecedence of all tokens. If token tk proceeds token tl in anyif the input sequences tk precedes tl in all result sequencesboth tokens are present in.

6. Experimental platform

Originally IPAPI was implemented and tested in the simu-lator developed for the WITAS UAV project. The WITASUAV simulator is composed of several components com-municating using real-time CORBA. The components ofthe simulator correspond to the subsystems of the onboardsystem in addition to components for simulating the envi-ronment, the physical behavior of the helicopter andrendering of 3D images. The rendered images are used asinput to the image processing algorithms. Currently thisimplementation is ported to the onboard hardware plat-form. During this porting the components are modiÞed tocoupe with the limitations of the onboard hardware whilemeeting constraints on timing and stability.

IPAPI has been implemented in Java which make it easyto move between different hardware platforms. To achievegood performance all image processing operations (actors)have been implemented in native methods. The onboardsystem has one PowerPC processor dedicated for imageprocessing. The native implementations of the image pro-cessing actors fully utilize the AltiVec unit of the PowerPCprocessor. The AltiVec unit is a vector processing unit with128 bit internal data-paths. It allows SIMD instructions on16, 8, or 4 parallel elements depending on the number ofbits per element (short, int, long or ßoating point). Ourexperiments show that this division of Java and native Cimplementation give a negligible overhead compared toimplementing the whole run time system in C. Java has the

S0 … Sn, ,{ } S0 … Sm, ,{ }→

init mapping:plan’ = planedp = unwrap(Þ x-points)

plane’ fp

result mapping:distance = wrap(dist)

fixpoints

distance

plane

distances

dist

Figure 6. A hierarchical actor for calculation of thedistance of all points in a set to one plane.



beneÞt of automatic memory management and stableexception handling while still meeting real time constraintsfor critical tasks [17].

7. Future work

In the WITAS UAV project we have encountered a need forhigher order functions. A higher order function is a func-tion that generates another function and in the context ofDFGs an actor that generates DFGs. This is similar to ahierarchical actor, but more powerful. Currently we areusing algorithms that is composed of calculations at differ-ent scales of an image and we need the number of scalelevels to be parametric. Today we solve this by allow a hier-archical actor to create its subgraph as it is being created.The creation of a subgraph is deÞned by a Java function,which creates the internal data structure used by IPAPIwhere the hierarchical actor Þres. To allow people notfamiliar with the internal structure of IPAPI to use thisfunctionality it is desirable to express higher order func-tions directly in IP-DFG.

Currently our onboard system for the WITAS helicopterhas one processor dedicated for image processing. ThusIPAPI is implemented for single processor. We plan toextend the onboard system with more processors in the nearfuture. For this purpose IPAPI will be extended with algo-rithms for distribution of image processing algorithms.

8. Acknowledgments

We would like to thank Johan Wiklund and the people inthe the computer Vision Laboratory at Linköping Univer-sity for their share of work with implementing IPAPI.

9. References

[1] P. Doherty et. al. “The WITAS Unmanned Aerial VehicleProject”, Proceedings of the 14th European Conference onArtiÞ cial Intelligence, 2000

[2] AUVS - The Association od Unmanned Vehicle Systems.http://www.auvsi.org/

[3] G. Buskey, G. Wyeth and J. Roberts. “Autonomoushelicopter hover using an artiÞcial neural network”,

Proceedings of Robotics and Automation 2001 ICRA. IEEEInternational Conference on, vol. 2, 2001

[4] J. Evans, G. Inalhan, Jung Soon Jang, R. Teo,C.J. Tomlin,“Dragonßy: a versatile UAV platform for the advancementof aircraft navigation and control”, Digital AvionicsSystems, 2001. DASC. 20th Conference, Volume: 1, 2001

[5] E. A. Lee and Thomas M. Parks. “Dataßow processnetworks”, Proceedings of the IEEE, vol. 83, no. 5, pp. 773-801, May, 1995

[6] E..A. Lee and David G. Messerschmitt. “Static Schedulingof Synchronous Data Flow Programs for Digital SignalProcessing”, IEEE transactions on computers vol. C-36,no1, January 1987

[7] J.T. Buck. “Scheduling dynamic dataßow graphs withbounded memory using the token ßow model”, Ph.D. thesisfrom University of California, 1993

[8] E. A. Lee. “Consistency in Dataßow Graphs”, IEEEtransactions on parallel and distributed system, vol.2 no. 2April 1991

[9] Arvind and K. P. Goestelow, “Some Relationships betweenAsynchronous Intepreters of a Dataßow Language”, FormalDescription of Programming Languages, IFIP WorkingGroup 2.2 1977

[10] Arvind and K. P. Goestelow, “The U-Intepreter”, IEEEComputer 15(2), February 1982

[11] E. A. Lee, “Overview of the Ptolemy Project,” TechnicalMemorandum UCB/ERL M01/11, University of California,Berkeley, March 6, 2001.

[12] C. S. Williams and J. R. Rasure, “A visual language forimage processing”, In Proc. 1990 IEEE Workshop on VisualLanguages, pages 86--91, 1990.

[13] R. Lauwereins, M. Engels, M. Ade and J.A. Peperstraete,“Grape-II: a system-level prototyping environment for DSPapplications”, IEEE Computer 28(2), February, 1995

[14] R. Szymanek, K. Kuchcinski, “Task Assignment andScheduling under Memory Constraints”, Euromicro, 2000

[15] P. K. Murthy and S. S Bhattacharyya, “Shared memoryimplementations of synchronous dataßow speciÞcations“,Proceedings of Design, Automation and Test in EuropeConference and Exhibition 2000, 2000

[16] P. Forssén, “Updating Camera Location and Heading Usinga Sparse Displacement Field”, technical report LinköpingUniversity LiTH-ISY-R-2318, November 2000

[17] A. Nilsson and T. Ekman, “Deterministic Java in TinyEmbedded Systems”, Proceedings of the fourth IEEEInternational Symposium on Object-Oriented Real-TimeDistributed Computing (ISORC), 2001



integrating a computational model and a run time system for image

Documents