fuzzy control for obstacle detection in object tracking

Mathematical and Computer Modelling 52 (2010) 1228–1236

Contents lists available at ScienceDirect

Mathematical and Computer Modelling

journal homepage: www.elsevier.com/locate/mcm

Fuzzy control for obstacle detection in object trackingEduardo Parrilla ∗, Jaime Riera, Juan-R. TorregrosaInstituto de Matemática Multidisciplinar, Universidad Politécnica de Valencia, Camino de Vera s/n, 46022 Valencia, Spain

a r t i c l e i n f o

Article history:Received 30 September 2009Accepted 5 February 2010

Keywords:Object trackingOptical flowOcclusionAdaptive filtersFuzzy controller

a b s t r a c t

In previous works, we have studied a fast and robust object tracking system based onoptical flow. To solve the occlusion problem, we have developed a combined trackingsystem based on optical flow and adaptive filters. The critical point of the system is thecoupling between optical flow and predictive algorithms. This coupling is governed byparameters such as tolerance, the absolute value of the difference between the value ofvelocity calculated by the optical flow and the estimated value. In this paper, we proposethe use of a fuzzy control system to solve this coupling problem between the differentvelocities. This technique will provide great robustness to the tracking algorithm.

© 2010 Elsevier Ltd. All rights reserved.

1. Introduction

Optical flow is an approximation to the 2D motion field of an image sequence, that is a projection of the 3D velocitiesof surface points onto the imaging surface. Optical flow computation is based on supposing that intensity variations ofthe image only depend on displacements of the objects, without considering other causes [1]. Formally, if I(x̄, t) =I(x(t), y(t), t) denotes the space–time intensity function of an image, the optical flow equation is

Ixu+ Iyv + It = 0, (1)

where Ix, Iy are the components of the spatial gradient ∇I , It denotes partial differentiation with respect to time andv = (u, v) denotes the components of the optical flow.For solving Eq. (1) it is necessary to consider additional restrictions because of the aperture problem.We have to compute

two components, u and v, and we only have one equation. There are many techniques that consider different restrictions tocalculate the optical flow. In this work, we have used a technique that was proposed by Lucas and Kanade in 1981 [2] andhas been successfully proved in different applications [3,4].On the other hand, an adaptive filter is a filter whose transfer function varies with time, as a function of the input signal.

Filter coefficients are unknown when the filter is designed, and they are calculated and updated according to an optimizingalgorithm. In this article, we use the recursive least squares (RLS) filter [5].We start from a signal xn, n = 0, . . . ,NT , and we assume for the signal an AR(p) model

xk = − (a1)k xk−1 − (a2)k xk−2 − · · · −(ap)k xk−p + εk. (2)

Assuming this model for k = p, . . . ,N , we obtain the system of equations

XN = CNaN + εN , (3)

∗ Corresponding author. Tel.: +34 963877000x75254; fax: +34 963879009.E-mail addresses: [email protected] (E. Parrilla), [email protected] (J. Riera), [email protected] (J.-R. Torregrosa).

0895-7177/$ – see front matter© 2010 Elsevier Ltd. All rights reserved.doi:10.1016/j.mcm.2010.02.014

http://www.elsevier.com/locate/mcm

http://www.elsevier.com/locate/mcm

mailto:[email protected]



http://dx.doi.org/10.1016/j.mcm.2010.02.014

E. Parrilla et al. / Mathematical and Computer Modelling 52 (2010) 1228–1236 1229

where

XN =

xpxp+1...xN

, CN =

xp−1 xp−2 · · · x0xp xp−1 · · · x1...

...xN−1 xN−2 · · · xN−p

,

aN =

− (a1)N− (a2)N...

−(ap)N

, εN =

εpεp+1...εN

.It can be demonstrated [6] that the coefficients

(aj)N can be obtained by using a recursive method. To use this recursive

least square (RLS) method, we first perform an initialization step, considering Nin samples of the signal and computing

PNin =[CTNinΛNCNin

]−1,

aNin = PNCTNinΛNinXNin ,

with

ΛN =

λN−p 0 · · · 0

0 λN−p−1...

......

0 · · · 1

,where 0 < λ ≤ 1 is a constant called the forgetting factor of the method.After this step, the following recursion is used

KN+1 =PNcN+1

λ+ cTN+1PNcN+1,

PN+1 =1λ

(PN − KN+1cTN+1PN

),

aN+1 = aN + KN+1(xN+1 − cTN+1aN

), (4)

where

cTN+1 =[xN , xN−1, . . . , xN−p+1

],

for N = Nin, . . . ,NT , obtaining in this way a new set of coefficients for the AR(p) model each time a new sample of the signalis considered. This kind of autoregressive method where the coefficients change in time is called a Dynamic Autoregressivemethod of order p, DAR(p) [7].In order to predict the velocities of the objects, we follow the next steps [8]:

1. We calculate velocities for Nin frames of each sequence by using Lucas and Kanade algorithm and we use this samples toinitialize the filter coefficients.

2. For the Nth frame, we calculate the velocity vN in the following way:a. We calculate vofN by using optical flow.

b. We estimate vafN by using an adaptive filter.

c. If |vofN − vafN | < tolN , then vN = v

ofN . Else vN = v

afN .

In a previous work [9], we have demonstrated that an optimum value of tolerance is

tolN = k |vN−1| . (5)

Tolerance values are a critical factor for the correct operation of the proposed algorithms. A very small value of thetolerance will cause the methods to select predicted velocities when there is no obstacle. On the other hand, a large valueof the tolerance will make that the methods can not detect any occlusion. For this reason, the choice of parameter k is verycritical. In this paper, we propose to use a variable value for parameter k controlled by a fuzzy system, instead of selectingits value a priori.

1230 E. Parrilla et al. / Mathematical and Computer Modelling 52 (2010) 1228–1236

Fuzzy controller

Inputsu(t)

Outputsy(t)

Process

Reference inputr(t)

Inferencemechanism

Rule-base Def

uzzi

fica

tion

Fuzz

ific

atio

nFig. 1. Fuzzy controller architecture.

Fig. 2. Membership functions. LN: large negative, MN: medium negative, S: small, MP: medium positive, LP: large positive.

2. Fuzzy controller

A fuzzy control system is a control system based on fuzzy logic [10]. Fuzzy control provides a formal methodology forrepresenting, manipulating and implementing a human’s heuristic knowledge about how to control a system [11]. The fuzzycontroller block diagram is given in Fig. 1, where we can see a fuzzy controller embedded in a closed-loop control system.To design a fuzzy controller, we have to specify the different blocks of the system:

• Input variables LHS.• Output variables RHS.• Input fuzzification method.• Fuzzy rules.• Inference engine parameters.• Defuzzification method.

FuzzificationThe function of the fuzzificator is the conversion of the real input x into a fuzzy set. To do this work, it is necessary to

define a set of membership functions. In Fig. 2, we can see an example of this set.The values of each membership function are labeled by µ (x) and determined by the input value x and the shape of each

membership function. In the example of Fig. 2, if we have an input value x = 2, the set of fuzzy variables will be:

µLN = 0µMN = 0µS = 0.6µMP = 0.4µLP = 0.

Fuzzy decision blocksFuzzy control requires a specification of the rules in terms of linguistic variables, that are conditional statements of type

IF-THEN. These rules relate input fuzzy values to output fuzzy value. A set of fuzzy rules Rl is:

Rl : IF x1 is F l1 and . . . and xn is Fln THEN y is G

l

where xi and y are input and output linguistic variables, and F li and Gl are fuzzy sets.


Membership

1.0

0.5

0

N S P

-2 -1 0 1 2y

Fig. 3. Membership output functions. N: negative, S: small, P: positive.

Membership

1.0

0.5

0

N S P

-2 -1 0

0.6

1 2y

µout (y)

0.4

Fig. 4. Global output.

In this case, we consider a systemwithmultiples inputs and one output. A systemwithmultiple outputs can be separatedin a set of simple fuzzy systems.Inference engineThe inference engine produces a global outputµout(y), by using the set of rules and considering themembership degree of

the inputwith eachmembership function. The global outputµout(y) is a fuzzy set that is calculated from a set ofmembershipfunctions for the output y.Continuing the previous example, if we have the two following rules:If x is small (S), Then y is small (S).If x is medium positive (MP), Then y is positive (P)

and we know that

µS = 0.6µMP = 0.4

by using the set of membership functions for the output y observed in Fig. 3, we will obtain the global output of the Fig. 4.Depending of the method of defuzzification, we can consider an only global output µout(y) as the union of all the

contributions, or a global output µkout(y) for each contribution.DefuzzificationThe defuzzification is a process to obtain a real value u∗ representative of µout(u). There are different methods for

defuzzification and method selection is very important for the efficiency and precision of the controller.The most used defuzzification techniques are:• COA o Center of Areas.

This technique calculates u∗ as the geometric center of the output fuzzy value µout(u), where µout(u) is thecombination of all the contributions. u∗ divides the area in two regionswith the same area. Some problems of thismethodare the computational cost, the preference for central values and that the overlapping area is only counted once.• COS o Center of Sums.

This method operates separately for each contribution. The overlapping areas are considered more than once. It isefficient and it is the most used method.• MOM o Mean of Maxima.

This technique calculates the mean value of points with maxima membership degree in the global output µout(u). Itis very fast, but it does not consider the shape of µout(u).


Fig. 5. Moving object.

200

150

100

50

00 20 40 60

Frame

|Vof

–Vaf

| (pp

f)

vel = 2 ppf200

150

100

50

00 20 40 60

Frame

|Vof

–Vaf

| (pp

f)

vel = 3 ppf200

150

100

50

00 20 40 60

Frame

|Vof

–Vaf

| (pp

f)

vel = 4 ppf

Fig. 6. Difference between optical flow and prediction.

10

8

6

4

010 20 30 40

Frame

|Vof

–Vaf

| (pp

f)

vel = 2 ppf vel = 3 ppf vel = 4 ppf

2

10

8

6

4

010 20 30 40

Frame

|Vof

–Vaf

| (pp

f)

2

10

8

6

4

010 20 30 40

Frame

|Vof

–Vaf

| (pp

f)

2

Fig. 7. Difference between optical flow and prediction (zoom).

• CA o Center Average.This method operates separately for each contribution and calculates a pondered average of the centers of the

contributions.

3. Obstacle analysis

Wewill analyze the occlusion problem by using the example observed in Fig. 5. It consists in an object with a horizontalmovement and a velocity vel pixels per frame.We have calculated |vofx − v

afx | for vel = 2, vel = 3 and vel = 4 pixels per frame (ppf) for the different frames of the

sequence. The results can be observed in Fig. 6. We can see that there are two different peaks for each value of vel thatcorresponds with the input and output frontiers of the obstacle. In these points there are two discontinuities that producea large error in the calculation of the optical flow. These peaks will always be detected. If we analyze what happens in theregion corresponding to the transition of the object through the obstacle, we obtain the Fig. 7.As we can see, the values of |vofx − v

afx | are very similar to the velocity values of the object because the optical flow

algorithm considers that obstacle is a static object so vofx,y = 0 in this region. In this way, we will use a variable value oftolerance in our algorithm for each frame N

tolN = k |vN−1| . (6)


NB M PB

–|Vn-1| |Vn-1|00

1

Δε

μ(Δε)

Fig. 8. Membership functions for the input1ε.

µ(k)

1

0

0.5 1k

L H

Fig. 9. Membership functions for the output k.

4. Obstacle detection

In our system, we want the value of tol to be small when there is an obstacle and large in any other case. The value of tolcan be controlled by the variable k. In this way, we have designed a fuzzy system to control the value of k.The system has two inputs,1ε and qV , and one output, k.The variable1ε is the increment error (EI). If we call error ε to the absolute value of the difference between the velocity

calculated by optical flow and the velocity estimated by the adaptive filter, we have that:

εn =∣∣vofn − vafn ∣∣ , (7)

1εn = εn − εn−1. (8)

The variable qV is a binary variable that indicates if we are using the velocities calculated by optical flow in the previousframes (qV = 1) or the ones estimated by the adaptive filter (qV = 0).In Fig. 8, we can see the set of membership functions used to the fuzzification of the input1ε. Fuzzy values for this input

can be negative big (NB), medium (M) or positive big (PB). NB values correspond to the final of the peaks that exist in thefrontiers of the obstacle as we can see in Fig. 6, PB values correspond to the beginning of these peaks andM value correspondto the other situations.In this system, the value of variable kmust be high (H), to select the velocity calculated by optical flow, when there is no

occlusion (EI =M, qV = 1) and when the target is in the output of the obstacle (EI= NB, qV = 0). In all the other cases, thevalue of kwill be low (L) to select the velocity estimated by the adaptive filter.In this way, the system fuzzy rules are the following:• IF EI is NB and qV = 0 THEN k is H.• IF EI is M and qV = 0 THEN k is L.• IF EI is PB and qV = 0 THEN k is L.• IF EI is NB and qV = 1 THEN k is L.• IF EI is M and qV = 1 THEN k is H.• IF EI is PB and qV = 1 THEN k is L.

The inference engine calculates the global output µout(k) by using these conditions and the set of output membershipfunctions that we can observe in Fig. 9. In this way, the value of the output variable k can oscillate between 0.5 and 1.Finally, the defuzzificator calculates from the global output µout(k) the real value for the variable k by using the center

of sums technique, that provides fast and satisfactory results.


Fig. 10. Object tracking in a synthetic sequence.

1

0.95

0.9

0.85

0.8

0.75

0.7

0.65

0.6

0.55

0.50 10 20 30 40 50 60 70 80 90 100

k

Frame

Fig. 11. k values in synthetic sequence.

5. Results

The algorithm exposed in this paper has been tested in different video sequences, studying synthetic and real trafficvideos. We have used windows of 7× 7 pixels for Lucas and Kanade algorithm, and a value of Nin = 7 frames, an order of 2and a forgetting factor of λ = 0.99 for the adaptive filter.In Fig. 10, we can observe different frames of a synthetic video sequence designed with 3D Studio Max r© that consists in

an object that follows a curved trajectory passing under an obstacle.In this example, the occlusion is produced in frames 37–43. As we can observe in Fig. 11, the value of k decreases in those

frames. In this way, the predicted velocity is selected while the target is hidden by the obstacle and the object is correctlytracked along all the sequence.The same result is obtained analyzing the example shown in Fig. 12. In this figure, we can see an urban route where there

is a partial vehicle occlusion because of a street light.In this case, the occlusion occurs in frames 21–27 and, as we can see in Fig. 13, k decreases in this position. On the other

hand, in this figure, we can observe that the value of k also decreases in frames 51–54. This change is due to the fact thatthere is an error in the optical flow calculation. The performance of the system in this point is very interesting since it is ableto correct possible errors in the optical flow calculation, besides handling occlusion.

6. Conclusions

In this article, we have analyzed the occlusion problem in object tracking in a video sequence. We have proposed the useof a fuzzy controller to combine the velocities calculated by an optical flow algorithm and the ones estimated by an adaptivefilter in order to predict the target movement and approximate its trajectory when the target disappears.


Fig. 12. Object tracking in a real sequence.

1

0.95

0.9

0.85

0.8

0.75

0.7

0.65

0.6

0.55

0.50 10 20 30 40 50 60 70

k

Frame

Fig. 13. k values in real sequence.

We have studied the effect that parameters values produce in the performance of the method, obtaining anoptimal system to control the tolerance value. This fuzzy system provides a great robustness and a low computationalcost.Finally, we have shown two examples that verify the efficiency of the algorithms, proving that this technique also can

detect and correct errors in the object tracking.


References

[1] B.K.P. Horn, B.G. Schunck, Determining optical flow, Artificial Intelligence 17 (1–3) (1981) 185–203.[2] B.D. Lukas, T. Kanade, An iterative image registration technique with an application to stereovision, in: Proceedings of Imaging UnderstandingWorkshop, 1981, pp. 121–130.

[3] F. Bourel, C.C. Chibelushi, A.A. Low, Recognition of facial expressions in the presence of occlusion, in: Proceedings of the Twelfth BritishMachine VisionConference, vol. 1, 2001, pp. 213–222.

[4] E. Parrilla, J. Riera, M.H. Giménez, J.R. Torregrosa, J.L. Hueso, Vehicle tracking in urban routes by means of Lucas & Kanade algorithm, in: Proceedingsof the International Conference on Computational and Mathematical Methods in Science and Engineering, 2005. pp. 438–445.

[5] L. Ljung, System Identification. Theory for the User, Prentice-Hall, Upper Saddle River, NJ, 1999.[6] K. Ogata, Discrete-Time Control Systems, Prentice Hall, Upper Saddle River, NJ, 1987.[7] P.C. Young, Nonstationary time series analysis and forecasting, Progress in Environmental Science 1 (1) (1999) 3–48.[8] E. Parrilla, J. Riera, J.R. Torregrosa, J.L. Hueso, Handling occlusion in object tracking in stereoscopic video sequences, Mathematical and ComputerModelling 50 (5–6) (2009) 823–830.

[9] E. Parrilla, D. Ginestar, J.L. Hueso, J. Riera, J.R. Torregrosa, Handling occlusion in optical flow algorithms for object tracking, Computers andMathematicswith Applications 56 (3) (2008) 733–742.

[10] Lotfi A. Zadeh, Fuzzy sets, Information and Control 8 (1965) 338–353.[11] K.M. Passino, S. Yurkovich, Fuzzy control, International Journal of General Systems 29 (2) (2000) 341–342.

fuzzy control for obstacle detection in object tracking

Documents