Inhibition of Returnin the Bayesian Strategy to Active Visual Search

Kai Welke, Tamim Asfour, Rudiger DillmannKarlsruhe Institute of Technology (KIT)

Institute for Anthropomatics, Humanoids and Intelligence Systems Lab{welke,asfour,ruediger.dillmann}@kit.edu

Abstract

The inhibition of return mechanism in systemsbased on visual attention allows to generate a sequenceof gaze directions (saccades) from saliency data. Inthis work we propose such a mechanism that seam-lessly integrates with the Bayesian Strategy to atten-tion. Taking into account a memory of attended loca-tions, the requirement for consistency constitutes thedrive for the generation of saccades. The general prob-abilistic model of active saliency is introduced, whichis applicable to systems based on saliency maps as wellas to landmark based systems. The model is furtherrefined and validated in the context of an active visualsearch task.

1 Introduction

The application of active camera systems is nowa-days common in state of the art humanoid platforms.The popularity of such active systems not only stemsfrom the resulting enhanced visual field but is also at-tributed to the fact that the gaze is an essential com-munication channel in human interaction. Robots thathave a similar ability of using their gaze as human aremore appealing and thus more likely to be accepted inhuman-centered environments.

The application of an active camera system raisesa wide range of problems. In this work we addressthe problem of directing the gaze of the active systemto interesting locations in the context of visual searchtasks. The problem of identifying and focusing on in-teresting locations in the scene is usually referred to asovert visual attention.

The work by Itti et al. [1] has been among the firstframeworks for the generation of attention on techni-cal systems. Their work focused on biological plausibleprocessing in the realization of bottom-up attentionalmechanisms. In the past years, several new approachesto the generation of attention have been proposed. Themost complete view has been provided by the BayesianStrategy which allows the integration of different as-pects of attention, such as bottom-up processing, top-down processing, and scene context ([2], [3]). In theBayesian Strategy, saliency is generated according tothe probability of detecting an object O at a spatial lo-cation X given the current visual input J . As shown in[4] the factorization of P (O = 1, X|J) allows to seam-lessly integrated the different aspects of attention, eachdescribed by one factor. While this strategy allows togenerate saliency data, it does not solve the problem ofgenerating gaze sequences from the saliency data in anintegrated manner. The question remains: What couldbe the mechanism that allows to determine a sequenceof distinct visual locations which are to be fixated by

the active system? In this work we propose and evalu-ate a novel model that allows to generate sequences ofgaze shifts based on the Bayesian Strategy.

The paper is organized as follows. In the next sectionthe general model of active saliency for the generationof gaze sequences is introduced. In Section 3 the appli-cation of the proposed model in active visual search isdiscussed before an experimental validation is providedin Section 4.

2 Active saliency

The proposed model is based on the assumption thattranssaccadic memory plays an important role in ac-tive visual search [5]. While this assumption is dis-cussed controversially in the context of human visualsearch, the application of such a memory on a tech-nical system seems to be feasible. The formulation ofthe Bayesian Strategy is extended in order to includethe requirement of a consistent transsaccadic memoryof salient locations. For this purpose, the inconsis-tency of the memory representation corresponding toa spatial location X is described with the random vari-able I. Thus, the desired posterior in our model be-comes P (O = 1, X, I|J), which we will refer to as activesaliency sa.

In the following, a distinction is made between pe-ripheral and foveal processing. Object recognition isperformed based on foveal processing, while periph-eral processing reports changes of the environment thatmight affect consistency of the memory. The periph-eral observation is then defined by the observation ofchange Zc. The foveal processing includes the mea-surement Zf which updates the believe of object exis-tence and location bel(O = 1, X). Further, the fovealprocessing results in the validation of memory entitieswhich is represented with the variable V . Reformulat-ing the above model using the introduced observationsyields

sa = P (O = 1, X, I|Zf , Zc, V ).

Assuming perfect knowledge of the current gaze di-rection, OX and I are conditionally independent givenall measurements. Further, exploiting the indepen-dences from observations and hidden variables as im-posed by the above model, the posterior for activesaliency under consideration of peripheral perception,foveal perception and memory is given with

sa = P (O = 1, X, I|Zf , Zc, V ) (1)

= P (O = 1, X|Zf )P (I|Zc, V ).

The first factor of the above model essentially cor-responds to the Bayesian Strategy as proposed in [4].The second factor allows to integrate the requirement

I− I

C V

Zc

Figure 1. Graphical model of the inconsistencyfiltering. Inconsistencies I are updated overtime under consideration of the measurement ofchange Zc and the validation V .

of the consistency of transsaccadic memory and formsthe basis for the inhibition of return mechanism. Adetailled model for the second factor is defined in thefollowing sections.

2.1 Filtering of inconsistencies

Inconsistencies are filtered over time, where a filter-ing step involves the integration of the observed changeZc and occurred validation V . Figure 1 illustrates agraphical model of all involved random variables. Inthe following, the sample spaces for all variables are de-fined for N memory entities. For each memory entitya binary random variable is used to encode its state.

• I: Encodes inconsistencies between memory andreal world. Inconsistency either takes the valueconsistent (con) or inconsistent (¬con), thus I ∈{con,¬con}N .

• I−: Previous inconsistencies. I− ∈ {con,¬con}N .

• C: Change that occurred in the world as relevantfor the memory. The world was either static (¬ch)or changed (ch) with respect to the memory entity,thus C ∈ {ch,¬ch}N .

• Z: Change of the world as measured by the changesensor. Z ∈ {ch,¬ch}N .

• V : Encodes whether validation has been per-formed for the respective entity in order to as-certain its consistency with the real world. V ∈{val,¬val}N .

Each random variable encodes the distribution forall memory entities. In order to keep the formalizationgeneral, we did not state whether memory entities arestored in a grid based manner as the case in proba-bilistic saliency map, or in a landmark based manner.For both representations, a common assumption is theindependence between entities. Thus, the joint distri-bution over all variables can be factored to

P (I, I−, C, V, Z) =∏

i=1,...,N

P (Ii, I−i , Ci, Vi, Zi).

According to Fig. 1, the inconsistency for each memoryentity i can then be factored using

P (Ii, I−i , Ci, Vi, Zi) (2)

= P (I−i )P (Ii|I−i , Ci, Vi)P (Zi|Ci)P (Ci)P (Vi).

The model parameters and the approach for infer-ence of the posterior are introduced in the followingfor the update of a single entry i.

2.2 Model parameters

The prediction model is defined by the conditionalprobability of inconsistency Ii given the change Ci, thevalidation Vi and the inconsistency from the last fil-tering step I−i . This dependency is expressed by theconditional probability table

P (Ii = ¬con|I−i , Ci, Vi)

=

0, if (I−i = con ∧ Ci = ¬ch)

1− pv, if (Vi = val ∧ I−i = ¬con ∧ Ci = ¬ch)

1, else

In the above definition, the first statement covers caseswhere the consistency from the previous step is pre-served since no change happened. Validation leadsto consistency of the memory if no change happenedand the memory was inconsistent, as stated in thesecond case. The parameter pv allows to define aconfidence for the validation success. In all othercases, the resulting inconsistency equals to one, thuschange always overrides validation. This is a neces-sary statement since the order of the performed vali-dation and measured change is not provided and thuschange might have occurred after validation has beenperformed within the last time interval.

The change sensor is modeled using the forwardmodel P (Zi|Ci). We define the sensor model for changein a general manner using the sensors sensitivity wc,1

and its specificity wc,0 in the following way

P (Zi = ch|Ci) =

{1− wc,0 if Ci = ¬ch,wc,1 if Ci = ch

.

Another parameter required to fully define the prob-abilistic model is the prior probability of change P (Ci).The prior allows to encode the likelihood of change inthe scene pc. This probability can be used as top-downcue in order to instruct the system to peform more val-idations in cases, where the scene is changing rapidly.The change prior is defined by

P (Ci = ch) = pc.

In summary, the model provides four free parameterswhich can be used to influence the inference of incon-sistencies. The sensitivity wc,1 and specificity wc,0 ofthe change sensor as well as the validation probabil-ity pv depend on the system build around the model.These parameters will be defined for the application ofactive visual search in Section 3. The change prior pcallows to tune the model according to the volatilenessof the current scene.

2.3 Inference of inconsistencies

The inference of the inconsistency of memory entitiesis formulated based on the model defined in Section2.1 and its parameters discussed in Section 2.2. Usingthe prior believe of inconsistency of a memory entitybel(I−i ), the observation of validation Vi = vi and of

Figure 2. The Karlsruhe Humanoid Head isequipped with a 3 DoF active camera system andoffers one perspective and one foveal camera pair.

measured change Zi = zi the posterior believe bel(Ii)can be calculated by performing the marginalization

P (Ii|Zi = zi, Vi = vi) (3)

=

∑I−i ,Ci

P (Ii, I−i , Ci, Vi = vi, Zi = zi)∑

Ii,I−i ,Ci

P (Ii, I−i , Ci, Vi = vi, Zi = zi)

.

The posterior believe is calculated using the factor-ization from (2).

3 Application in active visual search

The model proposed in the previous section is nowput into the context of active visual search. The ac-tive visual search task is performed using the Karls-ruhe Humanoid Head [6] as depicted in Fig. 2. Thehead provides an active camera system combined withfoveal and peripheral camera pairs that allow to ac-tively search for objects in the scene.

The goal of the active visual search task consistsin establishing and retaining a consistent memory oftarget object instances by performing object detectionin the foveal cameras. Resulting from the narrow fieldof view of the foveal cameras, the execution of saccadesis necessary in order to enhance the field of sight. Thesequence of saccades is generated using the proposedactive saliency.

3.1 Saliency calculation and representation

The saliency for the active visual search task is gen-erated according to the Bayesian Strategy. Similar toour previous work in [7] saliency is generated based ontop-down knowledge of the object appearance alone.As starting point for the object search, candidates oftarget objects are extracted in the peripheral views.The extracted candidates are stored in a landmarkbased map where each candidate is represented with anormally distributed location uncertainty Xi and theprobability of existence Oi. This memory represen-tation is updated with each foveal observation of thecandidates using the Bayes filter

P (O = 1, X|Zf ) = ηP (Zf |O = 1, X)P (O = 1, X).(4)

The above posterior corresponds to the top-down fac-tor of the Bayesian Strategy.

3.2 Extension to active saliency

In order to execute saccades which ensure a consis-tent memory based on the above saliency representa-tion, the active saliency is now put in the context of the

Figure 3. Setup used for the active visual searchtask. The goal consisted in building and retain-ing a consistent memory of the two cereal boxesbased on the active saliency.

active visual search task. While the top-down saliencyformulated in (4) corresponds to the first factor of thefactorization in (1), in the following the second factoris further defined. For each landmark in the memoryof object candidates the inconsistency is updated ac-cording to (3) separately.

The detection of change Zc is performed based onthe peripheral observations. Each object candidate inmemory is monitored over time by building correspon-dences between observed candidates and stored candi-dates according to [8]. If the observation differs fromthe memory representation, change has been detected.The change sensor in the model is assumed to provideperfect specificity wc,0 = 1. A sensitivity of wc,1 = 0.9is used to express the possibility of unobserved change.

The parameter for validation certainty pv expressesthe ability to validate the memory content based on thefoveal views. In order to model decreasing validationperformance toward the borders of the foveal images,the following validation certainty is used for the leftand right foveal image. Let the spatial 2D location ofthe object within left and right image be defined by~ul and ~ur. The validation certainty for each image isthen determined using

pv,l|r = e−(~ul|r−~cl|r)T Σv(~ul|r−~cl|r)

2 , (5)

where ~cl and ~cr are the centers of left and right im-ages and Σv is a diagonal matrix of variances. Thevalidation certainty pv is defined by the product of thevalidation certainty of both cameras

pv = pv,lpv,r. (6)

4 Experimental Validation

In the following, the feasibility of the active saliencymodel is validated. The goal of the validation consistsin illustrating the ability to generate gaze sequencedwhich assure memory consistency based on the top-down saliency data. The evaluation of the top-downgeneration of saliency is beyond the scope of this work.

A simple search task as illustrated in Fig. 3 waschosen as example that allows to illustrate the prop-erties of the inhibition of return mechanism. For this

00

5 10 15 20 25 30

0.2

0.4

0.6

0.8

1

activesa

liencysa

time (s)

entity 1 entity 2 validation change

phase 1 phase 2 phase 3

Figure 4. During the active visual search experiment, two object instances were visible to the system. De-pending on the current measurement of change and performed validation, the active saliency is inferred.Once the active saliency exceeds the threshold smax = 0.8 a saccade is executed.

purpose, two instances of a cereal box were succes-sively brought into the view of the peripheral cam-eras. Using the proposed model, the active saliencysa for the stored object instances was continuously up-dated. A saccade toward an object instance was exe-cuted once the active saliency exceeded a given thresh-old smax = 0.8. The active saliency was recorded fortwo object instances used during the experiment. Thetime course of active saliency as monitored during theexperiment is illustrated in Fig. 4. Time steps whereeither change was detected or validation was performedare marked with horizontal lines. As can be seen in theillustrated time course, measured change results in anincrease of the active saliency and an immediate valida-tion of the corresponding object instance. Validationonly is performed, once the active saliency reaches thethreshold smax. In the following the different phasesof the experiment and their counterparts in Fig. 4 areexplained.

The experiment consist of three phases: In the firstphase, only one object is visible to the peripheral cam-era system. For the second phase, an additional in-stance is positioned in the scene. Finally, the first in-stance is removed from the scene. The different phasesof the experiment are reflected in Fig. 4 with the ob-servation of change. In the first two phases objects arebrought into the scene which results in the observationof change and the execution of a saccade in order to de-tect the new instance based on the foveal views. In thethird phase the object is removed from the scene whichresults in the observation of change and the removal ofthe object instance from memory after the executionof a saccade and foveal recognition. In between theobserved changes the stored object instances are vali-dated frequently, where the frequency is defined by theprior of change pc, the sensitivity of the change sensor,and the success of the last performed validation. Eachtime the active saliency exceeds the threshold smax

validation is performed.

5 Conclusion

The concept of active saliency takes into account theconsistency of memory accumulated during saccadiceye movements. The resulting inhibition of returnmechanism is closely coupled to the requirement of aconsistent memory. The proposed probabilistic modelseamlessly integrates with the Bayesian Strategy tovisual attention and provides semantically descriptive

parameters. The behavior in terms of gaze sequencesas generated by the proposed model has been demon-strated in the context of an active visual search task.While a landmark based representation was used as un-derlying memory, the proposed approach is formulatedin a general way and can also be applied to grid basedrepresentations such as probabilistic saliency maps.

AcknowledgementThe research leading to these results has received

funding form the European Union Sixths and SeventhFramework Programme FP7/2007-2013 under grantagreement no.270273 and from the German ResearchFoundation (DFG: Deutsche Forschungsgemeinschaft)under the SFB 588.

References

[1] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” IEEETransactions on Pattern Analysis and Machine Intelli-gence, vol. 20, no. 11, pp. 1254–1259, 1998.

[2] A. Torralba, “Modeling global scene factors in atten-tion.” J Opt Soc Am A Opt Image Sci Vis, vol. 20,no. 7, pp. 1407–1418, 2003.

[3] A. Torralba, A. Oliva, M. S. Castelhano, and J. M. Hen-derson, “Contextual guidance of eye movements and at-tention in real-world scenes: the role of global featuresin object search.” Psychological Review, vol. 113, no. 4,pp. 766–786, 2006.

[4] L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W.Cottrell, “SUN: A Bayesian framework for saliency us-ing natural statistics,” Journal of Vision, vol. 8, no. 7,2008.

[5] P. D. Graef and K. Verfaillie, “Transsaccadic memoryfor visual object detail.” Prog Brain Res, vol. 140, pp.181–196, 2002.

[6] T. Asfour, K. Welke, P. Azad, A. Ude, and R. Dillmann,“The Karlsruhe Humanoid Head,” in IEEE/RAS Inter-national Conference on Humanoid Robots (Humanoids),2008, pp. 447–453.

[7] K. Welke, T. Asfour, and R. Dillmann, “Active multi-view object search on a humanoid head,” in IEEE Inter-national Conference on Robotics and Automation (ICRA),2009, pp. 417–423.

[8] ——, “Bayesian visual feature integration with saccadiceye movements,” in IEEE/RAS International Confer-ence on Humanoid Robots (Humanoids), 2009, pp. 256–262.