Plan of talk: Visual Indexes

Theoretical motivations The ubiquitous Correspondence Problem in

vision: When do two or more proximal tokens correspond to the same distal object?

• Detection of relational properties by “visual routines”• Incremental construction of visual representations

Must indexing & tracking be preconceptual?

Empirical demonstrations of visual indexing Multiple Object Tracking – how is it done? Other experimental results

• Subset selection, subitizing• Ballard & Hayhoe copying task

An important function of early vision is to individuate and select token elements (let’s call them “objects” for now)

The most basic perceptual operation is the individuation and selection that precedes the formulation of perceptual judgments.

Making visual judgments presupposes that the things (objects) that judgments are about have been individuated and selected (or indexed).Another way to put this is that the arguments of

perceptual predicates P(x,y,z,…) must be bound to things in the world in order for the judgment to have perceptual content.

Individuation, selection and indexing (or referring) are different from “discriminating.”

Several objects must be picked out at once in relational judgments

When we judge that certain objects are collinear, we must have picked out the relevant individual objects first.

Several objects must be picked out at once in relational judgments

The same is true for other relational judgments like inside or on-the-same-contour… etc. We must pick out the relevant individual objects first.

Individuating is different from discriminating

How do we individuate and select objects in our field of view?

The principal way we select individual objects is by foveating them - i.e., by looking directly at them. (NB Notice that this is a deictic reference, meaning “that which I am looking at now”).

That’s not the only way we select: we can also select with focal attention, independent of direction of gaze.

Focal attention appears to be unitary; yet we can also select more than one thing at a time (e.g., in making a relational judgment).

The BIG question: In virtue of what properties of objects are they individuated and indexed?

The ubiquitous “correspondenceproblem” in vision: Matching proximal tokens of a distal individual

Apparent motion, structure from motion, stereo

vision, and very many visual computations face

the problem of identifying which proximal

features of an image correspond to the same

individual distal object.

Less well known is the correspondence problem

faced when a single visual representation is

constructed incrementally over time – with or

without eye movements.

Example: Drawing a diagram and noticing its properties

Some of the distinct “views” while exploring the diagram

The correspondence problem for incremental construction of a visual representation

There is evidence that visual representations are built up incrementally over time, whether or not the construction involves eye movements;

When properties are noticed they are noticed as properties-of-individuals (objects);

When new properties are encoded they need to be assigned to the representation of the relevant distal object. If the object is one that had been encoded before, then it has to be so-identified;

If objects are identified only by their encoded properties, then the sequence of descriptors at each time t has to be retained in order to solve the correspondence problem.

Example of the correspondence problem for apparent motion

The grey disks correspond to the first flash and the black ones to the second flash. Which of the 24 possible matches will the visual system select as the solution to this correspondence problem? What principal does it use?

Curved matches Linear matches

Reprise: In virtue of what properties of objects are they individuated?

When you look at a uniform texture you can pick out one or more token elements without regard to their color, shape or other visual properties.

The most likely property used in picking out an object is its location (when objects are visually identical this may be the only property available).

It is widely believed that we access an object’s properties by first retrieving its location.• This assumption is made by every theory that deals with the

detection of patterns or conjunctions of features.• But there is also evidence that we can access an individual

object solely by virtue of its historical continuity qua individual.

The strongest candidate for the role of mediating property in picking out token individual objects is location. The case for the prior encoding of location is supported by:

1.Many theories of search, including Treisman’s Feature Integration Theory, assume that location provides the means for detecting property-conjunctions. To find a conjunction of properties one finds the first property, determines its location, and checks to see if the second property is also located there.

11

The case for the prior encoding of location …

2. It has been frequently reported that when people detect certain properties (e.g., color) in a display, they very often also know where these properties are located – even when they fail to report any other properties (e.g., shape).

* This is not uncontentious, though. There are many reports of the detection of properties without knowing where the properties occurred.

2

The case for the prior encoding of location …

3. A number of people have explicitly tested the location-mediation hypothesis by cuing a search display with one property and examining the resulting joint probability of detecting various other properties. For example, Mary-Jo Nissan cued a search display with a color C and measured (or estimated) the probability of detecting shape S, location L, and both shape and location S & L. She showed that:

P(L & S | C) = P(L| C) * P(S | L)which is what one would expect if location mediated the joint detection.

3

Although there is a great deal of evidence for the priority of reporting location, this is not the same as showing the necessity of accessing properties by their location.

In every case in which location is shown to play a special role, location is confounded with individuality because objects have fixed locations. In these cases, being at a different location entails being a different individual. But over time different locations may be occupied by the same individuals, and vice versa.

There are two possible ways to unconfound location and individuality:

1. use moving objects2. use objects whose identity and/or ‘motion’ is

independent of their spatial location.

But ….

1. Moving objects Priming across moving objects

(“reviewing object files”)

Inhibition of Return

Multiple Object Tracking (MOT)

2. Spatially coincident objects tracking in “feature space”

How to distinguish access by location and by individual

Object File Theory(Kahneman, Treisman & Gibbs, 1992)

Information is encoded and stored in files that are specific to particular individual objects.

When an object is encountered, an attempt is made to solve the correspondence problem and assign it to an existing object file, based on spatiotemporal properties.

When an assignment to an existing object file succeeds, the information in the existing file is first reviewed and is used as the default properties of that object. Thus there is a processing benefit for recognizing those properties that are listed in the object file for that object.

Moving object studies…

Kahneman & Treisman’s study of “Object File” priming

Demonstration of the Object File display: (Kahneman, Treisman & Gibbs, 1992)

Positive Example

Negative Example

The Case of Inhibition of Return

Moving object studies…

“Inhibition of Return” moves with the object that is inhibited

Moving object studies…

How do we do it? What properties of individual objects do we use?

How do we do it? What properties of individual objects do we use?

A possible location-based tracking algorithm1. While the targets are visually distinct, scan attention to each target

and encode its location on a list. When targets begin to move;

2. For n=1 to 4; Check the n’th position in the list and retrieve the location Loc(n) listed there.

3. Go to location Loc(n). Find the closest element to Loc(n).

4. Update the n’th position on the list with the actual location of the element found in #3. This becomes the new value of Loc(n).

5. Move attention to the location encoded in the next list position, Loc(n+1).

6. Repeat from #2 until elements stop moving.

7. Go to each Loc(n) in turn and report elements located there.

Use of the above algorithm assumes (1) focal attention is required to encode locations (i.e., encoding is not parallel), (2) focal attention is unitary and has to be scanned continuously from location to location. But it assumes no encoding (or dwell) time at each element.

Predicted performance for the serial tracking algorithm as a function of the speed of movement of attention

Some other findings concerning object tracking (1)

Detection of events on targets is better than on nontargets, but this does not generlize to locations between targets;

Objects can continue to be tracked when they disappear completely behind occluders, as long as the mode of disappearance is compatible with there being an occluding surface;

Objects can all disappear from view for as long as 330 ms without impairing tracking;

When objects that disappear behind an occluder and come out a different color or shape, the change is unnoticed;

Some other findings concerning object tracking (2)

Not all distinct feature clusters can be tracked; some, like the endpoints of a line, cannot;

People can track items that automatically attract attention, or they can decide which items to track; but in the latter case it appears that the may have to visit each object serially

Successful tracking of an object entails keeping track of it as a particular individual, yet people are poor at keeping track of which successfully tracked (initially numbered) item is which. This may be because: When observers make errors, they are more likely to switch

the identity of a target with that of another target than the identity of a target with that of a nontarget.

How do we do it? What properties of individual objects do we use?

Basic MOT with repulsionMOT with occlusionMOT with Virtual OccludersMOT with implosion/explosionMOT with self-occlusion (no repulsion)MOT with ‘target merging’MOT with rubber band connectionsMOT with IDs (which is which?)Track non-flashed (3 blinks)Track Non-flashed (one flash)

What properties are used in(a) selecting objects and (b) tracking objects?

Notice that these are different operations and need not involve the same properties

How do we select (index) an object?

The principal way we select individual objects is by foveating them - i.e., by looking directly at them. (NB Notice that this is a deictic reference, viz. “that which I am looking at now”).

That’s not the only way we select: we can also select with focal attention, independent of direction of gaze.

Focal attention appears to be unitary – yet we can also select more than one thing at a time (e.g., in making a relational judgment).

The BIG question: In virtue of what properties of objects are they individuated and indexed?

What properties can be used to select (index) an object in MOT?

We have evidence that selecting objects can be done either automatically or voluntarily, but only under certain conditions: Automatic selection requires “popout” features

(sudden appearance, motion, etc) Voluntary selection can use any discriminable

property, but the objects must be attended serially and the property must be available long enough for this to occur (Annan study)

Role of object properties

Selection of targets in MOT

When non-popout properties define the target subset, the objects can still be indexed and tracked, but only if sufficient time is provided.

The assumption is that when targets are selected (and indexed) by virtue of some property that does not automatically grab an index, the objects must be visited serially with focal attention in order to be indexed.

Role of object properties (continued)

What properties can be used to track indexed objects?

We have (suggestive) evidence that observers do not encode or use intrinsic object properties (e.g., color, shape) during tracking: There is evidence that observers can not recall object

properties (when we stop and ask) and do not notice when properties like color/shape change;

There is some evidence, with small numbers of objects, that tracking occurs even if it is not task-relevant (e.g., Kahneman & Treisman’s Object Files);

We have some evidence that when objects differ in non-identifying properties, they cannot be tracked any better than if they do not differ in these properties.

Role of object properties (continued)

Do observers use object locations in tracking?

Observers can’t use location-updating algorithm;Can observers use a historical trace of objects’

trajectories (space-time worms) to track objects? What kind of mechanism could respond to a

historical trace (or space-time worm)? “Historical trace” presupposes that it is the trace of

a single individual object (and not a sequence of time-slices of different objects) and therefore that the individual object has been selected and tracked. So responding to a historical trace may be the same as tracking an object’s numerical identity.

Observers can track non-spatial ‘virtual objects’ that move through ‘property space’: Tracking superimposed surfaces

Blaser, Pylyshyn & Holcombe (2000)

Two superimposed Gabor patches that vary in spatial frequence, color and angle

Changing feature dimensions

Surfaces move randomly in “feature-space”

snapshots taken every 250 msec

Snapshots

Such generalized ‘objects’ can be individually tracked, and they also show single-object superiority for change detection.

In conclusion: Some speculations about what we need and what early vision may provide (1)

1. We need a mechanism that puts us in causal contact with distal objects in a visual scene – a contact that does not depend on the object satisfying a certain description, but on a brute causal connection provided by an index or FINST.

We have seen many reasons for needing this, but we have not mentioned that such a mechanism is essential for connecting vision and action!

Speculations on what we need and what early vision may provide (2)

The assignment of an index can take place in one of several ways:An object can seize an index by virtue of possessing

certain attention-demanding properties (e.g., sudden onset, movement, other built-in index-grabbers).

An index can also be assigned indirectly by looking at an object or attending to it. Although the selection of which object to attend to may well depend on the object’s conceptualized properties, these need not be the properties that cause the index to be assigned once attention is focussed on it.

Notice that once an object is indexed, the index itself does not carry information about the object’s properties except insofar as these are independently encoded and stored. An index is like a pointer in a data structure or a proper name (although unlike a proper name, the referential relation holds only while the object is in view, which makes it more like a demonstrative).

Speculations on what we need and what early vision may provide (3)

2. We need a mechanism that keeps track of the identity of distal objects without using their encoded properties. Such a mechanism realizes a rudimentary identity-tracker, with its own ‘rules’.

3. This is not a general identity-maintenance process; it will not allow you to recognize the identity of a person in a picture and a person on the street. But it may provide a way to maintain same-objecthood within the modular early vision system. In that regard there is this final item … There is evidence for such a mechanism in

babies as young as 6 months (Leslie, Spelke)!

Relation to work on infants’ sensitivity to the numerosity of objects:

Alan Leslie’s “Object Indexes”

Infants as young as 4 months of age show surprise (longer looking time) when they watch two things they have seen together being placed behind a screen and then the screen is lifted to reveal only one thing. Below 10 months of age they are in general not surprised when the screen is lifted to reveal two things that are different from the ones they saw being placed behind the screen.

In some cases, infants (age 12 months) use the difference in color of the objects they see one-at-a-time to infer their numerosity, but they do not appear to record the colors and use them to identify the objects that are revealed when the screen is lifted.

Leslie & Tremoulet: Infants aged 10 and 12 months are shown a red and then a green object that are then hidden behind a screen. The 10 month old is surprised if lowering the screen reveals the wrong number of objects, not if it reveals the wrong color of objects. Color is used to individuate objects, but not to keep track of them! At 12 months children can use color to keep track of what went behind the screen.

Forms of representation for a robot: using indexicals

Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 197-207.

John Perry gives the following nice example of how the decision to take a particular course of action can arise from the realization that a particular object in a description and a particular thing one sees are one and the same object.

The author of the book, Hiker’s Guide to the Desolation Wilderness stands in the wilderness beside Gilmore Lake, looking at the Mt. Tallac trail as it leaves the lake and climbs the mountain. He desires to leave the wilderness. He believes that the best way out from Gilmore Lake is to follow the Mt. Tallac trail up the mountain … But he doesn’t move. He is lost. He is not sure whether he is standing beside Gilmore Lake, looking at Mt. Tallac, or beside Clyde Lake, looking at the Maggie peaks. Then he begins to move along the Mt. Tallac trail. If asked, he would have to explain the crucial change in his beliefs in this way: “I came to believe that this is the Mt. Tallac trail and that is Gilmore Lake”. (Perry, 1979, p 4)

The point of this example is that in order to understand and explain the action of the lost author it is essential to use demonstrative terms such as this and that. Without a way to directly pick out the referent of a descriptive term and to link the perceived object token with its cognitive representation, people would not be able to act on their knowledge and we, the theorists, would not be able to explain the people’s actions.

Summary: FINSTs keep us connected with the world

Pylyshyn, Z. W. (1998). Visual indexes in spatial vision and imagery. In R. D. Wright (Ed.), Visual Attention (pp. 215-231). New York: Oxford University Press.

Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 197-207.

Annan, V., & Pylyshyn, Z. W. (2002). Can indexes be voluntarily assigned in multiple object tracking? Paper presented at the Vision Sciences 2002, Sarasota, FL.

Blaser, E., Pylyshyn, Z. W., & Holcombe, A. O. (2000). Tracking an object through feature-space. Nature, 408(Nov 9), 196-199.

Burkell, J., & Pylyshyn, Z. W. (1997). Searching through subsets: A test of the visual indexing hypothesis. Spatial Vision, 11(2), 225-258.

Carey, S., & Xu, F. (2001). Infants' knowledge of objects: Beyond object files and object tracking. Cognition, 80(1/2), 179-213.

Kaplan, D. (1989). Demonstratives. In J. Almog & J. Perry & H. Wettstein (Eds.), Themes From Kaplan (pp. 481-563). New York: Oxford University Press.

Leslie, A. M., Xu, F., Tremolet, P. D., & Scholl, B. J. (1998). Indexing and the object concept: Developing `what' and `where' systems. Trends in Cognitive Sciences, 2(1), 10-18.

Lespérance, Y., & Levesque, H. J. (1995). Indexical knowledge and robot action - a logical account. Artificial Intelligence, 73, 69-115.

Annan, V. & Pylyshyn, Z. W. (2002). Can indexes be voluntarily assigned in multiple object tracking? Paper presented at the Vision Sciences 2002, Sarasota, FL.

Blaser, E., Pylyshyn, Z. W., & Holcombe, A. O. (2000). Tracking an object through feature-space. Nature, 408(Nov 9), 196-199.

Burkell, J., & Pylyshyn, Z. W. (1997). Searching through subsets: A test of the visual indexing hypothesis. Spatial Vision, 11(2), 225-258.

Carey, S., & Xu, F. (2001). Infants' knowledge of objects: Beyond object files and object tracking. Cognition, 80(1/2), 179-213.

Kaplan, D. (1989). Demonstratives. In J. Almog & J. Perry & H. Wettstein (Eds.), Themes From Kaplan (pp. 481-563). New York: Oxford University Press.

Leslie, A. M., Xu, F., Tremoulet, P. D., & Scholl, B. J. (1998). Indexing and the object concept: Developing `what' and `where' systems. Trends in Cognitive Sciences, 2(1), 10-18.

Lespérance, Y., & Levesque, H. J. (1995). Indexical knowledge and robot action - a logical account. Artificial Intelligence, 73, 69-115.

Ogawa, H., & Yagi, A. (2002). The processing of untracked objects during multiple object tracking. Paper presented at the Vision Sciences 2002, Sarasota, FL.

Perry, J. (1979). The problem of the essential indexical. Noûs, 13, 3-21.

Pylyshyn, Z. W. (1989). The role of location indexes in spatial perception: A sketch of the FINST spatial-index model. Cognition, 32, 65-97.

Selected references on FINSTs and Demonstratives

Ogawa, H., & Yagi, A. (2002). The processing of untracked objects during multiple object tracking. Paper presented at the Vision Sciences 2002, Sarasota, FL.

Perry, J. (1979). The problem of the essential indexical. Noûs, 13, 3-21.

Pylyshyn, Z. W. (1989). The role of location indexes in spatial perception: A sketch of the FINST spatial-index model. Cognition, 32, 65-97.

Pylyshyn, Z. W. (1998). Visual indexes in spatial vision and imagery. In R. D. Wright (Ed.), Visual Attention (pp. 215-231). New York: Oxford University Press.

Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 197-207.

Pylyshyn, Z. W. (2001a). Connecting vision and the world: Tracking the missing link. In J. Branquinho (Ed.), The Foundations of Cognitive Science (pp. 183-195). Oxford, UK: Clarendon Press.

Pylyshyn, Z. W. (2001b). Visual indexes, preconceptual objects, and situated vision. Cognition, 80(1/2), 127-158.

Schmidt, W. C., Fisher, B. D., & Pylyshyn, Z. W. (1998). Multiple-location access in vision: Evidence from illusory line motion. Journal of Experimental Psychology: Human Perception and Performance, 24(2), 505-525.

Scholl, B. J., & Pylyshyn, Z. W. (1999). Tracking multiple items through occlusion: Clues to visual objecthood. Cognitive Psychology, 38(2), 259-290.

Scholl, B. J., Pylyshyn, Z. W., & Feldman, J. (2001). What is a visual object: Evidence from target-merging in multiple-object tracking. Cognition, 80, 159-177.

Scholl, B. J., Pylyshyn, Z. W., & Franconeri, S. L. (submitted). The relationship between property-encoding and object-based attention: Evidence from multiple-object tracking.

Sears, C. R., & Pylyshyn, Z. W. (2000). Multiple object tracking and attentional processes. Canadian Journal of Experimental Psychology, 54(1), 1-14.

Slemmer, J. A., & Johson, S. P. (2002). Object tracking in ecologially valid occulsion events. Paper presented at the Vision Sciences 2002, Sarasota, FL.

Suganuma, M., & Yokosawa, K. (2002). Is multiple object tracking affected by three-dimensional rigidity? Paper presented at the Vision Sciences Society, Sarasota, FL.

Tremoulet, P. D., Leslie, A. M., & Hall, D. G. (2001). Infant individuation and identification of objects. Cognitive Development, 15(4), 499-522.

Trick, L. M., & Pylyshyn, Z. W. (1994). Why are small and large numbers enumerated differently? A limited capacity preattentive stage in vision. Psychological Review, 101(1), 80-102.

Viswanathan, L., & Mingolla, E. (1998). Attention in depth: disparity and occlusion cues facilitate multi-element visual tracking (Abstract). Investigative Ophthalmology and Visual Science, 39(4), 634.

Yantis, S. (1992). Multielement visual tracking: Attention and perceptual organization. Cognitive Psychology, 24, 295-340.