twelve

visual perception

Processing in the human visual system(maybe, sort of)

eyeearlyvision

(striate cortex)

contourdetection

salience attention

objectrecognition

(IT cortex)

spatial memory(PP cortex)

“what” pathway

“where” pathway

grouping

The eye

Lens Depth of field Chromatic aberration

Photoreceptors Rods Cones Fovea, macula, and

periphery Retinal processing

Gain control Edge enhancement? Simple motion detection

lens/irisrods

conesretinal

ganglion

Photoreceptors

Rods Found mostly in the macula and periphery Very sensitive to light But don’t detect color

Cones Found in the fovea Less sensitive Color sensitive

Colors seem to fade in low light

Trichromacy

Having different cones for every possible wavelength would be bad

We just have three kinds of cones “Blue” cones: short wavelengths “Green” cones: intermediate

wavelengths “Red” cones: long wavelengths However, their responses overlap

The eye reduces all the wavelengths at a given pixel to just the total “amount” of “red”, “green”, and “blue”

Components of a color image

Evolution of the color system

S to cortex

Evolution of the color system

S to cortex

L

+intensity

−chroma (B-Y)

cool/warm

Intensity and chromaticity

Color image Intensity (R+G+B) Chromaticity (B-Y)white = coolblack = warmgrey = neurtral

Evolution of the color system

S to cortex

I

+intensity

−chroma (B-Y)

L −chroma (R-G)

cool/warm

Intensity and chromaticity

Color image

Intensity

Chromaticity (B-Y)white = coolblack = warngrey = neutral

Chromaticity (R-G)white = redblack = greengrey = balanced

Code

[define gray [i → [color i i i]]]

[define signed-gray [i → [gray [+ 128 [∕ i 2]]]]]

[define b-y [c → [− [blue c] [∕ [+ [red c] [green c]]

2]]]]

[map-bitmap [c → [signed-gray [b-y c]]] cones]

Code

[define r-g [c → [− [red c] [green c]]]]

[map-bitmap [c → [signed-gray [r-g c]]] cones]

Image-level processing

Pixel-level processing Massively parallel Preattentive Very fast

Contrast versus absolute intensity

The visual system responds less to the amount of light or color

Than to variations in the amount

This is also known as contrast

Contrast detection

Many of the neurons in the visual system are designed to respond to contrast

Center/surround receptive fields Stimulated by the presence

of light in a specific area Inhibited by its presence in

the area immediately around it

− −+

inhibitory region(light prevents firing)

excitatory region(light stimulates firing)

Edge detection

When applied to image intensity (R+G+B),

Center/surround processing responds to lightness contrast

Which often occurs at object edges

Scale space

Contrast detection is performed at many different scales

Oriented edge detection

Other neurons Are stimulated by light

in an elongated region And inhibited by it in

the region next to it These can detect the

orientation of edges

− +

−+

−+

Oriented edges

Constancy

Illumination varies over a surface in intensity in chroma

But you (usually don’t perceive the color of the surface as changing

This is called perceptual constancy Color constancy Lightness constancy

Georges de la tour,The Repentant Magdalene, C. 1640

Failures of lightness constancy

Failures of lightness constancy

Failures of lightness constancy

Failures of color constancy

Failures of color constancy

Failures of color constancy

Failures of color constancy

Depth cues

The world is 3D But the image is 2D Your vision system uses

depth cues to recover lost information Stereopsis Motion Linear perspective Aerial perspective Relative size

Occlusion

The interruption of edges of one object by the edges of another object

Shows the interrupting object is in front

C. Coles Phillips,Brittish Tank on Fifth Avenue, 1917

Stereo vision

Your eyes are in slightly different positions

So they receive slightly different images The disparity between

images decreases with distance (simple case)

So by matching features between the two eyes Your brain computes

disparity And hence, depth

Motion parallax

Objects farther away seem to move slower So the background seems to remain stable And the foreground moves Or, the opposite, if the camera tracks the foreground

This is frequently used in cell animation

Linear perspective

Parallel lines converge at a vanishing point

One-point perspective One set of lines converging Gives a sense of immersion in

the scene Background seems to pop out

Two-point perspective Two sets of converging lines The object seems to pop out

Three-point perspective Rarely used Very dramatic

Aerial perspective

Surfaces get hazier as they recede into the distance

Relative size

Apparent size decreases with distance

So

Texture gradients

Textures get finer as objects recede

Gestalt theory

Theory of how we organize components of a percept into wholes Max Wertheimer, Kurt Koffka,

Wolfgang Köhler

Law of prägnanz (simplicity) We choose the simplest possible

interpretation of a scene

Formulated laws of grouping Disks or pacman?

Grouping by proximity

Grouping by similarity

Grouping by continuity

Grouping by closure

Grouping by common fate

Grouping by common fate

Figure and ground

We tend to separate the scene into One or more objects

(“figure”), in front of or on top of

A ground plane or background (“ground”)

We use a number of tricks to do it Depth, grouping

But it can be hard in degenerate cases

Attention

Remember our fovea is only about 2° So we have to view scenes by scanning

the fovea over the image

Attention is driven both top-down by cognition

Task Knowledge, expectations

And bottom-up by Size Intensity Local contrast Motion Odd-man-out

As well as recency(don’t look if we’ve just looked there)

Attention and scale space

Attention starts at coarse (large) scales

And narrows to finer scales

Attention and intensity

All things being equal, our attention is driven to intense stimuli Bright lights Saturated colors Rapid motion

Attention and color

Warm, bright colors draw attention

Cooler, darker colors tend to recede into the background

Claude Monet, Sunrise, 1872

Attention and contrast

But one of the biggest drivers of attention is contrast We focus on unusual

elements of the scene Elements that are

different from their neighbors

Attention follows curves

Attention follows perspective and depth

Top-down attention

Attention is also driven by expectations If I see something

face-like I look for the facial

features Or vice-versa

And by task

Change blindness

We only really see what the fovea is fixating

If anything changes, we don’t notice unless we see motion

Oh, and we don’t see motion while our eye is moving …

Continuity in films Pretty woman

Neisser et al’s demo Simons and Chabris’ demo

Supply Pattern

Work area

Adapted from Ballard et al.

Shape recognition

We don’t really know how this works

It’s really hard …

And there are a lot of high level semantic effects