perception and action guest lecture on navigation and ... · perception and action: locomotion and...

Perception and Action:Locomotion and Navigation

Robert AllisonDept. of Electrical Engineering & Computer Science

Centre for Vision ResearchYork University

• Two components to navigating– wayfinding– travel

Wayfinding

• Refers to determination of current location and path to desired location

• Need to maneuver through environment and avoid obstacles when wayfinding (or wandering)

• Need cognitive information to support location determination and wayfinding behaviour(navigational awareness)– people believed to form cognitive maps or

models of the environment

• Aids to wayfinding– landmarks (natural or artificial)– path following/leaving trails– maps– memorable place names– compass/instruments– exocentric views– coordinate display, grid structure– constrained travel

• Early models posited distinct stages of navigational learning– Landmark knowledge– Route knowledge– Survey knowledge

• More recent models suggests parallel rather than hierarchical use of these types of representations

Typical tasks• Scene oriented pointing or relative direction

• Path integration (Blind walking, triangle completion)

• Sketched maps

Images from Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum)

Questions

Where am I?

Where do I want to go?

How do I get there?

How do I remember where I have explored?

How do I get home?

How do I move safely and efficiently?

How do I know distance and direction?

Where am I?

• What direction am I facing?• What position do I occupy in space?• What are metric properties of my environment

(scale, distance from where I want to be,…)

• All imagine some internal concept of allocentric space – perhaps a cognitive map?

Where am I?

• Work primarily in rodents led discoveries indicating a neural substrate for a cognitive map– O’Keefe (1971, 1978) ‘place’ cells in the

hippocampus– Edvard and May-Britt Moser (2005) found

‘grid’ cells in entorhinal cortex– O’Keefe, Moser and Moser shared 2014

Nobel prize in Physiology or Medicine

https://nba.uth.tmc.edu/neuroscience/s4/chapter05.html

• Part of limbic system• Paleocortex (3-layer

not 6-layer neocortex)• Significant

connections with other limbic structures and entorhinal cortex in temporal lobe

Coronal section of the brain of a macaque monkey, showing hippocampus (circled)

By brainmaps.org, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=3907696

https://nba.uth.tmc.edu/neuroscience/s4/chapter05.html

From Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum) p 46

a) Place Cell b) Grid Cell

Animal path with action potentials

Firing rate per area occupied

Place cells• Selective pyramidal cells in hippocampus• Normally quiet but fire vigourously when animal

is at specific positions in the environment• Fire in dark or light, evidence of vestibular

contributions (multisensory not purely visual)• Do not appear to be topographically mapped

even though individual cells are spatially selective (c.f. visual cortex)

Place cells• Firing patterns remap with changes in

environment, familiarity– Population encoding of precise position in

multiple environments?• Firing rates/selectivity varies with familiarity

– hippocampus has role memory encoding• Place cells may be active during locomotor

planning, sleep (dreaming, imagery?)

Derdikman & Moser (2010) Trends in Cognitive Sciences

• Evidence for place cells mixed in primates– Evidence for parahippocampal ‘view cells’

responsive to particular views of specific places (Rolls & O’Mara 1995)

– fMRI evidence for parahippocampal response to landmarks

Ekstrom et al. 2003 Nature 425 (6954): 184– 88. Recordings of place and view cell like responses from implanted electrodes in an epilepsy patient.

Grid Cells• Found in medial entorhinal cortex (and other

areas)• Multiple firing locations collectively form a grid • Grids differ in spacing, orientation, phase• Population may encode specific position

Derdikman & Moser (2010) Trends in Cognitive Sciences

• Consistent with forming a “metric for space” that could allow the calculation of the distance

• Substrate for place cells? – Debatable, strong evidence they are not

necessary for place cells• Typically have preferred angle (position and

orientation)• Idea is that humans (and rats) encode distance

and direction relative to the external environment

Derdikman & Moser (2010) Trends in Cognitive Sciences

Head Direction Cells

• Found in several limbic areas

• Fire selectively when rat faces in a preferred direction

• Continue to fire in the absence of visual inputs

• First spatial cells to emerge in the developing rat brain

From Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum) p 46

• Hippocampal cognitive map theory is limited in explaining human navigation– Limited impact of lesions– Significant interspecies differences– Less clearly selective responses than in rat– Evidence for topographic versus metric

representations, abstraction – Nevertheless, very influential

Visual Guidance of Locomotion and Navigation

Visual Guidance of Locomotion and Navigation

• Locomotion through the world is a basic and essential skill for most animals. In humans we additonally locomote in vehicles.

• Many aspects:

24

– Biomechanics/reflex– Gait control

25

– Footstep placement

26Stepping Stones of Memory - by nwwes, DeviantArt

– Obstacle avoidance

27By John Pavliga [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

– Steering and interception

28

– Path planning

29By John Pavliga [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

– Wayfinding– Navigating

30Longleat Hedge Maze

Non-Visual Guidance of Locomotion

• Researchers in gait and locomotion often ignore visual guidance

31

– Can walk in the dark– Gait control is (semi-)

automated– Spinal and higher motor

programs and reflexes

Visual Guidance of Locomotion

– Vision often assumed to guide and modulate, often in a sampled control mode (Patla,1997)• Vision can predict and help plan for disturbances

rather than only react

32

LC closed

LC open

Patla Gait and Posture, 1997, 5:54

Self-motion Perception

• Multisensory– Visual, Vestibular, Audition, Proprioception,

Somatosensation• Heading

– Large literature discussing the extraction of direction of motion from visual motion fields

33

Self-motion Perception

• Vection– Visually-mediated self motion perception– Many processes do not require vection

(preconscious)– May have an influence or subserve visual

guidance and higher level control?• Path Integration

– Distance travelled, orientation• Cognitive mapping, orientation, spatial updating

34

• Anecdotal evidence and introspection suggests active self motion important for spatial learning

• But what is active versus passive– Efferent motor commands determining path– Proprioceptive and vestibular information for

self-motion – Attention to navigation-related features– Cognitive navigational decisions– Mental manipulation of spatial information

• Chrastil & Warren (2011) reviewed and concluded– Sensory–motor components of physically

walking contribute to survey knowledge– Little effect of locomotor decision making– Attention to spatial relations contributes to route

and survey knowledge but not landmarks or boundaries. Attention effects depend on presence of idiothetic cues.

– Route and survey information are differentially encoded in subunits of working memory

Chrastil & Warren, JEP: LMC 2-13

Chrastil & Warren, JEP: LMC 2-13

• Addition of leg proprioception and vestibular information improved accuracy

• No significant effect of decision making• Note though that errors were very large

Visual guidance of steering and interception

39

Optic flow

• Flow patterns in velocity field can be analysed

• Could provide direction of heading (e.g. from focus of expansion)

Gibson, 1950

Optic flow

• Traditionally considered to be essential for control of locomotion

• Visually-induced self motion can be produced by optic flow (known as vection)

Optic flow

• Large literature discussing the extraction of heading (direction of motion) from visual motion fields

• Neurophysiological correlates in MST, V6 and other areas

42

Duffy C J J Neurophysiol 1998;80:1816-1827

• Eye movements, object motion and degrees of freedom complicate– Optic flow refers to velocity patterns in ‘optic

arrays’ not the retina – Need to account for eye movements to extract

optic flow from retinal flow– Real flow fields are complex

44

Optic flow

• Optic flow parsing– Object versus self– Translation

versus Rotation– Layout and depth

of environment / structure from motion

45

MacNeilage et al. PLoS ONE 2012, 7: e40264

Visual direction based locomotion

• In recent decades the idea that locomotion relies on the processing of optic flow has been challenged

• Alternative is locomotion is based upon egocentric direction (Rushton et al. Current Biology, 1998)

Such strategies can work for robots too

-2 -1 0 1 2

0

1

2

3

4

5

6

18o6o 12o

Wen, Rushton, Allison, 2002 47

Such strategies can be readily calibrated (and straightened)

-1 0 1 20

1

2

3

4

5

6

7

8

9

10

drift over-compensation400%200%

48

49Fajen, Warren, Perception, 2004, 33:689

Rushton & Allison, Displays, 2012, 34: 105 50

0 0.5 1 1.50

1

2

3

4

5

6

x (m)

z (m)

0 0.5 1 1.50

1

2

3

4

5

6

x (m)

z (m

)

-0.5 0 0.5 10

1

2

3

4

5

6

x (m)z (

m)

−20 −15 −10 −5 00

2

4

6

8

10

12

14

Lateral Distance (m)

Ball CarrierPursuerHeuristic Pursuer

Rushton & Allison, Displays, 2012, 34: 105 51

• Gaze behaviour during steering– Land & Lee (above Nature, 1994) suggested

fixation of the tangent point around a bend– Wilkie & Wann (2002) look where you want to go

52

53Wilkie, Wann & Allison, JEP: HPP, 2008, 34:1150

• Adopting forced gaze patterns degrades steering performance (Wilkie et al. 2008)

54Hayhoe & Ballard, TICS, 9:188, 2005

• Wilkie, Wann & Allison ACM TAP, 2011

55

56

Visual guidance of obstacle avoidance

57

58

Goal directed locomotion:

Obstacle avoidance:

Warren, Fajen Ecological Psychology, 2004, 16:61

59

Wen, Rushton, Allison, 2002

−2 −1 0 1 20

1

2

3

4

5

6

7

8

9

X distance (m)

Y di

stan

ce (m

)

HeuristicHumanObstacle (o)

Rushton & Allison, Displays, 2012, 34: 105

Imminence of collision

-1 0 10

1

2

3

4

5

6

-1 0 10

1

2

3

4

5

6

-1 0 10

1

2

3

4

5

6

Depends on speed and distance60

• Time To Collision (TTC)– Can be calculated from retinal information

(Hoyle, 1957; Lee, 1976)

– Psychophysical and neurophysiological evidence for sensitivity

– Time to respond– Many limitations in practice (spherical,

constant velocity, direct impact with eye , …)61

τ =θ!θ

• Time to Passage (TTP)– Several potential cues

to estimation– Can be generalized for

other judgements

62

Tresilian TICS, 1999, 3:301

sinβB

=sinψX

!XX=−VX=!BB+ !ψ cotψ = −

1τ+ !ψ cotψ

immediacy = VX=1−τ !ψ cotψ

τ

• Crossing distance– Time to passage is not sufficient, also need to

consider whether we will clear it? – Can be calculated in various ways, for

example based on visual direction, α, and disparity, ϕ.

63

XDist = I ⋅!α!φ

after Regan (1993)

64Ripatransone's tiny alleyway (http://www.cnn.com/2014/05/27/travel/italy-alley-fight/)

• Warren & Whang(1987) – participants rotated torso when aperture less than 1.3 times shoulder width

• When avoiding obstacle need to take into account body size and obstacle size

65

Rushton & Allison, Displays, 2012, 34: 105

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

66

Rushton & Allison, Displays, 2012, 34: 105

• If direction does not change when an object nears à collision course

• Off-centre collisions and near misses • Could use speed ratio based correlate of

crossing distance to identify possible future collisions (with stationary or moving objects)

67

Perception of Layout for Path Planning, Guidance & Control

68

Perception of Layout for Path Planning, Guidance & Control

• Sampling requirements• Gaze patterns• Binocular vision

– Range– Type of judgements

69

Visual Guidance of Locomotion

• Some work has looked at control of placement of footsteps (e.g. Matthis & Fajen, JEP: HPP, 2014)

70

71

• Their results suggested that visibility of the ground for next two to three footsteps was critical

• But do not fixate object during step preceding/during step over an obstacle (Patla1997)

• Depends on terrain (Matthis et al, Curr Biol 2018)

• Depends on terrain (Matthis et al, Curr Biol 2018)

• Binocular vision and gaze seem important• For example, stepping over obstacles (Hayhoe

et al, Vis Neurosci, 2009, 26:73

74

Subjects started at the left near the wall, stepped over the two boxes, walked on the right side of the table, then between table and wall, and stepped over the boxes again on the return path to the starting point.

75

• Binocular vision reduced task completion time, foot clearance and fixation requirements

76

Range of Stereopsis

• Conventional textbook wisdom is that stereopsis is only useful in relatively near space– Reach space– Nearer than 6 m

(Gregory, 1966)• Most study at 2 m or

less

d

D

a

Binocular disparity

δ= a1-a2 ≈ da/D2

77

Range of Stereopsis

Disparity thresholds as small as a couple seconds of arc (Howard, 1919)

Depth Constancy

• Metric depth from disparity requires scaling by distance

• Also size, velocity from retinal size and motion

• Many tasks do not require metric depth

• 3D shape does not require metric depth but needs scaling for distance

d

D

a

Binocular disparity

δ= (a1-a2) ≈ da/D2

79

Depth Constancy

• Does depth constancy for stereoscopic depth persist beyond 2 m?

• Can rich natural distance cues such as perspective improve stereoscopic depth constancy?

80

General Methods

• Wireless constellation of 100+ LEDs– Placed at and beyond

• 4.5 or 9.0 m in dark/lit room• 20 or 40 m in an old railway tunnel

– Irregularly spaced– Controlled as Bluetooth sensor network

• Binocular or monocular viewing, head fixed

81

Dark Condition

Subject’s View

82

Centre for Vision Research, York University

True Depth (m)0.0 0.5 1.0 1.5

Estim

ated

Dep

th (m

)

0.0

0.2

0.4

0.6

0.8

1.0

Disparity (arcminutes)0 5 10

Dark/MonocularDark/BinocularLight/MonocularLight/Binocular

4.5 m Reference

84

True Depth (m)0.0 0.5 1.0 1.5

Estim

ated

Dep

th (m

)

0.0

0.2

0.4

0.6

0.8

1.0

Disparity (arcminutes)0 1 2 3 4

Dark/MonocularDark/BinocularLight/MonocularLight/Binocular

9 m Reference

85

π

20 m reference

86

40 m reference

87

Results

• Binocular depth estimates tended to be larger than monocular depth estimates

• Perceived depth increased when presented in a full cue environment under binocular but not monocular viewing

• Depths were generally underestimated

88

Conclusions• Stereo perception of depth magnitude can be much

better than monocular well beyond the distances previously investigated

• Depth underestimation might be partly due to use of verbal depth/distance estimates

• Perspective and other cues are able to provide the absolute distance (D) factor necessary for depth constancy– Relative role of monocular and binocular

information during active motion yet to be investigated

89

90

Slant Perception at Distance

Judgements of the support, passability and effort to traverse provided by terrain

• Essential for locomotion• Critical for sport and skilled

visuomotor behaviour

91

Slant Perception at Distance

• Does binocular vision contribute to judgements of ground surface properties, especially slant and curvature?

• Stereopsis is ineffective beyond modest distances?– Little empirical work beyond 500 cm; only

stereoscopic depth thresholds– Theoretical range is much larger; binocular vision

improves performance to at least 100+ m

92

Sequential-surface-integration-process (SSIP)

• Vergence and stereopsis calibrate and anchor depth percepts in near space

• Extended by integrating monocular cues over the continuous ground plane

He et al. (2004) Perception, 33: 789

93

General Methods

• Wireless constellation of LEDs in dark room– Centred at 4.5 or 9.0 m– Irregularly spaced

• LEDs could be selectively lit to create – A single ground plane – Two adjacent planes (with change in slant)– Two interleaved planes (simulating uneven terrain)

• Binocular or monocular viewing, head fixed

97

Experiment 2Slant Difference

• Does stereopsis help discern changes in relative slant over a surface?

• On each trial subjects viewed two adjacent slanted surfaces, one further than the other

• Subjects indicated whether the configuration formed a convex or concave angle

• Variety of absolute angles and slant differences

98

ConcaveConvex

99Concave Convex

100

105

Summary

• Binocular vision can contribute to precise judgements of ground surface properties:

– Improves discrimination of deviations from horizontal (ground plane)

– Improves detection of slant change

– Improves sensitivity to lack of smoothness of a surface

106

Conclusions

• Binocular vision is useful for judgements of the layout and regularity of terrain to at least 9.0 metres– A range of considerable importance for

locomotion and moment to moment wayfinding or path planning.

• Contribution is not simply limited to calibration and anchoring of monocular cues in personal space