perception and action guest lecture on navigation and ... · perception and action: locomotion and...
TRANSCRIPT
Perception and Action:Locomotion and Navigation
Robert AllisonDept. of Electrical Engineering & Computer Science
Centre for Vision ResearchYork University
• Two components to navigating– wayfinding– travel
Wayfinding
• Refers to determination of current location and path to desired location
• Need to maneuver through environment and avoid obstacles when wayfinding (or wandering)
• Need cognitive information to support location determination and wayfinding behaviour(navigational awareness)– people believed to form cognitive maps or
models of the environment
• Aids to wayfinding– landmarks (natural or artificial)– path following/leaving trails– maps– memorable place names– compass/instruments– exocentric views– coordinate display, grid structure– constrained travel
• Early models posited distinct stages of navigational learning– Landmark knowledge– Route knowledge– Survey knowledge
• More recent models suggests parallel rather than hierarchical use of these types of representations
Typical tasks• Scene oriented pointing or relative direction
• Path integration (Blind walking, triangle completion)
• Sketched maps
Images from Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum)
Questions
Where am I?
Where do I want to go?
How do I get there?
How do I remember where I have explored?
How do I get home?
How do I move safely and efficiently?
How do I know distance and direction?
Where am I?
• What direction am I facing?• What position do I occupy in space?• What are metric properties of my environment
(scale, distance from where I want to be,…)
• All imagine some internal concept of allocentric space – perhaps a cognitive map?
Where am I?
• Work primarily in rodents led discoveries indicating a neural substrate for a cognitive map– O’Keefe (1971, 1978) ‘place’ cells in the
hippocampus– Edvard and May-Britt Moser (2005) found
‘grid’ cells in entorhinal cortex– O’Keefe, Moser and Moser shared 2014
Nobel prize in Physiology or Medicine
https://nba.uth.tmc.edu/neuroscience/s4/chapter05.html
• Part of limbic system• Paleocortex (3-layer
not 6-layer neocortex)• Significant
connections with other limbic structures and entorhinal cortex in temporal lobe
Coronal section of the brain of a macaque monkey, showing hippocampus (circled)
By brainmaps.org, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=3907696
https://nba.uth.tmc.edu/neuroscience/s4/chapter05.html
From Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum) p 46
a) Place Cell b) Grid Cell
Animal path with action potentials
Firing rate per area occupied
Place cells• Selective pyramidal cells in hippocampus• Normally quiet but fire vigourously when animal
is at specific positions in the environment• Fire in dark or light, evidence of vestibular
contributions (multisensory not purely visual)• Do not appear to be topographically mapped
even though individual cells are spatially selective (c.f. visual cortex)
Place cells• Firing patterns remap with changes in
environment, familiarity– Population encoding of precise position in
multiple environments?• Firing rates/selectivity varies with familiarity
– hippocampus has role memory encoding• Place cells may be active during locomotor
planning, sleep (dreaming, imagery?)
Derdikman & Moser (2010) Trends in Cognitive Sciences
• Evidence for place cells mixed in primates– Evidence for parahippocampal ‘view cells’
responsive to particular views of specific places (Rolls & O’Mara 1995)
– fMRI evidence for parahippocampal response to landmarks
Ekstrom et al. 2003 Nature 425 (6954): 184– 88. Recordings of place and view cell like responses from implanted electrodes in an epilepsy patient.
Grid Cells• Found in medial entorhinal cortex (and other
areas)• Multiple firing locations collectively form a grid • Grids differ in spacing, orientation, phase• Population may encode specific position
Derdikman & Moser (2010) Trends in Cognitive Sciences
• Consistent with forming a “metric for space” that could allow the calculation of the distance
• Substrate for place cells? – Debatable, strong evidence they are not
necessary for place cells• Typically have preferred angle (position and
orientation)• Idea is that humans (and rats) encode distance
and direction relative to the external environment
Derdikman & Moser (2010) Trends in Cognitive Sciences
Head Direction Cells
• Found in several limbic areas
• Fire selectively when rat faces in a preferred direction
• Continue to fire in the absence of visual inputs
• First spatial cells to emerge in the developing rat brain
From Human Spatial Navigation ((2018) Ekstrom, Spiers, Bohbot, Rosenbaum) p 46
• Hippocampal cognitive map theory is limited in explaining human navigation– Limited impact of lesions– Significant interspecies differences– Less clearly selective responses than in rat– Evidence for topographic versus metric
representations, abstraction – Nevertheless, very influential
Visual Guidance of Locomotion and Navigation
Visual Guidance of Locomotion and Navigation
• Locomotion through the world is a basic and essential skill for most animals. In humans we additonally locomote in vehicles.
• Many aspects:
24
– Biomechanics/reflex– Gait control
25
– Footstep placement
26Stepping Stones of Memory - by nwwes, DeviantArt
– Obstacle avoidance
27By John Pavliga [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
– Steering and interception
28
– Path planning
29By John Pavliga [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
– Wayfinding– Navigating
30Longleat Hedge Maze
Non-Visual Guidance of Locomotion
• Researchers in gait and locomotion often ignore visual guidance
31
– Can walk in the dark– Gait control is (semi-)
automated– Spinal and higher motor
programs and reflexes
Visual Guidance of Locomotion
– Vision often assumed to guide and modulate, often in a sampled control mode (Patla,1997)• Vision can predict and help plan for disturbances
rather than only react
32
LC closed
LC open
Patla Gait and Posture, 1997, 5:54
Self-motion Perception
• Multisensory– Visual, Vestibular, Audition, Proprioception,
Somatosensation• Heading
– Large literature discussing the extraction of direction of motion from visual motion fields
33
Self-motion Perception
• Vection– Visually-mediated self motion perception– Many processes do not require vection
(preconscious)– May have an influence or subserve visual
guidance and higher level control?• Path Integration
– Distance travelled, orientation• Cognitive mapping, orientation, spatial updating
34
• Anecdotal evidence and introspection suggests active self motion important for spatial learning
• But what is active versus passive– Efferent motor commands determining path– Proprioceptive and vestibular information for
self-motion – Attention to navigation-related features– Cognitive navigational decisions– Mental manipulation of spatial information
• Chrastil & Warren (2011) reviewed and concluded– Sensory–motor components of physically
walking contribute to survey knowledge– Little effect of locomotor decision making– Attention to spatial relations contributes to route
and survey knowledge but not landmarks or boundaries. Attention effects depend on presence of idiothetic cues.
– Route and survey information are differentially encoded in subunits of working memory
Chrastil & Warren, JEP: LMC 2-13
Chrastil & Warren, JEP: LMC 2-13
• Addition of leg proprioception and vestibular information improved accuracy
• No significant effect of decision making• Note though that errors were very large
Visual guidance of steering and interception
39
Optic flow
• Flow patterns in velocity field can be analysed
• Could provide direction of heading (e.g. from focus of expansion)
Gibson, 1950
Optic flow
• Traditionally considered to be essential for control of locomotion
• Visually-induced self motion can be produced by optic flow (known as vection)
Optic flow
• Large literature discussing the extraction of heading (direction of motion) from visual motion fields
• Neurophysiological correlates in MST, V6 and other areas
42
Duffy C J J Neurophysiol 1998;80:1816-1827
• Eye movements, object motion and degrees of freedom complicate– Optic flow refers to velocity patterns in ‘optic
arrays’ not the retina – Need to account for eye movements to extract
optic flow from retinal flow– Real flow fields are complex
44
Optic flow
• Optic flow parsing– Object versus self– Translation
versus Rotation– Layout and depth
of environment / structure from motion
45
MacNeilage et al. PLoS ONE 2012, 7: e40264
Visual direction based locomotion
• In recent decades the idea that locomotion relies on the processing of optic flow has been challenged
• Alternative is locomotion is based upon egocentric direction (Rushton et al. Current Biology, 1998)
Such strategies can work for robots too
-2 -1 0 1 2
0
1
2
3
4
5
6
18o6o 12o
Wen, Rushton, Allison, 2002 47
Such strategies can be readily calibrated (and straightened)
-1 0 1 20
1
2
3
4
5
6
7
8
9
10
drift over-compensation400%200%
48
49Fajen, Warren, Perception, 2004, 33:689
Rushton & Allison, Displays, 2012, 34: 105 50
0 0.5 1 1.50
1
2
3
4
5
6
x (m)
z (m)
0 0.5 1 1.50
1
2
3
4
5
6
x (m)
z (m
)
-0.5 0 0.5 10
1
2
3
4
5
6
x (m)z (
m)
−20 −15 −10 −5 00
2
4
6
8
10
12
14
Lateral Distance (m)
Ball CarrierPursuerHeuristic Pursuer
Rushton & Allison, Displays, 2012, 34: 105 51
• Gaze behaviour during steering– Land & Lee (above Nature, 1994) suggested
fixation of the tangent point around a bend– Wilkie & Wann (2002) look where you want to go
52
53Wilkie, Wann & Allison, JEP: HPP, 2008, 34:1150
• Adopting forced gaze patterns degrades steering performance (Wilkie et al. 2008)
54Hayhoe & Ballard, TICS, 9:188, 2005
• Wilkie, Wann & Allison ACM TAP, 2011
55
56
Visual guidance of obstacle avoidance
57
58
Goal directed locomotion:
Obstacle avoidance:
Warren, Fajen Ecological Psychology, 2004, 16:61
59
Wen, Rushton, Allison, 2002
−2 −1 0 1 20
1
2
3
4
5
6
7
8
9
X distance (m)
Y di
stan
ce (m
)
HeuristicHumanObstacle (o)
Rushton & Allison, Displays, 2012, 34: 105
Imminence of collision
-1 0 10
1
2
3
4
5
6
-1 0 10
1
2
3
4
5
6
-1 0 10
1
2
3
4
5
6
Depends on speed and distance60
• Time To Collision (TTC)– Can be calculated from retinal information
(Hoyle, 1957; Lee, 1976)
– Psychophysical and neurophysiological evidence for sensitivity
– Time to respond– Many limitations in practice (spherical,
constant velocity, direct impact with eye , …)61
τ =θ!θ
• Time to Passage (TTP)– Several potential cues
to estimation– Can be generalized for
other judgements
62
Tresilian TICS, 1999, 3:301
sinβB
=sinψX
!XX=−VX=!BB+ !ψ cotψ = −
1τ+ !ψ cotψ
immediacy = VX=1−τ !ψ cotψ
τ
• Crossing distance– Time to passage is not sufficient, also need to
consider whether we will clear it? – Can be calculated in various ways, for
example based on visual direction, α, and disparity, ϕ.
63
XDist = I ⋅!α!φ
after Regan (1993)
64Ripatransone's tiny alleyway (http://www.cnn.com/2014/05/27/travel/italy-alley-fight/)
• Warren & Whang(1987) – participants rotated torso when aperture less than 1.3 times shoulder width
• When avoiding obstacle need to take into account body size and obstacle size
65
Rushton & Allison, Displays, 2012, 34: 105
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2
-1.5
-1
-0.5
0
0.5
66
Rushton & Allison, Displays, 2012, 34: 105
• If direction does not change when an object nears à collision course
• Off-centre collisions and near misses • Could use speed ratio based correlate of
crossing distance to identify possible future collisions (with stationary or moving objects)
67
Perception of Layout for Path Planning, Guidance & Control
68
Perception of Layout for Path Planning, Guidance & Control
• Sampling requirements• Gaze patterns• Binocular vision
– Range– Type of judgements
69
Visual Guidance of Locomotion
• Some work has looked at control of placement of footsteps (e.g. Matthis & Fajen, JEP: HPP, 2014)
70
71
• Their results suggested that visibility of the ground for next two to three footsteps was critical
• But do not fixate object during step preceding/during step over an obstacle (Patla1997)
• Depends on terrain (Matthis et al, Curr Biol 2018)
• Depends on terrain (Matthis et al, Curr Biol 2018)
• Binocular vision and gaze seem important• For example, stepping over obstacles (Hayhoe
et al, Vis Neurosci, 2009, 26:73
74
Subjects started at the left near the wall, stepped over the two boxes, walked on the right side of the table, then between table and wall, and stepped over the boxes again on the return path to the starting point.
75
• Binocular vision reduced task completion time, foot clearance and fixation requirements
76
Range of Stereopsis
• Conventional textbook wisdom is that stereopsis is only useful in relatively near space– Reach space– Nearer than 6 m
(Gregory, 1966)• Most study at 2 m or
less
d
D
a
Binocular disparity
δ= a1-a2 ≈ da/D2
77
Range of Stereopsis
Disparity thresholds as small as a couple seconds of arc (Howard, 1919)
Depth Constancy
• Metric depth from disparity requires scaling by distance
• Also size, velocity from retinal size and motion
• Many tasks do not require metric depth
• 3D shape does not require metric depth but needs scaling for distance
d
D
a
Binocular disparity
δ= (a1-a2) ≈ da/D2
79
Depth Constancy
• Does depth constancy for stereoscopic depth persist beyond 2 m?
• Can rich natural distance cues such as perspective improve stereoscopic depth constancy?
80
General Methods
• Wireless constellation of 100+ LEDs– Placed at and beyond
• 4.5 or 9.0 m in dark/lit room• 20 or 40 m in an old railway tunnel
– Irregularly spaced– Controlled as Bluetooth sensor network
• Binocular or monocular viewing, head fixed
81
Dark Condition
Subject’s View
82
Centre for Vision Research, York University
True Depth (m)0.0 0.5 1.0 1.5
Estim
ated
Dep
th (m
)
0.0
0.2
0.4
0.6
0.8
1.0
Disparity (arcminutes)0 5 10
Dark/MonocularDark/BinocularLight/MonocularLight/Binocular
4.5 m Reference
84
True Depth (m)0.0 0.5 1.0 1.5
Estim
ated
Dep
th (m
)
0.0
0.2
0.4
0.6
0.8
1.0
Disparity (arcminutes)0 1 2 3 4
Dark/MonocularDark/BinocularLight/MonocularLight/Binocular
9 m Reference
85
π
20 m reference
86
40 m reference
87
Results
• Binocular depth estimates tended to be larger than monocular depth estimates
• Perceived depth increased when presented in a full cue environment under binocular but not monocular viewing
• Depths were generally underestimated
88
Conclusions• Stereo perception of depth magnitude can be much
better than monocular well beyond the distances previously investigated
• Depth underestimation might be partly due to use of verbal depth/distance estimates
• Perspective and other cues are able to provide the absolute distance (D) factor necessary for depth constancy– Relative role of monocular and binocular
information during active motion yet to be investigated
89
90
Slant Perception at Distance
Judgements of the support, passability and effort to traverse provided by terrain
• Essential for locomotion• Critical for sport and skilled
visuomotor behaviour
91
Slant Perception at Distance
• Does binocular vision contribute to judgements of ground surface properties, especially slant and curvature?
• Stereopsis is ineffective beyond modest distances?– Little empirical work beyond 500 cm; only
stereoscopic depth thresholds– Theoretical range is much larger; binocular vision
improves performance to at least 100+ m
92
Sequential-surface-integration-process (SSIP)
• Vergence and stereopsis calibrate and anchor depth percepts in near space
• Extended by integrating monocular cues over the continuous ground plane
He et al. (2004) Perception, 33: 789
93
General Methods
• Wireless constellation of LEDs in dark room– Centred at 4.5 or 9.0 m– Irregularly spaced
• LEDs could be selectively lit to create – A single ground plane – Two adjacent planes (with change in slant)– Two interleaved planes (simulating uneven terrain)
• Binocular or monocular viewing, head fixed
97
Experiment 2Slant Difference
• Does stereopsis help discern changes in relative slant over a surface?
• On each trial subjects viewed two adjacent slanted surfaces, one further than the other
• Subjects indicated whether the configuration formed a convex or concave angle
• Variety of absolute angles and slant differences
98
ConcaveConvex
99Concave Convex
100
105
Summary
• Binocular vision can contribute to precise judgements of ground surface properties:
– Improves discrimination of deviations from horizontal (ground plane)
– Improves detection of slant change
– Improves sensitivity to lack of smoothness of a surface
106
Conclusions
• Binocular vision is useful for judgements of the layout and regularity of terrain to at least 9.0 metres– A range of considerable importance for
locomotion and moment to moment wayfinding or path planning.
• Contribution is not simply limited to calibration and anchoring of monocular cues in personal space