Perception: process of estimating world state What is Bayesian inference? How does it apply to perception? From Jonathan Pillow x + y = 9 What are x and y? This is an example of an ill-posed problem problem that has no unique solution Perception is also an ill-posed problem Example: Light Hitting Eye = Spectrum of Illuminant Reflectance function of surface Question we want to answer: what are the surface properties (i.e., color) of the surface? Equivalently: X Y = some cone responses R Given R, was Y? (youd have to know X to make it well-posed) Comparison patch Same light hits the eye from both patches Perception is an ill-posed problem Example #2: 3D world Question: whats out there in the 3D world? Ill-posed because there are infinitely many 3D worlds that give rise to the same 2D retinal image need some additional info to make it a well-posed problem 2D retinal image What might that information be? Figure 1. (a) The Necker Cube induces a bi-stable percept. (b) Disambiguation of the bi-stable Necker Cube percept by introducing an occlusion cue and a shadow. (c) An infinite number of 3D configurations could produce the same projection image. Here this fact is illustrated by the cast shadow on the tabletop, but the same projected images would be formed on the eyes retina. From Ernst, Bulthoff TICS 2004 Luckily, having some probabilistic information can help: x + y = 9 Tables showing past values of y: x y Given this information about past values, what would you guess to be the values of x? How confident are you in your answer? Bayes rule very simple formula for manipulating probabilities P(A | B) P(B) P(A) P(B | A) = conditional probability probability of B given that A occurred Formula for computing: P(whats in the world | sensory data) (This is what our brain wants to know!) P(sensory data | whats in the world) = likelihood P(whats in the world) = prior from (given by laws of physics; ambiguous because many world states could give rise to same sense data) (given by past experience) & Examples: Using Bayes rule to understand how the brain resolves ambiguous stimuli Many different 3D worlds can give rise to the same 2D retinal image The Ames Room How does our brain go about deciding which interpretation? A B P(image | A) and P(image | B) are equal! (both A and B could have generated this image) Lets use Bayes rule: P(A | image) = P(image | A) P(A) P(B | image) = P(image | B) P(B) Which of these is greater? Which dimples are popping out and which popping in? P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A Image equally likely to be OUT or IN given sensory data alone What we want to know: P(OUT | image) vs. P(IN | image) P(OUT | image) = P(image | OUT & light above) P(OUT) P(light above) P(IN | image) = P(image | IN & light below ) P(IN) P(light below) prior Which of these is greater? Apply Bayes rule: P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A P(OUT | image) = P(image | OUT & light above) P(OUT) P(light above) P(IN | image) = P(image | IN & light below ) P(IN) P(light below) P(OUT | image) = A 0.5 P(light above) P(IN | image) = A 0.5 P(light below) Lets say: Light above is 10 times more likely than light below P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A P(OUT | image) = P(image | OUT & light above) P(OUT) P(light above) P(IN | image) = P(image | IN & light below ) P(IN) P(light below) P(OUT | image) = A 0.5 P(light above) = 5 A P(IN | image) = A 0.5 P(light below) = 0.5 A Lets say: Light above is 10 times more likely than light below Bayesian account: Out is 10 times more likely! Perception is an ill-posed problem equivalently: the world is still ambiguous even given all our sensory information Probabilistic information can be used to solve ill-posed problems (via Bayes theorem) Bayes theorem: The brain takes into account prior knowledge to figure out whats in the world given our sensory information Summary P(world | sense data) P(sense data | world ) P(world) prior posterior likelihood (Note that the posterior can be considered the prior for the next time step in an ongoing learning process) Two take-home facts about what it means to be Bayesian in psychology / neuroscience: 1. Use priors we recognize that perception is ambiguous (ill-posed), and that the only / best way to deal with that is to use prior information (built up over last 2 minutes, last 2 days, last 2 decades, last 2M years, etc.) Weiss, Simoncelli & Adelson, Motion illusions as optimal percepts. Nature Neuroscience (2002) ref: 2. Keep around a full posterior distribution use probabilistic information (dont just store the most likely answer also store an estimate of our uncertainty) Ernst, M. & Banks, M. Humans integrate visual and haptic information in a statistically optimal fashion Nature, 2002 Krding, K. & Wolpert, D. Bayesian Integration in Sensorimotor Learning Nature, 2004 refs: (cue combination) Visual capture Vision and haptics view object through a cylindrical lens: vision dominates, but a small effect Of haptic info. Vision and audition sounds are localized to visual source eg speakers mouth In other instances, senses complement each other eg feeling an object where there is no conflict, info from back is given by haptics, front by vision. Auditory capture Number of beeps determines whether single or multiple flashes are seen Dominance determined by reliability of the estimate. Figure 3. Visualhaptic size-discrimination performance determined with a 2-interval forced-choice task [29]. The relative reliabilities of the individual signals feeding into the combined percept were manipulated by adding noise to the visual display. With these different relative reliabilities four discrimination curves were measured. As the relative visual reliability decreased, the perceived size as indicated by the point of subjective equality (PSE) was increasingly determined by the haptic size estimate (haptic standard, SH) and less by the visual size estimate (visual standard, SV). This demonstrates the weighting behaviour the brain adopts and the smooth change from visual dominance (red circles) to haptic dominance (orange triangles). As shown, the PSEs predicted from the individual visual and haptic discrimination performance (larger symbols with black outline) correspond closely to the empirically determined PSEs in the combined visualhaptic discrimination task. (JND. just noticeable difference.) What does it mean to perceive optimally? What is a Bayesian model of human perception? How can we formulate / fit Bayesian models? What does it mean to perceive optimally? 1. Our senses provide noisy measurements of the environment (e.g., something moving in the grass) 2. We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying wildebeast when the true stimulus was lion) 3. We have some beliefs about the underlying probability of different stimuli (e.g., wildebeast more common than lion) Assumptions: General Goal: Decide: what stimulus is present? (better to say act) What does it mean to perceive optimally? 1. Our senses provide noisy measurements of the environment (e.g., something moving in the grass) 2. We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying wildebeast when the true stimulus was lion) 3. We have some beliefs about the underlying probability of different stimuli (e.g., wildebeast more common than lion) Assumptions: Solution: Given a noisy measurement, pick stimulus that minimizes the expected cost. (better to say act) What does it mean to perceive optimally? 1. Our senses provide noisy measurements of the environment (e.g., something moving in the grass) 2. We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying wildebeest when the true stimulus was lion) 3. We have some beliefs about the underlying probability of different stimuli (e.g., wildebeest more common than lion) Assumptions: Solution: Given a noisy measurement, pick stimulus that minimizes the expected cost. Bayesian Decision Theory (better to say act) noise distribution / likelihood cost function prior Two basic approaches: 1. Objective approach: measure these three ingredients in the real world. Determine if observers are in fact optimal. (Bayesian ideal observer analysis) noise distribution / likelihood cost function prior What is the true sensory noise? (Or impose noise that swamps internal noise.) What is the true cost of saying wildebeest when its a lion, vs. saying lion when its a wildebeest? (Or impose costs: e.g., pay subjects) Measure the relative population size of wildebeests and lions (Or impose prior in an artificial task). Definitive answer: yes or no. Can quantify how close observers are to optimal Two basic approaches: 2. Subjective approach: measure stimuli and observer responses. Is there some setting of these three under which observer can be said to be Bayes optimal? noise distribution / likelihood cost function prior Estimate this! Under-constrained problem in general - have to make some assumptions Assume to be Gaussian (estimate variance) Assume some standard choice (e.g., mean-squared error) Not clear if the answer is ever no. (Would anyone publish a null result?) However: do the noise, cost and prior make sense? Is it reasonable to think that this is what subjects are really doing? (Offers parsimonious explanation of percepts) 1.What are the major transformations of the visual signals imposed by the retina and what are the perceptual consequences of those transformations? 2. Describe the organization of the central visual pathways. How is information about surfaces, depth and motion encoded? 3. Describe dorsal and ventral streams in visual processing. How do they differ. Discuss and evaluate Goodales interpretation of the different roles. 4. What is visual attention? How does it influence perception, and what are some of the neural mechanisms? 5. Describe the various ways that optic flow can be used in vision? 6. What are the different kinds of eye movements? Describe their properties, neural mechanisms, and their perceptual consequences. 7. What kinds of information are conveyed by the somatosensory system? How is that information encoded in the nervous system? How does it interact with control of movement? 8. What is known about sensory prediction? Why is it important? 9. What are similarities and differences between the different sensory modalities we have discussed. Major transformations of the light signal in the retina: 1.Temporal filtering visual response slower than input signal. 2. Spatial filtering local signals are combined across space to varying degrees. 3. Light adaptation retina modifies responsiveness depending on average light level. 4.Color coding trichromacy and color opponency Major transformations of the light signal in the retina: 1.Temporal filtering reduced response to high temporal frequencies Temporal integration a strong 1 msec flash is equivalent to a weaker 50 msec flash. 2. Spatial filtering: - Anatomical organization of photoreceptors provides high acuity in fovea with rapid fall-off in the periphery. (photoreceptor density) -Convergence of photoreceptors onto ganglion cells also leads to acuity limitations in the peripheral retina. (1 cone per midget cell in fovea) - Center-surround antagonism reduces sensitivity to uniform fields. 3. Light adaptation sensitivity regulation - adjustment of operating range to mean light level. (Light level range, ganglion cells, 10 2 range.) 4.Color opponency. Organization of 3 cone photoreceptors into color opponent signals (Luminance, Red-Green, Yellow-Blue) Describe how attention affects perception. How does attention affect neural responses? What kinds of information are conveyed by the somatosensory system? How is that information encoded in the nervous system? Describe the various ways that optic flow can be used in vision? What is known about sensory prediction? Why is it important?