neural correlates of degraded picture perception

Neural Correlates of Degraded Picture Perception

Tom Busey, Rob Goldstone and Bethany Knapp

General Method

• Record brain activity during perception of degraded pictures.

• Change knowledge by sometimes showing undegraded picture.

• Always record during presentation of degraded picture.

• Also change experience by showing primes.

(4 repetitions)

What was the picture?1 Raccoon2 Baseball player3 Chairs4 Two women talking5 Bird facing right6 Trees7 Monkey8 Don’t know (no clue)

(flip between degraded and undegraded images)

(4 repetitions)

Experimental Design

4 Repetitionsdegraded pict

QuestionsFlipping but blank screen instead of undegraded picture

Flipping with undegraded picture

4 Repetitions degraded pict

Pre-exposure Post-exposure

Process repeated 60 times30 Pictures in each condition.Each condition is replicated 120 timesper subject x 3 subjects.

One Trial

(4 repetitions)

What was the picture?1 Dog2 Woman with hands3 Camel facing left4 Horse5 Raccoon6 Bird facing right7 Baseball player8 Don’t know (no clue)

(flip between degraded image and gray screen)

(4 repetitions)

Experimental Design

4 Repetitionsdegraded pict

QuestionsFlipping but blank screen instead of undegraded picture

Flipping with undegraded picture

4 Repetitions degraded pict

Pre-exposure Post-exposure

Process repeated 60 times30 Pictures in each condition.Each condition is replicated 120 timesper subject x 3 subjects.

One Trial

Conditions Average

-3.87

+18

0Time (ms)

1000

Front of Head

ERP Data From Experiment 1

Back of Head

Left Side Right Side

Conditions Average

-3.87

+18

0Time (ms)

1000

Pz (Back Middle of Head)Cz (Middle of Head)


Back of Head

Left Side Right Side

Front of Head

Conditions Average

-3.87

+18

0Time (ms)

1000


KnowledgeNo

Knowledge

Prior to Flipping Blue Green

After Flipping Red Cyan

PzCz

ICA Decomposition

• Goal: Recover a set of independent signals (components) that were mixed together in the EEG electrodes.

• Recovered components can be considered as latent variables or factors like those in Factor Analysis.

• Not the same as dipole analysis, but dipoles can be fit to ICA components.

ICA Decomposition• Independence:

– Knowing something about one component tells you nothing about the state of the other components.

– Joint density equals the product of the marginal densities: p(y1,y2)= p(y1)p(y2)

– Independent components are uncorrelated, but uncorrelated factors (from PCA) need not be independent.

• Decomposition is an interative process– Knows nothing about time or conditions– Adjusts weight values assigned to each electrode to find

components with distributions that are as independent as possible.

– Decomposition is not unique, but we see good convergence over repeated simulations.

Component Maps from Experiment 1

ICA Decomposition• Visualization

– Compute components (which are defined by their weights) from concatenated individual subject data.

– Visualize using grand average data.– Look for components that differentiate between the

conditions– Back project each component to voltage, which simulates

what we would have recorded if this was the only active component.

• Statistical Issues– Statistical analysis of ICA components is still relatively

new.– Stress replication across experiments over hypothesis

testing.

ICA Component 4 Average

-2.14

+5.47

0

Time (ms)

1000

ICA Component From Experiment 1

KnowledgeNo

Knowledge



PzCz


-3.18

+2.87

0

Time (ms)

1000


KnowledgeNo

Knowledge



O2O1


-2.81

+5.63

0

Time (ms)

1000


KnowledgeNo

Knowledge



O2O1


-2.06

+5.43

0Time (ms)

1000


KnowledgeNo

Knowledge



PzCz


-1.37

+2.65

0

Time (ms)

1000


KnowledgeNo

Knowledge



PzCz

Experiment 1 Conclusions• Experience with the real image produces

large centrally-located changes in the ERP beginning around 400 ms.

• Also see an ICA component that separates out this condition in perceptual regions as early as 250 ms.

• Knowledge of the picture's gist but not its interpretation produces an ICA component that separates at 400 ms and is localized to the occipital portion of the head.

Experiment 2• How does prior knowledge about the

content of the picture help you interpret the degraded image?

• Precede the degraded image with a text description of the contents, called a prime.

• How are the components identified by ICA affected by the prime?

Experiment 2 Design Changes• Pre-expose half of the pictures at the

start of the experiment, along with their primes and degraded versions.

Prior to experiment

bird facing right

bird facing right

Prior to experiment

Experiment 2 Design Changes• Pre-expose half of the pictures at the

start of the experiment, along with their primes and degraded versions.

• The other half of the pictures are shown in their degraded form only.

Prior to experiment

baseball player

Experimental Design

Incorrect Description(comes fromother pictures)

Delay Degraded Picture

30 Pictures in each conditionEach condition is replicated 120 timesper subject x 4 subjects.

Prime

One trial

Correct Description

1000 ms

1000 ms

1000 ms

(One Trial)

baseball player

Experimental Design-EndAllow flipping with description (prime)Ask subject: How well did you figure this picture out?

End of experiment

baseball player

How well did you interpret this picture?

• No Clue: I never figured it out.• Partial: I got some of the details, or I figured it out

midway through the experiment.• Knew: I figured this picture out almost

immediately (or it was shown to me at the beginning of the experiment).

Exclude data from pictures in the No-Knowledge condition that subjects figure out.

Experimental DesignPrior Knowledge?

Knowledge No-KnowledgePrime veracity

Correct

Incorrect

Blue

Red Cyan

Green

Incorrect primes come from other Knowledge or No-Knowledge pictures. No information in the prime as to whether a Knowledge or No-Knowledge picture would appear. But, some No-Knowledge pictures will have primes associated with known pictures.

Bias Model

• Prime makes it more likely that picture will be interpreted in a manner consistent with the prime.– Bias or preference effect. Verbal/semantic in

nature.– Doesn't involve recall of image from memory.– Look at processing of prime to see if get

differences that may reflect recall of memory.

Picture Retrieval Model• Prime causes a retrieval of undegraded

picture from memory. Association between word and interpreted picture important.– Affects only pictures that had undegraded

versions presented, or those that the observer figured out.

– Predicts no effect of prime veracity on no-clue pictures (unless observer can recall and use uninterpreted blobs from degraded picture).

Perceptual Facilitation

• Prime affects the perceptual processing of the picture– Helps bind feature elements, separate figure

from ground.• e.g. outdoor scenes might be interpreted differently

– Evidence from Yu and Blake (1992). Degraded images from real scenes dominated binocular rivalry even though scene could not be interpreted.

Which image comes from a real Dalmatian picture?

Conditions Average

-4.8

+19.6

0Time (ms)

1000

ERP Data From Experiment 2- Eight Subjects

KnowledgeCorrect Prime Blue

Incorrect Prime Red

PzCz

Conditions Average

-4.8

+19.6

0Time (ms)

1000

ERP Data From Experiment 2- Eight Subjects

KnowledgeNo

KnowledgeCorrect Prime Blue Green

Incorrect Prime Red Cyan

PzCz


-2.66

+3.63

0

Time (ms)

1000


KnowledgeNo



PzCz


-4.79

+3.15

0

Time (ms)

1000


KnowledgeNo



O2O1

Experiment 2 Conclusions• Preceeding a known picture with a

correct prime produces differences in the onset and peak latency of a centrally-located source.– Differences begin at about 350-400 ms.– Is the correctly primed picture advanced,

or the incorrectly primed picture delayed?• A single ICA component captures both

correctly and incorrectly primed known images. Suggests that the neural processes are similar, but offset in time.

Experiment 2 Conclusions• ICA decomposition also reveals a

separation of the correctly-primed known condition from the other conditions that is located near the perceptual regions.– Differences begin at about 300 ms.– Consistent with a facilitation model that

involves perceptual regions of the brain.• Unknown pictures show an effect of the

prime!

Experiment 3 Design Changes• Included a neutral prime condition• Primes for unknown pictures come only

from pictures they never see– Can't form mental picture from these

primes, because never see the picture.• Forced-choice test at the end of the

experiment with only the degraded versions shown.– Again exclude pictures that subjects figure

out by themselves.• 17 Subjects (twice as many as

Experiment 2)

Experimental Design

Incorrect Description(comes from

never-seen pictures)

Delay Degraded Picture

30 Pictures in each conditionEach condition is replicated 120 timesper subject x 4 subjects.

Prime

One trial

Correct Description

1000 ms

1000 ms

1000 ms*******

Experimental DesignPrior Knowledge?

Knowledge No-KnowledgePrime Type

Correct

Neutral

Blue

Green Magenta

Cyan

Incorrect primes come from pictures they never see.Netural prime was '********'A given picture always had same incorrect prime for the 4 trials in which it was incorrectly primed.

Incorrect Red Black

Conditions Average

-1.8

+17.4

0 Time (ms) 1000


Prime Knowledge

Correct Blue

Neutral Green

Incorrect Red

PzCz

Conditions Average

-1.8

+17.4

0 Time (ms) 1000


Prime Knowledge No Knowledge

Correct Blue Cyan

Neutral Green Magenta

Incorrect Red Black

PzCz


-2.23

+5.15

0

Time (ms)

1000



Correct Blue Cyan


Incorrect Red Black

PzCz


-2.19

+5.15

0

Time (ms)

1000



Correct Blue Cyan


Incorrect Red Black

PzCz


-0.507

+9.67

0 Time (ms) 1000



Correct Blue Cyan


Incorrect Red Black

O2O1


-0.278

+8.44

0 Time (ms) 1000



Correct Blue Cyan


Incorrect Red Black

O2O1


-4.79

+3.15

0

Time (ms)

1000


KnowledgeNo



O2O1


-0.528

+8.09

0 Time (ms) 1000

ICA Component From Experiment 2 using Experiment 3 weights

KnowledgeNo



O2O1


-0.582

+8.18

0 Time (ms) 1000

ICA Component From Experiment 2 using Experiments 2 and 3 weights

KnowledgeNo



O2O1

Experiment 3 Conclusions• Correctly-primed pictures show a component with an

earlier onset and peak latency than either the incorrect or netural primed conditions.– One possibilitiy: A correct prime may speed up processing

of the degraded picture.• Again we see a component in the ICA

decomposition that has the correctly-primed picture processing separating out at about 250-300 ms in channel locations associated with perceptual regions.– Involvement of perceptual regions in facilitation.

Experiment 3 Conclusions• Unknown pictures showed no effect of the prime

veracity. No ICA component separated the correctly-primed unknown pictures from the neutral and incorrectly-primed unknown pictures.– Experiment 2 difference between correctly and

incorrectly primed unknown pictures is likely due to a mismatch between the picture engendered by the prime and the uninterpretable degraded image.

– No real evidence for perceptual facilitation of uninterpreted pictures.

General Conclusions• Knowledge of the pictures intepretation produces

large changes in the ERP in central recording sites.– Consistently revealed by ICA.

• Some evidence for changes in perceptual regions as well.

• A correct prime appears to speed up interpreation of the picture.

• May also produce changes in the perceptual regions of the brain.

How Does ICA Help?• Allows for comparison between conditions while

removing irrelevant activity (e.g. neutral primes).• Demonstrates that two sets of activity have the

same distributions on the scalp but differ in time (e.g. known pictures and 3 types of primes).

• Highlights perceptual activity.• Allows comparisons across experiments even if

they don't match in designs.• The potential for overinterpretation is vast, and

replication is important.

EEG Data From Experiment 3


-0.967

+2.44

0 1000


Correct Blue Cyan


Incorrect Red Black



-0.437

+2.44Prime Knowledge No Knowledge

Correct Blue Cyan


Incorrect Red Black



-2.38

+8.75

0 1000

KnowledgeNo



neural correlates of degraded picture perception

Documents