Committee UpdateBuilding a visual hierarchy

Andrew Smith

30 July 2008

Outline Confabulation theory

Summary Comparisons to other AI techniques

Human Visual System

Building A Visual Hierarchy Learning Inference

Texture modeling (applications)

Future work (dissertation defence, Spring 2009)

Confabulation Theory

• A theory of the mechanism of thought– Cortex/thalamus is divided

into thousands of modules (1,000,000s of neurons).

– Each module contains a lexicon of symbols.

– Symbols are sparse populations (100s) of neurons within a module.

– Symbols are stable states of a cortex-thalamus attractor circuit.

Confabulation theory (1/4)

Key concept 1:

Modules contain symbols, the atoms of our mental universe.

Smell module: Apple, flower, rotten, …Word module: ‘rose’ ‘the’ ‘and’ ‘it’ ‘France’ ‘Joe’ …Abstract planning modules, etc.

Modules are small patches of thalamocortical neurons.Each symbol is a sparse popuation of those neurons.

Confabulation theory (1/4)

Confabulation theory (2/4)

Key concept 2:

All cognitive knowledge is knowledge links between these symbols.

Smell module: Apple, flower, rotten, …Word module: ‘the’ ‘and’ ‘it’ ‘France’ ‘Joe’ ‘apple’ …

Only symbols that are meaningfully co-occurring may become linked.

Confabulation theory (3/4)

Confabulation theory (3/4)

Key concept 3:

A confabulation operation is the universal computational mechanism.

Given evidence a, b, c pick answer x such that:

x = argmaxx’ p(a, b, c | x’)

We say x has maximum cogency.

Confabulation theory (3/4) Fundamental Theorem of Cognition:[1]

p()4 = p()/p() ∙p()/p()∙p()/p() ∙p()/p() ∙p()p()p(g|)p()

If the first four terms remain nearly constant w.r.t , maximizing the fifth term maximizes cogency (the conditional joint).

Confabulation theory (3/4)

Confabulation theory (4/4)

Key concept 4:

Each confabulation operation launches a control signal to other modules.

Control mechanism of inference – studied by others in the lab.

(not here)

Similarities to other AI / ML

Bayesian networks – a special case A “confabulation network” is similar to a Bayesian Net with:

Symbolic variables (discrete & finite & exclusive state) with equal priors.

Naïve-Bayes assumption for CP tables. Can use similar learning algorithms (counting for CPs)

Hinton’s (unrestricted) Bolzman Machines – generalized: Do not require complete connectivity (many) more than two states. Can use stochastic (Monte Carlo) ‘execution’

Outline Confabulation theory

Summary Comparisons to other AI techniques

Human Visual System

A Visual Hierarchy Learning Inference

Texture modeling

Future Work (i.e. my thesis)

Human Visual System

1) Retina – “pixels”2) Lateral Geniculate Nucleus (LGN)

“center-surround” representation

3) Primary(…) Visual cortex (V1 …)• Simple cells:

• Hubel Weisel (1959)• Modeled by Dennis Gabor features[]

• Complex cells• more complicated (end-stops, bars, ???)

Take inspiration for our first and second-level features

Outline Confabulation theory

Summary Comparisons to other AI techniques

Human Visual System

Building A Visual Hierarchy Learning Inference

Texture modeling

Future Work (i.e. my thesis)

Confabulation & vision

Features (symbols) develop in a layer of the hierarchy as commonly seen inputs from their inputs.

Knowledge links are simple conditional probabilities: p(|) where and are symbols in connected modules) All knowledge can therefore be learned by simple co-

occurrence counting. p(|) = C(,) / C()

Building a vision hierarchy

• Can no longer use SSE to evaluate model

• Instead, make use of generative model:– Always be able to generate a plausible image.

Data set• 4,300 1.5 Mpix natural images (BW)

Vision Hierarch – level “0”

We know the first transformation from neuroscience research: simple cells approximate Gabor filters. 5 scales, 16 orientations (odd + even)

Vision Hierarch – level “0”

• Does the full convolution preserve information in images? (inverted by LS)

• Very closely.

Vision Hierarchy – level 1

• We now have a simple-cell like representation.• How to create a symbolic representation?

• Apply principle: Collect common sets of inputs from simple cells: similar to a Vector Quantizer.

• Keep the 5-scales separate – (quantize 16-dimensions, not 80)

Vision Hierarchy – level 1

• To create actual symbols, we use a vector quantizer– Trade-offs (threshold of quantizer) :

Number of symbols Preservation of information

Probability accuracy

• Solution Use angular distance metric (dot-product)– Keep only symbols that occurred in training set more than 200

times, to get accurate p().

– After training, ~95% of samples should be within threshold of at least one symbol.

– Pick a threshold so images can be plausibly generated.

Vision Hierarchy – level 1

Oops!

Ignoring wavelet

magnitude makes all

“texture features”

equally prominent.

Vision Hierarchy – level 1• Solution, use binning (into 5 magnitudes), then

apply vector quantizers).

Vision Hierarchy – level 1• ~10,000 symbols

are learned for each of the 5 scales.

• Complex features develop.

Vision Hierarchy – level 1

• Now image is re-represented as 5 “planes” of symbols:

Outline Confabulation theory

Summary Comparisons to other AI techniques

Human Visual System

Building A Visual Hierarchy Learning Inference

Texture modeling

Future Work (i.e. my thesis)

Texture modeling - Learning

We can now represent an image as five superimposed grids of symbols. Transform data set Learn which symbols are typically next to which. (knowledge links)

Knowledge links:

• Learn which symbols may be next to which symbols (conditional probabilities)

• Learn which symbols may be over/under which symbols.

• Go out to ‘radius’ 5.

Texture modeling – Inference 1

What if a portion of our image symbol representation is damaged? Blind spot CCD defect brain lesion

We can use confabulation (generation) to infer a plausible replacement.

Texture modeling – Inference 1

• Fill in missing region by confabulating from lateral & different scale neighbors (rad 5).

Texture modeling

Texture modeling

Texture modeling

Texture modeling

Conclusions

This visual hierarchy does an excellent job at capturing an image up to a certain order of complexity.

Given this visual hierarchy and its learned knowledge links, missing regions could plausibly filled in. This could be a reasonable explanation for what animals do.

Texture modeling – Inference 2

• Super-resolution:– If we have a low

resolution image, can we confabulate (generate) a high-resolution version?

– “Space out” the symbols, and confabulate values for the new neighbors

Texture modeling

Texture modeling

Texture modeling

Super-resolution: conclusions

Having learned the statistics of natural images, the generative properties of this hierarchy can confabulate (generate) plausible high-resolution versions of its input.

Outline Confabulation theory

Summary Comparisons to other AI techniques

Human Visual System

Building A Visual Hierarchy Learning Inference

Texture modeling

Future Work (Dissertation)

The next level…

Level 2 symbol hierarchy

• Collect commonly recurring regions of level 1 symbols.

• Symbols at Level 2 will fit together like puzzle pieces.

Thank you!