biological inspired system applied to computer vision

MotivationHMAX ModelImprovements

Summary

Biological Inspired Systemsapplied to Computer Vision

Federico Raue Rodriguez([email protected])

IUPR

July 2, 2012

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.

Federico Raue Rodriguez ([email protected]) Biological Inspired Systems applied to Computer Vision


Summary

Contents

1 Motivation

2 HMAX Model

3 ImprovementsSparsityPooling MechanismInput

4 Summary



Summary

(slide from Fundamentals of AI – Prof. De Schreye (KULeuven))



Summary



Summary

Neuroscience may begin to provide new ideas and approachesto machine learning, AI and computer vision (Tomaso Poggio)

Interesting properties for visual recognition

a Invarianceb Specificity

Visual processing in cortex is classically modeled as a hierarchy



Summary

(slide from Learning in Hierarchical Architectures – Tomaso Poggio (McGovern

Institute))



Summary

(Perception Strategies in Hierarchical Vision Systems. (Wolf et al))



Summary


Institute))



Summary

Contents

1 Motivation

2 HMAX Model


4 Summary



Summary

Goal: Object categorization based on human visual system

Assumptions:

a Invariance to position and scaleb Feature specificity must be built up through separate

mechanismsc Extending the model of simple and complex cells of Hubel and

Wieseld Hierarchical feedforward architecturee Pooling mechanism



Summary


Institute))



Summary

(Hierarchical models of Object recognition in cortex (Riesenhuber et al.))



Summary

General Description of HMAX model

The standard model consists of four layers of computational unitswhere simple S units, which combine their inputs with Gaussian-liketuning to increase object selectivity, alternate with complex Cunits, which pool their inputs through maximum operation, therebyintroducing gradual invariance to scale and translation



Summary


Institute))



Summary

Simple Cells(S1) is a battery of Gabor filters

G (x , y) = exp

(−X 2 + γ2Y 2

2σ2

)× cos

(2π

λX

)Complex Cells(C1) show some tolerance to shift and size

a Larger receptive fieldsb Shape Invariance: respond to oriented bars or edges anywhere

within their receptive fieldc Scale Invariance: more broadly tuned to spatial frequency than

simple cells



Summary

Pooling operation from S1 to C1

S1 units: 16 scales arranged in 8 bands

For each orientation, it contains two S1 maps, two filter size

C1 responses: these maps are sub-sampled using a grid cell ofsize NΣ × NΣ (8x8)

From each grid cell we obtain one measurement by taking themaximum of all 64 elements

As a last stage we take a max over the two scales, byconsidering for each cell the maximum value from the twomaps



Summary

(Object Recognition with Features Inspired by Visual Cortex (Serre et al.))



Summary

Learning Process

Large pool of K patches of various sizes at random positionsare extracted from a target set of images at the C1 level forall orientations

The patch size is n x n x 4 (The value 4 is due to 4orientations)

The training process ends by setting each of those patches asprototypes or centers of the S2 units, which behave as radialbasis function (RBF) units during recognition



Summary

Visual words in C2

When a new input is presented, each stored S2 unit isconvolved with the new (C1)Σ input image at all scales (thisleads to K x 8 (S2)Σ

i images), where the K factor correspondsto the K patches extracted during learning and the 8 factor,to the 8 scale bands

After taking a final max for each (S2)i map across all scalesand positions, we get the final set of K shift- andscale-invariant C2 units

The size of our final C2 feature vector thus depends only onthe number of patches extracted during learning and not nothe input image size



Summary



Summary

SparsityPooling MechanismInput

Contents

1 Motivation

2 HMAX Model


4 Summary



Summary


1 Extend the model using more biological information

Saliency ModelsNew Pooling mechanismRedefine the input image

2 Reduce the computational cost



Summary


Biological Motivation

Increase sparsity is to use a lateral inhibition model thateliminates weaker responses that disagree with the locallydominant ones

Our attention will be attracted to some locations mostlybecause their saliency, defined by contrasts in color, intensityor orientation

(Treisman) presented a theory about feature integration inhuman brain, which has two stages, the simple pre-attentionprocessing and complex attention processing. Some low levelfeatures will pop up automatically and generate the attentionarea in pre-attention processing



Summary


Computational Motivation

Simplifies structures and reduces computational costs

Feature or variable selection

Enhance the generalization ability of learning machines



Summary


(Multiclass Object Recognition with Sparse, Localized Features (Mutch and

Lowe)



Summary


|Fx(i)|+ |Fy(i)| ≥α

n

n∑k=1

(|Fx(k)|+ |Fy(k)|)

(Enhanced Biologically Inspired Model (Huang et al.))



Summary





Summary


1 Find the maximal response and its neighbors

2 Weak responses are removed due to inhibition effect3 New pooling Mechanism

a sum the energy of all responses remained by using differentweights for S1 units

C =1

NI0

∑xi ,yi∈I0

[wiS2(xi , yi )]

(Human age estimation using bio-inspired features (Guo et al.))b the STD operation is performed on the maximum map using a

cell grid of size Ns x Ns

std =

√√√√ 1

Ns × Ns

Ns×Ns∑i=1

(Fi − F̄

)2




Summary


C =1

NI0

∑xi ,yi∈I0

[wiS2(xi , yi )]



Summary





Summary


Relevant Component Analysis (RCA): finds a linear embeddingtransformation that minimizes the distances between points



Summary


Complement using dorsal stream (where)

Cells respond to colored stimuli more strongly than colorlessone in the Inferior Temporal (IT) and the visual areas V4 ofthe visual cortex

Analogous to the ’center-on surround-off’ center surroundprocessing that occurs in the retina and in the lateralgeniculate nucleus (LGN)

Some region (in the brain) are more active for face imageswhen compared to images of other objects



Summary


Hierarchical: gradually increase both the selectivity of neuronsalong with their invariance to 2D transformationsHypothesis: neurons in intermediate visual areas of the dorsalstream such as MT, MST and higher polysensory areas aretuned to spatio-temporal features of intermediate complexity,which pool over afferant input units tuned to differentdirections of motion



Summary


S1 Units

Gray-value video sequence at all positionThree different types of S1

a Space-time gradient-based: Space and time gradients

| ItIx + 1

| | ItIy + 1

|

b Optical flow based S1 units: Optical flow of the input usingLucas & Kanade’s alg.

b(θ, θp) = {1

2[1 + cos(θ − θp)]}q × exp(−|v − vp|)

4 directions and two speedsc Space-time oriented S1 units:

Add a temporal dimension to their receptive fields3rd derivatives fo Gaussians8 space-time filters tuned to 4 directions and 2 speedsSize of receptive fields was 9(pixels)x9(pixels)x9(frames)



Summary


Proposed by Plebe et al

Extract visual attribute for object recognition based oninfant’s brain

Children between 8 and 10 months old, their objectcategorization model is stable and flexible

10-month-old infants ’are sensitive to social cues but cannotrecruit them for word learning’

Early vocabulary is made up of the objects infants mostfrequency see

Connectionist model with backpropagation developed ageneral model based on similarities without taking intoaccount physiological and cognitive constraints



Summary


Implementation based on Laterally InterconnectedSynergetically Self Organizing Map architecture (LISSOM)

Hebbian Law: explains the adaptation of neurons in the brainduring the learning process“. . . , that any two cells or systems of cells that arerepeatedly active at the same time will tend to becomeassociated, so that activity in one facilitates activity in theother.”

Two paths: one for visual and the other for auditory channel



Summary




Summary


Exposure to stimuli

Visual path in the model develops in two stages.1 Random blobs, simulating pre-natal waves of spontaneous

activity, known to be essential in the early development of thevisual system

2 Natural images are used (After eye opening)

Auditory path there are different stages1 Random patches in frequency-time domain, with shorter

duration for HPC and longer for LPC2 7200 most common English words (lengths between 3 and 10

characters)

Last stage: an object is viewed and a word corresponding toits basic category is heard simultaneouly



Summary




Summary

Contents

1 Motivation

2 HMAX Model


4 Summary



Summary

Biological models are suitable features for visual recognition

RobustInvariance

Two Pathways1 Ventral stream (What?)2 Dorsal stream (Where?)

Depending on the task HMAX model changes

Parameters (Aging Detection)Pooling function (Energy model, Standard Deviation)


biological inspired system applied to computer vision

Education

hierarchical vision

maps federico raue rodriguez

model of simple

s1 maps

maximum operation

approachesto machine

visual cortex serre

c1 s1 units