adaptive cooperative systems chapter 8 synaptic plasticity 8.11 ~ 8.13 summary by byoung-hee kim...

Adaptive Cooperative SystemsAdaptive Cooperative Systems Chapter 8 Chapter 8

Synaptic PlasticitySynaptic Plasticity

8.11 ~ 8.13

Summary by Byoung-Hee KimBiointelligence Lab

School of Computer Sci. & Eng.

Seoul National University

(C) 2009 SNU CSE Biointelligence Lab

ContentsContents

8.11 Principal component neurons Introductory remarks Principal components and constrained optimization Hebbian learning and synaptic constraints Oja’s solution / Linsker’s model

8.12 Synaptic and phenomenological spin models Phenomenological spin models Synaptic models in the common input approximation

8.13 Objective function formulation of BCM theory Projection pursuit Objective function formulation of BCM theory

2

Goals and ContentsGoals and Contents

Goal: the information-processing functions of model neurons in the visual system

Contents Principal component neurons Special class of synaptic modification models Relation to phenomenological spin models Objective function formulation of BCM theory

(C) 2009 SNU CSE Biointelligence Lab 3

Introductory RemarksIntroductory Remarks

Images are highly organized spatial structures – some common statistical properties

Development of the visual system is influenced by the statistical properties of the images

knowledge of the statistical properties of natural scenes ~ understanding the behavior of cells in the visual system


Scale Invariance in Natural ImagesScale Invariance in Natural Images

Studies of image statistics reveal non-preferrence of angular scale Decimation procedure with the grey-valued pixels of the image

assuming the role of the spins p.d.f. of image constrasts and image gradients are unchanged

(Field 1987), (Ruderman and Bialek 1994), (Ruderman 1994)

Representing the scale invariance through the covariance matrix Gives a constraint on the form of the covariance matrix Starting point for the PCA (Hancock, et al. 1992), (Liu and Shouval 1995), (Liu and Shouval

1996)


Principal ComponentsPrincipal Components We are rotating the

coordinate system in order to find projections with desirable statistical properties

Projections: maximally preserve information content while compressing the data into a few leading components


Variance of the data projected onto the axis is maximal

Principal Components and Constrained Principal Components and Constrained OptimizationOptimization


n-component random vector

correlation matrix

If <X>=0, then the covariance matrix

Introducing a fixed vector that satisfies the normalization condition

use this to help us find interesting projections

Variance after operation:

• Optimization problem: find the vector a that satisfies the normalization condition, and maximizes the variance

The variance is equal to the eigenvalue

The maximum variance is given by the largest root

Hebbian Learning and Synaptic ConstraintsHebbian Learning and Synaptic Constraints


T Ti i

i

c m d m d d m

The simplest form Hebb’s rule for synaptic modification

[Problem]Unstable. The synaptic weights would undergo unbounded growth

c: output activitym: synaptic weight vectord: input activity vector

On reaching a fixed point

m is an eigenvector of the input correlation matrix with eigenvalue equal to zero

Solutions for the Unbounded Growth ProblemSolutions for the Unbounded Growth Problem

Oja’s solution

Linsker’s model


On reaching a fixed point

Results in a synaptic vector m for whichthe projection of the input activity has a maximum variance

Tm d

The synaptic system may be characterized as performing a principal component analysis of the input data

constraint on the total synaptic strength

Clipping- The sum of the synaptic weights are kept constant- each synaptic weight lies within a set range

Q kE E E

EQ: the variance in the input activity Ek: constraint

Properties of the Linsker’s ModelProperties of the Linsker’s Model

Stability corresponds to a global near minimum of the energy function Equivalent to the maximum in the input variance subject to the

constraint

Dynamics of the model system In different regimes for the parameters k1 and k2, different receptive field

structures dominate As k1 and k2 are varied, particular eigenvectors other than the principal one

gain in relative importance


Q kE E E

Synaptic and Phenomenological Spin Synaptic and Phenomenological Spin ModelsModels

Theory on synaptic modification Model to explain the emergence of

these highly ordered repeating structures


Phenomena Cells in the primate visual cortex self-organize onto ocular dominance

columns and iso-orientation patches The patterns observed experimentally are highly ordered

Phenomenological Spin ModelsPhenomenological Spin Models

2D Ising lattice of eye-specificity encoding spints (Cowan and Friedman 1991) Coupling strengths

If we take with , this type of coupling generates a short-range attraction plus a long-range repulsion between terminals from the same eye

Hamiltonian for iso-orientation


2 2

| | | |exp expij

i j i jJ a a

2 2/ /a a

| || | cos( )ij i j i ji j

E J s s

Synaptic Models in the Common Input Synaptic Models in the Common Input ApproximationApproximation Consider an LGN-cortico-cortico network with modifiable

geniculocortico synapses and fixed cortico-cortico-connections

Design of an energy function s.t. the fixed point of the network correspond to the minima of the energy function

The common input model by Shouval and Cooper hamiltonian in this model:


T Ti i

i

c m d m d d m

general form

correlational hamiltonian

Information-processing Activities by Information-processing Activities by Common Input NeuronsCommon Input Neurons For exclusive excitatory connections

symmetry breaking does not occur all receptive fields have the same orientation selectivity

Inhibition affects both the organization and structure of the receptive fields If there is sufficient inhibition, the network will develop

orientation selective receptive fields

The cortical cells self-organize into iso-orientation patches with pinwheel singularities


Objective Function Formulation of BCM Objective Function Formulation of BCM Theory - IntroTheory - Intro Distinguishment between information preservation

(variance maximization) and classification (multimodality)


Projection PursuitProjection Pursuit

Projection pursuit a method for finding the most interesting low-dimensional features

of high-dimensional data sets The objective is to find orthogonal projections that reveal

interesting structure in the data PCA is a particular case with the proportion of total variance as

the index of interestingness Why is it needed? High-dimensional spaces are inherently sparse,

or “curse of dimensionality”

For classification purpose Interesting projection is one that departs from normalcy


Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (1/3)Theory (1/3) In the objective (energy) function formulation of BCM

theory, a feature is associated with each projection direction

A one-dimensional projection may be interpreted as a single feature extraction

Goal: to find an objective (loss) function whose minimization produces a one-dimensional projection that is far from normal


Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (2/3)Theory (2/3)


Redefining the threshold function

Synaptic modification functions

How? Introduce a loss function

With some assumptions

Objective Function Formulation of BCM Objective Function Formulation of BCM Theory (3/3)Theory (3/3)


The risk, or expected value of the loss,which is continously differentiable

We are able to minimize the risk by means of gradient descent w.r.t. mi

Slightly modified, deterministic version of the stochastic BCM modification equation

A BCM neuron is extracting third-order statistical correlates of the dataThis would be a natural extension of principal component processing in the retina

2 2

( )( ( ), ( )) ( ) (8.19)

( ) ( ( )) ( ( ) ( )) (8.22)

d tc t t t

dt

t c t t t

md

m d

M

M

Take-Home MessageTake-Home Message

(Tomasso Poggio, NIPS 2007 tutorial)


adaptive cooperative systems chapter 8 synaptic plasticity 8.11 ~ 8.13 summary by byoung-hee kim...

Documents

synaptic constraintsc

synaptic plasticity8

synaptic weights

synaptic weight vectord

image constrasts

image gradients

common input approximation8

covariance matrixgives