Erro

r-driv

enlearn

ingan

dXCAL

•Self-o

rgan

izingHeb

bian

learningarises

from

biologically

-motiv

ated

XCALrule,ad

justin

gweig

htsas

afunctio

nofsynap

ticactiv

ityxi y

j

over

thesh

ort-term

,relativ

eto

long-term

valu

es(floatin

gthresh

old).

•TheXCALrule

caneasily

bead

apted

tosu

pporterro

r-driv

enlearn

ing,

byusin

gthe’m

edium-term

’synap

ticactiv

ityas

thecompariso

n.

•Med

ium-term

activity

:av

erageofboth

recentoutco

mean

dearlier

expectatio

n-justav

erages

synap

ticactiv

ityover

alonger

horizo

n.

Short-term

activity

reflects

outco

mepreferen

tially.

•So,th

edifferen

cebetw

eenthem

will

largely

reflect

differen

cesbetw

een

expectatio

nan

doutco

me-thedelta.

med

ium

termactiv

ityas

expectatio

nin

XCAL

∆w

≈xs y

s−

xmym

Thisrelatio

nsh

ipistru

eformost

oftheXCALfunctio

n(th

elin

earpart).

Then

goes

back

tozero

forlow

xs y

s–nolearn

ingfornoactiv

ity(C

a++).

Thisiswhyitiscalled

Contrastiv

eAttracto

rLearn

ing

Other

mech

anism

sforErro

r-driv

enLearn

ing

•Neu

romodulato

rysig

nals:

Dopam

ine,A

cetylch

olin

e,etc.

•“P

hasic”

signals

elicitedbybrain

system

scomputin

g’ex

pected

reward

’

anddev

iationsfro

mthisexpectatio

n

Other

mech

anism

sforErro

r-driv

enLearn

ing

•Neu

romodulato

rysig

nals:

Dopam

ine,A

cetylch

olin

e,etc.

•“P

hasic”

signals

elicitedbybrain

system

scomputin

g’ex

pected

reward

’

anddev

iationsfro

mthisexpectatio

n

•Resu

ltingsig

nals,w

hen

combined

with

target

inform

ation( w

hat

should

hav

ebeen

expected

)in

subseq

uen

tstate,can

enhan

ceco

ntrast

betw

eentw

osu

cceedingattracto

rstates

•Lots

ofev

iden

cethat

LTP,L

TD

under

neu

romodulato

rycontro

l

Other

mech

anism

sforErro

r-driv

enLearn

ing

•Neu

romodulato

rysig

nals:

Dopam

ine,A

cetylch

olin

e,etc.

•“P

hasic”

signals

elicitedbybrain

system

scomputin

g’ex

pected

reward

’

anddev

iationsfro

mthisexpectatio

n

•Resu

ltingsig

nals,w

hen

combined

with

target

inform

ation( w

hat

should

hav

ebeen

expected

)in

subseq

uen

tstate,can

enhan

ceco

ntrast

betw

eentw

osu

cceedingattracto

rstates

•Lots

ofev

iden

cethat

LTP,L

TD

under

neu

romodulato

rycontro

l

•Heb

bian

learningalw

aysoccu

rslocally,in

every

synap

se(m

odel

learning,statistics)

•Brain

regionsinnerv

atedbyDA,A

Chhav

een

han

cedweig

htch

anges

durin

gerro

rs,lead

ingto

contrastiv

eattracto

rlearn

ing(ap

proxim

ated

bydelta

rule)

Combined

Model

&Task

Learn

ing

1.Prosan

dCons:Use

Both.

2.Inhibitio

nisalso

anIm

portan

tBias.

Functio

nal:

Prosan

dCons

... ... ... ... ...E

rror

error−driven

is based onrem

ote errors

Hebbian

is local

Pro

Con

Heb

bian

autonomous,

myopic,

(local)

reliable

greed

yErro

r-driv

entask

-driv

en,

co-dep

enden

t,(rem

ote)

cooperativ

elazy

Erro

r-driv

en=Left-w

ing,H

ebbian

=Right-w

ing(?!)

CombiningErro

r-driv

en+Heb

bian

insin

gle

XCALrule

Get

ben

efits

ofboth:

∆wij

≈∆

hebb+

∆err

CombiningErro

r-driv

en+Heb

bian

insin

gle

XCALrule

Get

ben

efits

ofboth:

∆wij

≈∆

hebb+

∆err

Θp=

λyl+

(1−

λ)xmym

∆wij

=λl fxcal (xs y

s ,yl )

+λmfxcal (xs y

s ,xmym)

•λ

(“thr_l_mix”in

simulato

r)isaparam

eteraffectin

gdeg

reeto

which

XCAL

thresh

old

isdeterm

ined

byylorym.

•can

differ

betw

eenbrain

system

s,orev

enbemodulated

dynam

ically

(e.g.byneu

romodulato

rs)

Heb

bian

bias

help

sso

that

weig

htsare

constrain

edto

smaller

setof

solutio

ns(otherw

isetoointerd

epen

den

tin

err-driv

en)

Inhibito

ryCompetitio

nas

aBias

Inhibitio

n:

•Cau

sessp

arse,distrib

uted

represen

tations

(man

yaltern

atives,

only

afew

relevan

tat

anytim

e).

•Competitio

nan

dsp

ecialization:su

rvival

offittest.

•Self-o

rgan

izinglearn

ing.

(Often

more

importan

tthan

Heb

bian

bias)

TheWhole

Ench

ilada

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

nust

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

nust

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

nust

eachtim

eyouwalk

into

class,eachsocial

interactio

n,each

senten

ceyou

hear,etc.

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

nust

eachtim

eyouwalk

into

class,eachsocial

interactio

n,each

senten

ceyou

hear,etc.

We’re

constan

tlyfaced

with

new

situatio

ns,an

dgen

eralizereaso

nab

lywell

tothem

.

Gen

eralization

How

well

dowedeal

with

thingswe’v

enev

erseen

befo

re?

nust

eachtim

eyouwalk

into

class,eachsocial

interactio

n,each

senten

ceyou

hear,etc.

We’re

constan

tlyfaced

with

new

situatio

ns,an

dgen

eralizereaso

nab

lywell

tothem

.

How

dowedoit?

Heb

b:

•Sometim

esfails

tolearn

thetrain

ingset

•Rep

resents

mean

ingful“th

ings”

intheworld

(correlatio

ns)

•Showsgoodgen

eralization

Erro

r(G

eneR

ec/X

CA

Lw

ithxm

ym):

•Alw

ayslearn

sthetrain

ingset

•Rep

resentatio

nsare

“mush

y”

•Can

show

poorgen

eralization

Erro

r+

Heb

b:

•Learn

sthetrain

ingset

(more

quick

lythan

erroralo

ne)

•Rep

resents

mean

ingfulfeatu

res

•Showsgoodgen

eralization!

Deep

Netw

orks

Need

man

yhidden

layers

toach

ieveman

ystag

esoftran

sform

ations

(dram

aticallyre-rep

resentin

gtheproblem

).(cf.

recentsu

rgein

interest

in“d

eeplearn

ing”

inmach

inelearn

ingforsp

eechreco

gnitio

n,G

oogle,etc)

Butthen

theerro

rsig

nals

arevery

remote

&weak

.

Need

toad

dconstrain

tsan

dself-o

rgan

izinglearn

ing:

Deep

Netw

orks

Need

man

yhidden

layers

toach

ieveman

ystag

esoftran

sform

ations

(dram

aticallyre-rep

resentin

gtheproblem

).(cf.

recentsu

rgein

interest

in“d

eeplearn

ing”

inmach

inelearn

ingforsp

eechreco

gnitio

n,G

oogle,etc)

Butthen

theerro

rsig

nals

arevery

remote

&weak

.

Need

toad

dconstrain

tsan

dself-o

rgan

izinglearn

ing:

•Heb

bgives

eachlay

erlocal

guidan

ceonrep

resentatio

ns

•Inhib

competitio

nrestricts

flexibility

(only

certainstates

arevalid

)

•Combined

heb

b+err

→few

erdeg

reesoffreed

om

toad

apt

Exam

ple:

Fam

ilyTrees

(Hinton,1986)

Christo=

Penny

Andy=

Christi

Marge=

Art

Vicky=

James

Jenn=C

huck

Colin

Charlot

Rob=

Maria

Pierro=

Francy

Gina=

Em

ilioLucia=

Marco

Angela=

Tom

aso

Alf

Sophia

Agent

Agent_C

ode

Relation

Relation_C

odePatient

Patient_C

ode

Hidden

Exam

ple:

Fam

ilyTrees

(Hinton,1986)

Christo=

Penny

Andy=

Christi

Marge=

Art

Vicky=

James

Jenn=C

huck

Colin

Charlot

Rob=

Maria

Pierro=

Francy

Gina=

Em

ilioLucia=

Marco

Angela=

Tom

aso

Alf

Sophia

Agent

Agent_C

ode

Relation

Relation_C

odePatient

Patient_C

ode

Hidden

24peo

ple,

12relatio

nsh

ips(brother,

mother,g

randdau

ghter,etc)

Exam

ple:

Fam

ilyTrees

(Hinton,1986)

Christo=

Penny

Andy=

Christi

Marge=

Art

Vicky=

James

Jenn=C

huck

Colin

Charlot

Rob=

Maria

Pierro=

Francy

Gina=

Em

ilioLucia=

Marco

Angela=

Tom

aso

Alf

Sophia

Agent

Agent_C

ode

Relation

Relation_C

odePatient

Patient_C

ode

Hidden

24peo

ple,

12relatio

nsh

ips(brother,

mother,g

randdau

ghter,etc)

WhoisAlf’s

gran

dmother?

WhoisLucia’s

dau

ghter?

[family

trees.proj]