ist 4 information and logic - brain.caltech.edubrain.caltech.edu/ist4/lectures/lect 1715 p.pdf ·...

IST 4Information and Logic

mon tue wed thr fri30 M1 1= todayT

6 M1

13 1 2M2x= hw#x out

oh oh

oh oh

= todayT

M

20 227 M2

x= hw#x due

idt

oh oh

h h Students’ MQ27 M2

4 311 3 4Mx= MQx out

midtermsoh

oh oh

oh = office hours oh Students MQpresentations

11 3 418 4 525

Mx= MQx out

Mx= MQx dueoh oh

ohoh

25

1 5oh

oh

oh

oh

T

MQsMQs1. Everyone has a gift! (Tuesday)

2. Memory (Thursday)

Tuesday 6/2 2:30pm –

1. Christopher Haack: The gift of resilience

Tuesday, 6/2, 2:30pm

p g

2. Joon Lee: Settling is not an option

3 Spencer Strumwasser: The gift of dyslexia 3. Spencer Strumwasser: The gift of dyslexia

4. Richard Zhu: The gift of memory

5 A h i H i Th ift f i l iti5. Ashwin Hari: The gift of musical composition

6. Jessica Nassimi: Evolution—a gift in disguise

7. Serena Delgadillo: The gift of self-expression

8. Megan Keehan: Gift of motherliness

9. Zane Murphy: Grandmother and the piano

Thursday 6/4 2:30pm –Thursday, 6/4, 2:30pm

1. Connor Lee: Memory is a fickle thing – blessing or curse

2. Pallavi Aggarwal: The wonders of human memory

3. Peter Kundzicz and Anshul Ramachandran: Muscle memories

4. Siva Gangavarapu: A cultural retrospection. g p p

5. Philip Liu: The light of other days

6 Jason Simon: Math and Broadway6. Jason Simon: Math and Broadway

7. Yujie Xu: Memory v.s. ESL

8 C l Zh Wh 8. Celia Zhang: When memory sours

Last LectureGates and circuits AON: AND, OR, Not

LT: Linear ThresholdLT: Linear Threshold

>>

abc

ab >>

>>

abc

ab >

>

>

>

bc

abc

abc

>>>>

>>

>>

bc

abc

abc

Last LectureAON: AND, OR, Not LT: Linear Threshold

General construction for symmetric functions

AON 54

AONLT-l

*

*

2LT-nl *

Exponential gap in sizeWhat are the symmetric functions that can be computed functions that can be computed by a single LT gate? * = it is optimal

Linear Threshold and SYM

LT: Linear Threshold

Symmetric Functions and LT Circuits

Q: How is SYM related to LT1 ??

Q: Which class has more functions?

Definitions:

Q

Definitions:

(1) SYM = the class of Boolean symmetric functions

(2) LT1 = the class of Boolean functions that can be (2) LT1 the class of Boolean functions that can be realized by a single LT gate.

AND, OR, XOR and MAJ are symmetric functions

Q Whi h t i f ti i LT ?

|X| AND OR XOR MAJ

Q: Which symmetric functions are in LT1?

0 0 0 0 0

1 0 1 1 01 0 1 1 0

2 0 1 0 1

3 1 1 1 1

LT1 LT1 LT1not LT1

LT1 = the class of Boolean functions that can be realized by a single LT gate.

LT1 LT1 LT1not LT1

realized by a single LT gate.

Definition: A t i B l f ti i i TH if it h t t A symmetric Boolean function is in TH if it has at most a single transition in the symmetric function table

= a transition

|X| AND OR XOR MAJ

0 0 0 0 0

1 0 1 1 0

2 0 1 0 1

3 1 1 1 1

Not in THIn TH

The Class TH is in LT1

|X| TH0 TH1 TH2 TH3 TH0 TH1 TH2 TH3

0 1 0 0 0 0 1 1 10 1 0 0 0 0 1 1 11 1 1 0 0 0 0 1 1

1 1 1 0 0 0 0 12 1 1 1 0 0 0 0 1

3 1 1 1 1 0 0 0 0

Q: How is TH related to SYM and LT1 ??

We know that:

Q

We know that:

We Proved that:

SYM LT1TH

TH is exactly in the intersection of SYM and LT1

Theorem:

Q: What are the 4 functions?

Proof: Not today... you might want to try and prove it...

Q: What are the 4 functions?

SYM LT1TH

???

LT1 Function that is not SymmetricLT1 Function that is not Symmetric

0 00 1

-1-2

00

11

1 01 1

0-1

10

-1-1

lLinear Threshold Circuits

for symmetric functionsfor symmetric functions

5AON 54

AONLT-l

L l 2LT-nl

General construction for symmetric functions

|X| XOR Q: compute XOR with TH gates?

0 01 1

with TH gates?

2 0

1|X| TH1 TH2 TH1+TH2-1

0 0 1 01 1 1 12 1 0 02 1 0 0

LT Depth-2 Circuits

TH1

+-1

TH2 |X| TH1 TH2 TH1+TH2-1 |X| TH1 TH2

0 0 1 01 1 1 11 1 1 12 1 0 0

Generalization

|X| f( )

z

|X| f(x)0 0

1 1

2 1

3 0

4 04 0

Generalization

|X| f( )

z

|X| f(x)0 0

1 1

2 1

3 0

4 04 0

Generalization

|X| f( ) TH1

z

|X| f(x) TH1

0 0 0

1 1 1

2 1 1

3 0 1

4 0 14 0 1

Generalization

|X| f( ) TH1 TH3

z

|X| f(x) TH1 TH3

0 0 0 1

1 1 1 1

2 1 1 1

3 0 1 0

4 0 1 04 0 1 0

Generalization

|X| f( ) TH1 TH3 Σ 1

z

|X| f(x) TH1 TH3 Σ -10 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 04 0 1 0 0

|X| f( ) TH1 TH3 Σ 1|X| f(x) TH1 TH3 Σ -10 0 0 1 0

1 1 1 1 1

2 1 1 1 1

3 0 1 0 0

4 0 1 0 04 0 1 0 0

Generalization to EQz EQ

00

00 0

1

1

1

n

1

00

Generalization to EQz EQ

00 1

1

1

12

0

1

0 1

Generalization to SYMz M

+-1

Q:h h l fWhat is the generalization to arbitrary symmetric functions?

Generalization to SYMz M

Q:Q:What is the generalization to arbitrary symmetric functions?

A:A:Consider the symmetric function table, it is a sum of non-overlapping 1-intervals

0

0

1

1 Sum of two TH functions

Back to XORX

n TH gates for XOR of n variables

0 0

1

2

1

0

3

4

1

0

5 1

LT l Circuit Design Algorithm for SYMLT-l Circuit Design Algorithm for SYM

f(X)

0 1

f(X)

0

1

2

1

1

02

3

0

1Subtract 1 for every

4

5

1

0

Subtract 1 for everyisolated 1-block

67

11

The Layered Construction for SYM -Some History

Saburo Muroga1925- 2009

1959

Was born in Japan Majority Decision

PhD in 1958 from Tokyo U, Japan

1960-1964: Researcher at IBM Research, NY,

1964-2002: professor at the University of Illinois, Urbana-Champaign

Saburo Muroga1925- 2009

HW#5 problem 2a

neural circuits and logicg

some more historysome more history...

Being Homeless and Interdisciplinary Research

W M C ll hWarren McCulloch1899 - 1969

Walter Pitts1923 - 1969

Neurophysiologist, MD Logician, Autodidactg ,

Warren McCulloch arrived in early 1942 to the University of Chicago, invited Pitts, who was homeless, to live with his family

h ll h d P ll b d In the evenings McCulloch and Pitts collaborated. Pitts was familiar with the work of Leibniz on computing.They considered the question of whether the nervous system is a kind of universal computing device as described by Leibnizuniversal computing device as described by Leibniz

This led to their 1943 seminal neural networks paper:A Logical Calculus of Ideas Immanent in Nervous Activity

Warren McCulloch W l Pi

ImpactWarren McCulloch

1899 - 1969 Walter Pitts1923 - 1969

Neurophysiologist, MD Logician, Autodidactg ,

This led to their 1943 seminal neural networks paper:p pA Logical Calculus of Ideas Immanent in Nervous Activity

Neural networks d Logic Ti MNeural networks and Logic Time Memory

Threshold Logic and Learning

State Machines

neural circuits and memorym m y

computing with dynamicscomputing with dynamics

Linear ThresholdSome AdjustmentsSome Adjustments

Linear Threshold (LT) gateLinear Threshold (LT) gate

-tthreshold

t

-t

-1

t

1

-1

AND Function with {0,1}

-21

-21

0 00 1

1 0

-2-11

0001 0

1 1-10

01

AND Function with {-1,1}

Th AND f ti f t i bl ith { 1 1} ???The AND function of two variables with {-1, 1}: ???

-1-1-11

-3-1--

1-111

11

1-11-+

AND Function with {-1,1}

Th AND f ti f t i bl ith { 1 1} ???The AND function of two variables with {-1, 1}: ???

-1-1-11

-3-1

1-111

11

1-11

Linear Threshold with MemoryEl h t b l f i d i A i ltElephants are symbols of wisdom in Asian culturesand are famed for their exceptional memory

A memory nosey

Remembers the last f(X)

Feedback NetworksE lExample

-1 -1 00

weights thresholds

Th t t f th t k th t th t d The state of the network: the vector that corresponds to the states (noses…) of the gates

Feedback NetworksExampleExample

1 2Label the gates

-1 -1 00 1 1

1 2Label the gates

-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 1 1

1 2

1-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 1

1 2

1-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 1

1 2-1

-1 -1 00

11 i bl -11 is a stable state

Feedback NetworksExampleExample

1 2

-1 -1 00 1 1

1 2

-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 1 1

1 21

-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 1 -1

1 21

-1 -1 00

Feedback NetworksExampleExample

1 2

-1 -1 00 1 -1

1 2-1

-1 -1 00

1 1 i bl 1-1 is a stable state

Feedback NetworksExampleExample

1 2

-1 -1 00 1 1

1 2

-1 -1 00

State transition diagram (state space)1

The node

11 -11

2

1state

The node that computes -1-11-1

2Q:Is -1-1 a stable state?

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 -1

1 2-1

-1 -1 00

Answer: NoAnswer: No

Q:Is -1-1 a stable state?

Feedback NetworksExampleExample

1 2

-1 -1 00 1 -1

1 2-1

-1 -1 00

11 111

11 -11

2-1-11-1

1

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 -1

1 2-1

-1 -1 00

11 111

11 -11

2-1-11-1

1

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 1

1 2-1

-1 -1 00

11 111

11 -11

2 2-1-11-1

1

Feedback NetworksExampleExample

1 2

-1 -1 00 -1 1

1 2

-1 -1 00

11 111stable states

11 -11

2 2-1-11-1

1

neural circuits and memorym m y

associative memoryassociative memory

Feedback NetworksComputing with DynamicsComputing with Dynamics

1stable states11 -11

2

1

2

stable states

-1-11-1

2

1

2

Input:initial state

Output:stable state

FeedbackNetwork

n t al state

11 -1111 11

Feedback NetworksComputing with DynamicsComputing with Dynamics

1stable states11 -11

2

1

2

stable states

-1-11-1

2

1

2

Input:initial state

Output:stable state

FeedbackNetwork

n t al state

11 1-111 1 1

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

Feedback NetworksComputing with DynamicsComputing with Dynamics

Input: Output:Associative Memory“The Leibniz-Boole Machine”p

initial statep

stable stateThe Leibniz Boole Machine

FeedbackNetwork

????

Who is this person?????

John Hopfield

f ld d l ( l h )Feedback Networks

Hopfield Model (Caltech 1982)

Feedback NetworksJohn Hopfield

f ld d l ( l h )

1 2

Hopfield Model (Caltech 1982)

-1 -1 00 1 1

-11 20 0

1 1

1

i = node i 0 = threshold ti

-1

1 = state vi -1 = weight of edge (i,j)

The matrix descriptionm p

Feedback Networks/The Vector/Matrix Description

An n node feedback network can be specified by:An n node feedback network can be specified by:• W an nxn matrix of weights• T an n vector of thresholds f• V an n vector of states

1

4

2

34

5

The Matrix DescriptionE lExample

An n node feedback network can be specified by:An n node feedback network can be specified by:• W an nxn matrix of weights• T an n vector of thresholds f• V an n vector of states

1 20 0

-121

-1

11 1

-1

-1

The Matrix DescriptionComputationComputation

C t ti i N (W T)Computation in N= (W,T)1

4

2

34

5by column

Order of computationf mp

serial and parallel

Modes of OperationModes of Operation

Q: when do the nodes compute?Q: when do the nodes compute?

Serial mode: one node at a time (arbitrary order)

-1

( y )

21-1

Modes of OperationModes of Operation

Q: when do the nodes compute?Q: when do the nodes compute?

Serial mode: one node at a time (arbitrary order)

-1

( y )

21-1

Modes of OperationModes of Operation

Q: when do the nodes compute?Q: when do the nodes compute?

Serial mode: one node at a time (arbitrary order)

-1

( y )

21-1

Fully-Parallel mode: all nodes at the same time-1

21-1

Three examplesmp

Example 1Serial Mode – Symmetric Weight MatrixSerial Mode – Symmetric Weight Matrix

21-1

-11

The state space:

1stable states

The state space:

11 -11

2

1

2-1-11-1

1

Example 2Fully Parallel (FP) Mode Symmetric Weight MatrixFully-Parallel (FP) Mode – Symmetric Weight Matrix

Q: how does the state space look?

21-1

Q p

21-1

start with 11start with 11

It’s acycle!cycle!

Example 2Fully Parallel (FP) Mode Symmetric Weight MatrixFully-Parallel (FP) Mode – Symmetric Weight Matrix

21-1

21-1

h The state space:

stable states11 -11

l f l th 2-1-11-1

cycle of length 2

Example 2Fully Parallel Mode Symmetric Weight Matrix

3AntisymmetricFully-Parallel Mode – Symmetric Weight MatrixAntisymmetric

W T = −W21

-1

1

11W W

-1

Q: how does the state space look?p

Example 3Fully Parallel Mode Antisymmetric Weight Matrix

1

Fully-Parallel Mode – Antisymmetric Weight Matrix

21-1

Q: how does the state space look?

cycle of length 4

Example 3Fully Parallel Mode Antisymmetric Weight Matrix

1

Fully-Parallel Mode – Antisymmetric Weight Matrix

21-1

The state space:

11 -11

-1-11-1

cycle of length 4

The Three Cases

11 -1111 -11 11 -11

-1-11-1-1-11-1 -1-11-1

1 2 3

Wmode symmetric antisymmetricCycle lengths

serial 1 ?1Example #

fully-parallel 1,2 42 3

The Three Cases

Wmode symmetric antisymmetricCycle lengths

mode ymm ymm

serial 1 ?1

Example #fully-parallel 1,2 4

2 3

Example #

1 Hopfield 1982

2 G l 19852 Goles 1985

3 Goles 1986

Proof Ideas

C l l th W symmetric antisymmetricCycle lengths Wmode symmetric antisymmetric

serial 1 ?1Example #

fully-parallel 1,2 4

1

2 3The proofs of these three results use the conceptof an energy function

For the serial mode:

Sh th tShow that:

Namely, stable states are local max of the energy E

Questions on Convergence

Posted on the class web site

Cycle lengths Wd symmetric antisymmetric

Posted on the class web site

Cycle lengths Wmode symmetric antisymmetric

serial 1 ?1Example #

fully-parallel 1,2 42 3

1 Hopfield 1982

2 Goles 1985 Q1: Are the three cases “distinct”?2 Goles 1985

3 Goles 1986

Q

Q2: Elementary proof? (wo/energy)