a dynamic analog concurrently-processed adaptive chip malcolm stagg grade 11

A Dynamic Analog A Dynamic Analog Concurrently-Processed Concurrently-Processed

Adaptive ChipAdaptive Chip

Malcolm StaggMalcolm Stagg

Grade 11Grade 11

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Purpose:Purpose: To design a reconfigurable analog neural To design a reconfigurable analog neural

network on a chipnetwork on a chipA chip that can learn over timeA chip that can learn over timeFor any application neural networks are usedFor any application neural networks are used

Improvement over previous designs:Improvement over previous designs:High densityHigh densityHigh routabilityHigh routabilityOn-chip learningOn-chip learningMultiple learning algorithmsMultiple learning algorithms

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Neuron and synapse circuits are createdNeuron and synapse circuits are created For TSMC 0.35For TSMC 0.35µm CMOS (Complementary µm CMOS (Complementary

Metal Oxide Semiconductor) implementationMetal Oxide Semiconductor) implementation Dense layout of transistors for analog Dense layout of transistors for analog

arithmetic circuitsarithmetic circuits

High DensityHigh Density Neurons and Synapses are as small as Neurons and Synapses are as small as

possible without compromising performancepossible without compromising performance Supports both Backpropagation and Hebbian Supports both Backpropagation and Hebbian

learning on each celllearning on each cell

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Neural networks:Neural networks: Have been used for years in many applications, from Have been used for years in many applications, from

business and finance to science and engineeringbusiness and finance to science and engineering Ideally implemented as arrays of electronic cellsIdeally implemented as arrays of electronic cells Many learning algorithms have been designedMany learning algorithms have been designed

The first electronic neural network was made in The first electronic neural network was made in 1951, by Marvin Minsky (called “SNARC”)1951, by Marvin Minsky (called “SNARC”)

Carver Mead pioneered chip implementation of Carver Mead pioneered chip implementation of neural cells, designing a retina-like neural circuitneural cells, designing a retina-like neural circuit

Intel created a popular analog neural network Intel created a popular analog neural network chip in 1989 (called “ETANN”)chip in 1989 (called “ETANN”)

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Backpropagation:Backpropagation: Feedback-based learning algorithm (supervised Feedback-based learning algorithm (supervised

learning)learning) Requires calculation of the correct output for an input Requires calculation of the correct output for an input

patternpattern Will learn to represent the correct output over timeWill learn to represent the correct output over time

Hebbian:Hebbian: Non-feedback-based learning algorithm Non-feedback-based learning algorithm

(unsupervised learning)(unsupervised learning) Calculates synapse weights based on correlation Calculates synapse weights based on correlation

between input patternsbetween input patterns Will learn to identify and classify input patternsWill learn to identify and classify input patterns

Hopfield:Hopfield: Advanced memory recall algorithm, using Hebbian Advanced memory recall algorithm, using Hebbian

learninglearning

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Neuron Block DiagramNeuron Block Diagram

in out

BPin

HebbBP

BPout

Hebb

+ -

BP

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Neuron Cell DiagramNeuron Cell Diagram

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

The neuron design:The neuron design: Supports Backpropagation and HebbianSupports Backpropagation and Hebbian

Uses an SRAM cell to enable/disable algorithm-Uses an SRAM cell to enable/disable algorithm-dependent circuitsdependent circuits

Uses tanh sigmoid circuit for forwards-Uses tanh sigmoid circuit for forwards-propagation (range: [-1, 1])propagation (range: [-1, 1])

Derivative as sechDerivative as sech22 for backwards for backwards propagation (range: [0, 1])propagation (range: [0, 1])

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Synapse Block DiagramSynapse Block Diagram

in out

BPin

BP

BPout

Hebb

BP

Weight Update Parameters

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Synapse Cell DiagramSynapse Cell Diagram

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

The synapse design:The synapse design: Supports Backpropagation and HebbianSupports Backpropagation and Hebbian

Uses an SRAM cell to enable/disable algorithm-Uses an SRAM cell to enable/disable algorithm-dependent circuitsdependent circuits

Uses the Gilbert multiplier cell for forwards- Uses the Gilbert multiplier cell for forwards- and backwards- propagationand backwards- propagation

All inputs are ensured to be symmetrical (c+x, c-x)All inputs are ensured to be symmetrical (c+x, c-x) Differential pair-based circuits are usedDifferential pair-based circuits are used

Input range is small for good linearityInput range is small for good linearity

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

High routabilityHigh routability Multiple routing pathways between neurons Multiple routing pathways between neurons

and synapsesand synapsesUsing programmable wires and connectionsUsing programmable wires and connections

Enables use of advanced learning algorithmsEnables use of advanced learning algorithmsTo evolve/change the layout of the network over To evolve/change the layout of the network over timetime

This allows the network to become more dense This allows the network to become more dense and more efficient, while maintaining a low and more efficient, while maintaining a low synapse countsynapse count

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Programmable WireProgrammable Wire Programmable ConnectionProgrammable Connection

SRAM Analog Switch

SRAM Analog Switch

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Routing Cell DiagramRouting Cell Diagram

Neuron Synapse

Synapse Synapse

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Circuit DesignCircuit Design Created and modified analog and mixed-Created and modified analog and mixed-

signal circuits for each learning/propagation signal circuits for each learning/propagation tasktask

Most circuits are implemented using a small Most circuits are implemented using a small number of MOSFET transistorsnumber of MOSFET transistors

Transistors are relatively large (5Transistors are relatively large (5µm length) and µm length) and interdigitated for good matchinginterdigitated for good matchingSimulations have been completed for all cellsSimulations have been completed for all cells

CMOS layout has been completed for some of CMOS layout has been completed for some of the cellsthe cells

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Current SummationCurrent Summation

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Tanh and SechTanh and Sech22 (Sigmoid and Derivative) (Sigmoid and Derivative)

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive ChipDifferential Pairs (for multiplier input)Differential Pairs (for multiplier input)

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Gilbert Multiplier CellGilbert Multiplier Cell

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Layout:Layout: Completed for tanh/sechCompleted for tanh/sech22

First a stick diagram was plannedFirst a stick diagram was planned

Initial layout planned in Microsoft VisioInitial layout planned in Microsoft Visio

Drawn in Cadence Virtuoso using unmatched Drawn in Cadence Virtuoso using unmatched transistorstransistors

Drawn in Cadence Virtuoso using matched Drawn in Cadence Virtuoso using matched (interdigitated) transistors(interdigitated) transistors

Planned for other cells, but not yet completePlanned for other cells, but not yet complete

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Initial Virtuoso Layout of tanh (no matching)Initial Virtuoso Layout of tanh (no matching)

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Virtuoso Layout of tanh with interdigitated matchingVirtuoso Layout of tanh with interdigitated matching

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Initial Visio layout of tanh (no matching)Initial Visio layout of tanh (no matching)

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Simulation GraphsSimulation Graphs

Tanh SigmoidTanh Sigmoid

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Sigmoid:Sigmoid: Input Common Mode Range: 0.5V – 1.5VInput Common Mode Range: 0.5V – 1.5V Good accuracy compared to ideal function of Good accuracy compared to ideal function of

tanhtanh

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Simulation GraphsSimulation Graphs

SechSech22 Sigmoid Sigmoid

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Sigmoid Derivative:Sigmoid Derivative: Input Common Mode Range: 0.5V – 1.5VInput Common Mode Range: 0.5V – 1.5V Good accuracy compared to ideal function of Good accuracy compared to ideal function of

sechsech22

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Simulation GraphsSimulation Graphs

MultiplierMultiplier

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Multiplier:Multiplier: Input Common Mode Range: 2V – 3VInput Common Mode Range: 2V – 3V Symmetrical inputs around center pointSymmetrical inputs around center point Good linearity for input swing of Good linearity for input swing of ±±150mV150mV Performance is reduced as input range is Performance is reduced as input range is

extendedextendedPre-processing differential pair is requiredPre-processing differential pair is required

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Results:Results: A neural network is simulated in C++A neural network is simulated in C++

OCR problem, of 5 training sets of randomly OCR problem, of 5 training sets of randomly ordered numerical charactersordered numerical characters

10x10 pixels10x10 pixels 5% noise5% noise Position offset of Position offset of ±1 pixel horizontally and vertically±1 pixel horizontally and vertically

Comparing:Comparing: 100%-connected network100%-connected network 20%-connected network20%-connected network Simulation of a dynamically re-routed network by Simulation of a dynamically re-routed network by

removing 10% of the synapses over 1000 cyclesremoving 10% of the synapses over 1000 cycles

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

ResultsResults

0 to 1000 training cycles0 to 1000 training cycles0

5010

0P

erfo

rman

ce (

MS

E)

0 200 400 600 800 1000Cycle

Fully-connected N.Netw ork 20% Partially-connected N.Netw ork

Dynamically Re-routed N.Netw ork

Each Averaged Over 10 TrialsPERFORMANCE of NEURAL LEARNING METHODS

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

ResultsResults

200 to 1000 training cycles200 to 1000 training cycles0

510

15P

erfo

rman

ce (

MS

E)

200 400 600 800 1000Cycle

Fully-connected N.Network 20% Partially-connected N.Network

Dynamically Re-routed N.Network

Beginning at Cycle 200Each Averaged Over 10 Trials

PERFORMANCE of NEURAL LEARNING METHODS

05

1015

Per

form

ance

(M

SE

)

200 400 600 800 1000Cycle

Fully-connected N. Ntwk 20% Partially-connected N. Ntwk

Dynamically Re-routed N. Ntwk

Beginning at Cycle 200LINEAR PREDICTION

PERFORMANCE of NEURAL LEARNING METHODS

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

ResultsResults

800 to 1000 training cycles800 to 1000 training cycles0

.51

1.5

2P

erfo

rman

ce (

MS

E)

800 850 900 950 1000Cycle

Fully-connected N.Network 20% Partially-connected N.Network

Dynamically Re-routed N.Network

Beginning at Cycle 800Each Averaged Over 10 Trials

PERFORMANCE of NEURAL LEARNING METHODS

0.5

11.

52

Per

form

ance

(M

SE

)

800 850 900 950 1000Cycle

Fully-connected N. Ntwk 20% Partially-connected N. Ntwk

Dynamically Re-routed N. Ntwk

Beginning at Cycle 800LINEAR PREDICTION

PERFORMANCE of NEURAL LEARNING METHODS

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Conclusions:Conclusions: The dynamic routing algorithm is the ideal The dynamic routing algorithm is the ideal

method of optimizing performance in a method of optimizing performance in a hardware neural networkhardware neural network

Reducing the number of synapses to increase Reducing the number of synapses to increase densitydensity

The cell circuits perform very well with The cell circuits perform very well with system-made restrictions on input rangessystem-made restrictions on input ranges

Restrictions increase accuracy and linearityRestrictions increase accuracy and linearity The arrangement of routing cells between The arrangement of routing cells between

neurons and synapses allows the routing neurons and synapses allows the routing algorithm to perform very efficiently algorithm to perform very efficiently

A Dynamic Analog Concurrently-A Dynamic Analog Concurrently-Processed Adaptive ChipProcessed Adaptive Chip

Special Thanks To:Special Thanks To: Victoria StaggVictoria Stagg

My mother, for all of her help and supportMy mother, for all of her help and support Andrew StaggAndrew Stagg

My brother, for all of his help and supportMy brother, for all of his help and support Dr. Jim Haslett, University of CalgaryDr. Jim Haslett, University of Calgary

For use of the ATIPS laboratory, to use Cadence VirtuosoFor use of the ATIPS laboratory, to use Cadence Virtuoso John Carney, Cadence DesignJohn Carney, Cadence Design

For the demo license of Cadence Orcad LayoutFor the demo license of Cadence Orcad Layout Dr. Vance Tyree, MOSIS CorporationDr. Vance Tyree, MOSIS Corporation

For agreeing to accept a fabrication proposal under the For agreeing to accept a fabrication proposal under the MOSIS education programMOSIS education program

David Wells, Auton Engineering, Ltd.David Wells, Auton Engineering, Ltd.For printing my trifoldFor printing my trifold