Neural Networks

William LaiChris Rowlett

What are Neural Networks?

• A type of program that is completely different from functional programming.

• Consists of units that carry out simple computations linked together to perform a function

• Modeled after the decision making process of the biological network of neurons in the brain

The Biology of Neural Networks

• Neural Networks are models of neuron clusters in the brain• Each Neuron has a:

• Dendrites

• Axon

• Terminal buds

• Synapse

• Action potential is passed down the axon, which causes the release of neurotransmitters

Types of Neural Networks:General

• Supervised• During training, error is determined by subtracting

output from actual value

• Unsupervised• Nothing is known of results• Used to classify complicated data

• Nonlearning• Optimization

Types of Neural Networks:Specific

• Perceptrons• A subset of feed-forward networks, containing only one input layer,

one output layer, and each input unit links to only output units

• Feed-forward networks• a.k.a. Directed Acyclic Graphs

• Each unit only links to units in subsequent layers

• Allows for hidden layers

• Recurrent networks• Not very well understood

• Units can link to units in the same layer or even previous layers

• Example: The Brain

Neural Net Capabilities

• Neural Nets can do anything a normal digital computer can do (such as perform basic or complex computations)

• Functional Approximations/Mapping• Classification• Good at ignoring ‘noise’

Neural Net Limitations

• Problems similar to Y=1/X between (0,1) on the open interval

• (Pseudo)-random number predictors• Factoring integers or determining prime

numbers• Decryption

History of Neural Networks

• McColloch and Pitts (1943)• Co-wrote first paper on possible model for a

neuron

• Widrow Hoff (1959)• Developed MADALINE and ADALINE• MADALINE was the first neural network to try to

solve a real world problem• Eliminates echo in phone lines

• vonNeumann architecture took over for about 20 years (60’s-80’s)

Early Applications

• Checkers (Samuel, 1952)• At first, played very poorly as a novice• With practice games, eventually beat its author

• ADALINE (Widrow and Hoff, 1959)• Recognizes binary patterns in streaming data

• MADALINE (same)• Multiple ADAptive LINear Elements• Uses an adaptive filter that eliminates echoes on

phone lines

Modern Practical Applications

• Pattern recognition, including• Handwriting Deciphering• Voice Understanding• “Predictability of High-Dissipation Auroral Activity”

• Image analysis• Finding tanks hiding in trees (cheating)• Material Classification

• "A real-time system for the characterization of sheep feeding phases from acoustic signals of jaw sounds"

How Do Neural Networks Relate to Artificial

Intelligence?• Neural networks are usually geared towards

some application, so they represent the practical action aspect of AI

• Since neural networks are modeled after human brains, they are an imitation of human action. However, than can be taught to act rationally instead.

• Neural networks can modify their own weights and learn.

The Future of Neural Networks

• Pulsed neural networks• The AI behind a good Go playing agent• Increased speed through the making of chips• robots that can see, feel, and predict the world

around them• improved stock prediction • common usage of self-driving cars• Applications involving the Human Genome• Project self-diagnosis of medical problems

using neural networks

Past Difficulties

• Single-layer approach limited applications• Converting Widrow-Hoff Technique for use with

multiple layers• Use of poorly chosen and derived learning function• High expectations and early failures led to loss of

funding

Recurring Difficulties

• Cheating• Exactly what a neural net is doing to get its solutions is

unknown and therefore, it can cheat to find the solution as opposed to find a reliable algorithm

• Memorization• Overfitting without generalization

Describing Neural Net Units

• All units have input values, aj

• All input values are weighted, as in each aj is multiplied by the link’s weight, Wj,i

• All weighted inputs are summed, generating ini

• The unit’s activation function is called on ini, generating the activation value ai

• The activation value is output to every destination of the current unit’s links.

Perceptrons

• Single layer neural networks

• Require linearly separable functions

• Guarantees the one solution

OR

XOR

Back-Propagation

• Back-propagation uses a special function to divide the error of the outputs to all the weights of the network

• The result is a slow-learning method for solving many real world problems

€

Wj, i ← Wj, i +α × aj × Err × g' ini( )

Organic vs. Artificial

• Computer cycle times are in the order of nanoseconds while neurons take milliseconds

• Computers compute the results of each neuron sequentially, while all neurons in the brain fire simultaneously every cycle

• Result: massive parallelism makes brains a billion times faster than computers, even though computer bits can cycle a million times faster than neurons

Questions?