cp365 artificial intelligence - colorado...
TRANSCRIPT
![Page 1: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/1.jpg)
CP365Artificial Intelligence
![Page 2: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/2.jpg)
Tech News!
Apple news conference tomorrow?
![Page 3: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/3.jpg)
Tech News!
Apple news conference tomorrow?
Google cancels Project Ara modular phone
![Page 4: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/4.jpg)
Weather-Based Stock Market Predictions?
![Page 5: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/5.jpg)
Dataset Preparation
Clean – remove bogus data/fill in missing data
Normalize data – adjust features to be similar magnitudes
![Page 6: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/6.jpg)
Deal with Missing Data
Option 1: remove datapoints with any missing feature values
![Page 7: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/7.jpg)
Deal with Missing Data
Option 1: remove datapoints with any missing feature values
Option 2: fill in missing data with <data_missing> tags for categorical data
![Page 8: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/8.jpg)
Deal with Missing Data
Option 1: remove datapoints with any missing feature values
Option 2: fill in missing data with <data_missing> tags for categorical data
Option 3: fill in missing data with global means for numeric data
![Page 9: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/9.jpg)
Deal with Missing Data
Option 1: remove datapoints with any missing feature values
Option 2: fill in missing data with <data_missing> tags for categorical data
Option 3: fill in missing data with global means for numeric data
Option 4: fill in missing data with values from similar data points
![Page 10: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/10.jpg)
Remove Outliers
Some datapoints may have ridiculous feature values.
We can remove outliers from our dataset to increase performance.
What is an outlier?
![Page 11: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/11.jpg)
Outliers
Patient Height (cm)
Patient Weight (kg)
... Prognosis
131.2 59.2 ... Good
176.7 82.9 ... Good
12613.9 66.0 ... Poor
161.0 70.2 ... Poor
![Page 12: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/12.jpg)
Outliers
Patient Height (cm)
Patient Weight (kgs)
... Prognosis
131.2 59.2 ... Good
176.7 82.9 ... Good
12613.9 66.0 ... Poor
161.0 70.2 ... Poor
Obvious outlier. How can we define what makes an outlier?
We could use 3σ as the threshold.
![Page 13: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/13.jpg)
Outliers
Patient Height (cm)
Patient Weight (kgs)
... Prognosis
131.2 59.2 ... Good
176.7 82.9 ... Good
12613.9 66.0 ... Poor
161.0 70.2 ... Poor
This column has x = 156.3 and σ = 23.1 (without the possible
outlier).
The 3σ thresholds would be (156.3 - 3 * 23.1, 156.3 + 3 * 23.1)
or(87, 225.6)
![Page 14: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/14.jpg)
A Bad Dataset
Patient Height (nm)
Patient Weight (tons)
... Prognosis
1.31 x 109 0.065 ... Good
1.76 x 109 0.091 ... Good
1.23 x 109 0.073 ... Poor
1.61 x 109 0.077 ... Poor
![Page 15: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/15.jpg)
A Bad Dataset
Patient Height (nm)
Patient Weight (tons)
... Prognosis
1.31 x 109 0.065 ... Good
1.76 x 109 0.091 ... Good
1.23 x 109 0.073 ... Poor
1.61 x 109 0.077 ... Poor
How will these large differences affect learning?
![Page 16: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/16.jpg)
Data Normalization Procedure
Patient Height (nm)
1.31 x 109
1.76 x 109
1.23 x 109
1.61 x 109
Range of Extreme Values
1.76 x 109
1.23 x 109
![Page 17: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/17.jpg)
Data Normalization Procedure
Patient Height (nm)
1.31 x 109
1.76 x 109
1.23 x 109
1.61 x 109
Range of Extreme Values
1.76 x 109
1.23 x 109
Normalized Range
1.0
0.0 (-1.0)
Mapping
![Page 18: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/18.jpg)
Data Normalization Formula
Patient Height (nm)
1.31 x 109
1.76 x 109
1.23 x 109
1.61 x 109
Say we want the normalized value, newpt, for the first height, 1.31 x 109, called pt.
oldmax = 1.76 x 109 oldmin = 1.23 x 109
newmax = 1.0newmin = 0.0
![Page 19: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/19.jpg)
Data Normalization Formula
newpt=( pt−oldminoldmax−oldmin )⋅(newmax−newmin )+newmin
Patient Height (nm)
1.31 x 109
1.76 x 109
1.23 x 109
1.61 x 109
newpt=0.15
Say we want the normalized value, newpt, for the first height, 1.31 x 109, called pt.
oldmax = 1.76 x 109 oldmin = 1.23 x 109
newmax = 1.0newmin = 0.0
![Page 20: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/20.jpg)
How do we know if an ML model is any good?
![Page 21: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/21.jpg)
Overfitting
![Page 22: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/22.jpg)
Error
Epoch
Training
Testing
![Page 23: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/23.jpg)
A Biological Neuron
![Page 24: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/24.jpg)
Human Brain
![Page 25: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/25.jpg)
How many neurons?
AnimalNumber Neurons (cerebral cortex)
Rat 20,000,000
Dog 160,000,000
Cat 300,000,000
Pig 450,000,000
Horse 1,200,000,000
Dolphin 5,800,000,000
African Elephant 11,000,000,000
Human 20,000,000,000
![Page 26: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/26.jpg)
How many connections?
Human 100,000,000,000,000
![Page 27: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/27.jpg)
How many connections?
Human 100,000,000,000,000
Google (2012) 1,700,000,000
Google/Stanford (2013) 11,200,000,000
Digital Reasoning (2015) 160,000,000,000
![Page 28: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/28.jpg)
Artificial Neuron
Threshold Function
w1
w2 w3Input connectionsand weights
Output connections
![Page 29: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/29.jpg)
Hard Threshold
Threshold Function
w1
w2 w3
S = Sum up all inputi * weight
i
if S > THRESHOLD: output = 1else: output = 0
![Page 30: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/30.jpg)
Hard Threshold:Step Function
![Page 31: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/31.jpg)
Write down artificial neurons with weights and thresholds that model the following functions:
IdentityLogical ANDLogical OR
Logical XORConstant function
![Page 32: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/32.jpg)
Sigmoid Threshold
Threshold Function
w1
w2 w3
S = Sum up all inputi * weight
i
output = 1
1e−S
![Page 33: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/33.jpg)
Sigmoid Threshold:'S' Function
![Page 34: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/34.jpg)
sigmoid
w1 = 0.1
w2 = 0.2
w3 = 0.42
![Page 35: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/35.jpg)
sigmoid
w1 = 0.1
w2 = 0.2
w3 = 0.42
Features x1 = 0.66 x2 = 0.11 x3 = 0.20
![Page 36: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/36.jpg)
s = w1 * x1 + w2 * x2 + w3 * x3s = 0.1 * 0.66 + 0.2 * 0.11 + 0.42 * 0.2s = 0.09
1
1e−0.09=0.52
Output Calculations
![Page 37: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/37.jpg)
sigmoid
w1 = 0.1
w2 = 0.2
w3 = 0.42
Features x1 = 0.66 x2 = 0.11 x3 = 0.20
y1 = 0.52
![Page 38: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/38.jpg)
Perceptron Network
Input Layer
Output Layer
![Page 39: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/39.jpg)
Perceptron: Linear Boundary
![Page 40: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/40.jpg)
Linear Boundary?
![Page 41: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/41.jpg)
Multilayer Network
Input Layer
Hidden Layer(s)
Output Layer
![Page 42: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/42.jpg)
ANN Learning – How to get the weights?
![Page 43: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/43.jpg)
ANN Learning – How to get the weights?
weight1 weight2
error
![Page 44: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/44.jpg)
ANN Learning
● How do we get the right weights?
● Perceptron:● Gradient descent
● Multilayer Network:● Back propagation
![Page 45: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/45.jpg)
Node Activation Function
Activation (output) of node j.
a j=g(input j)=g(∑i=0
n
wij ai)
![Page 46: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/46.jpg)
Node Activation Function
g is the threshold activation function.
Activation (output) of node j.
a j=g(input j)=g(∑i=0
n
wij ai)
![Page 47: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/47.jpg)
Node Activation Function
a j=g(input j)=g(∑i=0
n
wij ai)
g is the threshold activation function.
Activation (output) of node j.
Sum of all weights and input values.
![Page 48: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/48.jpg)
Minimize Global Error Function
error=∑j
(t j−a j)2
For every output node, j, sum up...
![Page 49: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/49.jpg)
Minimize Global Error Function
error=∑j
(t j−a j)2
...the difference in target value vs. generated output
value and square it.For every output node, j, sum up...
![Page 50: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/50.jpg)
Perceptron Learning
Δwij=η(t j−a j)ai
Update the weight on connection i → j
![Page 51: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/51.jpg)
Perceptron Learning
Δwij=η(t j−a j)ai
Update the weight on connection i → j
The learning rate (0.3ish)
![Page 52: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/52.jpg)
Perceptron Learning
Δwij=η(t j−a j)ai
Update the weight on connection i → j
The learning rate (0.3ish)
Difference in target and generated output.
![Page 53: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/53.jpg)
Perceptron Learning
Δwij=η(t j−a j)ai
Update the weight on connection i → j
The learning rate (0.3ish)
Difference in target and generated output.
Input activation
![Page 54: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/54.jpg)
Let's learn NAND!
Input1 Input2 Label
0 0 1
0 1 1
1 0 1
1 1 0In1 In2
Out
Starting weight values: W1 = 0.81, W2 = 0.55, W3 = 0.16
η = 0.3
Use sigmoid threshold
W1 W2
Dataset: NAND
a j=g (input j)=g (∑i=0
n
w jiai)
Δwij=η(t j−a j)ai
1.0
W3
![Page 55: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/55.jpg)
ANN Learning - Backpropagation
Input Layer
Hidden Layer
Output Layer
Put in input
values and feed
the activation forward
to produce
the output.
![Page 56: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/56.jpg)
ANN Learning - Backpropagation
Input Layer
Hidden Layer
Output Layer
Calculate the error in the output layer and
then back-propagate it to update
lower weights.
![Page 57: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/57.jpg)
ANN Learning - Backpropagation
Δwij=ηδ j ai
Update the weight on connection i → j
![Page 58: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/58.jpg)
ANN Learning - Backpropagation
Δwij=ηδ j ai
Update the weight on connection i → j
Think of this as the error measure for node j.
Different for output and hidden weights.
![Page 59: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/59.jpg)
ANN Learning - Backpropagation
Δwij=ηδ j ai
Update the weight on connection i → j Input activation
Think of this as the error measure for node j.
Different for output and hidden weights.
![Page 60: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/60.jpg)
ANN Learning – Backpropagationfor Output Nodes
δ j=a j(1−a j)(t j−a j)
Error measure for output node, j.
![Page 61: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/61.jpg)
ANN Learning – Backpropagationfor Output Nodes
δ j=a j(1−a j)(t j−a j)
Derivative of sigmoid function.
Error measure for output node, j.
![Page 62: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/62.jpg)
ANN Learning – Backpropagationfor Output Nodes
δ j=a j(1−a j)(t j−a j)
Difference in target vs. generated output.
Derivative of sigmoid function.
Error measure for output node, j.
![Page 63: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/63.jpg)
ANN Learning – Backpropagationfor Hidden Nodes
δ j=a j(1−a j)∑k
δk w jk
Error measure for hidden node, j.
![Page 64: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/64.jpg)
ANN Learning – Backpropagationfor Hidden Nodes
δ j=a j(1−a j)∑k
δk w jk
Error measure for hidden node, j.
Derivative of sigmoid function.
![Page 65: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/65.jpg)
ANN Learning – Backpropagationfor Hidden Nodes
δ j=a j(1−a j)∑k
δk w jk
Error measure a combination of output errors that this weight
contributes to.
Error measure for hidden node, j.
Derivative of sigmoid function.
![Page 66: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/66.jpg)
ANN Learning
● Initialize random network weights● for epoch in range NUMBER_EPOCHS:
● Train network on random presentation of instances● Update weights with backpropagation● Report global error function value
![Page 67: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/67.jpg)
Choosing the Learning Rate, η
What happened when our learning rate was too high for linear regression?
How do we choose an appropriate learning rate for ANNs?
![Page 68: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/68.jpg)
Bold Driver
After each epoch...
if error went down:η = η * 1.05
else:η = η * 0.50
sodahead.com
![Page 69: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/69.jpg)
Choosing the Network Structure
Input Layer
Hidden Layer
Output Layer
How many nodes? What are their connections?
![Page 70: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/70.jpg)
Choosing the Network Structure
Input Layer
Hidden Layer
Output Layer
# of output nodes determined by the number of function
outputs.
![Page 71: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/71.jpg)
Choosing the Network Structure
Input Layer
Hidden Layer
Output Layer
# of input nodes determined by the number of function
inputs.
![Page 72: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/72.jpg)
Choosing the Network Structure
Input Layer
Hidden Layer
Output Layer
Too few hidden nodes: unable to get
a detailed enough approximation of the
target function
![Page 73: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/73.jpg)
Choosing the Network Structure
Input Layer
Hidden Layer
Output Layer
Too many hidden nodes: slower to train and easier to overfit
training data
![Page 74: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/74.jpg)
ANN Representational Power
● With one hidden layer:● Model all continuous functions
● With two hidden layers:● Model all functions
![Page 75: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/75.jpg)
Rules of Thumb
● Use 1 or 2 hidden layers
![Page 76: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/76.jpg)
Rules of Thumb
● Use 1 or 2 hidden layers
● Use about (2/3)n hidden nodes for reasonably complex functions
![Page 77: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/77.jpg)
Rules of Thumb
● Use 1 or 2 hidden layers
● Use about (2/3)n hidden nodes for reasonably complex functions
● Don't train for too many epochs
![Page 78: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/78.jpg)
Splitting up datasets
Training data – use to train your ML model
Validation data – use to improve your ML model while training
Testing data – use to test performance of your ML model
![Page 79: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/79.jpg)
K-Fold Cross Validation
Full Dataset Dataset split into k chunks
![Page 80: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/80.jpg)
K-Fold Cross Validation: Pass 1
Training Dataset Validation Dataset
![Page 81: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/81.jpg)
K-Fold Cross Validation: Pass 2
Training Dataset Validation Dataset
![Page 82: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/82.jpg)
K-Fold Cross Validation
Perform K training/validation passes
Each pass counts as a classification accuracy sample
Extreme case: K = datasetSizeLeave-one-out testing
![Page 83: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/83.jpg)
ANN Implementation?
![Page 84: CP365 Artificial Intelligence - Colorado Collegecs.coloradocollege.edu/~mwhitehead/courses/2016... · Google cancels Project Ara modular phone. Weather-Based Stock Market Predictions?](https://reader035.vdocuments.site/reader035/viewer/2022071007/5fc4fa4eaa796d07bb35df56/html5/thumbnails/84.jpg)
Break!