Download - More Classifiers
MORE CLASSIFIERS
AGENDA Key concepts for all classifiers
Precision vs recall Biased sample sets
Linear classifiers Intro to neural networks
RECAP: DECISION BOUNDARIES With continuous attributes, a decision
boundary is the surface in example space that splits positive from negative examples
x1>=20 x2
x1F
x2>=10
T
F
F
T
x2>=15
T F
T
4
BEYOND ERROR RATES
BEYOND ERROR RATE Predicting security risk
Predicting “low risk” for a terrorist, is far worse than predicting “high risk” for an innocent bystander (but maybe not 5 million of them)
Searching for images Returning irrelevant images is
worse than omitting relevant ones
5
BIASED SAMPLE SETS Often there are orders of magnitude more
negative examples than positive E.g., all images of Kris on Facebook If I classify all images as “not Kris” I’ll have
>99.99% accuracy
Examples of Kris should count much more than non-Kris!
FALSE POSITIVES
7x1
x2
True decision boundary Learned decision boundary
FALSE POSITIVES
8x1
x2
New query
An example incorrectly predicted
to be positive
True decision boundary Learned decision boundary
FALSE NEGATIVES
9x1
x2
New query
An example incorrectly predicted
to be negative
True decision boundary Learned decision boundary
PRECISION VS. RECALL Precision
# of relevant documents retrieved / # of total documents retrieved
Recall # of relevant documents retrieved / # of total
relevant documents Numbers between 0 and 1
10
PRECISION VS. RECALL Precision
# of true positives / (# true positives + # false positives)
Recall # of true positives / (# true positives + # false
negatives) A precise classifier is selective A classifier with high recall is inclusive
11
REDUCING FALSE POSITIVE RATE
12x1
x2
True decision boundary Learned decision boundary
REDUCING FALSE NEGATIVE RATE
13x1
x2
True decision boundary Learned decision boundary
PRECISION-RECALL CURVES
14
Precision
Recall
Measure Precision vs Recall as the decision boundary is tuned
Perfect classifier
Actual performance
PRECISION-RECALL CURVES
15
Precision
Recall
Measure Precision vs Recall as the decision boundary is tuned
Penalize false negatives
Penalize false positives
Equal weight
PRECISION-RECALL CURVES
16
Precision
Recall
Measure Precision vs Recall as the decision boundary is tuned
PRECISION-RECALL CURVES
17
Precision
Recall
Measure Precision vs Recall as the decision boundary is tuned
Better learningperformance
OPTION 1: CLASSIFICATION THRESHOLDS Many learning algorithms (e.g., probabilistic
models, linear models) give real-valued output v(x) that needs thresholding for classification
v(x) > t => positive label given to xv(x) < t => negative label given to x
May want to tune threshold to get fewer false positives or false negatives
18
OPTION 2: WEIGHTED DATASETS Weighted datasets: attach a weight w to
each example to indicate how important it is Instead of counting “# of errors”, count “sum of
weights of errors” Or construct a resampled dataset D’ where each
example is duplicated proportionally to its w As the relative weights of positive vs
negative examples is tuned from 0 to 1, the precision-recall curve is traced out
LINEAR CLASSIFIERS : MOTIVATION Decision tree produces axis-aligned decision
boundaries Can we accurately classify data like this?
x2
x1
PLANE GEOMETRY Any line in 2D can be expressed as the set of
solutions (x,y) to the equation ax+by+c=0 (an implicit surface) ax+by+c > 0 is one side of the line ax+by+c < 0 is the other ax+by+c = 0 is the line itself
y
x
b
a
PLANE GEOMETRY In 3D, a plane can be expressed as the set of
solutions (x,y,z) to the equation ax+by+cz+d=0 ax+by+cz+d > 0 is one side of the plane ax+by+cz+d < 0 is the other side ax+by+cz+d = 0 is the plane itself
a b
c
z
x
y
LINEAR CLASSIFIER In d dimensions,
c0+c1*x1+…+cd*xd =0 is a hyperplane. Idea:
Use c0+c1*x1+…+cd*xd > 0 to denote positive classifications
Use c0+c1*x1+…+cd*xd < 0 to denote negative classifications
24
PERCEPTRON
S gxi
x1
xn
ywi
y = f(x,w) = g(Si=1,…,n wi xi)
+ +
+
++ -
-
--
-x1
x2
w1 x1 + w2 x2 = 0
g(u)
u
25
A SINGLE PERCEPTRON CAN LEARN
S gxi
x1
xn
ywi
A disjunction of boolean literals x1 x2 x3
Majority function
26
A SINGLE PERCEPTRON CAN LEARN
S gxi
x1
xn
ywi
A disjunction of boolean literals x1 x2 x3
Majority functionXOR?
27
PERCEPTRON LEARNING RULE θ θ + x(i)(y(i)-g(θT x(i))) (g outputs either 0 or 1, y is either 0 or 1)
If output is correct, weights are unchanged If g is 0 but y is 1, then the value of g on
attribute i is increased If g is 1 but y is 0, then the value of g on
attribute i is decreased
Converges if data is linearly separable, but oscillates otherwise
28
PERCEPTRON
S gxi
x1
xn
ywi
+ +
+ +
+ -
-
- -
-
?
y = f(x,w) = g(Si=1,…,n wi xi)
g(u)
u
29
UNIT (NEURON)
S gxi
x1
xn
ywi
y = g(Si=1,…,n wi xi)g(u) = 1/[1 + exp(-u)]
30
NEURAL NETWORK Network of interconnected neurons
S gxi
x1
xn
ywi
S gxi
x1
xn
ywi
Acyclic (feed-forward) vs. recurrent networks
31
TWO-LAYER FEED-FORWARD NEURAL NETWORK
Inputs Hiddenlayer
Outputlayer
w1j w2k
32
NETWORKS WITH HIDDEN LAYERS Can represent XORs, other nonlinear
functions Common neuron types:
Soft perceptron (sigmoid), radial basis functions, linear, …
As the number of hidden units increase, so does the network’s capacity to learn functions with more nonlinear features
How to train hidden layers?
33
BACKPROPAGATION (PRINCIPLE) Treat the problem as one of minimizing
errors between the example label and the network output, given the example and network weights as input Error(xi,yi,w) = (yi – f(xi,w))2
Sum this error term over all examples E(w) = Si Error(xi,yi,w) = Si (yi – f(xi,w))2
Minimize errors using an optimization algorithm Stochastic gradient descent is typically used
Gradient direction is orthogonal to the level sets (contours) of E,points in direction of steepest increase
Gradient direction is orthogonal to the level sets (contours) of E,points in direction of steepest increase
Gradient descent: iteratively move in direction
Gradient descent: iteratively move in direction E
Gradient descent: iteratively move in direction E
Gradient descent: iteratively move in direction E
Gradient descent: iteratively move in direction E
Gradient descent: iteratively move in direction
Gradient descent: iteratively move in direction
STOCHASTIC GRADIENT DESCENT For each example (xi,yi), take a gradient
descent step to reduce the error for (xi,yi) only.
43
STOCHASTIC GRADIENT DESCENT Objective function values (measured over all
examples) over time settle into local minimum
Step size must be reduced over time, e.g., O(1/t)
44
NEURAL NETWORKS: PROS AND CONS Pros
Bioinspiration is nifty Can represent a wide variety of decision boundaries Complexity is easily tunable (number of hidden
nodes, topology) Easily extendable to regression tasks
Cons Haven’t gotten close to unlocking the power of the
human (or cat) brain Complex boundaries need lots of data Slow training Mostly lukewarm feelings in mainstream ML
(although the “deep learning” variant is en vogue now)
NEXT CLASS Another guest lecture