comp 328: midterm review spring 2010 nevin l. zhang department of computer science & engineering...
Post on 20-Dec-2015
217 views
TRANSCRIPT
COMP 328: Midterm Review
Spring 2010
Nevin L. Zhang
Department of Computer Science & Engineering
The Hong Kong University of Science & Technology
http://www.cse.ust.hk/~lzhang/
Can be used as cheat sheet
Page 2
Overview
Algorithms for supervised learning Decision trees
Naïve Bayes classifiers
Neural networks
Instance-based learning
Support vector machines
General issues regarding supervised learning Classification error and confidence interval
Bias-Variance tradeoff
PAC learning theory
Supervised Learning
Page 3
Decision Trees
Page 4
Decision trees
Page 5
Reduced-Error Pruning
Page 6
Decision Trees
Issues with attributes Continuous
Attributes with many values Use GainRatio instead of Gain
Missing values
Tree construction is a search process Local minimum
Page 7
Naïve Bayes Classifier
Page 8
Can classify using this rule:
But, joint too expensive to get
Naïve Bayes Classifier
Page 9
Learning Naïve Bayes Classifier
Page 10
Laplace smoothing Continuous attribute When independence not true, double counting of evidence Generalization: Bayesian networks
Neural Networks
Page 11
For classification and regression
Neural Networks
Activation function Step, sign
Sigmoid, tanh (hyperbolic tangent)
Page 12
Neural Network/Properties
Perceptrons are linear classifier
Two-layer network with enough perceptron units can
represent all Boolean functions
One layer with enough sigmoid units can approximate any
functions well
Page 13
Neural Network
Page 14
Converge only when linearly separable
Neural Network
Page 15
Adaline learning: Delta rule
Neural Network
Page 16
Instance-Based Learning
Lazy learning K-NN
Distance-weighted k-NN (kernel regression)
Locally weighted regression
Page 17
Support Vector Machines
Page 18
SVM
Page 19
SVM
Page 20
SVM
Page 21
SVM
Data not linearly separable
Page 22
SVM
Page 23
Nonlinear SVM
Page 24
Impact of σ and C
Page 25
Classifier Evaluation
Relationship between
Page 26
Algorithm Evaluation/Model Selection
Page 27
Which learning algorithm to use? Given algorithm, which model to use? (How many hidden units?)
Algorithm Evaluation/Model Selection
Page 28
Bias-Variance Decomposition
Page 29
Bias-Variance Tradeoff
Page 30
For classification problem also
PAC Learning Theory
Probably approximate correct (PAC)
Relationship between
Page 31
PAC Learning Theory
Page 32
VC Dimension
Page 33
Sample Complexity
Page 34