ethem alpaydin © the mit press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...nonparametric estimation...
TRANSCRIPT
![Page 1: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/1.jpg)
ETHEM ALPAYDIN© The MIT Press, 2010
[email protected]://www.cmpe.boun.edu.tr/~ethem/i2ml2e
Lecture Slides for
![Page 2: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/2.jpg)
![Page 3: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/3.jpg)
Nonparametric Estimation Parametric (single global model), semiparametric (small
number of local models)
Nonparametric: Similar inputs have similar outputs
Functions (pdf, discriminant, regression) change smoothly
Keep the training data;“let the data speak for itself”
Given x, find a small number of closest training instances and interpolate from these
Aka lazy/memory-based/case-based/instance-based learning
3Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 4: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/4.jpg)
Given the training set X={xt}t drawn iid from p(x)
Divide data into bins of size h
Histogram:
Naive estimator:
or
Density Estimation
Nh
xxxp
t as bin same the in #ˆ
4
Nh
hxxhxxp
t
2
#ˆ
otherwise
if
0
1211
1
uuw
h
xxw
Nhxp
N
t
t /ˆ
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 5: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/5.jpg)
5Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 6: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/6.jpg)
6Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 7: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/7.jpg)
Kernel Estimator
N
t
t
h
xxK
Nhxp
1
1ˆ
7
Kernel function, e.g., Gaussian kernel:
Kernel estimator (Parzen windows)
22
1 2uuK exp
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 8: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/8.jpg)
8Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 9: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/9.jpg)
Instead of fixing bin width h and counting the number of instances, fix the instances (neighbors) k and check bin width
dk(x), distance to kth closest instance to x
k-Nearest Neighbor Estimator
xNd
kxp
k2ˆ
9Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 10: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/10.jpg)
10Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 11: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/11.jpg)
Kernel density estimator
Multivariate Gaussian kernel
spheric
ellipsoid
Multivariate Data
N
t
t
d hK
Nhp
1
1 xxxˆ
11
uuu
uu
1
212
2
2
1
2
1
22
1
SS
T
d
d
K
K
exp
exp
//
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 12: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/12.jpg)
Estimate p(x|Ci) and use Bayes’ rule
Kernel estimator
k-NN estimator
Nonparametric Classification
ti
N
t
t
diii
ii
ti
N
t
t
di
i
rh
KNh
CPCpg
N
NCPr
hK
hNCp
1
1
1
1
xxxx
xxx
ˆ|ˆ
ˆ |ˆ
12
k
k
p
CPCpCP
VN
kCp iii
iki
ii
x
xx
xx
ˆ
ˆ|ˆ|ˆ |ˆ
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 13: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/13.jpg)
Condensed Nearest Neighbor
13
Time/space complexity of k-NN is O (N)
Find a subset Z of X that is small and is accurate in classifying X (Hart, 1968)
ZZXXZ ||' EE
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 14: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/14.jpg)
Condensed Nearest Neighbor Incremental algorithm: Add instance if needed
14Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 15: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/15.jpg)
Nonparametric Regression
otherwise
withbin same the in is if
where
0
1
1
1
xxxxb
xxb
rxxbxg
tt
N
t
t
tN
t
t
,
,
,ˆ
15
Aka smoothing models
Regressogram
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 16: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/16.jpg)
16Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 17: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/17.jpg)
17Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 18: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/18.jpg)
Running Mean/Kernel Smoother Running mean smoother
Running line smoother
Kernel smoother
where K( ) is Gaussian
Additive models (Hastie and Tibshirani, 1990)
18
otherwise
1i f
where
ˆ
0
1
1
1
uuw
h
xxw
rh
xxw
xgN
t
t
tN
t
t
N
t
t
tN
t
t
h
xxK
rh
xxK
xg
1
1
ˆ
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 19: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/19.jpg)
19Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 20: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/20.jpg)
20Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 21: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/21.jpg)
21Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 22: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/22.jpg)
How to Choose k or h? When k or h is small, single instances matter; bias is
small, variance is large (undersmoothing): High complexity
As k or h increases, we average over more instances and variance decreases but bias increases (oversmoothing): Low complexity
Cross-validation is used to finetune k or h.
22Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
![Page 23: ETHEM ALPAYDIN © The MIT Press, 2010ethem/i2ml2e/2e_v1-0/i2ml2...Nonparametric Estimation Parametric (single global model), semiparametric (small number of local models) Nonparametric:](https://reader034.vdocuments.site/reader034/viewer/2022042620/5f3f1000cb325c54fa2fbd37/html5/thumbnails/23.jpg)
23Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)