pattern’recognition’’ and’machine’learning’shirani/vision/prml_ch1.pdf ·...
TRANSCRIPT
![Page 1: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/1.jpg)
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 1: INTRODUCTION
![Page 2: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/2.jpg)
Polynomial Curve Fi0ng
![Page 3: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/3.jpg)
Sum-‐of-‐Squares Error Func9on
![Page 4: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/4.jpg)
0th Order Polynomial
![Page 5: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/5.jpg)
1st Order Polynomial
![Page 6: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/6.jpg)
3rd Order Polynomial
![Page 7: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/7.jpg)
9th Order Polynomial
![Page 8: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/8.jpg)
Over-‐fi0ng
Root-‐Mean-‐Square (RMS) Error:
![Page 9: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/9.jpg)
Polynomial Coefficients
![Page 10: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/10.jpg)
Data Set Size:
9th Order Polynomial
![Page 11: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/11.jpg)
Data Set Size:
9th Order Polynomial
![Page 12: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/12.jpg)
Regulariza9on
Penalize large coefficient values
![Page 13: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/13.jpg)
Regulariza9on:
![Page 14: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/14.jpg)
Regulariza9on:
![Page 15: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/15.jpg)
Regulariza9on: vs.
![Page 16: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/16.jpg)
Polynomial Coefficients
![Page 17: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/17.jpg)
The Rules of Probability
Sum Rule
Product Rule
![Page 18: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/18.jpg)
Bayes’ Theorem
Posterior likelihood × prior /
![Page 19: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/19.jpg)
Probability Densi9es
![Page 20: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/20.jpg)
Transformed Densi9es
![Page 21: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/21.jpg)
Expecta9ons
Condi9onal Expecta9on (discrete)
Approximate Expecta9on (discrete and con9nuous)
![Page 22: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/22.jpg)
Variances and Covariances
![Page 23: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/23.jpg)
The Gaussian Distribu9on
![Page 24: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/24.jpg)
Gaussian Mean and Variance
![Page 25: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/25.jpg)
The Mul9variate Gaussian
![Page 26: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/26.jpg)
Gaussian Parameter Es9ma9on
Likelihood func9on
![Page 27: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/27.jpg)
Maximum (Log) Likelihood
![Page 28: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/28.jpg)
Curve Fi0ng Re-‐visited
![Page 29: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/29.jpg)
Maximum Likelihood
Determine by minimizing sum-‐of-‐squares error, .
![Page 30: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/30.jpg)
Predic9ve Distribu9on
![Page 31: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/31.jpg)
MAP: A Step towards Bayes
Determine by minimizing regularized sum-‐of-‐squares error, .
![Page 32: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/32.jpg)
Bayesian Curve Fi0ng
![Page 33: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/33.jpg)
Bayesian Predic9ve Distribu9on
![Page 34: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/34.jpg)
Curse of Dimensionality
![Page 35: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/35.jpg)
Curse of Dimensionality
Polynomial curve fi0ng, M = 3 Gaussian Densi9es in higher dimensions
![Page 36: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/36.jpg)
Decision Theory
Inference step Determine either or .
Decision step For given x, determine op9mal t.
![Page 37: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/37.jpg)
Minimum Misclassifica9on Rate
![Page 38: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/38.jpg)
Minimum Expected Loss
Example: classify medical images as ‘cancer’ or ‘normal’
Decision Truth
![Page 39: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/39.jpg)
Minimum Expected Loss
Regions are chosen to minimize
![Page 40: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/40.jpg)
Reject Op9on
![Page 41: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/41.jpg)
Why Separate Inference and Decision?
• Minimizing risk (loss matrix may change over 9me) • Reject op9on • Unbalanced class priors • Combining models
![Page 42: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/42.jpg)
Decision Theory for Regression
Inference step Determine .
Decision step For given x, make op9mal predic9on, y(x), for t.
Loss func9on:
![Page 43: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/43.jpg)
The Squared Loss Func9on
![Page 44: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/44.jpg)
Genera9ve vs Discrimina9ve
Genera9ve approach: Model Use Bayes’ theorem
Discrimina9ve approach: Model directly
![Page 45: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/45.jpg)
Entropy
Important quan9ty in • coding theory • sta9s9cal physics • machine learning
![Page 46: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/46.jpg)
Entropy
Coding theory: x discrete with 8 possible states; how many bits to transmit the state of x?
All states equally likely
![Page 47: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/47.jpg)
Entropy
![Page 48: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/48.jpg)
Differen9al Entropy
Put bins of width ¢ along the real line Differen9al entropy maximized (for fixed ) when in which case
![Page 49: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/49.jpg)
Condi9onal Entropy
![Page 50: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/50.jpg)
The Kullback-‐Leibler Divergence
![Page 51: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t](https://reader033.vdocuments.site/reader033/viewer/2022042305/5ed0fe54d3b82946a13ab81f/html5/thumbnails/51.jpg)
Mutual Informa9on