boosted decision treesketelsen/files/courses/csci4622/slides/lesson37.pdfpreviously on csci 4622 2...

Post on 27-Sep-2020

4 Views

Category:

Documents

0 Downloads

Preview:

Click to see full reader

TRANSCRIPT

Boosted Decision Trees

Previously on CSCI 4622

2

Ensemble Learning: Build a bunch of slightly different models. Predict by majority vote.

Bagging: Train a bunch of decision trees on bootstrapped resamples of training data.

Random Forests: Train a bunch of decision trees on bootstrapped resamples of training data. On each node, consider only a random subset of features to split on.

A single tree is very sensitive to training data. An average over many trees is less sensitive.

Bagging can sometimes create correlated trees, especially if there’s a handful of strong features

Random Forests helps to de-correlate trees by leaving strong features out in some splits

Boosting Intuition

3

Boosting is another ensemble model, but with a slight twist

“How is it that a committee of blockheads can somehow arrive at highly reasoned decisions, despite the weak judgment of the individual members? How can the shaky separate views of a

panel of dolts be combined into a single opinion that is very likely to be correct?

Schapire and Freund, Boosting: Foundations and Algorithms”

"

I

Boosting Intuition

4

Boosting is another ensemble model, but with a slight twist

“How is it that a committee of blockheads can somehow arrive at highly reasoned decisions, despite the weak judgment of the individual members? How can the shaky separate views of a

panel of dolts be combined into a single opinion that is very likely to be correct?

Schapire and Freund, Boosting: Foundations and Algorithms”

Idea: Build a sequence of dumb models

Modify the training data along the way to focus on difficult to classify examples

Predict based on weighted majority vote of all dumb models

o What do we mean by dumb? o How do we promote difficult examples? o Which models get more say in the vote?

Dumb Models

5

What do we mean by dumb?

Each model in our sequence will be a weak learner

A model is a weak learner if its training error is just under 50%

Most common weak learner is a Decision Stump – a Decision Tree with a single split

Other common weak learners include slightly deeper Decision Trees and perceptrons

h(x)<latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit><latexit sha1_base64="DpZ0qqzuWcfNwUlQjJ+pzy3FXxM=">AAAB93icbVBNT8JAEJ3iF9YPqh69bAQTvJCWC3oj8eIREyskQMh22cKG7bbZ3Rqx4Zd48aDGq3/Fm//GBXpQ8CWTvLw3k5l5QcKZ0q77bRU2Nre2d4q79t7+wWHJOTq+V3EqCfVJzGPZCbCinAnqa6Y57SSS4ijgtB1Mrud++4FKxWJxp6cJ7Ud4JFjICNZGGjilij2uZr0gRI+zC7uCBk7ZrbkLoHXi5aQMOVoD56s3jEkaUaEJx0p1PTfR/QxLzQinM7uXKppgMsEj2jVU4IiqfrY4fIbOjTJEYSxNCY0W6u+JDEdKTaPAdEZYj9WqNxf/87qpDi/7GRNJqqkgy0VhypGO0TwFNGSSEs2nhmAimbkVkTGWmGiTlW1C8FZfXid+vXZV827r5WYjT6MIp3AGVfCgAU24gRb4QCCFZ3iFN+vJerHerY9la8HKZ07gD6zPHx5mkPE=</latexit>

err =1

m

mX

i=1

I(yi 6= h(xi)) =1

2� �

<latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit><latexit sha1_base64="N46xc2aFCg/Ztbtuk1eTBKVbj1k=">AAACUnicbVJNbxMxFPSGr3YpNNAjF4sUKT0Q7eZSOFSqxIXeitTQSnFYeb1vE1Pbu9hvUVfW/keE1EP/CBcOrZPmQFtGsjSaee/ZHjuvlXSYJFdR79HjJ0+fbWzGz7devNzuv3r91VWNFTARlarsWc4dKGlgghIVnNUWuM4VnObnn5b+6U+wTlbmBNsaZprPjSyl4BikrP99N2YIF2i1B2s7ekBZUVoufNp53bFCulrx1mGrgLlGZ14epN03TY+GbSYpM/CDLoae5SW96DK5t3dnwLij7ymbc615vEuz/iAZJSvQhyRdkwFZ4zjr/2ZFJRoNBoXizk3TpMaZ5xalUNDFrHFQc3HO5zAN1HANbuZXmXT0XVAKWlY2LIN0pf7b4bl2rtV5qNQcF+6+txT/500bLD/MvDR1g2DE7UZloyhWdBkwLaQFgaoNhAsrw1mpWPCQCIZniEMI6f0rPyST8ejjKP0yHhzur9PYIG/IWzIkKdknh+QzOSYTIsgv8odcRyS6jP72wi+5Le1F654dcge9rRt+/rHz</latexit>

Promoting Difficult Examples

6

How do we promote difficult examples?

After each iteration:

o Increase importance of training examples we got wrong on previous iteration

o Decrease importance of training examples we got right on previous iteration

Each training example will carry around a weight

Weights are nonnegative normalized so that they behave like a probability distribution

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

mX

i=1

wi = 1, wi � 0<latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit><latexit sha1_base64="o/WwpIg7D+Ld/IVkOPR7DtWc7HM=">AAACIHicbVA9T8MwEHX4LOGrwMhi0SIxoCrp0jIgIbEwFolCpaZEjnNtrdpJajugKOpfYeGvsDAAgg1+DW7pAC1POun5vTv57gUJZ0o7zqe1sLi0vLJaWLPXNza3tos7u9cqTiWFJo15LFsBUcBZBE3NNIdWIoGIgMNNMDgf+zd3IBWLoyudJdARpBexLqNEG8kv1su2FzKVcJIpnXHwVCr8nJ26o1uB732GT7F7jL1hSsLJ0+vBEDt2GfvFklNxJsDzxJ2SEpqi4Rc/vDCmqYBIU06UartOojs5kZpRDiPbSxUkhA5ID9qGRkSA6uSTC0f40Cgh7sbSVKTxRP09kROhVCYC0ymI7qtZbyz+57VT3a13chYlqYaI/nzUTTnWMR7HhUMmgWqeGUKoZGZXTPtEEqpNqLYJwZ09eZ40q5WTintZLZ3VpmkU0D46QEfIRTV0hi5QAzURRQ/oCb2gV+vRerberPef1gVrOrOH/sD6+gYWlaET</latexit>

{ C Ii, yD3I,

=

I = 1 , ... ,m

=

Weighted Majority Vote

7

Which models get more say in the weighted majority vote?

The models that performed better on the training data get more say in the vote

For our sequence of weak learners:

Boosted classifier defined by

The weight controls vote contribution of . Think of as measure of accuracy of

h1(x), h2(x), . . . , hK(x)<latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit><latexit sha1_base64="KmuhbOlzzpDHBoE6U+fHBMjY5eQ=">AAACHnicbVDLSsNAFJ3UV42vqEs3g61QoZSkC6u7ghvBTQVrC20Jk+mkGTrJhJmJWEL9Ejf+ihsXKoIr/RunbRBtPXDhzDn3MvceL2ZUKtv+MnJLyyura/l1c2Nza3vH2t27kTwRmDQxZ1y0PSQJoxFpKqoYaceCoNBjpOUNzyd+65YISXl0rUYx6YVoEFGfYqS05FonRTNwnVLa9Xx4Nz4uw/vArf5+dlmfK1mGgXv5IxehaxXsij0FXCRORgogQ8O1Prp9jpOQRAozJGXHsWPVS5FQFDMyNruJJDHCQzQgHU0jFBLZS6f3jeGRVvrQ50JXpOBU/T2RolDKUejpzhCpQM57E/E/r5Mo/7SX0ihOFInw7CM/YVBxOAkL9qkgWLGRJggLqneFOEACYaUjNXUIzvzJi6RZrZxVnKtqoV7L0siDA3AISsABNVAHF6ABmgCDB/AEXsCr8Wg8G2/G+6w1Z2Qz++APjM9vDQ2fdA==</latexit>

H(x) = sign

"KX

k=1

↵khk(x)

#

<latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit><latexit sha1_base64="YI891tToZdmhPJ3sylEthqlHFwI=">AAACSnicbVBNb9QwEHWWQkv46AJHLhZbpHJZJb0UDpUqcanEpZVYWmkdIsc72VixncieVI2i7O/jwqW3/gkuHKDigne7EtDypJGe3psZe15WK+kwiq6Cwb2N+w82tx6Gjx4/ebo9fPb8k6saK2AiKlXZs4w7UNLABCUqOKstcJ0pOM3K90v/9Bysk5X5iG0NieZzI3MpOHopHWY74dFux7KcXvRv6AFlCBdodefk3PSUKchxulhQNpOuVrx12CpgrtFpVx7E/ecPlHFVFzwtaZGWfxYtFszKeYFJuEPT4SgaRyvQuyRekxFZ4zgdXrJZJRoNBoXizk3jqMak4xalUNCHrHFQc1HyOUw9NVyDS7pVFj197ZUZzSvryyBdqX9PdFw71+rMd2qOhbvtLcX/edMG87dJJ03dIBhx81DeKIoVXQZLZ9KCQNV6woWV/q9UFNxygT7+0IcQ3z75Lpnsjd+N45O90eH+Oo0t8pK8IrskJvvkkByRYzIhgnwh38gP8jP4GnwProNfN62DYD3zgvyDwcZvGNeydw==</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

↵k<latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit><latexit sha1_base64="QiyFHT5MzMU1IklJTRnj++Msrrc=">AAAB83icbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0zgQuaWPdiw9+HuHgm58DtsLNTY+mfs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkZR5NRCIfA1RM8Jh5mmvBHlPJMAoE6wTjm7nfmTCpeBLf62nK/AiHMQ85RW0kv2b3UKQj7I/tGulXqk7dWYCsE7cgVSjQ7le+eoOEZhGLNRWoVNd1Uu3nKDWngs3sXqZYinSMQ9Y1NMaIKT9fHD0jF0YZkDCRpmJNFurviRwjpaZRYDoj1CO16s3F/7xupsMrP+dxmmkW0+WiMBNEJ2SeABlwyagWU0OQSm5uJXSEEqk2OdkmBHf15XXiNerXdfeuUW01izTKcAbncAkuNKEFt9AGDyg8wTO8wps1sV6sd+tj2VqyiplT+APr8wf0m5Br</latexit>

=

The Plan

8

o Look at example of popular Boosting method

o Unpack it for intuition

o Come back later and show the math

Usual binary classification assumptions:

o Training set

o Feature vectors

o Labels are

{(xi, yi)}mi=1<latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit><latexit sha1_base64="sqjdef+WkBOANhOscs3Ed4OjZyM=">AAACFnicbVDLSsNAFJ34rPEVdelmsBUqSEm6qS6EghuXFawtNDVMppN26OTBzI1YQv7Cjb/ixoWKW3Hn3zh9LLT1wIXDOfdy7z1+IrgC2/42lpZXVtfWCxvm5tb2zq61t3+r4lRS1qSxiGXbJ4oJHrEmcBCsnUhGQl+wlj+8HPuteyYVj6MbGCWsG5J+xANOCWjJsyol0xUsADfD5cz1A/yQe/wUjzyOT1zJ+wNwcy/jF05+F5ol7FlFu2JPgBeJMyNFNEPDs77cXkzTkEVABVGq49gJdDMigVPBctNNFUsIHZI+62gakZCpbjb5K8fHWunhIJa6IsAT9fdERkKlRqGvO0MCAzXvjcX/vE4KwVk341GSAovodFGQCgwxHoeEe1wyCmKkCaGS61sxHRBJKOgoTR2CM//yImlWK+cV57parNdmaRTQITpCZeSgGqqjK9RATUTRI3pGr+jNeDJejHfjY9q6ZMxmDtAfGJ8/Aqedfg==</latexit>

xi 2 Rp<latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit><latexit sha1_base64="K/IaIzQCvWEfm6x6GrBnNem7tRM=">AAACBnicbVA9T8MwEHX4LOErwIiELFokpirpUtgqsTAWRGilJkSO67RWHSeyHUQVdWPhr7AwAGLlN7Dxb3DaDNDypJOe3rvT3b0wZVQq2/42lpZXVtfWKxvm5tb2zq61t38rk0xg4uKEJaIbIkkY5cRVVDHSTQVBcchIJxxdFH7nnghJE36jxinxYzTgNKIYKS0F1lHNzL0wgg+TgEKPcujFSA3DML+e3KU1GFhVu25PAReJU5IqKNEOrC+vn+AsJlxhhqTsOXaq/BwJRTEjE9PLJEkRHqEB6WnKUUykn0//mMATrfRhlAhdXMGp+nsiR7GU4zjUncWVct4rxP+8XqaiMz+nPM0U4Xi2KMoYVAksQoF9KghWbKwJwoLqWyEeIoGw0tGZOgRn/uVF4jbq53XnqlFtNcs0KuAQHINT4IAmaIFL0AYuwOARPINX8GY8GS/Gu/Exa10yypkD8AfG5w9a25fh</latexit>

yi 2 {+1,�1}<latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit><latexit sha1_base64="wSYWNWE1VdElKwT63+784KrK8aA=">AAAB/nicbVBNS8NAEN3Urxq/ooIXL4utIKgl6aV6K3jxWMFYoQlhs920SzebsLsRQuzBv+LFg4pXf4c3/43bNgdtfTDweG+GmXlhyqhUtv1tVJaWV1bXquvmxubW9o61u3cnk0xg4uKEJeI+RJIwyomrqGLkPhUExSEj3XB0NfG7D0RImvBblafEj9GA04hipLQUWAd1Mw8o9CiHXnHqnMFzxxub9cCq2Q17CrhInJLUQIlOYH15/QRnMeEKMyRlz7FT5RdIKIoZGZteJkmK8AgNSE9TjmIi/WJ6/xgea6UPo0To4gpO1d8TBYqlzONQd8ZIDeW8NxH/83qZii78gvI0U4Tj2aIoY1AlcBIG7FNBsGK5JggLqm+FeIgEwkpHZuoQnPmXF4nbbFw2nJtmrd0q06iCQ3AEToADWqANrkEHuACDR/AMXsGb8WS8GO/Gx6y1YpQz++APjM8fFxSTFA==</latexit>

Adaboost Algorithm

9

1. Initialize training example weights to uniform distribution

2. For

3. Output H(x) = signhPK

k=1 ↵khk(x)i

<latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit>

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

k = 1, 2, . . . ,K :<latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit><latexit sha1_base64="RGAPVcQTsuavolgGhqL8ZCNpDfs=">AAACAHicbVBNS8NAEN34WeNX1IvgZbEVPJSS9FIVhIIXwUsFYwttKJvNpl262Q27G6GUevGvePGg4tWf4c1/47bNQVsfDDzem2FmXpgyqrTrfltLyyura+uFDXtza3tn19nbv1cik5j4WDAhWyFShFFOfE01I61UEpSEjDTDwdXEbz4Qqajgd3qYkiBBPU5jipE2Utc5LNkDeAm9MqyWYYdFQqsyvLkowa5TdCvuFHCReDkpghyNrvPViQTOEsI1ZkiptuemOhghqSlmZGx3MkVShAeoR9qGcpQQFYymH4zhiVEiGAtpims4VX9PjFCi1DAJTWeCdF/NexPxP6+d6fgsGFGeZppwPFsUZwxqASdxwIhKgjUbGoKwpOZWiPtIIqxNaLYJwZt/eZH41cp5xbutFuu1PI0COALH4BR4oAbq4Bo0gA8weATP4BW8WU/Wi/Vufcxal6x85gD8gfX5AzBakwo=</latexit>

a) Fit classifier to training data with weights

b) Compute weighted error

c) Compute vote weight

d) Update training example weights

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

errk =Pm

i=1 wi·I(yi 6=hk(xi))Pmi=1 wi

<latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit><latexit sha1_base64="DYudvI+ljGimk1yuXdxu0qV4n14=">AAACSHicbZBLbxMxFIU9KY8yvNJ2yeaKFCndRDPdFBaVKrGBXSsRWikTjMe501ixPYN9hzYazd9jw45d/wMbFgWxw0mzgIYjWTo6517Z/vJKK09JchV1Nu7cvXd/80H88NHjJ0+7W9vvfVk7iUNZ6tKd5cKjVhaHpEjjWeVQmFzjaT57vehPP6PzqrTvaF7h2IhzqwolBYWIdz/uxhnhJTnToHMtn8EhZIUTssl8bXijDtP2g4ELrjI5KQne9udcQWbxE0z5rN9keQGXLVd7e9CurbTxLvBuLxkkS8G6SVemx1Y65t1v2aSUtUFLUgvvR2lS0bgRjpTU2MZZ7bEScibOcRSsFQb9uFmSaOFFSCZQlC4cS7BM/95ohPF+bvIwaQRN/e1uEf6vG9VUvBw3ylY1oZU3FxW1BiphgRUmyqEkPQ9GSKfCW0FOReBIAX4cIKS3v7xuhvuDV4P0ZL93dLCiscmeseesz1J2wI7YG3bMhkyyL+w7u2Y/o6/Rj+hX9PtmtBOtdnbYP+p0/gDKcbEx</latexit>

↵k = 12 log((1� errk)/errk)

<latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit>

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

wi 1Zk

· wi exp [�↵kyihk(xi)]<latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit>

7

=( ADDING

MORE WEAK LEARNERS )

Adaboost Algorithm

10

1. Initialize training example weights to uniform distribution

Weights are initialized to uniform dist. At beginning, all training examples treated the same

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

Adaboost Algorithm

11

1. Initialize training example weights to uniform distribution

2a) Fit classifier to training data with weights

Decide Decision Stump split based on Impurity and Information gain as usual

Before: p was fraction of examples with positive label

Now: p is fraction of weights in pos class

wi =1

mfor i = 1, 2, . . . ,m

<latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit><latexit sha1_base64="oPRkYAeIMeueNJ5fkgd+uMoDQ2s=">AAACIHicbVC7TsNAEDzzxrwClDQrAhJFFNlpgAIJiYYSJAJIcRSdz2c4ceez7tZAZPlXaPgVGgpA0MHXcAkpeE01mtnVzk6cS2ExCN69sfGJyanpmVl/bn5hcam2vHJqdWEYbzMttTmPqeVSZLyNAiU/zw2nKpb8LL46GPhn19xYobMT7Oe8q+hFJlLBKDqpV9vZ8G96AvYgSlJDWRlWpaogQn6LRpWQagMViL2wAa0GRDLRaBug/A3o1epBMxgC/pJwROpkhKNe7S1KNCsUz5BJam0nDHLsltSgYJJXflRYnlN2RS94x9GMKm675fDDCjadkgzTpDpDGKrfN0qqrO2r2E0qipf2tzcQ//M6BaY73VJkeYE8Y1+H0kICahjUBYkwnKHsO0KZES4rsEvqmkJXqu9KCH+//Je0W83dZnjcqu9vj9qYIWtknWyRkGyTfXJIjkibMHJHHsgTefbuvUfvxXv9Gh3zRjur5Ae8j0/iVaBf</latexit>

hk(x)<latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit><latexit sha1_base64="3vQxizms4nIQmSBPqwBJtpLqCv8=">AAAB+nicbVBNT8JAEJ3iF9avikcvG8EEL6Tlgt5IvHjERIQEmma7bGHD9iO7WwNp+le8eFDj1V/izX/jAj0o+JJJXt6bycw8P+FMKtv+Nkpb2zu7e+V98+Dw6PjEOq08yjgVhHZJzGPR97GknEW0q5jitJ8IikOf054/vV34vScqJIujBzVPqBviccQCRrDSkmdVaubEm9azoR+gWX6FzBryrKrdsJdAm8QpSBUKdDzraziKSRrSSBGOpRw4dqLcDAvFCKe5OUwlTTCZ4jEdaBrhkEo3W96eo0utjFAQC12RQkv190SGQynnoa87Q6wmct1biP95g1QF127GoiRVNCKrRUHKkYrRIgg0YoISxeeaYCKYvhWRCRaYKB2XqUNw1l/eJN1m46bh3Der7VaRRhnO4QLq4EAL2nAHHegCgRk8wyu8GbnxYrwbH6vWklHMnMEfGJ8//1uR+Q==</latexit>

wi<latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit><latexit sha1_base64="oDiG3MzYIndo1l38OySeJflDm6w=">AAAB7nicbVBNT8JAEJ36ifUL9ehlI5h4Ii0X9EbixSMmVkigIdtlCxt2t3V3qyENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMyLUs608bxvZ219Y3Nru7Tj7u7tHxyWj47vdZIpQgOS8ER1IqwpZ5IGhhlOO6miWESctqPx9cxvP1KlWSLvzCSlocBDyWJGsLFSp+o+9ZlbRf1yxat5c6BV4hekAgVa/fJXb5CQTFBpCMdad30vNWGOlWGE06nbyzRNMRnjIe1aKrGgOszn907RuVUGKE6ULWnQXP09kWOh9UREtlNgM9LL3kz8z+tmJr4McybTzFBJFovijCOToNnzaMAUJYZPLMFEMXsrIiOsMDE2IteG4C+/vEqCeu2q5t/WK81GkUYJTuEMLsCHBjThBloQAAEOz/AKb86D8+K8Ox+L1jWnmDmBP3A+fwBB/45M</latexit>

I(D) = �p log2 p� (1� p) log2(1� p)<latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit><latexit sha1_base64="LA6upKk73lmL6XceCf7/jU3useA=">AAACEHicbZC9TsMwFIUdfkv4CzCyWLRI7dAq6VIYkCrBAFuRCK3URpXjOq1VJ7FsB6mK+gosvAoLAyBWRjbeBjfNAC1HsvT53Htl3+NzRqWy7W9jZXVtfWOzsGVu7+zu7VsHh/cyTgQmLo5ZLDo+koTRiLiKKkY6XBAU+oy0/fHlrN5+IELSOLpTE068EA0jGlCMlLb6Vrlk3pSvKvACVnmPxcN+ncMqLDtVXplfMzRLsG8V7ZqdCS6Dk0MR5Gr1ra/eIMZJSCKFGZKy69hceSkSimJGpmYvkYQjPEZD0tUYoZBIL802msJT7QxgEAt9IgUz9/dEikIpJ6GvO0OkRnKxNjP/q3UTFZx5KY14okiE5w8FCYMqhrN44IAKghWbaEBYUP1XiEdIIKx0iKYOwVlceRnceu285tzWi81GnkYBHIMTUAYOaIAmuAYt4AIMHsEzeAVvxpPxYrwbH/PWFSOfOQJ/ZHz+AH1Bl+8=</latexit>

p =

Pmi=1 wi · I(yi = +1)Pm

i=1 wi<latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit><latexit sha1_base64="9LEm29yj4UjovgL3uLk/JI0n8kM=">AAACLXicbVDLSsNAFJ3UV42vqks3g61QEUrSTXVRKIiguwrWFpoaJpNJO3QmCTMTpYR8kRt/RReCD9z6G04fC209MHA451zu3OPFjEplWW9Gbml5ZXUtv25ubG5t7xR2925llAhMWjhikeh4SBJGQ9JSVDHSiQVB3GOk7Q3Px377nghJo/BGjWLS46gf0oBipLTkFi5KZgzr0PEDgXDqyIS7Ka3b2R2HDy6FDvYjBa/KI83r8MQ+zhYymVmCbqFoVawJ4CKxZ6QIZmi6hRfHj3DCSagwQ1J2bStWvRQJRTEjmekkksQID1GfdDUNESeyl07OzeCRVnwYREK/UMGJ+nsiRVzKEfd0kiM1kPPeWPzP6yYqOO2lNIwTRUI8XRQkDKoIjruDPhUEKzbSBGFB9V8hHiBdnNINm7oEe/7kRdKqVs4q9nW12KjN2siDA3AIysAGNdAAl6AJWgCDR/AM3sGH8WS8Gp/G1zSaM2Yz++APjO8faK+l5Q==</latexit>

p =

Pmi=1 I(yi = +1)

m<latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit><latexit sha1_base64="2uC9p4I2Fqb43ljzO9gLpsbw4qI=">AAACE3icbVBNS8NAEN34WeNX1aOXxVaoFErSS/VQKHjRWwVrC20Nm82mXbqbhN2NEEJ+hBf/ihcPKl69ePPfuG1z0NYHA4/3ZpiZ50aMSmVZ38bK6tr6xmZhy9ze2d3bLx4c3skwFph0cMhC0XORJIwGpKOoYqQXCYK4y0jXnVxO/e4DEZKGwa1KIjLkaBRQn2KktOQUq2Uzgk048HyBcDqQMXdS2rSzew6vK4lDtVe1z7KUZ2XoFEtWzZoBLhM7JyWQo+0UvwZeiGNOAoUZkrJvW5EapkgoihnJzEEsSYTwBI1IX9MAcSKH6eypDJ5qxYN+KHQFCs7U3xMp4lIm3NWdHKmxXPSm4n9eP1b++TClQRQrEuD5Ij9mUIVwmhD0qCBYsUQThAXVt0I8RjoepXM0dQj24svLpFOvXdTsm3qp1cjTKIBjcAIqwAYN0AJXoA06AINH8AxewZvxZLwY78bHvHXFyGeOwB8Ynz/Vgpu2</latexit>

_

=

← pos pts=

0=←

Adaboost Algorithm

12

2b) Compute weighted errors:

o Still gives an error rate in [0, 1]

o Mistakes on highly weighted examples hurt more

o Mistakes on lowly weighted examples don’t hurt as much

errk =

Pmi=1 wi · I(yi 6= hk(xi))Pm

i=1 wi<latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit><latexit sha1_base64="//GA8Is51nY3KAR0Aj27RMeVcYM=">AAACSXicbZBLbxMxFIU9KY8yvAIs2VyRIqWbaKabtotKlbqBXZEIrZQJlsdzp7Fiewb7DjQaze/rhhU7fgQbFoBY4aRZQMORLB2dc69sf3mtlack+Rr1tm7dvnN3+158/8HDR4/7T56+81XjJI5lpSt3nguPWlkckyKN57VDYXKNZ/n8ZNmffUTnVWXf0qLGqREXVpVKCgoR74udOCO8JGdadK7jcziCrCidkG3mG8NbdZR27w184iqTRUXwerjgCjKLH2DG58M2y0u47Lja3YVuY6WLd4D3B8koWQk2Tbo2A7bWKe9/yYpKNgYtSS28n6RJTdNWOFJSYxdnjcdayLm4wEmwVhj003aFooOXISmgrFw4lmCV/r3RCuP9wuRh0gia+ZvdMvxfN2moPJi2ytYNoZXXF5WNBqpgyRUK5VCSXgQjpFPhrSBnInCkQD8OENKbX940473R4Sh9szc43l/T2GbP2Qs2ZCnbZ8fsFTtlYybZFfvGfrCf0efoe/Qr+n092ovWO8/YP+pt/QGsLrGf</latexit>

# gotis # wrong

=

Adaboost Algorithm

13

2c) Compute vote weight for the current weak learner

o Models with small weighted error get a larger vote

o Models with large weighted error get less vote (or, sometimes a negative vote!)

↵k = 12 log((1� errk)/errk)

<latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit><latexit sha1_base64="SfQna/o3NSXYOMTJbcHkpR5TUR0=">AAACKnicbVC7TsNAEDzzxrwClDQnAlIoCHaaQIGEoKEMEgGkOLLOl3VyyvmhuzUisvw/NPwKBRQQ0fIhXEIKCIy00tzMrm53glQKjY4ztGZm5+YXFpeW7ZXVtfWN0ubWjU4yxaHJE5mou4BpkCKGJgqUcJcqYFEg4TboX4z823tQWiTxNQ5SaEesG4tQcIZG8kvne7bHZNpjfp+eUi9UjOdukdcKTybdSsU99BAeUEU5KFX4/YOjqbe9R/1S2ak6Y9C/xJ2QMpmg4ZdevE7Cswhi5JJp3XKdFNs5Uyi4hML2Mg0p433WhZahMYtAt/PxrQXdN0qHhokyFSMdqz8nchZpPYgC0xkx7OlpbyT+57UyDI/buYjTDCHm3x+FmaSY0FFwtCMUcJQDQxhXwuxKeY+ZvNDEa5sQ3OmT/5JmrXpSda9q5bP6JI0lskN2SYW4pE7OyCVpkCbh5JE8kzfybj1Zr9bQ+vhunbEmM9vkF6zPL7a4plk=</latexit>

= -

×

×

× ja= - 0,5X

×

X 1

Adaboost Algorithm

14

2d) Update training example weights

o If example is misclassified then weight goes up

o If example is classified correctly then weight goes down

o How big of a jump up/down depends on how much vote the current learner has

o is just a normalizing constant to make it a probability distribution

o Don’t really have to compute , just scale weights so they sum to 1

wi 1Zk

· wi exp [�↵kyihk(xi)]<latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit><latexit sha1_base64="+Md+EfCAFodoBKQgI79wGNjvoIc=">AAACQXicbVA9b9QwGHbKVxu+rjCyWFyRysAp6VK6VWJhLBJHS89R9Mb35mLFiSP7De0pym9j4Rd06w9gYWgRaxd81xtoyyNZevR8yPaTNVo5iqLzYO3e/QcPH61vhI+fPH32fLD54oszrZU4lkYbe5SBQ61qHJMijUeNRagyjYdZ+WHhH35D65SpP9O8waSCWa1yJYG8lA6+boUnqeJCY05grTnhIrcgu7jvjtOy50JODXEfEXjaLFOTdwJ0U0Ba8rlvFmm53Yks56d9qt4Kq2YFJTzc4ulgGI2iJfhdEq/IkK1wkA7OxNTItsKapAbnJnHUUNKBJSU19qFoHTYgS5jhxNMaKnRJt5yg52+8MuW5sf7UxJfqv40OKufmVeaTFVDhbnsL8X/epKX8fdKpumkJa3l9Ud5qToYv9uRTZVGSnnsC0ir/Vi4L8BOSXz30I8S3v3yXjHdGe6P4085wf3e1xjp7xV6zbRazXbbPPrIDNmaSfWc/2QW7DH4Ev4LfwZ/r6Fqw6rxkNxBc/QWQoK+2</latexit>

Zk<latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit>

Zk<latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit><latexit sha1_base64="Bj3g7RvCGumNu7fhJbsHkjatqAI=">AAAB7nicbVBNT8JAEJ31E+sX6tHLRjDxRFou6I3Ei0dMrBChIdtlCxu227q7NSENf8KLBzVe/T3e/Dcu0IOCL5nk5b2ZzMwLU8G1cd1vtLa+sbm1Xdpxdvf2Dw7LR8f3OskUZT5NRKI6IdFMcMl8w41gnVQxEoeCtcPx9cxvPzGleSLvzCRlQUyGkkecEmOlTtV56I+dKu6XK27NnQOvEq8gFSjQ6pe/eoOEZjGThgqidddzUxPkRBlOBZs6vUyzlNAxGbKupZLETAf5/N4pPrfKAEeJsiUNnqu/J3ISaz2JQ9sZEzPSy95M/M/rZia6DHIu08wwSReLokxgk+DZ83jAFaNGTCwhVHF7K6Yjogg1NiLHhuAtv7xK/Hrtqubd1ivNRpFGCU7hDC7AgwY04QZa4AMFAc/wCm/oEb2gd/SxaF1DxcwJ/AH6/AEYo44x</latexit>

PREPlot .

L

- =a

- =

uol.net#Yane1strung #

Adaboost Algorithm

15

3) Algorithm produces the final boosted classifier

o Sum up weighted votes from each model

o Apply sign function so that it produces predictions in {+1,�1}<latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit><latexit sha1_base64="821zbZXswSZ6cqdHmYVyZcSBD4Q=">AAAB9HicbVBNT8JAEJ3iF9Yv1KOXjWBiopKWC3oj8eIREysktJLtssCG7bbZ3WpIw//w4kGNV3+MN/+NC/Sg4EsmeXlvJjPzwoQzpR3n2yqsrK6tbxQ37a3tnd290v7BvYpTSahHYh7LdogV5UxQTzPNaTuRFEchp61wdD31W49UKhaLOz1OaBDhgWB9RrA20kPF9rMz9/zC9Sd2BXVLZafqzICWiZuTMuRodktffi8maUSFJhwr1XGdRAcZlpoRTie2nyqaYDLCA9oxVOCIqiCbXT1BJ0bpoX4sTQmNZurviQxHSo2j0HRGWA/VojcV//M6qe5fBhkTSaqpIPNF/ZQjHaNpBKjHJCWajw3BRDJzKyJDLDHRJijbhOAuvrxMvFr1qure1sqNep5GEY7gGE7BhTo04Aaa4AEBCc/wCm/Wk/VivVsf89aClc8cwh9Ynz80ao/f</latexit>

H(x) = signhPK

k=1 ↵khk(x)i

<latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit><latexit sha1_base64="kdKSZawqyJ2SMSjAvU8WHIIVoU4=">AAACOXicbVBNa9wwFJTTtE3dptm2x15ENoXksth7SXoIBHoJ5JJAN7uwdoysfV4LS7KRnksW49/VS39Fbz30kkNbeu0fqPYD8rEZEAwzb3h6k1ZSWAyCH97Gk82nz55vvfBfvtp+vdN58/bSlrXhMOClLM0oZRak0DBAgRJGlQGmUgnDtPg094dfwFhR6s84qyBWbKpFJjhDJyWdiz3/dL+J0oxetwf0mEYI12hUY8VUtzSSkOGYRrZWSVMch+3VGY2YrHKWFDRPittkZMQ0x9jfo0mnG/SCBeg6CVekS1Y4Tzrfo0nJawUauWTWjsOgwrhhBgWX0PpRbaFivGBTGDuqmQIbN4vTW/rBKROalcY9jXSh3k00TFk7U6mbVAxz+9Cbi4954xqzo7gRuqoRNF8uympJsaTzHulEGOAoZ44wboT7K+U5M4yja9t3JYQPT14ng37vYy+86HdPDldtbJH3ZJfsk5AckhNySs7JgHDylfwkv8hv75t34/3x/i5HN7xV5h25B+/ff/X7q1E=</latexit>

-

Adaboost Example

16

Example: Suppose you have the following training data

Adaboost Example

17

o First Decision Stump:

0°0

Adaboost Example

18

o First Decision Stump:

err1 = 0.30↵1 = 0.42

<latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit>

=

=

=

Adaboost Example

19

o First Decision Stump:

err1 = 0.30↵1 = 0.42

<latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit><latexit sha1_base64="VfKIA+emCuEy8V1BOGlSXIJ9ZDg=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTFxYVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9EK+d7zHg/pBwKPIj0BlI/gnfm34Eep+6fZ3eK9aC+rBDPwtCUtSYyVavept1E9FnqAmocDaThhk1HUDSQqFbpfcYgbiAobYcVRDgrZbzC6f8F2n9PkgNe5p4jP1f0cBibXjJHadCdDIvq5NxfdqnZwGh91C6iwn1GL+0SBXnFI+jZH3pUFBauwICCPdrlyMwIAgF7bvQghfn/yWtBv17/XwR6PWPCrTqLDP7AvbZyH7xprslLVYmwl2ze7ZI3vybrwH79n7PW9d8ErPNnsB789fmV6pPA==</latexit>

Adaboost Example

20

o Second Decision Stump:

=

=

=

-

:-

Adaboost Example

21

o Second Decision Stump:

err2 = 0.21↵2 = 0.65

<latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit>

.

-

÷

Adaboost Example

22

o Second Decision Stump:

err2 = 0.21↵2 = 0.65

<latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit><latexit sha1_base64="pxGzwmjh9bwZluY3uPjjFj+47E4=">AAACOHicbVBNSyNBEO3RVbPjV1aPe2k2Kp7CTMBdVxACXjxmYbMKmRBqOpWksadn6K4Rw5C/5cV/4U3w4kFlr/sLtpMMy/rxoOHxql511YszJS0FwZ23sPhhaXml8tFfXVvf2Kx+2vpl09wIbItUpeY8BotKamyTJIXnmUFIYoVn8cXJtH52icbKVP+kcYbdBIZaDqQAclKv2trxoxiHUhdgDIwnhRFq4keEV2SSAo2Z9Bp873iPB/VGyKPIj0BlI/gnfj3wI9T90u3v8F61FtSDGfhbEpakxkq0etXbqJ+KPEFNQoG1nTDIqOsGkhQK3S65xQzEBQyx46iGBG23mF0+4btO6fNBatzTxGfq/44CEmvHSew6E6CRfV2biu/VOjkNDruF1FlOqMX8o0GuOKV8GiPvS4OC1NgREEa6XbkYgQFBLmzfhRC+PvktaTfq3+vhj0ateVSmUWGf2Re2z0L2jTXZKWuxNhPsmt2zR/bk3XgP3rP3e9664JWebfYC3p+/pISpQw==</latexit>

Adaboost Example

23

o Third Decision Stump:

Adaboost Example

24

o Third Decision Stump:

err3 = 0.14↵3 = 0.92

<latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit><latexit sha1_base64="aCuZFloAeoN85IH1WHegez00ejU=">AAACOHicbVBNa9tAEF2laeuqH3GbYy5L7JqejOQE2gQChlxydCFODJYxo/XYXrxaid1RqRH+W7nkX/RWyCWHNuTaX9C1I0Jq98HC48282ZkXZ0paCoKf3taz7ecvXlZe+a/fvH23U33/4cKmuRHYFalKTS8Gi0pq7JIkhb3MICSxwst4drqsX35DY2Wqz2me4SCBiZZjKYCcNKx26n4U40TqAoyB+aIwQi38iPA7maRAYxbDA944afCgGR7yKPIjUNkUHsWjlh+hHpVuv86H1VrQDFbgmyQsSY2V6AyrP6JRKvIENQkF1vbDIKOBG0hSKHS75BYzEDOYYN9RDQnaQbG6fME/OmXEx6lxTxNfqU8dBSTWzpPYdSZAU7teW4r/q/VzGn8ZFFJnOaEWDx+Nc8Up5csY+UgaFKTmjoAw0u3KxRQMCHJh+y6EcP3kTdJtNY+a4ddWrX1cplFhe2yffWIh+8za7Ix1WJcJdsVu2C/227v2br077/6hdcsrPbvsH3h//gKrH6lH</latexit>

Example: Boosted Classifier

25

0.42 + 0.65 + 0.92

=

" "a 1 1+1

-

'- -1

--1 L

42C1) t .65( 1) f. 921 D

×x

.yzf , ) + .

651 - 1) + r{2( D

= 7 0= -1.07+0-92=

- g.15<0⇒ +

Performance

26

Recall the standard experiment of measuring train and validation error vs model complexity

Once overfitting begins, validation error goes up

x

Performance

27

Boosting has a remarkable affect. Think of number of weak learners as model complexity

Even after training error becomes zero, validation error continues to improve!

tie

Performance

28

HOW CAN THIS BE?!

Boosting as Margin Maximizing Algo

29

Adaboost actually induces a margin, and tries to maximize it

Despite the fact that we make the model more complex with each weak learner, the model actually becomes more confident with each boosting iteration

This confidence can be represented as a margin

Our learn classifier with K weak learners is

Define an intermediate classifier by where

H(x) = signh PK

k=1 ↵khk(x)i

<latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit><latexit sha1_base64="FEylyjCW/EITZelDkNS1QMOHjQU=">AAACPXicbVA9a9xAFFw5sWPLX5e4dLPkbLCbQ7rGSWEwpDGkcSCKDSdZrPaeTot2V2L3KfgQuj+WJr8hXco0LpKQNm32zgdJbA8sDDNvePsmq6WwGARfvZUnT1fXnq1v+Jtb2zu7vecvPtiqMRwiXsnKXGXMghQaIhQo4ao2wFQm4TIr38z9y49grKj0e5zWkCg20SIXnKGT0l504J8ftXGW05vumJ7SGOEGjWqtmOiOxhJyHM1mNLaNStvyNOyu39KYybpgaUmLtPybnc1iIyYFJv4BTXv9YBAsQB+ScEn6ZImLtPclHle8UaCRS2btKAxqTFpmUHAJnR83FmrGSzaBkaOaKbBJuzi/o4dOGdO8Mu5ppAv130TLlLVTlblJxbCw9725+Jg3ajB/lbRC1w2C5neL8kZSrOi8SzoWBjjKqSOMG+H+SnnBDOPoGvddCeH9kx+SaDh4PQjfDftnJ8s21sk+eUmOSEhOyBk5JxckIpx8It/Id/LD++zdej+9X3ejK94ys0f+g/f7Dwo8rXE=</latexit>

H`(x) = signh P`

k=1 ↵khk(x)i

<latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit>

` K<latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit><latexit sha1_base64="XRmmFao92YqeU8l4WDesJQTMhL4=">AAAB+HicbVBNT8JAEN3iF9avqkcvG8HEE2m5oDcSLyZeMBEhoQ3ZLlPYsN3W3S0JafgnXjyo8epP8ea/cYEeFHzJJC/vzWRmXphyprTrfluljc2t7Z3yrr23f3B45ByfPKokkxTaNOGJ7IZEAWcC2pppDt1UAolDDp1wfDP3OxOQiiXiQU9TCGIyFCxilGgj9R2navvAOfY5POE7u4r7TsWtuQvgdeIVpIIKtPrOlz9IaBaD0JQTpXqem+ogJ1IzymFm+5mClNAxGULPUEFiUEG+uHyGL4wywFEiTQmNF+rviZzESk3j0HTGRI/UqjcX//N6mY6ugpyJNNMg6HJRlHGsEzyPAQ+YBKr51BBCJTO3YjoiklBtwrJNCN7qy+ukXa9d17z7eqXZKNIoozN0ji6RhxqoiW5RC7URRRP0jF7Rm5VbL9a79bFsLVnFzCn6A+vzB8lakVA=</latexit>

=

-

Boosting as Margin Maximizing Algo

30

Note that the classifier only returns +1 or -1, which doesn’t give us any measure of confidence

Define the normalized voting coefficients as

Define the margin of a training example after boosting iterations as

H`(x) = signh P`

k=1 ↵khk(x)i

<latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit><latexit sha1_base64="umnZDpFWX+FMRSuW0+Ga8B7yjrU=">AAACRXicbVBNb9QwFHRKCyUUWOiRi9UtUrmskl5KD5UqcemxlVhaaR0ix/uysWI7kf1SdRVl/xwXztz4CVx6aCuu4N2uRD8Y6UmjmTey32S1kg6j6Gew8mR17emz9efhi42Xr1733rz94qrGChiKSlX2LOMOlDQwRIkKzmoLXGcKTrPy09w/PQfrZGU+47SGRPOJkbkUHL2U9pLt8ChloNROy7KcXnQf6AFlCBdodevkxHSUKchxNJtR5hqdtuVB3H2dJyjjqi54WtIiLf/FZzNm5aTAJNymaa8fDaIF6GMSL0mfLHGc9n6wcSUaDQaF4s6N4qjGpOUWpVDQhaxxUHNR8gmMPDVcg0vaRQ0dfe+VMc0r68cgXah3Ey3Xzk115jc1x8I99Obi/7xRg/nHpJWmbhCMuH0obxTFis47pWNpQaCaesKFlf6vVBTccoG++dCXED88+TEZ7g72B/HJbv9wb9nGOnlHtsgOickeOSRH5JgMiSDfyC9yRa6D78FlcBP8vl1dCZaZTXIPwZ+/PNmxBw==</latexit>

↵̂k =↵kPKj=1 ↵j

<latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit><latexit sha1_base64="SZTz11lgK5a4CnIhcLXNBB/iTKk=">AAACJXicbZDLSsNAFIYnXmu8VV26GWwFVyXpprooFNwIbipYW2hqOJlO2mknF2YmQgl5Gje+ihsXVQRXvorTNgtt/WHg5zvncOb8XsyZVJb1Zaytb2xubRd2zN29/YPD4tHxg4wSQWiLRDwSHQ8k5SykLcUUp51YUAg8Ttve+HpWbz9RIVkU3qtJTHsBDELmMwJKI7dYL5vOEFTqAI+HkLljXMdO3xdAcuSOs9SRSeCmo7qdPd7iHI8ys4zdYsmqWHPhVWPnpoRyNd3i1OlHJAloqAgHKbu2FateCkIxwmlmOomkMZAxDGhX2xACKnvp/MwMn2vSx34k9AsVntPfEykEUk4CT3cGoIZyuTaD/9W6ifIveykL40TRkCwW+QnHKsKzzHCfCUoUn2gDRDD9V0yGoDNSOllTh2Avn7xqWtXKVcW+q5YatTyNAjpFZ+gC2aiGGugGNVELEfSMXtEUvRsvxpvxYXwuWteMfOYE/ZHx/QOvE6TW</latexit>

`<latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit><latexit sha1_base64="7U7CjYA21V1TBNVh7eD2hhgBsyA=">AAAB7nicbVA9TwJBEJ3DLzy/UEubjWBiRe5o0I7ExhITT0jgQvaWOdiw9+Hungm58CdsLNTY+nvs/DcucIWCL5nk5b2ZzMwLUsGVdpxvq7SxubW9U9619/YPDo8qxycPKskkQ48lIpHdgCoUPEZPcy2wm0qkUSCwE0xu5n7nCaXiSXyvpyn6ER3FPOSMaiN1a3YfhaiRQaXq1J0FyDpxC1KFAu1B5as/TFgWYayZoEr1XCfVfk6l5kzgzO5nClPKJnSEPUNjGqHy88W9M3JhlCEJE2kq1mSh/p7IaaTUNApMZ0T1WK16c/E/r5fp8MrPeZxmGmO2XBRmguiEzJ8nQy6RaTE1hDLJza2EjamkTJuIbBOCu/ryOvEa9eu6e9eotppFGmU4g3O4BBea0IJbaIMHDAQ8wyu8WY/Wi/VufSxbS1Yxcwp/YH3+ALtNjpw=</latexit>

margin`(x) =P`

k=1: hk(x)=y ↵̂k �P`

k=1: hk(x) 6=y ↵̂k<latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit>

=

=

g- ==

=

Boosting as Margin Maximizing Algo

31

Handwritten Digit Recognition with Adaboost

Note that Adaboost achieves zero training error after roughly 250 iterations

Boosting as Margin Maximizing Algo

32

Handwritten Digit Recognition with Adaboost. Two easy and two hard training examples.

margin`(x) =P`

k=1: hk(x)=y ↵̂k �P`

k=1: hk(x) 6=y ↵̂k<latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit><latexit sha1_base64="ypH/YlW7x4zWU0a1ByjHvi5W4bk=">AAACc3icdVFNb9QwFHRSPsrytS0HDnAw3UUqh66SXigVlSpx4VgkllbaLNGL92VjxXaC/YK6itI/wL/jxs/gwhnvdg+lpSNZGs28ke15Wa2koyj6FYQbd+7eu7/5oPfw0eMnT/tb219c1ViBY1Gpyp5l4FBJg2OSpPCstgg6U3ialR+W/ul3tE5W5jMtapxqmBuZSwHkpbT/Y9hLCM/J6laDnUvTpQkqxXfbJMv5efeGH/HENTpty6P48KJIy6vOovu6mk4KoDYBVRfQpSXfuzWSGPx2S2rI0/4gGkUr8JskXpMBW+Mk7f9MZpVoNBoSCpybxFFN0xYsSaGw6yWNwxpECXOceGpAo5u2q9I6/torM55X1h9DfKVeTbSgnVvozE9qoMJd95bi/7xJQ/nBtJWmbgiNuLwobxSnii83wGfSoiC18ASElf6tXBRgQZDfU8+XEF//8k0y3h+9G8Wf9gfH79dtbLIXbIftspi9ZcfsIzthYybY7+B5wINXwZ/wZbgTDi9Hw2Cdecb+Qbj3F5zyvHI=</latexit>

ownnunnmrnmmn

×

Practical Advantages of Boosting

33

o It’s fast!

o Simple and easy to program (you’ll find out on Monday cuz we’re gonna do it!)

o No parameters to really tune, except K

o Flexible, can choose any weak learner

o Shift in mindset: Now can look for weak classifiers instead of strong classifiers

o Can be used in lots of settings: text learning, images, discrete features, continuous features)

34

35

36

top related