overcast - academics | wpiweb.cs.wpi.edu/~cs4341/a17/lecturenotes/slides/mitchellsbook/ch3.pdf · (...

28

Upload: lamcong

Post on 19-Jul-2018

226 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� ����� � ������� ��� �

������� � !�"#�%$�&'�(� )+*�,�-��.0/21 1 �(3#�4�5� �768�9�:.0;=<>��< )�?A@2BC)4?EDF*

GIH ��.0;=<>;J/ 3 &K�-�(� �-�($��-��<>�938&-�%&';J/23GILMH ) NJ���%�-3�;J3PO �%NQO2/2�-;J&K"�1GIR 3S&K��/ $8T2B L 3�UV/2��1 �%&K;J/23 OW�X;J3G Y Z �(�K[#&K&K;J3�O

\A] ^`_babcEdfe�_7gh^`i`jf_bg'kmlAe>ch_bnAcpoflqlArtsXuJvbwqxzy|{~}�{puJ�py�x�yJ�A�J���f�Xi�c�a��q_b^`^����Xa���eh�����'i`^`^m�����J�J�

Page 2: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� ����� ���� � ������� ����� ��!

Outlook

Overcast

Humidity

NormalHigh

No Yes

Wind

Strong Weak

No Yes

Yes

RainSunny

"$# %'&)()*,+.-/&102%'3'4.&)06587$-9*2&):$*<;.7=7$>@?BADC)E=FHGJILKMI<ADN<GOFPGDQ$RDSUT.VB3W*X(ZY=&)%'%[ROVB(]\�-2^`_ba63'%'%8Rdc`eDeD#

Page 3: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� �������

� ������������� �� !�"� #�%$� #�&���%�'�)(*�������,+-/. (��10 ���2���% #�3(*4��5�267� ���&���#� (8� (8���� #��9�:����-/. (��10 9� ;(*�3�10 ���< ! #�&��$=���36��>��� (*���? !��95:���� @A(*4�:��-/. (��10 4���(CBD���26�� (<�!����E��3� ( �F4�(��#�G�IHJ�%(*�������

K �CL LM�<:�4�6 LM� !�%$5 !�&���%�2�%+- NPORQPO5S T U- VXW N Y Z�Q V\[ N ]_^ N ` Z- a �8Bcb

dfe gihkjklnmpoqhFrsgitiuphkr?vxwfoGlshkyfl{zpw�wf|~}8���k���������X�{���{�������f���;�p�8t�l�j���hkgig����8j��Ros�����?tigigx�������� 

Page 4: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ��� ��� � ��� ��� �� � ���������� � �����

���������! "��#%$&� '�$&�!#)(!*,+� "+�-,$ +/. "���0(1*,+�23��$54768 9-:23$ ;< "*,(=��?>� "(1@A$B��CD23��#%��*,EF� *G� '�*G�1#)(!$B��$ 68 "-,23$&'��H *I�KJL23�<#)��*,6F$ MN.O;PEA��M3$&��*I� Q R. +P$ (!$&ST23*,(!$&'��UVE7�1��*,+3-,. ��EF*I��. ��(W 9*,�3*,�3@ '3 "�!

XZY "Q ;3-,$R�\[� X ST23*];3Q $B�/�^EF(_Q $&'�*I#B "-`'�*I "@F�3E7�a*I�� bc(!$&'�*]� (1*G�1d "�< "-,./��*I��?e E/'�$f-,*,�3@ #B "-,$f��'3 "( �!#gM�$&'�23-,*]�3@ ;3(!$%CD$f(1$B�<#)$&�

h:i jlknmnoqpsrtk%uvjlwlxsknu0y{z:raovkn|:o~}szIz:���9�,�n�I���������~�,�~�����,�:�,�=�s�9w�o�m��Iknjlj����9m��5rv��� �0wljlj{�¢¡�i,i,£

Page 5: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ������� �� ��� ����������� ��� � ������������ � �������

� "!$# %$&�&('�)*,+�- . /102 35462871/:9 ;<28= ! 7 !$&(# /1/1> ! 4?@/12 A & > # 2CB�/ DFEHG�IJ�+�K 7L7 !$M(# - 7 ;N28= ! 7 !$&(# /1/1> ! 4?/O2 A & > DFEHG�IP<+RQ & >S2 =T0 U "% ?2 & AV- WX=Y>:2 /12 # 2[Z ;N287:=Y2 # ; "# / & A

D�EHG�I\ +(] & >:/^/O> "!_#!$#M 2`B "a '% 2b7^/ & % 2 A #& ;<287c�+RdeA�/1> "!$#!_#M 2CB "a '% 287 ' 2�>OAf28=Y/ %$g = %h 7L7 !ji 2b;kWmln02 #

]�l o pVW<q % 7r2 ! /O2�> /12 & U(2�> # 2�Z % 2 A #& ;<287

s 0 ! =T0 /O/1> ! 4?/12 ! 7t462871/vu

A1=? A2=?

ft ft

[29+,35-] [29+,35-]

[21+,5-] [8+,30-] [18+,33-] [11+,2-]

w$x y{z}|}~�����zC��y{�{��z}�O�����r~�z}��~��������R�H�$�}�������v�����$�������$���$ ¢¡�£H�¤~¥|§¦�z}y{y¨��£H|ª©���«­¬¯®O�{y{y��e°­±$±$²

Page 6: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ���������

Ent

ropy

(S)

1.0

0.5

0.0 0.5 1.0p+

� � �� � ����� ����� ��������� �!� �!�" �$#%��� �����&��('*) ��+�-,�� �����.�/�0���- ��0! � ���/�1�2 ��- �30� �$#%��� ���4�&�+ 4! ��('*5 ��+�-,�� �����.�/�0���- ��0! � ��!��6"7���� 430� �$#%��� �����&� �! ��98:!;�����.�=< � �>�0�2?����&���,�� �� ��?��� 4�@< � �:�

A BDCFE G '�HJIK�MLON PQ'*)R���.";S>'*) P '*5T���0";S>'U5

VXW Y[Z]\]^`_XabZ$cdY[e[fXZ]c-gihja2^dZ]kj^mlXh�hjnpo q�r]s�tvuxwzy{wmq�|mu}t~u��j�����X� e�^�\���Z]Y[Y��}� \���ad�����-e[Y[Yi�FW������

Page 7: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ���������

� �� ���������������� ���! "�$#&%'�)( *,+�- ."�0/�1324.�56%879*��:�)(;�)( %'1�:*<#&1,(;� #&=?>@7A7 ��B 1@/ C �D1324/E>3*<(;1@- =6F (;/�>$G�*- �:- ."�0/132H� �I+�*<(;�0/J%LK�� 1@ �%'5M- >N=POQ7RK�1S/A%'�$7R%UTV=M�:*�W@%'K#&1,(;�X�

Y K,F�Z[\*]2I1S/A- >3%'5M1@* %'K��01@/AF"^_1@ �%'5M- >N=�=M�:*�W@%LK #&1,(;� >S7A7'5MW@*<7

` =M1@W,aX� .�5M%b79%'1 - �$7A78>NW@� K<>$cd5M*�W �/[email protected]>N.�5M=M56%fF ��g

h 1�O]���! "�$#&%'�)( *!+�- ."�:/�1�24.�5M%87 %'1 �:*<#&1,(]� B 1@/ C 132/�>N*<(;1@- - �:- ."�:/1�2H�^

�eij� ` =M1SW,a$�<ik�ml �enj� ` =M1SW,a$�<nk�

� �� ��������������_o ` �<iJ=61@W,aX�<i ` �enJ=M1@W,ap�<n

qMr sutwvwxzy|{}t�~�su�u�|tw~L����{Rx�tw��x��|�����p�3�M�w���������P���M�������M���M���|�3��x�v¡ �twsus¢���3v¤£¥{�¦¨§ª©L�usus��V«¨¬M¬M­

Page 8: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

��� ������ ��� ���� � � ��

� ����������� � !�" #%$'&(#*),+-#/. 01#/.32),+-46587 497 #:7'+;0158&=< .32�# +-5> 580 +-497@? 5A7 �

� �����B�C���D� !�E F �HG�IKJML�NO�C��!QP RS TVU3W�XZY []\_^a`=b

c � S cc � c F �dG�IKJML�NO�C� S !

A1=? A2=?

ft ft

[29+,35-] [29+,35-]

[21+,5-] [8+,30-] [18+,33-] [11+,2-]

e9f gihkjklZmonph%qrgisitohkq;uwvxnylrhkzxla{ov|vx}�~K�9�k�|�����M���a�9�a�����9�x�9���o�Ks�l�j��|hkgig����Kj���nr�����;sigigw���� 9 9¡

Page 9: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ������� ��� � ��� � ����

� ��� � ����������� "!$# %&!$'(�)����'*! + ��# ,.-/,���� 0 ,�12- 34�.�5�6 "!$1�17,.8� 9 :��71�1;� + �<� + ,�=<> 0 !?�@� A �� B :��71�1;� + �<� + ,�=<> :���'C�<17= A �� D � E<!$'(F$�<8�� + �<� + ,�=<> 0 !?�@� G�!58� H I �J,�1 K ,L�.- + ,�=<> 0 !?�@� G�!58� M I �J,�1 N4���<� A �<'*# �)� 0 !?�@� G�!58� O I �J,�1 N4���<� A �<'*# �)� :���'C�<17= A �� P � E<!$'(F$�<8�� N4���<� A �<'*# �)� :���'C�<17= G�!58� Q :��71�1;� K ,L�.- + ,�=<> 0 !?�@� A �� R :��71�1;� N4���<� A �<'*# �)� 0 !?�@� G�!58� 95S I �J,�1 K ,L�.- A �<'*# �)� 0 !?�@� G�!58� 9@9 :��71�1;� K ,L�.- A �<'*# �)� :���'C�<17= G�!58� 9TB � E<!$'(F$�<8�� K ,L�.- + ,�=<> :���'C�<17= G�!58� 95D � E<!$'(F$�<8�� + �<� A �<'*# �)� 0 !?�@� G�!58� 9UH I �J,�1 K ,L�.- + ,�=<> :���'C�<17= A �

VXW Y[Z]\]^`_bacZedfY[g[hbZ]djilkman^fZ]om^qpbk.kmr�sJt�u]v.wyx{z}|~zqt��qx�w�x��m���(�b�Jg�^�\��.Z]Y[Y����J\���af�����jg[Y[Yl���������

Page 10: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

����������� � � ��� � � ��� � � ������ � � ���

Which attribute is the best classifier?

High Normal

Humidity

[3+,4-] [6+,1-]

Wind

Weak Strong

[6+,2-] [3+,3-]

= .940 - (7/14).985 - (7/14).592 = .151

= .940 - (8/14).811 - (6/14)1.0 = .048

Gain (S, Humidity ) Gain (S, )Wind

=0.940E =0.940E

=0.811E=0.592E=0.985E =1.00E

[9+,5-]S:[9+,5-]S:

��� ��� � !#"%$&�(')��*�+%� '-,/.0$1!)� 20!43%.5.0687:9�; <5=?>A@CBD@49�E4>F=G>�H0I�JLK%M:*N!O�QP5� ���RIFM:�TSU$)VXWZY-*����/I\[X]�]�^

Page 11: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

Outlook

Sunny Overcast Rain

[9+,5−]

{D1,D2,D8,D9,D11} {D3,D7,D12,D13} {D4,D5,D6,D10,D14}

[2+,3−] [4+,0−] [3+,2−]

Yes

{D1, D2, ..., D14}

? ?

Which attribute should be tested here?

Ssunny = {D1,D2,D8,D9,D11}

Gain (Ssunny , Humidity)

sunnyGain (S , Temperature) = .970 − (2/5) 0.0 − (2/5) 1.0 − (1/5) 0.0 = .570

Gain (S sunny , Wind) = .970 − (2/5) 1.0 − (3/5) .918 = .019

= .970 − (3/5) 0.0 − (2/5) 0.0 = .970

��� ��������� �������������������������������������! �"�#�$&%('*)+'� �,�%-$.%�/�0�132�4!�5�6�87������90-4!�;:<��=?>A@�������0CB?D�D�E

Page 12: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� � � ����� ��� � � ����� ����������� ��� ��� �

...

+ + +

A1

+ – + –

A2

A3+

...

+ – + –

A2

A4–

+ – + –

A2

+ – +

... ...

� � !#"%$%&('*)+"-,.!#/#0*"%,21435)6&."%75&98*3:35;=<?> @%A:BDCFEHGIE9> J9CKBLC M5N OQP*R?/S&T$VU:"%!#!WNKR?$YXZ).[]\_^2/#!#!4Na`]b b �

Page 13: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� � � ����� ��� � � ����� ����������� ��� ��� �

��� �!�"$#&%�')(&*+( (,!.-$/0' *+( /0"$1 !�23'4#&'$56 7 -98;:$'4#�<>=@?A/0#&*B"C? (&=�8;'42B *B? #&%�'D8E'$F+F+F

� G =@#&!�=�#;( - (&*B?�:C2B' %H �!�"C#&%�')(&*+( I>J�%�*+/K% "$?�'MLON6 P -9?RQS#�!�2+-T UWV XY=@')(,#Z*B"$?A(4F[F+F

��\ " ]A-$/_^ #Z8`-$/_^Y*B?�:6 a "b/4-92R1 *3?�*B1 -cF+F+F

� d�#;-9#&*e(,*+/D-92[2+ gfh]A-$(&')i (&')-98j/K% /`%�"$*+/0'T(6 k "$]@=A(,#�#&" ?�"$*+(& i@-9#;-cF+F[F

��lm?Aic=./0#&*Bn$' ]�*+-$(4op-9!�!@8E"rq st!�8;'u<>'D8 (,%@"$8E#Z')(,#�#&8;'4')v

wBx y{z}|}~�����zu��y{�{��z}�Z���[�,~�z}�[~S���+�[�����B�}�+���������S�B�S�����B�[ B¡j¢�£��¤~¥|§¦+z}y{y¨ �£�|ª©_��«­¬¯®Z�{y{y� h°­±B±B²

Page 14: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

��� � � ����� � � ����� ��� ��� �

� ����� � ��� ��� � !"�$# �&% �'�&�(�*)+�-,.���0/1,.23�4�65

7 8 ,:9 ��/����<;>=� ���?%0�</1@-@-ACBDB�B

EGF %0�H)I�&%J�K,>23� )I��% ��� ��%0�L�M%J�K�4�ONP/1,.; )I��% �M� �Q��� #?�-�M�� �-R�� �-,S)T��%0U /1���-�C, RQ/1�-, /1�M��%J�V9 W ���<� ,�</1%?��� � %0�����

EGX ��/��?��� / Y"Z0[]\3[KZ0[K^"_`[a)T��% �'��U � �bA�!"�C��� �<���<�KNc%d/1��� �&%���./1, / Z0[HeHfgZ`hi_<fjhik1^ �l)+�:A�!"����� �<����� �'!>/�23� �

E m 2&2K/1U no�L%p/1q&��%srt! %0�H)T�K%?��� � ��� ��%0���<���?�bA�!"������<�'�u����./1�Gvc�0�?��� � ; /l�0/

w-x y{z}|}~�����zH��y{�{��z}�M���D�'~�z}�D~������D���l�-�}�������`�g���-�������-�D�-�d �¡l�¢~£|¥¤�z}y{y¦��¡l|¨§©��ª¬«®­M�{y{y��°¯¬x-x-±

Page 15: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ������� �� ���� ��

� ��� ����������� ����� ��!"���#�$� !����%�&�'�)(* ��+ ,�- ��.�!�/0. �2143 � �'5

687 ��9��:� �&�;� �<!8�=�#�>��?@!��>1A. B0�C.�+ �=�#�D�E?F 1 ����� ��!G���#� !��>1A!8HI!�� J�1A!�1 ,�.�BK/MLA��B0� !�� N$�

O � /0. O /MJP��. O �F 1 BK� .�+ �=�#� !��>1A!QH>!�� J�1A!R1 - /0+ �=!8N$� O � /K. O /MJP�:. O �

* ��+ ,�- ��.�!�� ���$�S���'JT5

6VU �;���<� 1A��� - 1A.=� 981%���G!�� JP��HI.�� ��- 1ABKB�����!��G�W�X�=�#�>�

6 � ?Y+;?[Z>1WB0B�!������'�"9Q/0!\� 1 ���</K- � .=,�- N$�����W�].���JP�%�"!��>1A!,>�&� 1A!\!��</KN�,�!��'� N$��+ /K.�.�/0.�+ 9Q/0!�� ^`_ba

6 � �>1A!�cd� ��� �&�$� O /M1ABe1AN$� ,�! ��- 1ABKB�����!��QN>1 ���'J � . f)gMh'i�W�]�=�#�$� !\���'��/M�j(k(

l0m nporqrsYtvuwo�xynpzp{vorx\|~}[u&syor�[s��v}M}[�4�W�0�r�M�����)�2���0�������0�[�0�j�v�Wz�s�q��Mornpn����Wq���uy���¡ \zpnpn~�£¢�¤0¤0¥

Page 16: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ������� ��� � � � � ���� ������ � �������

�������! #"%$'& (�"�"% )��* ���� #�!+ ,!&-(. )�� /��* $10�(.2 3�4)$ 5 687:9

;�<�=>=@?BA C DFEGA H DJIJK L:M�A ;NEOIPDJ=RQBA S MTLU?WV XY=>=>Z [ \ H D] ^ (F, $1_R$a`b, ��� $a(F&c4) /$'& ,!&d$'$Je

Outlook

Overcast

Humidity

NormalHigh

No Yes

Wind

Strong Weak

No Yes

Yes

RainSunny

fhg ikjmlmnpohqrj1stikukvhjmsxwzy{q|ntjm}{n�~hy#y{�8�F�)�m�#�����������)�������)�{�)�-�h�Fu�n�l��#jmiki����Fl���qt���¡ xukikiz�¢g�£)£)¤

Page 17: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ������� ��� � �

��������������� ��������� �! #"%$'&(��)�"*�+�,�-� . �0/����1 )��324�5���5��6 ��27)�2�8:9<;=;4>=;+?A@CB DFE!GC.IH1 ���%)��J��� �����,)K���5L*M�)��5��� N �4 O�*27)�2�8P9<;=;!>=;�QOGC.IHR $'&(�S)�"��+����� . T U V'W(XZY0[]\+^ )��324�5���5��6 ��27)�2 �A O)�"������ ���

27� 27_J)�������27)��5/S� "%$'&(��)�"*�+�,�-� .�`�T U �,M�a3" )�"b27)9<;=;4>=;+?A@CB DFE!GC.IH�c 9<;0;4>=;<?A@dB DFESGC. ` H

27���9<;=;4>=; Q GC.IH�e 9<;0;4>=; Q Gd. ` H

f5g hjilklmonqpritsuhjvjwqilsKxzy{p,muil|{m~}qy�y{���4�5�l�����������~�5�~�����5�{�5���q�4v�m�k���ilhjh����4kF��pu���¡ Kvjhjhz�£¢�¤5¤5¥

Page 18: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� ��� � � ����������� � ����� � �������� ���

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0 10 20 30 40 50 60 70 80 90 100

Acc

urac

y

Size of tree (number of nodes)

On training dataOn test data

� � !#"%$%&('*)+"-,.!#/#0*"%,21435)6&."%75&98*3:35;�<�= >%?:@BADCFE�C9= G9AH@IA J5K LNM*O�/P&Q$SR:"%!#!TKHO�$VUW).XZY\[2/#!#!4K^]Z_ _ `

Page 19: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� ����������������

� ��� ���! �#" ��$%�%&(' ��$%"*),+.-/-/&0 2143576 -/�98 19):���;&< =1 �;>2"* '=�!-?� 6 8=@0&0- =�%- 6 -?�!-/& 6 -/&(���!@A@(B6 &019 =&C+D�*�E 4-

5 1%):��� FHG=@<@I-/)?"*"%J2-/>="* 8D� 6 -LKM8=):G2 ="

� ��� -,� 6 "*@0"N�O- PLQD" 6 -SR -/)?"*"%T57U "V� 6 G=)?" 8D"�)/FW�%):X �! Y�O" ��$9"*)Z-/)[�E&0 =&0 =1 '=�!-?�57U "V� 6 G=)?" 8D"�)/FW�%):X �! Y�O" ��$9"*) 6 "*8\�!)[�E-/" $��!@0&('=�E-/&A�9 '=�!-?� 6 "*-

57U ] ^ T_X &0 2&0X &0`*"a*bLc%d�egfMhEd�d!i�j a*bLc%d�eWk bLaVl*mWn4a�a*bLo.bplqn�fMbprtsuavegfMhEd�d!i/i

wCx y{z}|}~�����zq��y{�{��z}�,���A��~�z}�A~����(�A���E�0�}�(�������g���0�������0�A�0 ¢¡�£E�¤~¥|§¦(z}y{y¨��£E|ª©«��¬®­°¯,�{y{y��M±®²0²0³

Page 20: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ��� � ����� � ���� �� � ��� � ��� �

��������� ��� �!� ��"#�%$ &('*),+(-�+(-/. � "�� 01)#23+54�)�&6+578- 9;:<�= $ >�"1�;����?@>�A!�;B�:<A ��AC>�"���"�D �E9FB��*ACG ?@>��IHJLKNMPO � ��>�� �;: ��G ���QR�S$" 01)#23+54�)�&(+578- 9;:<� $*?T��AC>�"���"�D

:U�QVB �W$X9!9Y��Z���: "�$1�[: \]����>�9 �;B�$^9Y: ZW:<��$`_ ���bac1Kd AC:e:U�[����f A!:<G $ O : �%B�: $"�: �;B�� �SG $^9Y�g��G ��A!$ O :h9

01)#23+54�)�&6+578- 9Y:<� �Q<QR>�Ai�QRf

j ��AC$1�k>�QR:U9 9;G � ����:U9;� O :<Ai9Y��$" $*?TG $X9;�S�QeQR>�Ai� �;:9Y>�Z��;AC:e:

jgl B�� �S�m?n��� �!� �o9 ����G ���;:U�qp

r�s tvuxwxy{z}|~u���tv�v�}ux�%����|Yy�ux��y��}�o���N�*���x�o�����b�I�������3���������i�}�*��y�w �ouxtvt¡�3�*w£¢¤|�¥§¦©¨%�vtvt��(ª§«�«�¬

Page 21: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� � ����� �� ��� � � ��� ��� ������� � ��� � ��� �

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0 10 20 30 40 50 60 70 80 90 100

Acc

urac

y

Size of tree (number of nodes)

On training dataOn test data

On test data (during pruning)

��� ���! !"$#&%'�)(*��+�,&�!(.-0/1%2"*�!31"54&/6/1798;:�<!=6>@?BADCEA5:�F5?G>H?�I1J�KML&N;+O"P RQ6�!���SJGN; UTV%*WYX[Z.+����0J]\Y^�^�_

Page 22: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� � ��� � ����� � ��� � ��� �

��� ������� ���! " #�!��� #� �%$'&)(*�,+.-/�0�1 2#�� �435�!&)-*�6271�98��!&)�)� �%+ :<; �=&>-*� (*��?@��AB����?C�D�� #-*E �435� #;>���F2G@� HI� �! KJ���+.-L�=&>-*�%2M(*�1 #� ?C�62N(*�!�%? 2#�%$'&)����:O� 3P� � &�2N�

8Q���!;�+.A�2KR �S2# T3P�!�%$'&)�D�� #-*E &�2#�%? R �� #;)�1? UP� �WV)�YXLZ []\*^)_

`*a bdcfefgihkjlcOmnbdodpkcfmrqtsujNgncfvugWwksYsux9y4z*{f|Y}�~������Wz*�W~�}�~*�u�*���k�4o�g�e��Ycfbdb����4e���jn�����rodbdbt�����*�*a

Page 23: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ����������� � � � � ����� �� � � �����

Outlook

Overcast

Humidity

NormalHigh

No Yes

Wind

Strong Weak

No Yes

Yes

RainSunny

��� ��� � !#"%$&�(')��*�+%� '-,/.0$1!)� 20!43%.5.0687:9�; <5=?>A@CBD@49�E4>F=G>�H0I�JLK%M:*N!O�QP5� ���RIFM:�TSU$)VXWZY-*����/I\[X]�]�^

Page 24: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

��� ��� ����� ��� � ������������� ��� ��� � �!� �"� � � �$#&%'�(*) +-, . 0/!�21 34�����65 � 7 �

��� ��� ����� ��� � ������������� ��� ��� � �!� �"� � 7 �989� /:��(*) +-, . 0/!�21 34�����65 � ; 395

<=<=<

>@? ACBEDEFHGJIKBMLNACOCPJBELRQTSUIVFNBEWUFYXJSZSU[]\_^@`EaZbdcfehgieY^@jYckblc@mUn@oqpJr_OsFtDvuZBEACAwnkr_DyxzIN{}|�~ROCACATn��}?@?@�

Page 25: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ��������� �� � �� � ��� � ��������� ����

�����! #"$� %'&)(+*,���-"$� ."$"$�+&0/�1�"$� "$2 "$�3($" *,2546"$&04�1�271�(8:9 ;=< >�;=?.@BADCE?F; G HJI�KML8 NO9 ;=< >�;=?F@BAPC�?.; Q R.I6KTSVUWG AYX[Z

\�]_^ `�]_a+b5ced�a�]_f gih g H j h R.I H h k7hl:m0b.no\�]_p�p�qsr[f t 2 t 2 u��!( u��3( u��3( t 2

v0w xzy|{|}�~s��y���xz�z�sy|��������}�y|��}��s�)�����.�0�|�)�����[�����0�������0 �¡0¢¤£s¥.�¦}§{©¨)y|xzxª¡�¥.{M«¬��­¯®�°��zxzx�¡D±¯²0²0v

Page 26: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ��������� �� � � ����� ����� � ���� ��

����� ��!#"%$ &'�(�)+*-,., ��/#��0 , " 1 * 2 $ *-354 67* !#08" 2:9<; =?>�@ A /#!#! 2 "%!#"CB , / ,'�( $ *-D / 3 " 0 2 / 3�D E =GFIH J KMLN@ O PCQ Q R * 2S*-,., ��/#��0 , "

T 3 " *-U8U �V� * BW1X&Y0 2 " ; =?>�@[Z =GF�>]\ / 3N2., " * ^

; =?>�@_Z =GF`>]\�a`b ced f+g ; =?>�@ha`bicVd fbkjmln>]FIo?@[pk\rq7s =GF`>]\r@ha`b ctd f

bkjkln>]F]o?@[pk\rqrs =GF`>]\r@ha`b c�d f+g u vwxzy|{} b x }} b }

!#� D5~} b x }} b }

A 18"%�V" b x / 2 2 08� 2 " , � )�b ) � � A 18/�BW1 d 1 * 2�67* !�08" � x

��� �������������������������������t�����������������¡ #¢�£�¤¦¥¨§ª©«§� #¬�¥n¤­¥#®�¯#°e±�²¡�³�´�¶µ������·¯n²¡�¹¸º��»½¼|¾�������¯��½¿#¿#�

Page 27: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� ��������� �� � � ����� � �������

��������� �"!$#%'& !(�"� )+*-,.�"� *-/����0�1�2�4365 798:8:;=< >@?(ACB�*�� )D�0��E FHGJI:K% #L��MN�OE�� )$�$3QP RS;�AUT V�WX8@Y GZVNA[B\*��])D�0��E ^:_ ��!()-`

a �@b Ec� ,d!(*-#e� * )D������� ��E�!$��E E�#L!+! b]�fEcB ,f�@b !hg�iN!j)DE�!(�)D�0��Elkm ��! *-i�i�#e�0*�)nBpoq#e!$i�,2*�)D! /0*:�f� M�r

%ts *-� *-�\� u")vB�,f� & & !$# wUG(x�x�K�yz {=R�|.}lw�~ ��� y

� 8�?(Alw�� y �

%'� � ��!$� wUG(x�����y^@���U���J���������"� G

w�� 8X?(A�w�� yp� G@y��b�B�!$#L! � �  ¡K��$Gh¢ �"Ec!$# & �f���£� & iN��#eEe*-�\)D! �:¤C)D�¥�1E

¦f§ ¨ª©�«�¬®­°¯±©h²¡¨ª³ª´°©�²cµ·¶¸¯1¬¡©�¹¸¬»º°¶ ¶¸¼J½:¾f¿�À ÁÃÂÅÄlÆ�Ä»¾fÇ»ÂÈÁ�ÂfɸÊfË�Ì°Í:³Î¬Ï«ÑÐ ©�¨ª¨ÒÊÈÍ:«ÔÓ�¯¡Õ�ÖØ×c³ª¨ª¨·Ê�Ù�ÚfÚf¦

Page 28: Overcast - Academics | WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/MitchellsBook/ch3.pdf · ( 10 2 % # 3(*4 5 267 & # (8 (8 # 9 : -/. ( 10 9 ;(* 3 10 < ! # & $= 36 > (*

� � � � ��� � � ������ � ��� � ����� ���

� ����� �! #"%$'& ( (*)��+& ,.-/(0"1& �2"3"4�65.7 89�+-/:.(;" $< #= >? "%( �%@A�+�/5.�65.7 (*)��+& ,.-/( �+5CBEDF�0B'G"%$'@3�1�H�.@I$J:.7'� �%@3(K(

LFM N5.$COP( Q �%(0"4�3" = G�'"3"4�/7J5 & $R"%�TSU$J& & $'5 89�+-/:.( $� = �+& $'57 $'�%�.(K@ (V)��+& ,.-/(;" "%$'@3�%(;O �%$ 5.$COP( Q

L �'"I"%�/7'5 & $W"4�XSU$'& & $J5 89�+-/:.( $< #= �+& $'57 $'�%�(K@(V)��+& ,.-6(;"1DT�/�%� "3�+& ( �3�+@37'(K�T89�+-/:(

L �'"I"%�/7'5 ,.@3$'Y���Y.�/-/�/�ZB [.\]�%$ (;�JS^� ,_$R"3"4�/Y-/( 89��-/:.( `�\]$< =

a �'"3"4�67'5 b@A�'SU�H�/$'5 [.\c$� N(*)��+& ,.-/( �%$ (;�JS^�OP(;"3SU(K5dO.�+5��T�/5 �%@3(K(

e -2�'"3"4�f bB 5(KD (V)���& ,.-/(;" �/5 "3�+& ( g�'"%�.�/$'5

h/i jlknmnoqpsrtkVuvjlwlxsknuHy{z|r4ovkn}|o�~sz2z|�����/�n�2�����������/�������/�|�/�A�s��w�o�m��2knjlj�����m���rv� �¢¡Hwljlj{�¤£ ¥/¥/h