roc 1.medical decision making 2.machine learning 3.data mining research communities a technique for...
TRANSCRIPT
![Page 1: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/1.jpg)
ROC
1. Medical decision making2. Machine learning3. Data mining research communities
A technique for visualizing, organizing , selecting classifiers based on their performance
![Page 2: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/2.jpg)
ROC Confusion matrix
![Page 3: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/3.jpg)
benefits
costs
ROC spaceAny classifier on the diagonal may be said to have on information about the class
![Page 4: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/4.jpg)
ROC curve
A discrete classifierdecision trees rule sets
Y or N Produces a single point
a Naive Bayes classifier a neural network
probability score
Each threshold value
produces a different point
Vary a threshold from −∞ to +∞ and
tracing a ROC curve
![Page 5: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/5.jpg)
ROC curve
![Page 6: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/6.jpg)
ROC curve
Threshold= + ∞
![Page 7: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/7.jpg)
ROC curve
ROC curves have an attractive property: they are insensitive to changes in class distribution.
![Page 8: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/8.jpg)
ROC curve
![Page 9: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/9.jpg)
ROC curve
![Page 10: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/10.jpg)
AUCDefinition: Area under an ROC Curve
The AUC has an important statistical property
1. It is equivalent to the Wilcoxon test of ranks2. It is also closely related to the Gini coefficient Gini + 1 = 2 × AUC
![Page 11: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/11.jpg)
Averaging ROC curvesThe error bars
![Page 12: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/12.jpg)
Decision problems with more than two classes
Multi-class ROC graphs
Multi-class AUC
![Page 13: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/13.jpg)
Iso-performance line
ability: 1. class skew 2. error costs
This equation defines the slope of an iso-performance line.
Conclusion: Lines “more northwest” (having a larger TP-intercept) are better because they correspond to classifiers with lower expected cost.
![Page 14: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/14.jpg)
Combining classifiers
![Page 15: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/15.jpg)
Conditional combinations of classifiers to remove concavities
1.idiosyncracies in learning 2.small test set effects
![Page 16: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/16.jpg)
Conditional combinations of classifiers to remove concavities
![Page 17: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based](https://reader031.vdocuments.site/reader031/viewer/2022032313/56649e575503460f94b4f90f/html5/thumbnails/17.jpg)
Logically combining classifiers
2. c4= c1 c2∨
1. c3 = c1 c2∧