introduction to chap13 - statistical learning and machine...
TRANSCRIPT
![Page 1: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/1.jpg)
Introduction to Statistical Learning
and Machine Learning
Chap13 - Semi-supervised Learning
Fudan-SDS Confidential - Do Not Distribute
![Page 2: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/2.jpg)
Chap12 - Semi-supervised Learning
Fudan-SDS Confidential - Do Not Distribute
![Page 3: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/3.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleBooks
![Page 4: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/4.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitle
![Page 5: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/5.jpg)
Fudan-SDS Confidential - Do Not Distribute
Why bother?
![Page 6: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/6.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitlewhile it may be dangerous to obtain labelled data.
![Page 7: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/7.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitlewhile it may be dangerous to obtain labelled data.
![Page 8: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/8.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleExamples of hard-to-get labels
![Page 9: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/9.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleNotations
![Page 10: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/10.jpg)
Fudan-SDS Confidential - Do Not Distribute
Learning paradigms
![Page 11: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/11.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleWhy the name
![Page 12: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/12.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleWhy the name
We will mainly discuss semi-supervised classification.
![Page 13: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/13.jpg)
Semi-supervised Learning
Intuition understanding
Fudan-SDS Confidential - Do Not Distribute
![Page 14: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/14.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleHow semi-supervised learning is helpful?
![Page 15: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/15.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleHow semi-supervised learning is helpful
![Page 16: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/16.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleHow semi-supervised learning is helpful?
![Page 17: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/17.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleTwo moons
There are 2 labeled points (the triangle and the cross) and 100 unlabeled points. The global optimum of S3VM correctly identifies the decision boundary (black line).
This figure comes from Chapelle et al. “Optimization Techniques for Semi-Supervised Support Vector Machines”, JMLR 2007.
![Page 18: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/18.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleDoes unlabeled data always help?
![Page 19: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/19.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleDoes unlabeled data always help?
Unfortunately, this is not the case, yet.
![Page 20: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/20.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleDoes unlabeled data always help?
Unfortunately, this is not the case, yet.
Not until recently, “Safe Semi-supervised learning”, Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. IEEE Transactions on Pattern Analysis and Machine Intelligence 2014.
![Page 21: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/21.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAssumption and Intuitions
• Points in the same cluster likely to be of the same class; • Decision line should lie in a low-density region.
Guess the name?
![Page 22: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/22.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAssumption and Intuitions
• Points in the same cluster likely to be of the same class; • Decision line should lie in a low-density region.
Guess the name?
Cluster Assumption
![Page 23: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/23.jpg)
Fudan-SDS Confidential - Do Not Distribute
Assumption and Intuitions
• The data lie approximately on a manifold of much lower dimension than the input space. • Swiss Roll
Guess the name?
![Page 24: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/24.jpg)
Fudan-SDS Confidential - Do Not Distribute
Assumption and Intuitions
• The data lie approximately on a manifold of much lower dimension than the input space. • Swiss Roll
Guess the name?
Manifold Assumption
![Page 25: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/25.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAssumption and Intuitions
If two points x1, x2 in a high-density region are close, then so should be the corresponding outputs y1, y2.
Guess the name?
Points which are close to each other are more likely to share a label. This is also generally assumed in supervised learning and yields a preference for geometrically simple decision boundaries. In the case of semi-supervised learning, the smoothness assumption additionally yields a preference for decision boundaries in low-density regions, so that there are fewer points close to each other but in different classes.
![Page 26: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/26.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAssumption and Intuitions
If two points x1, x2 in a high-density region are close, then so should be the corresponding outputs y1, y2.
Guess the name?
Smoothness AssumptionPoints which are close to each other are more likely to share a label. This is also generally assumed in supervised learning and yields a preference for geometrically simple decision boundaries. In the case of semi-supervised learning, the smoothness assumption additionally yields a preference for decision boundaries in low-density regions, so that there are fewer points close to each other but in different classes.
![Page 27: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/27.jpg)
Semi-supervised Learning
Key methods of SSL
Fudan-SDS Confidential - Do Not Distribute
![Page 28: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/28.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleMethods and Algorithms of SSL
• Self-training • Co-training -Multi-view Algorithm • Graph-based SVM • Semi-supervised SVM (S3VM)
![Page 29: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/29.jpg)
Self-training
Fudan-SDS Confidential - Do Not Distribute
![Page 30: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/30.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleSelf-Training algorithm
![Page 31: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/31.jpg)
Fudan-SDS Confidential - Do Not Distribute
Variations in self-training
![Page 32: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/32.jpg)
Fudan-SDS Confidential - Do Not Distribute
Self-training example: image categorisation
![Page 33: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/33.jpg)
Fudan-SDS Confidential - Do Not Distribute
Self-training example: image categorisation
![Page 34: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/34.jpg)
Fudan-SDS Confidential - Do Not Distribute
The bag-of-word representation of images
![Page 35: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/35.jpg)
Fudan-SDS Confidential - Do Not Distribute
Self-training example: image categorization
![Page 36: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/36.jpg)
Fudan-SDS Confidential - Do Not Distribute
Self-training example: image categorization
![Page 37: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/37.jpg)
Fudan-SDS Confidential - Do Not Distribute
Comments on Self-trainingAdvantages of self-training
Disadvantages of self-training
![Page 38: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/38.jpg)
Co-training one type of Multi-view algorithm
Fudan-SDS Confidential - Do Not Distribute
![Page 39: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/39.jpg)
Fudan-SDS Confidential - Do Not Distribute
Co-training
![Page 40: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/40.jpg)
Fudan-SDS Confidential - Do Not Distribute
Feature split
![Page 41: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/41.jpg)
Fudan-SDS Confidential - Do Not Distribute
Co-training assumptions
![Page 42: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/42.jpg)
Fudan-SDS Confidential - Do Not Distribute
Co-training algorithm
![Page 43: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/43.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitlePros and cons of co-training
![Page 44: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/44.jpg)
Fudan-SDS Confidential - Do Not Distribute
Variants of co-training
![Page 45: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/45.jpg)
Graph-based semi-supervised
learning
Fudan-SDS Confidential - Do Not Distribute
![Page 46: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/46.jpg)
Fudan-SDS Confidential - Do Not Distribute
Graph-based semi-supervised learning
![Page 47: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/47.jpg)
Fudan-SDS Confidential - Do Not Distribute
Graph-based semi-supervised learning
![Page 48: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/48.jpg)
Fudan-SDS Confidential - Do Not Distribute
Graph construction
![Page 49: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/49.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleGraph-based semi-supervised learning
![Page 50: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/50.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAdjacency matrix
![Page 51: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/51.jpg)
Fudan-SDS Confidential - Do Not Distribute
Incidence matrixThe unoriented incidence matrix (or simply incidence matrix) of an undirected graph is a n × m matrix B, where n and m are the numbers of vertices and edges respectively, such that Bi,j = 1 if the vertex vi and edge ej are incident and 0 otherwise.
For example the incidence matrix of the undirected graph shown on the right is a matrix consisting of 4 rows (corresponding to the four vertices, 1-4) and 4 columns (corresponding to the four edges, e1-e4):
![Page 52: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/52.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleQuestions:Incidence matrix & Adjacency matrix
![Page 53: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/53.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleQuestions:Incidence matrix & Adjacency matrix
![Page 54: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/54.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleQuestions:Incidence matrix & Adjacency matrix
![Page 55: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/55.jpg)
Fudan-SDS Confidential - Do Not Distribute
The graph
![Page 56: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/56.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleLabel Propagation on graphs
![Page 57: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/57.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleHarmonic function f
Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions. Xiaojin Zhu, Zoubin Ghahramani, John Lafferty. ICML-2003. Awarded the classic paper prize in ICML 2013
![Page 58: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/58.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleAn electric network interpretation
![Page 59: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/59.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleA random walk interpretation
![Page 60: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/60.jpg)
Fudan-SDS Confidential - Do Not Distribute
Label Propagation
Zhou D.-Y. Learning with Local and Global Consistency, NIPS 2004.
![Page 61: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/61.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleLabel Propagation
![Page 62: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/62.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleLabel Propagation
![Page 63: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/63.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleLabel Propagation
![Page 64: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/64.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleGraph partition
![Page 65: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/65.jpg)
Fudan-SDS Confidential - Do Not Distribute
Min-cuts
In graph theory, a minimum cut of a graph is a cut (a partition of the vertices of a graph into two disjoint subsets that are joined by at least one edge) that is minimal in some sense.
Max-flow min-cut theorem:
The cut in a flow network that separates the source and sink vertices and minimizes the total weight on the edges that are directed from the source side of the cut to the sink side of the cut. As shown in the max-flow min-cut theorem, the weight of this cut equals the maximum amount of flow that can be sent from the source to the sink in the given network.
Karger's Min Cut Algorithm
![Page 66: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/66.jpg)
Fudan-SDS Confidential - Do Not Distribute
Min-cutsGraph-cut Normalised Cut
In graph theory, a minimum cut of a graph is a cut (a partition of the vertices of a graph into two disjoint subsets that are joined by at least one edge) that is minimal in some sense.
Max-flow min-cut theorem:
The cut in a flow network that separates the source and sink vertices and minimizes the total weight on the edges that are directed from the source side of the cut to the sink side of the cut. As shown in the max-flow min-cut theorem, the weight of this cut equals the maximum amount of flow that can be sent from the source to the sink in the given network.
Karger's Min Cut Algorithm
![Page 67: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/67.jpg)
Fudan-SDS Confidential - Do Not Distribute
Karger's minimum cut algorithm
For his dissertation "Random Sampling in Graph Optimization Problems."
David Karger ACM Doctoral Dissertation Award United States – 1994
![Page 68: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/68.jpg)
Semi-supervised Learning
(Transductive) SVM
Fudan-SDS Confidential - Do Not Distribute
![Page 69: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/69.jpg)
Fudan-SDS Confidential - Do Not Distribute
S3VM [Joachims98]• Suppose we believe target separator goes through low density regions of the space/large margin. • Aim for separator with large margin w.r.t labeled and unlabeled data. (L+U)
+
+
_
_
Labeled data only
+
+
_
_
+
+
_
_
S^3VMSVM
![Page 70: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/70.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleSemi-supervised SVM
![Page 71: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/71.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleSemi-supervised SVM
![Page 72: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/72.jpg)
Fudan-SDS Confidential - Do Not Distribute
Optional subtitleSemi-supervised SVM
![Page 73: Introduction to Chap13 - Statistical Learning and Machine ...yanweifu.github.io/courses/SLML/chap11_semi... · Fudan-SDS ConÞdential - Do Not Distribute Incidence matrix The unoriented](https://reader036.vdocuments.site/reader036/viewer/2022070923/5fbb650525eb526111425f9b/html5/thumbnails/73.jpg)
Appendix
Fudan-SDS Confidential - Do Not Distribute