cvpr2010: semi-supervised learning in vision: part 3: algorithms and applications

Semi-Supervised Learning in Computer VisionPart II

Amir Saffari,Christian Leistner,Horst Bischof

Institute for Computer Graphics and Vision, Graz University of Technology

June 18th, 2010

Graz University of Technology

SemiBoost & Visual Similarity Learning On-line Semi-supervised Boosting Semi-Supervised Random Forests On-line Manifold Regularization Conclusion & Outlook

Outline

1 SemiBoost & Visual Similarity Learning

2 On-line Semi-supervised BoostingTracking

3 Semi-Supervised Random ForestsMILForestsOn-line Random Forests

4 On-line Manifold Regularization

5 Conclusion & Outlook

Amir Saffari,Christian Leistner,Horst Bischof Semi-Supervised Learning in Computer Vision Part II



SemiBoost[Mallapragada et al.,PAMI’09] [Leistner et al.,CVPR’08]

Loss function ∑(x,y)∈XL e−yF(x)+∑

x∈XU

(λu

∑x ′∈XU

s(x, x ′) cosh(F (x) − F (x ′)) + λl

∑(x ′,y ′)∈XL

s(x, x ′)e−2y ′F(x))

Optimization Problem

arg minf (x),α

=∑

x ′∈XU

( ∑(x,y)∈XL

s(x, x ′)e−2y(F(x ′)+αf (x ′))

+λu

∑x ′∈XU

s(x, x ′)e((F(x ′)−F(x))eα(f (x)−f (x ′)))




SemiBoost

px = λl

∑(x ′,y ′)∈XL

I(y ′ = 1)s(x, x ′)e−2F(x ′)+λu

2

∑x∈XU

s(x, x ′)eF(x ′)−F(x)

and

qx = λl

∑(x ′,y ′)∈XL

I(y ′ = −1)s(x, x ′)e−2F(x ′)+λu

2

∑x∈XU

s(x, x ′)eF(x)−F(x ′)




SemiBoost

Pseudo Labels and Weights

yx = sign(px − qx)

wx = |px − qx|

Optimal α

α =14

ln

∑x∈XU

(piI(f (x) = 1) + qiI(f (x) = −1)

)∑x∈XU

(piI(f (x) = −1) + qiI(f (x) = 1)

)




SemiBoost

I labeled training data (x, y) ∈ XL and unlabeled data x ′ ∈ XU

I Similarity measure s(x, x ′)I Weak learners fi

I weight parameters λu, λl

I max iterations T

1 For t = 1, 2, . . . , T2 Compute pi and qi for every given sample3 yx = sign(px − qx)

4 wx = |px − qx|

5 Train weak classifier ft (x)

6 Compute αt

7 F (x)← F (x) + αt ft (x)

8 EndFor




SemiBoost with learned Similarities[Hertz et al.,CVPR’04]

Radial Basis Function [Zhu et al.,ICML’03]

s(x, x ′) = e

(−

d(x,x ′)2

σ2

)

d(x, x ′) . . . distance between points




Learning Distance Functions

Idea

Learn distance or metric function on labeled data which thencan discriminatively support task-specific classification.

Distance Function

F d : X× X→ Y = [−1 1]

Training Pairs of “same” or “different” [Hertz et al.,CVPR’04]

Dd = {(x, x ′, +1)|y = y ′, x, x ′ ∈ DL} ∪∪{(x, x ′, −1)|y , y ′, x, x ′ ∈ DL}




SemiBoost with learned Distance Functions

Number of Training Pairs (Symmetric case)n·(n−1)

2

SemiBoost+

-

?-

-++

+ ??

?

?




Using Arbitrary Classifiers

Approximate pair-wise classifier

|F (x, x ′)| ≈ |F (x) − F (x ′)|

+

-

?

?

+




Reusing Prior Classifiers[Schapire et al,ML’02]

Classifier Combination

F C(x) = α0F P(x) + F (x)

SemiBoost+

-

?

??

?

?




SemiBoost Applications

Car Detection




Similarity Performance

Accuracy depending on the number of samples





Car Detection





Face Detection

(a) prior (b) trained (c) combined




Simple Data mining method[Levin et al.,ICCV’03][Rosenberg et al.,2005]

1 Labeled training data (x, y) ∈ XL

2 Train cascaded detector F P(x) on XL using [Viola & Jones,2001]

3 Use a web image search engine in order to collect hugeamounts of possibly useful images XU ; pass phrases that aremuch likely related to your target object

4 Apply F P(x) in a sliding window manner on XU and copy alldetections to XU∗

5 Train a SemiBoost classifier F (x) on XL and XU∗ using F P(x)

as prior

6 Output the final classifier F (x)





Transfer Learning




Outline









On-line Boosting




Tracking

[Oza,PhD-Thesis’01], [Grabner & Bischof,CVPR’06]




Tracking




Tracking

Tracking is an One-Shot Semi-supervised Learning Problem




On-line SemiBoost

px ≈ e−Fn−1(x)∑

xi∈X+

S(x, xi) ≈ e−Fn−1(x)F +(x) ≈ e−Fn−1(x)eF P(x)

eF P(x) + e−F P(x)

qx ≈ eFn−1(x)∑

xi∈X−

S(x, xi) ≈ eFn−1(x)F −(x) ≈ eFn−1(x)e−F P(x)

eF P(x) + e−F P(x)

pn(x)−qn(x) =sinh(F P(x) − Fn−1)

cosh(F P(x))= tanh(F P(x))−tanh(Fn−1(x))




Tracking




Tracking

Problem: Rapid Appearance Changes




Tracking




Exploration-Exploitation Dilemma

Convex Trade-off

`(F (x)) = (1 − α)`l(F (x)) + α`u(F (x))

We need more Robustness when minimizing the labeled loss!




Loss Functions

Random classification noise defeats all convex potential boosters[Long and Servidio,ICML’08]




On-line Gradient Boost

Gradient Descent Functional Gradient Descent

GradientBoost [Friedman et al.,Annals of Statistics’01]

ft (x) = arg maxf (x)

−∇LT f (x)




On-line Gradient Boost

I A training sample: (xn, yn), A differentiable loss function `(·)I Number of selectors M , Number of weak learners per selector K

1 Set F0(xn) = 0.

2 Set the initial weight wn = −`′(0).

3 For m = 1 to M

4 For k = 1 to K

5 Train kth weak learner f km(x) with sample (xn, yn) and weight wn.

6 ekm ← ek

m + wnI(sign(f km(xn)) , yn) //Compute the error

7 EndFor

8 Find the best weak learner with the least total weighted error:j = arg min

kek

m.

9 Set fm(xn) = f jm(xn).

10 Set Fm(xn) = Fm−1(xn) + fm(xn).

11 Set the weight wn = −`′(ynFm(xn)).

12 EndFor

13

→Output the final model: F (x)




Weight Updates




Co-Training of Pedestrian Detectors

Exponential Loss Logit Loss




SERBoost

Expectation Regularization [Mann and MacCallum,ICML’07]

Penalize model predictions on unlabeled data that deviate fromcertain expectation.

SERBoost [Saffari et al.,ECCV’08]

L(H (x), X) = Ll(H (x), Xl) + βLu(H (x), Xu)

L(H (x), X) =∑

x∈XL

e−yH(x) +∑

x∈XU

e−ypH(x)cosh(H (x))

Pseudo Label

yp = 2P+p (x) − 1




On-line SERBoost with logistic loss

Supervised Loss

Ll(XL) =

∑(x,y)∈Xl

log(

1 + e−2yF(x))

=∑

(x,y)∈XL

log(

e−yF(x)(eyF(x) + e−yF(x)))

=∑

(x,y)∈XL

−yF (x) + log(

eF(x) + e−F(x))

.





Minimize the cross entropy

H (Pp, P) = −∑

z∈{−1,1}

Pp(y = z|x) log P(y = z|x)

= −(

2Pp(y = 1|x) − 1)︸︷︷︸

yp(x)

F (x) + log(

eF(x) + e−F(x))





Unsupervised Loss

Lu(XU ) =∑

x∈XU

H (Pp, P) =∑

x∈XU

−yp(x)F (x) + log(

eF(x) + e−F(x))

Unlabeled Update

∀ x ∈ XU :wx =∣∣yp(x) − tanh(F (x))

∣∣yx = sign

(yp(x) − tanh(F (x))

)




OSER Tracking

λ = 0.5




Influence of convex combination




Multiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09]




Multiple Instance Boosting[Viola et al.,NIPS’05][Babenko et al.,CVPR’09]

Bags

{(B1, y1), . . . , (Bn, yn)}

Bi = {x1i , x2

i , . . . , xnii }

Minimize binary log-likelihood

log L =∑

i

(yi log p(yi) + (1 − yi) log p(yi))




Semi-Supervised Multiple Instance Boosting[Zeisl et al.,CVPR’10]

Combine benefits of MIL and SSL





Unlabeled Loss of the Bags

Lu(XBu ) = −

Nu∑i=1

∑z∈Y

Pp(z|Bui ) log(P(z|Bu

i ))

Approximate max with geometric mean

P(y = 1|Bi) = 1 −[NBi∏

j=1

(1 − P(y = 1|xij)

)]1/NBi





Experimental Results

Sequence MILSER MIL OSB OAB

sylv 0.64 0.61 0.46 0.50david 0.71 0.54 0.31 0.32faceocc2 0.78 0.65 0.63 0.64coke11 0.18 0.29 0.12 0.20tiger1 0.60 0.51 0.17 0.27tiger2 0.46 0.50 0.08 0.25faceocc1 0.68 0.63 0.71 0.47girl 0.64 0.53 0.69 0.38




On-line Co-Training[Liu et al.,ICCV’09][Saffari et al.,ECCV’10]

Performance measured in average location center errors in pixels

Approach sylv david faceocc2 tiger1 tiger2 coke faceocc1 girl

MV-GPBoost 17 20 10 15 16 20 12 15CoBoost 15 33 11 22 19 14 13 17SemiBoost 22 59 43 46 53 85 41 52MILBoost 11 23 20 15 17 21 27 32




End Part I




Random Forests

[Breiman,ML’01]

Ensemble of n decision trees

F (x) =∑N

n=1 f (x)

Information Gain

∆H = −|Il |

|Il |+|Ir |H (Il) −

|Ir ||Il |+|Ir |

H (Ir)




Random Forests

Advantages:I speedI parallelismI noise robustI inherently multi-class

Applications:I Object Detection, Semantic Segmentation, Categorization,

Tracking, etc.

Disadvantage:I RFs demand a huge amount of data in order to leverage their

full potential [Caruana et al.,ICML’08]




Optimization

Incorporate labels for the unlabeled data as additionaloptimization variables!

Deterministic Annealing [Rose,IJCNN’98]

p∗ = arg minp∈P

Ep(F(y)) − TH(p)

T0 > T1 > . . . > T∞ = 0

p∗ . . . distributions over the label predictions




Two Step Optimization

First Stage

f∗n = arg minf

1|Xl |

∑(x,y)∈Xl

`(fy(x))+

+α

|Xu |

∑x∈Xu

`(fyu(x))




Finding the optimal Distributions

Take the derivate w.r.t. each class

hi(p, x) = p(i|x)(α`(gi(x)) + T log(p(i|x))) (1)

dhi

dpi= α`(gi(x)) + T log(p(i|x)) + T (2)

Optimal Distribution

p∗(i|x) = exp(−α`(fi(x))+T

T )/Z(x)

Z(x) =∑K

i=1 p∗(i|x) is the partition function




Experiments

Classification Accuracy in %

Method SVM TSVM SER RMSB RF DAS-RF

g50c 91.7 93.1 91.9 94.2 89.1 93.3Letter 70.3 65.9 76.5 79.9 76.4 79.7SensIt 80.2 79.9 81.9 83.7 76.5 84.3

Train and Test time in Seconds

Method SVM TSVM SER RMSB RF DAS-RF GPU

Letter 25 74 3124 125 35 72 29SensIt 195 687 1158 514 125 410 137




Caltech-101

binary classification error

Class RF DAS-RF ImprovementC4 0.0081 0.0033 58%C5 0.0078 0.002 65%C20 0.011 0.0013 87.5%C33 0.007 0.003 52%C81 0.0027 0.001 62.5%

classification error over different numbers of labeled samples

Algorithm l = 15 l = 30

RF 0.72 0.64DAS-RF 0.70 0.60LinSVM 0.74 0.65improvement 2% 4%




Prior Regularization

Potential Information Gain

∆H = −|Il |

|Il |+|Ir |H (Il) −

|Ir ||Il |+|Ir |

H (Ir)

Kullback-Leibler Divergence

DKL(q‖p) = H (q, p) − H (q)

DSKL(q‖p) = 12(DKL(q‖p) + DKL(p‖q))

Prior-regularized node score

∆H∗ = ∆H + λ∆DSKL(q‖p)




Airbag

OOBE : em−1F − em

F




Outline









Multiple Instance Forests[Leistner et al.,ECCV’10]

--

-

-

-

-

+

-

---

-

+

+

[Dietterich,AI’97]

I Content-based Image RetrievalI Object Detection and CategorizationI TrackingI Action Recognition




Multiple Instance Forests

Multiple Instance Learning is a special case of semi-supervisedLearning!





Multi-class Instance Classifier

F (x) : X→ Y = {1, . . . , K }

{(B1, y1), . . . , (Bn, yn)}, where yi ∈ {1, . . . , K }

Objective Function

({y ji }∗, F∗) =arg min

{y ji },F(·)

n∑i=1

ni∑j=1

`(Fy j

i(xj

i))

s.t. ∀i :

ni∑j=1

I(yi = arg maxk∈Y

Fk(xji)) > 1.





DA Loss Function

LDA(F , p) =

n∑i=1

ni∑j=1

K∑k=1

p(k|xji)`(Fk(xj

i)) + Tn∑

i=1

H (pi)

Entropy of the distribution inside a bag

H (pi) = −

ni∑j=1

K∑k=1

p(k|xji) log(p(k|xj

i))




Evaluation

Method Elephant Fox Tiger Musk1 Musk2

RandomForest[Breiman,2001] 74 60 77 85 78

MILForest 84 64 82 85 82

MI-Kernel[Andrews,2003] 84 60 84 88 89

MI-SVM[Zhou,2009] 81 59 84 78 84

mi-SVM[Zhou,2009] 82 58 79 87 84

MILES[Chen,2006] 81 62 80 88 83

SIL-SVM[Bunescu,2007] 85 53 77 88 87

AW-SVM[Gehler,2007] 82 64 83 86 84

AL-SVM[Gehler,2007] 79 63 78 86 83

EM-DD[Zhang,2001] 78 56 72 85 85

MILBoost-NOR[Viola,2006] 73 58 56 71 61




Corel Data Set




Corel Data Set

Results for the COREL image categorization benchmark

Method Corel-1000 Corel-2000 Testing[sec.] Training[sec.]

MILForest 59 66 4.6 22.0

MILES 58 67 180 960




Semantic Segmentation

[Vezhnevets & Buhmann,CVPR’10]




Outline









On-line Random Forests




Interactive Segmentation




Tracking with On-line RF




Tracking




Tracking

RT ∩ RGT/RT ∪ RGT

Sequence OSERB MILBoost OSB OAB ORF MILForest RF

sylv 0.64 0.61 0.46 0.50 0.53 0.59 0.50david 0.69 0.54 0.31 0.32 0.69 0.72 0.32faceocc2 0.77 0.65 0.63 0.64 0.72 0.77 0.79tiger1 0.65 0.51 0.17 0.27 0.38 0.55 0.34tiger2 0.42 0.50 0.08 0.25 0.43 0.53 0.32coke 0.2 0.33 0.08 0.25 0.35 0.35 0.15faceocc1 0.77 0.63 0.71 0.47 0.71 0.77 0.77girl 0.77 0.53 0.69 0.38 0.70 0.71 0.74




On-line Manifold Regularization[Goldberg et al.,ECML’08]

I Based on Convex Programming in kernel space usingstochastic gradient descent

I Random Projection Trees [Dasgupta & Freund, TR, 2007]




On-line Manifold Regularization[Goldberg et al.,ECML’08]




On-line Graph-based SSL[Kveton et al.,OLCV’10]

Harmonic Function Solution





Merge the two most similar vertices and add the new vertex





Face recognition of 8 people




Conclusion

Semi-supervised Learning is a powerful learning paradigm withmany potential applications in Computer Vision

I It is often also the way how learning is done in natureI It can be applied virtually everywhere where classifiers are

appliedI On-line SSL can be used in order to make

tracking-by-detection systems more robust




Outlook

We need to increase the robustness of SSL algorithms in order toleverage more applications

I Demand for more on-line Semi-Supervised MethodsI SSL from weakly-related unlabeled data




References

BooksI O. Chapelle and B. Schoelkopf and A. Zien, “Semi-Supervised Learning”, The MIT Press, 2006

I Xiaojin Zhu and Andrew B. Goldberg, “Introduction to Semi-Supervised Learning”, Morgan & Claypool, 2009

Papers and ArticlesI C. Leistner, A. Saffari and H. Bischof, “MILForests: Multiple-Instance Learning with Randomized

Trees”,ECCV’10I C. Leistner, A. Saffari, J. Santner and H. Bischof: “,Semi-Supervised Random Forests”,ICCV’09I C. Leistner, A. Saffari, P.M. Roth and H. Bischof: “On Robustness of On-line Boosting – A Competitive

Study”,(ICCV) OLCV’09

I H. Grabner, C. Leistner and H. Bischof: “On-line Semi-Supervised Boosting for Robust Tracking”,ECCV’08

I B. Zeisl, C. Leistner, A. Saffari and H. Bischof: “On-line Semi-supervised Multiple-Instance Boosting”,CVPR’10

I C. Leistner, “Semi-Supervised Ensemble Methods for Computer Vision”, PhD-Thesis, Graz University ofTechnology, 2010

I A. Saffari, C. Leistner, M. Godec, J. Santner and H. Bischof, “On-line Random Forests”, (ICCV) OLCV’09I A. Saffari, C. Leistner, M. Godec and H. Bischof, “Robust Multi-View Multi-Class Boosting with Priors”,ECCV’10

I B. Kveton, M. Valko, M. Philipose and L. Huang, “Online Semi-Supervised Perception: Real-Time Learningwithout Explicit Feedback”, (CVPR) OLCV’10

I A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09

I C. Leistner, H. Grabner and H. Bischof, “Semi-Supervised Boosting using Visual Similarity Learning”,CVPR’08

I A. Saffari, C. Leistner and H. Bischof, “Regularized Multi-Class Semi-Supervised Boosting”,CVPR’09

I A. Saffari, H. Grabner and H. Bischof, “SERBoost: Semi-supervised Boosting with ExpectationRegularization”,ECCV’08


cvpr2010: semi-supervised learning in vision: part 3: algorithms and applications

Education