guest lecture: visual tracking - artificial...

Guest Lecture - A. Alahi - !

Guestlecture:VisualTracking

AlexandreAlahiStanfordVisionLab/CVGL

17-Nov-151

Guest Lecture - A. Alahi - ! 2 17-Nov-15


WhyisVisualTrackingrelevant?

17-Nov-159

•  MediaproducNon(basketball,football)&augmentedreality(Hololens,magicleap)

Iwillputnicevideos

Guest Lecture - A. Alahi - ! 12


Allstartedintheearly60s

•  WithKalmanfilterformilitary

•  AbookonVideoTracking:TheoryandPracNce

18 17-Nov-15


Whatistrackingabout?

•  DataassociaNon•  Similaritymeasurement•  CorrelaNon•  Matching/Retrieval

•  Reasoningwith“strong”priors•  DetecNonwithverysimilarexamples

19 17-Nov-15


1.  Problemstatement2.  Challenges3.  ObjectrepresentaNon4.  Singletargettracking5.  MulN-targettracking6.  Tips&references

20

Outline

17-Nov-15


Problemstatement

•  Input:target•  ObjecNve:EsNmatetargetstateoverNme(space)•  State:

– PosiNon– Appearance– Shape– Velocity– AffinetransformaNonw.r.t.previouspatch

21 17-Nov-15


Problemstatement

•  Input:target•  ObjecNve:EsNmatetargetstateoverNme•  State:e.g.posiNon

•  Design/pipelineelements:(O.S.S.)– ObjectrepresentaNon– Similaritymeasure– Searchingprocess

22 17-Nov-15


1.  Problemstatement2.   Challenges3.  ObjectrepresentaNon4.  Singletargettracking5.  MulN-targettracking6.  Tips&references

23

Outline

17-Nov-15


Whatarethechallenges?

•  VariaNonsduetogeometricchanges (pose,arNculaNon,scale)

•  VariaNonsduetophotometricfactors (illuminaNon,appearance)

•  Occlusions•  Non-linearmoNon•  VerylimitedresoluNon,blurry

(standardrecogniNonmightfail)•  Similarobjectsinthescene

24

Seelivedemo17-Nov-15


Algorithmscommonissues

•  TrackiniNaNon&terminaNon•  Occlusionhandling•  Merging/switching•  Dridingduetowrongupdateofthetargetmodel

25

Seelivedemo17-Nov-15


Outline

1.  Problemstatement2.  Challenges3.   Objectrepresenta5on

1.  Low/mid/highlevelfeatures2.  Grid/Pyramid/Cascade3.  Patch/keypoints

4.  Singletargettracking5.  MulN-targettracking6.  Tips&references

26 17-Nov-15


ObjectrepresentaNon

•  Goal:wewantarepresentaNonthatis:

– DescripNveenoughtodisambiguatetargetVSbackground

– Flexibleenoughtocopewith:•  Scale•  Pose•  IlluminaNon•  ParNalocclusions

27 17-Nov-15


ObjectrepresentaNon

•  ObjectapproximaNon:– SegmentaNon/PolygonalapproximaNon– Boundingellipse/box– PosiNononly

•  Goal:Measureaffinity

28

ImagefromA.Yilmazet.Al:Objecttracking:Asurvey.ACMCompuNngSurveys,2006

17-Nov-15

Lecture 2 - !!

MeasuringAffinity

•  Ingeneral:

•  Examples:-  Distance:-  Intensity:-  Color:-  Texture:

•  Note:Canalsomodifydistancemetric

slidecredit:Forsyth&Ponce

FromLecture2

Pixels=>Regions


ObjectrepresentaNon:FromlighttousefulinformaNon

•  Low/mid/highlevelfeatures

30

histograms

17-Nov-15


Fromcs231

17-Nov-15


Low-levelfeatures

•  Integerresponses

32

0

5

10

15

Gradient orientation

Oc

cu

rre

nc

es

0

5

10

15


Oc

cu

rre

nc

es

HoG feature used in SIFT-like descriptor Haar feature used in SURF

17-Nov-15


Low-levelfeatures

•  Binaryresponses

33

01


Re

spo

nse

BRIEF/ORB

… 01


Re

spo

nse

FREAK

17-Nov-15


SpaNal-Frequency-BasedDescriptor

DifferenNalDescriptor

DistribuNon-BasedDescriptor

Vectorofpixelintensi5es

Histogramofpixelintensi5es

HOG

Haar-waveletresponses

Covarianceofsetoffeatures

Steerablefilters

Gaussianderiva5ves

Complexfilters

Momentinvariants

GLOH

ShapecontextSpinimages

SIFT

Gabor-waveletresponsesSURF

Low Performance High Performance

BinaryDescriptor

BRIEFORBBRISKFREAK

AbulkofLow-levelfeatures

Mikolajczyket.al."AperformanceevaluaNonoflocaldescriptors."PAMI2005

17-Nov-15


Recenttrend:CNNfeatures

35

Fromcs231n

17-Nov-15


ObjectrepresentaNon:Samplingstrategies

•  Grid/pyramid/cascadeofcoarse-to-fine

36

ImagefromL.Seidenari

17-Nov-15


ObjectrepresentaNon:Samplingstrategy

37

•  Localpatches/Keypoints[1]

[1]A.Alahiet.al.,Biologically-inspiredkeypoint,tobepublishedbyWiley


Outline

1.  Problemstatement2.  Challenges3.  ObjectrepresentaNon4.   Singletargettracking

1.   Bayesianes5ma5on2.   On-linelearning

5.  MulN-targettracking6.  Tips&references

38 17-Nov-15


Singletargettracking

•  FormulaNon–  Input:boundingboxatstarNngframe–  Output:nextboundingboxesacrossthenextframes

39 17-Nov-15


Singletargettracking-ProbabilisNctracking-

•  TrackingasaBayesiannetwork•  HiddenMarkovModel

•  MarkovassumpNons

40

ObservaNons

States

p(xk | x1:k−1) = p(xk | xk−1)

p(zk | x1:k ) = p(zk | xk )

17-Nov-15



•  RecursiveBayesfilters•  Findposterior•  Stateeq.(moNondynamics)•  ObservaNoneq.(image)

41

p(xk | z1:k )f (xk | xk−1)g(zk | xk )

17-Nov-15



•  SolvingBayesEquaNons–  Gaussian&Linear

•  Kalmanfilter[1]–  Gaussiannon-linear

•  ExtendedKalmanfilter–  Non-Gaussiannon-linear

•  MonteCarlomethods(CondensaNon[2])–  Hill-climbingonposterior

•  Mean-shid

[1]Kalman,RudolphEmil."AnewapproachtolinearfilteringandpredicNonproblems."JournalofFluidsEngineering,1960[2]Isard,Michael,andAndrewBlake."CondensaNon—condiNonaldensitypropagaNonforvisualtracking.”IJCV1998

17-Nov-15



•  Kernel-basedtracking[1]

•  Mean-shid–  Non-parametricfeaturespace–  LocatethemaximaofadensityfuncNon–  Colorhistogram/Bhauacharyya

44

[1]Comaniciu,Dorin,VisvanathanRamesh,andPeterMeer."Kernel-basedobjecttracking."PAMI(2003)

17-Nov-15


Mean-Shid

45

Region of interest

Center of mass

Mean Shift vector

SlidebyY.Ukrainitz&B.Sarel


Region of interest

Center of mass

Mean Shift vector

46


Mean-Shid


Region of interest

Center of mass

Mean Shift vector

47


Mean-Shid


Region of interest

Center of mass

48


Mean-Shid



•  Mean-shidPros:

–  Fast–  Noneedfortexture–  Tolerateforminorchangeofappearance

Cons:–  Onlyonehypothesis,nofallbackiftrackerislost–  AsinglehistogramdoesnotcapturevariaNonofappearance–  LimiteddiscriminaNvepowerwithbackground

49 17-Nov-15


Singletargettracking-On-linelearning-

•  DiscriminaNvemodeling(tracking-by-detecNon)

•  Learnandapplyadetectororpredictor

•  Challenges:– Whataretrainingdata?Labeled?–  Howtoavoiddrid?Handleocclusion?–  Howtocontrolcomplexity?

50 17-Nov-15



51

SlidefromCollins,PSU

17-Nov-15



•  On-linediscriminaNvelearning•  Oneshotlearning•  On-lineupdateoftheclassifier

52

FigurefromGrabnerandBischofCVPR06

17-Nov-15



•  Examplesofon-linediscriminaNvelearning– MulNpleInstanceLearning[1]–  KernelizedStructuredSVM[2]–  Combineshorttrack+detector[3]

53

[1]Babenko,Boris,Ming-HsuanYang,andSergeBelongie."VisualtrackingwithonlinemulNpleinstancelearning."CVPR2009[2]Hare,Sam,AmirSaffari,andPhilipHSTorr."Struck:Structuredoutputtrackingwithkernels.”ICCV2011[3]Kalal,Zdenek,KrysNanMikolajczyk,andJiriMatas."Tracking-learning-detecNon."PAMI2012 17-Nov-15



•  On-linediscriminaNvelearningPros:

–  Canhandleseveralappearancechanges–  Candetectaderfullocclusion

Cons:–  Candrid–  Learningisnottrivial

54 17-Nov-15


Outline

1.  Problemstatement2.  Challenges3.  ObjectrepresentaNon4.  Singletargettracking5.   Mul5-targettracking

1.   Formula5on2.   Graph-based

6.  Tips&references

55 17-Nov-15


MulN-targettracking

•  FormulaNon–  Input:asetofdetecNons(fromnextmoduleR-CNN)– Output:state(id)foreachdetecNons

56 17-Nov-15


•  DataassociaNon

•  Assignmentproblems

•  DiscretecombinatorialopNmizaNon

57

WhatisMulN-targettrackingabout?

17-Nov-15


MulN-targettracking

58


17-Nov-15


MulN-targettracking

59


17-Nov-15


MulN-targettracking

60


17-Nov-15


MulN-targettracking

61


17-Nov-15



MulN-targettracking

Non-opNmal!

17-Nov-15


Penn State

Robert Collins

VLPR 2012

Linear Assignment Problem

constraints that say

X is a permutation matrix

subject to:

The permutation matrix ensures that we only match up one

object from each row and from each column.

maximize:

minimize: note: alternately, we can minimize

costs rather than maximize weights

Mathematical Definition

Hungarianalgorithmfindstheop5malassignment

MulN-targettracking•  MathemaNcaldefiniNon

Wherewistheaffinitymatrixandxistheassignments


17-Nov-15


Penn State

Robert Collins

VLPR 2012

Greedy Solution to LAP

0.95 0.76 0.62 0.41 0.06

0.23 0.46 0.79 0.94 0.35

0.61 0.02 0.92 0.92 0.81

0.49 0.82 0.74 0.41 0.01

0.89 0.44 0.18 0.89 0.14

1 2 3 4 5

1

2

3

4

5

0.95 0.76 0.62 0.41 0.06

0.23 0.46 0.79 0.94 0.35

0.61 0.02 0.92 0.92 0.81

0.49 0.82 0.74 0.41 0.01

0.89 0.44 0.18 0.89 0.14

1 2 3 4 5

1

2

3

4

5

Score=3.77 Score=4.26

Greedy Solution Optimal Solution

No!

Greedy method is easy to program; quick to run; and

yields “pretty good” solutions in practice.

But it often does not yield the optimal solution.


MulN-targettracking

17-Nov-15


MulN-targettracking

65

•  Hungarianalgorithm

•  Pro– OpNmalsingleframeassignment

•  Con– NotopNmalformulNpleframes

17-Nov-15


MulN-targettracking

66

•  Goal:seekagloballyopNmalsoluNonacrossseveralframes

17-Nov-15


MulN-targettracking

67

ObjecNve:minimumcutmaximumflow

c( f ) = αi fi +∑ βij fij∑

argminf

c( f )

Whereαi,βij,γODarethecosts,andfitheflows

17-Nov-15



Costαibased:-DetecNonlikelihood

ObjecNve:minimumcostmaximumflow

argminf

c( f )

MulN-targettracking

17-Nov-15

Guest Lecture - A. Alahi - ! 69 22-April-15

Costβijbased:-spaNal-velocity


ObjecNve:minimumcutmaximumflow

argminf

c( f )

MulN-targettracking


MulN-targettracking•  OpNmalassignmentforfullyconnectedgraph[1]

[1]Zamir,AmirRoshanet.al."Gmcp-tracker:GlobalmulN-objecttrackingusinggeneralizedminimumcliquegraphs.”ECCV2012

17-Nov-15


42 million of collected trajectories

71

Density

17-Nov-15


Outline

1.  Problemstatement2.  Challenges3.  ObjectrepresentaNon4.  Singletargettracking5.  MulN-targettracking6.   Tips&references

72 17-Nov-15


Tips•  Modelcontext(apopularstrategysinceearly90sinCVcommunity)

•  DiscriminaNvelearning•  Sparsitydriven

73 17-Nov-15


Somereadings•  Trackingbymatching

–  Isard,Michael,andAndrewBlake."CondensaNon—condiNonaldensitypropagaNonforvisualtracking."InternaNonaljournalofcomputervision29.1(1998):5-28.

–  S.Oron,A.Bar-Hillel,D.Levi,andS.Avidan.LocallyOrderlessTracking.InCVPR,2012•  Trackingbymatchingwithanextendedappearancemodel

–  D.Ross,J.Lim,R.-S.Lin,andM.-H.Yang.IncrementalLearningforRobustVisualTracking.IJCV,77(1):125–141,2008.

•  Trackingwithsparsityconstraint–  W.Zhong,H.Lu,andM.-H.Yang.RobustObjectTrackingviaSparsity-based

CollaboraNveModel.InCVPR,2012.–  Kwon,Junseok,andKyoungMuLee."VisualtrackingdecomposiNon."Computer

VisionandPauernRecogniNon(CVPR),2010IEEEConferenceon.IEEE,2010.–  Li,Hanxi,ChunhuaShen,andQinfengShi."Real-Nmevisualtrackingusing

compressivesensing."ComputerVisionandPauernRecogniNon(CVPR),2011IEEEConferenceon.IEEE,2011.

74 17-Nov-15


Somereadings•  TrackingbydetecNons(MLapproach,usingadiscriminaNveclassificaNon)

–  Babenko,Boris,Ming-HsuanYang,andSergeBelongie."VisualtrackingwithonlinemulNpleinstancelearning."ComputerVisionandPauernRecogniNon,2009.CVPR2009.IEEEConferenceon.IEEE,2009.

–  Z.Kalal,K.Mikolajczyk,andJ.Matas,“Tracking-Learning-DetecNon,”PauernAnalysisandMachineIntelligence2011.

–  S.Hare,A.Saffari,andP.H.S.Torr.Struck:StructuredOutputTrackingwithKernels.InICCV,2011.

–  F.Henriques,R.Caseiro,P.MarNns,andJ.BaNsta.ExploiNngtheCirculantStructureofTracking-by-DetecNonwithKernels.InECCV,2012

–  Nebehay,Georg,andRomanPflugfelder."Consensus-basedmatchingandtrackingofkeypointsforobjecttracking."ApplicaNonsofComputerVision(WACV),2014IEEEWinterConferenceon.IEEE,2014.

75 17-Nov-15


Somereadings•  MulN-targettracking(dataassociaNon)

–  Berclaz,Jerome,etal."MulNpleobjecttrackingusingk-shortestpathsopNmizaNon."PauernAnalysisandMachineIntelligence,IEEETransacNonson33.9(2011):1806-1819.

–  Pirsiavash,Hamed,DevaRamanan,andCharlessC.Fowlkes."Globally-opNmalgreedyalgorithmsfortrackingavariablenumberofobjects.”(CVPR),2011

–  Zamir,AmirRoshan,AfshinDehghan,andMubarakShah."Gmcp-tracker:GlobalmulN-objecttrackingusinggeneralizedminimumcliquegraphs."ComputerVision–ECCV2012.SpringerBerlinHeidelberg,2012.343-356.

–  Liu,Jingchen,etal."Trackingsportsplayerswithcontext-condiNonedmoNonmodels."(CVPR),2013.

76 17-Nov-15

guest lecture: visual tracking - artificial...

Documents