StefanosZafeiriouThursday19,January2017

FromLukas-KanadetoMnemonicDescentandDeepDenseShapeRegression:Abriefhistoryof(deformable)imagealignment

Image/Objectalignment

2

Image/Object alignment (or registration) is the process oftransformingdifferentsetsofdataintoonecoordinatesystem

•Globalalignment

• Alignment using object parts (i.e., semantically meaningfulcomponents)

DeformableObjectAlignment

3

DeformableModelsaimtomodeltheshapeandappearancevariationsofanobjectclass.

WhatareDeformableModels?

Trainedusing“In-the-Wild”data(i.e.imagescapturedundertotallyunconstrainedconditions)

4

DeformableModelsaimtomodeltheshapeandappearancevariationsofanobjectclass.

WhatareDeformableModels?

Trainedusing“In-the-Wild”data(i.e.imagescapturedundertotallyunconstrainedconditions)

4

6DeepfacebyFacebook,inCVPR2014

RecentsuccessesinComputerVisionFacerecognition/verification~97%inLFW

•Shouldthissuccessreallyattributedindeeplearning?•Thenetworkwithoutalignmentproduces:87.9%

Trainedon4.4millionlabeledfacialimages

•Averysimpleclassifieronaligneddata:~92%

•

•InthesameyearKitttler’sgroupreported:~96%(MRFalignmentandsimplefeatures)

7

Facebook’sDeepface

The success is mainly due to an elaborate image alignment and imagenormalization(frontalization)procedure

Google’sFaceNet

• Betterperformancewithouttheneedtoapplysuchelaboratealignment(stillalignmentimprovesperformance)

• Butitistrainedontherecordnumberof260,000,000images!!!!

8

Whyaretheseimagesdisturbing?

11/12/169

HolisticObjectalignment

10

Max(minim)ize

W is the motion model, p are the motion parameters

Lukas-Kanade(LK)inanutshell

11

• Findmotionparametersbysolving

• Exampleofwarpsandparameters(n-params):

Affine(6-parameters):

Translation(2-parameters):

• xisthetestimageandmthetemplate(N-pixels)

[1]B.Lucas,andT.Kanade."Aniterativeimageregistrationtechniquewithanapplicationtostereovision."IJCAI.Vol.81.No.1.1981.

LK:Forward-Additive

12

Jacobian: Derivativeofthewarp:

Errorimage:

Step1:Linearisearoundthecurrentestimate

Step2:Solve

Step3:Update

13

JacobianandHessiancanbepre-computed

LK:InverseCompositional(IC)Step1:Linearisearoundzero(parametersinthetemplate)

Step2:Update

Step3:Update

[1]Baker,S.andMatthews,I.."Lucas-kanade20yearson:Aunifyingframework."Internationaljournalofcomputervision56.3(2004):221-255.

LinearComplexity!!!!

FromLKtostatisticaldeformablemodels

14

ActiveAppearanceModels(AAMs)

Timeline

1998

2004

2008

2009

2013

2014

2015

2016

[Cootesetal.,1998]AAMswithsinglestepregressionfitting!

[Matthews&Baker,2004]Project-OutInverseCompositionalSimultaneousInverseCompositional[Papandreou&Maragos,2008]AlternatingInverseCompositional[Ambergetal.,2009][Tzimiropoulosetal.,2013]Project-OutForwardCompositional [Asthana,Zafeiriouetal.,2013]

[Xiong&DelaTorre,2013][Caoetal.,2014]Cascadedregressionfittingframework

[Alabort&Zafeiriou,2014,2016]Unifiedframeworkforcompositionalfitting[Antonakosetal.,2015]ActivePictorialStructures

[Trigeorgisetal.,2016]MnemonicDescentMethod

GenerativeModels DiscriminativeModels

[Antonakosetal.,2016]AdaptiveCascadedRegression 23

Timeline

1998

2004

2008

2009

2013

2014

2015

2016

[Cootesetal.,1998]AAMswithsinglestepregressionfitting!

[Matthews&Baker,2004]Project-OutInverseCompositionalSimultaneousInverseCompositional[Papandreou&Maragos,2008]AlternatingInverseCompositional[Ambergetal.,2009][Tzimiropoulosetal.,2013]Project-OutForwardCompositional [Asthana,Zafeiriouetal.,2013]

[Xiong&DelaTorre,2013][Caoetal.,2014]Cascadedregressionfittingframework

[Alabort&Zafeiriou,2014,2016]Unifiedframeworkforcompositionalfitting[Antonakosetal.,2015]ActivePictorialStructures,FeaturebasedAAMs [Trigeorgisetal.,2016]

MnemonicDescentMethod

GenerativeModels DiscriminativeModels

[Antonakosetal.,2016]AdaptiveCascadedRegression 24

17

MeanShapeShapesafterprocrustes

PCAonthatspace

FromLKtoAAMs

18

Warpusingmeanshape

PCAonthatspace

FromLKtoAAMs

[1]Cootes,T.,Edwards,G.andTaylor,C."Activeappearancemodels."IEEET-PAMI23.6(2001):681-685.

• Thewarpingfunctionisdrivenbytheshape.• Insteadofthetemplateimagewehavealineartexturemodel.

• Theparametersthatdrivethemodelare(a)theweightsofthelinearshape,(b)aglobalsimilaritytransformand(c)theweightsofthelineartexturemodel.

19

FromLKtoAAMs

Statisticalparametricmodeloftheshapeandappearanceofanobject.

FromLKtoAAMs

Recovershapeandappearanceparametersthatminimizethereconstructionerrorofthewarpedimage.

ShapePCA AppearancePCA

[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 28

21

• Notrobusttooutliers(occlusions,illumination,castshadowsetc.)• Replacetheleastsquaresfunctionwithrobusterrorfunctions•Noisemodelforoutliershardtodefine-Aretheyalwaysrobust?•Approximationsforefficiency

RobustLK-AAMs

[1]Baker,S.andMatthews,I.."Lucas-kanade20yearson:Aunifyingframework."Internationaljournalofcomputervision56.3(2004):221-255.[2]Dowson,N.,andR.Bowden."Mutualinformationforlucas-kanadetracking(milk):Aninversecompositionalformulation."IEEET-PAMI30.1(2008):180-185.[3]Evangelidis,G.,&Psarakis,E.(2008).Parametricimagealignmentusingenhancedcorrelationcoefficientmaximization.IEEET-PAMI,30(10),1858-1865.[4]Lucey,Simon,etal."Fourierlucas-kanadealgorithm."IEEET-PAMI35.6(2013):1383-1396.

GivenanimageofsizeHxW,thedensefeatureextractionfunctionisdefinedas:

whereDisthenumberofchannels.

Feature-BasedLK-AAMs

[1]Antonakos,Alabort-i-Medina,TzimiropoulosandZafeiriou,“Feature-BasedLucas-KanadeandActiveAppearanceModels”,IEEETIP,2015[2]Tzimiropoulos,Georgios,StefanosZafeiriou,andMajaPantic."Robustandefficientparametricfacealignment.",ICCV,2011(oral)[3]Tzimiropoulos,G.,Argyriou,V.,Zafeiriou,S.,&Stathaki,T.(2010).RobustFFT-basedscale-invariantimageregistrationwithimagegradients.IEEET-PAMI,32(10),1899-1906.

19

HOG,SIFT,...

multi-channelDenseimagefeatures

Patch-basedwarping

patchesextraction multi-

channelpatches

Feature-BasedLK-AAMs

[1]Antonakos,Alabort-i-Medina,TzimiropoulosandZafeiriou,“Feature-BasedLucas-KanadeandActiveAppearanceModels”,IEEETIP,2015

27

Feature-BasedAAMs● Warpthefeaturesimages

● Extractfeaturesonthewarpedimage

20

AAM:Project-OutInverseCompositional

distancewithinA(0forallp)

distancetoA(i.e.distancewithinorthogonalcomplement)

linearize

with and

Project-out

Inverseandlinearization

[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 30

AAM:Project-OutInverseCompositional

[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 31

AAM:AlternatingInverseCompositional

withand

Inverseandlinearization

Project-Out

[1]G.PapandreouandP.Maragos.“Adaptiveandconstrainedalgorithmsforinversecompositionalactiveappearancemodelfitting”,CVPR,2008.

linearize

32

AAM:Project-OutInverseCompositional

[1]G.PapandreouandP.Maragos.“Adaptiveandconstrainedalgorithmsforinversecompositionalactiveappearancemodelfitting”,CVPR,2008. 33

FromLKtoAAMs:What’snext?

29

AAMs:LKalgorithmwithashapedrivenmodelandaspecialweightedleast-squarescostfunction(akindofMahalanobis)

WhatkindofMahalanobis(whytheaboveweights?)

Howtoachievereal-timeperformance?

ActivePictorialStructures

30[1]Antonakos,AlabortiMedina,Zafeiriou,ActivePictorialStructures,CVPR2015

PictorialStructuresQuickOverview

➜ Gaussiandistributionforthepatch-basedappearanceofeachlandmark

➜ Gaussiandistributionforeachpairwiserelativelocationbetween

landmarksbasedonatree(acyclicgraph).

➜ Costfunction

○ Mahalanobisfortheappearanceofeachlandmark○ Spring-likedeformationcostbetweenpairsoflandmarks

➜ Globaloptimumusinganefficientdynamicprogrammingalgorithm.[1]P.FelzenszwalbandD.Huttenlocher.“PictorialStructuresforobjectrecognition”,IJCV2005.[2]P.FelzenszwalbandD.Huttenlocher.“Distancetransformsofsampledfunctions”,2004.[3]P.Felzenszwalb,etal.“Objectdetectionwithdiscriminativelytrainedpart-basedmodels”,IEEET-PAMI2010.[4]X.ZhuandD.Ramanan.“Facedetection,landmarklocalizationinthewild”,CVPR2012.[5]Fischler,M.A.andElschlager,R.A.1973.Therepresentationand1376matchingofpictorialstructures.IEEETransactionsonComputer,137722(1):67–92.

ActivePictorialStructures

[1]E.Antonakos,J.Alabort-i-Medina,S.Zafeiriou.“ActivePictorialStructures”,CVPR2015.

ShapePCA

Blocksparseprecisionmatrix

AppearanceGMRF

Blocksparseprecisionmatrix

DeformationGMRF

ThecostfunctionconsistsoftheappearanceMahalanobisdistance,thedeformationpriorandaweighthyperparameterbetweenthem.

36

ActivePictorialStructures

32

ActivePictorialStructures

[1]E.AntonakosandS.Zafeiriou.“ActivePictorialStructures”,CVPR2015.

• WeightedInverseCompositionalGauss-NewtonwithfixedJacobianandHessian• Fastestinversecompositionalfittingwithfastestconvergencerate• Deformationpriormakesmodelveryrobust

pre-computed pre-computed

residual

38

ActivePictorialStructures

[1]E.Antonakos,J.Alabort-i-MedinaandS.Zafeiriou.“ActivePictorialStructures”,CVPR,2015. 39

ActivePictorialStructures

DiscriminativeAlignment

41

[1]Xiong,X.,andF.DelaTorre."Superviseddescentmethodanditsapplicationstofacealignment."CVPR,2013.[2]Asthana,A.,Zafeiriou,S.,Cheng,S.,&Pantic,M.Robustdiscriminativeresponsemapfittingwithconstrainedlocalmodels,CVPR,2013[3]Asthana,A.,Zafeiriou,S.,Cheng,S.,&Pantic,M.Incrementalfacealignmentinthewild.InCVPR2014[4]Asthana,A.,Zafeiriou,S.,Tzimiropoulos,G.,Cheng,S.,&Pantic,M.Frompixelstoresponsemaps:Discriminativeimagefilteringforfacealignmentinthewild.IEEET-PAMI,37(6),1312-1320.2015

LearningtheDescentDirections

Gauss-Newton

Newton

InsteadofcomputingtheJacobianfromtheimageateachiterationwhynotpre-computeit?

GeneralDescentDirection

42

38

DiscriminativeAlignment

Groundtruth

Perturbation

Solve:

Learnthemapping

39

Applyingthemapping

45

AdaptiveCascadedRegression

Costfunctionwithrespecttoparametricshapemodel.

Motivation

47

Costfunctionwithrespecttoparametricshapemodel.

Realisticinitializationfromfacedetector.

Motivation

48

Costfunctionwithrespecttoparametricshapemodel.

Realisticinitializationfromfacedetector.

Gauss-Newtonfailsduetobadinitialization.

Motivation

49

Costfunctionwithrespecttoparametricshapemodel.

Realisticinitializationfromfacedetector.

Gauss-Newtonfailsduetobadinitialization.

Cascadedregressionmovestowardsthecorrectdirectionbutnotcloseenough.

Motivation

50

Costfunctionwithrespecttoparametricshapemodel.

Realisticinitializationfromfacedetector.

Gauss-Newtonfailsduetobadinitialization.

Cascadedregressionmovestowardsthecorrectdirectionbutnotcloseenough.

ApplyingGauss-Newtondescentdirectionsrightaftercascadedregressiondirectionsreachesoptimum!

Motivation

51

AdaptiveCascadedRegression

DiscriminativeCascadedRegressiondescentdirections

GenerativeGauss-Newtondescentdirections

AdaptiveCascadedRegression

Linearcombinationofdescentdirectionswithadaptiveweights:

Adaptivedescentsteps

Projected-OutResidual

with

52

Matrixwithgroundtruthshapeparameters

Matrixwithparametersfromnoisyshapes

Matrixwithprojected-outfeaturevectors

[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.

AdaptiveCascadedRegression

AdaptiveCascadedRegression

[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.54

AdaptiveCascadedRegression

[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.

ConsistentlylowererrorcomparedtocascadedregressiondiscriminativeandtoGauss-Newtongenerativemodel.

55

AdaptiveCascadedRegression

[ACR]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.[300W1]E.Zhouetal.“Extensivefaciallandmarklocalizationwithcoarse-to-fineconvolutionalnetworkcascade”,ICCV-W2013.[CFSS]S.Zhuetal.“Facealignmentbycoarse-to-fineshapesearching”,CVPR2015.[PO-CR]G.Tzimiropoulos.“Project-Outcascadedregressionwithanapplicationtofacealignment”,CVPR2015.[300W2]J.Yanetal.“Learntocombinemultiplehypothesesforaccuratefacealignment”,ICCV-W2013.[ERT]V.KazemiandJ.Sullivan.“Onemillisecondfacealignmentwithanensembleofregressiontrees”,CVPR2014.[Intraface]X.XiongandF.DelaTorre.“SupervisedDescentMethodanditsApplicationstoFaceAlignment”,CVPR2013.[Chehra]A.Asthana,S.Zafeiriou,S.Cheng,andM.Pantic.“Incrementalfacealignmentinthewild”,CVPR2014. 56

AdaptiveCascadedRegression

52

MnemonicDescent

End-to-EndTrainingofannon-linearcascaderegressionmethod

54

• Why HoGs and not any other features? (Learn a non-lineartransformforfeatures)

• Theprocessbearssimilaritieswithadynamical system.Whynotmodel itexplicitlymodel? (Learnanon-lineardynamicalsystemthatmodelsthecascade).

MnemonicDescentMethod

55

Trigeorgis,G.,Snape,P.,Nicolaou,M.A.,Antonakos,E.,&Zafeiriou,S.(2016,June).MnemonicDescentMethod:Arecurrentprocessappliedforend-to-endfacealignment.CVPR’16,LasVegas,NV,USA.

MnemonicDescentMethod

56

• Updates

57

MnemonicDescentMethod

Clusteringofthestate

58

MnemonicDescentMethod

3DMM

11/12/1659

3DMM

11/12/1660

3DMM“in-the-wild”

11/12/1661

Fittinga3DMM

11/12/1662

DeepLearning

11/12/1663

IntroducingMenpo• OpenSourcePythonpackageforgenerativeanddiscriminativemodelling

• UnifiedframeworkforAAMs,CLMs,&SDMs

• Underactivedevelopment,MorphableModelscoming

• www.menpo.org

• github.com/menpo76

ThankYou