adaptive deep learning for visual understanding › wp-content › uploads › 2018 › 06 ›...

Adaptive)Deep)Learning)for)VisualUnderstanding

Kate)SaenkoBoston)University

Harvard&University,&January&22&2019

Boston&University Slideshow*Title*Goes*Here

Prof.*Kate*Saenko


AIR :*AI*Research*at*BU*

AIR


Prof.*Kate*Saenko


Goal:*teach*machines*to*see,$talk,$actResearch*in*my*lab*********************************************************************************************************** Kate*Saenko

A baseball game in progress with the batter up to plate

A man is riding a bicycle

Find*“window*upper*right”

A: skateboard

Q:*What*is*the*child*standing*

on?

Find*the*moment*when*“girl*looks*

up*at*the*camera*and*smiles”

pick*up*what*you*see

A

I

Vision Lang&uage

Action

“Go*out*of*the*bedroom,*down*

the*stairs,*turn*left*and*stop*at*

the*dining*table”

deep$learning$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$human$learning

?

adaptiveexplainablemodular

train test

HUMAN

● learns/from/a/single/example● generalizes/knowledge

adaptive,/explainable,/modular

train test

SUPERVISED0DEEP0LEARNING

● learns0from010000’s0examples● fails0on0new0domains

adaptive,0explainable,0modular

imagenet.org

HUMAN

● can*explain*decisions● grounds*language*in*world

adaptive,*explainable,*modular

why*is*this*a*dog?tail,*four*legs,*...

DEEP$LEARNING

● cannot$explain$decisions● black$box

adaptive,$explainable,$modular

why$is$this$a$dog?

HUMAN

● disentangles0properties● compositional

adaptive,0explainable,0modular

dog0running cat0sitting cat0running

adaptive,)explainable,)modular

dog)running cat)sitting ??

DEEP)LEARNING

● entangles)properties● not)compositional


this2talk

Kate%Saenko,%Boston%University

Domain'shift

“Dataset'Bias”“Domain'Shift”

What%your%net%is%trained%on What%it’s%asked%to%label


Example(shift

train test

from+simulation++++++++++++++++++++++++++++++++++++++++++++++++to+reality


Shift&from&simulation&to&reality

road

sidewalk

people

treesca

r

car

sidewalk

sidewalk

road

trees

Input1Image True1Segmentation

Model1Output

grass


Solution:)Domain)Adaptation

road

sidewalk

people

trees

car

car

sidewalk

sidewalk

road

trees

Adapted/Model/Output

Input/Image True/Segmentation

Model/Output

grass


Applications+of+Domain+AdaptationFrom+dataset+to+dataset

From+simulated+to+real+control

From+RGB+to+depth

From+CAD+models+to+real+images+


Source'Data'+'Labelsbackpack chair

Encoder'CNN Classifierloss

Adversarial*domain*adaptation

Classifier



Unlabeled'Target'Data

?

Encoder'CNN



Classifier




?

Encoder'CNN



Adversarial'loss

Classifier

Discriminator




?

Classifierloss


Adversarial'loss

ClassifierEncoder'CNN

Encoder'CNNDiscriminator

Classifier


Results'on'Digits'Classification

● Domain(Confusion(Loss((Tzeng(2015)

● Adversarial(Domain(Alignment((ADDA)((Tzeng(2017)

Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep Transfer Across Domains and Tasks” ICCV 2015Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.

Domain(Confusion(Loss((Tzeng(2015)GradReversal((Ganin(&(Lempitsky(2015)

ADDA((Tzeng(2017)




Problem:)ambiguous)features

Classifier




?

Encoder'CNN



Classifier




?

Encoder'CNN



Adversarial'loss

Classifier

Discriminator




?

Classifierloss


Adversarial'loss

ClassifierEncoder'CNN

Encoder'CNNDiscriminator


Source' Target'

Before'Adaptation Adapted



Source' Target'

Goal:&avoid&generating&ambiguous&featuresBefore'Adaptation Adapted

Adversarial'Dropout'Regularization,'Kuniaki'Saito, Yoshitaka'Ushiku, Tatsuya'Harada, Kate'Saenko,'ICLR'2018


Solution:)use)the)decision)boundaryTrain&a&critic&(C)&(=discriminator)&that&can&detect&target&samples&near&decision&boundaryTrain&a&generator&(G)&that&can&fool&the&critic

Slightly&change&the&boundary&and&measure&the&change&of&p(y|x)!&&(sensitivity)

Samples&near&the&boundary&have&larger&sensitivity

Original&Boundary

Adversarial&Dropout&Regularization,&Kuniaki&Saito, Yoshitaka&Ushiku, Tatsuya&Harada, Kate&Saenko,&ICLR&2018


Data FeatureClassifiers

Sampling2by2dropout

Predictions

Measure2Sensitivity

Fix G"and2train C"to2maximize2d(p1,p2)"for2target2samples.Train G"and C"to2minimize2Cross2Entropy2for2source2samples.For2k2=1:nFix2C"and2train G"to2minimize2d(p1,p2)"for"target.

Adversarial"Dropout"Regularization

Adversarial2Dropout2Regularization,2Kuniaki2Saito, Yoshitaka2Ushiku, Tatsuya2Harada, Kate2Saenko,2ICLR22018


Results'on'Digits'Classification

● Adversarial+Dropout+Regularization+(ADR)+(Saito+et+al+2018)

Adversarial+Dropout+Regularization,+Kuniaki+Saito, Yoshitaka+Ushiku, Tatsuya+Harada, Kate+Saenko,+ICLR+2018

Domain+Confusion+Loss+(Tzeng+2015)GradReversal+(Ganin+&+Lempitsky+2015)

ADDA+(Tzeng+2017)


Adaptation)for)semantic)segmentation

Adapted

Ground,TruthInput,image

Source5only,model


Pixel&to&pixel*adaptation


Strong'weak,feature,alignment

Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018


ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline

Domain%shift:%from%PASCAL%VOC%to%CLIPART


Evidence(for(target(domain(label((GradCAM)(shows(that(the(feature(extractor(seems(to(deceive(the(domain(classifier(in(regions(with(car.

Domain(shift:(from(Grand(Theft(Auto(game(to(Cityscapes

target

source


ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline

Domain%shift:%from%Cityscapes%to%Foggy%Cityscapes

CNNs learn to predict pneumonia by detecting hospital which took the image

Variable(generalization(performance(of(a(deep(learning(model(to(detect(pneumonia(in(chest(radiographs:(A(cross8sectional(study.(Zech%JR1,%Badgeley%MA2,%Liu%M2,%Costa%AB3,%Titano%JJ4,%Oermann%EK3.%https://www.ncbi.nlm.nih.gov/pubmed/30399157

● Study%on%detecting%pneumonia%using%158,323%chest%radiographs

● CNNs%robustly%identified%hospital%system%and%department%within%a%hospital

● CNN%has%learned%to%detect%a%metal%token%that%radiology%technicians%place%on%the%patient%in%the%corner%of%the%image%field%of%view%at%the%time%they%capture%the%image

https://www.ncbi.nlm.nih.gov/pubmed/?term=Zech%20JR%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

https://www.ncbi.nlm.nih.gov/pubmed/?term=Badgeley%20MA%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

https://www.ncbi.nlm.nih.gov/pubmed/?term=Liu%20M%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

https://www.ncbi.nlm.nih.gov/pubmed/?term=Costa%20AB%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

https://www.ncbi.nlm.nih.gov/pubmed/?term=Titano%20JJ%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

https://www.ncbi.nlm.nih.gov/pubmed/?term=Oermann%20EK%5BAuthor%5D&cauthor=true&cauthor_uid=30399157

RISE:&randomly&mask&input,&measure&output

42

P[shark]

shark

RISE:&Randomized&Input&Sampling&for&Explanation&of&Black:box&Models,&Petsiuk,&Das,&Saenko,&BMVC&2018

Explainable*Neural*Computation*via*Stack*Neural*Module*Nets*Networksinput:*There*is*a*small*gray*block;*are*there*any*spheres*to*the*left*of*it?

Hu,$Andreas,$Darrell,$Saenko,$Explainable$Neural$Computation$via$Stack$Neural$Module$Networks,$ECCV’18

input*image

Disentangling*properties*via*generation

A"Two&Stream"Variational"Adversarial"Network"for"Video"Generation

generated"video


A*Two2Stream*Variational*Adversarial*Network*for*Video*Generation,*arxiv*2018


Disentangling*properties*for*domain*transfer

adaptive deep learning for visual understanding › wp-content › uploads › 2018 › 06 ›...

Documents