Diverse Image-to-Image Translation via Disentangled RepresentationsHsin-Ying Lee*1 Hung-Yu Tseng*1 Jia-Bin Huang2 Maneesh Singh3 Ming-Hsuan Yang1,4

1University of California, Merced 2Virginia Tech 3Verisk Analytics 4Google Cloud AI

Code available!http://bit.ly/DRIT-ECCV18

Image-to-image translation

Challenges

Contributions

Disentangled representation for image-to-image translation (DRIT)

!"##

$%&

$%'

(%

$%&

$%'

(%

!"##

$)&

$)'

()

$)&

$)'

()

!adv#-./0./ !adv

1-234. !adv1-234.

5

6

7

8

95

96

1. Disentangled representation & cross cycle consistency

2. Diverse translation from unpaired data

3. Competitive performance on domain adaptation

Experimental resultsVisual results

60.876.7

42.3

39.223.3

57.7

25.3 19.6 16.237.5

74.7 80.4 83.862.5

Realism:Preference percentage

Training

1. Lack of aligned training pairs

2. Multiple possible outputs given singe input image

Winter Summer

()

$%'

$)&

()

E(0,1)…

$%'

Example guided translation Randomly sampled translation

Photo Artistic portrait

% domain

) domain

Loss

Prior distribution

Content encoder$'

Attribute encoder$&

Generator(

Diversity: LPIPS scoreDRIT (ours)Cycle/Bicycle

UNITCycleGAN

Real images

Domain adaptation

References[1] Liu et al. Unsupervised Image-to-Image Translation Networks. In NIPS, 2017[2] Zhu et al. Unpaired Image-to-Image Translation using Cycle-Consistent[2] Adversarial Networks. In ICCV, 201[3] Zhu et al. Toward Multimodal Image-to-Image Translation. In NIPS, 2017

Other training loss

(%

$%'

$%&

!KLE(0,1)

!"recon

()

$%'

E(0,1) !"latent

!adv1-234.

$)&

(a) MNIST-M (b) LineMODSource

Targetexample

Randomly sampledtranslation

Paired data Unpaired data

One-to-one Pix2pix [Isola et al.]DiscoGAN [Kim et al.]CycleGAN [Zhu et al.]

UNIT [Liu et al.]

One-to-many BicycleGAN [Zhu et al.]Pix2pixHD [Wang et al.]

DRIT (Ours)1

2

2

Real images