generative adversarial nets: applications and...
TRANSCRIPT
![Page 1: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/1.jpg)
Generative Adversarial Nets: Applications and Extensions
Wangmeng Zuo
Vision Perception and Cognition CentreHarbin Institute of Technology
![Page 2: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/2.jpg)
LeCun, NIPS 2016
• Reinforcement learning (cherry)
• Supervised learning (Chocolate)
• Unsupervised/Predictive learning (Cake)• Generative adversarial nets (GAN)
![Page 3: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/3.jpg)
For Most Application Tasks
• For most applications, GANs only serve as the accessories to the existing solutions.
• How to Make Latte Art (i.e. improve the trainability of generator)
• How to make a perfect Latte Coffee (i.e. incorporate with other models for solving real problems)
GAN
Other Learning Models
![Page 4: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/4.jpg)
Content
• Improve the trainability of GANs: An Application Perspective• Theoretical solution
• Incorporating with other learning models
• Designing generator based on signal/image characteristics
• Applications• Adversarial learning
• Low level vision
• Domain adaptation
• Image translation
![Page 5: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/5.jpg)
Improve the trainability of GANs
![Page 6: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/6.jpg)
Generative Adversarial Networks (Goodfellow et al., NIPS 2014)
• Update the generator to generate more realistic image
• Update the discriminator to discriminate the synthetic images from real ones
![Page 7: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/7.jpg)
Mode Collapse
• D in inner loop: convergence to correct distribution
• G in inner loop: place all mass on most likely point
![Page 8: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/8.jpg)
Let's first turn to supervised deep learning
• Unprecedented successes in:• Image classification
• Image denoising, image super-resolution
• ...
• Can we exploit these achievements to improve GAN training?• How to train a good generator (the later half of image restoration?)
• How to train a good discriminator (classification?)
![Page 9: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/9.jpg)
Auto-encoder
• Auto-encoder
• Denoising auto-encoder
![Page 10: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/10.jpg)
Variational AutoEncoder
• Variational AutoEncoder
• Relaxation of discrete variables
![Page 11: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/11.jpg)
VAE/GAN (Larsen et al., ICML 2016)
• VAE
• GAN
• VAE/GAN
![Page 12: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/12.jpg)
Classifier Discriminator
• Na Lei, Kehua Su, Li Cui, Shing-Tung Yau, David Xianfeng Gu, A Geometric View of Optimal Transportation and Generative Model, Arxiv 2017.
![Page 13: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/13.jpg)
Nguyen et al., NIPS 2016
• Optimize the hidden code input (red bar) of a deep image generator network (DGN) to produce an image that highly activates h
![Page 14: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/14.jpg)
InfoGAN (Chen et al., NIPS 2016)
• GAN
• InfoGAN (Chen et al., NIPS 2016)• Input: z, c
• Interpretable and disentangled representations
• Easy to train
![Page 15: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/15.jpg)
AC-GAN (Odena et al., ICML 2017)
• Class-conditional image synthesis with Auxiliary Classifier GANs
• The log-likelihood of the correct source:
• The log-likelihood of the correct class:
![Page 16: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/16.jpg)
Arbitrary Facial Attribute Editing
• One model for all tasks (He et al., Arxiv 2018)
![Page 17: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/17.jpg)
A Favorable Framework
• Auto-encoder
![Page 18: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/18.jpg)
AttGAN
![Page 19: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/19.jpg)
Extension for attribute style manipulation
![Page 20: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/20.jpg)
Single task
![Page 21: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/21.jpg)
Multi-task
![Page 22: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/22.jpg)
Continuous attribute
![Page 23: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/23.jpg)
Attribute Style Manipulation
•
![Page 24: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/24.jpg)
Take home message
• Incorporating auto-encoder to improve the trainability of generator;
• Incorporating deep classification model to improve the trainability of discriminator
![Page 25: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/25.jpg)
Let's then turn to the objective of GANs
• Image generation
• What's the characteristics of an image• Multi-scale property
• Manifold property
• What makes a high quality image• Deep image prior
• Deep image quality assessment
![Page 26: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/26.jpg)
LAPGANs (Denton et al., NIPS 2015)
![Page 27: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/27.jpg)
LAPGANs (Denton et al., 2015)
![Page 28: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/28.jpg)
Stack-GAN (Zhang et al., ICCV 2017)
![Page 29: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/29.jpg)
Stack-GAN (Zhang et al., ICCV 2016)
• Stage-I GAN
• Stage-II GAN
![Page 30: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/30.jpg)
Cascaded Refinement Networks (Chen & Koltun, ICCV 2017)
• CRN: not rely on adversarial training
![Page 31: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/31.jpg)
Manifold property (Benaim & Wolf, NIPS 2017)
• Distance Constraints
• Self-distance Constraints
![Page 32: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/32.jpg)
Total Variation
• Deep feature visualization
• Total variation (TV) regularization
• Better (deep) image prior?
![Page 33: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/33.jpg)
Insight from deep image denoising
• DnCNN for image denoising (Zhang et al., TIP 2017)
• For noisy image,
• For clean image,
• Perceptual regularization (Li et al., Arxiv 2016)
ˆ ( ; )CNN x y y
( ; )CNN y y x2 2( ; )CNN mn y
2( ; ) 0CNN y
2( ; )CNN y
![Page 34: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/34.jpg)
Deep image prior (Ulyanov et al., CVPR 2018)
• Energy
• Image restoration
• A randomly-initialized neural network can be used as a handcrafted prior
• The structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning
![Page 35: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/35.jpg)
Deep Features as a Perceptual Metric (Zhang et al., CVPR 2018)
• Perceptual loss
• Deep features outperform all previous metrics by huge margins.
• This result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised).
• Deep Non-reference Image Quality Assessment?
![Page 36: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/36.jpg)
Take home message
• Exploiting image property to improve GANs
• Developing deep models/GANs for better revealing image priors/quality
• Object-oriented design
![Page 37: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/37.jpg)
Applications
![Page 38: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/38.jpg)
Adversarial learning (Szegedy et al., ICLR 2014)
• Deep neural networks learn input-output mappings that are fairly discontinuous to a significant extent.
• We can cause the network to misclassify an image by applying a certain hardly perceptible perturbation, which is found by maximizing the network’s prediction error.
2018-4-22
![Page 39: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/39.jpg)
Intriguing properties of neural networks (Szegedy et al., ICLR 2014)
2018-4-22
![Page 40: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/40.jpg)
Deep Neural Networks are Easily Fooled (Nguyen et al., CVPR 2015)
2018-4-22
![Page 41: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/41.jpg)
2018-4-22
> 99.6%confidences
![Page 42: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/42.jpg)
Adversarial Attacks and Defences Competition (Kurakin et al., Arxiv 2018)
• 1st place in defense track: team TsAIL
• Team members: Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu and Xiaolin Hu.
• Solution: Denoising U-net
![Page 43: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/43.jpg)
Adversarially-augmented training (Simon-Gabriel et al., Arxiv 2018)
• Adversarially-augmented training
• Replacing strided by average-pooling layers
• Increase generalization performance
![Page 44: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/44.jpg)
Object detection: A-Fast-RCNN (Wang et al., CVPR 2017)
•
![Page 45: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/45.jpg)
Visual tracking
• CVPR 2018
• VITAL: VIsual Tracking via Adversarial Learning
• SINT++: Robust Visual Tracking via Adversarial Hard Positive Generation
![Page 46: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/46.jpg)
Low level vision
• SRGAN for super-resolution
![Page 47: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/47.jpg)
![Page 48: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/48.jpg)
DSLR-Quality Photos on Mobile Devices (Ignatov et al., ICCV 2017)
![Page 49: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/49.jpg)
• Color loss
• Texture loss
• Content loss
• TV regularizer
• Discriminator
![Page 50: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/50.jpg)
WESPE: Weakly Supervised Photo Enhancer (NTIRE 2018)
• Only require two distinct datasets
![Page 51: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/51.jpg)
Image inpainting: more freedom and non-uniqueness
![Page 52: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/52.jpg)
Context-encoders (Pathak et al., 2016)
• The first key: Auto-encoder
![Page 53: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/53.jpg)
Problem with auto-encoder
• Information bottleneck
![Page 54: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/54.jpg)
Adversarial loss is helpful
• But remains limited ...
![Page 55: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/55.jpg)
Analyzing U-Net (Ronneberger et al., 2015)
• Fine-details
• Unfortunately, also not work for inpainting
![Page 56: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/56.jpg)
Return to traditional patch-based inpainting
• Patch processing order
• PatchMatch
![Page 57: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/57.jpg)
CNN and Patch-based Solutions are Complementary
• CNN-based solution• Poor texture
• Better structure
• Patch-based solution• Better details
• Poor structure
• Can we combine them in an end-to-end learning framework?
Context-encoders
![Page 58: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/58.jpg)
CNN architecture
![Page 59: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/59.jpg)
Objective and learning
• Objective
• Learning
![Page 60: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/60.jpg)
Results
• Speed• MNPS: 40mins -> 40s
• Ours: 82 ms
• PSNR
![Page 61: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/61.jpg)
Random mask
![Page 62: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/62.jpg)
Real images
![Page 63: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/63.jpg)
Guided face enhancement (Li et al., Arxiv 2018)
Film Restoration, Smartphones
![Page 64: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/64.jpg)
Challenges
• 1. Blind enhancement: the degradation model is sophisticated and unknown
• blur, downsampling, noise, compression
• 2. The guided and degraded images are of different pose, expression and illumination
![Page 65: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/65.jpg)
Challenge 1
• Train on realistic synthetic degraded images, test on real degraded image
• The degradation model:
![Page 66: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/66.jpg)
Challenge 2: GFRNet
•
![Page 67: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/67.jpg)
Model and losses for WarpNet
• Landmark loss
• TV regularization
![Page 68: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/68.jpg)
Model objective
• Reconstruction loss
• Adversarial loss
• Objective
![Page 69: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/69.jpg)
Appearance Flow
![Page 70: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/70.jpg)
Results
MDnCNN MARCNN MDeblurGAN Ours
![Page 71: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/71.jpg)
More images
![Page 72: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/72.jpg)
Video
![Page 73: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/73.jpg)
Domain Adaptation
• Domain Adaptation: learning from a (labeled) source data distribution a well performing model on a different (but related) (labeled or unlabled) target data distribution (wikipedia)
• Three categories:• Supervised domain adaptation
• Semi-supervised domain adaptation
• Unsupervised domain adaptation
![Page 74: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/74.jpg)
The Future of Real-Time SLAM (ICCV 2015 Workshop)
• Panel discussion: Deep Learning vs SLAM
• Newcombe's Proposal: Use SLAM to fuel Deep Learning
• Today's SLAM systems are large-scale "correspondence engines" which can be used to generate large-scale datasets
• Graphics for CNN
![Page 75: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/75.jpg)
The need of domain adaptation
Synthetic:
Real:
Domain Transfer
![Page 76: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/76.jpg)
Unsupervised domain adaptation
• Only the class labels of source samples are known, all class labels of the target samples are unknown.
• Goal: a feature extractor f and a classifier c• P(f(xs)) = P(f(xt))
• Better classification performance on xs
• Key issue: Discrepancy metric between two complex distributions• D(P(f(xs)), P(f(xt)))
![Page 77: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/77.jpg)
Weighted MMD
• Let
• Define
• Weighted MMD
![Page 78: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/78.jpg)
Office-10+Caltech-10
![Page 79: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/79.jpg)
Unsupervised Domain Adaptation by Backpropagation (Ganin & Lempitsky, ICML 2015)
![Page 80: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/80.jpg)
Simultaneous Deep Transfer Across Domains and Tasks (Tzeng et al., ICCV 2015)
• “maximally confuse” the two domains
• uniform distribution over domain labels
![Page 81: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/81.jpg)
Domain cocktail network (Xu et al., CVPR 2018)
•
![Page 82: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/82.jpg)
SimGAN (CVPR 2017)
• Learning from Simulated and Unsupervised Images through Adversarial Training (Shrivastava, Arxiv 2016)
• Realism loss
• Self-regularization
• is also pixel-level DA
![Page 83: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/83.jpg)
Unsupervised Pixel–Level Domain Adaptation (CVPR 2017)
![Page 84: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/84.jpg)
Image translation (Zhu et al., CVPR 2017)
![Page 85: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/85.jpg)
Pix2pix: supervised image translation (Isola et al., CVPR 2017)
• Positive pair: • (Input, groundtruth)
• Negative pair:• (Input, synthesis)
![Page 86: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/86.jpg)
Learning Residual Images (Shen & Liu, CVPR 2017)
![Page 87: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/87.jpg)
Cycle-Consistent supervision (Zhu et al., ICCV 2017)• Cycle consistency loss
![Page 88: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/88.jpg)
BicycleGAN: Multimodal Image-to-Image Translation (Zhu et al., NIPS 2017)
![Page 89: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/89.jpg)
Suggestion
• Problem-oriented• Generator
• Discriminator+
![Page 90: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/90.jpg)
References
• I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, NIPS 2014.
• D.P. Kingma, M. Welling, Auto-encoding variational bayes, arXiv:1312.6114, 2013.
• N. Lei, K. Su, L. Cui, S.-T. Yau, D. X. Gu, A Geometric View of Optimal Transportation and Generative Model, Arxiv 2017.
• A.B.L. Larsen, S. K. Sønderby, H. Larochelle, O. Winther, Autoencoding beyond pixels using a learned similarity metric, ICML 2016.
• A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, NIPS 2016.
![Page 91: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/91.jpg)
• X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel, InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets, NIPS 2016.
• A. Odena, C. Olah, J. Shlens, Conditional image synthesis with auxiliary classifier GANs, ICML 2017.
• Z. He, W. Zuo, M. Kan, S. Shan, X. Chen, Arbitrary Facial Attribute Editing: Only Change What You Want, arXiv:1711.10678, 2017.
• E.L. Denton, S. Chintala, R. Fergus, Deep generative image models using a laplacian pyramid of adversarial networks, NIPS 2015.
• H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D Metaxas, StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks, ICCV 2017.
![Page 92: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/92.jpg)
• Q. Chen, V. Koltun, Photographic image synthesis with cascaded refinement networks, ICCV 2017.
• S. Benaim, and Lior Wolf, One-Sided Unsupervised Domain Mapping, NIPS 2017.
• K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep cnn for image denoising, IEEE T-IP 2017.
• M. Li, W. Zuo, D. Zhang, Deep Identity-aware Transfer of Facial Attributes, arXiv:1610.05586.
• D. Ulyanov, A. Vedaldi, V. Lempitsky, Deep Image Prior, CVPR 2018.
• R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, CVPR 2018.
![Page 93: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/93.jpg)
• C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, ICLR 2014.
• A. Nguyen, J. Yosinski, J. Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, CVPR 2015.
• C.-J. Simon-Gabriel, Y. Ollivier, B. Schölkopf, L. Bottou, D. Lopez-Paz, Adversarial Vulnerability of Neural Networks Increases With Input Dimension, arXiv:1802.01421.
• X. Wang, A. Shrivastava, A. Gupta, A-Fast-RCNN: Hard positive generation via adversary for object detection, CVPR 2017.
• C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, W. Shi, Photo-realistic single image super-resolution using a generative adversarial network, CVPR 2017.
![Page 94: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/94.jpg)
• A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, L. Van Gool, DSLR-quality photos on mobile devices with deep convolutional networks, ICCV 2017.
• A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, L. Van Gool, WESPE: Weakly supervised photo enhancer for digital cameras, NTIRE 2018.
• D. Pathak,P. Krahenbuhl, J. Donahue, T. Darrell, A.A. Efros, Context encoders: Feature learning by inpainting, CVPR 2016.
• Z. Yan, X. Li, M. Li, W. Zuo, S. Shan, Shift-Net: Image Inpainting via Deep Feature Rearrangement, arXiv:1801.09392.
• X. Li, M. Liu, Y. Ye, W. Zuo, L. Lin, R. Yang, Learning Warped Guidance for Blind Face Restoration, arXiv:1804.04829.
• H. Yan, Y. Ding, P. Li, Q. Wang, Y. Xu, W. Zuo, Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation, CVPR 2017.
![Page 95: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/95.jpg)
• Y. Ganin, V. Lempitsky, Unsupervised Domain Adaptation by Backpropagation, ICML 2015.
• E. Tzeng, J. Hoffman, T. Darrell, Simultaneous Deep Transfer Across Domains and Tasks, ICCV 2015.
• R. Xu, Z. Chen, W. Zuo, J. Yan, L. Lin, Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift, CVPR 2017.
• A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, R. Webb, Learning from simulated and unsupervised images through adversarial training, CVPR 2017.
• K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, D. Krishnan, Unsupervised pixel-level domain adaptation with generative adversarial networks, CVPR 2017.
![Page 96: Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision](https://reader030.vdocuments.site/reader030/viewer/2022040307/5ed408e08d46b66d226352c4/html5/thumbnails/96.jpg)
• P. Isola, J.Y. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with conditional adversarial networks, CVPR 2017.
• W. Shen, R. Liu, Learning residual images for face attribute manipulation, CVPR 2017.
• J.Y. Zhu, T. Park, P. Isola, A.A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, ICCV 2017.
• J.Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A.A. Efros, O Wang, E Shechtman, Toward Multimodal Image-to-Image Translation, NIPS 2017.