Upload File

generative adversarial text to image synthesis

31
Generative Adversarial Text to Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran [GitHub ] [Arxiv ] Slides by Víctor Garcia [GDoc ] Computer Vision Reading Group (30/09/2016)

Upload: xavier-giro

Post on 14-Jan-2017

193 views

Category:

Data & Analytics


5 download

Report

TRANSCRIPT

Page 1: Generative adversarial text to image synthesis

Generative Adversarial Text to Image Synthesis

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran

[GitHub] [Arxiv]

Slides by Víctor Garcia [GDoc]Computer Vision Reading Group (30/09/2016)

https://github.com/paarthneekhara/text-to-image
https://github.com/paarthneekhara/text-to-image
https://github.com/paarthneekhara/text-to-image
https://github.com/paarthneekhara/text-to-image
https://docs.google.com/presentation/d/1pvyOFygLzhG3fifa8G1j46EavuUxsQELMIKxKlbydrU/edit?usp=sharing
https://imatge.upc.edu/web/teaching/computer-vision-reading-group
https://imatge.upc.edu/web/teaching/computer-vision-reading-group
Page 2: Generative adversarial text to image synthesis

Index● Introduction ● State of the Art● Method

○ Network Architecture○ Losses

● Experiments○ Qualitative Results○ Sentence interpolation○ Style Transfer

● Conclusions

Page 3: Generative adversarial text to image synthesis

Introduction

Text → Image

GANs

Page 4: Generative adversarial text to image synthesis

Index● Introduction ● State of the Art● Method

○ Network Architecture○ Losses

● Experiments○ Qualitative Results○ Sentence interpolation○ Style Transfer

● Conclusions

Page 5: Generative adversarial text to image synthesis

GANs

Discriminator

1/0

True

World

Fake

Generator

Page 6: Generative adversarial text to image synthesis

GANs

DiscriminatorD(·)

1/0

True

World

Fake

Generator

q(x) xG(z) zx’

Page 7: Generative adversarial text to image synthesis

GANs

DiscriminatorD(·)

MAX → E[log(D(X))]

True

World

Fake

Generator

q(x) xG(z) zx’

Page 8: Generative adversarial text to image synthesis

GANs

DiscriminatorD(·)

MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ]

True

World

Fake

Generator

q(x) xG(z) zx’

Page 9: Generative adversarial text to image synthesis

GANs

DiscriminatorD(·)

MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ]

True

World

Fake

Generator

q(x) xG(z) zx’

Page 10: Generative adversarial text to image synthesis

GANs

DiscriminatorD(·)

True

World

Fake

Generator

q(x) xG(z) zx’

MIN → E[ log(1 - D(G(Z))) ]

Page 11: Generative adversarial text to image synthesis

GANs with Join DistributionsHow do we generate the image from text?

Page 12: Generative adversarial text to image synthesis

GANs with Join DistributionsHow do we generate the image from text?

Discriminator

1/0

f(x,t) f(x’,t)

Page 13: Generative adversarial text to image synthesis

GANs with Join Distributions

Discriminator

1/0

Real Image

+Text

Gen. Image

+Text

Generator +Text

Page 14: Generative adversarial text to image synthesis

GANs with Join Distributions

Discriminator

1/0

Real Image

+Text

Gen. Image

+Text

Generator +Text

Page 15: Generative adversarial text to image synthesis

Text EmbedddingIn order to represent the text in a vector...

MIN

WHERE

Page 16: Generative adversarial text to image synthesis

Text EmbedddingIn order to represent the text in a vector...

MIN

WHERE

This is the recurrent text encoder

Page 17: Generative adversarial text to image synthesis

Index● Introduction ● State of the Art● Method

○ Network Architecture○ Losses

● Experiments○ Qualitative Results○ Sentence interpolation○ Style Transfer

● Conclusions

Page 18: Generative adversarial text to image synthesis

Network Architecture

Page 19: Generative adversarial text to image synthesis

Losses - CLS

log(D(x,t)) log(1-D(G(z,t)))

True Image +

True Text

Fake Image +

True Text

Real Images match the text content?

Page 20: Generative adversarial text to image synthesis

Losses - CLS

log(D(x,t)) log(1-D(G(z,t))) log(1-D(G(zi,tk)))

True Image +

True Text

Fake Image +

True Text

True Image (i) +

True Text (j)Unmatched

Page 21: Generative adversarial text to image synthesis

Losses - INT

They train interpolating between different text embedding vector (t1~t2).

So the generator learns to fill GAPS on the data manifold.

Page 22: Generative adversarial text to image synthesis

Index● Introduction ● State of the Art● Method

○ Network Architecture○ Losses

● Experiments○ Qualitative Results○ Sentence interpolation○ Style Transfer

● Conclusions

Page 23: Generative adversarial text to image synthesis

Qualitative Results - Birds

Page 24: Generative adversarial text to image synthesis

Sentence Interpolation

Gen.

z0

+Text1

Gen.

z1

+Text3

Gen.

z0

+Text2

Gen.

z1

+Text4

Page 25: Generative adversarial text to image synthesis

Disentangling style and content

Generator.

z+

Text

If ‘text’ is describing the content? What is ‘z’ describing?

Page 26: Generative adversarial text to image synthesis

Disentangling style and content

Generator.

z+

Text

If ‘text’ is describing the content? What is ‘z’ describing?

Style → Pose, Background…, let’s extract ‘z’

Page 27: Generative adversarial text to image synthesis

Disentangling style and contentz0 z1 z2 z3 z4 z5

Page 28: Generative adversarial text to image synthesis

Qualitative Results - Flowers

Page 29: Generative adversarial text to image synthesis

Qualitative Results - MSCOCO

Page 30: Generative adversarial text to image synthesis

Conclusions

Discriminator

1/0

f(x,t) f(x’,t)

x~t

Page 31: Generative adversarial text to image synthesis

1 An Introduction to Image Synthesis with Generative Adversarial … · 2018. 11. 20. · 1 An Introduction to Image Synthesis with Generative Adversarial Nets He Huang, Philip S

High-Resolution Mammogram Synthesis using Progressive Generative Adversarial … · 2020. 9. 21. · High-Resolution Mammogram Synthesis using Progressive Generative Adversarial Networks

Generative Adversarial Networkstat.snu.ac.kr/mcp/Lecture10_GAN.pdf · 2019-11-11 · Generative Adversarial Network Generative adversarial network: Setup Let y = 1 if the data is

Lecture 9: Unsupervised, Generative & Adversarial Networks · Lecture 9: Unsupervised, Generative & Adversarial Networks ... GENERATIVE & ADVERSARIAL NETWORKS - 20 o Autoencoders

Generative adversarial networks...Generative adversarial nets. NeurIPS. Longer version Goodfellow, (2016). NIPS 2016 Tutorial: Generative Adversarial Networks. Extra Radford et al,

Generative Adversarial Nets - courses.cs.washington.edu

Adversarial Distillation of Bayesian Neural Network Posteriorsproceedings.mlr.press/v80/wang18i/wang18i.pdf · 2.3. Generative Adversarial Networks Generative Adversarial Networks

Generative Adversarial Nets - Semantic Scholargenerative adversarial text to image synthesis. Scott Reed, ZeynepAkata. ... Conditional Sequence Generative Adversarial Nets[J]. arXiv

Mode Seeking Generative Adversarial Networks for …openaccess.thecvf.com/content_CVPR_2019/papers/Mao_Mode...Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis

Unsupervised Generative Adversarial Alignment

MC-GAN: Multi-conditional Generative Adversarial Network for Image ... · MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis Hyojin Park 1 [email protected]

Non-Adversarial Image Synthesis With Generative Latent Nearest …openaccess.thecvf.com/content_CVPR_2019/papers/Hoshen... · 2019-06-10 · Non-Adversarial Image Synthesis with Generative

Generative modeling Generative Adversarial Nets Use case ... · • Generate images from text Reed et al., «Generative Adversarial Text to Image Synthesis», 2016 • Segment images

Generative Adversarial Networksbhiksha/courses/deeplearning/Spring.2018/... · 2018. 3. 29. · Generative Adversarial Networks Generative adversarial networks (GANs) are relatively

Generative adversarial networks

Data Synthesis based on Generative Adversarial Networks · each other. We propose a data synthesis method based on gener-ative adversarial networks (GANs). GANs are a generative model

Unsupervised Learning – Generative Models (PixelRNNs, PixelCNNs) – Generative ... · 2019. 3. 28. · Generative Adversarial Networks Ian Goodfellow et al., “Generative Adversarial

Generative Adversarial Network based Synthesis for Supervised … ·  · 2017-10-11Generative Adversarial Network based Synthesis for ... In this work we study this ... which we

Generative Adversarial Networkcse.iitkgp.ac.in/~sudeshna/courses/DL17/GAN-6-april-17.pdfGenerative Adversarial Network 14 min ... Generative Adversarial Text to Image Synthesis, ICML

Generative Adversarial Networks - sahin olut...Image to text (Image captioning - Generative Adversarial Text to Image Synthesis) There are many applications which are not covered above

CPGAN: Content-Parsing Generative Adversarial Networks for ... · CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis Jiadong Liang1 ;y, Wenjie Pei2,

Triple Generative Adversarial Netspapers.nips.cc/paper/6997-triple-generative-adversarial-nets.pdf · the labels. To address the problems, we present triple generative adversarial

COMP9444 Neural Networks and Deep Learning 9b. Generative Adversarial … · 2020. 7. 25. · COMP9444 20T2 Generative Adversarial Networks 13 Generative Adversarial Networks GAN

A Survey of Image Synthesis and Editing with Generative ...kun/papers/gan_survey_final.pdf · 2 Generative Adversarial Networks Generative adversarial networks (GANs) were proposed

Generative Adversarial Networks (GANs)

Relational Data Synthesis using Generative Adversarial ...including a representative deep generative model, variational auto-encoder (VAE) [33, 49], and the state-of-the-art data synthesis

論文輪読: Generative Adversarial Text to Image Synthesis

Generative Adversarial Transformers

Markovian Generative Adversarial Networks …Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks Chuan Li and Michael Wand Institut for Informatik,

DM-GAN: Dynamic Memory Generative Adversarial Networks for … · 2019. 6. 10. · DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis Minfeng Zhu1,3∗

Text to Image Synthesis Using Generative Adversarial Networks

High-Quality Statistical Parametric Speech Synthesis Using Generative Adversarial Networkssython.org › papers › thesis › saito17_MS_presen.pdf · 2019-06-06 · /23 Speech Synthesis

Generative Adversarial Nets: Applications and Extensionsice.dlut.edu.cn/valse2018/ppt/Generative... · Generative Adversarial Nets: Applications and Extensions Wangmeng Zuo Vision

EmotiGAN: Emoji Art using Generative Adversarial Networkscs229.stanford.edu/proj2017/final-reports/5244346.pdfA. Generative Adversarial Networks A Generative Adversarial Network[4]

Generative Adversarial Networks (GANs) - Emanuele Sansone · List of recent papers on Generative Adversarial Networks: PAPERs about GANs 20142016 Generative Adversarial Networks ICLR