practice: vae and gan - deep generative models

Practice: VAE and GANHao Dong

Peking University

1

2

Practice: VAE + GAN

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

3

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

4

Hello World: MNIST Classification

28x28x1 = 784 binary values/image

28

28

MNIST dataset

• Image X is a list of row vectors:>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 784))>>> print(X_train.shape)… (50000, 784)

• Image X is a list of images:>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))>>> print(X_train.shape)… (50000, 28, 28, 1)

If RGB image, we will have 3 channels

(height, width, channels)

5

Hello World: MNIST Classification

• Simple Iteration

6

Hello World: MNIST Classification

• Dataset API

7

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

8

Introduction of VAE

!𝑿!𝒁X

L2

KLD

• Two network architectures

• Two loss functions

• Reparameterization trick

9

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

10

VAE Architecture

!𝑿!𝒁X

L2

KLD

28

28

28x28x1 or 784

𝑥~𝑝𝑑𝑎𝑡𝑎

11

VAE Architecture

!𝑿!𝒁X

L2

KLD

• Architecture of Encoder

12

VAE Architecture

!𝑿!𝒁X

L2

KLD

• Architecture of Decoder/Generator

13

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

14

VAE Training

!𝑿!𝒁X

L2

KLD

Reconstruction Loss

15

VAE Training

!𝑿!𝒁X

L2

KLD

KL-Divergence loss

16

VAE Training

!𝑿!𝒁X

L2

KLD

Training Pipeline

17

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

18

VAE Interpolation

19

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

20

Sampling

!𝑿Z

21

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

22

Introduction of DCGAN

• Two network architectures

• Two loss functions!𝑿

X

real

fakeZ

23

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

24

DCGAN Architecture

!𝑿

X

real

fakeZ

28

28

28x28x1 or 784

𝑧~𝑁(0,1)

𝑥~𝑝𝑑𝑎𝑡𝑎

25

DCGAN Architecture

!𝑿

X

real

fakeZ

Architecture of generator

26

DCGAN Architecture

!𝑿

X

real

fakeZ

Architecture of discriminator

27

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

28

DCGAN Training

!𝑿

X

fake

realZ

Loss of discriminator

29

DCGAN Training

!𝑿 realZ

Loss of generator

30

DCGAN Training

Training pipeline

!𝑿

X

real

fakeZ

31

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

32

DCGAN Interpolation

!𝑿

Z1

Z2

(1-a) Z1+a Z2

33

More

34

Improved GANLSGANWGAN

WGAN-GPBiGAN

VAE-GAN…

with MNIST

35

Pix2PixCycleGAN

SRGAN…

with other datasets

36

Proposal Your Projects

Thanks

37