CS – 3750

Machine Learning

Generative Adversarial
Network
Khushboo Thaker
[email protected]
Exponential Growth in GAN Papers

Ian Goodfellow

Khushboo Thaker 2
 • Why Generative Modeling ?
• Existing Generative Models – A review
• Properties of GAN
• GAN Framework
Outline • Minimax Play for GAN
• Why GAN training is Hard ?
• Tricks to train GAN
• Examples of some common extension to GAN
• Conclusion and future reading

Khushboo Thaker 3
 Generative Modeling
• Input is Training examples and output is
some representation of probability
distribution which defines this example
space.

• Un-Supervised
Data – X
Goal – Learn Hidden structure of data
Training Examples Generated Samples
Figure from Dr. Fei-Fei Li slides

• Supervised
Data – X , y
Goal – Learn mapping from X -> Y

Khushboo Thaker Sample Generation 4

 Features
Noisy Input Simulated Data Representative
of Data
Prediction of Semi-Supervised
Future State Missing Data Learning

Khushboo Thaker 5
 Maximum Likelihood based Models

P(x)

Maximum likelihood tries increase the

likelihood of data given the parameters

Khushboo Thaker 6
Khushboo Thaker 7
 Tractable Model - PixelRNN / PixelCNN /
WaveNet
Fully visible belief Network
• Generate image pixels from corner

• Training Faster

• Generation Slow / Sequential

• Cannot generate samples based on

some latent code
Chain Rule
Khushboo Thaker 8
Maximum Likelihood based Training
Non Tractable Model - Variational
Approximation
Variational Auto-encoder
• Model is able to achieve high
likelihood
• Model is not asymptotically
consistent unless q is perfect
• Samples tend to have lower quality

Khushboo Thaker 9
Non Tractable Model - MCMC Approximation
Boltzmann Machine
• Energy Function based models
• Markov chains don’t work for long
sequences
• Hard to scale on large dataset

Khushboo Thaker 10
Khushboo Thaker 11
Where do GANs fall ?
•Can Use Latent Information while sample generation
•Asymptotically consistent ( claims to recover true
distribution)
•No Markov Chain assumption
•Samples produced are high quality

Khushboo Thaker 12
Generated Samples - GAN

Khushboo Thaker 13
Next Video Frame Prediction

• Sharp image
• Better estimation of Ear
position
• Much crisp eyes

Khushboo Thaker 14
Generative Adversarial Networks

Generator Discriminator

Khushboo Thaker 15
Generative Adversarial Networks

Khushboo Thaker 16
Classic GAN Framework

Z – random Noise
(latent representation of data)
Zd <= Xd

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
Khushboo Thaker 17
Training Discriminator

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
Khushboo Thaker 18
Training Generator

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
Khushboo Thaker 19
 Mini-max Game Approach

Discriminator output for Discriminator output for

real data x fake data G(z)

• Generator minimizes the log-probability of the discriminator being correct

• Resembles Jensen-Shannon divergence
• Saddle Point of discriminators loss
Khushboo Thaker 20
 Mini-max Game Approach

Nash Equilibrium / Saddle Point

• Generator minimizes the log-probability of the discriminator being correct

• Resembles Jensen-Shannon divergence
• Saddle Point of discriminators loss
Khushboo Thaker 21
Vanishing Gradient Problem with Generator

Gradient goes to 0 if D is confident , ie D(G(z)) -> 0

As can be seen that whenever the
discriminator becomes very confident the
loss value will be zero

Nothing to improve for Generator

Khushboo Thaker 22
Heuristic Non Saturating Game

Generator maximizes the log probability of the discriminator’s mistake

Does not change when discriminator is successful

Khushboo Thaker 23
Comparison of Generator Losses Able to learn even if the
Gradient signal is low

• Generators cost is a function D(G(z))

Khushboo Thaker 24
 • Why Generative Modeling ?
• Existing Generative Models – A review
• Properties of GAN
• GAN Framework
Outline • Minimax Play for GAN
• Why GAN training is Hard ?
• Tricks to train GAN
• Examples of some common extension to GAN
• Conclusion and future reading

https://www.youtube.com/watch?v=mObnwR-u8pc
Khushboo Thaker 25
Why GAN are hard to train ?

Khushboo Thaker 26
 Non-Convergence • Differential Equation’s solution has sinusoidal
terms
D & G nullifies each others learning in every iteration • Even with a small learning rate, it will not
converge
Train for a long time – without generating good quality samples
• Discrete time gradient descent can spiral
outward for large step size

Khushboo Thaker 27
 Sample
Coverage
Sample
Mode Collapse Accuracy

http://www.youtube.com/watch?v=ktxhiKhWoEE&t=0m30s

Generator excels in a subspace but

does-not cover entire real distribution

Unroll GAN

GAN Luke et al. 2016

Khushboo Thaker 28
Why GAN are hard to train ?

• Generator keeps generating similar images – so nothing to learn

• Maintain trade-off of generating more accurate vs high coverage samples

• The two learning tasks need to have balance to achieve stability

• If Discriminator is not sufficiently trained – it can worse generator

• If Discriminator is over-trained - will produce no gradients

Khushboo Thaker 29
Tricks to Train GAN
• One sided label smoothing
• Historical generated batches
• Feature Matching
• Batch Normalization
• Regularizing discriminator gradient in region around real data
(DRAGAN)

Khushboo Thaker 30
 One Side Label Smoothening
• Generator is very sensitive to
Discriminators output
• Prevents discriminator to give
high gradients
• Does-not reduce accuracy.
• Increase confidence
• Only smooth positive samples

Salimans, Tim, et al. "Improved techniques for training gans." Advances in Neural
Information Processing Systems. 2016.

Khushboo Thaker 31
Historical generated batches
Help stabilize discriminator training at early stages
Don’t Let discriminator
forget what it already
learned

Shrivastava, Ashish, et al. "Learning from Simulated and Unsupervised Images through Adversarial Training." CVPR. Vol. 2. No. 4. 2017.

Khushboo Thaker 32
Feature Matching
• Generated images must match
statistics of real images
• Discriminator defines the statistics
• Generator is trained such that the
expected value of statistics
matches the expected value of real
statistics
• Generator tries to minimize the L2
distance in expected values in
some arbitrary space
• Discriminator defines that arbitrary
space

Khushboo Thaker 33
Batch Normalization
• Construct different mini-batches
for real and fake

• Each mini-batch needs to

contain only all real images or all
generated images.

• Makes samples with-in a batch

less dependent

Khushboo Thaker 34
DRAGAN
• Failed GANs typically have extreme gradients/sharp peaks around
real data
• Regularize GANs to reduce the gradient of the discriminator in a
region around real data

Khushboo Thaker 35
Few variations of GAN
• Conditional GAN
• LapGAN
• DCGAN
• CatGAN
• InfoGAN
• AAE
• DRAGAN
• IRGAN

Khushboo Thaker 36
 • Generator Learns P (X | Z, Y)
• Discriminator Learns P (L | X,Y)
• Much better samples

Mirza, M. and Osindero, S., 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784.

Khushboo Thaker 37
Khushboo Thaker 38
DCGAN
• Multiple Convolutional Layers
• Batch Normalization
• Strides with Convolution
• Leaky ReLUs

Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." (2015

Khushboo Thaker 39
DCGAN

Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." (2015).

Khushboo Thaker 40
InfoGAN
• Rewards Disentanglement – ( individual dimensions capturing key
attributes of images)

• Z – partitioned into two parts

z – capture slight variation in the images
y – captures the main attributes of the images

Mutual Information – maximizing mutual information

Between the code and generator output

Khushboo Thaker 41
InfoGAN

Khushboo Thaker 42
BiGANs
• Encoder
• Decoder
• Discriminator

Khushboo Thaker 43
LapGANs
• To Scale GAN for large image

• Laplacian pyramid function is

used to generate different
scales of image

Denton EL, Chintala S, Fergus R. Deep generative image models using a laplacian pyramid of adversarial networks. NIPS 2015 (pp. 1486-1494).

Khushboo Thaker 44
LapGAN

Denton EL, Chintala S, Fergus R. Deep generative image models using a laplacian pyramid of adversarial networks. InAdvances in neural information processing
systems 2015 (pp. 1486-1494).

Khushboo Thaker 45
DCGAN
• Multiple Convolutional Layers
• Batch Normalization
• Strides with Convolution
• Leaky ReLUs

Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." (2015

Khushboo Thaker 46
Khushboo Thaker 47
Adversarial Autoencoder (GAN + VAE)

Khushboo Thaker 48
Khushboo Thaker 49
GAN for Text
• GANs for Language Generation (Yu et al. 2017)
• GANs for MT (Yang et al. 2017)
• GANs for Dialogue Generation (Li et al. 2016)
• GANs for fake news detection (Yang et al. 2017)
• GANs for Information Retrieval

Khushboo Thaker 50
GAN and RL connection
• GANs – Inverse Reinforcement
Learning
• GANs - Imitate Learning
• GANs – actor critic framework

• REINFORCE - Policy Gradient Based

learning
• Gumbel Softmax

Khushboo Thaker 51
Conclusion
• GAN is an active area of research
• GAN architecture is flexible to support variety of learning problems
• GAN does not guarantee to converge
• GAN is able to capture perceptual similarity and generates better
images than VAE
• Needs a lot of work in theoretic foundation of Network
• Evaluation of GAN is still an open research (Theis et. al)

Khushboo Thaker 52
Important Papers to dig into GAN
• NIPS 2016 Tutorial: - Ian Goodfellow
• Arjovsky, Martin, and Léon Bottou. "Towards principled methods for training generative adversarial
networks." arXiv preprint arXiv:1701.04862 (2017).
• Roth, Kevin, et al. "Stabilizing training of generative adversarial networks through regularization." Advances
in Neural Information Processing Systems. 2017.
• Li, Jerry, et al. "Towards understanding the dynamics of generative adversarial networks." arXiv preprint
arXiv:1706.09884 (2017).
• Kodali, Naveen, et al. "On convergence and stability of GANs." arXiv preprint arXiv:1705.07215 (2017).
• Fedus, William, et al. "Many Paths to Equilibrium: GANs Do Not Need to Decrease aDivergence At Every
Step." arXiv preprint arXiv:1710.08446 (2017).
• https://github.com/soumith/ganhacks#authors
• http://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/
• https://www.araya.org/archives/1183

Khushboo Thaker 53
Startup code, Tools and Tricks
• https://github.com/soumith/ganhacks#authors

• https://medium.com/@utk.is.here/keep-calm-and-train-a-gan-pitfalls-and-tips-on-training-generative-adver
sarial-networks-edd529764aa9

• https://jhui.github.io/2017/03/05/Generative-adversarial-models/

Khushboo Thaker 54
References
• Deep Learning Book
• GAN paper: https://arxiv.org/abs/1701.00160
• GAN slides: http://slazebni.cs.illinois.edu/spring17/lec11_gan.pd
• GAN Tutorial: https://www.youtube.com/watch?v=HGYYEUSm-0Q
• GAN for text:
http://www.phontron.com/class/nn4nlp2017/assets/slides/nn4nlp-1
7-adversarial.pdf

Khushboo Thaker 55
Not the end..

Khushboo Thaker 56
Thank You for Listening
Questions ?

Khushboo Thaker 57