Scribd
Upload
43 views

CNNs Pytorch

The document discusses different training tricks for CNNs using PyTorch, including ReLUs, dropout, and batch normalization. ReLUs train neural networks several times faster than sigmoids. Dropout prevents overfitting by randomly dropping neurons during training, forcing neurons to learn more robust features. Batch normalization standardizes the inputs to each layer to speed up training by reducing the internal covariate shift problem between layers. It normalizes each dimension independently and adds scale and shift parameters to allow the network flexibility.

Uploaded by

Alex Saave
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
43 views

CNNs Pytorch

The document discusses different training tricks for CNNs using PyTorch, including ReLUs, dropout, and batch normalization. ReLUs train neural networks several times faster than sigmoids. Dropout prevents overfitting by randomly dropping neurons during training, forcing neurons to learn more robust features. Batch normalization standardizes the inputs to each layer to speed up training by reducing the internal covariate shift problem between layers. It normalizes each dimension independently and adds scale and shift parameters to allow the network flexibility.

Uploaded by

Alex Saave
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Training Tricks for CNNs + PyTorch

IALAB UC

Computer Science Department, PUC

IALAB UC Deep Learning I DCC 1 / 20


Today’s Schedule

Training Tricks
ReLUs
Dropout
Batch Normalization
PyTorch - in depth!

IALAB UC Deep Learning I DCC 2 / 20


Training Tricks: Rectified Linear Units (ReLUs)

Usually, NNs use sigmoid activation functions, such as tanh(x) or


(1 + exp−x )−1 . Here, however, they use Rectified Linear Units
(ReLU).
Empirical observation: Deep convolutional neural networks with
ReLUs train several times faster than their equivalents with
sigmoid units.
IALAB UC Deep Learning I DCC 3 / 20
Training Tricks: Rectified Linear Units (ReLUs)

Example: A four-layer convolutional neural network with ReLUs


(solid line) reaches a 25% training error rate on CIFAR-10 six
times faster than an equivalent network with tanh(x) neurons
(dashed line).

IALAB UC Deep Learning I DCC 4 / 20


Training Tricks: Dropout1

In general, combining different models can be very useful (Mixture


of experts, majority voting, boosting, etc.).
Training many different models, however, is very time consuming.
Here, they introduce Dropout.

1
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
http://jmlr.org/papers/v15/srivastava14a.html
IALAB UC Deep Learning I DCC 5 / 20
Training Tricks: Dropout

At each iteration (epoch) set the output of each hidden neuron to


zero with a certain probability.
The neurons which are “dropped out” in this way do not contribute
to the forward pass and do not participate in backpropagation.
For every input, the neural network samples a different
architecture, but all these architectures share weights.
This technique reduces complex co-adaptations of neurons, since
each neuron cannot rely on the presence of other neurons.

IALAB UC Deep Learning I DCC 6 / 20


Training Tricks: Dropout

Each neuron is forced to learn more robust features that are


useful in conjunction with many different random subsets of other
neurons.
Without dropout, the network exhibits substantial overfitting.
Downside: Dropout increases the number of iterations required to
converge.

IALAB UC Deep Learning I DCC 7 / 20


Training Tricks: Dropout

If you want to know of a few extensions to Dropout (which we won’t


cover here):
DropBlock2 : Dropping whole sets of neurons.
DropConnect3 : Dropping weights and biases randomly.

2
DropBlock: A regularization method for convolutional networks https://arxiv.org/abs/1810.12890
3
Regularization of Neural Networks using DropConnect
http://yann.lecun.com/exdb/publis/pdf/wan-icml-13.pdf
IALAB UC Deep Learning I DCC 8 / 20
Training Tricks: Batch Normalization4

As a ML problem, it is desirable that the distribution of training and


test data matches.
However, the distribution of the input to each layer changes during
training (why?).
This slows down the training by requiring lower learning rates and
careful parameter initialization.
We would like to keep the distribution of x̄ fixed over time.
Then, θ2 does not have to readjust to compensate for the change
in the distribution of x̄.

4
https://arxiv.org/abs/1502.03167
IALAB UC Deep Learning I DCC 9 / 20
Batch Normalization (BN), Ioffe and Szegedy, 2015

BN takes a step towards reducing the internal covariate shift


problem between layers.
Specifically, BN introduces a normalization step that fixes the
means and variances of layer inputs.

x̄ (k ) − E[x̄ (k ) ]
x̂ (k ) = p (1)
Var [x̄ (k ) ]
Expectation and variance are computed over the mini-batch for
each dimension k . So each dimension is normalized
independently.
Therefore, BN normalizes each scalar feature independently
trying to make them unit gaussian, i.e., each dimension has zero
mean and unit variance.

IALAB UC Deep Learning I DCC 10 / 20


Batch Normalization (BN)

A small constant  is added to the denominator to avoid an eventual


division by zero.

IALAB UC Deep Learning I DCC 11 / 20


Batch Normalization (BN)

Simply normalizing each input of a layer might change the


representational space of the layer.
Ex. normalizing the inputs of a sigmoid would constrain them to
lay in the linear part of the sigmoid.
We need to add a mechanism to compensate for this effect.
IALAB UC Deep Learning I DCC 12 / 20
Batch Normalization (BN)

BN introduces a transformation that, if needed, allows the network


to cancel the operation of the BN operator.
In other words, the transformation provides the network with the
flexibility to convert the BN operator in the identity function.
Specifically, BN introduces for each activation x (k ) , parameters γ k
and β k that scale and shift the normalized value.
Using this transformation, if needed, the network can recover the
original activations by setting γ k = Var [x̄ (k ) ] and β k = E[x̄ (k ) ].
(k ) (k )
Recall that: x̂ (k ) = x̄√ −E[x̄(k ) ]
Var [x̄ ]

Parameters γk and βk are learned during training.

IALAB UC Deep Learning I DCC 13 / 20


Batch Normalization (BN)

IALAB UC Deep Learning I DCC 14 / 20


What happens at test time?
Do we have a mini batch?
How mini-batch normalization operates at test time?

Mini-batches are normalized using the


population, rather than mini-batch statistics.

IALAB UC Deep Learning I DCC 15 / 20


Batch Normalization (BN)

IALAB UC Deep Learning I DCC 16 / 20


Batch Normalization (BN)

IALAB UC Deep Learning I DCC 17 / 20


Batch Normalization (BN)

IALAB UC Deep Learning I DCC 18 / 20


Batch Normalization (BN) More Than Just Normalization

IALAB UC Deep Learning I DCC 19 / 20

You might also like