Scribd
Upload
15 views13 pages

4 NN Regularization

The document discusses regularization techniques for neural networks. It describes overfitting in neural networks and explains that regularization helps reduce overfitting to improve performance. The two most common regularization techniques are L2 regularization, which penalizes large weights, and dropout regularization, which randomly removes nodes during training.

Uploaded by

virgulatiit21a1068
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views13 pages

4 NN Regularization

The document discusses regularization techniques for neural networks. It describes overfitting in neural networks and explains that regularization helps reduce overfitting to improve performance. The two most common regularization techniques are L2 regularization, which penalizes large weights, and dropout regularization, which randomly removes nodes during training.

Uploaded by

virgulatiit21a1068
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Regularization of Neural

Networks

Dinesh K. Vishwakarma, Ph.D.


PROFESSOR, DEPARTMENT OF INFORMATION TECHNOLOGY

DELHI TECHNOLOGICAL UNIVERSITY, DELHI.


Webpage: http://www.dtu.ac.in/Web/Departments/InformationTechnology/faculty/dkvishwakarma.php
Regularization
▪To improve the performance of the NN,
regularization is done.
▪ An NN performs incredibly well on the
training set, but not nearly as good on the
test set.
▪NN has a very high variance and it cannot
generalize well to data it has not been
trained on.
▪These are the sign of overfitting.
1/28/2022 Dinesh K. Vishwakarma, Ph.D. 2
Solution of Overfitting
▪Get more data
▪Use regularization
✔ Getting more data is sometimes impossible,
and other times very expensive.
✔ Therefore, regularization is a common method
to reduce overfitting and consequently improve
the model’s performance.

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 3


Solution of Overfitting…
▪Two most common approach used as
Regularization for NN:
▪ L2 regularization
▪ Dropout.

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 4


L2 regularization
▪

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 5


L2 regularization…
▪ Lambda: regularization parameter. The
addition of the Frobenius norm, denoted by the
subscript F.
▪ lambda is a parameter that can be tuned.
✔ Larger weight values will be more penalized if the
value of lambda is large.
✔ Similarly, for a smaller value of lambda, the
regularization effect is smaller.
▪ This makes sense, because the cost function must be minimized.
▪ By adding the squared norm of the weight matrix and multiplying it
by the regularization parameters, large weights will be driven down
in order to minimize the cost function.
1/28/2022 Dinesh K. Vishwakarma, Ph.D. 6
How Regularization Works?
▪

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 7


Dropout Regularization
▪ Dropout involves going over all the layers in a
neural network and setting probability of keeping
a certain nodes or not.
▪ The input layer and the output layer are kept the
same.
▪ The probability of keeping each node is set at
random. Only threshold is decided: a value that
will determine if the node is kept or not.
▪ For example, if you set the threshold to 0.8, then
there is a probability of 20% that a node will be
removed from the network.
▪ Therefore, this will result in a much smaller and
simpler neural network.
1/28/2022 Dinesh K. Vishwakarma, Ph.D. 8
Dropout Regularization…

▪ Dropout means that the NN cannot rely on any input node,


since each have a random probability of being removed.
Therefore, the NN will be reluctant to give high weights to
certain features, because they might disappear.
▪ Consequently, the weights are spread across all features,
making them smaller. This effectively shrinks the model and
regularizes it.

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 9


▪https://towardsdatascience.com/how-to-i
mprove-a-neural-network-with-regularizat
ion-8a18ecda9fe3

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 10


When to use Deep Learning?
▪Data size is large Deep
▪High end Learning
infrastructure

Performance
▪Lack of domain Machine
understanding Learning

▪Complex problem
such as image
classification, Amount of Data
speech recognition
etc. Fuel of deep learning is the big data
by Andrew Ng
1/28/2022 Dinesh K. Vishwakarma, Ph.D. 11
Limitations of Deep
Learning
▪Very slow to train
▪Models are very complex, with lot of
parameters to optimize:
✔ Initialization of weights
✔ Layer-wise training algorithm
✔ Neural architecture
• Number of layers
• Size of layers
• Type – regular, pooling, max pooling, soft max
✔ Fine-tuning of weights using back propagation

1/28/2022 Dinesh K. Vishwakarma, Ph.D. 12


Thank you!
[email protected]

1/28/2022 Dinesh K. Vishwakarma, Ph.D.


Slide 13 of 74

You might also like