Scribd
Upload
139 views

Building Blocks of DNN PDF

Umair bin Mansoor gives a lecture on the building blocks of deep neural networks (DNNs). He discusses forward and backward propagation through a single layer, including the calculations for the output, input, weights, and biases. He also defines hyperparameters as parameters that determine the weights and biases for each layer, such as the learning rate, number of iterations, number of hidden layers and units, activation functions, and more. Finally, he notes that DNN application is empirical and requires tuning hyperparameters for optimal performance.

Uploaded by

Ambreen Abdul razzaque
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
139 views

Building Blocks of DNN PDF

Umair bin Mansoor gives a lecture on the building blocks of deep neural networks (DNNs). He discusses forward and backward propagation through a single layer, including the calculations for the output, input, weights, and biases. He also defines hyperparameters as parameters that determine the weights and biases for each layer, such as the learning rate, number of iterations, number of hidden layers and units, activation functions, and more. Finally, he notes that DNN application is empirical and requires tuning hyperparameters for optimal performance.

Uploaded by

Ambreen Abdul razzaque
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Building Blocks of DNN

CS-421 Machine Learning

Umair bin Mansoor

DHA Su¤a University

June 2020

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 1/6
Building Blocks of DNN

Let us revise and consider calculations for a single layer l.

Forward propagation:
Input: A[l 1 ] , Output: A[l ] , Z[l ]
Z[l ] = W [l ] A[l 1 ] + b[l ]
A[l ] = g [l ] Z[l ]
Backward propagation:
Input: dA[l ] , Z[l ] , Output: dA[l 1 ] , dW [l ] , db[l ]

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 2/6
Building Blocks of DNN

Let us see the Backward propagation steps in detail:

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN

Let us see the Backward propagation steps in detail:


dZ[l ] = dA[l ] g [l ] Z[l ]

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN

Let us see the Backward propagation steps in detail:


dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN

Let us see the Backward propagation steps in detail:


dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T
db[l ] = dZ[l ]

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
Building Blocks of DNN

Let us see the Backward propagation steps in detail:


dZ[l ] = dA[l ] g [l ] Z[l ]
dW[l ] = dZ[l ] A[l 1 ]T
db[l ] = dZ[l ]
dA[l 1 ] = W[l ]T dZ[l ]
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 3/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.
Getting to know the optimum value of the hyperparameters can
become another optimization problem.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
What are Hyperparameters?

Hyperparameters are parameters that are responsible for determining


W[l ] and b[l ] for any layer l.
Learning rate α
Number of iterations
Hidden layers L
Hidden units/Neurons n[1 ] , n[2 ] ,. . .
Choice of Activation functions: Sigmoid, ReLU, Tanh, etc.
There are a few more like momentum, minibatch size, etc.
Getting to know the optimum value of the hyperparameters can
become another optimization problem.
In order to have a deep knowledge about them requires a greater
understanding of how they a¤ect each other.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 4/6
Applied DNN is Emperical in Nature

- Hyperparameter tuning is a
repetitive process.
- There are a number of applications
to DNN e.g. Computer vision,
Speech, NLP, etc. currently, all
of them require tuning of
hyperparameters for optimal
performance.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 5/6
Human Brain v.s. Arti…cial Neural Network

No one knows how a biological neuron works, not yet.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network

No one knows how a biological neuron works, not yet.


All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network

No one knows how a biological neuron works, not yet.


All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.

What we have discovered is an e¢ cient method of learning using this


structure.

Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6
Human Brain v.s. Arti…cial Neural Network

No one knows how a biological neuron works, not yet.


All that we mimic from a biological neuron in an ANN is its apparent
structure, what is happening inside is still unknown.

What we have discovered is an e¢ cient method of learning using this


structure.
This led us to think that our brain works in a similar fashion.
Umair bin Mansoor (DHA Su¤a University) Lecture# 14 June 2020 6/6

You might also like