Scribd
Upload
6 views16 pages

Ch2_ANN_BB

Uploaded by

Boch
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views16 pages

Ch2_ANN_BB

Uploaded by

Boch
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Neural network architecture

In this example, we will build a neural network with three layers:


• Input layer with two inputs neurons

• One hidden layer with two neurons

• Output layer with a single neuron

1
Update weights
• Neural network training is about finding
weights that minimize prediction error.

• We usually start our training with a set of


randomly generated weights.

• Then, backpropagation is used to update


the weights in an attempt to correctly
map arbitrary inputs to outputs

• Our initial weights will be as following:

w1 = 0.11, w2 = 0.21, w3 = 0.12, w4 = 0.08,

w5 = 0.14 and w6 = 0.15


2
Dataset
• Our dataset has one sample with two inputs and one output.

• Our single sample is as following inputs=[2, 3] and output=[1].

3
Forward propagation
• We will use given weights and

inputs to predict the output.

• Inputs are multiplied by weights;

the results are then passed

forward to next layer.

4
Calculating Error
• Now, it’s time to find out how our

network performed by calculating

the difference between the actual

output and predicted one.

➢ Our network output, or prediction,

is not even close to actual output.

We can calculate the difference or

the error as following:

5
Reducing Error
• Our main goal of the training is to reduce the error.
• Since actual output is constant, “not changing”, the only way to reduce the error is to change prediction value.
➢ How to change prediction value?
• In order to change prediction value, we need to change weights values.

6
Backpropagation
How to update the weights value so that the error is reduced?

The answer is Backpropagation

• Backpropagation, short for “backward propagation of errors”, is a mechanism used to update the
weights using gradient descent.
• It calculates the gradient of the error function with respect to the neural network’s weights.
• The calculation proceeds backwards through the network.

7
Backpropagation: Gradient descent
Gradient descent is an iterative optimization algorithm for finding the
minimum of a function.
In our case we want to minimize the error function.

8
Backpropagation: Update w6
• In order to update w6, we take the current w6 and subtract the partial derivative of error
function with respect to w6.

• Optionally, we multiply the derivative of the error function by a selected number to make
sure that the new updated weight is minimizing the error function; this number is called
learning rate.

9
Backpropagation: Update w6
• The derivation of the error function is evaluated by applying the chain rule as following

10
Backpropagation: Update w6 and w5

• So to update w6 we can apply the following formula:

• Similarly, we can derive the update formula for w5 and any other weights
existing between the output and the hidden layer.

11
Backpropagation: Update w1
• However, when moving backward to update w1, w2, w3 and w4 existing between input and hidden layer, the
partial derivative for the error function with respect to w1, for example, will be as following.

12
Backpropagation: Update wi
• We can find the update formula for the remaining weights w2, w3 and w4 in the same way.
• In summary, the update formulas for all weights will be as following:

13
Backpropagation: Update wi
• We can rewrite the update formulas in matrices as following:

14
Backward Pass
• Using derived formulas we can find the new weights.

15
• Now, using the new weights
we will repeat the forward
passed.

• We can notice that the


prediction 0.26 is a little bit
closer to actual output than
the previously predicted
one 0.191. We can repeat
the same process of
backward and forward pass
until error is close or equal
to zero.

16

You might also like