BACK PROPAGATION

NETWORK
BY
ASST. PROF. S. S. SONTAKKE
&
ASST. PROF. N. L. MUDEGOL
HOW BPN WORKS?
• inputs x, arrive through the preconnected path
• input is modeled using real weights W. the weights are
usually randomly selected.
• calculate the output for every neuron from the input layer,
to the hidden layers, to the output layer.
• calculate the error in the outputs
error= actual output – desired output
• travel back from the output layer to the hidden layer to
adjust the weights such that the error is decreased.
• keep repeating the process until the desired output is
achieved
 WHY WE NEED
BACKPROPAGATION?
• It is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge
about the network
• It is a standard method that generally works well
• It does not need any special mention of the features of the
function to be learned.
• It is especially useful for deep neural networks working on
error-prone projects, such as image or speech recognition.
• It takes advantage of the chain and power rules allows it to
function with any number of outputs.
 MULTI LAYER FEED
FORWARD NETWORK (BPN)
Input Layer Hidden Layer Output Layer

II1 OI1 IO1 OO1

1 IH1 1
OH1 1

II2 OI2 IH2 IO2 OO2

2 2 OH2 2

. . .
. . .
. . .

IIl OIl IHm OHm IOn OOn

l m n
 II1 No 1
Input of
Input Neuron

OI No 4
Output of 4
Input Neuron
 IH1 No 1
Input of
Hidden Neuron

OH5 No 5
Output of
Hidden Neuron
 IO1 No 1
Input of
Output Neuron

OO3 No 3
Output of
Output Neuron
 MULTI LAYER FEED
FORWARD NETWORK (BPN)
Input Layer Hidden Layer Output Layer

II1 V11 W11 OO1

1 1 1

V21 W12
II2 OO2
2
V22 2 W22 2

V32 W23
II3 V33 W33 OO3
3 3 3
V is the weight on the link from ith input neuron to jth hidden neuron.

Weight
Between Input
Vij jth Hidden
Neuron
and Hidden
Neuron ith input
Neuron

Weight
Between Input
V25 5th Hidden
Neuron

and Hidden
Neuron 2nd input
Neuron
W is the weight on the link from ith Hidden neuron to jth Output neuron.

Weight
Between
Wij jth Output
Neuron
Hidden and
Output Neuron ith Hidden
Neuron

Weight
Between
W43 3rd Output
Neuron

Hidden and
Output Neuron 4th Hidden
Neuron
 INTRODUCTION

• THREE PHASES:
• FEED FORWARD
• BACK PROPAGATION OF ERROR
• WEIGHT UPDATION
STEP 1:
Initialize weights and learning rate to a small random
values.

STEP 2:
Perform steps 3-17 until stopping condition is false.

STEP 3:
Perform steps 4-16 for each training pair.
 PHASE – 1
FEED-FORWARD
PHASE-1: FEED-FORWARD PHASE
STEP 4: OUTPUT OF INPUT LAYER
Present each input pair to input layer. Apply linear
activation function to it.
i.e. 𝜙 𝐼 =𝑥 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑥

So, output of input layer will be same as it’s input.

i.e. 𝑂 = I 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑥
PHASE-1: FEED-FORWARD PHASE
STEP 5: INPUT FOR HIDDEN LAYER
Compute the net Input for each hidden neuron.

𝐼𝐻𝑝 = σ𝑙𝑖=1 𝑉𝑖𝑗 𝑂𝐼𝑖

j=p and p = 1, 2, …, m
(Here Input and Output is same for input layer, so we can take either of
two i.e. input or output for net input computation)
Add Bias in net Input.

𝐼𝐻𝑝 = 𝐼𝐻𝑝 + 𝑏𝐻𝑝

Note: Bias can be same for all inputs or different for each hidden layer.
PHASE-1: FEED-FORWARD PHASE
STEP 6: OUTPUT OF HIDDEN LAYER
Output of hidden layer can be computed by applying binary
sigmoidal function on it’s input.

p = 1, 2, . . ., m

We shall have ‘m’ such outputs which will be passed to the output
layer.
PHASE-1: FEED-FORWARD PHASE
STEP 7: INPUT FOR OUTPUT LAYER
Compute the net Input for each output neuron.

σ𝑛
𝐼𝑂𝑞 = 𝑖=1 𝑊𝑖𝑗 𝑂𝐻𝑖
j = q and q = 1, 2, . . .n

Add Bias in net Input.

Note: Bias can be same for all inputs or different for each hidden
layer.
PHASE-1: FEED-FORWARD PHASE
STEP 8: OUTPUT OF OUTPUT LAYER
Output of output layer can be computed by applying binary
sigmoidal function on it’s input.

We shall have ‘n’ such outputs which will be passed to the output
layer.
(In case of logic gates n = 1)
 PHASE – 2
ERROR BACK-PROPAGATION
PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 9: ERROR CALCULATION
For given training pair, calculate the difference between desired
output and actual output.

r = 1, 2, …, n
PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 10: CALCULATION OF DERIVATIVE OF OUTPUT
Calculate the derivative of the output of output layer.

r = 1, 2, …, n

Note: This is a derivative of binary sigmoidal function.

We shall have ‘n’ such derivatives.
PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 11: CALCULATION OF ERROR CORRECTION TERM
Calculate the error correction term.

𝐸𝑂𝑟 = 𝑒𝑟 𝑑𝑂𝑟
= (𝑂𝑡𝑎𝑟𝑔𝑒𝑡𝑟 − 𝑂𝑎𝑐𝑡𝑢𝑎𝑙𝑟 )(1 − 𝑂𝑂𝑟 )(𝑂𝑂𝑟 )
r = 1, 2, …, n

We shall have ‘n’ such error correction terms.

PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 12: CHANGE IN WEIGHTS AND BIAS
Calculate change in weights and bias between hidden and output
layer.

And

Where,

𝛼 = Learning rate
PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 13: DELTA INPUTS TO HIDDEN LAYER
Compute the delta inputs to every hidden layer.

Where,

We shall have ‘m’ such a delta inputs which will be passed to hidden
layer.
PHASE-2: ERROR BACK-PROPAGATION PHASE
STEP 14: CHANGE IN WEIGHTS AND BIAS
Calculate change in weights and bias between input and hidden
layer.

And

Where,

𝛼 = Learning rate
 PHASE – 3
WEIGHT AND BIAS UPDATE
PHASE-3: WEIGHT UPDATION PHASE
STEP 15: UPDATE WEIGHTS AND BIAS
Update the weights and bias between hidden and output layer.

And
PHASE-3: WEIGHT UPDATION PHASE
STEP 16: UPDATE WEIGHTS AND BIAS
Update the weights and bias between input and hidden layer.

And
PHASE-3: WEIGHT UPDATION PHASE
STEP 17:
Check for the stopping condition.
A stopping condition may be certain number of epochs (iterations) or
when actual output matches with the targets output.
STOPPING CONDITION:
Compute the Error Norm as follows:
𝐸𝑁𝑘 = 1ൗ2 𝑒𝑘 2
For every training pair,

n = total outputs

Also called as ‘Mean-squared Error’

If E is acceptably low than tolerance, then stop.
STOPPING CONDITION:

The Euclidean norm of Error:

n = total outputs
SOLVED EXAMPLE:

SR. NO. INPUTS OUTPUT

I1 I2 O
1 0.4 -0.7 0.1
2 0.3 -0.5 0.05
3 0.6 0.1 0.3
4 0.2 0.4 0.25
5 0.1 -0.2 0.12
SOLVED EXAMPLE:

I1 0.1

0.2
0.4 -0.2
O

I2 -0.5

0.2
Given:
I1 = 0.4, I2 = -0.7
V11 = 0.1, V12 = 0.4, V21 = -0.2, V22 = 0.2
W11 = 0.2, W21 = -0.5
a=1
1. Output of Input Layer will be same as its Input
because of Linear Activation Function.

II1 = 0.4, II2 = -0.7

OI1 = 0.4 OI2 = -0.7

2. Input to Hidden Layer
IH1 = V11 * I1 + V21 * I2
= (0.1 * 0.4) + (-0.2 * -0.7)
= 0.04 + 0.14
IH1 = 0.18
IH2 = V12 * I1 + V22 * I2
= (0.4 * 0.4) + (0.2 * -0.7)
= 0.16 - 0.14
IH2 = 0.02
3. Output of Hidden Layer
OH1 = 1 / (1 + 𝑒 −0.18)
= 0.5448

OH2 = 1 / (1 + 𝑒 −0.02)
= 0.505
4. Input to Output Layer
IO1 = W11 * OH1 + W21 * OH2
= (0.2 * 0.5448) + (-0.5 * 0.505)
= 0.10896 - 0.2525
IO1 = -0.14354

5. Output of Output Layer

OO1 = 1 / (1 + 𝑒 0.14354 )
OO1 = 0.4642
6. Error
e = (0.1 – 0.4642)
e = -0.3642

7. Derivative of Output of Output layer

d = (1-OO1) * OO1
= (1 – 0.4642) * 0.4642
= 0.5358 * 0.4642
d = 0.2487
8. Error Correction Term
E=e*d
= -0.3642 * 0.2487
E = -0.09058
9. Change in weights of H-O layer
∆𝑊 11 = a * E * OH1
= 1 * (-0.09258) * 0.5448
∆𝑾11 = -0.0493

∆𝑊 21 = a * E * OH2
= 1 * (-0.09258) * 0.505
∆𝑾21 = -0.0457
10. Derivative of output of Hidden Layer
dH1 = (1 – OH1) * OH1
= (1-0.5448) * 0.5448
= 0.4552 * 0.5448
dH1 = 0.2479
dH2 = (1 – OH2) * OH2
= (1-0.505) * 0.505
= 0.495 * 0.505
dH2 = 0.2499
10. Delta inputs to Hidden Layer
d1 = dH1 * W11 * E
= 0.2479 * 0.2 * -0.09058
d1 = -0.00449

d2 = dH2 * W21 * E
= 0.2499 * -0.5 * -0.09058
d2 = 0.01131
11. Change in weights of I-H layer
∆𝑉11 = a * d1 * OI1
= 1 * (-0.00449) * 0.4
∆𝑽11 = -0.001796

∆𝑉21 = a * d1 * OI2
= 1 * (-0.00449) * (-0.7)
∆𝑽12 = 0.003143
∆𝑉12 = a * d2 * OI1
= 1 * (0.01131) * 0.4
∆𝑽21 = 0.004524

∆𝑉22 = a * d2 * OI2
= 1 * (0.01131) * (-0.7)
∆𝑽22 = -0.007917
12. Weight Update (H-O layer)
W11 = W11 + ∆W11
= 0.2 – 0.0493
W11 = 0.1507

W21 = W21 + ∆W21

= -0.5 – 0.0457
W21 = -0.5457
12. Weight Update (I-H layer)
V11 = V11 + ∆V11
= 0.1 – 0.001796
V11 = 0.098204

V12 = V12 + ∆V12

= 0.4 + 0.004524
V12 = 0.404524
12. Weight Update (I-H layer)
V21 = V21 + ∆V21
= -0.2 + 0.003143
V21 = -0.196857

V22 = V22 + ∆V22

= 0.2 – 0.007917
V22 = 0.192083
STOPPING CONDITION:

EN = ½ * e * e
= ½ * (-0.3642) * (-0.3642)
= ½ * 0.13264
EN = 0.06632
ADDING MOMENTUM IN WEIGHTS

To improve the rate of convergence, we can add momentum term in

weights.

And