11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

Backpropagation in Neural Network

Last Updated : 02 Nov, 2024

Backpropagation (short for "Backward Propagation of Errors") is a

method used to train artificial neural networks. Its goal is to reduce the
difference between the model’s predicted output and the actual output
by adjusting the weights and biases in the network.

In this article, we will explore what backpropagation is, why it is

crucial in machine learning, and how it works.

Table of Content
What is Backpropagation?
Working of Backpropagation Algorithm
Example of Backpropagation in Machine Learning
Backpropagation Implementation in Python for XOR Problem
Advantages of Backpropagation for Neural Network Training
Challenges with Backpropagation

A neural network is a structured system composed of computing units

called neurons, which enable it to compute functions. These neurons are
interconnected through edges and assigned an activation function,
along with adjustable parameters. These parameters allow the neural
network to compute specific functions. Regarding activation functions,
higher activation values indicate greater neuron activation in the
network.

What is Backpropagation?
Backpropagation is a powerful algorithm in deep learning, primarily
used to train artificial neural networks, particularly feed-forward
https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 1/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

networks. It works iteratively, minimizing the cost function by adjusting

weights and biases.

In each epoch, the model adapts these parameters, reducing loss by

following the error gradient. Backpropagation often utilizes optimization
algorithms like gradient descent or stochastic gradient descent. The
algorithm computes the gradient using the chain rule from calculus,
allowing it to effectively navigate complex layers in the neural network
to minimize the cost function.

fig(a) A simple illustration of how the backpropagation works by adjustments of weights

Why is Backpropagation Important?

Backpropagation plays a critical role in how neural networks improve
over time. Here's why:

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 2/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

1. Efficient Weight Update: It computes the gradient of the loss

function with respect to each weight using the chain rule, making it
possible to update weights efficiently.
2. Scalability: The backpropagation algorithm scales well to networks
with multiple layers and complex architectures, making deep
learning feasible.
3. Automated Learning: With backpropagation, the learning process
becomes automated, and the model can adjust itself to optimize its
performance.

Working of Backpropagation Algorithm

The Backpropagation algorithm involves two main steps: the Forward
Pass and the Backward Pass.

How Does the Forward Pass Work?

In the forward pass, the input data is fed into the input layer. These
inputs, combined with their respective weights, are passed to hidden
layers.

For example, in a network with two hidden layers (h1 and h2 as shown
in Fig. (a)), the output from h1 serves as the input to h2. Before applying
an activation function, a bias is added to the weighted inputs.

Each hidden layer applies an activation function like ReLU (Rectified

Linear Unit), which returns the input if it’s positive and zero otherwise.
This adds non-linearity, allowing the model to learn complex
relationships in the data. Finally, the outputs from the last hidden layer
are passed to the output layer, where an activation function, such as
softmax, converts the weighted outputs into probabilities for
classification.

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 3/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

The forward pass using weights and biases

How Does the Backward Pass Work?

In the backward pass, the error (the difference between the predicted
and actual output) is propagated back through the network to adjust
the weights and biases. One common method for error calculation is the
Mean Squared Error (MSE), given by:

MSE = (Predicted Output − Actual Output)2

Once the error is calculated, the network adjusts weights using

gradients, which are computed with the chain rule. These gradients
indicate how much each weight and bias should be adjusted to minimize
the error in the next iteration. The backward pass continues layer by
layer, ensuring that the network learns and improves its performance.
The activation function, through its derivative, plays a crucial role in
computing these gradients during backpropagation.

Example of Backpropagation in Machine Learning

Let’s walk through an example of backpropagation in machine learning.
Assume the neurons use the sigmoid activation function for the forward
and backward pass. The target output is 0.5, and the learning rate is 1.

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 4/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

Example (1) of backpropagation sum

Here's how backpropagation is implemented:

Forward Propagation

1. Initial Calculation
The weighted sum at each node is calculated using:

aj = ∑(wi , j ∗ xi )
​ ​ ​

Where,

aj is the weighted sum of all the inputs and weights at each node,

wi,j represents the weights associated with the j th input to the ith

neuron,
xi represents the value of the j th input,
​

2. Sigmoid Function
The sigmoid function returns a value between 0 and 1, introducing non-
linearity into the model.
1
yj =
​

1+e−aj
​
​

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 5/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

To find the outputs of y3, y4 and y5

3. Computing Outputs
At h1 node,
a1 = (w1,1 x1 ) + (w2,1 x2 )
​ ​ ​ ​ ​

= (0.2 ∗ 0.35) + (0.2 ∗ 0.7)

​

= 0.21

Once, we calculated the a1 value, we can now proceed to find the y3

value:
1
yj = F (aj ) =
​ ​

1+e−a1 ​
​

1
y3 = F (0.21) =
​

1+e−0.21
​

y3 = 0.56
​

Similarly find the values of y4 at h2 and y5 at O3 ,

a2 = (w1,2 ∗ x1 ) + (w2,2 ∗ x2 ) = (0.3 ∗ 0.35) + (0.3 ∗ 0.7) = 0.315

​ ​ ​ ​ ​

1
y4 = F (0.315) =
​

1+e−0.315
​

a3 = (w1,3 ∗ y3 ) + (w2,3 ∗ y4 ) = (0.3 ∗ 0.57) + (0.9 ∗ 0.59) = 0.702

​ ​ ​ ​

1
y5 = F (0.702) =
​

1+e−0.702
​ = 0.67

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 6/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

Values of y3, y4 and y5

4. Error Calculation
Note that, our actual output is 0.5 but we obtained 0.67.

To calculate the error, we can use the below formula:

Errorj = ytarget − y5
​ ​

Error = 0.5 − 0.67 = −0.17

Using this error value, we will be backpropagating.

Backpropagation

1. Calculating Gradients
The change in each weight is calculated as:

Δwij = η × δj × Oj
​ ​

Where:

δj ​is the error term for each unit,

η is the learning rate.

2. Output Unit Error

For O3:

δ5 = y5 (1 − y5 )(ytarget − y5 )
​ ​ ​ ​ ​

= 0.67(1 − 0.67)(−0.17) = −0.0376

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 7/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

3. Hidden Unit Error

For h1:

δ3 = y3 (1 − y3 )(w1,3 × δ5 )
​ ​ ​ ​ ​

= 0.56(1 − 0.56)(0.3 × −0.0376) = −0.0027

For h2:

δ4 = y4 (1 − y4 )(w2,3 × δ5 )
​ ​ ​ ​ ​

= 0.59(1 − 0.59)(0.9 × −0.0376) = −0.0819

4. Weight Updates
For the weights from hidden to output layer:

Δw2,3 = 1 × (−0.0376) × 0.59 = −0.022184

​

New weight:

w2,3 (new) = −0.22184 + 0.9 = 0.67816

​

For weights from input to hidden layer:

Δw1,1 = 1 × (−0.0027) × 0.35 = 0.000945

​

New weight:

w1,1 (new) = 0.000945 + 0.2 = 0.200945

​

Similarly, other weights are updated:

w1,2 (new) = 0.271335

​

Statistics with Python Data Analysis Tutorial Python – Data visualization tutorial NumPy Pandas
w1,3 (new) = 0.08567
​

w2,1 (new) = 0.29811

​

w2,2 (new) = 0.24267

​

The updated weights are illustrated below,

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 8/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

Through backward pass the weights are updated

Final Forward Pass:

After updating the weights, the forward pass is repeated, yielding:

y3 = 0.57
​

y4 = 0.56
​

y5 = 0.61
​

Since y5 = 0.61 is still not the target output, the process of calculating
​

the error and backpropagating continues until the desired output is

reached.

This process demonstrates how backpropagation iteratively updates

weights by minimizing errors until the network accurately predicts the
output.

Error = ytarget − y5​

= 0.5 − 0.61 = −0.11

This process is said to be continued until the actual output is gained by

the neural network.

Backpropagation Implementation in Python for XOR

Problem
This code demonstrates how backpropagation is used in a neural
network to solve the XOR problem. The neural network consists of:

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 9/18
11/12/24, 8:20 PM Backpropagation in Neural Network - GeeksforGeeks

Epoch 8000, Loss:0.0029801470220045504

Predictions after training:

[[0.02330965]
[0.95658721]
[0.95049451]
[0.05896647]]

Advantages of Backpropagation for Neural Network

Training
The key benefits of using the backpropagation algorithm are:

Ease of Implementation: Backpropagation is beginner-friendly,

requiring no prior neural network knowledge, and simplifies
programming by adjusting weights via error derivatives.
Simplicity and Flexibility: Its straightforward design suits a range of
tasks, from basic feedforward to complex convolutional or recurrent
networks.
Efficiency: Backpropagation accelerates learning by directly updating
weights based on error, especially in deep networks.
Generalization: It helps models generalize well to new data,
improving prediction accuracy on unseen examples.
Scalability: The algorithm scales efficiently with larger datasets and
more complex networks, making it ideal for large-scale tasks.

Challenges with Backpropagation

While backpropagation is powerful, it does face some challenges:

1. Vanishing Gradient Problem: In deep networks, the gradients can

become very small during backpropagation, making it difficult for the
network to learn. This is common when using activation functions
like sigmoid or tanh.
2. Exploding Gradients: The gradients can also become excessively
large, causing the network to diverge during training.
3. Overfitting: If the network is too complex, it might memorize the
training data instead of learning general patterns.

https://www.geeksforgeeks.org/backpropagation-in-neural-network/ 12/18