Net Input /Summation

Net Input /Summation

Z = W1X1 + W2X2 + …+ WnXn

But, it is incomplete.
Z=
Net Input /Summation/ Weighted_Sum

Z = W1X1 + W2X2 + …+ WnXn

But, it is incomplete.
Z=

Z = Bias + W1X1 + W2X2 + …+ WnXn

Z=
Bias
• It is an extra input (all layers including hidden layers )
• It is a weight whose activation is always=1
• Normally, it is denoted by W0 , b etc.
• It is used to control the input
• Similar to the weight, bias can be initialized randomly or with zero.
• It also participate in net input calculation
Net= ∑ (weight * Input) + bias
Net= ∑ (Wi * Xi) + bias

Net = + w0
Activation Function
• An Activation Function decides whether a neuron
should be activated or not.

• The purpose of an activation function is to

• add non-linearity to the neural network.

• Let’s suppose we have a neural network working without the

activation functions.

• In that case, every neuron will only be performing a linear

transformation on the inputs using the weights and biases. It’s
because it doesn’t matter how many hidden layers we attach
in the neural network; all layers will behave in the same
Activation Function

• The activation function defines the output of the neuron/node given a

net input.
• The o/p of the activation function acts as input to the next layers (o/p
layer, hidden layer), this is called FORWARD PROPAGATION.
• It controls the “neuron firing”.
• A neuron firing implies “it will be activated or not”, i.e., is it able to
output or not?

Suppose, Net= ∑ (Wi * Xi) + bias = 2.56 What is the meaning?

How it affects the output
How it affects the neuron firing
Neuron Firing
• Neuron firing is depend of:
 Net I/P
 Activation Function
 Threshold

• The activation function takes two inputs:

 Net I/P Discussion
-Net i/p range:
 Threshold -Neuron unable to bound net i/p
-Neuron unable to decide firing pattern
-Activation Function (AF) used to bound it
-AF first bound it and then decided neuron
firing
pattern
-AF introduces non-linearity (otherwise, it
is
Threshold

A msg/email is: spam / not spam ?

if net i/p is greater than a value (T) = its spam otherwise no

Threshold

Value ≥ T Spam
Value < T Not Spam Discussion
-unit step activation function
-shifted unit step activation
function
-binary activation function
Threshold

Neuron fired
Spam(1) if value ≥ T
f(x) = value =
Spam(0) if value < T Neuron fired

1 If (∑ (Wi * Xi) + b ≥ T Neuron fired

o/p =
0 If (∑ (Wi * Xi) + b < T Neuron fired
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 1-Binary Step Function
Binary step function depends on a threshold value that
decides whether
a neuron should be activated or not.
-The input fed to the activation function is compared to a
certain threshold; if the input is greater than it, then the
neuron is activated, else it is deactivated, meaning that its
output is not passed on to the next hidden layer.
1-Binary Step Function
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 2-Linear Activation Function
The linear activation function, also known as “identity
function" (multiplied x1.0), is where the activation is
proportional to the input.
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 2-Linear Activation Function
The linear activation function, also known as “identity
function" (multiplied x1.0), is where the activation is
proportional to the input.
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 3-Non-Linear Activation Functions
• The linear activation function shown is simply a linear regression model.

• Because of its limited power, this does not allow the model to create
complex mappings between the network’s inputs and outputs.

• Non-linear activation functions solve the following limitations of linear

activation functions:
• They allow backpropagation because now the derivative function would be
related to the input, and it’s possible to go back and understand which
weights in the input neurons can provide a better prediction.
• They allow the stacking of multiple layers of neurons as the output would
now be a non-linear combination of input passed through multiple layers.
Any output can be represented as a functional computation in a neural
network.
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 3.1-Sigmoid / Logistic Activation Function
• This function takes any real value as input and outputs
values in the range of 0 to 1.
The larger the input (more positive), the closer the output
value will be to 1.0, whereas the smaller the input (more
negative), the closer the output will be to 0.0,
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 3.1-Sigmoid / Logistic Activation Function
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 3.2-Tanh Function (Hyperbolic Tangent)

• Tanh function is very similar to the sigmoid/logistic

activation function, and even has the same S-shape with
the difference in output range of -1 to 1.
• In Tanh, the larger the input (more positive), the closer the
output value will be to 1.0, whereas the smaller the input
(more negative), the closer the output will be to -1.0.
 Activation Functions: Some Common Choices

3 Types of Neural Networks Activation Functions

• 3.2-Tanh Function (Hyperbolic Tangent)
3.3 ReLU Function

• ReLU stands for Rectified Linear Unit.

• Although it gives an impression of a linear function,

ReLU has a derivative function and allows for
backpropagation while simultaneously making it
computationally efficient.

• The main catch here is that the ReLU function does not
activate all the neurons at the same time.

• The neurons will only be deactivated if the output of the

linear transformation is less than 0.
3.3 ReLU Function
3.4 Leaky ReLU Function

• Leaky ReLU is an improved version of ReLU function to

solve the Dying ReLU problem as it has a small positive
slope in the negative area.
Activation Functions: Some Common Choices
Preferred more than
sigmoid For sigmoid as well as tanh
sigmoid. Helps keep
tanh, gradients the mean of the next
saturate (become layer’s inputs close
close to zero as the
h h to zero (with
function tends to its sigmoid, it is close to
extreme values) 0.5)

a a

Leaky ReLU
ReLU ReLU and Leaky
Helps fix the dead ReLU are among
neuron problem the most popular
h of ReLU when is a ones
negative number

a
 23

Examples of some basic NN/MLP architectures

 24
Single Hidden Layer and Single Outputs
 One hidden layer with nodes and a single output (e.g., scalar-
valued regression or binary classification)
 25
Single Hidden Layer and Multiple Outputs
 One hidden layer with nodes and a vector of output (e.g., vector-
valued regression or multi-class classification or multi-label
classification)
 26
Multiple Hidden Layers (One/Multiple Outputs)
 Most general case: Multiple hidden layers with (with same or
different number of hidden nodes in each) and a scalar or vector-
valued output
Forward Propagation in Neural Networks?
• Forward propagation is the process in a neural network
where the input data is passed through the network’s
layers to generate an output.
• It calculates the output of the network for a given input
based on the current values of the weights and biases.
Steps for Forward Propagation in Neural Network

Step 1: Parameters Initialization

We will first initialize the weight matrices and the bias vectors. it’s
important to randomly initialize the parameters to a value between
0 and 1. The learning rate will be recommended as 0.01 to make
the activation function active.

Step 2- Compute net input

Step 3: Activation function
based on specific requirements activation function must be chosen.

Step 4: Feeding forward

Pass the out of the layer to next layer as input.
Forward Propagation: understand with example

a1[0] denotes the first neuron of the input layer

a2[0] denotes the second neuron of the input layer and so on…
a1[1] denotes the first neuron of the hidden layer
a2[1] denotes the second neuron of the hidden layer and so on…
Lastly, a1[2] is the first and the only neuron in the output layer. There can be more than one
neuron in the final layer as well.
Forward Propagation: understand with example

a1[1] = f(w11[1]*a1[0] + w12[1]*a2[0] +

w13[1]*a3[0] + B1)
Forward Propagation: understand with example

A0: all inputs at input layer (0)

A1: Output of the first hidden layer (1). B1 and B2: bias
A2: Output of the second hidden layer (2)/final output if there is no second
hidden layer.
W1: weights between input and first hidden layer.
Error Calculation

• Forward pass give you a output/prediction of the network: o

• There is already available target/correct value: t

Error: t-o
Error: True_value – Predicted_value
Back Propagation in Neural Networks?

• Idea: after each round of training (Forward pass),' the network reviews its
performance on tasks. It calculates the difference between its output and the
correct answer, known as the error. Then, it adjusts its internal parameters, or
'weights,' to reduce this error next timeBackpropagation is a process involved in
training a neural network. It takes the error of a forward propagation and feeds
this loss backward through the neural network layers to fine-tune (update) the
weights.

• Proper tuning of the weights ensures lower error, making the model reliable.

• Backpropagation is an essential part of modern neural network training,

enabling these sophisticated algorithms to learn from training datasets and
improve over time. . This method is essential for tuning the neural network's
accuracy and is a foundational strategy in learning to make better predictions or
Back Propagation in Neural Networks?
Back Propagation in Neural Networks?

https://www.geeksforgeeks.org/backpropagation-in-neural-network/
Forward and Back Propagation in Neural Networks?
Training-Neural Networks?

There are overall four main steps in the deep learning

model training ( forward and backpropagation :

•Forward pass
•Errors calculation
•Backward pass
•Weights update
Training-Neural Networks?

Forward pass
This is the first step of the backpropagation process,
and it’s illustrated below:

•The data ( inputs X1 and X2) is fed to the input layer

•Then, each input is multiplied by its corresponding

weight, and the results are passed to the neurons N1X
and N2X of the hidden layers.

•Those neurons apply an activation function to the

weighted inputs they receive, and the result passes to
Training-Neural Networks?

Errors calculation

•The process continues until the output layer

generates the final output (o/p).

•The output of the network is then compared to the

ground truth (desired output), and the difference is
calculated, resulting in an error value.
Training-Neural Networks?

Backward pass

•The error value obtained previously is used to

calculate the gradient of the loss function.

•The gradient of the error is propagated back through

the network, starting from the output layer to the
hidden layers.

•As the error gradient propagates back, the weights

are updated according to their contribution to the
error. This involves taking the derivative of the error
Training-Neural Networks?

Weights update

•The weights are updated in the opposite direction of

the gradient, leading to the name “gradient descent.”
It aims to reduce the error in the next forward pass.

•This process of forward pass, error calculation,

backward pass, and weights update continues for
multiple epochs until the network performance
reaches a satisfactory level or stops improving
significantly.