UNIT -4

ARTIFICIAL NEURAL
NETWORKS, Deep Learning
 Biological Motivation

• The study of artificial neural networks (ANNs) has been inspired by the
observation that biological learning systems are built of very complex webs of
interconnected Neurons
• Human information processing system consists of brain neuron: basic building
block cell that communicates information to and from various parts of body
• Simplest model of a neuron: considered as a threshold unit –a processing element
(PE)
• Collects inputs & produces output if the sum of the input exceeds an internal
threshold value

2
 Facts of Human Neurobiology

• Number of neurons ~ 1011

• Connection per neuron ~ 10 4 – 5
• Neuron switching time ~ 0.001 second or 10 -3
• Scene recognition time ~ 0.1 second
• Highly parallel computation based on distributed representation

3
 Properties of Neural Networks

• Many neuron-like threshold switching units

• Many weighted interconnections among units
• Highly parallel, distributed process
• Emphasis on tuning weights automatically
• Input is a high-dimensional discrete or real-valued (e.g, sensor input)

4
 When to consider Neural Networks ?

• Input is a high-dimensional discrete or real-valued (e.g., sensor input)

• Output is discrete or real-valued
• Output is a vector of values
• Possibly noisy data
• Form of target function is unknown

Examples:
1. Speech recognition
2. Image classification
3. Financial Prediction

5
Neuron

6
Neuron

7
Neuron

8
Neuron

9
N EURA L N E T W O R K REPRESENTATION S

10
11
• A prototypical example of ANN learning is provided by Pomerleau's (1993)
system ALVINN, which uses a learned ANN to steer an autonomous vehicle
driving at normal speeds on public highways.

• The input to the neural network is a 30x32 grid of pixel intensities obtained from
a forward-pointed camera mounted on the vehicle.

• The network output is the direction in which the vehicle is steered.

12
• Figure illustrates the neural network representation.
• The network is shown on the left side of the figure, with the input camera image
depicted below it.
• Each node (i.e., circle) in the network diagram corresponds to the output of a
single network unit, and the lines entering the node from below are its inputs.
• There are four units that receive inputs directly from all of the 30 x 32 pixels in
the image. These are called "hidden" units because their output is available only
within the network and is not available as part of the global network output. Each
of these four hidden units computes a single real-valued output based on a
weighted combination of its 960 inputs
• These hidden unit outputs are then used as inputs to a second layer of 30 "output"
units.
• Each output unit corresponds to a particular steering direction, and the output
values of these units determine which steering direction is recommended most
strongly.
13
• The diagrams on the right side of the figure depict the learned weight values
associated with one of the four hidden units in thisANN.
• The large matrix of black and white boxes on the lower right depicts the weights
from the 30 x 32 pixel inputs into the hidden unit. Here, a white box indicates a
positive weight, a black box a negative weight, and the size of the box indicates
the weight magnitude.
• The smaller rectangular diagram directly above the large matrix shows the
weights from this hidden unit to each of the 30 output units.

14
 Architectures of Artificial Neural Networks
An artificial neural network can be divided into three parts (layers), which are
known as:
• Input layer: This layer is responsible for receiving information (data), signals,
features, or measurements from the external environment. These inputs are usually
normalized within the limit values produced by activation functions
• Hidden, intermediate, or invisible layers: These layers are composed of neurons
which are responsible for extracting patterns associated with the process or system
being analysed. These layers perform most of the internal processing from a
network.
• Output layer : This layer is also composed of neurons, and thus is responsible for
producing and presenting the final network outputs, which result from the
processing performed by the neurons in the previous layers.

15
 Architectures of Artificial Neural Networks
The main architectures of artificial neural networks, considering the neuron
disposition, how they are interconnected and how its layers are composed, can be
divided as follows:

1. Single-layer feedforward network

2. Multi-layer feedforward networks
3. Recurrent or Feedback networks
4. Mesh networks

16
Single-Layer Feedforward Architecture
• This artificial neural network has just one input layer and a single neural layer, which is also the
output layer.
• Figure illustrates a simple-layer feedforward network composed of n inputs and m outputs.
• The information always flows in a single direction (thus, unidirectional), which is from the input
layer to the output layer

17
Multi-Layer Feedforward Architecture
• This artificial neural feedforward networks with multiple layers are composed of one or more
hidden neural layers.
• Figure shows a feedforward network with multiple layers composed of one input layer with n
sample signals, two hidden neural layers consisting of n1 and n2 neurons respectively, and, finally,
one output neural layer composed of m neurons representing the respective output values of the
problem being analyzed .

18
Recurrent or Feedback Architecture
• In these networks, the outputs of the neurons are used as feedback inputs for other neurons.
• Figure illustrates an example of a Perceptron network with feedback, where one of its output
signals is fed back to the middle layer.

19
Mesh Architectures
• The main features of networks with mesh structures reside in considering the spatial arrangement
of neurons for pattern extraction purposes, that is, the spatial localization of the neurons is directly
related to the process of adjusting their synaptic weights and thresholds.
• Figure illustrates an example of the Kohonen network where its neurons are arranged within a two-
dimensional space

20
 PERCEPTRONS
• Perceptron is a single layer neural network.
• A perceptron takes a vector of real-valued inputs, calculates a linear combination
of these inputs, then outputs a 1 if the result is greater than some threshold and -1
otherwise
• Given inputs x1 through xn, the output O(x1, . . . , xn) computed by the perceptron
is

• where each wi is a real-valued constant, or weight, that determines the contribution

of input xi to the perceptron output.
• -w0 is a threshold that the weighted combination of inputs w1x1 + . . . + wnxn must
surpass in order for the perceptron to output a 1.

21
Sometimes, the perceptron function is written as,

Learning a perceptron involves choosing values for the weights w0 , . . . , wn .

Therefore, the space H of candidate hypotheses considered in perceptron learning is
the set of all possible real-valued weight vectors

Why do we need Weights and Bias?

Weights shows the strength of the particular node.
A bias value allows you to shift the activation function curve up or down

22
23
24
25
Representational Power of Perceptrons

*The perceptron can be

viewed as representing a
hyperplane decision surface
in the n-dimensional space
of instances.
*The perceptron outputs a
1 for instances lying on one
side of the hyperplane and
outputs a -1 for instances
lying on the other side

26
A single perceptron can be used to represent many Boolean functions
AND function

• IfA=0 & B=0 → 0*0.6 + 0*0.6 = 0.

This is not greater than the threshold of 1, so the output = 0.
• IfA=0 & B=1 → 0*0.6 + 1*0.6 = 0.6.
This is not greater than the threshold, so the output = 0.
• IfA=1 & B=0 → 1*0.6 + 0*0.6 = 0.6.
This is not greater than the threshold, so the output = 0.
• IfA=1 & B=1 → 1*0.6 + 1*0.6 = 1.2.
This exceeds the threshold, so the output = 1.

27
 The Perceptron Training Rule

The learning problem is to determine a weight vector that causes the perceptron to
produce the correct + 1 or - 1 output for each of the given training examples.

To learn an acceptable weight vector

• Begin with random weights, then iteratively apply the perceptron to each training
example, modifying the perceptron weights whenever it misclassifies an example.
• This process is repeated, iterating through the training examples as many times as
needed until the perceptron classifies all training examples correctly.
• Weights are modified at each step according to the perceptron training rule,
which revises the weight wi associated with input xi according to the rule.

28
• The role of the learning rate is to moderate the degree to which weights are
changed at each step. It is usually set to some small value (e.g., 0.1) and is
sometimes made to decay as the number of weight-tuning iterations increases
Drawback: The perceptron rule finds a successful weight vector when the training
examples are linearly separable, it can fail to converge if the examples are not
linearly separable.

29
 Gradient Descent and the Delta Rule

• If the training examples are not linearly separable, the delta rule converges toward
a best-fit approximation to the target concept.
• The key idea behind the delta rule is to use gradient descent to search the
hypothesis space of possible weight vectors to find the weights that best fit the
training examples.
To understand the delta training rule, consider the task of training an unthresholded
perceptron. That is, a linear unit for which the output O is given by

30
To derive a weight learning rule for linear units, specify a measure for the training
error of a hypothesis (weight vector), relative to the training examples.

Where,
• D is the set of training examples,
• td is the target output for training example d,
• od is the output of the linear unit for training example d
• E is simply half the squared difference between the target output td and the linear unit output
od, summed over all training examples.

31
 Visualizing the Hypothesis Space

• To understand the gradient descent algorithm, it is helpful to visualize the entire

hypothesis space of possible weight vectors and their associated E values as
shown in below figure.
• Here the axes w0 and wl represent possible values for the two weights of a simple
linear unit. The w0, wl plane therefore represents the entire hypothesis space.
• The vertical axis indicates the error E relative to some fixed set of training
examples.
• The arrow shows the negated gradient at one particular point, indicating the
direction in the w0, wl plane producing steepest descent along the error surface.
• The error surface shown in the figure thus summarizes the desirability of every
weight vector in the hypothesis space

32
• Given the way in which we chose to define E, for linear units this error surface must always be
parabolic with a single global minimum.
Gradient descent search determines a weight vector that minimizes E by starting with an arbitrary
initial weight vector, then repeatedly modifying it in small steps.
At each step, the weight vector is altered in the direction that produces the steepest descent along the
error surface depicted in above figure. This process continues until the global minimum error is
reached.

33
34
 Derivation of the Gradient Descent Rule

How to calculate the direction of steepest descent along the error surface?
The direction of steepest can be found by computing the derivative of E with respect
to each component of the vector . This vector derivative is called the gradient of E
with respect to , written as

35
• The gradient specifies the direction of steepest increase of E, the training rule for
gradient descent is

• Here η is a positive constant called the learning rate, which determines the step
size in the gradient descent search.
• The negative sign is present because we want to move the weight vector in the
direction that decreases E
• This training rule can also be written in its component form

36
 𝜕𝐸
Calculate the gradient at each step. The vector of derivatives that form the gradient can be obtained by
𝜕𝑤
differentiating E from Equation (2), as 𝑖

37
GRADIENT DESCENT algorithm for training a linear unit

38
To summarize, the gradient descent algorithm for training linear units is as follows:
• Pick an initial random weight vector.
• Apply the linear unit to all training examples, then computeΔwi for each weight
according to Equation (7).
• Update each weight wi by addingΔwi, then repeat this process

39
 Features of Gradient Descent Algorithm

Gradient descent is an important general paradigm for learning. It is a strategy for

searching through a large or infinite hypothesis space that can be applied whenever
1. The hypothesis space contains continuously parameterized hypotheses
2. The error can be differentiated with respect to these hypothesis parameters

The key practical difficulties in applying gradient descent are

1. Converging to a local minimum can sometimes be quite slow
2. If there are multiple local minima in the error surface, then there is no guarantee
that the procedure will find the global minimum

40
 Stochastic Approximation to Gradient Descent
• The gradient descent training rule presented in Equation (7) computes weight
updates after summing over all the training examples in D
• The idea behind stochastic gradient descent is to approximate this gradient descent
search by updating weights incrementally, following the calculation of the error
for each individual example

where t, o, and xi are the target value, unit output, and ith input for the training
example in question

41
42
One way to view this stochastic gradient descent is to consider a distinct error
function Ed ( ) for each individual training example d as follows

Where, td and od are the target value and the unit output value for training example
d.
• Stochastic gradient descent iterates over the training examples d in D, at each
iteration altering the weights according to the gradient with respect to Ed( )
• The sequence of these weight updates, when iterated over all training examples,
provides a reasonable approximation to descending the gradient with respect to
our original error function Ed ( )
• By making the value of η sufficiently small, stochastic gradient descent can be
made to approximate true gradient descent arbitrarily closely

43
The key differences between standard gradient descent and stochastic gradient
descent are
• In standard gradient descent, the error is summed over all examples before
updating weights, whereas in stochastic gradient descent weights are updated upon
examining each training example.
• Summing over multiple examples in standard gradient descent requires more
computation per weight update step. On the other hand, because it uses the true
gradient, standard gradient descent is often used with a larger step size per weight
update than stochastic gradient descent.
• In cases where there are multiple local minima with respect to stochastic gradient
descent can sometimes avoid falling into these local minima because it uses the
various ❑Ed ( ) rather than ❑E ( ) to guide its search

44
45
 MULTILAYER NETWORKS AND THE
BACKPROPAGATION ALGORITHM

Multilayer networks learned by the BACKPROPACATION algorithm are capable

of expressing a rich variety of nonlinear decision surfaces

46
• Decision regions of a multilayer feedforward network. The network shown here was trained to recognize 1 of
10 vowel sounds occurring in the context "h_d" (e.g., "had," "hid"). The network input consists of two
parameters, F1 and F2, obtained from a spectral analysis of the sound. The 10 network outputs correspond to
the 10 possible vowel sounds. The network prediction is the output whose value is highest.
• The plot on the right illustrates the highly nonlinear decision surface represented by the learned network.
Points shown on the plot are test examples distinct from the examples used to train the network.

47
 A Differentiable Threshold Unit

• Sigmoid unit-a unit very much like a perceptron, but based on a smoothed,
differentiable threshold function.
• The sigmoid unit first computes a linear combination of its inputs, then applies a
threshold to the result. In the case of the sigmoid unit, however, the threshold
output is a continuous function of its input.
• More precisely, the sigmoid unit computes its output O as

σ is the sigmoid function

48
49
 The BACKPROPAGATION Algorithm

• The BACKPROPAGATION Algorithm learns the weights for a multilayer network, given a
network with a fixed set of units and interconnections. It employs gradient descent to attempt to
minimize the squared error between the network output values and the target values for these
outputs.
• In BACKPROPAGATION algorithm, we consider networks with multiple output units rather than
single units as before, so we redefine E to sum the errors over all of the network output units.

where,
• outputs - is the set of output units in the network
• tkd and Okd - the target and output values associated with the kth output unit
• d - training example

50
51
52
Backpropagation is a supervised learning algorithm, for training Multi-layer Perceptron
(Artificial Neural Networks).

To calculate gradient vector or gradient of error surface, derivative of activation function will be
required. Hence before getting into the computation of back propagation algorithm, Let us go
through some preliminary work to find out the derivatives of commonly used activation
functions.
 REMARKS ON THE BACKPROPAGATION
ALGORITHM

Convergence and Local Minima

• The BACKPROPAGATION multilayer networks is only guaranteed to converge
toward some local minimum in E and not necessarily to the global minimum error.
• Despite the lack of assured convergence to the global minimum error,
BACKPROPAGATION is a highly effective function approximation method in
practice.
• Local minima can be gained by considering the manner in which network weights
evolve as the number of training iterations increases.

63
Common heuristics to attempt to alleviate the problem of local minima include:
1. Add a momentum term to the weight-update rule. Momentum can sometimes
carry the gradient descent procedure through narrow local minima
2. Use stochastic gradient descent rather than true gradient descent
3. Train multiple networks using the same data, but initializing each network with
different random weights

64
Representational Power of Feedforward Networks
What set of functions can be represented by feed-forward networks?
The answer depends on the width and depth of the networks. There are three quite
general results are known about which function classes can be described by which
types of Networks

1. Boolean functions – Every boolean function can be represented exactly by some

network with two layers of units, although the number of hidden units required
grows exponentially in the worst case with the number of network inputs
2. Continuous functions – Every bounded continuous function can be
approximated with arbitrarily small error by a network with two layers of units
3. Arbitrary functions – Any function can be approximated to arbitrary accuracy by
a network with three layers of units.

65
Generalization, Overfitting, and Stopping Criterion
What is an appropriate condition for terminating the weight update loop?
• One choice is to continue training until the error E on the training examples falls
below some predetermined threshold.
• To see the dangers of minimizing the error over the training data, consider how the
error E varies with the number of weight iterations

66
• Consider first the top plot in this figure. The lower of the two lines shows the monotonically
decreasing error E over the training set, as the number of gradient descent iterations grows. The
upper line shows the error E measured over a different validation set of examples, distinct from the
training examples. This line measures the generalization accuracy of the network-the accuracy
with which it fits examples beyond the training data.

67
• Consider first the top plot in this figure. The lower of the two lines shows the
monotonically decreasing error E over the training set, as the number of gradient
descent iterations grows. The upper line shows the error E measured over a
different validation set of examples, distinct from the training examples. This line
measures the generalization accuracy of the network-the accuracy with which it
fits examples beyond the training data.

68
 Unsupervised learning

•Self-organising neural nets group similar input vectors together without the use of target vectors.
Only input vectors are given to the network during training. The learning algorithm then modifies
the network weights so that similar input vectors produce the same output vector.
•Unsupervised training is good for finding hidden structure or features within a set of patterns. E.g.
the network is allowed to cluster without being biased towards looking for a pre-defined output or
classification. What the network learns is entirely data driven. After learning it is often necessary to
examine what the clusters or output units have come to mean.
 Self-Organizing Maps (SOMs)

• Biologically inspired by human brain

and nervous system
•Clustering algorithm that maps
input to best matching unit
• Maps n-dimensional input space
onto a two dimensional map
• Preserves topological relationships
in the data

11 May 2023
SOM algorithm – Finding winning neuron

Find neuron whose weights closest match the inputs

Y j  a rg m in j (  || X i  W i j ||)

winner
SOM algorithm – weight adaptation

W i j ( t  1)  W i j ( t )   h i j ( X ( t )  W i j ( t ))

h ij  e xp (  d i j
2
/2  2)
Kohonen Learning Algorithm
1. Initialise weights (random values) and set topological neighbourhood and learning rate parameters
2. While stopping condition is false, do steps 3-8
3. For each input vector X, do steps 4-6
4. For each neuron j, compute Euclidean distance:
5. Find index j such that YJ is a minimum

Y j  a rg m in j (  || X i  W i j ||)

6. For all units j within a specified neighbourhood of j, and for all i:

W i j ( t  1)  W i j ( t )   h i j ( X ( t )  W i j ( t ))
7. Update learning rate ()
8. Reduce topological neighbourhood at specified times
9. Test stopping condition.
Example
SOM applications

• Soms have been used in many applications; most often in vision systems for vector
quantisation.
•Vision based autonomous robot navigation
systems historically use pre-determined models of
the environment.
•Kohonen SOMs have recently been investigated
to allow such systems to develop their own
internal representation of the external
environment.

Vector quantisation

Grey scale video images are pre-processed into multiple local texture energy feature images.
The resultant feature vectors for each pixel are then clustered using a Kohonen SOM.
“Traditional” machine learning:

handcrafted learned
cat
features classifier

Deep, “end-to-end” learning:

learned learned learned

learned
low-level mid-level high-level cat
classifier
features features features
Concept of 1-D and 2-D Convolution Layers
Case Study of CNN for Diabetic retinopathy :

Diabetic retinopathy also known as diabetic eye disease, is a medical state in which
destruction occurs to the retina due to diabetes mellitus, It is a major cause of
blindness in advance countries.
Diabetic retinopathy influence up to 80 percent of those who have had diabetes for 20
years or more.
The overlong a person has diabetes, the higher his or her chances of growing diabetic
retinopathy.
It is also the main cause of blindness in people of age group 20-64.
Diabetic retinopathy is the outcome of destruction to the small blood vessels and
neurons of the retina.
Method and Network Architecture:

Large number of convolution layers are used to allow the network to learn deeper
features. The first layer learns edges, the last convolutional layer, should learn the
features of classification of DR.
The network starts with convolution blocks with activation and then batch
normalisation after each convolution layer. As the number of feature maps
increases we move to one batch normalisation per block.
All maxpooling is performed with kernel size 3x3 and 2x2 strides. After the final
convolutional block the network is flattened to one dimension. We then perform
dropout on dense layers until we reach the dense five node classification layer
which uses a softmax activation function to predict the final classification. The
leaky rectified linear unit activation function was used applied with a value of 0.01
to stop over reliance on certain nodes in the network.
The study is used for five-class problem screening of DR can be approached using a CNN
method. The CNN has shown promising signs of being able to learn the features required
to classify accurately.

The potential benefit of using CNN is it can classify thousands of images every minute
allowing it to be used in real-time whenever a new image is acquired. In practice images
are sent to clinicians for grading and not accurately graded when the patient is in for
screening. The trained CNN makes a quick diagnosis and instant response to a patient
possible.
Case Study of CNN Self -Driving Car:

The first self-driving car was invented in 1989, it was the Automatic Land
Vehicle in Neural Network (ALVINN). It used neural networks to detect lines,
segment the environment, navigate itself, and drive. It worked well, but it was
limited by slow processing powers and insufficient data.
With today’s high-performance graphics cards, processors, and huge amounts of
data, self-driving is more powerful than ever. If it becomes mainstream, it will
reduce traffic congestion and increase road safety.
Self-driving cars are autonomous decision-making systems. They can process
streams of data from different sensors such as cameras, LiDAR, RADAR, GPS, or
inertia sensors. This data is then modeled using deep learning algorithms, which
then make decisions relevant to the environment the car is in.
Convolutional neural networks (CNN) are used to model spatial information, such as
images. CNNs are very good at extracting features from images, and they’re often
seen as universal non-linear function approximators.
CNNs can capture different patterns as the depth of the network increases. For
example, the layers at the beginning of the network will capture edges, while the deep
layers will capture more complex features like the shape of the objects (leaves in trees,
or tires on a vehicle). This is the reason why CNNs are the main algorithm in self-
driving cars.
The key component of the CNN is the convolutional layer itself. It has a convolutional
kernel which is often called the filter matrix. A CNN Arcitecture for self driving car is
given in the following figure: