Neural Network: A Brief

Overview
Introduction
 What are Neural Networks?
◦ Neural networks are a new method of
programming computers.

◦ They are exceptionally good at performing

pattern recognition and other tasks that are
very difficult to program using conventional
techniques.

◦ Programs that employ neural nets are also

capable of learning on their own and adapting
to changing conditions.
Background
 An Artificial Neural Network (ANN) is an information
processing paradigm that is inspired by the biological nervous
systems, such as the human brain’s information processing
mechanism.

 It is composed of a large number of highly interconnected

processing elements (neurons) working in unison to solve
specific problems. NNs, like people, learn by example.

 ANN is configured for a specific application, such as pattern

recognition or data classification, through a learning process.
Learning in biological systems involves adjustments to the
synaptic connections that exist between the neurons. This is
true of ANNs as well.
 How the Human Brain learns

 In the human brain, a typical neuron collects signals from others

through a host of fine structures called dendrites.
 The neuron sends out spikes of electrical activity through a long,
thin stand known as an axon, which splits into thousands of
branches.
 At the end of each branch, a structure called a synapse converts
the activity from the axon into electrical effects that inhibit or
excite activity in the connected neurons.
 A Neuron Model
 When a neuron receives excitatory input that is sufficiently large
compared with its inhibitory input, it sends a spike of electrical activity
down its axon.
 Learning occurs by changing the effectiveness of the synapses so that
the influence of one neuron on another changes.
A Simple Neuron

 An artificial neuron is a device with many inputs

and one output.
 The neuron has two modes of operation;

◦ Training mode and

◦ Using mode.
A Simple Neuron (Cont.)
 In the training mode,
◦ the neuron can be trained to fire (or not), for particular
input patterns.

 In the using mode,

◦ when a taught input pattern is detected at the input, its
associated output becomes the current output.
◦ If the input pattern does not belong in the taught list of
input patterns, the firing rule is used to determine whether
to fire or not.

 A firing rule determines whether a neuron should fire for any

input pattern. It relates to all the input patterns, not only the
ones on which the node was trained on previously.
Pattern Recognition
 Pattern recognition can be implemented by using a feed-
forward neural network that has been trained accordingly.

 During training,
◦ the network is trained to associate outputs with input
patterns.
 When the network is used,
◦ it identifies the input pattern and tries to output the
associated output pattern.
 The power of neural networks comes to life when a pattern
that has no output associated with it, is given as an input.
◦ In this case, the network gives the output that corresponds
to a taught input pattern that is least different from the
given pattern.
Pattern Recognition (cont.)

Suppose a network is trained to

recognize the patterns T and H. The
associated patterns are all black for T
and all white for H as shown above.
Pattern Recognition (cont.)

Since the input pattern looks more like a ‘T’, when

the network classifies it, it sees the input closely
resembling ‘T’ and outputs the pattern that
represents a ‘T’.
Pattern Recognition (cont.)

The input pattern here closely

resembles ‘H’ with a slight difference.
The network in this case classifies it as
an ‘H’ and outputs the pattern
representing an ‘H’.
Pattern Recognition (cont.)

 Here the top row is 2 errors away from a ‘T’ and 3 errors away
from an H. So the top output is a black.
 The middle row is 1 error away from both T and H, so the
output is random.
 The bottom row is 1 error away from T and 2 away from H.
Therefore the output is black.
 Since the input resembles a ‘T’ more than an ‘H’ the output of
the network is in favor of a ‘T’.
A Complicated Perceptron

 A more sophisticated Neuron is know as the McCulloch and Pitts model (MCP)

 The difference is that in the MCP model, the inputs are weighted and the effect
that each input has at decision making, is dependent on the weight of the
particular input.

 The weight of the input is a number which is multiplied with the input to give
the weighted input.
A Complicated Perceptron
(cont.)
 The weighted inputs are then added together and if
they exceed a pre-set threshold value, the
perceptron / neuron fires.
 Otherwise it will not fire and the inputs tied to that
perceptron will not have any effect on the decision
making.
 In mathematical terms, the neuron fires if and only
if;
X1W1 + X2W2 + X3W3 + ... > T
A Complicated Perceptron
(contd)
 The MCP neuron has the ability to adapt to a
particular situation by changing its weights and/or
threshold.

 Various algorithms exist that cause the neuron to

'adapt'; the most used ones are the Delta rule and
the back error propagation.
Diagram of an NN

Fig: A simple Neural Network

Network Layers
 InputLayer - The activity of the input units
represents the raw information that is fed into the
network.

 Hidden Layer - The activity of each hidden unit is

determined by the activities of the input units and
the weights on the connections between the input
and the hidden units.

 Output Layer- The behavior of the output units

depends on the activity of the hidden units and the
weights between the hidden and output units.
Different types of Neural Networks

Feed-forward networks
◦ Feed-forward NNs allow signals to travel one way only;
from input to output. There is no feedback (loops) i.e. the
output of any layer does not affect that same layer.

◦ Feed-forward NNs tend to be straight forward networks

that associate inputs with outputs. They are extensively
used in pattern recognition.
Continued
 Feedback networks

◦ Feedback networks can have signals traveling in both

directions by introducing loops in the network.

◦ Feedback networks are dynamic; their 'state' is changing

continuously until they reach an equilibrium point.

◦ They remain at the equilibrium point until the input changes

and a new equilibrium needs to be found.

◦ Feedback architectures are also referred to as interactive or

recurrent, although the latter term is often used to denote
feedback connections in single-layer organizations.
Feedback Networks
Continued
 This simpletype of network is interesting because
the hidden units are free to construct their own
representations of the input.

 Theweights between the input and hidden units

determine when each hidden unit is active, and so
by modifying these weights, a hidden unit can
choose what it represents.
Network Structure
 Thenumber of layers and number of neurons
depend on the specific task. In practice this issue is
solved by trial and error.

 Two types of adaptive algorithms can be used:

◦ start from a large network and successively remove some

neurons and links until network performance degrades.

◦ begin with a small network and introduce new neurons

until performance is satisfactory.
Network Parameters
How are the weights initialized?

How many hidden layers and how

many neurons?

How many examples in the training

set?
Weights
In general, initial weights are
randomly chosen, with typical values
between -1.0 and 1.0 or -0.5 and 0.5.
There are two types of NNs. The first
type is known as
◦ Fixed Networks – where the weights are
fixed
◦ Adaptive Networks – where the weights
are changed to reduce prediction error.
Number of neurons in each layer
Input Layer
◦  the number of neurons comprising input layer is equal to
the number of features (columns) in your data. Some NN
configurations add one additional node for a bias term.

 Output Layer
◦ Machine mode: returns a class label (e.g., "Premium
Account"/"Basic Account").
 If the NN is a classifier, then it also has a single node unless softmax is
used in which case the output layer has one node per class label in your
model.
◦ Regression Mode returns a value (e.g., price)
 If the NN is a regressor, then the output layer has a single node.
Hidden Layer
◦ number of hidden layers equals one
 the situations in which performance improves
with a second (or third, etc.) hidden layer are
very few.
◦ the number of neurons in that layer is the
mean of the neurons in the input and
output layers.(may change depending on
application, use trial and error method)
Size of Training Data
 Rule of thumb:
◦ the number of training examples should be at least
five to ten times the number of weights of the
network.

 Other rule:

|W|= number of weights

|W|
N a = expected accuracy on
(1 - a) test set
Training Basics
 The most basic method of training a neural network is trial
and error.

 If the network isn't behaving the way it should, change the

weighting of a random link by a random amount. If the
accuracy of the network declines, undo the change and make
a different one.

 It takes time, but the trial and error method does produce
results.
Training: Backprop algorithm
 The Backprop algorithm searches for weight values that
minimize the total error of the network over the set of
training examples (training set).

 Backprop consists of the repeated application of the

following two passes:

◦ Forward pass: in this step the network is activated on

one example and the error of (each neuron of) the
output layer is computed.

◦ Backward pass: in this step the network error is used

for updating the weights. Starting at the output layer,
the error is propagated backwards through the network,
layer by layer. This is done by recursively computing the
local gradient of each neuron.
Back Propagation
 Backpropadjusts the weights of the NN in order to
minimize the network total mean squared error

Network activation
Forward Step

Error
propagation
Backward Step
The Learning Process
 Every neural network possesses knowledge which is
contained in the values of the connection weights.

 Modifying the knowledge stored in the network as a

function of experience implies a learning rule for
changing the values of the weights.
The Learning Process (cont.)
 Recall:Adaptive networks are NNs that allow the
change of weights in its connections.

 The learning methods can be classified in two

categories:
◦ Supervised Learning
◦ Unsupervised Learning
Supervised Learning
 Incorporates an external teacher

 Each output unit is told the desired response to input

signals
 An important issue concerning supervised learning is
the problem of error convergence, ie the minimization
of error between the desired and computed unit values.
 The network then learns from its error, that is, it
changes its weight to reduce its prediction error.

 The aim is to determine a set of weights which

minimizes the error.

 One well-known method, which is common to many

learning paradigms is the least mean square (LMS)
convergence.
Unsupervised Learning
 No external teacher and is based upon only local
information.
 It is also referred to as self-organization, in the
sense that it self-organizes data presented to the
network and detects their emergent collective
properties.
 The network is then used to construct clusters of
similar patterns.

 useful in domains were instances are checked to

match previous scenarios.
 For example, detecting credit card fraud.
Disadvantage of Neural Network

The sample size has to be large.

Requires lot of trial and error so
training can be time consuming.
Where can neural network systems help…

when we can't formulate an

algorithmic solution.
when we can get lots of examples
of the behavior we require.
‘learning from experience’
when we need to pick out the
structure from existing data.

37
Who is interested?...
Electrical Engineers – signal
processing, control theory
Computer Engineers – robotics
Computer Scientists – artificial
intelligence, pattern recognition
Mathematicians – modelling tool
when explicit relationships are
unknown

38
Uses
 Classification — A neural network can be trained
to classify given pattern or data set into predefined
class. It uses feed forward networks.
 Prediction — A neural network can be trained to
produce outputs that are expected from given
input. Eg: — Stock market prediction.
 Clustering — The Neural network can be used to
identify a special feature of the data and classify
them into different categories without any prior
knowledge of the data.
Coronary
Disease

Classification

STOP

01 10
Neural 11
Net 11 10 00 00 Input patterns
00 11

Input layer

Output layer

00 01 10 11
Sorted
00 10 11 patterns
.

00 11
40
 Clustering

00 11
10
11 10

00 11

00
01

41
 ANN Applications

Medical Applications

Information
Searching & retrieval
Chemistry

Education
Business & Management
 Applications of ANNs
 Signal processing
 Pattern recognition, e.g. handwritten characters or
face identification.
 Diagnosis or mapping symptoms to a medical case.
 Speech recognition
 Human Emotion Detection
 Educational Loan Forecasting
 Recognition of speakers in communications
 Texture analysis
 Three-dimensional object recognition
 Hand-written word recognition
 Facial recognition

43