Neural Networks

Lecture 5
Neural Network Architectures (1)
Md. Mijanur Rahman, Prof. Dr.
Dept. of Computer Science and Engineering, Jatiya Kabi Kazi Nazrul Islam University, Bangladesh. Email: [email protected]
1
Contents of This Chapter
• This chapter covers the following topics:
– Basic Architecture of an ANN
– ANN Model and Net Input
– How do ANNs work?
– Building Blocks of ANN
– Network Topology
– Adjustments of weights or learning
– Activation functions
– Machine Learning Methods
– Activation functions
– McCulloch-Pitts Neuron

2
Basic Architecture of an ANN…
• The arrangement of neurons into layers and the connection patterns
within and between layers is called the net architecture.
• To understand the concept of the architecture of an artificial neural
network, we have to understand what a neural network consists of. In
order to define a neural network that consists of a large number of artificial
neurons, which are termed units arranged in a sequence of layers.
• Artificial Neural Network primarily consists of three layers, input layer,
hidden layer and output layer.

3
Basic Architecture of an ANN
• Input Layer: As the name suggests, it accepts inputs in several different
formats provided by the programmer.
• Hidden Layer: The hidden layer presents in-between input and output
layers. It performs all the calculations to find hidden features and
patterns.
• Output Layer: The input goes through a series of transformations using
the hidden layer, which finally results in output that is conveyed using
this layer.
• The artificial neural network takes input and computes the weighted sum
of the inputs and includes a bias. This computation is represented in the
form of a transfer function.

4
ANN Model and Net Input…
• The following diagram represents the general model of ANN
followed by its processing.

• For the above general model of artificial neural network, the net
input can be calculated as follows −

5
ANN Model and Net Input
• The output can be calculated by applying the activation function
over the net input.

Output = Function (Net_Input_Calculated)

• It determines weighted total is passed as an input to an activation function to
produce the output. Activation functions choose whether a node should fire or not.
Only those who are fired make it to the output layer. There are distinctive activation
functions available that can be applied upon the sort of task we are performing.

6
How do ANNs work?...
• Artificial Neural Network can be best represented as a weighted directed
graph, where the artificial neurons form the nodes. The association
between the neurons outputs and neuron inputs can be viewed as the
directed edges with weights.
• The Artificial Neural Network receives the input signal from the external
source in the form of a pattern and image in the form of a vector.

7
How do ANNs work?...
• These inputs are then mathematically assigned by the notations x(n) for
every n number of inputs. The weighted input is passed to an activation
function to produce output, y(n).
• Information flows through a neural network in two ways. When it's
learning (being trained) or operating normally (after being trained), patterns
of information are fed into the network via the input units, which trigger the
layers of hidden units, and these in turn arrive at the output units.

8
How do ANNs work?
• This common design is called a feedforward network. Not all units "fire"
all the time. Each unit receives inputs from the units to its left, and the
inputs are multiplied by the weights of the connections they travel
along.
• Every unit adds up all the inputs it receives in this way. If the sum is more
than a certain threshold value, the unit "fires" and triggers the units it's
connected to (those on its right).
• For a neural network to learn, there has to be an element of feedback. The
feedback in involved to compare the desired outcome with actual
output. It needs to figure out the difference between the two (errors), and
minimize the errors by adjusting the weights.

9
Building Blocks of ANN
• Neural networks are made of shorter modules or building blocks,
same as atoms in matter and logic gates in electronic circuits. Once
we know what the blocks are, we can combine them to solve a
variety of problems.
• Processing of Artificial neural network depends upon the given
three building blocks:
• Network Topology
• Adjustments of weights or learning
• Activation functions

10
Network Topology…
• A network topology is the arrangement of a network along with its nodes
and connecting lines. It refers to the way how Neurons are associated, and
it is a significant factor in network functioning and learning.
1. A common topology in unsupervised learning is a direct mapping of
inputs to a group of units that represents categories, for example, self-
organizing maps.
2. The most widely recognized topology in supervised learning is
completely associated, three-layer, feedforward network
(Backpropagation, Radial Basis Function Networks).
• All input values are associated with all neurons in the hidden layer (hidden
because they are not noticeable in the input or the output), the output of the
hidden neurons are associated to all neurons in the output layer, and the
activation functions of the output neurons establish the output of the entire
network.

11
Network Topology…
• According to the topology, ANN can be classified as the following
kinds −
– Feedforward Network
• Single layer feedforward network
• Multilayer feedforward network

– Feedback Network
• Recurrent networks
• Fully recurrent network
• Jordan network

12
Network Topology…
• Feedforward Network: Single layer and Multilayer feedforward
networks

Single layer feedforward networks Multilayer feedforward networks

13
Network Topology
• Feedback Network: Recurrent, Fully recurrent and Jordan
networks

Recurrent Network Fully Recurrent Network

Jordan Network
14
 Feedforward Network…
• The advancement of layered feed-forward networks initiated in the
late 1950s, given by Rosenblatt's perceptron and Widrow's Adaptive
linear Element (ADLINE).
• The perceptron and ADLINE can be defined as a single layer networks
and are usually referred to as single-layer perceptron's. Single-layer
perceptron's can only solve linearly separable problems.
• The limitations of the single-layer network have prompted the
advancement of multi-layer feed-forward networks with at least one
hidden layer, called multi-layer perceptron (MLP) networks.
• MLP networks overcome various limitations of single-layer perceptron's
and can be prepared to utilize the backpropagation algorithm. In 1974,
Werbos created a backpropagation training algorithm

15
 Feedforward Network…
• Single-layer feedforward network:
– Rosenblatt first constructed the single-layer feedforward
network in the late 1950s and early 1990s.
– The concept of feedforward artificial neural network
having just one weighted layer.
– In other words, we can say that the input layer is
completely associated with the output layer.

16
 Feedforward Network
• Multilayer feedforward network:
– A multilayer feedforward neural network is a linkage of perceptrons in which
information and calculations flow are uni-directional, from the input data to
the outputs.
– The total number of layers in a neural network is the same as the total number
of layers of perceptrons. The easiest neural network is one with a single input
layer and an output layer of perceptrons.
– The concept of feedforward artificial neural network having more than one
weighted layer. As the system has at least one layer between the input and the
output layer, it is called the hidden layer.

17
 Feedback Network…
• A feedback based prediction refers to an approximation of an outcome in
an iterative way where each iteration's operation depends on the present
outcome. Feedback is a common way of making predictions in different
fields, ranging from control hypothesis to psychology.
• Using feedback associations is also additionally exercised by biological
organisms, and the brain is proposing a vital role for it in complex
cognition.
• In other words, we can say that a feedback network has feedback paths,
which implies the signal can flow in both directions using loops. It makes
a non-linear dynamic system, which changes continuously until it reaches
the equilibrium state.
• It may be divided into the following types:
 Recurrent networks
 Fully recurrent network
 Jordan network

18
 Feedback Network…
• Recurrent network:
– The human brain is a recurrent neural network that refers to a network of
neurons with feedback connections. They are feedback networks with closed
loops. It can learn numerous behaviors, sequence, processing tasks
algorithms, and programs that are not learnable by conventional learning
techniques.
– It explains the rapidly growing interest in artificial recurrent networks for
technical applications. For example, general computers that can learn
algorithms to map input arrangements to output arrangements, with or
without an instructor. They are computationally more dominant and
biologically more conceivable than other adaptive methodologies. For
example, Hidden Markov models (no continuous internal states), feedforward
networks, and supportive vector machines (no internal states).
– Following are the two types of recurrent networks.
• Fully recurrent network
• Jordan network

19
 Feedback Network…
• Fully recurrent network:
– The most straightforward form of a fully recurrent neural network is a Multi-
Layer Perceptron (MLP) with the previous set of hidden unit activations,
feeding back along with the inputs. In other words, it is the easiest neural
network design because all nodes are connected to all other nodes and each
node works as both input and output.

• Note that the time 't' has to be

discretized, with the activations updated
at each time interval. The time scale
may compare to the activity of real
neurons, or for artificial systems
whenever step size fitting for the given
problem can be used. A delay unit
should be introduced to hold activations
until they are prepared at the next time
interval.

20
 Feedback Network
• Jordan network:
– The Jordan network refers to a simple neural structure in which only one value of
the process input signal (from the previous sampling) and only one value of the
delayed output signal of the model (from the previous sampling) are utilized as
the inputs of the network. It is a closed loop network in which the output will go
to the input again as feedback as shown in the following diagram.

 In order to get a computationally basic

MPC (Model Predictive Control)
algorithm, the nonlinear Jordan neural
model is repeatedly linearized on line
around an operating point, which
prompts a quadratic optimization issue.
 Adequacy of the described MPC
algorithm is compared with that of the
nonlinear MPC scheme with on-line
nonlinear optimization performed at
each sampling instant.

21
NEURAL NETWORK
ARCHITECTURES To be continued…