Soft Computing 8702661

Dr. Labib Arafeh,

Associate Professor
[email protected]

Ch5 SC Fall 2016 1

 Advanced Topics
• Introduction to System Identification;
• Neuro Fuzzy Techniques;
• Artificial Neural Networks;
• NeuroFuzzy Examples
• ANFIS

Ch5 SC Fall 2016 2

 System Identification
• Goal: To determine a mathematical model for an
unknown system (or target system) by observing its
input-output data pairs.
• Purposes: To
– Predict a system’s behavior, as in Time Series Prediction,
Load, Weather Forecasting, etc.;
– Explain the Input-Output’s interactions and relationships
of a system;
– Design a controller based on the model of a system, as an
aircraft or ship control;
– Simulate the system under control once the model is
known.
• To identify a system: Two main steps are involved:
S t r u c t u r e identification and P a r a m e t e r identification
Ch5 SC Fall 2016 3
–Structure identification :
• Apply a-priori knowledge about the target system to
determine a class of models within which the search for
the most suitable model is to be conducted; this C l a s s
o f M o d e l is denoted by y  f (u, ) where:
– y is the model output
– u is the input vector
–  is the parameter vector
•f depends on the:
– Problem Under Processing,
– Designer’s Experience, &
– Laws of nature governing the target system.
Ch5 SC Fall 2016 4
–Parameter identification :
• Once, the structure of the model is known (previous
Stage), we need to apply optimization techniques in
order to determine the parameter vector   ˆ
ˆ f
such that the resulting model y (u,ˆ)
describes the system appropriately:
yi  yˆ  0 with yi assigned to ui

Block diagram
for parameter
identification

Ch5
• The Training Data  Data set composed of m desired
Input-Output pairs (u i ; y i ), (i = 1,…,m);
• System Identification needs to perform Structure and
Parameter Identification repeatedly until a Satisfactory
model is found, making use of the following procedure:
1. Specify & Parameterize a Class of Mathematical Models
representing the System to be identified;
2. Perform Parameter Identification to choose the
parameters that best fit the training data set;
3. Conduct validation set to see if the identified model
responds correctly to an unseen data set (Cross
Validation!);
4. Terminate the procedure once the results of the
validation test are satisfactory. Otherwise, another class
of model is selected & repeat step 2 to 4
Ch5 SC Fall 2016 6
 NeuroFuzzy Techniques
• Soft computing approach combines in various ways Artificial
Neural Networks, NN, and fuzzy concepts. Refer to
• Has the potential to capture the benefits of both in a single
framework.
• Inference System corresponds to a set of fuzzy IF–THEN rules
that have learning capability to approximate nonlinear
functions. Different combinations include:
• Co-operative-neural algorithm adapt fuzzy systems: During
system operation, NN & FS work independently from each
other. NN tries to learn
the parameters from the
FS. This adaptation can
be performed offline or
online while FS is applied.
Ch5 SC Fall 2016 7
 • Concurrent: The two techniques are applied one after
another as pre- or post-
processing
• Hybrid: Fuzzy system being
represented as a network
structure, making it possible to take advantage of learning
algorithm inherited from NN.
• Fuzzy models fall into two categories:
• Linguistic Models: Based on a collection of If-Then rules
and use fuzzy reasoning such as Mamdani's model:
If x is A and y is B then z is C
Where A, B, C are fuzzy sets of the universes of discourse X,
Y & Z, respectively and x & y are values of I/P variables.
• Sugeno's Model: Characterized with functional type
conclusions. A first-order Sugeno TISO model has the
If x is A and y is B then z  px  qy  r
Ch5 SC Fall 2016 8
 Artificial Neural Networks
• NN: Information Processing Paradigm inspired by
Biological Nervous Systems (our brain);
• Structure: Large number of highly interconnected processing
elements (neurons) working together;
• Learn from experience / by example (Like people) complex
functional relations by generalizing from a limited amount
of training data (I/O data observed on the system).
• Can serve as black-box models of nonlinear, multivariable
static and dynamic systems
• In a biological system: Learning involves adjustments to the
synaptic connections among neurons: Weights for NNs!
• Brain uses massively parallel computation: 1011 neurons &
104 connections per neuron in contrast with common NNs
that consist of layers of simple interconnected neurons and
weights
Ch5
assigned to theseSCinterconnections.
Fall 2016 9
• Information relevant to net’s I-O mapping is stored in weights
• Used in Image recognition and predictions based on past
knowledge problems.
• Strengths of NNs: Power: Model complex functions,
nonlinearity built into NN; Ease of Use: Learn by example;
Very little user domain-specific expertise needed; Naturally
Attractive: Based on modeling of biology, will it lead to
intelligent computers/robots/applications?
• NNs can’t do anything that can’t be done using traditional
computing techniques, BUT they can do some things which
would otherwise be very difficult.
• PROS: Adaptability to unknown situations, Robustness: fault
tolerance (network redundancy), Autonomous Learning and
Generalization. Can: Perform tasks that a linear program can
not; Be implemented in any application without any problem
• CONS: Large complexity of network structure; Needs
training & Requires high processing time for large NNs.
Ch5 SC Fall 2016 10
• A neuron Fires only if the total signal received at the cell
body Exceeds a certain (threshold) level. The neuron either
fires or it doesn't, there aren't different grades of firing.
• Entire brain is composed of the interconnected electro-
chemical transmitting neurons. Each neuron performs a
Weighted Sum of its inputs, and then fires a binary signal if
the total input exceeds a certain level. This is the model on
which ANNs are based.
• A Cell Body (soma), a branching I/P structure (DendrIte)
and a branching O/P structure (AxOn). Axons connect to
dendrites via synapses.

A Biological Neuron An Artificial Neuron (Perceptron)

Ch5 SC Fall 2016 11
• Model neurons have been implemented in:
– H/W (VLSI);
– Running huge nets of model neurons as S/W simulations in
sound time. This has advantages over its H/W counterpart.
• A Simple Artificial Neuron (Node / Unit):
– Receives inputs from other units or an external source.
– Inputs have associated weight w , which can be modified
so as to model synaptic learning.
– Unit computes activation function
f of the weighted sum (net input
to unit i (n e t i )) of its inputs;
–w i j refers to the weight from
unit j to unit i.
– The function f is the unit's activation function.
– Its output, in turn, can serve as input to other units.
Ch5 SC Fall 2016 12
• Single-Input neuron: Scalar I/P p is
multiplied by scalar weight w to
form wp that is sent to summer.
– Other I/P, 1, is multiplied by a bias
(offset) b and passed to the summer.
– The summer O/P n (net input) goes into an activation
(transfer) function f that produces the scalar neuron O/P a
• Relating this simple model back to the biological neuron:
– Weight w corresponds to the strength of a synapse,
– Summer & f (activation function) represent the cell body,
– Neuron O/P a represents the signal on the axon.
• The neuron output is calculated as: a  f ( wp  b)
• f is chosen by the designer & then w, b will be adjusted by
a learning rule so that the neuron I/O relationship meets
some specific goal.
Ch5 SC Fall 2016 13
• Network Architectures: Generally, need several neurons
operating in parallel as a layer, instead of just one neuron.
• A Layer of Neurons: A single-layer network
of neurons is shown in the figure.
– Each of R I/Ps is connected to each of neurons
– Weight matrix W has S rows (neurons).
– The layer includes W, summers, bias vector b,
the transfer function f and the O/P vector a.
– Each element of the I/P vector p is
connected to each neuron through W.
Each neuron has a bias bi, a summer, a
transfer function f and an O/P ai.
– Taken together, the O/Ps form vector a.
– It is common for the number of I/Ps to a layer to be
different from the number of neurons (R≠S).
– Neurons in a layer might have the same transfer function.
Ch5 SC Fall 2016 14
• Multiple Layers of Neurons: A network with several layers.
Each layer has its own: W, b, I/P n and an O/P vector a.
• 1st layer Weight matrix: W1,
2nd layer Weight matrix: W2
• Output Layer / Hidden Layers:
–O/P (output layer): A layer
whose O/P is the network O/P.
– Other layers are hidden layers
(1st and 2nd layers).
• # of I/Ps to the network and
# of O/Ps from the network
are defined by external problem specifications (4 external
variables (I/Ps), have 4 I/Ps to the network. Also, 7 O/Ps
from the network, have 7 neurons in the O/P layer.
• Desired characteristics of the O/P help select O/P layer f.
• For O/P is to be either -1 or 1, then a symmetrical hard limit
transfer function is appropriate.
Ch5 SC Fall 2016 15
• Choosing an Architecture: Problem specifications help
define the network in the following ways:
1. Number of network I/Ps = Number of problem inputs;
2. Number of neurons in O/P layer = Number of problem O/Ps
3. O/P layer transfer function choice at least partly
determined by problem specification of the outputs.
• Most practical NNs have two to three layers. Higher layers
are rarely used.
• The use of biases: One can choose neurons with or without
biases. The bias gives the network an extra variable, and so
you might expect that networks with biases would be more
powerful than those without, and that is true. Note that a
neuron without a bias will always have a net I/P n of zero
when the network I/Ps p are zero. This may not be
desirable and can be avoided by the use of a bias.
Ch5 SC Fall 2016 16
• Learning (Training) Scheme: Procedures to modify weights &
biases of an NN that are applied to train the NN to perform
some particular task. Two kinds of learning in NN:
– Structure Learning: Focuses on the change in the network
structure like number of neurons & their connection types
– Parameter Learning: Updating weights and biases;
• Learning rules’ wide categories: supervised & unsupervised
–Supervised learning: Training is done by providing NN
with training sets I/P & desired O/P Pairs {P1,t1}, … {Pq,tq}.
Parameters (weights & biases )are adjusted based on error
signal step-by-step to move its O/Ps closer to the targets.
–Unsupervised Learning (Self-organization): An O/P
unit is trained to respond to a finite number of clusters
within I/P. Weights & biases are modified in respond to
NN I/Ps only.
Ch5 SC Fall 2016 17
• S t a n d a r d s t e p s t o d e s i g n N N s to solve problems in
application areas like: Function Fitting, Pattern Recognition,
Clustering and Time Series Analysis:
1. Data collection (Lot of good (Appropriate & Complete)
data)): Includes Analysis, Assessing & Preprocessing;
2.Network’s construction and Configuration:
1.# of network I/Ps = # of problem I/Ps
2.# of neurons in O/P layer = # of problem O/Ps.
3.O/P layer transfer function (partly determined by
problem specification of the O/Ps.)
3. Weights and Biases Initialization;
4. Network Training: Using I/Ps – Target O/Ps;
5. Network Validation: Using New data (Cross Validation);
6. Network Implementation: Making use of it using new data
•Claimed: Only one NN algorithm (Back Propagation) has
proven to be widely effective!
Ch5 SC Fall 2016 18
• NN topologies: Pattern of connections between the units
and the propagation of data:
• Feed-forward neural networks, where
the data flow from I/P to O/P units is
strictly feedforward. Data processing
can extend over layers of units and
connections extend from O/Ps of units
to I/Ps of units in the same layer or
previous layers. E.g. Perceptron, Adaline.
• Recurrent neural networks: Have
feedback connections (connections to
the neurons of the same layer or to the
previous layers are allowed “Have
Closed Loops”). E.g. : Kohonen, Hopfield.
Ch5 SC Fall 2016 19
 NeuroFuzzy
• The Integration of learning techniques with fuzzy systems;
• Recalling NN: For an NN that has n-dimensional I/P vector,
P, from either external sources or PEs at previous layers,
and processing the data to deliver a scalar O/P, a, which is
a function of: present I/P, and the weights (W) of these
connections: a  f ( p, w)  a  f (Wp  b)
p 
• Combining NN & FSs, O/P a is given as: a  f  ( wi  pi ) 
 i 1 
• NeuroFuzzy modeling is possible by applying learning
techniques to fuzzy inference systems;
• Unsupervised learning is suitable for finding clusters of
data indicating the presence of fuzzy rules;
• Supervised and reinforcement learning are useful for
adapting the rules or MFs for the desired output in FSs.
Ch5 SC Fall 2016 20
 NeuroFuzzy Cont…
• Several NF structures have been reported in the literature
[Lin, Wang, Zia, Iisk, Jang, …etc.];
• There can be two types of cooperative combinations:
– Fuzzy-NN: NNs play the primary role of the system and the
fuzzy system support to estimate system parameters.
– NN-fuzzy: Fuzzy system plays the primary role of the
system and NN supports for learning a set of parameters.
• Jang’s Adaptive NeuroFuzzy Inference System (ANFIS): Has
used a hybrid-learning rule that combines:
– Back propagation gradient descent method and
– Least-squares estimates to rapidly identify parameters;
– ANFIS identifies parameter structure of the model;
– Clustering-based techniques may be applied for the
structure identification.
Ch5 SC Fall 2016 21
ANFIS

Ch5 SC Fall 2016 22

 ANFIS

Ch5 SC Fall 2016 23

 ANFIS-Based Forecasters
Forecasted Load
Time
Weight

Layer1: Every node stores parameters to define MBF; the outputs are the MB
values of the input part.
Layer2: Every rule is represented by one
Node (Every node multiplies the
incoming signals & outputs their
product, which represents the
Firing strength of a rule).
Layer3: Node N is the normalized
firing strength. Other nodes are
connected to the two inputs and to
exactly one node in Layer1.
Layer4: The node generates the sum (Σi Oi) of all of the outputs produced by the
previous layer (Layer3) Overall output (Load)=Σi Oi/Σi α i =(piTime+qiWeight+ri)
Ch5 SC Fall 2016 24