Scribd
Upload
71 views

Neural Network Learning Rules

This document discusses various neural network learning rules, including: - Hebbian learning, where weights increase proportionally to the product of input and output. - Perceptron learning, where weight changes equal the learning rate times the error between actual and desired output. - Delta learning for continuous activations, where weight changes equal the learning rate times the derivative of the activation function times the error. - Widrow-Hoff learning minimizes mean square error between actual and desired output. Other rules discussed include correlation, winner-take-all, and outstar learning rules. Examples are provided to illustrate how different rules update weights during training.

Uploaded by

Rajiv Punjabi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
71 views

Neural Network Learning Rules

This document discusses various neural network learning rules, including: - Hebbian learning, where weights increase proportionally to the product of input and output. - Perceptron learning, where weight changes equal the learning rate times the error between actual and desired output. - Delta learning for continuous activations, where weight changes equal the learning rate times the derivative of the activation function times the error. - Widrow-Hoff learning minimizes mean square error between actual and desired output. Other rules discussed include correlation, winner-take-all, and outstar learning rules. Examples are provided to illustrate how different rules update weights during training.

Uploaded by

Rajiv Punjabi
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 33

Neural Network

Learning Rules
By
Prof.Arun Kulkarni
Associate Professor and Head department of IT
Thadomal Shahani Engineering College
Bandra Mumbai
General symbol of neuron consisting of processing node
and synaptic connections.
 This symbolic representation shows a set of weights and the neuron's
processing unit, or node.
 The neuron output signal is given by the following relationship:
 The function f (wtx) is often referred to as an activation function.
 Its domain is the set of activation values, net, of the neuron
model, we thus often use this function as f(net).The variable net is
defined as a scalar product of the weight and input vector
 The soft-limiting activation functions are often called
sigmoidal characteristics, as opposed to the hard-limiting
activation functions.
 Typical activation functions used are

where

The above Activation functions are called bipolar continuous


and bipolar binary functions, respectively
Supervised and Unsupervised Learning
NEURAL NETWORK LEARNING RULES
 The weight vector increases in proportion to the
product of input x and learning signal r.
 The learning signal r is in general a function of wi,x, and sometimes of
the teacher's signal d,.We thus have
r = r(wi, X, di)
 The increment of the weight vector wi produced by the learning step at
time t according to the general learning rule is

 where c is a positive number called the learning constant that


determines the rate of learning. The weight vector adapted at time t
becomes at the next instant, or learning step,
 The superscript convention will be used in this text to index the
discrete-time training steps. For the k'th step we thus have from using
this convention
Hebbian Learning Rule
 The Hebbian learning rule represents a purely feedforward,
unsupervised learning.

 For the Hebbian learning rule the learning signal is equal simply to the
neuron's outputWe have

 The increment of the weight vector becomes

 The single weight Wij is adapted using the following increment:

 This can be written briefly as

 This learning rule requires the weight initialization at small random values
aroundWi = 0 prior to learning.
The Following example illustrates Hebbian learning with binary
and continuous activation functions of a very simple network
where C=1. Assume the network shown below with the initial
weight vector
Using Bipolar Binary function
Using continuous bipolar activation
function f(net), using input x1, and initial weights w1
we obtain neuron output values and the updated weights for
h = 1 as summarized in Step 1. The only difference compared
with the previous case is that instead of f(net) = sgn (net)
The training continues till you get the minor difference in new
weight and previous weight.
Perceptron Learning Rule
 For the Perceptron learning rule, the learning signal is the difference
between the desired and actual neuron's response Thus, learning is
supervised and the learning signal is equal to

Note that this rule is applicable only for binary neuron response,
In bipolar binary case the output will be +1 and -1.
So Change in weight equals to
Example
 This example illustrates the Perceptron learning rule of the network
The set of input training vectors and initial weight vector is as
follows:

 The learning constant is assumed to be c = 0.1. The teacher's desired


responses for xl , x2, x3 are dl = - 1, d2 = - 1, and d3 = 1,
respectively. The learning according to the Perceptron learning rule
progresses as follows.
This terminates the sequence of learning steps unless the training set is
recycled
Delta Learning Rule
 The delta learning rule is only valid for continuous activation
functions and in the supervised training mode. The learning signal for
this rule is called delta and is defined as follows

 The term is the derivative of the activation function f (net)


computed for
 Calculating the gradient vector with respect to wi of the squared error
defined as

the weight adjustment becomes


Example
 The delta learning requires that the value f’(net) be computed in each
step.Here assume
 for the bipolar continuous activation function
Widrow-Hoff Learning Rule
Correlation Learning Rule
 By substituting r = di into the general learning rule we obtain the
correlation learning rule. The adjustments for the weight vector and the
single weights, respectively are
Winner-Take all Learning rule
 This rule is an example of competitive learning ,and it is used for
unsupervised network training.
 The learning is based on the premise that one of the neurons in the layer,
say the m'th, has the maximum response due to input x, as shown
Outstar Learning Rule
 This rule is concerned with supervised learning; however, it is
supposed to allow the network to extract statistical properties of the
input and output signals. The weight adjustments in this rule are
computed as follows
Summary

You might also like