Types Of Learning Rules in ANN

The learning rule enhances the Artificial Neural Network’s performance by applying this rule over the network. Learning in ANNs involves adjusting the weights of the connections between neurons to improve performance on a given task. This adjustment is governed by learning rules, which define how weights are updated based on input data, output, and error.

Let's explore the different types of learning rules in ANNs.

Types Of Learning Rules in ANN

1. Hebbian Learning Rule

Principle: This rule is based on the biological concept that "neurons that fire together, wire together.

Hebbian Learning Rule is an unsupervised learning algorithm used in neural networks to adjust the weights between nodes. It is based on the principle that the connection strength between two neurons should change depending on their activity patterns.

The rule can be summarized as follows:

• When two neighboring neurons operate in the same phase at the same time, the weight between them increases.
• If the neurons operate in opposite phases, the weight between them decreases.
• When there is no signal correlation between the neurons, the weight remains unchanged.

The sign of the weight between two nodes is determined by the sign of their input signals:

• If both nodes receive inputs that are either positive or negative, the resulting weight is strongly positive.
• If one node's input is positive while the other's is negative, the resulting weight is strongly negative.

Mathematical Formulation:

\delta w = \alpha x_i y

Here:

• \delta w is the change in weight.
• \alpha is the learning rate.
• x_i​ represents the input vector.
• y is the output.

This rule forms the foundation of many learning processes in artificial neural networks.

2. Perceptron Learning Rule

Principle: This rule adjusts weights to minimize classification errors.

Perceptron Learning Rule is an error-correcting algorithm designed for single-layer feedforward networks. It is a supervised learning approach that adjusts weights based on the error calculated between the desired and actual outputs. Weight adjustments are made only when an error is present.

The process is computed as follows:

1. Definitions:
   • (x_1, x_2, x_3, \dots, x_n): Set of input vectors
   • (w_1, w_2, w_3, \dots, w_n): Set of weights
   • y: Actual output
   • w_0​: Initial weight
   • w_{\text{new}}​: New weight
   • \delta w: Change in weight
   • \alpha: Learning rate
2. Computation:
   • The actual output is given by: y = w_i x_i
   • The learning signal, representing the error, is calculated as: e_j = t_i - y where t_i is the desired output.
   • The change in weight (\delta w) is determined by:\delta w = \alpha x_i e_j
   • The new weight is updated as: w_{\text{new}} = w_0 + \delta w
3. Output Calculation: The final output is based on the net input and the activation function applied to it:
   y = \begin{cases} 1, & \text{if net input} \geq \theta \\ 0, & \text{if net input} < \theta \end{cases}

This rule provides the foundation for learning in perceptrons, enabling them to make adjustments and improve performance iteratively.

3. Delta Learning Rule

Principle: Minimizes the error between the actual output and the desired output using gradient descent.

Delta Learning Rule is a supervised learning algorithm that uses a continuous activation function. Also known as the Least Mean Square (LMS) method, it aims to minimize the error across all training patterns. This rule is based on the gradient descent approach, which iteratively reduces error by updating the weights of the network.

Computed as follows:

1. Definitions:
   • (x_1, x_2, x_3, \dots, x_n): Set of input vectors
   • (w_1, w_2, w_3, \dots, w_n): Set of weights
   • y: Actual output
   • w_0​: Initial weight
   • w_{\text{new}}​: Updated weight
   • \delta w: Change in weight
   • \text{Error} = t_i - y: Difference between the target (t_i​) and actual output (y)
2. Learning Signal: The learning signal (e_j​) is calculated as:e_j = (t_i - y) \cdot y'′ where y' is the derivative of the activation function.
3. Weight Adjustment:
   • The weight change is given by: \delta w = \alpha x_i e_j = \alpha x_i (t_i - y) y'
   • The updated weight is computed as: w_{\text{new}} = w_0 + \delta w
4. Activation Function: The output y is determined by the net input and the activation function: y = f(\text{net input}) = \int w_i x_i

Weight Update Conditions:

• Case I: When t = y:
  No error exists, so there is no change in the weight.
• Case II: When t \neq y:
  The weights are updated as: w_{\text{new}} = w_0 + \delta w

This method ensures that weights are updated only when an error is present, driving the network toward reducing the overall error in successive iterations.

4. Correlation Learning Rule

Principle: Updates weights based on the correlation between input and output signals.

Correlation Learning Rule is based on principles similar to the Hebbian Learning Rule. Specifically, it states that:

• If two neighboring neurons operate in the same phase at the same time, the weight between them becomes more positive.
• Conversely, if the neurons operate in opposite phases, the weight becomes more negative.

However, unlike the Hebbian rule, the Correlation Rule is supervised. This means it uses the target response for calculating the change in weight.

The change in weight (\delta w) is calculated as:

\delta w = \alpha x_i t_j

where:

• \delta w: Change in weight
• \alpha: Learning rate
• x_i​: Input vector
• t_j​: Target value

This supervised approach allows for more precise adjustments based on the desired output, improving the learning process.

5. Out Star Learning Rule

Principle: Focuses on updating the weights of connections originating from a specific neuron to multiple target neurons.

Out Star Learning Rule was introduced by Grossberg, is a supervised training procedure designed for networks where nodes are organized in layers. In this approach, the weights connected to a specific node are adjusted to match the desired outputs of the nodes they connect to.

The weight change is calculated as:

\delta w = \alpha (t - y)

Where:

• \delta w: Change in weight
• α: Learning rate
• t: Desired output
• y: Actual output

This rule ensures that the weights adapt to produce the target outputs for the connected nodes, enabling the network to learn effectively.

6. Competitive Learning Rule

Principle: Neurons compete to activate, and only the winner gets updated (Winner-Takes-All).

In Competitive Learning Rule, output nodes compete to represent the input pattern, and only one node, the "winner," is activated.

1. Competition Among Nodes: All output nodes compete to represent the input pattern. The node with the highest output is declared the winner and is assigned an output of 1, while all other nodes are assigned 0.
2. Neuron Activation: A subset of neurons with randomly initialized weights undergoes activation. However, only one neuron—the winner—remains active at any given time.
3. Weight Update: Only the weights of the winning neuron are updated. The weights of all other neurons remain unchanged.

This rule is widely used in clustering and feature detection tasks, where the network learns to identify patterns in the input data without supervision.

7. Reinforcement Learning Rule

Principle: Learning occurs by rewarding desirable outcomes and penalizing undesirable ones.

Reinforcement Learning Rule is inspired by behavioral psychology and focuses on training a network based on feedback from its environment. This approach is neither entirely supervised nor unsupervised but relies on a reward signal to guide learning.

• Rewards and Penalties: Desirable outcomes (correct actions) are rewarded to reinforce the corresponding weights, while undesirable outcomes (incorrect actions) are penalized to discourage those weights.
• Trial and Error: The network learns through repeated interactions with the environment, refining its weights based on feedback over time.

The change in weight (\delta w) is proportional to the product of the reward signal and the input: \delta w = \alpha r x_i

Where:

• \delta w: Change in weight
• \alpha: Learning rate
• r: Reward signal
• x_i: Input vector

This rule helps networks improve performance over time by reinforcing connections associated with positive outcomes.

8. Boltzmann Learning Rule

Principle: Inspired by statistical mechanics, this rule uses probabilities to adjust weights.

Boltzmann Learning Rule is a probabilistic learning method commonly used in stochastic networks such as Boltzmann Machines. It utilizes principles from statistical mechanics to find the optimal weight configuration that minimizes energy in the network.

• Probabilistic Framework: Learning is governed by the probability of a node's state, influenced by the system's energy.
• Energy Minimization: Weights are updated to minimize the network's energy, ensuring a stable and efficient configuration.

The change in weight (\delta w) is calculated using the correlation between the states of the input and output nodes:

\delta w = \alpha (P_{\text{data}} - P_{\text{model}})

Where:

• \delta w: Change in weight
• \alpha: Learning rate
• P_{\text{data}}​: Probability of the data
• P_{\text{model}}​: Probability of the model's predictions

This rule provides a probabilistic approach to weight adjustment, making it suitable for tasks involving uncertainty and complex energy landscapes.