Scribd
Upload
0 views

Unit 3_IntrotoNN (1)

The document outlines the evolution of neural networks from their inception in the 1940s to the dominance of deep learning in the 2010s, highlighting key developments such as perceptrons, multilayer perceptrons, and convolutional and recurrent neural networks. It explains the structure and components of neural networks, including neurons, activation functions, and learning rules. Additionally, it discusses types of neural networks and the concept of linear separability in binary classification problems.

Uploaded by

nayakpragatiumesh26
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0 views

Unit 3_IntrotoNN (1)

The document outlines the evolution of neural networks from their inception in the 1940s to the dominance of deep learning in the 2010s, highlighting key developments such as perceptrons, multilayer perceptrons, and convolutional and recurrent neural networks. It explains the structure and components of neural networks, including neurons, activation functions, and learning rules. Additionally, it discusses types of neural networks and the concept of linear separability in binary classification problems.

Uploaded by

nayakpragatiumesh26
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Unit 3 – Neural Networks

3.1 Evolution of Neural Networks

➢ 1940s-1950s: Early Concepts


Neural networks began with the introduction of the first mathematical model of artificial
neurons by McCulloch and Pitts. But computational constraints made progress difficult.

Biological Neuron vs McCulloch and Pitts Neuron

➢ 1960s-1970s: Perceptrons
This era is defined by the work of Rosenblatt on perceptrons. Perceptrons are single-layer
networks whose applicability was limited to linearly separable problems.

➢ 1980s: Multilayer Perceptron and Backpropagation


Multi-layer network training was made possible by Rumelhart, Hinton, and Williams’
invention of the backpropagation method. With its emphasis on learning through
interconnected nodes, connectionism gained appeal.

➢ 1990s: Boom and Winter


With applications in image identification, finance, and other fields, neural networks saw a
boom. Neural network research however experienced a “winter” due to exorbitant
computational costs and inflated expectations.

➢ 2000s: Resurgence and Deep Learning


Larger datasets, innovative structures, and enhanced processing capability spurred a
comeback. Deep learning has shown amazing effectiveness in a number of disciplines by
utilizing numerous layers.

➢ 2010s-Present: Deep Learning Dominance


Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), two deep
learning architectures, dominated machine learning. Their power was demonstrated by
innovations in gaming, picture recognition, and natural language processing.

3.2 What are Neural Networks?

Neural networks are complex systems that mimic some features of the functioning of the human
brain. It is composed of an input layer, one or more hidden layers, and an output layer.

Network components include neurons, connections, weights, biases, propagation functions, and a
learning rule. Neurons receive inputs, governed by thresholds and activation functions.
Connections involve weights and biases regulating information transfer.

Bias: It is the constant which is added to the product of features and weights. It is used to offset
the result. It helps the models to shift the activation function towards the positive or negative side
by a constant value.

Activation Functions: It decides whether a neuron should be activated or not by calculating the
weighted sum and adding bias to it. The purpose of the activation function is to introduce non-
linearity into the output of a neuron. This nonlinearity allows neural networks to develop
complex representations and functions based on the inputs that would not be possible with a
simple linear regression model. Some types of activation functions are:
Learning Rule: The learning rules updates the weights and bias levels of a network when a
network simulates in a specific data environment. Applying learning rule is an iterative process.
It helps a neural network to learn from the existing conditions and improve its performance.
Example: Hebbian Rule, Delta Rule, Perceptron Rule etc.

3.3 Types of Neural Networks

➢ Feed-forward Networks: A feed-forward neural network is a simple artificial neural


network architecture in which data moves from input to output in a single direction. It has
input, hidden, and output layers; feedback loops are absent. Its straightforward
architecture makes it appropriate for a number of applications, such as regression and
pattern recognition.
➢ Multilayer Perceptron (MLP): MLP is a type of feed-forward neural network with
three or more layers, including an input layer, one or more hidden layers, and an output
layer. It uses nonlinear activation functions.
➢ Convolutional Neural Network (CNN): A Convolutional Neural Network (CNN) is a
specialized artificial neural network designed for image processing. It employs
convolutional layers to automatically learn hierarchical features from input images,
enabling effective image recognition and classification. CNNs have revolutionized
computer vision and are pivotal in tasks like object detection and image analysis.
➢ Recurrent Neural Network (RNN): An artificial neural network type intended for
sequential data processing is called a Recurrent Neural Network (RNN). It is appropriate
for applications where contextual dependencies are critical, such as time series prediction
and natural language processing, since it makes use of feedback loops, which enable
information to survive within the network.

Backpropagation

1. Loss Calculation: The network’s output is evaluated against the real goal values, and a
loss function is used to compute the difference.
2. Loss Reduction: A method like dradient descent is then used by the network to reduce the
loss. To lower the inaccuracy, weights are changed based on the derivative of the loss
with respect to each weight.
3. Adjusting weights: The weights are adjusted at each connection by applying this iterative
process, or backpropagation, backward across the network.
4. Training: During training with different data samples, the entire process of forward
propagation, loss calculation, and backpropagation is done iteratively, enabling the
network to adapt and learn patterns from the data.
5. Activation Functions: Model non-linearity is introduced by activation functions like the
rectified linear unit (ReLU) or sigmoid. Their decision on whether to “fire” a neuron is
based on the whole weighted input.
3.2 Linear-separability?

Linear Separability refers to the data points in binary classification problems which can be
separated using linear decision boundary. If the data points can be separated using a line, linear
function, or flat hyperplane, the problem is considered to be linearly separable.

Implementation of Linearly separable Boolean Functions (AND, OR, NOT, NAND, NOR):

Implementation of Non-linearly separable Boolean Functions (XOR, XNOR):


− MLP to implement a non-liner function like Boolean Function XOR

You might also like