Introduction to Artificial Neural
Networks
Andrew L. Nelson
Visiting Research Faculty
University of South Florida
1
Overview
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Outline to the left
Current topic in red
Introduction
History and Origins
Biologically Inspired
Applications
Perceptron
Activation Functions
Hidden Layer Networks
Training with BP
Examples
Neural Networks
References
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
W. S. McCulloch, W. Pitts, "A logical calculus
of the ideas immanent in nervous activity,"
Bulletin of Mathematical Biophysics, vol. 5 pp.
115-133, 1943.
J. L. McClelland, D. E. Rumelhart, Parallel
Distributed Processing: Explorations in the
Microstructure of Cognition, The MIT Press,
1986.
C. M. Bishop, Neural Networks for Pattern
Recognition, Oxford University Press, 1995.
Neural Networks
Introduction
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Artificial Neural Networks (ANN)
Connectionist computation
Parallel distributed processing
Computational models
Biologically Inspired computational
models
Machine Learning
Artificial intelligence
Neural Networks
History
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
McCulloch and Pitts introduced the
Perceptron in 1943.
Simplified model of a biological neuron
Fell out of favor in the late 1960's
(Minsky and Papert)
Perceptron limitations
Resurgence in the mid 1980's
2/9/2004
Nonlinear Neuron Functions
Back-propagation training
Neural Networks
Summary of Applications
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Function approximation
Pattern recognition
Signal processing
Modeling
Control
Machine learning
Neural Networks
Biologically Inspired
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Electro-chemical signals
Threshold output firing
Neural Networks
The Perceptron
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Binary classifier functions
Threshold activation function
Neural Networks
The Perceptron: Threshold
Activation Function
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Binary classifier functions
Threshold activation function
Neural Networks
Linear Activation functions
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Output is scaled sum of inputs
y u wn xn
n 1
Neural Networks
10
Nonlinear Activation Functions
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Sigmoid Neuron unit function
1
y hid (u )
1 e u
Neural Networks
11
Layered Networks
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Inputs
Hidden Neurons
Output Neurons
Outputs
x1
y h1 ( u )
y o1 ( u )
y1
x2
y h2 ( u )
y o2 ( u )
y2
xn
y hn ( u )
y om ( u )
ym
Neural Networks
12
SISO Single Hidden Layer Network
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
Can represent and single input
single output functions: y = f(x)
Hidden Neurons
hid,1
hid,1
(u )
hid,2
(u )
hid,2
out,1
out,2
Output Neuron
Input
Output
hid,N
2/9/2004
hid,N
out
(u )
out,N
(u )
Neural Networks
13
Training Data Set
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
Adjust weights (w) to learn a given target
function: y = f(x)
Given a set of training data XY
Hidden Neurons
hid,1
hid,1
(u )
hid,2
(u )
hid,2
out,1
out,2
Output Neuron
Input
Output
hid,N
2/9/2004
hid,N
out
(u )
out,N
(u )
Neural Networks
14
Training Weights: Error Back-Propagation (BP)
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Weight update formula:
w( k 1) w( k ) w
e(i )
w(i ) *
w
Neural Networks
15
Error Back-Propagation (BP)
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Training error term: e
1
e ( y out y train ) 2
2
Neural Networks
16
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
e( y out , y train ) e( y out (u out ,1 ), y train )
e( y out ( wout ,1 y hid ,1 ), y train )
e( y out ( wout ,1 y hid (u hid ,1 )), y train )
e( y out ( wout ,1 y hid ( whid ,1 x )), y train )
2/9/2004
Neural Networks
17
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
e
e y out u out ,1 y hid u hid ,1
whid y out u out ,1 y hid u hid ,1 whid ,1
u hid ,1 yhid uout ,1 yout e
e
whid
whid ,1 u hid ,1 y hid uout ,1 yout
Neural Networks
18
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
u hid ,1
whid ,1
whid ,1 x
whid ,1
Neural Networks
19
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
dy hid (u ) d
1
du
du 1 e u
(1 e u ) 2 ( e u )
1 e u 1
(1 e u ) 2
1 e u
1
(1 e u ) 2 (1 e u ) 2
1
1
(1 e u ) (1 e u ) 2
1
1
1
(1 e u )
(1 e u )
y hid (u hid ,1 )1 y hid (u hid ,1 )
2/9/2004
Neural Networks
20
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
u out ,1
y hid
wout ,1 y hid ,1
y hid
wout ,1
Neural Networks
21
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
e
1
( y out y train ) 2
y out y out 2
( y out y train )
Neural Networks
22
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
u hid ,1 y hid u out ,1 y out e
e
whid whid ,1 u hid ,1 y hid u out ,1 y out
( x ) y hid (u hid ,1 )1 y hid (u hid ,1 ) ( wout ,1 )(1)( y out y train )
Neural Networks
23
BP Formulation
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
u out ,1 y out e
e
wout
wout u out ,1 y out
u out ,1 u out ,2 ... u out , N
wout ,1 y hid ,1
wout
u out ,1
( y hid )(1)( y out y train )
Neural Networks
1
( y out y train ) 2
y out 2
24
Example: The XOR problem:
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Single hidden layer: 3 Sigmoid
neurons
2 inputs, 1 output
x1
x2
Example 1
Example 2
Example 3
Example 4
Neural Networks
Desired I/O table (XOR):
25
Example: The XOR problem:
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Training error over epoch
Neural Networks
26
Example: The XOR problem:
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
initial_weights =
0.0654 0.2017 0.0769 0.1782 0.0243
0.0806 0.0174 0.1270 0.0599 0.1184
0.1335 0.0737 0.1511
final_weights =
4.6970 -4.6585 2.0932 5.5168 -5.7073
-3.2338 -0.1886 1.6164 -0.1929
-6.8066 6.8477 -1.6886 4.1531
Neural Networks
27
Example: The XOR problem:
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Mapping produced by the trained neural net:
x1
x2
Example 1
0.0824
Example 2
0.9095
Example 3
0.9470
Example 4
0.0464
Neural Networks
28
Example: Overtraining
References
Introduction
History
Biologically
Inspired
Applications
The Perceptron
Activation
Functions
Hidden Layer
Networks
Training with BP
Examples
2/9/2004
Single hidden layer: 10 Sigmoid
neurons
1 input, 1 output
Neural Networks
29
