Photogrammetry & Robotics Lab

Intro to Neural Networks

Part 1: Network Basics

Cyrill Stachniss

The slides have been created by Cyrill Stachniss.

1
5 Minute Preparation for Today

https://www.ipb.uni-bonn.de/5min/
2
Image Classification

input classifier output

“cat”

“5”
3
Semantic Segmentation

“a label for
each pixel”

4
Neural Networks
§ Machine learning technique
§ Often used for classification, semantic
segmentation, and related tasks
§ First ideas discussed in the 1950/60ies
§ Theory work on NNs in the 1990ies
§ Increase in attention from 2000 on
§ Deep learning took off around 2010
§ CNNs for image tasks from 2012 on

5
 Part 1
Neural Networks Basics

6
 Neural Network

What is a neuron? What is a network?

fundamental unit connected elements

(of the brain)

neural networks are connected

elementary (computing) units

7
Biological Neurons
Biological neurons are the fundamental
units of the brain that
§ Receive sensory input from the
external world or from other neurons
§ Transform and relay signals
§ Send signals to other neurons and
also motor commands to the muscles

8
Artificial Neurons
Artificial neurons are the fundamental
units of artificial neural networks that
§ Receive inputs
§ Transform information
§ Create an output

9
Neurons
§ Receive inputs / activations from
sensors or other neurons
§ Combine / transform information
§ Create an output / activation

10
Neurons as Functions
We can see a neuron as a function
§ Input given by
§ Transformation of the input data can
be described by a function
§ Output

11
Neural Network
§ NN is a network/graph of neurons
§ Nodes are neurons
§ Edges represent input-output
connections of the data flow

input computations output

12
Neural Network as a Function
§ The whole network is again a function
§ Multi-layer perceptron or MLP is often
seen as the “vanilla” neural network

input layer hidden layers output layer

13
Neural Networks are Functions
§ Neural networks are functions
§ Consist of connected artificial neurons
§ Input layer takes (sensor) data
§ Output layer provides the function
result (information or command)
§ Hidden layers do some computations

input layer hidden layers output layer

14
Different Types of NNs
§ Perceptron
§ MLP – Multilayer perceptron
§ Autoencoder
§ CNN – Convolutional NN
§ RNN – Recurrent NN
§ LSTM – Long/short term memory NN
§ GANs – Generative adversarial network
§ Graph NN
§ Transformer
§ ...
15
 [Image courtesy: van Veen]
16
 [Image courtesy: van Veen]
17
Multi-layer Perceptron (MLP)

18
Multi-layer Perceptron
Seen as a Function

input layer hidden layers output layer

19
Image Classification Example

“cat”

input function that maps label

image images to labels

20
What is the Network’s Input?

An image consists
of individual pixels.

image 21
What is the Network’s Input?

An image consists
of individual pixels.
pixel intensities
Each pixel stores
an intensity value.

image 22
What is the Network’s Input?
pixel intensities

An image consists
of individual pixels.
Each pixel stores
an intensity value.

image 23
What is the Network’s Input?

An image consists
of individual pixels.
Each pixel stores
an intensity value.
We have N+1 such
intensity values.

24
What is the Network’s Input?

Arrange all the

intensity values
in a N+1 dim vector.
25
What is the Network’s Input?

Arrange all the

intensity values
in a N+1 dim vector.
26
What is the Network’s Input?

Arrange all the

intensity values
in a N+1 dim vector.
27
Input Layer of the Network

This vector is
the input layer
of our network!
28
What is the Network’s Output?

“cat”

29
What is the Network’s Output?

Is it a...
cat or a
dog or a
human or a
...?

30
What is the Network’s Output?

Is it a...
cat or a
dog or a
human or a
...?

indicator
vector
31
What is the Network’s Output?

Is it a...
cat or a
dog or a
human or a
...?

indicator
vector
32
What is the Network’s Output?

Is it a...
cat or a
dog or a
human or a
...?

we are
never
certain...
33
Output of the Network

the output layer is vector

indicating an activation/
likelihood for each label
34
Image Classification

“cat”

largest
value

pixels intensities output layer is a

are the values of vector of likelihoods
the input layer for the possible labels
35
Multi-layer Perceptron
Let’s Look at a Single Neuron

36
Multi-layer Perceptron
Let’s Look at a Single Neuron

37
Perceptron (Single Neuron)

output

How does this

inputs function look like?
38
Perceptron (Single Neuron)

output activation
for the next layer

activations from weights

previous layer
39
Function Behind a Neuron

(input) activations

weights

bias

activation function

output activation

40
Function Behind a Neuron
A neuron gets activated ( ) through
§ A weighted sum of input activations
§ A bias activation
§ An activation function

41
Similarity to Convolutions?
§ A neuron is similar to a convolution
§ Remember linear shift-invariant
kernels used as local operators

This part looks like the convolutions

used for defining local operators

Additionally: activation function and bias

42
Activation Function
§ Biological neurons are either active
or not active
§ We can see this as a step function:

“activated”

“no activation”

§ Bias tells us where the activation

happens
43
Activation Function

“activated”

“no activation”

§ We can model this behavior through

§ Non-smooth functions (eg, steps) have

disadvantages later down the line...
44
Sigmoid Activation Function
§ Common activation function is a
sigmoid (also called logistic function)
§ Smooth function
§ Squeezes values to [0,1]

45
ReLU Activation Function
§ Most commonly used one is the so-
called “rectified linear unit” or ReLU
§
§ Often advantages for deep networks

46
Neuron Activation
§ A neuron is only activated if

“no activation” “activated” (identity)

§ If
§ the weighted activations are larger than the
negative bias
47
Common Activation Functions
There are different activation functions
§ sigmoid()
§ ReLU()
§ tanh()
§ atan()
§ softplus()
§ identity()
§ step-function()
§ …
ReLU is often used
48
 Illustration

[Courtesy of S. Sharma]
49
Function Behind a Neuron
§ Neuron gets activated if the weighted
sum of input activations is large
enough (larger than the negative bias)

§ This is the case for all neurons in the

neural network
50
For All Neurons...

These are a lot of values! 51

Let’s Use a Matrix Notation

52
Each Layer Can Be Expressed
Through Matrix Multiplications
layer 1 layer 0

53
Do It Layer by Layer...

54
Do It Layer by Layer...

input = layer 0

layer 1

layer 2

layer k = output

That not much more than linear algebra...

55
Feedforward Networks
§ MLPs are feedforward networks
§ The information flows form left to right
§ There are no loops

§ Such networks are called feedforward

networks
§ There exist other variants (eg, RNNs)
56
 Example:
Handwritten Digit Recognition

57
Handwritten Digit Recognition

=5
28x28 pixel image
[Image courtesy: Nielsen] 58
Handwritten Digit Recognition

[Image courtesy: Nielsen/Lecun] 59

A Basic MLP Recognizing Digits

[Image courtesy: Nielsen] 60

Images to Digits - A Mapping
from 784 to 10 Dimensions

28x28
pixel
input
images output
vector
(784 dim)
(10 dim)
[Partial image courtesy: Nielsen] 61
What Happens in the Layers?

62
What Happens in the 1st Layer?

pixel
values

784 input activations = pixel intensities

784 weights = weights for pixel intensities

63
What Happens in the 1st Layer?
784 input activations = pixel intensities
784 weights = weights for pixel intensities
treat activations and weights as images

-1 0 +1
white
black
(rest doesn’t
matter)

pixel values weights effect on the

weighted sum
64
What Happens in the 1st Layer?

-1 0 +1

pixel
values weights tell us
what matters for
activating the neuron!

individual “weight images” for a neuron

support individual patterns in the image
65
Link to Local Operators Defines
Through Convolutions
§ Direct link to defining
image operators through
-1 0 +1 convolutions

Here:
§ Global (not local) operators
§ Weight matrix does not
weights tell us (yet) “slide over image”
what matters

66
Weights & Bias = Patterns
§ Weights define the patterns to look
for in the image
§ Bias tells us how well the image must
match the pattern
§ Activation functions “switches the
neuron on” if it matches the pattern

67
What Happens in the 2nd Layer?
§ The weights in layer 2 tell us which
1st layer patterns should be combined
§ The deeper we go, the more patterns
get arranged and combined

§ The last layer decides, which final

patterns make up a digit
68
What Happens in the Layers?

raw simple combined patterns

pixels patterns patterns to digits
[Image courtesy: Nielsen] 69
No Manual Features

Compared to several other classifiers,

this network also includes the feature
computations – it operates directly on
the input data, no manual features!

raw simple combined patterns

pixels patterns patterns to digits
[Image courtesy: Nielsen] 70
Classification Performance

Such a simple MLP achieves a correct

classification for ~96% of the examples

[Partial image courtesy: Nielsen] 71

Classification Performance
§ A simple MLP achieves a classification
accuracy of ~96%
§ Note that there are tricky cases

§ That is a good performance for a

simple model!
§ Improved networks achieve ~99%
72
How to Design a
Neural Network?

73
How to Make the Network
Compute What We Want?
§ So far, the network is a recipe for
sequentially performing computations
§ Structure and parameters are the
design choices
§ How to set them?

Learning!

74
Summary – Part 1
§ What are neurons and neural networks
§ Lots of different networks exists
§ Focus: multi-layer perceptrons (MLP)
§ Activations, weights, bias
§ Networks have many parameters
§ “It’s just a bunch of matrices and
vectors”
§ MLP for simple image classification
§ Part 2: Learning the parameters
75
Literature & Resources
§ Online Book by Michael Nielsen, Chapter 1:
http://neuralnetworksanddeeplearning.com/chap1.html
§ Nielsen, Chapter 1, Python3 code:
https://github.com/MichalDanielDobrzanski/DeepLearningPython
§ MNIST database:
§ http://yann.lecun.com/exdb/mnist/
§ Grant Sanderson, Neural Networks
https://www.3blue1brown.com/
§ Alpaydin, Introduction to Machine Learning

76
Slide Information
§ The slides have been created by Cyrill Stachniss as part of the
photogrammetry and robotics courses.
§ I tried to acknowledge all people from whom I used
images or videos. In case I made a mistake or missed
someone, please let me know.
§ Huge thank you to Grant Sanderson (3blue1brown) for
his great educational videos that influenced this lecture.
§ Thanks to Michael Nielsen for his free online book & code
§ If you are a university lecturer, feel free to use the course
material. If you adapt the course material, please make sure
that you keep the acknowledgements to others and please
acknowledge me as well. To satisfy my own curiosity, please
send me email notice if you use my slides.

Cyrill Stachniss, [email protected]

77