Motivation—Image Data

 So far, the structure of our neural network treats all inputs interchangeably.
 No relationships between the individual inputs
 Just an ordered set of variables
 We want to incorporate domain knowledge into the architecture of a Neural
Network.

2
Motivation
Image data has important structures, such as;
 ”Topology” of pixels
 Translation invariance
 Issues of lighting and contrast
 Knowledge of human visual system
 Nearby pixels tend to have similar values
 Edges and shapes
 Scale Invariance—objects may appear at different sizes in the
image.

3
Motivation

4
Motivation

5
Motivation
High CONTRAST Low Lightening Vs. High Lightening

6
Motivation—Image Data
 Fully connected would require a vast number of parameters
 MNIST images are small (32 x 32 pixels) and in grayscale
 Color images are more typically at least (200 x 200) pixels x 3 color channels (RGB)
= 120,000 values.
 A single fully connected layer would require (200x200x3)2 = 14,400,000,000
weights!
 Variance (in terms of bias-variance) would be too high
 So we introduce “bias” by structuring the network to look for certain kinds of
patterns

7
Motivation
 Features need to be “built up”
 Edges -> shapes -> relations between shapes
 Textures
 Cat = two eyes in certain relation to one another + cat fur
texture.
 Eyes = dark circle (pupil) inside another circle.
 Circle = particular combination of edge detectors.
 Fur = edges in certain pattern.

8
Motivation

9
Motivation

10
Kernels
 A kernel is a grid of weights “overlaid” on image, centered on one
pixel
 Each weight multiplied with pixel underneath it
 Output over the centered pixel is

 Used for traditional image processing techniques:

– Blur
– Sharpen
– Edge detection
– Emboss

11
Kernel: 3x3 Example
Input Kernel Output

3 2 1 -1 0 1

1 2 3 -2 0 2

1 1 1 -1 0 1

12
Kernel: 3x3 Example
Input Kernel Output

3 2 1 -1 0 1

1 2 3 -2 0 2 2

1 1 1 -1 0 1

= 3⋅ + 2 ⋅ 0+ 1⋅
−1 1
+ 1⋅ + 2 ⋅ 0+ 3⋅
−2 2
+ −3
= 1 ⋅ + 1+
− 21 +
⋅ 06 +
− 11+
⋅ 1
−12
= 1
13
Kernel: 3x3 Example
Output

-1 0 1
3 2 1
-2 0 2
1 2 3
-1 0 1
1 1 1

14
Kernels as Feature Detectors
Can think of kernels as a ”local feature
detectors”

Vertical Line Horizontal Line

Detector Detector Corner Detector

-1 1 -1 -1 -1 -1 -1 -1 -1

-1 1 -1 1 1 1 -1 1 1

-1 1 -1 -1 -1 -1 -1 1 1

15
Convolutional Neural Nets
Primary Ideas behind Convolutional Neural Networks:

 Let the Neural Network learn which kernels are most useful
 Use same set of kernels across entire image (translation invariance)
 Reduces number of parameters and “variance” (from bias-variance point of
view)

16
Convolutions

17
Convolution Settings—Grid Size
Grid Size (Height and Width):
 The number of pixels a kernel “sees” at once
 Typically use odd numbers so that there is a “center” pixel
 Kernel does not need to be square

Height: 3, Width: 3 Height: 1, Width: 3

Height: 3, Width: 1

18
Convolution Settings—Padding
Padding
 Using Kernels directly, there will be an “edge effect”
 Pixels near the edge will not be used as “center pixels” since there are not
enough surrounding pixels
 Padding adds extra pixels around the frame
 So every pixel of the original image will be a center pixel as the kernel
moves across the image
 Added pixels are typically of value zero (zero-padding)

19
Without Padding

1 2 0 3 1 -1 1 2 -2

1 0 0 2 2 1 1 0

2 1 2 1 1 -1 -2 0

0 0 1 0 0 Kernel Output

1 2 1 1 1

Input

20
With Padding
0 0 0 0 0 0 0

0 1 2 0 3 1 0 -1 1 2 -1

0 1 0 0 2 2 0
1 1 0
0 2 1 2 1 1 0
-1 -2 0
0 0 0 1 0 0 0

0 1 2 1 1 1 0 Kernel
0 0 0 0 0 0 0 Output

Input

21
Convolution Settings
Stride
 The ”step size” as the kernel moves across the image
 Can be different for vertical and horizontal steps (but usually is the same
value)
 When stride is greater than 1, it scales down the output dimension

22
Stride 2 Example—No Padding

1 2 0 3 1 -1 1 2 -2 3

1 0 0 2 2 1 1 0 0

2 1 2 1 1 -1 -2 0 Output

0 0 1 0 0 Kernel

1 2 1 1 1

Input

23
Stride 2 Example—with Padding
0 0 0 0 0 0 0

0 1 2 0 3 1 0 -1 1 2 -1 2
0 1 0 0 2 2 0
1 1 0 3
0 2 1 2 1 1 0
-1 -2 0
0 0 0 1 0 0 0

0 1 2 1 1 1 0 Kernel Output
0 0 0 0 0 0 0

Input

24
Convolutional Settings—Depth
 In images, we often have multiple numbers associated with each pixel location.
 These numbers are referred to as “channels”
– RGB image—3 channels
– CMYK—4 channels(CMYK refers to the four ink plates
used in some color printing: cyan,
magenta, yellow, and key).

 The number of channels is referred to as the “depth”

 So the kernel itself will have a “depth” the same size as the number of input
channels
 Example: a 5x5 kernel on an RGB image
– There will be 5x5x3 = 75 weights

25
Convolutional Settings—Depth
 The output from the layer will also have a depth
 The networks typically train many different kernels
 Each kernel outputs a single number at each pixel location
 So if there are 10 kernels in a layer, the output of that layer will have depth
10.

26
Pooling
 Idea: Reduce the image size by mapping a patch of pixels to a single value.
 Shrinks the dimensions of the image.
 Does not have parameters, though there are different types of pooling
operations.

27
Pooling: Max-pool
 For each distinct patch, represent it by the
maximum
 2x2 maxpool shown below

2 1 0 -1

-3 8 2 5 8 5

1 -1 3 4 maxpool 1 4
0 1 1 -2

28
Pooling: Average-pool
 For each distinct patch, represent it by the
average
 2x2 avgpool shown below

2 1 0 -1

-3 8 2 5 2 1.5

1 -1 3 4 avgpool
.25 1.5

0 1 1 -2

29
CNN – Using Keras
We will use the Malaria Cell Image dataset. This dataset consists of 27,558 images of
microscopic blood samples. The dataset consists of 2 folders – folders-Parasitized and
Uninfected. Sample Images-
a) parasitized blood sample
b) Uninfected blood sample

We will discuss the building of CNN along with CNN working in following 6 steps –
Step1 – Import Required libraries
Step2 – Initializing CNN & add a convolutional layer
Step3 – Pooling operation

30
CNN – Using Keras
 Step4 – Add two convolutional layers
 Step5 – Flattening operation
 Step6 – Fully connected layer & output layer

 These 6 steps will explain the working of CNN, which is shown in the below image –

31
CNN – Using Keras
 ,,

32
CNN – Using Keras
1. Import Required libraries
 Python Code :
import tensorflow as tf
from tensorflow.keras.layers import Input, Lambda, Dense, Flatten,Conv2D
from tensorflow.keras.models import Model
from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.applications.resnet50 import preprocess_input
from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import ImageDataGenerator,load_img
from tensorflow.keras.models import Sequential
import numpy as np
from glob import glob
import matplotlib.pyplot as plt
from tensorflow.keras.layers import MaxPooling2D
33
CNN – Using Keras
2. Initializing CNN & add a convolutional layer
 Python Code :
model=Sequential()
model.add(Conv2D(filters=16,kernel_size=2,padding="same",activation="relu",input_
shape=(224,224,3)))

 We first need to initiate sequential class since there are various layers to build
CNN which all must be in sequence. Then we add the first convolutional layer
where we need to specify 5 arguments. So, let’s discuss each argument and its
purpose.

34
CNN – Using Keras
Filters:
 The primary purpose of convolution is to find features in the image using a feature
detector. Then put them into a feature map, which preserves distinct features of
images.

 Feature detector which is known as a filter also is initialized randomly and then after
a lot of iteration, filter matrix parameter selected which will be best for separating
images. For instance, animals’ eye, nose, etc. will be considered as a feature which
is used for classifying images using filter or feature detectors. Here we are using 16
features.

Kernel_size:
 Kernel_size refers to filter matrix size. Here we are using a 2*2 filter size.

35
CNN – Using Keras
 Padding:

36
CNN – Using Keras
 Let’s discuss what is problem with CNN and how the padding operation will solve the
problem.
 a. For a gray scale (n x n) image and (f x f) filter/kernel, the dimensions of the image
resulting from a convolution operation is (n – f + 1) x (n – f + 1).
 So for instances, a 5*7 image and 3*3 filter kernel size, the output result after
convolution operation would be a size of 3*5. Thus, the image shrinks every time after
the convolutional operation
 b. Pixels, located on corners are contributed very little compared to middle pixels.

 So, then to mitigate these problems, padding operation is done. Padding is a simple
process of adding layers with 0 or -1 to input images so to avoid above mentioned
problems.
 Here we are using Padding = Same arguments, which depicts that output images have
the same dimensions as input images.

37
CNN – Using Keras
Activation Function – Relu:

 Since images are non-linear, to bring

non-linearity, the relu activation function is
applied after the convolutional operation.

 Relu stands for Rectified linear activation

function. Relu function will output the input
directly if it is positive, otherwise, it will
output zero.

38
CNN – Using Keras
Input shape
 This argument shows image size – 224*224*3. Since the images in RGB format so,
the third dimension of the image is 3.
3. Pooling Operation
 Python Code :

model.add(MaxPooling2D(pool_size=2))

 We need to apply the pooling operation after initializing CNN. Pooling is an operation
of down sampling of the image. The pooling layer is used to reduce the dimensions of
the feature maps. Thus, the Pooling layer reduces the number of parameters to learn
and reduces computation in the neural network.

39
CNN – Using Keras

40
CNN – Using Keras
 Future operations are performed on summarized features created by the pooling
layer. instead of precisely positioned features generated by the convolution
layer. This leads the model more robust to variations in the orientation of the
feature in the image.

 There are mainly 3 types of pooling: –

 1. Max Pooling
 2. Average Pooling
 3. Global Pooling

41
CNN – Using Keras
4. Add two convolutional Layer
 In order to add two more convolutional layers, we need to repeat steps 2 &3 with
slight modification in the number of filters.

 Python Code :
model.add(Conv2D(filters=32,kernel_size=2,padding="same",activation ="relu"))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=64,kernel_size=2,padding="same",activation="relu"))
model.add(MaxPooling2D(pool_size=2))

 We modified the 2nd and 3rd convolutional layers with filter numbers 32 & 64
respectively.

42
CNN – Using Keras
5. Flattening Operation
 Python Code :
model.add(Flatten())

 Flattening operation is converting the dataset

into a 1-D array for input into the next layer
which is the fully connected layer.

 After finishing the 3 steps, now we have pooled feature map. We are now flattening
our output after two steps into a column. Because we need to insert this 1-D data
into an artificial neural network layer.

43
CNN – Using Keras
6. Fully Connected layer and output layer:
 The output of the flattening
operation work as input for
the neural network. The aim
of the artificial neural
network makes the
convolutional neural network
more advanced and capable
enough of classifying images.

44
CNN – Using Keras
 Here we are using a dense class from the Keras library from creating a fully
connected layer and output layer.

 Python Code :
model.add(Dense(500,activation="relu"))
model.add(Dense(2,activation="softmax"))

 The softMax activation function is used

for building the output layer. Let’s discuss
the softmax activation function.

45
CNN – Using Keras
 It is used as the last activation function of a neural network to bring the output of the
neural network to a probability distribution over predicting classes. The output of
Softmax is in probabilities of each possible outcome for predicting class. The
probabilities sum should be one for all possible predicting classes.

 Now, let’s discuss training and evaluation of the Convolutional neural network. We
will be discussing this section in 3 steps;-

 Step 1 – Compile CNN model

 Step 2 – Fit model on training set
 Step 3 – Evaluate Result

46
CNN – Using Keras
Step 1 – Compile CNN Model:
 Code line:
model.compile(loss=’categorical_crossentropy’,optimizer=’adam’,metrics=[‘accuracy’])
 Here we are using 3 arguments:-
1- Loss function
 We are using the categorical_crossentropy loss function that is used in the classification
task. This loss is a very good measure of how distinguishable two discrete probability
distributions are from each other.

47
CNN – Using Keras
2- Optimizer
 We are using adam Optimizer that is used to update neural network weights and
learning rate. Optimizers are used to solve optimization problems by minimizing the
function.

3- Metrics arguments
 Here, we are using Accuracy as a metrics to evaluate the performance of the
Convolutional neural network algorithm.

48
CNN – Using Keras
Step 2 – Fit Model on Training Set
 Code line:
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
Or
mypath = 'cell_images/'
training_set = glob(mypath +'train/' 'C33P1thinF_IMG_/' + '.jpg')
test_set = glob(mypath +'test/' 'C33P1thinF_IMG_/' + '.jpg')
model.fit_generator(training_set,validation_data=test_set,epochs=50,
steps_per_epoch=len(training_set), validation_steps=len(test_set) )

 We are fitting the CNN model on the training dataset with 50 iterations and each
iteration has different steps for training and evaluating steps based on the
length of the test and training set.

49
CNN – Using Keras
Step3:- Evaluate Result
 We compare the accuracy and loss function for both the training and test dataset.
 Code: Plotting loss graph
plt.plot(r.history['loss'], label='train loss') plt.plot(r.history['val_loss'], label='val
loss') plt.legend() plt.show()
plt.savefig('LossVal_loss')

 OUTPUT:

50
CNN – Using Keras
 Code: Plotting Accuracy graph
plt.plot(r.history['accuracy'], label='train acc') plt.plot(r.history['val_accuracy'], label='val
acc')
plt.legend() plt.show()
plt.savefig('AccVal_acc')

 Output:

51