Deep Learning —

Introduction, Keras Intro,

Tensor Manipulation
Machine Learning
Sher Muhammad Daudpota
Sukkur IBA University
AI, Machine Learning and Deep
Learning
Deep Learning
• Deep Learning is subfield of Machine Learning
• Layered representation of learning
• Hierarchical representation of learning
• “Deep” in Deep Learning is number of layers or depth of hierarchy
• Layered representation is learned through neural network
• Remember people will say neural networks are modeled the way
human brain works, but believe me that is not the truth…
Deep Learning
Deep Learning
Deep Learning
• Three important things in deep learning,
• Weights
• Loss Function
• Optimizer
Deep Learning: Weights
Deep Learning: Loss Function
Deep Learning: Optimizer
Deep Learning Wonders So far..
Deep Learning Wonders So far..
What makes deep learning different
or so successful?
• Feature Engineering
• DL automates it
Why Deep Learning Now?
• Hardware
• GPU, NVIDIA’s CUDA and TITAN X
• Google’s TPU (Tensor Pro…..)
• Datasets and benchmarks
• Algorithmic advancements
• Better activation functions
• Weight initialization schemes
• Optimization schemes
Let’s run MNIST hand written digits
recognition..

Hello World of Deep Learning 

Data Representation for Tensors
• Tensor is a container for data, mostly numerical data
• Matrices are 2D tensors
• Scalars
• Tensor with only one number
• 0D tensor
Data Representation for Tensors
• Vectors
• 1D tensors
Data Representation for Tensors
• Matrices (2D tensors)
• Array of vectors
Data Representation for Tensors
• 3D tensor
Data Representation for Tensors
• Tensor Key Attributes
• Number of axes: that is returned through ndim
• Shape: define dim along each axis and returned through shape
• Data type: It can be float32, uint8, int32, float64 and char
Data Representation for Tensors
• Real-world tensors examples
Data Representation for Tensors
• Vector Data
Data Representation for Tensors
• Timeseries data (3D tensor)
Data Representation for Tensors
Data Representation for Tensors
• Image Data (4D tensors)
• A batch of 128 grayscale images of size
256 x 256 can be stored in a tensor of
shape (128,256,256,1)
• A batch of 128 color images could be
stored in a tensor of shape
(128,256,256,3)
Data Representation for Tensors
• Video Data (5D tensors)
Data Representation for Tensors
• Example

• ndim

• Shape and dtype

Data Representation for Tensors
Data Representation for Tensors
• Tensor slicing is selecting specific elements in an tensor,
Data Representation for Tensors
• Data batches
MNIST Complete Example
Classifying movie reviews (IMDB
Practical)
Classifying newswire: A multiclass
classification (practical)
Playing with BBC dataset
Assignment
• Available on eLearning
• Tweet Emotion Classification
• BBC Sports News Classification (try doing it in class, if time permits)
Fundamentals of Machine Learning
• Four Branches of Machine Learning
• Supervised Learning
• Sequence Generation
• Syntax Tree Prediction
• Object Detection
• Image Segmentation
• Unsupervised Learning
• Dimensionality Reduction
• Clustering
• Self Supervised Learning
• Reinforcement Learning
Evaluating Machine Learning Models
• Training set
• Validation Set
• Test Set
Simple Hold Out Validation
Simple Hold Out Validation
K-Fold Validation
Keep in mind….
• Data Representation in Cross Validation
• Arrow of time
• Redundancy in your data
Data Preprocessing, feature
engineering and feature learning
• Preprocessing
• Vectorization
• Value normalization
• Handling missing values
• Feature Engineering
Overfitting, Underfitting,
Optimization, Generalization
• Regularization
• Process to fight overfitting
• Too much capacity vs memory starvation
Overfitting, Underfitting,
Optimization, Generalization
Overfitting, Underfitting,
Optimization, Generalization
Overfitting, Underfitting,
Optimization, Generalization
Overfitting, Underfitting,
Optimization, Generalization
Regularization
Regularization
Regularization
Regularization
• Adding dropout
Regularization
• Adding dropout
Regularization
To fight with overfitting, increase
generalization and discourage
optimization..
• Get more training data
• Reduce network capacity
• Add weight regularization
• Add dropout
Universal Workflow of Machine
Learning Problems
• Define problem and dataset
• Data inputs outputs
• Binary classification, regression, multiclass?
• Non-stationary data consideration
Universal Workflow of Machine
Learning Problems
• Choosing a measure of success
• Accuracy for class-balance problems
• Precision and recall for class-imbalance problems
Universal Workflow of Machine
Learning Problems
• Deciding on an evaluation protocol
• Hold out cross validation: loads of data available
• K-Fold cross validation: too few samples for hold out cross validation
• Iterated K-Fold: for high accurate model too little data, say in 100s
Universal Workflow of Machine
Learning Problems
• Preparing your data
• Covert data in tensors
• Range data to [-1,1] or [0,1]
• Normalize all features, bring to same range
• Feature engineering if required
Universal Workflow of Machine
Learning Problems
• Developing a model that does better than baseline
• Random classification accuracy 0.5
Universal Workflow of Machine
Learning Problems
Universal Workflow of Machine
Learning Problems
Universal Workflow of Machine
Learning Problems
• Scaling up: developing a model that overfits
• Cross the border that exists between optimization and generalization
• Border between overfitting and underfitting
Universal Workflow of Machine
Learning Problems
• Regularizing your model and tuning hyper parameters
• This step deserve your highest attention