Course Of Machine Learning

By

Professor Abdellatif El Afia

ENSIAS- University in Rabat
Course of M.L. by Pr. Abdellatif El Afia 1
 Example: Bank Credit

Develop a software (machine) able to take one of the following decisions

(Task):
- Credit approval : Classification Task
- Amount of credit allocation: Regression Task
- Probability of credit approval : Probability Distribution Task

To build this machine, we should have a customer historical data.

The objective is to let the machine learn (Best Model in the Task’s type
chosen) from the data.

 Machine learning.
Course of M.L. by Pr. Abdellatif El Afia 2
 Example: Digits Recognition
Develop a software (machine) able to recognize handwritten digits
Task: Multi-class Classification.
To build this machine, we should have a set of pictures historical data.

The objective is to let the machine learn(Best Model) from the data.
 Machine learning.
Course of M.L. by Pr. Abdellatif El Afia 3
 Overview of this Course

• Introduction to Machine Learning

• Machine Learning Theory
• Tasks Type: Classification’s Models – Regression’s Models
• Overfitting : How we improve the model’s learning

Course of M.L. by Pr. Abdellatif El Afia 4

Introduction to Machine Learning

Course of M.L. by Pr. Abdellatif El Afia 5

 Definition of Machine Learning

Scenario: Task

Historical Data Learning Model Optimal Expertise

• Machine Learning is a process of experience (Historical Data) to gain expertise

Scenarios: Supervised Learning - Unsupervised Learning - Reinforced Learning

Tasks: Classification - Regression - Probability Distribution Task

Course of M.L. by Pr. Abdellatif El Afia 6
 Using conditions of machine learning
The use of ML requires the fulfillment of three conditions:
1- Existence of a model to learn:
There is a correlation between input and output variables. We know that a
model exists even if we do not know it.
2- Mathematical modeling is impossible:
We can not solve the model mathematically (no analytical solution).
3- Existence of data: (sufficient condition)
There is data that represents the model.

Course of M.L. by Pr. Abdellatif El Afia 7

 Machine learning scenarios

There are different learning scenarios to adapt with different

situations and conditions.
The three main learning scenarios are:
 Supervised learning
 Unsupervised learning
 Reinforced learning

Course of M.L. by Pr. Abdellatif El Afia 8

 Machine learning scenarios

Supervised Learning
- The data form is: (Inputs, Correct Outputs)
- Learn from a dataset tagged by target variables.
- Classification, regression and ranking are tasks that belong to the scenario of supervised
learning.
Don’t
- Example: Credit Approval approve

approve

Course of M.L. by Pr. Abdellatif El Afia 9

 Machine learning scenarios

Unsupervised Learning
- The form of the data is: (Inputs)
- Learn from a non-tagged dataset by target variables.
- Partitioning and dimension reduction are tasks that belong to the scenario of unsupervised
learning.

Example: Customer clustering.

Course of M.L. by Pr. Abdellatif El Afia 10

 Machine learning scenarios

Reinforced Learning

- The data form is: (Input, Some Outputs, Reward for each output).

- Learn by interaction with the environment and by observing the result of

certain actions.

- It can be used for classification, regression tasks if the training data is

insufficient.

Example: Child learning. Course of M.L. by Pr. Abdellatif El Afia 11

 Types of data reception
 Active reception of data:
The learning algorithm selects the data.
 Passive reception of data:
The user provides the data to the learning algorithm. This form owns two types:
- Offline reception.
The data is presented to the algorithm as a batch (all at once).
- Online reception.
The data is presented to the algorithm incrementally (one by one).

Course of M.L. by Pr. Abdellatif El Afia 12

 Supervised Learning : Passive - offline
• 

Task

   
 
Data Predictive Model Optimal predictor

, : feature

: Model, : set of hypotheses , : empirical error function and is vector of parameters.

It concerns the use of the best features , to build the best model by minimizing in order to
solve the best tasks. : Generalization Error

Course of M.L. by Pr. Abdellatif El Afia 13

 Data features
• 
- The features : can take different forms:
 Quantitative: a real number.
 Qualitative: image, chain of letters, ...

- Labels can take two forms:

 Real values:
 Regression Task
 Ranking Task = logistic regression task
 Discrete values: or Classification Task
Course of M.L. by Pr. Abdellatif El Afia 14
 Supervised Learning Passive Offline Algorithm (SLPOA)
 Goal: Find the Optimal Predictor
, : Feature
: target function Historical Data
It consists on using the training sample to
find the best hypothesis that Minimizes the
• Approximation Error
• Empirical Error   Training sample   Testing sample
• Loss Function
Or Maximizes the Approximation Capacity
Error
Then using the testing sample to measure 𝑺   𝑳𝑺 ( 𝒉 𝑺 ) measure  𝑳𝑫 ,𝒇 ( 𝒉𝑺 )
the of :
• Generalization Error   Model
• Generalization Capacity   Optimal predictor
𝒉  𝒉
  𝑺
• F isTarget Function,
• D is a ditribution for measuring the
quality of ,   h

Course of M.L. by Pr. Abdellatif El Afia 15

• 
Classification:
(ahmed)= oui ou non, est un séparateur, un heperlan
• : le nombre de point mal classifiée: training (cours et TD) 18
• >> ? overfitting
• : Generalization Error;: testing (examen) 5
• ?
• Y: label set
regression
• (ahmed)= , est une fonction: linéaire, non linéaire
Logistic Regression: distribution
(ahmed)= probabilite=, = P(Credit Approval / ahmed)

Course of M.L. by Pr. Abdellatif El Afia 16

 Bank Credit: Credit Approval
•   Binary Classification
Task:
Aim: Given the history , find an optimal prediction model (separator) for future data.
Tool: Machine learning. ,
Inputs of the training algorithm:
- Labels set:
Features Type
- Training set: ,
Present employment Qualitative
- Testing set Duration in month  Numerical
- Feature set: Credit history  Qualitative
Purpose Qualitative
Age Numerical
Number of existing credits Numerical
Outputs of the training algorithm: at this bank

- Optimal hypothesis: = the best separator using Credit amount Numerical

- Generalized model using

Course of M.L. by Pr. Abdellatif El Afia 17
 Bank Credit: Credit Allocation
•   Regression
Task:
Aim: Given the history 𝑆, find an optimal prediction model (function) for future data.
Tool: Machine learning.

Inputs of the training algorithm:

Features Type
- Training set: ,
Present employment Qualitative
- Testing set Duration in month  Numerical
- Feature set: Credit history  Qualitative
- Labels set: Purpose Qualitative
Age in years Numerical
Number of existing credits Numerical
Outputs of the training algorithm: at this bank

- Optimal hypothesis: = the best function using Credit amount Numerical

- Generalized model using

Course of M.L. by Pr. Abdellatif El Afia 18
 Bank Credit: Probability of Credit Approval
•   Logistic Regression
Task:
Aim: Given the history 𝑆, find an optimal prediction model (probability of distribution ) because are random and
follow a binomial distribution) for future data.
Example: two customers with the same information but different credit approval decisions.
Tool: Machine learning.

Features Type
Inputs of the training algorithm:
Present employment Qualitative
- Training set: ,
Duration in month  Numerical
- Testing set
Credit history  Qualitative
- Feature set: Purpose Qualitative
- Labels set: Age in years Numerical
Number of existing credits Numerical
Outputs of the training algorithm: at this bank

- Optimal hypothesis: = the best probability distribution using Credit amount Numerical

- Generalized model using

Course of M.L. by Pr. Abdellatif El Afia 19
 Digits Recognition: Handwritten Digits Recognition
•   Multi-class Classification
Task:
Aim: Given the history 𝑆, find an optimal prediction model (separator) for future data.
Inputs of the training algorithm:
- Labels set:
- Training set: ,
- Testing set
- Feature set: number of pixels (18x18)

Outputs of the training algorithm:

- Optimal hypothesis: = the best separator using
- Generalized model using
Course of M.L. by Pr. Abdellatif El Afia 20
• 
•, ,
Cas: separation de 0, 1, 2

Course of M.L. by Pr. Abdellatif El Afia 21

 Models: Regression
Definition:
The objective of regression task is to find a function, in order to approximate real-
valued targets. Regression

Linear Regression Non-Linear Regression

 
 
(polynomial regression)

Course of M.L. by Pr. Abdellatif El Afia 22

 Models: Regression

Nonlinear models

SVR MLP (ANN)

Using the concept of Using the concept of the

the marge combination of many nonlinear
functions

Course of M.L. by Pr. Abdellatif El Afia 23

• On utilise ANN ou SVR lorsque nous ne pouvons pas avoir une
fonction linéaire ou non linéaire
Lorsque il n’y a pas une fonction donc on utilise l’un des :
• SVR: on utilise la marge entre les supports vecteur
• ANN: combinaison des fonctions

Course of M.L. by Pr. Abdellatif El Afia 24

• Classification:

• La nature de data soit

• Linearly separable data
• Nonlinearly separable data

Course of M.L. by Pr. Abdellatif El Afia 25

 Models: Classification

Linearly separable data

Without noise With noise

Perceptron Adaline Perceptron Soft SVM

Hard SVM

PLA Best separator Widrow-Hoff Pocket Best separator

Course of M.L. by Pr. Abdellatif El Afia 26

 Models: Classification
Nonlinearly separable data

Nonlinear Transformation Nonlinear models

Linear model ANN Twin-SVM

Course of M.L. by Pr. Abdellatif El Afia 27

Supervised Learning Passive Offline Algorithm (SLPOA)
  Labeling the inputs by unknown   Input generation by unknown
target function   distribution
 𝒚 𝟏 , 𝒚𝟐 … 𝒚𝑴
 
 𝒙𝟏 , 𝒙𝟐 … 𝒙 𝑴
Data preprocessing

  Training sample   Testing sample

𝑺   𝑳𝑺 ( 𝒉 𝑺 ) Error
measure  𝑳𝑫 ,𝒇 ( 𝒉𝑺 )
  Model
  Final hypothesis
𝒉 
 𝒉𝑺

  h

Course of M.L. by Pr. Abdellatif El Afia 28

 Training Process
• 
Tasks (classification or regression)

    Approximation Capability
Training Data  
Predictive Model

(regression case)

(classification case)

: Empirical error function ; , : Feature

: Model, : set of hypotheses , and is vector of parameters.

It concerns the use of the best features , to build the best model by minimizing in order to
solve the best tasks.
Course of M.L. by Pr. Abdellatif El Afia 29
 Testing process
• 
   
Testing data Optimal hypothesis  
Generalization Capability

Such that:

, : features

: test set, : optimale hypothesis. : Probability Distibution, : general error.

: General error.
(regression case)

(classification case)

Course of M.L. by Pr. Abdellatif El Afia 30

 Overfitting
• :  empirique error (approximation capability).
 : general error (generalization capability).
If :

We say that the algorithm has a poor generalization capacity.

To remedy this problem, the following techniques are used:

Regularization: imposing a constraint
Validation & Cross validation: selection of the best , or the best model
Course of M.L. by Pr. Abdellatif El Afia 31
Supervised Learning Passive Offline Algorithm (SLPOA)
  Labeling the inputs by unknown target   Input generation by unknown
function 𝒇  distribution
   𝒚 𝟏 , 𝒚 𝟐 … 𝒚 𝑴
    𝒙𝟏 , 𝒙𝟐 … 𝒙 𝑴
Data preprocessing
Feature selection

  Training sample   Testing sample

Validation &
∗ Error
Cross-Validation  𝑳𝑺 ( 𝒉 ) measure ∗
 𝑳𝑫 ,𝒇 ( 𝒉 )
  Model
  Final hypothesis
∗
𝒉   𝒉

  h
Regularization
Course of M.L. by Pr. Abdellatif El Afia 32
 Objective
•   is learning?
What
 PAC learning.
How can a machine learn?
 ERM.
Is data learnable? S
 Uniform convergence.
What is the amount of data needed for learning?
 Uniform convergence.
How learning might fail?
 No-free-lunch theorem.
How can we measure the complexity of a model?
 VC dimension and covering number.
Is the model a good learner?
 Tradeoff Bias/Variance.
How to deal with models having an infinite complexity measure?
 Non-uniform learning, SRM.
How can we improve the model’s learning?
 Regularization and cross-validation.

Course of M.L. by Pr. Abdellatif El Afia 33

 Outline
Part 1: Machine learning theory Part 2: Classification and regression
Discover the basic concepts of ML. Models
Study the relationship between and  Discover classification models.
Study the probability distribution of the data .  Implement classification models.
Study the labeling function  Discover regression models.
Study the hypothesis set .  Implement regression models.
Study the model .  Study nonlinear transformations.
Find the best approximation of .

Part 3 Regularization, Validation and Cross-Validation

 Fight against the overfitting.

Course of M.L. by Pr. Abdellatif El Afia 34

 Outline
Part 1 Machine Learning Theory
Learning framework
Uniform convergence
Learnability of infinite size hypotheses classes
Tradeoff Bias/Variance
Non-Uniform learning.
Part 2: Learning Models TP
Classification
Regression
Part 3: Overfitting course and TP
Kernel
Regularization, Validation et Cross-Validation
Feature Selection
Course of M.L. by Pr. Abdellatif El Afia 35
 References

• Abdellatif El Afia: Machine Learning from Theory to Algorithms

Youtube: https://www.youtube.com/watch?v=lS3FAlCiuTs&t=531s

• Yaser S. Abu-Mostafa, Malik Magdon-Ismail, Hsuan-Tien Lin. Learning from data. Sci-hub

Youtube: https://www.youtube.com/watch?v=mbyG85GZ0PI&list=PLnIDYuXHkit4LcWjDe0EwlE57WiGlBs08

• Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms.

Youtube: https://www.youtube.com/watch?v=b5NlRg8SjZg&list=PLFze15KrfxbH8SE4FgOHpMSY1h5HiRLMm

Livre:
https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf

Course of M.L. by Pr. Abdellatif El Afia 36