Madan Mohan Malaviya Univ.

of Technology, Gorakhpur

Introduction to Machine Learning

(BCS-41)

By
Sushil Kumar Saroj
Assistant Professor

Department of Computer Science & Engineering

Madan Mohan Malaviya University of Technology Gorakhpur
(UP State Govt. University)

Email: [email protected]
17-08-2020 Side 1
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Syllabus

Unit-I

FOUNDATIONS OF LEARNING- Components of Learning – Learning

Models – Geometric Models – Probabilistic Models – Logic Models –
Grouping and Grading – Learning Versus Design – Types of Learning –
Supervised – Unsupervised – Reinforcement – Theory of Learning –
Feasibility of Learning – Error and Noise – Training versus Testing – Theory
of Generalization – Generalization Bound – Approximation – Generalization
Tradeoff – Bias and Variance – Learning Curve

17-08-2020 Side 2
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

What is Learning?

“Learning denotes changes in a system that ... enable a system to do the same
task … more efficiently the next time.” - Herbert Simon

“Learning is constructing or modifying representations of what is being

experienced.” - Ryszard Michalski

“Learning is making useful changes in our minds.” - Marvin Minsky

“Machine learning refers to a system capable of the

autonomous acquisition and integration of knowledge.”

17-08-2020 Side 3
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

What is Learning?

17-08-2020 Side 4
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Why Machine Learning?

• No human experts
• industrial/manufacturing control
• mass spectrometer analysis, drug design, astronomic discovery
• Black-box human expertise
• face/handwriting/speech recognition
• driving a car, flying a plane
• Rapidly changing phenomena
• credit scoring, financial modeling
• diagnosis, fraud detection
• Need for customization/personalization
• personalized news reader
• movie/book recommendation

17-08-2020 Side 5
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Related Fields

data
mining control theory

statistics
decision theory

information theory machine

learning cognitive science

databases
psychological models
evolutionary
models neuroscience

• Machine learning is primarily concerned with the accuracy and

effectiveness of the computer system
17-08-2020 Side 6
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Components of Learning

• Collecting and preparing data

• Choosing and training a model
• Evaluating a model
• Hyperparameter tuning and Prediction

17-08-2020 Side 7
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

• Geometric Models
• Probabilistic Models
• Logical Models

17-08-2020 Side 8
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Geometric Models
In Geometric models, features could be described as points in two dimensions
(x- and y-axis) or a three-dimensional space (x, y, and z). Even when features
are not intrinsically geometric, they could be modelled in a geometric manner
(for example, temperature as a function of time can be modelled in two axes).
In geometric models, there are two ways we could impose similarity.

• We could use geometric concepts like lines or planes to segment (classify)

the instance space. These are called Linear models
• Alternatively, we can use the geometric notion of distance to represent
similarity. In this case, if two points are close together, they have similar
values for features and thus can be classed as similar. We call such models
as Distance-based models

17-08-2020 Side 9
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Probabilistic Models
Probabilistic models see features and target variables as random variables.
The process of modelling represents and manipulates the level of uncertainty
with respect to these variables. There are two types of probabilistic models:
Predictive and Generative.
• Predictive probability models use the idea of a conditional probability
distribution P (Y |X) from which Y can be predicted from X
• Generative models estimate the joint distribution P (Y, X). Once we
know the joint distribution for the generative models, we can derive any
conditional or marginal distribution involving the same variables

Probabilistic models use the idea of probability to classify new entities

Naïve Bayes is an example of a probabilistic classifier.

17-08-2020 Side 10
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Logical Models
Logical models use a logical expression to divide the instance space into
segments and hence construct grouping models. A logical expression is an
expression that returns a Boolean value, i.e., a True or False outcome. Once
the data is grouped using a logical expression, the data is divided into
homogeneous groupings for the problem we are trying to solve.
There are two types of logical models: Tree models and Rule models.
• Rule models consist of a collection of implications or IF-THEN rules. For
tree-based models, the ‘if-part’ defines a segment and the ‘then-part’
defines the behaviour of the model for this segment. Rule models follow
the same reasoning
• Tree models can be seen as a particular type of rule model where the if-
parts of the rules are organised in a tree structure. Both Tree models and
Rule models use the same approach to supervised learning

17-08-2020 Side 11
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Groping and Grading Models

17-08-2020 Side 12
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Groping Models

17-08-2020 Side 13
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Grading Models

17-08-2020 Side 14
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Models

Groping versus Grading Models

17-08-2020 Side 15
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Versus Design

• Machine learning is a powerful tool that drives everything from curated

content recommendations to optimized user interfaces
• Machine learning answers questions about user behavior
• Machine learning customizes interfaces to users needs
• Digital product designers need to get familiar with machine learning
• Many warn that designers who don’t start learning about ML will be left
behind. But I haven’t seen one that has explored what design and machine
learning have to offer each other
• Design and machine learning function like a flywheel: when connected,
each provides value to the other. Together, they open up new product
experiences and business value
• Design helps machine learning gather better data

17-08-2020 Side 16
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning Versus Design

• Machine learning is a hungry beast. To deliver the best results, learning

algorithms need vast amounts of detailed data, clean of any confounding
factors or built-in biases
• Designers can help create user experiences that eliminate noise in data,
leading to more accurate and efficient ML-powered applications
• Design helps set expectations and establish trust with users

17-08-2020 Side 17
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Types of Learning

• Supervised learning
• Unsupervised learning
• Reinforcement learning

17-08-2020 Side 18
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Types of Learning
Supervised learning
In this type of learning, the data set on which the machine is trained consists
of labelled data or simply said, consists both the input parameters as well as
the required output.
Supervised Machine Learning Algorithms can be broadly divided into two
types of algorithms; Classification and Regression.
• Classification: Supervised learning problem that involves predicting a
class label
• Regression: Supervised learning problem that involves predicting a
numerical label

Examples: Linear Regression, Logistic Regression, KNN classification,

Support Vector Machine (SVM), Decision Trees, Random Forest, Naive
Bayes’ theorem.
17-08-2020 Side 19
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Types of Learning
Unsupervised learning
Unlike supervised learning algorithms, where we deal with labelled data for
training, the training data will be unlabelled for Unsupervised Machine
Learning Algorithms. The clustering of data into a specific group will be done
on the basis of the similarities between the variables.
• Clustering: Unsupervised learning problem that involves finding groups
in data
• Density estimation: Unsupervised learning problem that involves
summarizing the distribution of data
• Visualization: Unsupervised learning problem that involves creating plots
of data
• Projection: Unsupervised learning problem that involves creating lower-
dimensional representations of data
Examples: K-means clustering, neural networks
17-08-2020 Side 20
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Types of Learning
Reinforcement learning
Reinforcement Learning is a type of Machine Learning in which the machine
is required to determine the ideal behaviour within a specific context, in order
to maximize its rewards. It works on the rewards and punishment principle
which means that for any decision which a machine takes, it will be either be
rewarded or punished. Thus, it will understand whether or not the decision
was correct. This is how the machine will learn to take the correct decisions to
maximize the reward in the long run.

Examples: Q-learning, temporal-difference learning, and deep reinforcement

learning.

17-08-2020 Side 21
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Error and Noise

Error

Error measures are a tool in ML that quantify the question “how wrong was
our estimation”. It is a function that compares the output of a learned
hypothesis with the output of the real target function. What this means in
practice is that we compare the prediction of our model with the real value in
data. An error measure is expressed as E(h, f) (a hypothesis h ∈ H, and f is the
target function). E is almost always pointwise. It is defined by the difference
at two points, therefore, we use the pointwise definition of the error measure
e() to compute this error in the different points: e(h(x), f(x)).

Examples:
Squared error: e(h(x), f(x)) = (h(x)- f(x))²
Binary error: e(h(x), f(x)) = ⟦h(x) ≠ f(x)⟧ (the number of wrong
classifications)
17-08-2020 Side 22
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Error and Noise

Noise

It refers to the irrelevant information or randomness in a dataset.

We can express noisy target as follows:
Noisy target= deterministic target + noise = 𝔼[y|x] + ε where ε = (y - f(x)) is
the difference between the outcome and the predicted value.
𝔼[y|x] is the expected value of y knowing x, y is our prediction using the
target function h(x) and f(x) is the real value of the data point.
We introduced P(y|x) into our learning scheme to account for the fact that
there will always be noise in the relationship between x and y while P(x)
represents the random variable x and is necessary for us to use Hoeffding‘s
inequality.

17-08-2020 Side 23
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Training versus Testing

• In a dataset, a training set is implemented to build up a model, while a test

(or validation) set is to validate the model built. Data points in the training
set are excluded from the test (validation) set
• In Machine Learning, we basically try to create a model to predict the test
data
• Usually, a dataset is divided into a training set, a validation set (some
people use ‘test set’ instead) in each iteration, or divided into a training
set, a validation set and a test set in each iteration

17-08-2020 Side 24
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Training versus Testing

Sets:

• Training Set: Here, you have the complete training dataset. You can
extract features and train to fit a model and so on
• Validation Set: This is crucial to choose the right parameters for your
estimator. We can divide the training set into a train set and validation set.
Based on the validation test results, the model can be trained(for instance,
changing parameters, classifiers)
• Testing Set: Here, once the model is obtained, you can predict using the
model obtained on the training set

17-08-2020 Side 25
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Training versus Testing

17-08-2020 Side 26
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Theory of Generalization

• In machine learning, generalization usually refers to the ability of an

algorithm to be effective across a range of inputs and applications
• Our key working assumption is that data is generated by an underlying,
unknown distribution D. Rather than accessing the distribution directly,
statistical learning assumes that we are given a training sample S, where
every element of S is i.i.d and generated according to D. A learning
algorithm chooses a function (hypothesis h) from a function space
(hypothesis class) H where H = {f(x, α)} where α is the parameter vector
• We can then define the generalization error of a hypothesis h as the
difference between the expectation of the error on a sample x picked
from the distribution D and the empirical loss

17-08-2020 Side 27
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

• The objective of learning is to achieve good generalization to new cases,

otherwise just use a look-up table
• Generalization can be defined as a mathematical interpolation or
regression over a set of training points:

17-08-2020 Side 28
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

17-08-2020 Side 29
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

17-08-2020 Side 30
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

17-08-2020 Side 31
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

• The objective of learning is to achieve good generalization to new cases,

otherwise just use a look-up table
• Generalization can be defined as a mathematical interpolation or
regression over a set of training points:

17-08-2020 Side 32
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

Over-Training

• Is the equivalent of over-fitting a set of data points to a curve

which is too complex
• Occam’s Razor (1300s): “plurality should not be assumed
without necessity”
• The simplest model which explains the majority of the data is
usually the best

17-08-2020 Side 33
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

Preventing Over-training

• Use a separate test or tuning set of examples

• Monitor error on the test set as network trains
• Stop network training just prior to over-fit error occurring-
early stopping or tuning
• Number of effective weights is reduced
• Most new systems have automated early stopping methods

17-08-2020 Side 34
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization

How can we control number of effective weights?

• Manually or automatically select optimum number of hidden

nodes and connections
• Prevent over-fitting = over-training
• Add a weight-cost term to the bp error equation

17-08-2020 Side 35
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization Bound

17-08-2020 Side 36
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization Bound

17-08-2020 Side 37
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Generalization Bound

• There are two types of bound

• VC generalization bound
• Distributed function based bound

17-08-2020 Side 38
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Approximation- Generalization Tradeoff

17-08-2020 Side 39
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Approximation- Generalization Tradeoff

17-08-2020 Side 40
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 41
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 42
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 43
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 44
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 45
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 46
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 47
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Overfitting

17-08-2020 Side 48
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Bias and Variance

Bias
Bias is the difference between the Predicted Value and the Expected Value.
Mathematically, let the input variables be X and a target variable Y. We map
the relationship between the two using a function f. Therefore,
Y = f(X) + e
Here ‘e’ is the error that is normally distributed. The aim of our model f'(x)
is to predict values as close to f(x) as possible. Here, the Bias of the model
is:
Bias[f'(X)] = E[f'(X) – f(X)]
As I explained above, when the model makes the generalizations i.e. when
there is a high bias error, it results in a very simplistic model that does not
consider the variations very well. Since it does not learn the training data
very well, it is called Underfitting.

17-08-2020 Side 49
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Bias and Variance

Variance
Contrary to bias, the Variance is when the model takes into account the
fluctuations in the data i.e. the noise as well. So, what happens when our
model has a high variance?
The model will still consider the variance as something to learn from. That
is, the model learns too much from the training data, so much so, that when
confronted with new (testing) data, it is unable to predict accurately based
on it.
Mathematically, the variance error in the model is:
Variance[f(x))=E[X^2]−E[X]^2
Since in the case of high variance, the model learns too much from the
training data, it is called overfitting.

17-08-2020 Side 50
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Bias and Variance

17-08-2020 Side 51
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning curves
• Learning curves are plots that show changes in learning performance
over time in terms of experience
• Learning curves of model performance on the train and validation
datasets can be used to diagnose an underfit, overfit, or well-fit model
• Learning curves of model performance can be used to diagnose whether
the train or validation datasets are not relatively representative of the
problem domain
• Generally, a learning curve is a plot that shows time or experience on the
x-axis and learning or improvement on the y-axis

Learning curves are deemed effective tools for monitoring the performance
of workers exposed to a new task. LCs provide a mathematical
representation of the learning process that takes place as task repetition
occurs.
17-08-2020 Side 52
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning curves
• Train Learning Curve: Learning curve calculated from the training
dataset that gives an idea of how well the model is learning
• Validation Learning Curve: Learning curve calculated from a hold-out
validation dataset that gives an idea of how well the model is
generalizing
• Optimization Learning Curves: Learning curves calculated on the
metric by which the parameters of the model are being optimized, e.g.
loss
• Performance Learning Curves: Learning curves calculated on the
metric by which the model will be evaluated and selected, e.g. accuracy
• There are three common dynamics that you are likely to observe in
learning curves; they are:
• Underfit
• Overfit
• Good Fit

17-08-2020 Side 53
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning curves

• Underfitting refers to a model that cannot learn the training dataset.

• A plot of learning curves shows underfitting if:
• The training loss remains flat regardless of training
• The training loss continues to decrease until the end of training

• Overfitting refers to a model that has learned the training dataset too
well, including the statistical noise or random fluctuations in the training
dataset.
• A plot of learning curves shows overfitting if:
• The plot of training loss continues to decrease with experience
• The plot of validation loss decreases to a point and begins increasing again

17-08-2020 Side 54
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

Learning curves

• A good fit is the goal of the learning algorithm and exists between an
overfit and underfit model
• A good fit is identified by a training and validation loss that decreases to
a point of stability with a minimal gap between the two final loss values
• A plot of learning curves shows a good fit if:
• The plot of training loss decreases to a point of stability
• The plot of validation loss decreases to a point of stability and has a small gap with
the training loss

17-08-2020 Side 55
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

References
Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky. “Spectrally-
[1]

normalized margin bounds for neural networks.” Advances in Neural

Information Processing Systems. 2017.
[2]Shawe-Taylor, J. and Rivasplata, O., “Statistical Learning Theory: A
Hitchhiker’s Guide”, 2012, [link]

17-08-2020 Side 56
 Madan Mohan Malaviya Univ. of Technology, Gorakhpur

17-08-2020 Side 57