Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning:
Basic Concepts
by Samuel I. G. Situmeang

Modified slides provided by: Ansaf Salleb-Aouissi

Artificial Intelligence, Columbia University, 2018
Machine Learning: Basic Concepts Artificial Intelligence

Objectives

• Machine Learning Concepts

• Training-Testing
• Overfitting-Underfitting and Regularization

2
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning Concepts

3
Machine Learning: Basic Concepts Artificial Intelligence

Terminology

Machine Learning, Data Science, Data Mining, Data Analysis, Statistical

Learning, Knowledge Discovery in Databases, Pattern Discovery.

4
Machine Learning: Basic Concepts Artificial Intelligence

A Few Quotes

• "A breakthrough in machine learning would be worth ten Microsofts"

(Bill Gates, Chairman, Microsoft)
• "Machine learning is the next Internet"
(Tony Tether, former Director, DARPA - Defense Advanced Research Projects Agency)
• Machine learning is the hot new thing"
(John Hennessy, former President, Stanford)
• "Web rankings today are mostly a matter of machine learning"
(Prabhakar Raghavan, former Dir. Research, Yahoo)
• "Machine learning is going to result in a real revolution"
(Greg Papadopoulos, CTO, Sun Microsystems)
• "Machine learning is today’s discontinuity"
(Jerry Yang, former CEO, Yahoo)

5
Machine Learning: Basic Concepts Artificial Intelligence

Data everywhere!

Source: https://www.domo.com/
6
Machine Learning: Basic Concepts Artificial Intelligence

Data types

Data comes in different sizes and also flavors (types):

 Texts
 Numbers
 Clickstreams
 Graphs
 Tables
 Images
 Transactions
 Videos
 Some or all of the above!

7
Machine Learning: Basic Concepts Artificial Intelligence

Smile, we are 'DATAFIED'!

• Wherever we go, we are "datafied".

• Smartphones are tracking our locations.
• We leave a data trail in our web browsing.
• Interaction in social networks.
• Privacy is an important issue in Data Science.

8
Machine Learning: Basic Concepts Artificial Intelligence

The Data Science process

9
Machine Learning: Basic Concepts Artificial Intelligence

Applications of ML

• We all use it on a daily basis. Examples:

10
Machine Learning: Basic Concepts Artificial Intelligence

Applications of ML

• Spam filtering
• Credit card fraud detection
• Digit recognition on checks, zip codes
• Detecting faces in images
• MRI image analysis
• Recommendation system
• Search engines
• Handwriting recognition
• Scene classification
• etc...

11
Machine Learning: Basic Concepts Artificial Intelligence

Now is a great time to study ML

Research progress

Image classification Audio synthesis Games

Products

Voice recognition Translation Self-driving cars

12
Machine Learning: Basic Concepts Artificial Intelligence

Progress in ML is driven by…

• More compute
• More data
• Better algorithms → Need more people who understand the
algorithms!

13
Machine Learning: Basic Concepts Artificial Intelligence

Interdisciplinary field

14
Machine Learning: Basic Concepts Artificial Intelligence

ML versus Statistics

Statistics: Machine Learning:

• Hypothesis testing • Decision trees
• Experimental design • Rule induction
• Analysis of variance (ANOVA) • Neural Networks
• Linear regression • Support Vector Machines (SVMs)
• Logistic regression • Clustering method
• Generalized Linear Models (GLM) • Association rules
• Principal Component Analysis (PCA) • Feature selection
• Visualization
• Graphical models
• Genetic algorithm

http://statweb.stanford.edu/~jhf/ftp/dm-stat.pdf

15
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning definition

Alan Turing proposed the concept of a learning machine in 1950 (in the
same paper that proposed the Turing test).

Idea: Divide the problem into two parts:

1. A machine that simulates a child’s brain (analogous to a blank notebook:
should function by simple mechanisms and have lots of blank sheets).
2. A way of teaching the child machine (should be simple since we know how
to teach a human child).

Teacher rewards good behaviour and penalizes bad behaviour.

16
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning definition

"An important feature of a learning machine is that its teacher will

often be very largely ignorant of quite what is going on inside."
Alan Turing

• While we don’t know how our brain converts input to output, we

know what the output should be for every input.
• We can use this knowledge to teach the machine.

17
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning definition

“How do we create computer programs that improve with

experience?”
Tom Mitchell
http://videolectures.net/mlas06_mitchell_itm/

18
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning definition

"How do we create computer programs that improve with

experience?"
Tom Mitchell
http://videolectures.net/mlas06_mitchell_itm/

"A computer program is said to learn from experience 𝐸 with respect

to some class of tasks 𝑇 and performance measure 𝑃, if its
performance at tasks in 𝑇, as measured by 𝑃, improves with
experience 𝐸."
Tom Mitchell. Machine Learning 1997.

19
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning definition

• A branch of artificial intelligence, concerned with the design and

development of algorithms that allow computers to evolve behaviors
based on empirical data.

• As intelligence requires knowledge, it is necessary for computers to

acquire knowledge.

Data
• Traditional programming Computer Output
Program

• Machine learning Data

Computer Program
Output

20
Machine Learning: Basic Concepts Artificial Intelligence

Types of machine learning Algorithms

There some variations of how to define the types of Machine Learning

Algorithms but commonly they can be divided into categories
according to their purpose and the main categories are the following:
• Supervised learning (predictive model, "labeled" data).
• Classification
• Numeric prediction/forecasting/regression
• Unsupervised learning (descriptive model, "unlabeled" data).
• Clustering
• Pattern Discovery
• Semi-supervised learning (mixture of "labeled" and "unlabeled" data).
• Reinforcement learning. Using this algorithm, the machine is trained to make
specific decisions. It works this way: the machine is exposed to an
environment where it trains itself continually using trial and error. This
machine learns from past experience and tries to capture the best possible
knowledge to make accurate business decisions.

21
Machine Learning: Basic Concepts Artificial Intelligence

Common algorithms for each categories

• Supervised learning
• Classification. e.g. Logistic Regression, Decision Tree, KNN, Random Forest,
SVM, & Naive Bayes
• Numeric prediction/forecasting/regression. e.g. Linear Regression, KNN,
Gradient Boosting & AdaBoost
• Unsupervised learning
• Clustering. e.g. K-Means
• Pattern Discovery. e.g. Apriori, FP-Growth, & Eclat
• Semi-supervised learning
• Reinforcement learning.
• e.g. Q-Learning, Temporal Difference (TD), & Deep Adversarial Networks

22
Machine Learning: Basic Concepts Artificial Intelligence

A brief of ML types with sample use cases

Source: https://en.proft.me/

23
Machine Learning: Basic Concepts Artificial Intelligence

Supervised vs. Unsupervised

Given: Training data: 𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑 and 𝑦𝑖 is the label.

example 𝑥1 → 𝑥11 𝑥12 … 𝑥1𝑑 𝑦1 ← label
… … … … … …
example 𝑥𝑖 → 𝑥𝑖1 𝑥𝑖2 … 𝑥𝑖𝑑 𝑦𝑖 ← label
… … … … … …
example 𝑥𝑛 → 𝑥𝑛1 𝑥𝑛2 … 𝑥𝑛𝑑 𝑦𝑛 ← label

fruit length width weight label

fruit 1 165 38 172 Banana
fruit 2 218 39 230 Banana
fruit 3 76 80 145 Orange
fruit 4 145 35 150 Banana
fruit 5 90 88 160 Orange
…
fruit n … … … …

24
Machine Learning: Basic Concepts Artificial Intelligence

Supervised vs. Unsupervised

fruit length width weight label

fruit 1 165 38 172 Banana
fruit 2 218 39 230 Banana
fruit 3 76 80 145 Orange
fruit 4 145 35 150 Banana
fruit 5 90 88 160 Orange
…
fruit n … … … …

Unsupervised learning:
Learning a model from unlabeled data.
Supervised learning:
Learning a model from labeled data.

25
Machine Learning: Basic Concepts Artificial Intelligence

Unsupervised Learning

Training data: "examples" 𝑥.

𝑥1 , … , 𝑥𝑛 , 𝑥𝑖 ∈ 𝑋 ⊂ ℝ𝑛

• Clustering/segmentation:

𝑓: ℝ𝑑 → 𝐶1 , … , 𝐶𝑘 set of cluster

Example: Find clusters in the population, fruits, species.

26
Machine Learning: Basic Concepts Artificial Intelligence

Unsupervised Learning

Clustering/segmentation:

27
Machine Learning: Basic Concepts Artificial Intelligence

Unsupervised Learning

Clustering/segmentation:

28
Machine Learning: Basic Concepts Artificial Intelligence

Unsupervised Learning

Clustering/segmentation:

Methods: K-means, gaussian mixtures, hierarchical clustering, spectral

clustering etc.

29
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Training data: "examples" 𝑥 with "labels" 𝑦.

𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑

• Classification: 𝑦 is discrete. To simplify, 𝑦 ∈ −1. +1

𝑓: ℝ𝑑 → −1, +1 𝑓 is called a binary classifier

Example: Approve credit yes/no, spam/ham, banana/orange.

30
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Classification:

31
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Classification:

32
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Classification:

Methods: Support Vector Machines, neural networks, decision trees,

K-nearest neighbors, naive Bayes, etc.

33
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Classification:

34
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Non linear classification

35
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Training data: "examples" 𝑥 with "labels" 𝑦.

𝑥1 , 𝑦1 , … , 𝑥𝑛 , 𝑦𝑛 Τ𝑥𝑖 ∈ ℝ𝑑

• Regression: 𝑦 is a real value, 𝑦 ∈ ℝ

𝑓: ℝ𝑑 → ℝ 𝑓 is called a regressor

Example: amount of credit, weight of fruit.

36
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Regression:

Example: Income in function of age, weight of the fruit in function

of its length.

37
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Regression:

38
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Regression:

39
Machine Learning: Basic Concepts Artificial Intelligence

Supervised Learning

Regression:

40
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

41
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

Training set

ML Algorithm

Model (f)

42
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

Training set

ML Algorithm

Income,
gender,
age, Credit amount $
family Model (f) Credit yes/np
status,
zipcode

43
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

Training set

ML Algorithm

Income,
gender,
age, Credit amount $
family Model (f) Credit yes/np
status,
zipcode

Question: How can we be confident about 𝒇?

44
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

• We calculate 𝐸 𝑡𝑟𝑎𝑖𝑛 the in-sample error (training error or empirical

error/risk). prediction
𝑛
label
𝑡𝑟𝑎𝑖𝑛
𝐸 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1 true label

45
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

• We calculate 𝐸 𝑡𝑟𝑎𝑖𝑛 the in-sample error (training error or empirical

error/risk).
𝑛

𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1
• Examples of loss functions:
• Classification error:
1 𝑠𝑖𝑔𝑛 𝑦𝑖 ≠ 𝑠𝑖𝑔𝑛 𝑓 𝑥𝑖
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 =ቊ
0 otherwise

46
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

• We calculate 𝐸 𝑡𝑟𝑎𝑖𝑛 the in-sample error (training error or empirical

error/risk).
𝑛

𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1
• Examples of loss functions:
• Classification error:
1 𝑠𝑖𝑔𝑛 𝑦𝑖 ≠ 𝑠𝑖𝑔𝑛 𝑓 𝑥𝑖
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 =ቊ
0 otherwise
• Least square loss:
2
𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 = 𝑦𝑖 − 𝑓 𝑥𝑖

47
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

• We calculate 𝐸 𝑡𝑟𝑎𝑖𝑛 the in-sample error (training error or empirical

error/risk).
𝑛

𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 = ෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖
𝑖=1
• We aim to have 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 small, i.e., minimize 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓

• We hope that 𝐸 𝑡𝑟𝑎𝑖𝑛 𝑓 , the out-sample error (test/true error), will

be small too.

48
Machine Learning: Basic Concepts Artificial Intelligence

Overfitting-Underfitting and
Regularization

49
Machine Learning: Basic Concepts Artificial Intelligence

Overfitting/underfitting

• Overfitting: keadaan ketika model memiliki kinerja baik hanya untuk

training data/seen examples tetapi tidak memiliki kinerja baik untuk
unseen examples.
• Terjadi ketika model terlalu fleksibel (memiliki kemampuan yang terlalu tinggi
untuk mengestimasi banyak fungsi) atau terlalu mencocokkan diri terhadap
training data.
• Underfitting: keadaan ketika model memiliki kinerja buruk baik untuk
training data dan unseen examples.
• Terjadi akibat model yang telalu tidak fleksibel (memiliki kemampuan yang
rendah untuk mengestimasi variasi fungsi.

50
Machine Learning: Basic Concepts Artificial Intelligence

Structural Risk Minimization

High Bias Low Bias
Prediction Error LowVariance High Variance

Test error
Training error

UnderfiAng Good models OverfiAng

Low Complexity of the model High

51
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

52
Machine Learning: Basic Concepts Artificial Intelligence

Training and Testing

53
Machine Learning: Basic Concepts Artificial Intelligence

Avoid overfitting

• In general, use simple models!

• Reduce the number of features manually or do feature selection.
• Do a model selection (ML course).
• Use regularization (keep the features but reduce their importance by setting
small parameter values) (ML course).
• Do a cross-validation to estimate the test error.

54
Machine Learning: Basic Concepts Artificial Intelligence

Regularization: Intuition

We want to minimize:

Classification term + 𝐶 × Regularization term

෍ 𝑙𝑜𝑠𝑠 𝑦𝑖 , 𝑓 𝑥𝑖 +𝐶×𝑅 𝑓
𝑖=1

55
Machine Learning: Basic Concepts Artificial Intelligence

Regularization: Intuition

Hint: Avoid high-degree polynomials.

56
Machine Learning: Basic Concepts Artificial Intelligence

Train, Validation and Test

Example: Split the data randomly into 60% for training, 20% for
validation and 20% for testing.

Source: https://towardsdatascience.com/

57
Machine Learning: Basic Concepts Artificial Intelligence

Train, Validation and Test

Training set is a set of examples used for learning a model (e.g., a

classification model).

Source: https://towardsdatascience.com/

58
Machine Learning: Basic Concepts Artificial Intelligence

Train, Validation and Test

Training set is a set of examples used for learning a model (e.g., a

classification model).
Validation set is a set of examples that cannot be used for learning the
model but can help tune model parameters (e.g., selecting K in K-NN).
Validation helps control overfitting.

Source: https://towardsdatascience.com/

59
Machine Learning: Basic Concepts Artificial Intelligence

Train, Validation and Test

Training set is a set of examples used for learning a model (e.g., a

classification model).
Validation set is a set of examples that cannot be used for learning the
model but can help tune model parameters (e.g., selecting K in K-NN).
Validation helps control overfitting.
Test set is used to assess the performance of the final model and
provide an estimation of the test error.

Source: https://towardsdatascience.com/

60
Machine Learning: Basic Concepts Artificial Intelligence

Train, Validation and Test

Training set is a set of examples used for learning a model (e.g., a

classification model).
Validation set is a set of examples that cannot be used for learning the
model but can help tune model parameters (e.g., selecting K in K-NN).
Validation helps control overfitting.
Test set is used to assess the performance of the final model and
provide an estimation of the test error.

Note: Never use the test set in any

way to further tune the parameters
or revise the model.

Source: https://towardsdatascience.com/

61
Machine Learning: Basic Concepts Artificial Intelligence

Classifier Evaluation Method: Holdout &

Cross-Validation
• Holdout method
• Given data is randomly partitioned into two independent sets
• Training set (e.g., 2/3) for model construction
• Test set (e.g., 1/3) for accuracy estimation
• Repeated random sub-sampling validation: a variation of holdout
• Repeat holdout 𝑘 times, accuracy = avg. of the accuracies
obtained
• Cross-validation (𝑘-fold, where 𝑘 = 10 is most popular)
• Randomly partition the data into 𝑘 mutually exclusive subsets, each
approximately equal size
• At 𝑖-th iteration, use 𝐷𝑖 as test set and others as training set
• Leave-one-out: 𝑘 folds where 𝑘 = # of tuples, for small sized data
• *Stratified cross-validation*: folds are stratified so that class
distribution, in each fold is approximately the same as that in the
initial data

62
Machine Learning: Basic Concepts Artificial Intelligence

K-fold Cross Validation

A method for estimating test error using training data.

Algorithm:
Given a learning algorithm 𝒜 and a dataset 𝒟
Step 1: Randomly partition D into k equal-size subsets D1; : : : ; Dk
Step 2:
For 𝑗 = 1 to 𝑘
Train 𝒜 on all 𝐷𝑖 , 𝑖 ∈ 1, … , 𝑘 and 𝑖 ≠ 𝑗, and get 𝑓𝑗
Apply 𝑓𝑗 to 𝐷𝑗 and compute 𝐸𝐷𝑗
Step 3: Average error over all folds.
𝑘

෍ 𝐸 𝐷𝑗
𝑗=1

63
Machine Learning: Basic Concepts Artificial Intelligence

Confusion matrix

64
Machine Learning: Basic Concepts Artificial Intelligence

Classification Model Evaluation Metrics

𝐅𝟏 − 𝐒𝐜𝐨𝐫𝐞 2𝑃𝑅 Harmonic mean of precision and recall

𝑃+𝑅

65
Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Binary Classification

• Accuracy:
• How many of the samples are classified correctly? C1

• A = 9/10 = 0.9

C2

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 66

Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Binary Classification

System 2 System 1
• A = 7/10 = 0.7 System 1
System 2 C1 C1
• A = 7/10 = 0.7
• When classes are highly unbalanced
• Precision/recall/F1 for the rare class
• e.g. Spam classification (detection)

C2 C2

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 67

 Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Binary Classification

System 1 System 2 System 2 System 1

Precision 1/3 = 0.33 0/1 = 0 C1 C1

Recall 1/2 = 0.5 0/2 = 0

F1 0.4 0

C2 C2

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 68

 Machine Learning: Basic Concepts Artificial Intelligence

Error Analysis
𝑪𝟏
• Confusion Matrix
• How classes get confused?
Predicted
𝐶1 𝐶2 𝐶3 𝑪𝟐

3 0 1
Actual 0 3 1
1 0 1 𝑪𝟑

• Useful:
• Find classes that get confused with others
• Develop better features to solve the problem

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 69

Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Multi-class
𝑪𝟏
𝐶1 𝐶2 𝐶3

P 0.75 1 0.333
R 0.75 0.75 0.5
F1 0.75 0.86 0.4 𝑪𝟐

• Accuracy = (3+3+1)/10 = 0.7

• Good measure when 𝑪𝟑
• Classes are nearly balanced
• Preferred:
• Precision/recall/F1 for each class

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 70

Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Multi-class

σ𝑁
𝑖=1 TP𝑖
• Micro-average of Precision= σ𝑁 𝑁
𝑖=1 TP𝑖 +σ𝑖=1 FP𝑖

σ𝑁
𝑖=1 TP𝑖
• Micro-average of Recall= σ𝑁 𝑁
𝑖=1 TP𝑖 +σ𝑖=1 FN𝑖

σ𝑁
𝑖=1 P𝑖
• Macro-average of Precision=
𝑁

σ𝑁
𝑖=1 R𝑖
• Macro-average of Recall=
𝑁

71
Machine Learning: Basic Concepts Artificial Intelligence

Evaluation: Multi-class
𝑪𝟏
• Majority class baseline
• Accuracy = 0.8
• Macro-F1 = 0.296
𝑪𝟐

• Macro-F1:
• Should be used in binary classification when
two classes are important
• e.g.: males/females while distribution is 80/20% 𝑪𝟑

Thursday, Apr. 12, 2018 SGS/ ISS4011-Sistem Temu Balik Informasi 72

Machine Learning: Basic Concepts Artificial Intelligence

Example of Confusion Matrix:

Use the same confusion matrix, calculate the measure just introduced.
Actual Class\Predicted class cancer = yes cancer = no Total Recognition(%)
cancer = yes 90 210 300 30.00 (sensitivity)
cancer = no 140 9560 9700 98.56 (specificity)
Total 230 9770 10000 96.50 (accuracy)

• Sensitivity = TP/P = 90/300 = 30%

• Specificity = TN/N = 9560/9700 = 98.56%
• Accuracy = (TP + TN)/All = (90+9560)/10000 = 96.50%
• Error rate = (FP + FN)/All = (140 + 210)/10000 = 3.50%
• Precision = TP/(TP + FP) = 90/(90 + 140) = 90/230 = 39.13%
• Recall = TP/ (TP + FN) = 90/(90 + 210) = 90/300 = 30.00%

73
Machine Learning: Basic Concepts Artificial Intelligence

Regression Model Evaluation Metrics

• Mean Squared Error (MSE)

• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)
• R Squared (R²)
• Adjusted R Squared (R²)
• Mean Square Percentage Error (MSPE)
• Mean Absolute Percentage Error (MAPE)
• Root Mean Squared Logarithmic Error (RMSLE)

74
Machine Learning: Basic Concepts Artificial Intelligence

Mean Squared Error (MSE)

• It is perhaps the most simple and common metric for regression

evaluation, but also probably the least useful. It is defined by the
equation.

• where 𝑦𝑖 is the actual expected output and 𝑦ො𝑖 is the model’s prediction.
• The higher this value, the worse the model is. It is never negative,
since we’re squaring the individual prediction-wise errors before
summing them, but would be zero for a perfect model.

75
Machine Learning: Basic Concepts Artificial Intelligence

Root Mean Squared Error (RMSE)

• RMSE is the square root of MSE. The square root is introduced to

make scale of the errors to be the same as the scale of targets.

𝑁
1 2
RMSE = ෍ 𝑦𝑖 − 𝑦ො𝑖 = MSE
𝑁
𝑖=1

76
Machine Learning: Basic Concepts Artificial Intelligence

Terminology review

Review the concepts and terminology:

Instance, example, feature, label, supervised learning, unsupervised

learning, classification, regression, clustering, prediction, training set,
validation set, test set, K-fold cross validation, classification error, loss
function, overfitting, underfitting, regularization.

77
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning Books

1. Tom Mitchell, Machine Learning.

2. Abu-Mostafa, Yaser S. and Magdon-Ismail, Malik and Lin, Hsuan-
Tien, Learning From Data, AMLBook.
3. The elements of statistical learning. Data mining, Inference, and
Prediction T. Hastie, R. Tibshirani, J. Friedman.
4. Christopher Bishop. Pattern Recognition and Machine Learning.
5. Richard O. Duda, Peter E. Hart, David G. Stork. Pattern
Classification. Wiley

78
Machine Learning: Basic Concepts Artificial Intelligence

Machine Learning Resources

• Major journals/conferences: ICML, NIPS, UAI, ECML/PKDD, JMLR,

MLJ, etc.
• Machine learning video lectures:
http://videolectures.net/Top/Computer_Science/Machine_Learning/
• Machine Learning (Theory):
http://hunch.net/
• LinkedIn ML groups: \Big Data" Scientist, etc.
• Women in Machine Learning:
https://groups.google.com/forum/#!forum/women-in-machine-learning
• KDD nuggets
http://www.kdnuggets.com/

79
Machine Learning: Basic Concepts Artificial Intelligence

References

• T. Mitchell, Machine Learning, 1997.

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning: Data Mining, Inference, and Prediction (2nd Edition), 2009.
• S. J. Russell and P. Borvig, Artificial Intelligence: A Modern Approach
(3rd Edition), Prentice Hall International, 2010.

80