Algorithms: K Nearest Neighbors

1
 Simple Analogy..
• Tell me about your friends(who your
neighbors are) and I will tell you who you are.

2
 KNN – Different names
• K-Nearest Neighbors
• Memory-Based Reasoning
• Example-Based Reasoning
• Instance-Based Learning
• Lazy Learning

3
Instance-based Learning

Its very similar to a

Desktop!!

4
 What is instance based Learning?
In machine learning, instance-based
learning (sometimes called memory-based learning)
is a family of learning algorithms that, instead of
performing explicit generalization, compare new
problem instances with instances seen in training, which
have been stored in memory.

Because computation is postponed until a new instance

is observed, these algorithms are sometimes referred to
as "lazy."
 What is instance based Learning?(Cont..)
It is called instance-based because it constructs
hypotheses directly from the training instances
themselves. This means that the hypothesis complexity
can grow with the data: in the worst case, a hypothesis
is a list of n training items and the computational
complexity of classifying a single new instance is O(n).

Advantage :
Ability to adapt its model to previously unseen data.
Examples are:
• KNN
• RBF networks
Why Is The KNN Called A “Lazy Learner” Or A
“Lazy Algorithm”?
KNN is called a lazy learner because when we supply training data
to this algorithm, the algorithm does not train itself at all.

KNN does not learn any discriminative function from the training
data. But it memorizes the entire training dataset instead.

There is no training time in KNN.

But, this skipping of training time comes with a cost.

Each time a new data point comes in and we want to make a

prediction, the KNN algorithm will search for the nearest
neighbors in the entire training set.

Hence the prediction step becomes more time-consuming and

computationally expensive.
 What is KNN?
• A powerful classification algorithm used in pattern
recognition.

• K nearest neighbors stores all available cases and

classifies new cases based on a similarity measure(e.g
distance function)

• One of the top data mining algorithms used today.

• A non-parametric lazy learning algorithm (An Instance-

based Learning method).

8
 KNN: Classification Approach

• An object (a new instance) is classified by a

majority votes for its neighbor classes.
• The object is assigned to the most common class
amongst its K nearest neighbors.(measured by a
distant function )

9
10
 Distance Measure

Compute
Distance
Test
Record

Training
Records Choose k of the
“nearest” records

11
Different Distance Measure

12
Distance measure for Continuous Variables
Euclidean Distance

Manhattan distance

Minkowski distance

13
 Distance Between Neighbors
• Calculate the distance between new example
(E) and all examples in the training set.

• Euclidean distance between two examples.

– X = [x1,x2,x3,..,xn]
– Y = [y1,y2,y3,...,yn]

– The Euclidean distance between X and Y is defined

as: n
D( X ,Y ) =  (x − y )
i i
2

i=1 11
 K-Nearest Neighbor Algorithm
• All the instances correspond to points in an n-dimensional
feature space.

• Each instance is represented with a set of numerical

attributes.

• Each of the training data consists of a set of vectors and a

class label associated with each vector.

• Classification is done by comparing feature vectors of

different K nearest points.

• Select the K-nearest examples to E in the training set.

• Assign E to the most common class among its K-nearest

neighbors.
15
KNN: Pseudocode
 3-KNN: Example
Customer Age Income No. Class Distance from John
credit
cards
George 35 35K 3 No sqrt [(35-37)2+(35-50)2 +(3-
2)2]=15.16

Rachel 22 50K 2 Yes sqrt [(22-37)2+(50-50)2 +(2-

2)2]=15

Steve 63 200K 1 No sqrt [(63-37)2+(200-50)2 +(1-

2)2]=152.23

Tom 59 170K 1 No sqrt [(59-37)2+(170-50)2 +(1-

2)2]=122

Anne 25 40K 4 Yes sqrt [(25-37)2+(40-50)2 +(4-

2)2]=15.74

John 37 50K 2 ? YES

17
 How to choose K?

• If K is too small it is sensitive to noise points.

• Larger K works well. But too large K may include majority

points from other classes.

• Rule of thumb is K < sqrt(n), n is number of examples.

18
19
 X X X

(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor

K-nearest neighbors of a record x are data points

that have the k smallest distance to x

20
 KNN Feature Weighting

• Scale each feature by its importance for

classification

• Can use our prior knowledge about which features

are more important
• Can learn the weights wk using cross‐validation

21
 Feature Normalization
• Distance between neighbors could be dominated
by some attributes with relatively large numbers.
e.g., income of customers in our previous example.

• Arises when two features are in different scales.

• Important to normalize those features.

– Mapping values to numbers between 0 – 1.

22
 Nominal/Categorical Data
• Distance works naturally with numerical attributes.

• Binary value categorical data attributes can be regarded as 1

or 0.

23
 KNN Classification
$250,000

$200,000

$150,000

Loan$ Non-Default
$100,000 Default

$50,000

$0
0 10 20 30 40 50 60 70
Age

24
KNN Classification – Distance
Age Loan Default Distance
25 $40,000 N 102000
35 $60,000 N 82000
45 $80,000 N 62000
20 $20,000 N 122000
35 $120,000 N 22000
52 $18,000 N 124000
23 $95,000 Y 47000
40 $62,000 Y 80000
60 $100,000 Y 42000
48 $220,000 Y 78000
33 $150,000 Y 8000

48 $142,000 ?

D = (x − x ) 2 +( y − y ) 2
1 2 1 2

25
KNN Classification – Standardized Distance
Age Loan Default Distance
0.125 0.11 N 0.7652
0.375 0.21 N 0.5200
0.625 0.31 N 0.3160
0 0.01 N 0.9245
0.375 0.50 N 0.3428
0.8 0.00 N 0.6220
0.075 0.38 Y 0.6669
0.5 0.22 Y 0.4437
1 0.41 Y 0.3650
0.7 1.00 Y 0.3861
0.325 0.65 Y 0.3771

0.7 0.61 ?
X − Min
Xs =
Max − Min
26
 Strengths of KNN
• Very simple and intuitive.
• Can be applied to the data from any distribution.
• Good classification if the number of samples is large enough.

Weaknesses of KNN
• Takes more time to classify a new example.
• need to calculate and compare distance from new
example to all other examples.
• Choosing k may be tricky.
• Need large number of samples for accuracy.
27