UNIT :II and III

Compiled by : Rashmi Bohra

 Data Mining:
Concepts and Techniques

Compiled by :Rashmi Bohra 2

 Efficient Computation of Data
Cubes
• Preliminary cube computation tricks (Agarwal et al.’96)
• Computing full/iceberg cubes: 3 methodologies
– Top-Down: Multi-Way array aggregation (Zhao, Deshpande & Naughton,
SIGMOD’97)
– Bottom-Up:
• Bottom-up computation: BUC (Beyer & Ramarkrishnan, SIGMOD’99)
• H-cubing technique (Han, Pei, Dong & Wang: SIGMOD’01)
– Integrating Top-Down and Bottom-Up:
• Star-cubing algorithm (Xin, Han, Li & Wah: VLDB’03)
• High-dimensional OLAP: A Minimal Cubing Approach (Li, et al.
VLDB’04)
• Computing alternative kinds of cubes:
– Partial cube, closed cube, approximate cube, etc.
Compiled by :Rashmi Bohra 3
 Preliminary Tricks (Agarwal et al.
VLDB’96)
• Sorting, hashing, and grouping operations are applied to the
dimension attributes in order to reorder and cluster related tuples
• Aggregates may be computed from previously computed
aggregates, rather than from the base fact table
– Smallest-child: computing a cuboid from the smallest, previously computed
cuboid
– Cache-results: caching results of a cuboid from which other cuboids are
computed to reduce disk I/Os
– Amortize-scans: computing as many as possible cuboids at the same time to
amortize disk reads
– Share-sorts: sharing sorting costs cross multiple cuboids when sort-based
method is used
– Share-partitions: sharing the partitioning cost across multiple cuboids when
hash-based algorithms are used
Compiled by :Rashmi Bohra 4
 Multi-Way Array Aggregation
• Array-based “bottom-up” algorithm
• Using multi-dimensional chunks a ll

• No direct tuple comparisons

• Simultaneous aggregation on multiple A B C

dimensions
• Intermediate aggregate values are re- A B A C B C
used for computing ancestor cuboids
• Cannot do Apriori pruning: No iceberg A B C
optimization

Compiled by :Rashmi Bohra 5

 Multi-way Array Aggregation for
Cube Computation (MOLAP)
• Partition arrays into chunks (a small subcube which fits in memory).
• Compressed sparse array addressing: (chunk_id, offset)
• Compute aggregates in “multiway” by visiting cube cells in the order which minimizes the # of
times to visit each cell, and reduces memory access and storage cost.

C c3 61
c2 45
62 63 64
46 47 48
c1 29 30 31 32 What is the best
c0
b3 B13 14 15 16 60 traversing order
44
9
28 56 to do multi-way
b2
B 40
24 52 aggregation?
b1 5 36
20
b0 1 2 3 4
a0 a1 a2 a3Compiled by :Rashmi Bohra 6
A
Multi-way Array Aggregation
for Cube Computation

C c3 61
c2 45
62 63 64
46 47 48
c1 29 30 31 32
c0
B13 14 15 16 60
b3 44
B 28 56
b2 9
40
24 52
b1 5
36
20
b0 1 2 3 4
a0 a1 a2 a3
A

Compiled by :Rashmi Bohra 7

 Multi-way Array
Aggregation for Cube
Computation

C c3 61
c2 45
62 63 64
46 47 48
c1 29 30 31 32
c0
B13 14 15 16 60
b3 44
B 28 56
b2 9
40
24 52
b1 5
36
20
b0 1 2 3 4
a0 a1 a2 a3
A

Compiled by :Rashmi Bohra 8

 Multi-Way Array Aggregation
for Cube Computation
(Cont.)
• Method: the planes should be sorted and computed
according to their size in ascending order
– Idea: keep the smallest plane in the main memory, fetch and
compute only one chunk at a time for the largest plane
• Limitation of the method: computing well only for a
small number of dimensions
– If there are a large number of dimensions, “top-down”
computation and iceberg cube computation methods can be
explored

Compiled by :Rashmi Bohra 9

 Bottom-Up Computation (BUC)
a ll

• BUC (Beyer & Ramakrishnan,

SIGMOD’99)
A B C D

• Bottom-up cube computation A B A C A D B C B D C D

(Note: top-down in our view!) A B C A B D A C D B C D

• Divides dimensions into partitions

and facilitates iceberg pruning A B C D

1 a ll
– If a partition does not satisfy
min_sup, its descendants can be 2 A 10 B 14 C 16 D
pruned
– If minsup = 1 ⇒ compute full CUBE! 3 A B 7 A C 9 A D 11 BC 13 BD 15 C D

• No simultaneous aggregation
4 A BC 6 A BD 8 A C D 12 BC D

Compiled by :Rashmi Bohra 10

5 A BC D
 BUC: Partitioning
• Usually, entire data set
can’t fit in main memory
• Sort distinct values, partition into blocks that fit
• Continue processing
• Optimizations
– Partitioning
• External Sorting, Hashing, Counting Sort
– Ordering dimensions to encourage pruning
• Cardinality, Skew, Correlation
– Collapsing duplicates
• Can’t do holistic aggregates anymore!
Compiled by :Rashmi Bohra 11
 H-Cubing: Using H-Tree
Structure a ll

• Bottom-up computation
A B C D

• Exploring an H-tree structure

A B A C A D B C B D C D
• If the current computation of an H-
tree cannot pass min_sup, do not
A B C A B D A C D B C D
proceed further (pruning)
• No simultaneous aggregation A B C D

Compiled by :Rashmi Bohra 12

 H-tree: A Prefix Hyper-tree
Attr. Val. Quant-Info Side-link
Edu Sum:2285 … root

Hhd …
Header Bus … edu hhd bus
table … …
Jan …
Feb … Jan Mar Jan Feb
… …
Tor …
Van … Tor Van Tor Mon
Month CityMon
Cust_grp Prod… Cost Price
… …
Jan Tor Edu Printer 500 485
Quant-Info Q.I. Q.I. Q.I.
Jan Tor Hhd TV 800 1200
Jan Tor Edu Camera 1160 1280 Sum: 1765
Cnt: 2
Feb Mon Bus Laptop 1500 2500 by :Rashmi Bohra
Compiled 13
Mar Van Edu HD 540 520 bins
 H-Cubing: Computing Cells Involving
Dimension City
Attr. Val. Q.I. Side-link
From (*, *, Tor) to (*, Jan, Tor)
Header Edu … root
Table Hhd …
Bus …
HTor … … Edu. Hhd. Bus.
Jan …
Feb …
… …
Attr. Val. Quant-Info Side-link Jan. Mar. Jan. Feb.
Edu Sum:2285 …
Hhd …
Bus …
… … Tor. Van. Tor. Mon.
Jan …
Feb … Quant-Info Q.I. Q.I. Q.I.
… …
Tor …
Van … Sum: 1765
Mon … Cnt:Bohra
2
Compiled by :Rashmi 14
… …
Computing Cells Involving Month But
No City
1. Roll up quant-info root

2. Compute cells
Edu. Hhd. Bus.
involving month but
noVal.city
Attr. Quant-Info Side-link
Edu. Sum:2285 … Jan. Mar. Jan. Feb.
Hhd. …
Bus. …
Q.I. Q.I. Q.I. Q.I.
… …
Jan. …
Feb. …
Mar. …
… …
Tor. … Tor. Van. Tor. Mont.
Van. … Top-k OK mark: if Q.I. in a child
Mont. … passes top-k avg threshold, so does
… …
Compiledits parents.
by :Rashmi Bohra No binning is needed!
15
 Computing Cells Involving Only
Cust_grp
root

Check header table edu hhd bus

directly
Attr. Val. Quant-Info Side-link Jan Mar Jan Feb
Edu Sum:2285 …

Hhd … Q.I. Q.I. Q.I. Q.I.

Bus …
… …
Jan …
Feb …
Mar … Tor Van Tor Mon
… …
Tor …
Van …
Mon … Compiled by :Rashmi Bohra 16
… …
 Star-Cubing: An Integrating
•
Method
Integrate the top-down and bottom-up methods
• Explore shared dimensions
– E.g., dimension A is the shared dimension of ACD and AD
– ABD/AB means cuboid ABD has shared dimensions AB
• Allows for shared computations
– e.g., cuboid AB is computed simultaneously as ABD
• Aggregate in a top-down manner but with the bottom-up sub-
C / C D
layer underneath which will allow Apriori pruning
• Shared dimensions grow in bottom-up
A C / fashion
A AC D / A B C / B BC D / B C D

A B C / AA BB CD / A AB C D / A B C D

Compiled by :Rashmi Bohra A B C D / a l l 17

 Iceberg Pruning in Shared
Dimensions
• Anti-monotonic property of shared dimensions
– If the measure is anti-monotonic, and if the aggregate value on a shared
dimension does not satisfy the iceberg condition, then all the cells extended
from this shared dimension cannot satisfy the condition either

• Intuition: if we can compute the shared dimensions before the actual

cuboid, we can use them to do Apriori pruning
• Problem: how to prune while still aggregate simultaneously on
multiple dimensions?

Compiled by :Rashmi Bohra 18

 Cell Trees
• Use a tree structure similar to
H-tree to represent cuboids
• Collapses common prefixes to
save memory
• Keep count at node
• Traverse the tree to retrieve a
particular tuple

Compiled by :Rashmi Bohra 19

 Star Attributes and Star Nodes
• Intuition: If a single-dimensional
aggregate on an attribute value p
does not satisfy the iceberg A B C D Count
condition, it is useless to distinguish a1 b1 c1 d1 1
them during the iceberg computation a1 b1 c4 d3 1
– E.g., b2, b3, b4, c1, c2, c4, d1, d2, d3 a1 b2 c2 d2 1
a2 b3 c3 d4 1
• Solution: Replace such attributes by
a2 b4 c3 d4 1
a *. Such attributes are star
attributes, and the corresponding
nodes in the cell tree are star nodes

Compiled by :Rashmi Bohra 20

 Example: Star Reduction
• Suppose minsup = 2
• Perform one-dimensional
A B C D Count
aggregation. Replace attribute
a1 b1 * * 1
values whose count < 2 with *. And
a1 b1 * * 1
collapse all *’s together
a1 * * * 1
• Resulting table has all such attributes a2 * c3 d4 1
replaced with the star-attribute a2 * c3 d4 1

• With regards to the iceberg A B C D Count

computation, this new table is a a1 b1 * * 2
loseless compression of the original a1 * * * 1
table a2 * c3 d4 2
Compiled by :Rashmi Bohra 21
 Star Tree
• Given the new compressed table, it is
possible to construct the corresponding
cell tree—called star tree

• Keep a star table at the side for easy

lookup of star attributes

• The star tree is a loseless compression

of the original cell tree

Compiled by :Rashmi Bohra 22

 Star-Cubing Algorithm—DFS on
Lattice Tree
a ll
BC D : 51

A /A B /B C /C D /D
b*: 33 b1: 26

ro o t: 5
c*: 14 c3: 211 c* : 27
A B /A B A C /A C A D /A B C /B C B D /B C D
a1: 3 a2: 2
d*: 15 d4 : 2 12 d*: 28

A B C /A B C A B D /A B A C D /A BC D
b*: 1 b1: 2 b*: 2

c*: 1 c*: 2 c3: 2

A BC D

d*: 1 d*: 2 d4: 2

Compiled by :Rashmi Bohra 23

 Multi-Way B C D A C D / A A B D / AA B B C / A B C

Aggregation
A B C D

Compiled by :Rashmi Bohra 24

Star-Cubing Algorithm—DFS on
Star-Tree

Compiled by :Rashmi Bohra 25

 Multi-Way Star-Tree
Aggregation
• Start depth-first search at the root of the base star tree
• At each new node in the DFS, create corresponding star tree that are
descendents of the current tree according to the integrated traversal
ordering
– E.g., in the base tree, when DFS reaches a1, the ACD/A tree is created
– When DFS reaches b*, the ABD/AD tree is created

• The counts in the base tree are carried over to the new trees

Compiled by :Rashmi Bohra 26

 Multi-Way Aggregation (2)
• When DFS reaches a leaf node (e.g., d*), start backtracking
• On every backtracking branch, the count in the corresponding
trees are output, the tree is destroyed, and the node in the base
tree is destroyed
• Example
– When traversing from d* back to c*, the a1b*c*/a1b*c* tree is
output and destroyed
– When traversing from c* back to b*, the a1b*D/a1b* tree is output
and destroyed
– When at b*, jump to b1 and repeat similar process

Compiled by :Rashmi Bohra 27

 The Curse of Dimensionality

• None of the previous cubing method can handle

high dimensionality!
• A database of 600k tuples. Each dimension has
cardinality of 100 and zipf of 2.

Compiled by :Rashmi Bohra 28

 Motivation of High-D OLAP
• Challenge to current cubing methods:
– The “curse of dimensionality’’ problem
– Iceberg cube and compressed cubes: only delay the
inevitable explosion
– Full materialization: still significant overhead in
accessing results on disk
• High-D OLAP is needed in applications
– Science and engineering analysis
– Bio-data analysis: thousands of genes
– Statistical surveys: hundreds of variables

Compiled by :Rashmi Bohra 29

 Fast High-D OLAP with Minimal
Cubing
• Observation: OLAP occurs only on a small subset of
dimensions at a time
• Semi-Online Computational Model
1. Partition the set of dimensions into shell fragments
2. Compute data cubes for each shell fragment while retaining
inverted indices or value-list indices
3. Given the pre-computed fragment cubes, dynamically
compute cube cells of the high-dimensional data cube online

Compiled by :Rashmi Bohra 30

 Properties of Proposed Method

• Partitions the data vertically

• Reduces high-dimensional cube into a set of lower

dimensional cubes

• Online re-construction of original high-dimensional space

• Lossless reduction

• Offers tradeoffs between the amount of pre-processing and

the speed of online computation

Compiled by :Rashmi Bohra 31

Example Computation

tid A B C D E
1 a1 b1 c1 d1 e1
2 a1 b2 c1 d2 e1
3 a1 b2 c1 d1 e2
4 a2 b1 c1 d1 e2
5 a2 b1 c1 d1 e3

Compiled by :Rashmi Bohra 32

 Online Query Computation
a,a,K ,a
n:M
• A query has the general form 1 2

• Each ai has 3 possible values

1. Instantiated value

2. Aggregate * function

3. Inquire ? function
3??*1
:c
o
un
t
 For example, returns a 2-D
data cube.
Compiled by :Rashmi Bohra 33
 Online Query Computation (2)

• Given the fragment cubes, process a query as

follows
1. Divide the query into fragment, same as the shell
2. Fetch the corresponding TID list for each fragment
from the fragment cube
3. Intersect the TID lists from each fragment to
construct instantiated base table
4. Compute the data cube using the base table with any
cubing algorithm
Compiled by :Rashmi Bohra 34
Online Query Computation (3)

A B C D E F GH I J K L MN …

Instantiated Online
Base Table Cube
Compiled by :Rashmi Bohra 35
Experiment: Size vs. Dimensionality
(50 and 100 cardinality)

• (50-C): 106 tuples, 0 skew, 50 cardinality, fragment size 3.

• (100-C): 106 tuples, 2 skew, 100 cardinality,
Compiled by :Rashmifragment
Bohra size 2. 36
 Experiments on Real World Data

• UCI Forest CoverType data set

– 54 dimensions, 581K tuples
– Shell fragments of size 2 took 33 seconds and 325MB to
compute
– 3-D subquery with 1 instantiate D: 85ms~1.4 sec.
• Longitudinal Study of Vocational Rehab. Data
– 24 dimensions, 8818 tuples
– Shell fragments of size 3 took 0.9 seconds and 60MB to
compute
– 5-D query with 0 instantiated D: 227ms~2.6 sec.
Compiled by :Rashmi Bohra 37
 Discovery-Driven Exploration of
Data Cubes

• Hypothesis-driven
– exploration by user, huge search space
• Discovery-driven (Sarawagi, et al.’98)
– Effective navigation of large OLAP data cubes
– pre-compute measures indicating exceptions, guide user in the
data analysis, at all levels of aggregation
– Exception: significantly different from the value anticipated,
based on a statistical model
– Visual cues such as background color are used to reflect the
degree of exception of each cell
Compiled by :Rashmi Bohra 38
 What is Concept Description?
• Descriptive vs. predictive data mining
– Descriptive mining: describes concepts or task-relevant
data sets in concise, summarative, informative,
discriminative forms
– Predictive mining: Based on data and analysis, constructs
models for the database, and predicts the trend and
properties of unknown data
• Concept description:
– Characterization: provides a concise and succinct
summarization of the given collection of data
– Comparison: provides descriptions comparing two or
more collections of data
Compiled by :Rashmi Bohra 39
 Data Generalization and
Summarization-based
Characterization
• Data generalization
– A process which abstracts a large set of task-relevant
data in a database from a low conceptual levels to
higher ones.
1
2
3
4
Conceptual levels
5

– Approaches:
• Data cube approach(OLAP approach)
• Attribute-oriented induction approach
Compiled by :Rashmi Bohra 40
 Concept Description vs.
OLAP
• Similarity:
– Data generalization
– Presentation of data summarization at multiple levels of abstraction.
– Interactive drilling, pivoting, slicing and dicing.
• Differences:
– Can handle complex data types of the attributes and their aggregations
– Automated desired level allocation.
– Dimension relevance analysis and ranking when there are many
relevant dimensions.
– Sophisticated typing on dimensions and measures.
– Analytical characterization: data dispersion analysis
Compiled by :Rashmi Bohra 41
 Attribute-Oriented
Induction
• Proposed in 1989 (KDD ‘89 workshop)
• Not confined to categorical data nor particular measures
• How it is done?
– Collect the task-relevant data (initial relation) using a relational
database query
– Perform generalization by attribute removal or attribute
generalization
– Apply aggregation by merging identical, generalized tuples and
accumulating their respective counts
– Interactive presentation with users

Compiled by :Rashmi Bohra 42

Basic Principles of Attribute-Oriented
Induction
• Data focusing: task-relevant data, including dimensions, and the result
is the initial relation
• Attribute-removal: remove attribute A if there is a large set of distinct
values for A but (1) there is no generalization operator on A, or (2) A’s
higher level concepts are expressed in terms of other attributes
• Attribute-generalization: If there is a large set of distinct values for A,
and there exists a set of generalization operators on A, then select an
operator and generalize A
• Attribute-threshold control: typical 2-8, specified/default
• Generalized relation threshold control: control the final relation/rule size

Compiled by :Rashmi Bohra 43

 Attribute-Oriented Induction: Basic
Algorithm
• InitialRel: Query processing of task-relevant data, deriving
the initial relation.
• PreGen: Based on the analysis of the number of distinct
values in each attribute, determine generalization plan for
each attribute: removal? or how high to generalize?
• PrimeGen: Based on the PreGen plan, perform
generalization to the right level to derive a “prime
generalized relation”, accumulating the counts.
• Presentation: User interaction: (1) adjust levels by drilling,
(2) pivoting, (3) mapping into rules, cross tabs,
visualization presentations.
Compiled by :Rashmi Bohra 44
 Example
• DMQL: Describe general characteristics of graduate
students in the Big-University database
use Big_University_DB
mine characteristics as “Science_Students”
in relevance to name, gender, major, birth_place,
birth_date, residence, phone#, gpa
from student
where status in “graduate”
• Corresponding SQL statement:
Select name, gender, major, birth_place, birth_date,
residence, phone#, gpa
from student
where status in {“Msc”, “MBA”, “PhD” }
Compiled by :Rashmi Bohra 45
 Class Characterization: An
Example
Name Gender Major Birth-Place Birth_date Residence Phone # GPA

Initial Jim M CS Vancouver,BC, 8-12-76 3511 Main St., 687-4598 3.67

Woodman Canada Richmond
Relation Scott M CS Montreal, Que, 28-7-75 345 1st Ave., 253-9106 3.70
Lachance Canada Richmond
Laura Lee F Physics Seattle, WA, USA 25-8-70 125 Austin Ave., 420-5232 3.83
… … … … … Burnaby … …
…
Removed Retained Sci,Eng, Country Age range City Removed Excl,
Bus VG,..
Gender Major Birth_region Age_range Residence GPA Count
Prime M Science Canada 20-25 Richmond Very-good 16
Generalized F Science Foreign 25-30 Burnaby Excellent 22
Relation … … … … … … …

Birth_Region
Canada Foreign Total
Gender
M 16 14 30
F 10 22 32
Total 26 36 62
Compiled by :Rashmi Bohra 46
 Presentation of Generalized
Results
• Generalized relation:
– Relations where some or all attributes are generalized, with counts
or other aggregation values accumulated.
• Cross tabulation:
– Mapping results into cross tabulation form (similar to contingency
tables).
– Visualization techniques:
– Pie charts, bar charts, curves, cubes, and other visual forms.
• Quantitative characteristic rules:
– Mapping generalized result into characteristic rules with quantitative
information
grad ( x) ∧ male(associated
x) ⇒ with it, e.g.,
birth _ region ( x) ="CanadaCompiled
"[t :53%] ∨ birth Bohra
by :Rashmi _ region ( x) =" foreign"[t : 47%]47
.
 Mining Class Comparisons
• Comparison: Comparing two or more classes
• Method:
– Partition the set of relevant data into the target class and the
contrasting class(es)
– Generalize both classes to the same high level concepts
– Compare tuples with the same high level descriptions
– Present for every tuple its description and two measures
• support - distribution within single class
• comparison - distribution between classes
– Highlight the tuples with strong discriminant features
• Relevance Analysis:
– Find attributes (features) which best distinguish different classes
Compiled by :Rashmi Bohra 48
 Class Description
Quantitative characteristic rule
•

–
∀ X, target_class(X) ⇒ condition(X) [t : t_weight]
necessary
Quantitative discriminant rule
•

sufficient
–
Quantitative description rule
•
∀ X, target_cla ss(X) ⇐ condition( X) [d : d_weight]

necessary and sufficient

–

∀ X, target_cla ss(X) ⇔
condition 1(X) [t : w1, d : w ′1] ∨ ... ∨ condition n(X) [t : wn, d : w ′n]

Compiled by :Rashmi Bohra 49

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 50

 Classification vs. Prediction
• Classification
– predicts categorical class labels (discrete or nominal)
– classifies data (constructs a model) based on the training set
and the values (class labels) in a classifying attribute and uses it
in classifying new data
• Prediction
– models continuous-valued functions, i.e., predicts unknown or
missing values
• Typical applications
– Credit approval
– Target marketing
– Medical diagnosis
– Fraud detection
Compiled by :Rashmi Bohra 51
 Classification—A Two-Step
Process
• Model construction: describing a set of predetermined
classes
– Each tuple/sample is assumed to belong to a predefined class, as
determined by the class label attribute
– The set of tuples used for model construction is training set
– The model is represented as classification rules, decision trees, or
mathematical formulae
• Model usage: for classifying future or unknown objects
– Estimate accuracy of the model
• The known label of test sample is compared with the classified
result from the model
• Accuracy rate is the percentage of test set samples that are
correctly classified by the model
• Test set is independent of training set, otherwise over-fitting will
occur
– If the accuracy is acceptable, use the model to classify data tuples
whose class labels are not known
Compiled by :Rashmi Bohra 52
 Process (1): Model
Construction

Classification
Algorithms
Training
Data

NAM E RANK YEARS TENURED Classifier

M ike Assistant Prof 3 no (Model)
M ary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes
IF rank = ‘professor’
Dave Assistant Prof 6 no
OR years > 6
Anne Associate Prof 3 no
Compiled by :Rashmi Bohra THEN tenured = ‘yes’
53
 Process (2): Using the Model in
Prediction

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)
N AM E R AN K YEARS TENUR ED
Tom Assistant Prof 2 no Tenured?
M erlisa Associate Prof 7 no
G eorge Professor 5 yes
Joseph Assistant Prof Compiled
7 by :Rashmi
yesBohra 54
 Supervised vs. Unsupervised
Learning

• Supervised learning (classification)

– Supervision: The training data (observations, measurements,
etc.) are accompanied by labels indicating the class of the
observations
– New data is classified based on the training set
• Unsupervised learning (clustering)
– The class labels of training data is unknown
– Given a set of measurements, observations, etc. with the aim
of establishing the existence of classes or clusters in the data

Compiled by :Rashmi Bohra 55

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 56

 Issues: Data Preparation

• Data cleaning
– Preprocess data in order to reduce noise and
handle missing values
• Relevance analysis (feature selection)
– Remove the irrelevant or redundant attributes
• Data transformation
– Generalize and/or normalize data

Compiled by :Rashmi Bohra 57

Issues: Evaluating Classification Methods
• Accuracy
– classifier accuracy: predicting class label
– predictor accuracy: guessing value of predicted attributes
• Speed
– time to construct the model (training time)
– time to use the model (classification/prediction time)
• Robustness: handling noise and missing values
• Scalability: efficiency in disk-resident databases
• Interpretability
– understanding and insight provided by the model
• Other measures, e.g., goodness of rules, such as
decision tree size or compactness of classification rules
Compiled by :Rashmi Bohra 58
 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 59

 Decision Tree Induction: Training
Dataset
age income student credit_rating buys_computer
<=30 high no fair no
This <=30 high no excellent no
follows 31…40 high no fair yes
an >40 medium no fair yes
>40 low yes fair yes
example >40 low yes excellent no
of 31…40 low yes excellent yes
Quinlan’s <=30 medium no fair no
<=30 low yes fair yes
ID3 >40 medium yes fair yes
(Playing <=30 medium yes excellent yes
Tennis) 31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no Bohra
Compiled by :Rashmi excellent no 60
 Output: A Decision Tree for
“buys_computer”

age?

<=30 overcast
31..40 >40

student? yes credit rating?

no yes excellent fair

no yes no yes

Compiled by :Rashmi Bohra 61

 Algorithm for Decision Tree
Induction
• Basic algorithm (a greedy algorithm)
– Tree is constructed in a top-down recursive divide-and-conquer
manner
– At start, all the training examples are at the root
– Attributes are categorical (if continuous-valued, they are
discretized in advance)
– Examples are partitioned recursively based on selected attributes
– Test attributes are selected on the basis of a heuristic or statistical
measure (e.g., information gain)
• Conditions for stopping partitioning
– All samples for a given node belong to the same class
– There are no remaining attributes for further partitioning – majority
voting is employed for classifying the leaf
Compiled
– There are no samples left by :Rashmi Bohra 62
 Attribute Selection Measure:
Information Gain (ID3/C4.5)
 Select the attribute with the highest information
gain
 Let pi be the probability that an arbitrary tuple in D
belongs to class Ci, estimated by |Ci, D |/|D|
m
 Expected information (entropy) needed
Info ( D) = to classify
−∑ pi log 2 ( pi )
a tuple in D: i =1

| Dj | v
 Information needed (after using = ∑split D
InfoA ( DA) to × I into
(D j ) v
partitions) to classify D: j =1 | D |

 Information gained by branching

Gain(A) on attribute
= Info(D) A
− Info (D) A
Compiled by :Rashmi Bohra 63
 Attribute Selection: Information Gain
5 4
Info age ( D) = I (2,3) + I (4,0)
 Class P: buys_computer 14 14
= “yes” 9 9 5 5 5
Info ( D) = I (9,5) = − log ( ) − log ( ) =0.940
 Class N:
+ I (3,2) = 0.694
14buys_computer
2 2
14 14 14 14
= “no” 5
age pi ni I(pi, ni) I (2,3) means “age <=30” has 5 out
14
<=30 2 3 0.971 of 14 samples, with 2 yes’es and
31…40 4 0 0 3 no’s. Hence
age >40 income student3 credit_rating
2 0.971 buys_computer Gain ( age ) = Info ( D ) − Info age ( D ) = 0.246
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
Similarly,
>40 medium no fair yes
>40
>40
low
low
yes
yes
fair
excellent
yes
no Gain(income) = 0.029
31…40 low yes excellent yes
<=30 medium no fair no Gain( student ) = 0.151
<=30 low yes fair yes
>40 medium yes fair yes
Compiled
Gain(credit _ rating ) = 0.048
by :Rashmi Bohra 64
<=30 medium yes excellent yes
31…40 medium no excellent yes
 Computing Information-Gain for
Continuous-Value Attributes
• Let attribute A be a continuous-valued attribute
• Must determine the best split point for A
– Sort the value A in increasing order
– Typically, the midpoint between each pair of adjacent
values is considered as a possible split point
• (ai+ai+1 )/2 is the midpoint between the values of ai and ai+1

– The point with the minimum expected information

requirement for A is selected as the split-point for A
• Split:
– D1 is the set of tuples in D satisfying A ≤ split-point, and D2
is the set of tuples Compiled
in D satisfying A > split-point
by :Rashmi Bohra 65
 Gain Ratio for Attribute Selection
(C4.5)
• Information gain measure is biased towards attributes with a
large number of values
• C4.5 (a successor of ID3) uses gain ratio to overcome the
problem (normalization to information gain)
v | Dj | | Dj |
SplitInfo A ( D ) = −∑ × log 2 ( )
j =1 |D| |D|

– GainRatio(A) = Gain(A)/SplitInfo(A)
4 4 6 6 4 4
SplitInfo A ( D) = − ×log 2 ( ) − ×log 2 ( ) − ×log 2 ( ) = 0.926
• Ex. 14 14 14 14 14 14
– gain_ratio(income) = 0.029/0.926 = 0.031
• The attribute with the maximum gain ratio is selected as the
splitting attribute
Compiled by :Rashmi Bohra 66
 Gini index (CART, IBM
IntelligentMiner)
• If a data set D contains examples from n classes, gini index, gini(D) is
defined as n
gini ( D) =1− ∑ p 2j
j =1
where pj is the relative frequency of class j in D
• If a data set D is split on A into two subsets D1 and D2, the gini index gini(D)
is defined as
|D1| |D |
gini A ( D) = gini ( D1) + 2 gini ( D 2)
|D| |D|
• Reduction in Impurity:
∆gini ( A) = gini ( D) − gini A ( D)

• The attribute provides the smallest ginisplit (D) (or the largest reduction in
impurity) is chosen to split the node (need to enumerate all the possible
splitting points for each attribute)
Compiled by :Rashmi Bohra 67
 Gini index (CART, IBM
IntelligentMiner)
• Ex. D has 9 tuples in buys_computer = “yes” and 5 in “no”
2 2
9 5
gini ( D) = 1 −   −   = 0.459
 14   14 
• Suppose the attribute income partitions D into 10 in D1: {low,
 10  4
medium} and 4 in D2 giniincome∈{low , medium} ( D ) =  Gini ( D1 ) +  Gini ( D1 )
 14   14 

but gini{medium,high} is 0.30 and thus the best since it is the lowest

• All attributes are assumed continuous-valued

• May need other tools, e.g., clustering, to get the possible split values
• Can be modified for categorical attributes
Compiled by :Rashmi Bohra 68
 Comparing Attribute Selection
Measures
• The three measures, in general, return good results
but
– Information gain:
• biased towards multivalued attributes
– Gain ratio:
• tends to prefer unbalanced splits in which one partition is
much smaller than the others
– Gini index:
• biased to multivalued attributes
• has difficulty when # of classes is large
• tends to favor tests that result in equal-sized partitions
and purity in bothCompiled
partitions
by :Rashmi Bohra 69
Other Attribute Selection Measures
• CHAID: a popular decision tree algorithm, measure based on χ2 test for
independence
• C-SEP: performs better than info. gain and gini index in certain cases
• G-statistics: has a close approximation to χ2 distribution
• MDL (Minimal Description Length) principle (i.e., the simplest solution is
preferred):
– The best tree as the one that requires the fewest # of bits to both (1) encode the tree,
and (2) encode the exceptions to the tree

• Multivariate splits (partition based on multiple variable combinations)

– CART: finds multivariate splits based on a linear comb. of attrs.

• Which attribute selection measure is the best?

– Most give good results, none is significantly superior than others

Compiled by :Rashmi Bohra 70

 Overfitting and Tree Pruning
• Overfitting: An induced tree may overfit the training
data
– Too many branches, some may reflect anomalies due to noise or outliers
– Poor accuracy for unseen samples

• Two approaches to avoid overfitting

– Prepruning: Halt tree construction early—do not split a node if this would
result in the goodness measure falling below a threshold
• Difficult to choose an appropriate threshold
– Postpruning: Remove branches from a “fully grown” tree—get a sequence
of progressively pruned trees
• Use a set of data different from the training data to decide which is the
“best pruned tree”
Compiled by :Rashmi Bohra 71
 Enhancements to Basic Decision Tree
Induction
• Allow for continuous-valued attributes
– Dynamically define new discrete-valued attributes that
partition the continuous attribute value into a discrete set
of intervals
• Handle missing attribute values
– Assign the most common value of the attribute
– Assign probability to each of the possible values
• Attribute construction
– Create new attributes based on existing ones that are
sparsely represented
– This reduces fragmentation, repetition,
Compiled by :Rashmi Bohra and replication 72
 Classification in Large Databases
• Classification—a classical problem extensively studied by
statisticians and machine learning researchers
• Scalability: Classifying data sets with millions of examples
and hundreds of attributes with reasonable speed
• Why decision tree induction in data mining?
– relatively faster learning speed (than other classification methods)
– convertible to simple and easy to understand classification rules
– can use SQL queries for accessing databases
– comparable classification accuracy with other methods

Compiled by :Rashmi Bohra 73

 Scalable Decision Tree Induction
Methods
• SLIQ (EDBT’96 — Mehta et al.)
– Builds an index for each attribute and only class list and the current
attribute list reside in memory
• SPRINT (VLDB’96 — J. Shafer et al.)
– Constructs an attribute list data structure
• PUBLIC (VLDB’98 — Rastogi & Shim)
– Integrates tree splitting and tree pruning: stop growing the tree earlier
• RainForest (VLDB’98 — Gehrke, Ramakrishnan & Ganti)
– Builds an AVC-list (attribute, value, class label)
• BOAT (PODS’99 — Gehrke, Ganti, Ramakrishnan & Loh)
– Uses bootstrapping to create several small samples

Compiled by :Rashmi Bohra 74

 Scalability Framework for
RainForest
• Separates the scalability aspects from the criteria that
determine the quality of the tree
• Builds an AVC-list: AVC (Attribute, Value, Class_label)
• AVC-set (of an attribute X )
– Projection of training dataset onto the attribute X and class label
where counts of individual class label are aggregated

• AVC-group (of a node n )

– Set of AVC-sets of all predictor attributes at the node n

Compiled by :Rashmi Bohra 75

 Rainforest: Training Set and Its AVC
Sets
Training Examples AVC-set on Age AVC-set on income
age income studentc redit_rating
buys_computer income Buy_Computer
<=30 high no fair no Age Buy_Computer
<=30 high no excellent no yes no
31…40 high no fair yes yes no
high 2 2
>40 medium no fair yes <=30 3 2
medium 4 2
>40 low yes fair yes 31..40 4 0
>40 low yes excellent no
low 3 1
31…40 low yes excellent yes >40 3 2
<=30 medium no fair no
<=30 low yes fair yes AVC-set on
AVC-set on Student
>40 medium yes fair yes
student Buy_Computer credit_rating
<=30 medium yes excellent yes
Credit Buy_Computer
31…40 medium no excellent yes rating
yes no
31…40 high yes fair yes
yes no
>40 medium no excellent no yes 6 1
no 3 4 fair 6 2

Compiled by :Rashmi Bohra excellent 3 76 3

 Data Cube-Based Decision-Tree
Induction
• Integration of generalization with decision-tree
induction (Kamber et al.’97)
• Classification at primitive concept levels
– E.g., precise temperature, humidity, outlook, etc.
– Low-level concepts, scattered classes, bushy classification-
trees
– Semantic interpretation problems
• Cube-based multi-level classification
– Relevance analysis at multi-levels
– Information-gain analysis
Compiled bywith dimension
:Rashmi Bohra + level 77
 BOAT (Bootstrapped Optimistic
Algorithm for Tree Construction)
• Use a statistical technique called bootstrapping to create several smaller
samples (subsets), each fits in memory

• Each subset is used to create a tree, resulting in several trees

• These trees are examined and used to construct a new tree T’
– It turns out that T’ is very close to the tree that would be generated using the whole data
set together

• Adv: requires only two scans of DB, an incremental alg.

Compiled by :Rashmi Bohra 78

Presentation of Classification Results

Compiled by :Rashmi Bohra 79

Visualization of a Decision Tree in SGI/MineSet
3.0

Compiled by :Rashmi Bohra 80

Interactive Visual Mining by Perception-
Based Classification (PBC)

Compiled by :Rashmi Bohra 81

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 82

 Bayesian Classification: Why?
• A statistical classifier: performs probabilistic prediction, i.e.,
predicts class membership probabilities
• Foundation: Based on Bayes’ Theorem.
• Performance: A simple Bayesian classifier, naïve Bayesian
classifier, has comparable performance with decision tree and
selected neural network classifiers
• Incremental: Each training example can incrementally
increase/decrease the probability that a hypothesis is correct —
prior knowledge can be combined with observed data
• Standard: Even when Bayesian methods are computationally
intractable, they can provide a standard of optimal decision
making against which other methods can be measured

Compiled by :Rashmi Bohra 83

 Bayesian Theorem: Basics
• Let X be a data sample (“evidence”): class label is unknown
• Let H be a hypothesis that X belongs to class C
• Classification is to determine P(H|X), the probability that the hypothesis
holds given the observed data sample X
• P(H) (prior probability), the initial probability
– E.g., X will buy computer, regardless of age, income, …
• P(X): probability that sample data is observed
• P(X|H) (posteriori probability), the probability of observing the sample X,
given that the hypothesis holds
– E.g., Given that X will buy computer, the prob. that X is 31..40, medium income

Compiled by :Rashmi Bohra 84

 Bayesian Theorem
• Given training data X, posteriori probability of a hypothesis H,
P(H|X), follows the Bayes theorem
P(H | X) = P(X | H )P(H )
P(X)
• Informally, this can be written as
posteriori = likelihood x prior/evidence

• Predicts X belongs to C2 iff the probability P(Ci|X) is the highest

among all the P(Ck|X) for all the k classes
• Practical difficulty: require initial knowledge of many probabilities,
significant computational cost
Compiled by :Rashmi Bohra 85
 Towards Naïve Bayesian
Classifier
• Let D be a training set of tuples and their associated class labels, and
each tuple is represented by an n-D attribute vector X = (x1, x2, …, xn)
• Suppose there are m classes C1, C2, …, Cm.
• Classification is to derive the maximum posteriori, i.e., the maximal
P(Ci|X)
• This can be derived from Bayes’ theorem

P(X | C ) P(C )
P(C | X) = i i
i P(X)
• Since P(X) is constant for all classes, only
P(C | X) = P(X | C ) P(C )
needs to be maximized i i i

Compiled by :Rashmi Bohra 86

 Derivation of Naïve Bayes
•
Classifier
A simplified assumption: attributes are conditionally
independent (i.e., no dependence relation between attributes):
n
P ( X | C i ) = ∏ P( x | C i ) = P ( x | C i ) × P( x | C i ) ×... × P ( x | C i )
• This greatly reduces thek computation
=1
k cost:
1 Only2 counts then
class distribution
• If Ak is categorical, P(xk|Ci) is the # of tuples in Ci having value
xk for Ak divided by |Ci,D | (# of tuples of Ci in D)
• If Ak is continous-valued, P(xk|Ci) is usually computed based on
Gaussian distribution with a mean μ and standard deviation σ
( x −µ) 2
1 −
and P(xk|Ci) is g ( x, µ, σ) = e 2σ 2
2πσ

P ( X | C i ) = g ( xk , µ C i , σ C i )
Compiled by :Rashmi Bohra 87
 Naïve Bayesian Classifier: Training
Dataset age income studentcredit_rating
buys_computer
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
>40 medium no fair yes
Class:
>40 low yes fair yes
C1:buys_computer = >40 low yes excellent no
‘yes’ 31…40 low yes excellent yes
C2:buys_computer = ‘no’<=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
Data sample <=30 medium yes excellent yes
X = (age <=30, 31…40 medium no excellent yes
31…40 high yes fair yes
Income = medium,
>40 medium no excellent no
Student = yes
Credit_rating = Fair)

Compiled by :Rashmi Bohra 88

 Naïve Bayesian Classifier: An
Example
• P(Ci): P(buys_computer = “yes”) = 9/14 = 0.643
P(buys_computer = “no”) = 5/14= 0.357

• Compute P(X|Ci) for each class

P(age = “<=30” | buys_computer = “yes”) = 2/9 = 0.222
P(age = “<= 30” | buys_computer = “no”) = 3/5 = 0.6
P(income = “medium” | buys_computer = “yes”) = 4/9 = 0.444
P(income = “medium” | buys_computer = “no”) = 2/5 = 0.4
P(student = “yes” | buys_computer = “yes) = 6/9 = 0.667
P(student = “yes” | buys_computer = “no”) = 1/5 = 0.2
P(credit_rating = “fair” | buys_computer = “yes”) = 6/9 = 0.667
P(credit_rating = “fair” | buys_computer = “no”) = 2/5 = 0.4

• X = (age <= 30 , income = medium, student = yes, credit_rating = fair)

P(X|Ci) : P(X|buys_computer = “yes”) = 0.222 x 0.444 x 0.667 x 0.667 = 0.044

P(X|buys_computer = “no”) = 0.6 x 0.4 x 0.2 x 0.4 = 0.019
P(X|Ci)*P(Ci) : P(X|buys_computer = “yes”) * P(buys_computer = “yes”) = 0.028
P(X|buys_computer = “no”) * P(buys_computer = “no”) = 0.007

Therefore, X belongs to class (“buys_computer = yes”)

Compiled by :Rashmi Bohra 89
 Avoiding the 0-Probability
Problem
• Naïve Bayesian prediction requires each conditional prob. be non-
zero. Otherwise, the predicted prob. will be zero
n
P( X | C i) = ∏P ( x k | C i )
k =1

• Ex. Suppose a dataset with 1000 tuples, income=low (0), income=

medium (990), and income = high (10),
• Use Laplacian correction (or Laplacian estimator)
– Adding 1 to each case
Prob(income = low) = 1/1003
Prob(income = medium) = 991/1003
Prob(income = high) = 11/1003
– The “corrected” prob. estimates are close to their “uncorrected” counterparts

Compiled by :Rashmi Bohra 90

 Naïve Bayesian Classifier:
Comments
• Advantages
– Easy to implement
– Good results obtained in most of the cases
• Disadvantages
– Assumption: class conditional independence, therefore loss of
accuracy
– Practically, dependencies exist among variables
• E.g., hospitals: patients: Profile: age, family history, etc.
Symptoms: fever, cough etc., Disease: lung cancer, diabetes, etc.
• Dependencies among these cannot be modeled by Naïve Bayesian
Classifier
• How to deal with these dependencies?
– Bayesian Belief Networks
Compiled by :Rashmi Bohra 91
 Bayesian Belief Networks
• Bayesian belief network allows a subset of the
variables conditionally independent
• A graphical model of causal relationships
– Represents dependency among the variables
– Gives a specification of joint probability distribution
 Nodes: random variables
 Links: dependency
X Y  X and Y are the parents of Z, and
Y is the parent of P
Z  No dependency between Z and P
P
CompiledbyHas noBohra
:Rashmi loops or cycles 92
 Bayesian Belief Network: An
Example
Family The conditional probability
Smoker
History table (CPT) for variable
LungCancer:
(FH, S) (FH, ~S) (~FH, S) (~FH, ~S)

LC 0.8 0.5 0.7 0.1

LungCancer Emphysema ~LC 0.2 0.5 0.3 0.9

CPT shows the conditional probability

for each possible combination of its
parents

PositiveXRay Dyspnea Derivation of the probability of a

particular combination of values
of X, from CPT:
n
Bayesian Belief Networks P ( x1 ,..., xn ) = ∏ P ( x i | Parents (Y i ))
Compiled by :Rashmi Bohra
i =1 93
 Training Bayesian Networks
• Several scenarios:
– Given both the network structure and all variables observable:
learn only the CPTs
– Network structure known, some hidden variables: gradient
descent (greedy hill-climbing) method, analogous to neural
network learning
– Network structure unknown, all variables observable: search
through the model space to reconstruct network topology
– Unknown structure, all hidden variables: No good algorithms
known for this purpose
• Ref. D. Heckerman: Bayesian networks for data mining

Compiled by :Rashmi Bohra 94

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 95

 Using IF-THEN Rules for
Classification
• Represent the knowledge in the form of IF-THEN rules
R: IF age = youth AND student = yes THEN buys_computer = yes
– Rule antecedent/precondition vs. rule consequent
• Assessment of a rule: coverage and accuracy
– ncovers = # of tuples covered by R
– ncorrect = # of tuples correctly classified by R
coverage(R) = ncovers /|D| /* D: training data set */
accuracy(R) = ncorrect / ncovers
• If more than one rule is triggered, need conflict resolution
– Size ordering: assign the highest priority to the triggering rules that has the
“toughest” requirement (i.e., with the most attribute test)
– Class-based ordering: decreasing order of prevalence or misclassification cost
per class
– Rule-based ordering (decision list): rules are organized into one long priority
list, according to some measure
Compiledof
byrule quality
:Rashmi or by experts
Bohra 96
Rule Extraction from a Decision Tree age?

<=30 31..40 >40

• Rules are easier to understand than large trees
student? credit rating?
• One rule is created for each path from the root to a leaf yes

• Each attribute-value pair along a path forms a conjunction: the leaf no yes excellent fair
holds the class prediction no yes
no yes
• Rules are mutually exclusive and exhaustive

• Example: Rule extraction from our buys_computer decision-tree

IF age = young AND student = no THEN buys_computer = no
IF age = young AND student = yes THEN buys_computer = yes
IF age = mid-age THEN buys_computer = yes
IF age = old AND credit_rating = excellent THEN buys_computer = yes
IF age = young AND credit_rating = fair THEN buys_computer = no
Compiled by :Rashmi Bohra 97
 Rule Extraction from the Training
Data
• Sequential covering algorithm: Extracts rules directly from training data
• Typical sequential covering algorithms: FOIL, AQ, CN2, RIPPER
• Rules are learned sequentially, each for a given class Ci will cover many tuples of Ci
but none (or few) of the tuples of other classes
• Steps:
– Rules are learned one at a time
– Each time a rule is learned, the tuples covered by the rules are removed
– The process repeats on the remaining tuples unless termination condition, e.g., when no more
training examples or when the quality of a rule returned is below a user-specified threshold

• Comp. w. decision-tree induction: learning a set of rules simultaneously

Compiled by :Rashmi Bohra 98

 How to Learn-One-Rule?
• Star with the most general rule possible: condition = empty
• Adding new attributes by adopting a greedy depth-first strategy
– Picks the one that most improves the rule quality
• Rule-Quality measures: consider both coverage and accuracy
– Foil-gain (in FOIL & RIPPER): assesses info_gain by extending condition
pos ' pos
FOIL _ Gain = pos '×(log
It favors rules that have high accuracy and cover − log )
pos 'many
+ neg ' positivepos
2 2 tuples
+ neg
• Rule pruning based on an independent set of test tuples

pos − neg
FOIL _ Prune ( R ) =
pos + neg
Pos/neg are # of positive/negative tuples covered by R.
If FOIL_Prune is higher for the pruned version of R, prune R

Compiled by :Rashmi Bohra 99

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 100

Classification: A Mathematical Mapping
• Classification:
– predicts categorical class labels
• E.g., Personal homepage classification
– xi = (x1, x2, x3, …), yi = +1 or –1
– x1 : # of a word “homepage”
– x2 : # of a word “welcome”
• Mathematically
– x ∈ X = ℜn, y ∈ Y = {+1, –1}
– We want a function f: X  Y
Compiled by :Rashmi Bohra 101
 Linear Classification
• Binary Classification
problem
• The data above the red line
belongs to class ‘x’
x
x • The data below red line
x x x belongs to class ‘o’
x
x x o • Examples: SVM,
x Perceptron, Probabilistic
o
x o o Classifiers
ooo
o o
o o o o

Compiled by :Rashmi Bohra 102

 Discriminative Classifiers
• Advantages
– prediction accuracy is generally high
• As compared to Bayesian methods – in general
– robust, works when training examples contain errors
– fast evaluation of the learned target function
• Bayesian networks are normally slow
• Criticism
– long training time
– difficult to understand the learned function (weights)
• Bayesian networks can be used easily for pattern discovery
– not easy to incorporate domain knowledge
• Easy in the form ofCompiled
priors byon:Rashmi
the data
Bohra
or distributions 103
 Perceptron & Winnow
• Vector: x, w
x2
• Scalar: x, y, w
Input: {(x1, y1), …}
Output: classification function
f(x)
f(xi) > 0 for yi = +1
f(xi) < 0 for yi = -1
f(x) => wx + b = 0
• or w1x1+w2x2update
Perceptron: +b = 0
W additively
• Winnow: update W
multiplicatively
x1
Compiled by :Rashmi Bohra 104
 Classification by
Backpropagation
• Backpropagation: A neural network learning algorithm
• Started by psychologists and neurobiologists to develop and test computational
analogues of neurons

• A neural network: A set of connected input/output units where each connection

has a weight associated with it
• During the learning phase, the network learns by adjusting the weights so as
to be able to predict the correct class label of the input tuples
• Also referred to as connectionist learning due to the connections between units

Compiled by :Rashmi Bohra 105

 Neural Network as a Classifier
• Weakness
– Long training time
– Require a number of parameters typically best determined
empirically, e.g., the network topology or ``structure."
– Poor interpretability: Difficult to interpret the symbolic meaning
behind the learned weights and of ``hidden units" in the network
• Strength
– High tolerance to noisy data
– Ability to classify untrained patterns
– Well-suited for continuous-valued inputs and outputs
– Successful on a wide array of real-world data
– Algorithms are inherently parallel
– Techniques have recently been developed for the extraction of rules
from trained neural networks
Compiled by :Rashmi Bohra 106
 A Neuron (= a perceptron)
-µ k
x0 w0
x1 w1
∑ f
output y
xn wn
For Example
n
Input weight weighted Activation y = sign(∑ wi xi + µ k )
vector x vector w sum function i =0

• The n-dimensional input vector x is mapped into variable y by means

of the scalar product and a nonlinear function mapping

Compiled by :Rashmi Bohra 107

 A Multi-Layer Feed-Forward Neural
Network

Output vector

Err j = O j (1 − O j )∑ Errk w jk
Output layer k

θ j = θ j + (l) Err j
wij = wij + (l ) Err j Oi
Hidden layer Err j = O j (1 − O j )(T j − O j )
wij 1
Oj = −I j
1+ e
Input layer
I j = ∑ wij Oi + θ j
i
Input vector: X Compiled by :Rashmi Bohra 108
 How A Multi-Layer Neural Network
Works?
• The inputs to the network correspond to the attributes measured for each training tuple
• Inputs are fed simultaneously into the units making up the input layer
• They are then weighted and fed simultaneously to a hidden layer
• The number of hidden layers is arbitrary, although usually only one
• The weighted outputs of the last hidden layer are input to units making up the output layer,
which emits the network's prediction

• The network is feed-forward in that none of the weights cycles back to an input unit or to an
output unit of a previous layer
• From a statistical point of view, networks perform nonlinear regression: Given enough hidden
units and enough training samples, they can closely approximate any function

Compiled by :Rashmi Bohra 109

 Defining a Network Topology
• First decide the network topology: # of units in the input
layer, # of hidden layers (if > 1), # of units in each hidden
layer, and # of units in the output layer
• Normalizing the input values for each attribute measured in
the training tuples to [0.0—1.0]
• One input unit per domain value, each initialized to 0
• Output, if for classification and more than two classes, one
output unit per class is used
• Once a network has been trained and its accuracy is
unacceptable, repeat the training process with a different
network topology or a different set of initial weights
Compiled by :Rashmi Bohra 110
 Backpropagation
• Iteratively process a set of training tuples & compare the
network's prediction with the actual known target value
• For each training tuple, the weights are modified to minimize the
mean squared error between the network's prediction and the
actual target value
• Modifications are made in the “backwards” direction: from the
output layer, through each hidden layer down to the first hidden
layer, hence “backpropagation”
• Steps
– Initialize weights (to small random #s) and biases in the network
– Propagate the inputs forward (by applying activation function)
– Backpropagate the error (by updating weights and biases)
– Terminating condition (when error is very small, etc.)
Compiled by :Rashmi Bohra 111
Backpropagation and Interpretability
• Efficiency of backpropagation: Each epoch (one interation through
the training set) takes O(|D| * w), with |D| tuples and w weights, but
# of epochs can be exponential to n, the number of inputs, in the
worst case
• Rule extraction from networks: network pruning
– Simplify the network structure by removing weighted links that have the least
effect on the trained network
– Then perform link, unit, or activation value clustering
– The set of input and activation values are studied to derive rules describing
the relationship between the input and hidden unit layers
• Sensitivity analysis: assess the impact that a given input variable
has on a network output. The knowledge gained from this analysis
can be represented in rules
Compiled by :Rashmi Bohra 112
 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 113

 SVM—Support Vector Machines
• A new classification method for both linear and nonlinear data
• It uses a nonlinear mapping to transform the original training data
into a higher dimension
• With the new dimension, it searches for the linear optimal
separating hyperplane (i.e., “decision boundary”)
• With an appropriate nonlinear mapping to a sufficiently high
dimension, data from two classes can always be separated by a
hyperplane
• SVM finds this hyperplane using support vectors (“essential”
training tuples) and margins (defined by the support vectors)

Compiled by :Rashmi Bohra 114

 SVM—History and Applications
• Vapnik and colleagues (1992)—groundwork from
Vapnik & Chervonenkis’ statistical learning theory in
1960s
• Features: training can be slow but accuracy is high
owing to their ability to model complex nonlinear
decision boundaries (margin maximization)
• Used both for classification and prediction
• Applications:
– handwritten digit recognition, object recognition, speaker
identification, benchmarking time-series prediction tests
Compiled by :Rashmi Bohra 115
 SVM—General Philosophy

Small Margin Large Margin

Support Vectors
Compiled by :Rashmi Bohra 116
SVM—Margins and Support
Vectors

Compiled by :Rashmi Bohra 117

 Associative Classification
• Associative classification
– Association rules are generated and analyzed for use in classification
– Search for strong associations between frequent patterns (conjunctions of
attribute-value pairs) and class labels
– Classification: Based on evaluating a set of rules in the form of

P1 ^ p2 … ^ pl  “Aclass = C” (conf, sup)

• Why effective?
– It explores highly confident associations among multiple attributes and may
overcome some constraints introduced by decision-tree induction, which
considers only one attribute at a time
– In many studies, associative classification has been found to be more accurate
than some traditional classification methods, such as C4.5
Compiled by :Rashmi Bohra 118
 Typical Associative Classification
Methods
• CBA (Classification By Association: Liu, Hsu & Ma, KDD’98)
– Mine association possible rules in the form of
• Cond-set (a set of attribute-value pairs)  class label
– Build classifier: Organize rules according to decreasing precedence based on
confidence and then support
• CMAR (Classification based on Multiple Association Rules: Li, Han, Pei, ICDM’01)
– Classification: Statistical analysis on multiple rules
• CPAR (Classification based on Predictive Association Rules: Yin & Han, SDM’03)
– Generation of predictive rules (FOIL-like analysis)
– High efficiency, accuracy similar to CMAR
• RCBT (Mining top-k covering rule groups for gene expression data, Cong et al.
SIGMOD’05)
– Explore high-dimensional classification, using top-k rule groups
Compiled
– Achieve high classification by :Rashmi
accuracy Bohrarun-time efficiency
and high 119
 A Closer Look at CMAR
• CMAR (Classification based on Multiple Association Rules: Li, Han, Pei, ICDM’01)
• Efficiency: Uses an enhanced FP-tree that maintains the distribution
of class labels among tuples satisfying each frequent itemset
• Rule pruning whenever a rule is inserted into the tree
– Given two rules, R1 and R2, if the antecedent of R1 is more general than that of
R2 and conf(R1) ≥ conf(R2), then R2 is pruned
– Prunes rules for which the rule antecedent and class are not positively
correlated, based on a χ2 test of statistical significance
• Classification based on generated/pruned rules
– If only one rule satisfies tuple X, assign the class label of the rule
– If a rule set S satisfies X, CMAR
• divides S into groups according to class labels
• uses a weighted χ2 measure to find the strongest group of rules, based on
the statistical correlation of rules within a group
• assigns X the class label of the strongest group
Compiled by :Rashmi Bohra 120
 Associative Classification May Achieve
High Accuracy and Efficiency (Cong et al.
SIGMOD05)

Compiled by :Rashmi Bohra 121

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 122

 Lazy vs. Eager Learning
• Lazy vs. eager learning
– Lazy learning (e.g., instance-based learning): Simply stores
training data (or only minor processing) and waits until it is
given a test tuple
– Eager learning (the above discussed methods): Given a set of
training set, constructs a classification model before receiving
new (e.g., test) data to classify
• Lazy: less time in training but more time in predicting
• Accuracy
– Lazy method effectively uses a richer hypothesis space since it
uses many local linear functions to form its implicit global
approximation to the target function
– Eager: must commit to a single hypothesis that covers the
entire instance space
Compiled by :Rashmi Bohra 123
 Lazy Learner: Instance-Based
Methods
• Instance-based learning:
– Store training examples and delay the processing (“lazy
evaluation”) until a new instance must be classified
• Typical approaches
– k-nearest neighbor approach
• Instances represented as points in a Euclidean space.
– Locally weighted regression
• Constructs local approximation
– Case-based reasoning
• Uses symbolic representations and knowledge-based
inference

Compiled by :Rashmi Bohra 124

 The k-Nearest Neighbor
Algorithm
• All instances correspond to points in the n-D
space
• The nearest neighbor are defined in terms of
Euclidean distance, dist(X1, X2)
• Target function could be discrete- or real-
valued
• For discrete-valued, k-NN returns the most
common value among the k training examples
nearest
_ to xq .
_ _
• Vonoroi _
diagram: the decision surface induced
+
by 1-NN for
_ .
q
+
+ . .
x a typical set of training. examples
_
+ Compiled by :Rashmi Bohra
. 125
Discussion on the k-NN Algorithm
• k-NN for real-valued prediction for a given unknown tuple
– Returns the mean values of the k nearest neighbors
• Distance-weighted nearest neighbor algorithm
– Weight the contribution of each of the k neighbors according to their distance to
the query xq
• Give greater weight to closer neighbors
1 w≡
• Robust to noisy data by averaging k-nearest neighbors d ( xq , x )2
i
• Curse of dimensionality: distance between neighbors could be
dominated by irrelevant attributes
– To overcome it, axes stretch or elimination of the least relevant attributes

Compiled by :Rashmi Bohra 126

 Case-Based Reasoning (CBR)
• CBR: Uses a database of problem solutions to solve new problems
• Store symbolic description (tuples or cases)—not points in a Euclidean space
• Applications: Customer-service (product-related diagnosis), legal ruling
• Methodology
– Instances represented by rich symbolic descriptions (e.g., function graphs)
– Search for similar cases, multiple retrieved cases may be combined
– Tight coupling between case retrieval, knowledge-based reasoning, and problem solving
• Challenges
– Find a good similarity metric
– Indexing based on syntactic similarity measure, and when failure, backtracking, and
adapting to additional cases

Compiled by :Rashmi Bohra 127

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 128

 Genetic Algorithms (GA)
• Genetic Algorithm: based on an analogy to biological evolution
• An initial population is created consisting of randomly generated rules
– Each rule is represented by a string of bits
– E.g., if A1 and ¬A2 then C2 can be encoded as 100
– If an attribute has k > 2 values, k bits can be used
• Based on the notion of survival of the fittest, a new population is formed
to consist of the fittest rules and their offsprings
• The fitness of a rule is represented by its classification accuracy on a set
of training examples
• Offsprings are generated by crossover and mutation
• The process continues until a population P evolves when each rule in P
satisfies a prespecified threshold
• Slow but easily parallelizable
Compiled by :Rashmi Bohra 129
 Rough Set Approach
• Rough sets are used to approximately or “roughly” define
equivalent classes
• A rough set for a given class C is approximated by two sets: a lower
approximation (certain to be in C) and an upper approximation
(cannot be described as not belonging to C)
• Finding the minimal subsets (reducts) of attributes for feature
reduction is NP-hard but a discernibility matrix (which stores the
differences between attribute values for each pair of data tuples) is
used to reduce the computation intensity

Compiled by :Rashmi Bohra 130

 Fuzzy Set
Approaches

• Fuzzy logic uses truth values between 0.0 and 1.0 to

represent the degree of membership (such as using
fuzzy membership graph)
• Attribute values are converted to fuzzy values
– e.g., income is mapped into the discrete categories {low,
medium, high} with fuzzy values calculated
• For a given new sample, more than one fuzzy value
may apply
• Each applicable rule contributes a vote for membership
in the categories
• Typically, the truth values for each predicted category
are summed, and theseCompiledsums are
by :Rashmi combined
Bohra 131
 What Is Prediction?
• (Numerical) prediction is similar to classification
– construct a model
– use model to predict continuous or ordered value for a given input
• Prediction is different from classification
– Classification refers to predict categorical class label
– Prediction models continuous-valued functions
• Major method for prediction: regression
– model the relationship between one or more independent or predictor
variables and a dependent or response variable
• Regression analysis
– Linear and multiple regression
– Non-linear regression
– Other regression methods: generalized linear model, Poisson
regression, log-linear Compiled
models,byregression
:Rashmi Bohratrees 132
 Linear Regression
• Linear regression: involves a response variable y and a single
predictor variable x
y = w 0 + w1 x
where w0 (y-intercept) and w1 (slope) are regression coefficients

• Method of least squares: estimates the best-fitting straight line

| D|

∑( x − x )( yi − y )
w = w = y −w x
i
i =1

1 | D|
0 1
∑( x i =1
i − x )2

• Multiple linear regression: involves more than one predictor

variable
– Training data is of the form (X1, y1), (X2, y2),…, (X|D| , y|D| )
– Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2
– Solvable by extension of least square method or using SAS, S-Plus
Compiled by :Rashmi Bohra 133
– Many nonlinear functions can be transformed into the above
 Nonlinear Regression
• Some nonlinear models can be modeled by a polynomial
function
• A polynomial regression model can be transformed into linear
regression model. For example,
y = w0 + w1 x + w2 x2 + w3 x3
convertible to linear with new variables: x2 = x2, x3= x3
y = w0 + w1 x + w2 x2 + w3 x3
• Other functions, such as power function, can also be
transformed to linear model
• Some models are intractable nonlinear (e.g., sum of exponential
terms)
– possible to obtain least square estimates through extensive calculation
on more complex formulae
Compiled by :Rashmi Bohra 134
Other Regression-Based Models
• Generalized linear model:
– Foundation on which linear regression can be applied to modeling
categorical response variables
– Variance of y is a function of the mean value of y, not a constant
– Logistic regression: models the prob. of some event occurring as a
linear function of a set of predictor variables
– Poisson regression: models the data that exhibit a Poisson
distribution
• Log-linear models: (for categorical data)
– Approximate discrete multidimensional prob. distributions
– Also useful for data compression and smoothing
• Regression trees and model trees
– Trees to predict continuous values rather than class labels
Compiled by :Rashmi Bohra 135
 Regression Trees and Model
Trees
• Regression tree: proposed in CART system (Breiman et al. 1984)
– CART: Classification And Regression Trees
– Each leaf stores a continuous-valued prediction
– It is the average value of the predicted attribute for the training tuples that reach
the leaf

• Model tree: proposed by Quinlan (1992)

– Each leaf holds a regression model—a multivariate linear equation for the
predicted attribute
– A more general case than regression tree

• Regression and model trees tend to be more accurate than linear

regression when the data are not represented well by a simple linear
model
Compiled by :Rashmi Bohra 136
Predictive Modeling in Multidimensional
Databases
• Predictive modeling: Predict data values or construct
generalized linear models based on the database data
• One can only predict value ranges or category distributions
• Method outline:
– Minimal generalization
– Attribute relevance analysis
– Generalized linear model construction
– Prediction
• Determine the major factors which influence the prediction
– Data relevance analysis: uncertainty measurement, entropy analysis,
expert judgement, etc.
• Multi-level prediction: drill-down and roll-up analysis

Compiled by :Rashmi Bohra 137

Prediction: Numerical Data

Compiled by :Rashmi Bohra 138

Prediction: Categorical Data

Compiled by :Rashmi Bohra 139

 Classifier Accuracy C1 C2

Measures C1 True positive False negative

C2 False positive True negative

classes buy_computer = yes buy_computer = no total recognition(%)

buy_computer = yes 6954 46 7000 99.34

buy_computer = no 412 2588 3000 86.27

• Accuracy of a classifier M, acc(M): percentage of test set tuples that
total 7366 2634 10000 95.52
are correctly classified by the model M
– Error rate (misclassification rate) of M = 1 – acc(M)
– Given m classes, CMi,j , an entry in a confusion matrix, indicates # of tuples in
class i that are labeled by the classifier as class j
• Alternative accuracy measures (e.g., for cancer diagnosis)
sensitivity = t-pos/pos /* true positive recognition rate */
specificity = t-neg/neg /* true negative recognition rate */
precision = t-pos/(t-pos + f-pos)
accuracy = sensitivity * pos/(pos + neg) + specificity * neg/(pos + neg)
– This model can also be used for cost-benefit analysis
Compiled by :Rashmi Bohra 140
 Predictor Error Measures
• Measure predictor accuracy: measure how far off the predicted value is from
the actual known value
• Loss function: measures the error betw. yi and the predicted value yi’
– Absolute error: | yi – yi’|
– Squared error: (yi – yi’)2
• Test error (generalization error): the average loss over the test set
d d
– Mean absolute error: ∑Mean
| yi − ysquared
i =1
i '|
error:
∑(y i − yi ' ) 2
i =1
d
d
– Relative absolute error: Relative
d
| y − ysquared error: d
∑
i =1
d
i '|i
∑ ( yi − yi ' ) 2
i =1
∑| y i −y | d
The mean squared-error exaggerates
i =1the presence of outliers
∑(y i − y)2
Popularly use (square) root mean-square error, similarly, root relative squared error
i =1

Compiled by :Rashmi Bohra 141

 Evaluating the Accuracy of a
Classifier or Predictor (I)
• Holdout method
– Given data is randomly partitioned into two independent sets
• Training set (e.g., 2/3) for model construction
• Test set (e.g., 1/3) for accuracy estimation
– Random sampling: a variation of holdout
• Repeat holdout k times, accuracy = avg. of the accuracies
obtained
• Cross-validation (k-fold, where k = 10 is most popular)
– Randomly partition the data into k mutually exclusive subsets,
each approximately equal size
– At i-th iteration, use Di as test set and others as training set
– Leave-one-out: k folds where k = # of tuples, for small sized data
– Stratified cross-validation: folds are stratified so that class dist. in
each fold is approx. the same
Compiled as thatBohra
by :Rashmi in the initial data 142
 Evaluating the Accuracy of a
Classifier or Predictor (II)
• Bootstrap
– Works well with small data sets
– Samples the given training tuples uniformly with replacement
• i.e., each time a tuple is selected, it is equally likely to be selected again
and re-added to the training set
• Several boostrap methods, and a common one is .632 boostrap
– Suppose we are given a data set of d tuples. The data set is sampled d times, with
replacement, resulting in a training set of d samples. The data tuples that did not
make it into the training set end up forming the test set. About 63.2% of the original
data will end up in the bootstrap, and the remaining 36.8% will form the test set
(since (1 – 1/d)d ≈ e-1 = 0.368)
– Repeat the sampling procedue k times, overall accuracy of the model:
k
acc ( M ) = ∑(0.632 × acc ( M i ) test _ set +0.368 × acc ( M i ) train _ set )
i =1

Compiled by :Rashmi Bohra 143

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 144

 Ensemble Methods: Increasing the
Accuracy

• Ensemble methods
– Use a combination of models to increase accuracy
– Combine a series of k learned models, M1, M2, …, Mk,
with the aim of creating an improved model M*
• Popular ensemble methods
– Bagging: averaging the prediction over a collection of
classifiers
– Boosting: weighted vote with a collection of classifiers
– Ensemble: combining a set of heterogeneous classifiers
Compiled by :Rashmi Bohra 145
 Bagging: Boostrap Aggregation
• Analogy: Diagnosis based on multiple doctors’ majority vote
• Training
– Given a set D of d tuples, at each iteration i, a training set Di of d tuples is
sampled with replacement from D (i.e., boostrap)
– A classifier model Mi is learned for each training set Di
• Classification: classify an unknown sample X
– Each classifier Mi returns its class prediction
– The bagged classifier M* counts the votes and assigns the class with the most
votes to X
• Prediction: can be applied to the prediction of continuous values by
taking the average value of each prediction for a given test tuple
• Accuracy
– Often significant better than a single classifier derived from D
– For noise data: not considerably worse, more robust
– Proved improved accuracy in prediction
Compiled by :Rashmi Bohra 146
 Boosting
• Analogy: Consult several doctors, based on a combination of weighted
diagnoses—weight assigned based on the previous diagnosis accuracy
• How boosting works?
– Weights are assigned to each training tuple
– A series of k classifiers is iteratively learned
– After a classifier Mi is learned, the weights are updated to allow the subsequent classifier,
Mi+1 , to pay more attention to the training tuples that were misclassified by Mi
– The final M* combines the votes of each individual classifier, where the weight of each
classifier's vote is a function of its accuracy
• The boosting algorithm can be extended for the prediction of continuous
values
• Comparing with bagging: boosting tends to achieve greater accuracy, but it
also risks overfitting the model to misclassified data

Compiled by :Rashmi Bohra 147

Adaboost (Freund and Schapire,
1997)
• Given a set of d class-labeled tuples, (X1, y1), …, (Xd, yd)
• Initially, all the weights of tuples are set the same (1/d)
• Generate k classifiers in k rounds. At round i,
– Tuples from D are sampled (with replacement) to form a training set Di of the
same size
– Each tuple’s chance of being selected is based on its weight
– A classification model Mi is derived from Di
– Its error rate is calculated using Di as a test set
– If a tuple is misclssified, its weight is increased, o.w. it is decreased
• Error rate: err(Xj) is the misclassification error of tuple Xj. Classifier Mi
error rate is the sum of the weights of the misclassified tuples:
d
error ( M i ) = ∑ w j × err ( X j )
j
1 − error ( M i )
log
• The weight of classifier Mi’s vote is error ( M i )

Compiled by :Rashmi Bohra 148

 Chapter 6. Classification and
Prediction
• What is classification? What is • Support Vector Machines (SVM)

prediction? • Associative classification

• Issues regarding classification and • Lazy learners (or learning from your
neighbors)
prediction
• Other classification methods
• Classification by decision tree
• Prediction
induction
• Accuracy and error measures
• Bayesian classification
• Ensemble methods
• Rule-based classification • Model selection
• Classification by back propagation • Summary

Compiled by :Rashmi Bohra 149

Model Selection: ROC
Curves
• ROC (Receiver Operating Characteristics) curves:
for visual comparison of classification models
• Originated from signal detection theory
• Shows the trade-off between the true positive rate
and the false positive rate
• The area under the ROC curve is a measure of the
accuracy of the model • Vertical axis represents
• Rank the test tuples in decreasing order: the one the true positive rate
that is most likely to belong to the positive class
• Horizontal axis rep. the
appears at the top of the list
false positive rate
• The closer to the diagonal line (i.e., the closer the
area is to 0.5), the less accurate is the model • The plot also shows a
diagonal line
• A model with perfect
accuracy will have an
Compiled by :Rashmi Bohra
area of 1.0 150
 Summary (I)
• Classification and prediction are two forms of data analysis that can be used to extract
models describing important data classes or to predict future data trends.
• Effective and scalable methods have been developed for decision trees induction, Naive
Bayesian classification, Bayesian belief network, rule-based classifier, Backpropagation,
Support Vector Machine (SVM), associative classification, nearest neighbor classifiers,
and case-based reasoning, and other classification methods such as genetic algorithms,
rough set and fuzzy set approaches.
• Linear, nonlinear, and generalized linear models of regression can be used for prediction.
Many nonlinear problems can be converted to linear problems by performing
transformations on the predictor variables. Regression trees and model trees are also
used for prediction.

Compiled by :Rashmi Bohra 151

 Summary (II)
• Stratified k-fold cross-validation is a recommended method for accuracy estimation.
Bagging and boosting can be used to increase overall accuracy by learning and
combining a series of individual models.
• Significance tests and ROC curves are useful for model selection
• There have been numerous comparisons of the different classification and prediction
methods, and the matter remains a research topic
• No single method has been found to be superior over all others for all data sets
• Issues such as accuracy, training time, robustness, interpretability, and scalability must
be considered and can involve trade-offs, further complicating the quest for an overall
superior method

Compiled by :Rashmi Bohra 152

 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 153
 What is Cluster Analysis?
• Cluster: a collection of data objects
– Similar to one another within the same cluster
– Dissimilar to the objects in other clusters
• Cluster analysis
– Finding similarities between data according to the
characteristics found in the data and grouping similar data
objects into clusters
• Unsupervised learning: no predefined classes
• Typical applications
– As a stand-alone tool to get insight into data distribution
– As a preprocessing stepbyfor
Compiled other
:Rashmi algorithms
Bohra 154
 Clustering: Rich Applications
and Multidisciplinary Efforts
• Pattern Recognition
• Spatial Data Analysis
– Create thematic maps in GIS by clustering feature spaces
– Detect spatial clusters or for other spatial mining tasks
• Image Processing
• Economic Science (especially market research)
• WWW
– Document classification
– Cluster Weblog data to discover groups of similar access
patterns
Compiled by :Rashmi Bohra 155
 Examples of Clustering
Applications
• Marketing: Help marketers discover distinct groups in their customer
bases, and then use this knowledge to develop targeted marketing
programs
• Land use: Identification of areas of similar land use in an earth
observation database
• Insurance: Identifying groups of motor insurance policy holders with
a high average claim cost
• City-planning: Identifying groups of houses according to their house
type, value, and geographical location
• Earth-quake studies: Observed earth quake epicenters should be
clustered along continent faultsby :Rashmi Bohra
Compiled 156
 Quality: What Is Good
Clustering?
• A good clustering method will produce high quality clusters with
– high intra-class similarity
– low inter-class similarity

• The quality of a clustering result depends on both the similarity

measure used by the method and its implementation
• The quality of a clustering method is also measured by its ability to
discover some or all of the hidden patterns

Compiled by :Rashmi Bohra 157

 Measure the Quality of
Clustering
• Dissimilarity/Similarity metric: Similarity is expressed in terms of a
distance function, typically metric: d(i, j)
• There is a separate “quality” function that measures the “goodness” of a
cluster.
• The definitions of distance functions are usually very different for
interval-scaled, boolean, categorical, ordinal ratio, and vector variables.
• Weights should be associated with different variables based on
applications and data semantics.
• It is hard to define “similar enough” or “good enough”
– the answer is typically highly subjective.

Compiled by :Rashmi Bohra 158

Requirements of Clustering in Data
Mining
• Scalability
• Ability to deal with different types of attributes
• Ability to handle dynamic data
• Discovery of clusters with arbitrary shape
• Minimal requirements for domain knowledge to determine
input parameters
• Able to deal with noise and outliers
• Insensitive to order of input records
• High dimensionality
• Incorporation of user-specified constraints
• Interpretability and usability
Compiled by :Rashmi Bohra 159
 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 160
 Data Structures
• Data matrix  x11 ... x1f ... x1p 
 
– (two modes)  ... ... ... ... ... 
x ... xif ... xip 
 i1 
 ... ... ... ... ... 
x ... xnf ... xnp 
 n1 

 0 
 d(2,1) 
• Dissimilarity matrix  0 
 d(3,1 ) d ( 3,2) 0 
– (one mode)  
 : : : 
d ( n,1) d ( n,2) ... ... 0
Compiled by :Rashmi Bohra 161
 Type of data in clustering
analysis
• Interval-scaled variables
• Binary variables
• Nominal, ordinal, and ratio variables
• Variables of mixed types

Compiled by :Rashmi Bohra 162

 Interval-valued variables

• Standardize data
– Calculate the mean absolute deviation:
s f = 1n (| x1 f − m f | + | x2 f − m f | +...+ | xnf − m f |)

where
m f = 1n (x1 f + x2 f + ... + xnf )
.

– Calculate the standardized measurement (z-score)

xif − m f
zif =
• Using mean absolute deviation s fis more robust than using
standard deviation

Compiled by :Rashmi Bohra 163

 Similarity and Dissimilarity
Between Objects

• Distances are normally used to measure the similarity or

dissimilarity between two data objects
• Some popular ones include: Minkowski distance:
d (i, j) = q (| x − x |q + | x − x | q +...+ | x − x |q )
i1 j1 i2 j2 ip jp
where i = (xi1 , xi2 , …, xip ) and j = (xj1 , xj2 , …, xjp ) are two p-
dimensional data objects, and q is a positive integer

• If q = 1, d is Manhattan distance

d (i, j) =| x − x | + | x − x | + ...+ | x − x |
i1 j1 i2 j 2 ip jp
Compiled by :Rashmi Bohra 164
 Similarity and Dissimilarity
Between Objects (Cont.)
• If q = 2, d is Euclidean distance:
d (i, j) = (| x − x | 2 + | x − x |2 +...+ | x − x | 2 )
i1 j1 i2 j2 ip jp
– Properties
• d(i,j) ≥ 0
• d(i,i) = 0
• d(i,j) = d(j,i)
• d(i,j) ≤ d(i,k) + d(k,j)
• Also, one can use weighted distance,
parametric Pearson product moment correlation,
or other disimilarity
Compiledmeasures
by :Rashmi Bohra 165
 Binary Variables Object j
1 0 sum
• A contingency table for binary data
1 a b a +b
Object i
0 c d c+d
• Distance measure for symmetric binary sum a + c b + d p
variables:

• Distance measure for asymmetric binary d (i, j) = b +c

variables: a +b +c +d
• Jaccard coefficient (similarity measure
for asymmetric binary variables): d (i, j) = b +c
a +b +c

sim Jaccard (i, j) = a

a +b +c
Compiled by :Rashmi Bohra 166
 Dissimilarity between Binary
Variables
• Example
Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4
Jack M Y N P N N N
Mary F Y N P N P N
Jim M Y P N N N N

– gender is a symmetric attribute

– the remaining attributes are asymmetric binary
0 +1
– let thed values
( jack ,Y
maand )=
ry P =0.the
be set to 1, and 33 value N be set to 0
2 +0 +1
1 +1
d ( jack , jim ) = =0.67
1 +1 +1
1 +2
d ( jim , mary ) = =0.75
1+
Compiled by +2 Bohra
1 :Rashmi 167
 Nominal Variables

• A generalization of the binary variable in that it can take more

than 2 states, e.g., red, yellow, blue, green
• Method 1: Simple matching
– m: # of matches, p: total # of variables

d (i, j) = p −
p
m

• Method 2: use a large number of binary variables

– creating a new binary variable for each of the M nominal states

Compiled by :Rashmi Bohra 168

 Ordinal Variables

• An ordinal variable can be discrete or continuous

• Order is important, e.g., rank
• Can be treated like interval-scaled
rif ∈{1,..., M f }
– replace xif by their rank
– map the range of each variable onto [0, 1] by replacing i-th
object in the f-th variable by
rif −1
zif =
Mf − 1

– compute the dissimilarity using methods for interval-scaled

variables
Compiled by :Rashmi Bohra 169
 Ratio-Scaled Variables

• Ratio-scaled variable: a positive measurement on a

nonlinear scale, approximately at exponential scale,
such as AeBt or Ae-Bt
• Methods:
– treat them like interval-scaled variables—not a good choice!
(why?—the scale can be distorted)
– apply logarithmic transformation
yif = log(xif )
– treat them as continuous ordinal data treat their rank as interval-
scaled
Compiled by :Rashmi Bohra 170
 Variables of Mixed Types

• A database may contain all the six types of variables

– symmetric binary, asymmetric binary, nominal, ordinal, interval and
ratio
• One may use a weighted formula to combine their effects
Σpf =1δij( f ) d ij( f )
d (i, j ) =
–f is binary or nominal: Σpf =1δij( f )
dij (f) = 0 if xif = xjf , or dij (f) = 1 otherwise
– f is interval-based: use the normalized distance
– f is ordinal or ratio-scaled
• compute ranks rif and
• and treat zif as interval-scaled
zif =
r
if −1
M f −1
Compiled by :Rashmi Bohra 171
 Vector Objects
• Vector objects: keywords in documents, gene features in
micro-arrays, etc.
• Broad applications: information retrieval, biologic
taxonomy, etc.
• Cosine measure

• A variant: Tanimoto coefficient

Compiled by :Rashmi Bohra 172

 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 173
 Major Clustering
Approaches (I)
• Partitioning approach:
– Construct various partitions and then evaluate them by some criterion, e.g.,
minimizing the sum of square errors
– Typical methods: k-means, k-medoids, CLARANS
• Hierarchical approach:
– Create a hierarchical decomposition of the set of data (or objects) using some
criterion
– Typical methods: Diana, Agnes, BIRCH, ROCK, CAMELEON
• Density-based approach:
– Based on connectivity and density functions
– Typical methods: DBSACN, OPTICS, DenClue

Compiled by :Rashmi Bohra 174

 Major Clustering
Approaches (II)
• Grid-based approach:
– based on a multiple-level granularity structure
– Typical methods: STING, WaveCluster, CLIQUE
• Model-based:
– A model is hypothesized for each of the clusters and tries to find the best fit of
that model to each other
– Typical methods: EM, SOM, COBWEB
• Frequent pattern-based:
– Based on the analysis of frequent patterns
– Typical methods: pCluster
• User-guided or constraint-based:
– Clustering by considering user-specified or application-specific constraints
– Typical methods: COD (obstacles), constrained
Compiled by :Rashmi Bohra clustering 175
 Typical Alternatives to Calculate the
Distance between Clusters
• Single link: smallest distance between an element in one cluster and an element in
the other, i.e., dis(Ki, Kj) = min(tip , tjq )

• Complete link: largest distance between an element in one cluster and an element
in the other, i.e., dis(Ki, Kj) = max(tip , tjq )

• Average: avg distance between an element in one cluster and an element in the
other, i.e., dis(Ki, Kj) = avg(tip , tjq )

• Centroid: distance between the centroids of two clusters, i.e., dis(Ki, Kj) = dis(Ci, Cj)
• Medoid: distance between the medoids of two clusters, i.e., dis(Ki, Kj) = dis(Mi, Mj)
– Medoid: one chosen, centrally located object in the cluster

Compiled by :Rashmi Bohra 176

Centroid, Radius and Diameter of a
Cluster (for numerical data sets)
• Centroid: the “middle” of a cluster ΣiN= 1(t )
C m=
ip

• Radius: square root of average

N distance from
2 any point
Σ (t − cm )
Rm = i =1 ip
of the cluster to its centroid
N

Σ N Σ N (t −t ) 2
• Diameter: square root of average
i =1 i mean
=1 ip squared
iq
Dm =
N ( N −1)
distance between all pairs of points in the cluster

Compiled by :Rashmi Bohra 177

 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 178
 Partitioning Algorithms: Basic
Concept
• Partitioning method: Construct a partition of a database D of n
objects into a set of k clusters, s.t., min sum of squared distance

Σkm=1Σtmi ∈Km (Cm − tmi ) 2

• Given a k, find a partition of k clusters that optimizes the chosen

partitioning criterion
– Global optimal: exhaustively enumerate all partitions
– Heuristic methods: k-means and k-medoids algorithms
– k-means (MacQueen’67): Each cluster is represented by the center of the
cluster
– k-medoids or PAM (Partition around medoids) (Kaufman & Rousseeuw’87):
Each cluster is represented by one of the objects in the cluster

Compiled by :Rashmi Bohra 179

 The K-Means Clustering Method
• Given k, the k-means algorithm is implemented in
four steps:
– Partition objects into k nonempty subsets
– Compute seed points as the centroids of the clusters of the
current partition (the centroid is the center, i.e., mean point,
of the cluster)
– Assign each object to the cluster with the nearest seed
point
– Go back to Step 2, stop when no more new assignment

Compiled by :Rashmi Bohra 180

 The K-Means Clustering Method
• Example
10
10
10 9
9
9 8
8
8 7

6 7
7
5 6
6
4 5
5 3
4
4
Assign
2
Update 3
3 1

2 each 0 the 2

cluster
0 1 2 3 4 5 6 7 8 9 10
1
1
objects 0
0
0 1 2 3 4 5 6 7 8 9 10 to means 0 1 2 3 4 5 6 7 8 9 10

most
similar reassign reassign
center 10 10

K=2
9 9

8 8

7 7

Arbitrarily choose 6 6

K object as initial
5 5

4 4

cluster center 3

2
Update 3

1 the 1

cluster
0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10

means
Compiled by :Rashmi Bohra 181
 Comments on the K-Means
Method
• Strength: Relatively efficient: O(tkn), where n is # objects, k is #
clusters, and t is # iterations. Normally, k, t << n.
• Comparing: PAM: O(k(n-k)2 ), CLARA: O(ks2 + k(n-k))
• Comment: Often terminates at a local optimum. The global
optimum may be found using techniques such as: deterministic
annealing and genetic algorithms
• Weakness
– Applicable only when mean is defined, then what about categorical data?
– Need to specify k, the number of clusters, in advance
– Unable to handle noisy data and outliers
– Not suitable to discover clusters with non-convex shapes

Compiled by :Rashmi Bohra 182

 Variations of the K-Means Method
• A few variants of the k-means which differ in
– Selection of the initial k means

– Dissimilarity calculations

– Strategies to calculate cluster means

• Handling categorical data: k-modes (Huang’98)

– Replacing means of clusters with modes

– Using new dissimilarity measures to deal with categorical objects

– Using a frequency-based method to update modes of clusters

– A mixture of categorical and numerical data: k-prototype method

Compiled by :Rashmi Bohra 183
What Is the Problem of the K-Means
Method?
• The k-means algorithm is sensitive to outliers !
– Since an object with an extremely large value may substantially
distort the distribution of the data.

• K-Medoids: Instead of taking the mean value of the object

in a cluster as a reference point, medoids can be used,
which is the most centrally located object in a cluster.
10 10
9 9
8 8
7 7
6 6
5 5
4 4
3 3
2 2
1 1
0 0
0 1 2 3 4 5 6
Compiled
78 9 10
by :Rashmi Bohra
0 1 2 3 4 5 6 7 8 9 10
184
 The K-Medoids Clustering Method
• Find representative objects, called medoids, in clusters
• PAM (Partitioning Around Medoids, 1987)
– starts from an initial set of medoids and iteratively replaces one of
the medoids by one of the non-medoids if it improves the total
distance of the resulting clustering
– PAM works effectively for small data sets, but does not scale well
for large data sets

• CLARA (Kaufmann & Rousseeuw, 1990)

• CLARANS (Ng & Han, 1994): Randomized sampling
• Focusing + spatial data structure (Ester et al., 1995)
Compiled by :Rashmi Bohra 185
 A Typical K-Medoids Algorithm (PAM)
Total Cost = 20
10 10
10
9 9
9
8 8
8
7 7

7
Arbitrar Assign
6 6

6 5 5

5
y 4 each 4

4 choose 3
remaini 3

2 2
3
k object 1
ng 1
2
as 0
0 1 2 3 4 5 6 7 8 9 10
object 0
0 1 2 3 4 5 6 7 8 9 10

initial to
1

0
0 1 2 3 4 5 6 7 8 9 10
medoid nearest
s medoid
K=2 s Randomly select a
Total Cost = 26 nonmedoid
object,Oramdom
10 10

Do loop 9

8 Compute
9

8
Swapping 7 total cost 7

Until no O and 6
of 6

Oramdom
change
5 5

4
swapping 4

If quality is 3

2
3

improved. 1 1

Compiled by :Rashmi Bohra

0
0 1 2 3 4 5 6 7 8 9 10
0
0 1 2 3 4 5 6
186
7 8 9 10
PAM (Partitioning Around Medoids)
(1987)

• PAM (Kaufman and Rousseeuw, 1987), built in Splus

• Use real object to represent the cluster
– Select k representative objects arbitrarily
– For each pair of non-selected object h and selected object i,
calculate the total swapping cost TCih
– For each pair of i and h,
• If TCih < 0, i is replaced by h
• Then assign each non-selected object to the most similar
representative object
– repeat steps 2-3 until there is no change
Compiled by :Rashmi Bohra 187
PAM Clustering: Total swapping cost
TCih =∑jCjih
10 10

9 9

j
t
8 8

7
t 7

6 6

4
j 5

h
3 3

2 i h 2

1 1 i
0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10

Cjih = d(j, h) - d(j, i) Cjih = 0

10
10

9
9
8
8

7 h 7

5
j 6

4
4
i
i 3

h j
t
3
2
2

t
1
0
0
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10

Compiled by :Rashmi Bohra 188

Cjih = d(j, t) - d(j, i) Cjih = d(j, h) - d(j, t)
 What Is the Problem with
PAM?
• Pam is more robust than k-means in the presence of noise
and outliers because a medoid is less influenced by outliers
or other extreme values than a mean
• Pam works efficiently for small data sets but does not scale
well for large data sets.
– O(k(n-k)2 ) for each iteration

where n is # of data,k is # of clusters

Sampling based method,
CLARA(Clustering LARge Applications)

Compiled by :Rashmi Bohra 189

 CLARA (Clustering Large
Applications) (1990)
• CLARA (Kaufmann and Rousseeuw in 1990)
– Built in statistical analysis packages, such as S+
• It draws multiple samples of the data set, applies PAM on each
sample, and gives the best clustering as the output
• Strength: deals with larger data sets than PAM
• Weakness:
– Efficiency depends on the sample size
– A good clustering based on samples will not necessarily represent a
good clustering of the whole data set if the sample is biased

Compiled by :Rashmi Bohra 190

 CLARANS (“Randomized” CLARA)
(1994)
• CLARANS (A Clustering Algorithm based on Randomized Search)
(Ng and Han’94)
• CLARANS draws sample of neighbors dynamically
• The clustering process can be presented as searching a graph where
every node is a potential solution, that is, a set of k medoids
• If the local optimum is found, CLARANS starts with new randomly
selected node in search for a new local optimum
• It is more efficient and scalable than both PAM and CLARA
• Focusing techniques and spatial access structures may further
improve its performance (Ester et al.’95)

Compiled by :Rashmi Bohra 191

 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 192
 Hierarchical Clustering
• Use distance matrix as clustering criteria. This
method does not require the number of clusters
k as an input, but needs a termination condition
Step 0 Step 1 Step 2 Step 3 Step 4
agglomerative
(AGNES)
a ab
b abcde
c
cde
d
de
e
divisive
Step 4 Step 3 StepCompiled
2 Stepby1 :Rashmi
Step 0Bohra (DIANA) 193
 AGNES (Agglomerative
Nesting)
• Introduced in Kaufmann and Rousseeuw (1990)
• Implemented in statistical analysis packages, e.g.,
Splus
• Use the Single-Link method and the dissimilarity
matrix.
• Merge nodes that have the least dissimilarity
• Go on in a non-descending fashion
10 10 10

9 9 9

8 8 8

7 7 7

• Eventually all nodes belong to the same cluster

6 6 6

5 5 5

4 4 4

3 3 3

2 2 2

1 1 1

0 0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10

Compiled by :Rashmi Bohra 194

Dendrogram: Shows How the Clusters are Merged

Decompose data objects into a several levels of nested

partitioning (tree of clusters), called a dendrogram.

A clustering of the data objects is obtained by cutting the

dendrogram at the desired level, then each connected
component forms a cluster.

Compiled by :Rashmi Bohra 195

 DIANA (Divisive Analysis)

• Introduced in Kaufmann and Rousseeuw (1990)

• Implemented in statistical analysis packages, e.g., Splus
• Inverse order of AGNES
• Eventually each node forms a cluster on its own

10 10
10

9 9
9
8 8
8

7 7
7
6 6
6

5 5
5
4 4
4

3 3
3
2 2
2

1 1
1
0 0
0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10

Compiled by :Rashmi Bohra 196

 Recent Hierarchical Clustering
Methods
• Major weakness of agglomerative clustering methods
– do not scale well: time complexity of at least O(n2), where n is the
number of total objects
– can never undo what was done previously
• Integration of hierarchical with distance-based clustering
– BIRCH (1996): uses CF-tree and incrementally adjusts the quality of
sub-clusters
– ROCK (1999): clustering categorical data by neighbor and link
analysis
– CHAMELEON (1999): hierarchical clustering using dynamic modeling

Compiled by :Rashmi Bohra 197

 BIRCH (1996)
• Birch: Balanced Iterative Reducing and Clustering using
Hierarchies (Zhang, Ramakrishnan & Livny, SIGMOD’96)
• Incrementally construct a CF (Clustering Feature) tree, a
hierarchical data structure for multiphase clustering
– Phase 1: scan DB to build an initial in-memory CF tree (a multi-level
compression of the data that tries to preserve the inherent
clustering structure of the data)
– Phase 2: use an arbitrary clustering algorithm to cluster the leaf
nodes of the CF-tree

• Scales linearly: finds a good clustering with a single scan

and improves the quality with a few additional scans
• Weakness: handles only numeric data, and sensitive to the
order of the data record.
Compiled by :Rashmi Bohra 198
 Clustering Feature Vector in
BIRCH
Clustering Feature: CF = (N, LS, SS)
N: Number of data points
LS: ∑Ni=1 =Xi
SS: ∑Ni=1 =Xi2 CF = (5, (16,30),(54,190))
10

8
(3,4)
7

5
(2,6)
(4,5)
4

(4,7)
1

0
0 1 2 3 4 5 6 7 8 9 10

(3,8)
Compiled by :Rashmi Bohra 199
 CF-Tree in BIRCH
• Clustering feature:
– summary of the statistics for a given subcluster: the 0-th, 1st and 2nd
moments of the subcluster from the statistical point of view.
– registers crucial measurements for computing cluster and utilizes
storage efficiently

A CF tree is a height-balanced tree that stores the clustering

features for a hierarchical clustering
– A nonleaf node in a tree has descendants or “children”
– The nonleaf nodes store sums of the CFs of their children

• A CF tree has two parameters

– Branching factor: specify the maximum number of children.
– threshold: max diameter of sub-clusters stored at the leaf nodes
Compiled by :Rashmi Bohra 200
 The CF Tree Structure
Root

B=7 CF1 CF2 CF3 CF6

L=6 child1 child2 child3 child6

Non-leaf node
CF1 CF2 CF3 CF5
child1 child2 child3 child5

Leaf node Leaf node

prev CF1 CF2 CF6 next prev CF1 CF2 CF4 next

Compiled by :Rashmi Bohra 201

 Clustering Categorical Data: The
ROCK Algorithm
• ROCK: RObust Clustering using linKs
– S. Guha, R. Rastogi & K. Shim, ICDE’99
• Major ideas
– Use links to measure similarity/proximity
– Not distance-based
– Computational complexity: O ( n 2
+ nmmma + n 2 log n)
• Algorithm: sampling-based clustering
– Draw random sample
– Cluster with links
– Label data in disk
• Experiments
– Congressional voting, mushroom data
Compiled by :Rashmi Bohra 202
 Similarity Measure in ROCK
• Traditional measures for categorical data may not work
well, e.g., Jaccard coefficient
• Example: Two groups (clusters) of transactions
– C1. <a, b, c, d, e>: {a, b, c}, {a, b, d}, {a, b, e}, {a, c, d}, {a, c, e},
{a, d, e}, {b, c, d}, {b, c, e}, {b, d, e}, {c, d, e}
– C2. <a, b, f, g>: {a, b, f}, {a, b, g}, {a, f, g}, {b, f, g}
• Jaccard co-efficient may lead to wrong clustering result
– C1: 0.2 ({a, b, c}, {b, d, e}} to 0.5 ({a, b, c}, {a, b, d})
– C1 & C2: could be as high as 0.5 ({a, b, c}, {a, b, f}) T1 ∩T2
Sim ( T1 , T2 ) =
• Jaccard co-efficient-based similarity function: T1 ∪T2

{c} 1
– Ex. LetST
im1 =({a,
T 1,b,
T c}, T2 = {c, d, e}
2) = = =0.2
{a, b, c, d , e} 5
Compiled by :Rashmi Bohra 203
 Link Measure in ROCK
• Links: # of common neighbors
– C1 <a, b, c, d, e>: {a, b, c}, {a, b, d}, {a, b, e}, {a, c, d}, {a, c, e}, {a, d, e},
{b, c, d}, {b, c, e}, {b, d, e}, {c, d, e}
– C2 <a, b, f, g>: {a, b, f}, {a, b, g}, {a, f, g}, {b, f, g}

• Let T1 = {a, b, c}, T2 = {c, d, e}, T3 = {a, b, f}

– link(T1, T2) = 4, since they have 4 common neighbors
• {a, c, d}, {a, c, e}, {b, c, d}, {b, c, e}

– link(T1, T3) = 3, since they have 3 common neighbors

• {a, b, d}, {a, b, e}, {a, b, g}

• Thus link is a better measure than Jaccard coefficient

Compiled by :Rashmi Bohra 204
 CHAMELEON: Hierarchical
Clustering Using Dynamic Modeling
(1999)
• CHAMELEON: by G. Karypis, E.H. Han, and V. Kumar’99
• Measures the similarity based on a dynamic model
– Two clusters are merged only if the interconnectivity and closeness
(proximity) between two clusters are high relative to the internal
interconnectivity of the clusters and closeness of items within the
clusters
– Cure ignores information about interconnectivity of the objects,
Rock ignores information about the closeness of two clusters
• A two-phase algorithm
1. Use a graph partitioning algorithm: cluster objects into a large number
of relatively small sub-clusters
2. Use an agglomerative hierarchical clustering algorithm: find the
genuine clusters by repeatedly combining
Compiled by :Rashmi Bohra these sub-clusters 205
 Overall Framework of
CHAMELEON
Construct
Sparse Graph Partition the Graph

Data Set

Merge Partition

Final Clusters

Compiled by :Rashmi Bohra 206

CHAMELEON (Clustering Complex
Objects)

Compiled by :Rashmi Bohra 207

 Chapter 7. Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 208
 Density-Based Clustering
Methods
• Clustering based on density (local cluster criterion),
such as density-connected points
• Major features:
– Discover clusters of arbitrary shape
– Handle noise
– One scan
– Need density parameters as termination condition
• Several interesting studies:
– DBSCAN: Ester, et al. (KDD’96)
– OPTICS: Ankerst, et al (SIGMOD’99).
– DENCLUE: Hinneburg & D. Keim (KDD’98)
– CLIQUE: Agrawal, et al.by(SIGMOD’98)
Compiled :Rashmi Bohra (more grid-based)
209
 Density-Based Clustering: Basic
Concepts
• Two parameters:
– Eps: Maximum radius of the neighbourhood
– MinPts: Minimum number of points in an Eps-neighbourhood
of that point

• NEps (p): {q belongs to D | dist(p,q) <= Eps}

• Directly density-reachable: A point p is directly density-
reachable from a point q w.r.t. Eps, MinPts if
– p belongs to NEps (q)
p MinPts = 5
– core point condition: q
Eps = 1 cm
|NEps (q)| >= MinPts
Compiled by :Rashmi Bohra 210
 Density-Reachable and Density-
Connected
• Density-reachable:
– A point p is density-reachable from a p
point q w.r.t. Eps, MinPts if there is a
p1
chain of points p1, …, pn, p1 = q, pn = q
p such that pi+1 is directly density-
reachable from pi

• Density-connected
– A point p is density-connected to a p q
point q w.r.t. Eps, MinPts if there is a
point o such that both, p and q are o
density-reachable from o w.r.t. Eps
and MinPts Compiled by :Rashmi Bohra 211
 DBSCAN: Density Based Spatial
Clustering of Applications with Noise
• Relies on a density-based notion of cluster: A
cluster is defined as a maximal set of density-
connected points
• Discovers clusters of arbitrary shape in spatial
databases with noise
Outlier

Border
Eps = 1cm
Core MinPts = 5

Compiled by :Rashmi Bohra 212

 DBSCAN: The Algorithm
• Arbitrary select a point p

• Retrieve all points density-reachable from p w.r.t. Eps

and MinPts.

• If p is a core point, a cluster is formed.

• If p is a border point, no points are density-reachable

from p and DBSCAN visits the next point of the
database.

• Continue the process until all of the points have been

processed. Compiled by :Rashmi Bohra 213
DBSCAN: Sensitive to
Parameters

Compiled by :Rashmi Bohra 214

CHAMELEON (Clustering Complex
Objects)

Compiled by :Rashmi Bohra 215

 OPTICS: A Cluster-Ordering
Method (1999)
• OPTICS: Ordering Points To Identify the
Clustering Structure
– Ankerst, Breunig, Kriegel, and Sander (SIGMOD’99)
– Produces a special order of the database wrt its
density-based clustering structure
– This cluster-ordering contains info equiv to the density-
based clusterings corresponding to a broad range of
parameter settings
– Good for both automatic and interactive cluster
analysis, including finding intrinsic clustering structure
– Can be represented graphically or using visualization
techniques Compiled by :Rashmi Bohra 216
 OPTICS: Some Extension from
DBSCAN
• Index-based:
• k = number of dimensions
• N = 20
• p = 75% D
• M = N(1-p) = 5
– Complexity: O(kN2)
• Core Distance p1

o
• Reachability Distance
p2 o
Max (core-distance (o), d (o, p))
MinPts = 5
r(p1, o) = 2.8cm. r(p2,o)Compiled
= 4cmby :Rashmi Bohra 217
ε = 3 cm
 Reachability
-distance

undefined
ε
ε
ε ‘

Cluster-order
Compiled by :Rashmi Bohra 218
of the objects
Density-Based Clustering: OPTICS & Its
Applications

Compiled by :Rashmi Bohra 219

 DENCLUE: Using Statistical
Density Functions
• DENsity-based CLUstEring by Hinneburg & Keim (KDD’98)
2
d ( x,y)
−
• Using statistical density functions: f Gaussian( x , y ) = e 2σ 2

d ( x , xi ) 2
−
( x ) =∑
N
D 2σ2
f Gaussian i =1
e
d ( x , xi ) 2
−
( x, xi ) = ∑i =1 ( xi − x) ⋅ e
N
∇f D
Gaussian
2σ 2

• Major features
– Solid mathematical foundation
– Good for data sets with large amounts of noise
– Allows a compact mathematical description of arbitrarily shaped clusters in high-
dimensional data sets
– Significant faster than existing algorithm (e.g., DBSCAN)
– But needs a large number of parameters

Compiled by :Rashmi Bohra 220

 Denclue: Technical Essence
• Uses grid cells but only keeps information about grid cells that
do actually contain data points and manages these cells in a
tree-based access structure
• Influence function: describes the impact of a data point within
its neighborhood
• Overall density of the data space can be calculated as the sum
of the influence function of all data points
• Clusters can be determined mathematically by identifying
density attractors
• Density attractors are local maximal of the overall density
function

Compiled by :Rashmi Bohra 221

Density Attractor

Compiled by :Rashmi Bohra 222

Center-Defined and Arbitrary

Compiled by :Rashmi Bohra 223

 Chapter 7. Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 224
 Grid-Based Clustering Method
• Using multi-resolution grid data structure
• Several interesting methods
– STING (a STatistical INformation Grid approach) by Wang, Yang and
Muntz (1997)
– WaveCluster by Sheikholeslami, Chatterjee, and Zhang (VLDB’98)
• A multi-resolution clustering approach using wavelet method
– CLIQUE: Agrawal, et al. (SIGMOD’98)
• On high-dimensional data (thus put in the section of clustering high-
dimensional data

Compiled by :Rashmi Bohra 225

STING: A Statistical Information Grid
Approach
• Wang, Yang and Muntz (VLDB’97)
• The spatial area area is divided into rectangular
cells
• There are several levels of cells corresponding to
different levels of resolution

Compiled by :Rashmi Bohra 226

 The STING Clustering
Method
– Each cell at a high level is partitioned into a number of smaller
cells in the next lower level
– Statistical info of each cell is calculated and stored
beforehand and is used to answer queries
– Parameters of higher level cells can be easily calculated from
parameters of lower level cell
• count, mean, s, min, max
• type of distribution—normal, uniform, etc.
– Use a top-down approach to answer spatial data queries
– Start from a pre-selected layer—typically with a small number
of cells
– For each cell in the current level compute the confidence
interval
Compiled by :Rashmi Bohra 227
 Comments on STING

• Remove the irrelevant cells from further consideration

• When finish examining the current layer, proceed to the next lower level
• Repeat this process until the bottom layer is reached
• Advantages:
– Query-independent, easy to parallelize, incremental update
– O(K), where K is the number of grid cells at the lowest level
• Disadvantages:
– All the cluster boundaries are either horizontal or vertical, and no diagonal
boundary is detected

Compiled by :Rashmi Bohra 228

 WaveCluster: Clustering by Wavelet
Analysis (1998)
• Sheikholeslami, Chatterjee, and Zhang (VLDB’98)
• A multi-resolution clustering approach which applies wavelet
transform to the feature space
• How to apply wavelet transform to find clusters
– Summarizes the data by imposing a multidimensional grid structure
onto data space
– These multidimensional spatial data objects are represented in a n-
dimensional feature space
– Apply wavelet transform on feature space to find the dense regions in
the feature space
– Apply wavelet transform multiple times which result in clusters at
different scales from fine to coarse
Compiled by :Rashmi Bohra 229
 Wavelet Transform
• Wavelet transform: A signal processing technique that decomposes a signal into different
frequency sub-band (can be applied to n-dimensional signals)
• Data are transformed to preserve relative distance between objects at different levels of resolution
• Allows natural clusters to become more distinguishable

Compiled by :Rashmi Bohra 230

 The WaveCluster Algorithm
• Input parameters
– # of grid cells for each dimension
– the wavelet, and the # of applications of wavelet transform
• Why is wavelet transformation useful for clustering?
– Use hat-shape filters to emphasize region where points cluster, but
simultaneously suppress weaker information in their boundary
– Effective removal of outliers, multi-resolution, cost effective
• Major features:
– Complexity O(N)
– Detect arbitrary shaped clusters at different scales
– Not sensitive to noise, not sensitive to input order
– Only applicable to low dimensional data
• Both grid-based and density-based
Compiled by :Rashmi Bohra 231
 Quantization
& Transformation
• First, quantize data into m-D grid
structure, then wavelet transform
– a) scale 1: high resolution
– b) scale 2: medium resolution
– c) scale 3: low resolution

Compiled by :Rashmi Bohra 232

 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 233
 Model-Based Clustering
• What is model-based clustering?
– Attempt to optimize the fit between the given data and
some mathematical model
– Based on the assumption: Data are generated by a
mixture of underlying probability distribution
• Typical methods
– Statistical approach
• EM (Expectation maximization), AutoClass
– Machine learning approach
• COBWEB, CLASSIT
– Neural network approach
• SOM (Self-Organizing Feature
Compiled Map)
by :Rashmi Bohra 234
 EM — Expectation Maximization
• EM — A popular iterative refinement algorithm
• An extension to k-means
– Assign each object to a cluster according to a weight (prob.
distribution)
– New means are computed based on weighted measures
• General idea
– Starts with an initial estimate of the parameter vector
– Iteratively rescores the patterns against the mixture density produced
by the parameter vector
– The rescored patterns are used to update the parameter updates
– Patterns belonging to the same cluster, if they are placed by their
scores in a particular component
• Algorithm converges Compiled
fast but may not be in global optima
by :Rashmi Bohra 235
The EM (Expectation Maximization)
Algorithm
• Initially, randomly assign k cluster centers
• Iteratively refine the clusters based on two steps
– Expectation step: assign each data point Xi to cluster Ci
with the following probability

– Maximization step:
• Estimation of model parameters

Compiled by :Rashmi Bohra 236

 Conceptual Clustering
• Conceptual clustering
– A form of clustering in machine learning
– Produces a classification scheme for a set of unlabeled
objects
– Finds characteristic description for each concept (class)
• COBWEB (Fisher’87)
– A popular a simple method of incremental conceptual
learning
– Creates a hierarchical clustering in the form of a
classification tree
– Each node refers to a concept and contains a
probabilistic description of that concept
Compiled by :Rashmi Bohra 237
 COBWEB Clustering
Method
A classification tree

Compiled by :Rashmi Bohra 238

 More on Conceptual Clustering
• Limitations of COBWEB
– The assumption that the attributes are independent of each other is often
too strong because correlation may exist
– Not suitable for clustering large database data – skewed tree and
expensive probability distributions
• CLASSIT
– an extension of COBWEB for incremental clustering of continuous data
– suffers similar problems as COBWEB
• AutoClass (Cheeseman and Stutz, 1996)
– Uses Bayesian statistical analysis to estimate the number of clusters
– Popular in industry

Compiled by :Rashmi Bohra 239

 Neural Network Approach

• Neural network approaches

– Represent each cluster as an exemplar, acting as a
“prototype” of the cluster
– New objects are distributed to the cluster whose exemplar
is the most similar according to some distance measure
• Typical methods
– SOM (Soft-Organizing feature Map)
– Competitive learning
• Involves a hierarchical architecture of several units (neurons)
• Neurons compete in a “winner-takes-all” fashion for the object
currently being presented

Compiled by :Rashmi Bohra 240

 Self-Organizing Feature Map
(SOM)
• SOMs, also called topological ordered maps, or Kohonen Self-Organizing
Feature Map (KSOMs)
• It maps all the points in a high-dimensional source space into a 2 to 3-d
target space, s.t., the distance and proximity relationship (i.e., topology)
are preserved as much as possible
• Similar to k-means: cluster centers tend to lie in a low-dimensional
manifold in the feature space
• Clustering is performed by having several units competing for the current
object
– The unit whose weight vector is closest to the current object wins
– The winner and its neighbors learn by having their weights adjusted
• SOMs are believed to resemble processing that can occur in the brain
• Useful for visualizing high-dimensional data in 2- or 3-D space
Compiled by :Rashmi Bohra 241
 Web Document Clustering
Using SOM
• The result of SOM
clustering of 12088
Web articles
• The picture on the
right: drilling down
on the keyword
“mining”
• Based on
websom.hut.fi Web
page

Compiled by :Rashmi Bohra 242

 Chapter 6. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 243
 Clustering High-Dimensional
Data
• Clustering high-dimensional data
– Many applications: text documents, DNA micro-array data
– Major challenges:
• Many irrelevant dimensions may mask clusters
• Distance measure becomes meaningless—due to equi-distance
• Clusters may exist only in some subspaces
• Methods
– Feature transformation: only effective if most dimensions are relevant
• PCA & SVD useful only when features are highly correlated/redundant
– Feature selection: wrapper or filter approaches
• useful to find a subspace where the data have nice clusters
– Subspace-clustering: find clusters in all the possible subspaces
• CLIQUE, ProClus, and frequent pattern-based clustering
Compiled by :Rashmi Bohra 244
The Curse of Dimensionality
(graphs adapted from Parsons et al. KDD
Explorations 2004)
• Data in only one dimension is
relatively packed
• Adding a dimension “stretch” the
points across that dimension, making
them further apart
• Adding more dimensions will make
the points further apart—high
dimensional data is extremely sparse
• Distance measure becomes
meaningless—due Compiled
to equi-distance
by :Rashmi Bohra 245
 Why Subspace
Clustering?
(adapted from Parsons et al. SIGKDD
• Explorations
Clusters may exist only in some2004)
subspaces
• Subspace-clustering: find clusters in all the subspaces

Compiled by :Rashmi Bohra 246

 CLIQUE (Clustering In QUEst)
• Agrawal, Gehrke, Gunopulos, Raghavan (SIGMOD’98)
• Automatically identifying subspaces of a high dimensional data space
that allow better clustering than original space
• CLIQUE can be considered as both density-based and grid-based
– It partitions each dimension into the same number of equal length interval
– It partitions an m-dimensional data space into non-overlapping rectangular units
– A unit is dense if the fraction of total data points contained in the unit exceeds
the input model parameter
– A cluster is a maximal set of connected dense units within a subspace

Compiled by :Rashmi Bohra 247

 CLIQUE: The Major Steps
• Partition the data space and find the number of points
that lie inside each cell of the partition.
• Identify the subspaces that contain clusters using the
Apriori principle
• Identify clusters
– Determine dense units in all subspaces of interests
– Determine connected dense units in all subspaces of interests.
• Generate minimal description for the clusters
– Determine maximal regions that cover a cluster of connected
dense units for each cluster
– Determination of minimal cover for each cluster

Compiled by :Rashmi Bohra 248

 Vacation
(10,000)

(week)
Salary

0 1 2 3 4 5 6 7
0 1 2 3 4 5 6 7

age age
20 30 40 50 60 20 30 40 50 60

τ =3
Vacation

ry 30 50
a la age
S

Compiled by :Rashmi Bohra 249

 Strength and Weakness of
CLIQUE
• Strength
– automatically finds subspaces of the highest
dimensionality such that high density clusters exist in
those subspaces
– insensitive to the order of records in input and does not
presume some canonical data distribution
– scales linearly with the size of input and has good
scalability as the number of dimensions in the data
increases
• Weakness
– The accuracy of the clustering result may be degraded
at the expense of simplicity of the method
Compiled by :Rashmi Bohra 250
 Frequent Pattern-Based
Approach
• Clustering high-dimensional space (e.g., clustering text
documents, microarray data)
– Projected subspace-clustering: which dimensions to be projected on?
• CLIQUE, ProClus
– Feature extraction: costly and may not be effective?
– Using frequent patterns as “features”
• “Frequent” are inherent features
• Mining freq. patterns may not be so expensive

• Typical methods
– Frequent-term-based document clustering
– Clustering by pattern similarity in micro-array data (pClustering)
Compiled by :Rashmi Bohra 251
 Clustering by Pattern Similarity (p-
Clustering)
• Right: The micro-array “raw” data shows 3 genes
and their values in a multi-dimensional space
– Difficult to find their patterns

• Bottom: Some subsets of dimensions form nice

shift and scaling patterns

Compiled by :Rashmi Bohra 252

 Why p-Clustering?
• Microarray data analysis may need to
– Clustering on thousands of dimensions (attributes)
– Discovery of both shift and scaling patterns
• Clustering with Euclidean distance measure? — cannot find shift patterns
• Clustering on derived attribute Aij = ai – aj? — introduces N(N-1) dimensions
• Bi-cluster using transformed mean-squared residue score matrix (I, J)

1
d = ∑ d 1 1
– Where ij | J | j ∈ J ij d =
∑ d d = ∑ d
Ij | I | ij IJ | I || J | i ∈ I , j ∈ J ij
i∈I
– A submatrix is a δ-cluster if H(I, J) ≤ δ for some δ > 0
• Problems with bi-cluster
– No downward closure property,
– Due to averaging, it may contain outliers but still within δ-threshold

Compiled by :Rashmi Bohra 253

 p-Clustering:
Clustering by
Pattern Similarity
• Given object x, y in O and features a, b in T, pCluster is a 2 by 2 matrix

d xa d xb 
pScore (  ) =| (d xa − d xb ) − (d ya − d yb ) |
for
• A pair (O, T) is in δ-pCluster
d ya d if
yb 
any 2 by 2 matrix X in (O, T), pScore(X) ≤
δ for some δ > 0
• Properties of δ-pCluster
– Downward closure
– Clusters are more homogeneous than bi-cluster (thus the name: pair-wise Cluster)

• Pattern-growth algorithm has been developed for efficient mining

• For scaling patterns, one can observe, taking logarithmic on will lead
to the pScore form

d xa / d ya
<δ
d xb / d yb
Compiled by :Rashmi Bohra 254
 Chapter 6. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 255
 Why Constraint-Based Cluster
Analysis?
• Need user feedback: Users know their applications the best
• Less parameters but more user-desired constraints, e.g., an ATM allocation
problem: obstacle & desired clusters

Compiled by :Rashmi Bohra 256

 A Classification of Constraints in
Cluster Analysis
• Clustering in applications: desirable to have user-
guided (i.e., constrained) cluster analysis
• Different constraints in cluster analysis:
– Constraints on individual objects (do selection first)
• Cluster on houses worth over $300K
– Constraints on distance or similarity functions
• Weighted functions, obstacles (e.g., rivers, lakes)
– Constraints on the selection of clustering parameters
• # of clusters, MinPts, etc.
– User-specified constraints
• Contain at least 500 valued customers and 5000 ordinary ones
– Semi-supervised: giving small training sets as “constraints”
or hints Compiled by :Rashmi Bohra 257
Clustering With Obstacle Objects
• K-medoids is more preferable since k-means
may locate the ATM center in the middle of a
lake
• Visibility graph and shortest path
• Triangulation and micro-clustering
• Two kinds of join indices (shortest-paths) worth
pre-computation
– VV index: indices for any pair of obstacle vertices
– MV index: indices for any pair of micro-cluster and
obstacle indices

Compiled by :Rashmi Bohra 258

 An Example: Clustering With
Obstacle Objects

Not Taking obstacles into Taking obstacles into

Compiled by :Rashmi Bohra 259
account account
 Clustering with User-Specified
Constraints
• Example: Locating k delivery centers, each serving at
least m valued customers and n ordinary ones
• Proposed approach
– Find an initial “solution” by partitioning the data set into k
groups and satisfying user-constraints
– Iteratively refine the solution by micro-clustering relocation
(e.g., moving δ μ-clusters from cluster Ci to Cj) and “deadlock”
handling (break the microclusters when necessary)
– Efficiency is improved by micro-clustering
• How to handle more complicated constraints?
– E.g., having approximately same number of valued customers
in each cluster?! — Can you solve it?
Compiled by :Rashmi Bohra 260
 Chapter 7. Cluster
Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 261
 What Is Outlier Discovery?
• What are outliers?
– The set of objects are considerably dissimilar from the
remainder of the data
– Example: Sports: Michael Jordon, Wayne Gretzky, ...
• Problem: Define and find outliers in large data sets
• Applications:
– Credit card fraud detection
– Telecom fraud detection
– Customer segmentation
– Medical analysis

Compiled by :Rashmi Bohra 262

Outlier Discovery:
Statistical
Approaches

❆ Assume a model underlying distribution that generates

data set (e.g. normal distribution)
• Use discordancy tests depending on
– data distribution
– distribution parameter (e.g., mean, variance)
– number of expected outliers
• Drawbacks
– most tests are for single attribute
– In many cases, data distribution may not be known
Compiled by :Rashmi Bohra 263
Outlier Discovery: Distance-Based
Approach
• Introduced to counter the main limitations imposed by
statistical methods
– We need multi-dimensional analysis without knowing data
distribution
• Distance-based outlier: A DB(p, D)-outlier is an object
O in a dataset T such that at least a fraction p of the
objects in T lies at a distance greater than D from O
• Algorithms for mining distance-based outliers
– Index-based algorithm
– Nested-loop algorithm
– Cell-based algorithm
Compiled by :Rashmi Bohra 264
 Density-Based
Local Outlier
Detection
• Distance-based outlier detection is
based on global distance
distribution
• It encounters difficulties to identify
outliers if data is not uniformly
distributed • Local outlier factor
• Ex. C1 contains 400 loosely (LOF)
distributed points, C2 has 100 – Assume outlier is not
crisp
tightly condensed points, 2 outlier
– Each point has a LOF
points o1, o2
• Distance-based method cannot
identify o2 as an outlier
• Need the concept of local outlier
Compiled by :Rashmi Bohra 265
 Outlier Discovery: Deviation-Based
Approach
• Identifies outliers by examining the main
characteristics of objects in a group
• Objects that “deviate” from this description are
considered outliers
• Sequential exception technique
– simulates the way in which humans can distinguish
unusual objects from among a series of supposedly like
objects
• OLAP data cube technique
– uses data cubes to identify regions of anomalies in large
multidimensional data
Compiled by :Rashmi Bohra 266
 Chapter 7. Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10.Constraint-Based Clustering
11.Outlier Analysis
12.Summary Compiled by :Rashmi Bohra 267
 Summary
• Cluster analysis groups objects based on their similarity and has wide
applications
• Measure of similarity can be computed for various types of data
• Clustering algorithms can be categorized into partitioning methods, hierarchical
methods, density-based methods, grid-based methods, and model-based
methods
• Outlier detection and analysis are very useful for fraud detection, etc. and can
be performed by statistical, distance-based or deviation-based approaches
• There are still lots of research issues on cluster analysis

Compiled by :Rashmi Bohra 268

 Problems and Challenges
• Considerable progress has been made in scalable
clustering methods
– Partitioning: k-means, k-medoids, CLARANS
– Hierarchical: BIRCH, ROCK, CHAMELEON
– Density-based: DBSCAN, OPTICS, DenClue
– Grid-based: STING, WaveCluster, CLIQUE
– Model-based: EM, Cobweb, SOM
– Frequent pattern-based: pCluster
– Constraint-based: COD, constrained-clustering
• Current clustering techniques do not address all the
requirements adequately, still an active area of research
Compiled by :Rashmi Bohra 269