What Is Prediction?

• (Numerical) prediction is similar to classification

– construct a model
– use model to predict continuous or ordered value for a given input
• Prediction is different from classification
– Classification refers to predict categorical class label
– Prediction models continuous-valued functions
• Major method for prediction: regression
– model the relationship between one or more independent or predictor
variables and a dependent or response variable
• Regression analysis
– Linear and multiple regression
– Non-linear regression
– Other regression methods: generalized linear model, Poisson regression,
log-linear models, regression trees
January 30, 2025 Data Mining: Concepts and Techniqu 1
es
 Linear Regression
• Linear regression: involves a response variable y and a single predictor
variable x

y = w 0 + w1 x
where w0 (y-intercept) and w1 (slope) are regression coefficients

• Method of least squares: estimates the best-fitting straight line

| D|

 ( x  x )( y  y )
i i w y  w x
w i 1
0 1
1  (x  x) | D|

i
2

i 1

January 30, 2025 Data Mining: Concepts and Techniques 2

Example of Linear regression
Multiple linear regression:
Multiple linear regression is an extension of straight-line regression so as to
involve more than one predictor variable
• It allows response variable y to be modeled as a linear function of, say, n
predictor variables or attributes, A1, A2, : : : , An, describing a tuple, X.
(That is, X = (x1, x2, : : : , xn).)
• training data set, D, contains data of the form (X1, y1), (X2, y2), : : : , (X
• jDj, yjDj), where the Xi are the n-dimensional training tuples with associated
class labels, yi.
– Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2
– Here x1, x2,… are values of the attributes A1, A2… etc for Xi
– Solvable by extension of least square method
– Many nonlinear functions can be transformed into the above
– commonly solved with the use of statistical software packages, such as
SAS, SPSS, and S-Plus
 Nonlinear Regression
• Some nonlinear models can be modeled by a polynomial function
• A polynomial regression model can be transformed into linear
regression model. For example,
y = w 0 + w 1 x + w 2 x 2 + w3 x 3
convertible to linear with new variables: x2 = x2, x3= x3
y = w 0 + w 1 x + w 2 x 2 + w3 x 3
• Other functions, such as power function, can also be transformed
to linear model
• Some models are intractable nonlinear (e.g., sum of exponential
terms)
– possible to obtain least square estimates through extensive
calculation on more complex formulae
January 30, 2025 Data Mining: Concepts and Techniqu 6
es
 Classifier Accuracy Measures
• Accuracy of a classifier M, acc(M): percentage of test set tuples that are correctly
classified by the model M(recognition rate)
• Error rate (misclassification rate) of M = 1 – acc(M)
The confusion matrix is a useful tool for analyzing how well your classifier can
recognize tuples of different classes.
– Given m classes, CMi,j, an entry in a confusion matrix, indicates # of tuples in
class i that are labeled by the classifier as class j
classes buy_computer = buy_computer = total recognition(%
yes no )
buy_computer = 6954 46 7000 99.34
yes
buy_computer = 412 2588 3000 86.27
no
• For atotal
classifier to have good accuracy, ideally2634
7366 most of the tuples
1000 would be
95.52
0
represented along the diagonal of the confusion matrix, from entry CM1, 1 to
entry CMm, m, with the rest of the entries being close to zero.
January 30, 2025 Data Mining: Concepts and Techniques 14
• True positives refer to the positive tuples that were correctly
labeled by the classifier,
• while true negatives are the negative tuples that were correctly
labeled by the classifier.
• False positives are the negative tuples that were incorrectly
labeled (e.g., tuples of class buys computer = no for which the
classifier predicted buys computer = yes). Similarly,
• false negatives are the positive tuples that were incorrectly
labeled (e.g., tuples of class buys computer = yes for which the
classifier predicted buys computer = no).

C1 C2
C1 True positive False
negative
C2 False True negative
positive
• Alternative accuracy measures
• Sensitivity or Recall (completeness) is also referred to as the true positive
(recognition) rate (that is, the proportion of positive tuples that are correctly
identified).
• specificity is the true negative rate (that is, the proportion of negative tuples
that are correctly identified).
• These measures are given as
sensitivity or Recall (completeness) = t-pos/pos /* true positive
recognition rate */
specificity = t-neg/neg /* true negative recognition rate */
precision (exactness )= t-pos/(t-pos + f-pos)
accuracy = sensitivity * pos/(pos + neg) + specificity * neg/(pos + neg)
– This model can also be used for cost-benefit analysis
• Sensitivity of the classifier is 90/ 300=30.00%.
• The specificity is 9560/ 9700= 98.56%.
• The classifier’s overall
accuracy is 9650/ 10,000 = 96.50%.
• The precision of the classifier for the yes class is
90/230 = 39.13%.
• The recall is 90 /300 = 30.00%
 Predictor Error Measures
• Measure predictor accuracy: measure how far off the predicted value is from the actual
known value
• Loss function: measures the error betw. yi and the predicted value yi’
– Absolute error: | yi – yi’|
– Squared error: (yi – yi’)2
• Test error (generalization error):
d the average loss over the test set d

– Mean absolute error:  | yi  yi ' | Mean squared error: (y

i 1
i  yi ' ) 2
i 1
d d

– d: no of samples in test set d

d
– Relative absolute error:  | yi  yRelative
i '|
squared error: (y
i 1
i  yi ' ) 2
i 1 d
d

| y i  y| (y
i 1
i  y)2
i 1

The mean squared-error exaggerates the presence of outliers

Popularly use (square) root mean-square error, similarly, root relative squared error
January 30, 2025 Data Mining: Concepts and Techniques 20
• For training data (first y1’=67.54
8 records from exam Y2’=44.47
Y3’=83.29
problem of linear Y4’=78.44

Regression) Mean absolute error = 9.2

Mean Squared error=87.5
• X- =74.87 Relative Absolute
error=0.8
• Y-=74.37 Relative squared
error=0.46
• W1=0.687 Root relative=.68
Root absolute=9.37
• W0=22.80
• Model :y=22.80+0.68
7x
 Evaluating the Accuracy of a Classifier or
Predictor (I)
• Holdout method
– Given data is randomly partitioned into two independent sets
• Training set (e.g., 2/3) for model construction
• Test set (e.g., 1/3) for accuracy estimation
– Random sampling: a variation of holdout
• Repeat holdout k times, accuracy = avg. of the accuracies obtained
• Cross-validation (k-fold, where k = 10 is most popular)
– Randomly partition the data into k mutually exclusive subsets, each
approximately equal size
– At i-th iteration, use Di as test set and others as training set
– Leave-one-out: k folds where k = # of tuples, for small sized data
– Stratified cross-validation: folds are stratified so that class dist. in each
fold is approx. the same as that in the initial data

January 30, 2025 Data Mining: Concepts and Techniqu 22

es
 Evaluating the Accuracy of a Classifier or Predictor
(II)
• Bootstrap
– Works well with small data sets
– Samples the given training tuples uniformly with replacement
• i.e., each time a tuple is selected, it is equally likely to be selected
again and re-added to the training set
• Several boostrap methods, and a common one is .632 boostrap
– Suppose we are given a data set of d tuples. The data set is sampled d times, with
replacement, resulting in a training set of d samples. The data tuples that did not
make it into the training set end up forming the test set. About 63.2% of the
original data will end up in the bootstrap, and the remaining 36.8% will form the
test set (since (1 – 1/d)d ≈ e-1 = 0.368)
– Repeat the sampling procedue k times, overall accuracy of the model:
k
acc( M )  (0.632 acc ( M i ) test _ set 0.368 acc ( M i ) train_ set )
i 1

January 30, 2025 Data Mining: Concepts and Techniques 23

 Chapter 6. Classification and Prediction

• What is classification?
Support Vector MachinesWhat is prediction?
(SVM)
• Issues regarding
Associative classification and prediction
classification
• Classification
Lazy learners by
(ordecision
learningtree
frominduction
your neighbors)
• Bayesian classification
Other classification methods
• Rule-based
Prediction classification
• Classification
Accuracy andby back
error propagation
measures
• Ensemble methods
• Model selection
• Summary
January 30, 2025 Data Mining: Concepts and Techniqu 24
es
 Ensemble Methods: Increasing the Accuracy

• Ensemble methods
– Use a combination of models to increase accuracy
– Combine a series of k learned models, M1, M2, …, Mk, with
the aim of creating an improved model M*
• Popular ensemble methods
– Bagging: averaging the prediction over a collection of
classifiers
– Boosting: weighted vote with a collection of classifiers
– Ensemble: combining a set of heterogeneous classifiers
January 30, 2025 Data Mining: Concepts and Techniqu 25
es
 Bagging: Boostrap Aggregation
• Analogy: Diagnosis based on multiple doctors’ majority vote
• Training
– Given a set D of d tuples, at each iteration i, a training set Di of d tuples is
sampled with replacement from D (i.e., boostrap)
– A classifier model Mi is learned for each training set Di
• Classification: classify an unknown sample X
– Each classifier Mi returns its class prediction
– The bagged classifier M* counts the votes and assigns the class with the
most votes to X
• Prediction: can be applied to the prediction of continuous values by taking
the average value of each prediction for a given test tuple
• Accuracy
– Often significant better than a single classifier derived from D
– For noise data: not considerably worse, more robust
– 30,Proved
January 2025 improved accuracy in prediction
Data Mining: Concepts and Techniqu 26
es
 Boosting
• Analogy: Consult several doctors, based on a combination of weighted
diagnoses—weight assigned based on the previous diagnosis accuracy
• How boosting works?
– Weights are assigned to each training tuple
– A series of k classifiers is iteratively learned
– After a classifier Mi is learned, the weights are updated to allow the
subsequent classifier, Mi+1, to pay more attention to the training tuples
that were misclassified by Mi
– The final M* combines the votes of each individual classifier, where the
weight of each classifier's vote is a function of its accuracy
• The boosting algorithm can be extended for the prediction of continuous
values
• Comparing with bagging: boosting tends to achieve greater accuracy, but it
also risks overfitting the model to misclassified data
January 30, 2025 Data Mining: Concepts and Techniqu 27
es
 Adaboost (Freund and Schapire, 1997)
• Given a set of d class-labeled tuples,
• Initially, all the weights of tuples are set the same (1/d)
• Generate k classifiers in k rounds. At round i,
– Tuples from D are sampled (with replacement) to form a training set Di
of the same size
– Each tuple’s chance of being selected is based on its weight
– A classification model Mi is derived from Di
– Its error rate is calculated using Di as a test set
– If a tuple is misclssified, its weight is increased, o.w. it is decreased
• Error rate: err(Xj) is the misclassification error of tuple Xj. Classifier Mi error
rate is the sum of the weights of the misclassified tuples:
d
error ( M i )  w j err ( X j )
j
• The weight of classifier Mi’s vote is 1  error ( M i )
log
error ( M i )
January 30, 2025 Data Mining: Concepts and Techniques 28
the closer the ROC curve of a model
is to the diagonal line, the less accurate the model
 Chapter 6. Classification and Prediction

• What is classification?
Support Vector MachinesWhat is prediction?
(SVM)
• Issues regarding
Associative classification and prediction
classification
• Classification
Lazy learners by
(ordecision
learningtree
frominduction
your neighbors)
• Bayesian classification
Other classification methods
• Rule-based
Prediction classification
• Classification
Accuracy andby back
error propagation
measures
• Ensemble methods
• Model selection
• Summary
January 30, 2025 Data Mining: Concepts and Techniqu 30
es
 Model Selection: ROC Curves

• ROC (Receiver Operating Characteristics)

curves: for visual comparison of classification
models
• Originated from signal detection theory
• Shows the trade-off between the true positive
rate and the false positive rate
•
 Vertical axis
The area under the ROC curve is a measure of
represents the true
the accuracy of the model
positive rate
• Rank the test tuples in decreasing order: the  Horizontal axis rep.
one that is most likely to belong to the positive the false positive
class appears at the top of the list rate
• The closer to the diagonal line (i.e., the closer  The plot also shows
the area is to 0.5), the less accurate is the a diagonal line
model  A model with perfect
January 30, 2025 Data Mining: Concepts and Techniques
accuracy will have 31
 Chapter 6. Classification and Prediction

• What is classification?
Support Vector MachinesWhat is prediction?
(SVM)
• Issues regarding
Associative classification and prediction
classification
• Classification
Lazy learners by
(ordecision
learningtree
frominduction
your neighbors)
• Bayesian classification
Other classification methods
• Rule-based
Prediction classification
• Classification
Accuracy andby back
error propagation
measures
• Ensemble methods
• Model selection
• Summary
January 30, 2025 Data Mining: Concepts and Techniqu 32
es
 Summary (I)
• Classification and prediction are two forms of data analysis that can be used
to extract models describing important data classes or to predict future
data trends.
• Effective and scalable methods have been developed for decision trees
induction, Naive Bayesian classification, Bayesian belief network, rule-based
classifier, Backpropagation, Support Vector Machine (SVM), associative
classification, nearest neighbor classifiers, and case-based reasoning, and
other classification methods such as genetic algorithms, rough set and fuzzy
set approaches.
• Linear, nonlinear, and generalized linear models of regression can be used
for prediction. Many nonlinear problems can be converted to linear
problems by performing transformations on the predictor variables.
Regression trees and model trees are also used for prediction.
January 30, 2025 Data Mining: Concepts and Techniqu 33
es
 Summary (II)
• Stratified k-fold cross-validation is a recommended method for accuracy
estimation. Bagging and boosting can be used to increase overall accuracy by
learning and combining a series of individual models.
• Significance tests and ROC curves are useful for model selection
• There have been numerous comparisons of the different classification and
prediction methods, and the matter remains a research topic
• No single method has been found to be superior over all others for all data
sets
• Issues such as accuracy, training time, robustness, interpretability, and
scalability must be considered and can involve trade-offs, further
complicating the quest for an overall superior method

January 30, 2025 Data Mining: Concepts and Techniqu 34

es
 References (1)
• C. Apte and S. Weiss. Data mining with decision trees and decision rules. Future Generation
Computer Systems, 13, 1997.
• C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press, 1995.
• L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth
International Group, 1984.
• C. J. C. Burges. A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and
Knowledge Discovery, 2(2): 121-168, 1998.
• P. K. Chan and S. J. Stolfo. Learning arbiter and combiner trees from partitioned data for scaling
machine learning. KDD'95.
• W. Cohen. Fast effective rule induction. ICML'95.
• G. Cong, K.-L. Tan, A. K. H. Tung, and X. Xu. Mining top-k covering rule groups for gene expression
data. SIGMOD'05.
• A. J. Dobson. An Introduction to Generalized Linear Models. Chapman and Hall, 1990.
• G. Dong and J. Li. Efficient mining of emerging patterns: Discovering trends and differences.
KDD'99.

January 30, 2025 Data Mining: Concepts and Techniqu 35

es
 References (2)
• R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification, 2ed. John Wiley and Sons, 2001
• U. M. Fayyad. Branching on attribute values in decision tree generation. AAAI’94.
• Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an
application to boosting. J. Computer and System Sciences, 1997.
• J. Gehrke, R. Ramakrishnan, and V. Ganti. Rainforest: A framework for fast decision tree
construction of large datasets. VLDB’98.
• J. Gehrke, V. Gant, R. Ramakrishnan, and W.-Y. Loh, BOAT -- Optimistic Decision Tree Construction.
SIGMOD'99.
• T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning: Data Mining,
Inference, and Prediction. Springer-Verlag, 2001.
• D. Heckerman, D. Geiger, and D. M. Chickering. Learning Bayesian networks: The combination of
knowledge and statistical data. Machine Learning, 1995.
• M. Kamber, L. Winstone, W. Gong, S. Cheng, and J. Han. Generalization and decision tree
induction: Efficient classification in data mining. RIDE'97.
• B. Liu, W. Hsu, and Y. Ma. Integrating Classification and Association Rule. KDD'98.
• W. Li, J. Han, and J. Pei, CMAR: Accurate and Efficient Classification Based on Multiple Class-
Association Rules, ICDM'01.
January 30, 2025 Data Mining: Concepts and Techniqu 36
es
 References (3)

• T.-S. Lim, W.-Y. Loh, and Y.-S. Shih. A comparison of prediction accuracy, complexity, and training
time of thirty-three old and new classification algorithms. Machine Learning, 2000.
• J. Magidson. The Chaid approach to segmentation modeling: Chi-squared automatic
interaction detection. In R. P. Bagozzi, editor, Advanced Methods of Marketing Research,
Blackwell Business, 1994.
• M. Mehta, R. Agrawal, and J. Rissanen. SLIQ : A fast scalable classifier for data mining. EDBT'96.
• T. M. Mitchell. Machine Learning. McGraw Hill, 1997.
• S. K. Murthy, Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey,
Data Mining and Knowledge Discovery 2(4): 345-389, 1998
• J. R. Quinlan. Induction of decision trees. Machine Learning, 1:81-106, 1986.
• J. R. Quinlan and R. M. Cameron-Jones. FOIL: A midterm report. ECML’93.
• J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.
• J. R. Quinlan. Bagging, boosting, and c4.5. AAAI'96.

January 30, 2025 Data Mining: Concepts and Techniqu 37

es
 References (4)

• R. Rastogi and K. Shim. Public: A decision tree classifier that integrates building and pruning.
VLDB’98.
• J. Shafer, R. Agrawal, and M. Mehta. SPRINT : A scalable parallel classifier for data mining.
VLDB’96.
• J. W. Shavlik and T. G. Dietterich. Readings in Machine Learning. Morgan Kaufmann, 1990.
• P. Tan, M. Steinbach, and V. Kumar. Introduction to Data Mining. Addison Wesley, 2005.
• S. M. Weiss and C. A. Kulikowski. Computer Systems that Learn: Classification and Prediction
Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. Morgan
Kaufman, 1991.
• S. M. Weiss and N. Indurkhya. Predictive Data Mining. Morgan Kaufmann, 1997.
• I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques, 2ed.
Morgan Kaufmann, 2005.
• X. Yin and J. Han. CPAR: Classification based on predictive association rules. SDM'03
• H. Yu, J. Yang, and J. Han. Classifying large data sets using SVM with hierarchical clusters.
KDD'03.

January 30, 2025 Data Mining: Concepts and Techniqu 38

es
January 30, 2025 Data Mining: Concepts and Techniques 39