Concepts and

Techniques

— Chapter 2 —

Jiawei Han, Micheline Kamber, and Jian Pei

University of Illinois at Urbana-Champaign
Simon Fraser University
©2011 Han, Kamber, and Pei. All rights
reserved.
Chapter 2: Getting to Know Your
Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Summary
 Types of Data Sets
 Record
 Relational records
 Data matrix, e.g., numerical matrix,

timeout

season
coach

game
score
team

ball

lost
pla

wi
crosstabs

n
y
 Document data: text documents:
term-frequency vector
Document 1 3 0 5 0 2 6 0 2 0 2
 Transaction data
 Graph and network Document 2 0 7 0 2 1 0 0 3 0 0
 World Wide Web
Document 3 0 1 0 0 1 2 2 0 3 0
 Social or information networks
 Molecular Structures
 Ordered TID Items
 Video data: sequence of images
1 Bread, Coke, Milk
 Temporal data: time-series
 Sequential Data: transaction 2 Beer, Bread
sequences 3 Beer, Coke, Diaper, Milk
 Genetic sequence data 4 Beer, Bread, Diaper, Milk
 Spatial, image and multimedia: 5 Coke, Diaper, Milk
 Spatial data: maps
 Image data:
 Video data:
 Important Characteristics of
Structured Data

 Dimensionality
 Curse of dimensionality
 Sparsity
 Only presence counts
 Resolution

Patterns depend on the scale
 Distribution
 Centrality and dispersion
 Data Objects

 Data sets are made up of data objects.

 A data object represents an entity.
 Examples:
 sales database: customers, store items, sales
 medical database: patients, treatments
 university database: students, professors,
courses
 Also called samples , examples, instances, data
points, objects, tuples.
 Data objects are described by attributes.
 Database rows -> data objects; columns -
 Attributes
 Attribute (or dimensions, features,
variables): a data field, representing a
characteristic or feature of a data object.
 E.g., customer _ID, name, address

 Types:
 Nominal

 Binary

 Numeric: quantitative


Interval-scaled

Ratio-scaled
 Attribute Types
 Nominal:
 Nominal means “relating to names.” The values of a

nominal attribute are symbols or names of things.

 categories, states, or “names of things”

 Hair_color = {auburn, black, blond, brown, grey, red,

white}
 marital status, occupation, ID numbers, zip codes
 Binary
 It is a Nominal attribute with only 2 states (0 and 1)
 Symmetric binary: both outcomes equally important

e.g., gender
 Asymmetric binary: outcomes not equally important.

e.g., medical test (positive vs. negative)

Convention: assign 1 to most important outcome
(e.g., HIV positive)
 Ordinal
 Values have a meaningful order (ranking) but magnitude
between successive values is not known.
 Size = {small, medium, large}, grades, army rankings
 Numeric Attribute Types
 Quantity (integer or real-valued)
 Interval

Measured on a scale of equal-sized units

Values have order
 E.g., temperature in C˚or F˚, calendar
dates

No true zero-point
 Ratio

Inherent zero-point

We can speak of values as being an order of
magnitude larger than the unit of
measurement (10 K˚ is twice as high as 5
K˚).
 e.g., temperature in Kelvin, length,
counts, monetary quantities
 Discrete vs. Continuous
Attributes
 Discrete Attribute
 Has only a finite or countably infinite set of

values

E.g., zip codes, profession, or the set of words
in a collection of documents
 Sometimes, represented as integer variables

 Note: Binary attributes are a special case of

discrete attributes
 Continuous Attribute
 Has real numbers as attribute values


E.g., temperature, height, or weight
 Practically, real values can only be measured and

represented using a finite number of digits

 Continuous attributes are typically represented

as floating-point variables
Chapter 2: Getting to Know Your
Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Summary
Basic Statistical Descriptions of
Data
 Motivation
 To better understand the data: central
tendency, variation and spread
 Central tendency: A measure of central tendency (also
referred to as measures of centre or central location) is a
summary measure that attempts to describe a whole set of
data with a single value that represents the middle or centre
of its distribution.
 Variation of the Data: Measures of variation are statistics of
how far away the values in the observations (data points)
are from each other. There are different measures of
variation such as Range, Quartiles and Percentiles,
Interquartile Range, Standard Deviation.
 Spread: Measures of spread describe how similar or varied
the set of observed values are for a particular variable (data
item). Measures of spread include the range, quartiles and
Basic Statistical Descriptions of
Data
 Data dispersion characteristics: It should be based
on all the observations of the series. It should be
rigidly defined. It should not be affected by
extreme values.
 median, max, min, quantiles, outliers, variance,
etc.
 Numerical dimensions correspond to sorted
intervals
 Data dispersion: analyzed with multiple
granularities of precision
 Boxplot or quantile analysis on sorted intervals
 Population: A population is the entire group that
you want to draw conclusions about.
 Sample:A sample is defined as a smaller and more
Measuring the Central Tendency
 Mean (algebraic measure) (sample vs. population): 1 n
x   xi   x
Note: n is sample size and N is population size. n i 1 N
n
 Weighted arithmetic mean

w x i i
trimmed mean: Extreme Values x  i 1n
w
i 1
i

 Median: The data are already sorted in increasing

order
 Middle value if odd number of values, or
average of the middle two values otherwise
 Mode
 Value that occurs most frequently in the data
 Unimodal, bimodal, trimodal
mean  mode 3 (mean  median)
 Empirical formula:
 Examples

Median:
(52+56)/2=108/2=54
Mode: The two modes are
$52,000 and $70,000
 Symmetric vs.
Skewed Data
 Median, mean and mode of symmetric

symmetric, positively and

negatively skewed data

positively skewed negatively

skewed

Data Mining: Concepts and

Techniques
 Symmetric vs.
Skewed Data

 In a unimodal frequency curve with perfect

symmetric data distribution, the mean, median,
and mode are all at the same center value
 Data in most real applications are not symmetric.
 Positively skewed: The mode occurs at a value
that is smaller than the median.
 Negatively skewed: The mode occurs at a value
greater than the median.

Data Mining: Concepts and

Techniques
 Measuring the Dispersion of
Data
 Quartiles, outliers and boxplots
 Quartiles: Q1 (25th percentile), Q3 (75th percentile)
 Inter-quartile range: IQR = Q3 – Q1
 Five number summary: min, Q1, median, Q3, max
 Boxplot: ends of the box are the quartiles; median is marked; add
whiskers, and plot outliers individually
 Outlier: usually, a value higher/lower than 1.5 x IQR
 Variance and standard deviation (sample: s, population: σ)
 Variance: (algebraic, scalable computation)
1 n 1 n 2 1 n 2 1 n 1 n
2
s  
n  1 i 1

2
( xi  x )  
[ xi  ( xi ) ]
n  1 i 1 n i 1
   ( xi   ) 2 
2

N i 1 N
 xi   2
i 1
2

 Standard deviation s (or σ) is the square root of variance s2 (or

σ2)
 Quantiles
 Quantiles are points taken at regular intervals of a data
distribution, dividing it into essentially equalsize
consecutive sets.
 The 4-quantiles are the three data points that split the
data distribution into four equal parts; each part
represents one-fourth of the data distribution.
 The 100-quantiles are more commonly referred to as
percentiles; they divide the data distribution into 100
equal-sized consecutive sets.
 Example

Let we have the following values for salary

(in thousands of dollars), shown in increasing
order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70,
70, 110.

 Q1 is $47,000 and Q3 is $63,000

 Q2 or Median is $52,000
 IQR is $63,000-$47,000=$16,000
 Boxplot Analysis
 Five-number summary of a distribution
 Minimum, Q1, Median, Q3, Maximum
 Boxplot: visualizing a data distribution
 Data is represented with a box
 The ends of the box are at the first and
third quartiles, i.e., the height of the box
is IQR
 The median is marked by a line within the
box
 Whiskers: two lines outside the box
extended to Minimum and Maximum
 Outliers: points beyond a specified outlier
Note: A common rule for identifying suspected outliers is to single out
valuesthreshold, plotted
falling at least 1.5 individually
times IQR above the third quartile or below the
first quartile.
 Example

Let we have four branches of an Electronics company.

The first box represernts: median price sold is $80, Q1
is $60, and Q3 is $100.
Notice that two outlying observations for this branch
are the values of 175 and 202 which are more than 1.5
times the IQR here of 40.
Variance and Standard Deviation
 Variance and standard deviation are measures of
data dispersion.
 They indicate how spread out a data distribution is.
 A low standard deviation means that the data
observations tend to be very close to the mean
 High standard deviation indicates that the data are
spread out over a large range of values.
 Example
 Ex: Let we have the following values for
salary (in thousands of dollars), shown in
increasing order: 30, 36, 47, 50, 52, 52, 56,
60, 63, 70, 70, 110. Mean is 58

_x001B_ = 0 only when there is no spread, that is,

when all observations have the same value.
Otherwise, _x001B_ > 0.
 Properties of Normal Distribution
Curve

 The normal (distribution) curve

 From μ–σ to μ+σ: contains about 68% of the

measurements (μ: mean, σ: standard deviation)

 From μ–2σ to μ+2σ: contains about 95% of it
 From μ–3σ to μ+3σ: contains about 99.7% of it
Graphic Displays of Basic Statistical
Descriptions

 Boxplot: graphic display of five-number summary

 Histogram: x-axis are values, y-axis repres.
frequencies
 Quantile plot: each value xi is paired with fi
indicating that approximately 100 fi % of data are
 xi
 Quantile-quantile (q-q) plot: graphs the
quantiles of one univariant distribution against the
corresponding quantiles of another
 Scatter plot: each pair of values is a pair of
 Quantile Plot
 A quantile plot is a simple and effective way to have a first
look at a univariate (one attribute) data distribution.
 First, it displays all of the data for the given attribute
 Second, it plots quantile information
 Quantile–Quantile Plot
 A quantile–quantile plot, or q-q plot, graphs the quantiles of
one univariate distribution against the corresponding
quantiles of another.
 It is a powerful visualization tool in that it allows the user to
view whether there is a shift in going fromone distribution to
another.
 Histogram Analysis
 “Histos” means pole or mast, and
“gram” means chart, so a
40
histogram is a chart of poles.
35
 Histogram: Graph display of
tabulated frequencies, shown as 30
bars 25
 It shows what proportion of cases 20
fall into each of several categories
15
 Differs from a bar chart in that it
is the area of the bar that denotes 10
the value, not the height as in bar 5
charts, a crucial distinction when 0
the categories are not of uniform 10000 30000 50000 70000 90000

width
 The categories are usually
specified as non-overlapping
 Histogram Analysis
 The range of values is partitioned into disjoint consecutive
subranges. The subranges, referred to as buckets or bins, are
disjoint subsets of the data distribution.
 The range of a bucket is known as the width.
 Buckets (orbins) are defined by equal-width ranges

A histogram
Histograms Often Tell More than
Boxplots

 The two histograms

shown in the left
may have the same
boxplot
representation
 The same values
for: min, Q1,
median, Q3, max
 But they have
rather different data
distributions
 Scatter plot
 A scatter plot is one of the most effective graphical
methods for determining if there appears to be a
relationship, pattern, or trend between two numeric
attributes.
 To construct a scatter plot, each pair of values is
treated as a pair of coordinates in an algebraic sense
and plotted as points in the plane.
 Scatter plot
 Provides a first look at bivariate data to see clusters
of points, outliers, etc
 Each pair of values is treated as a pair of coordinates
and plotted as points in the plane
 Explores the possibility of correlation relationships
between two attributes, X, and Y, are correlated if one
attribute implies the other.
 Correlations can be positive, negative, or null
(uncorrelated).
Positively and Negatively Correlated
Data

 The left half fragment is

positively correlated
 The right half is negative
correlated
Uncorrelated Data
 Summary
 Data attribute types: nominal, binary, ordinal, interval-scaled,
ratio-scaled
 Many types of data sets, e.g., numerical, text, graph, Web,
image.
 Gain insight into the data by:
 Basic statistical data description: central tendency,
dispersion, graphical displays
 Above steps are the beginning of data preprocessing.
 Many methods have been developed but still an active area of
research.
 References
 W. Cleveland, Visualizing Data, Hobart Press, 1993
 T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley, 2003
 U. Fayyad, G. Grinstein, and A. Wierse. Information Visualization in Data Mining and
Knowledge Discovery, Morgan Kaufmann, 2001
 L. Kaufman and P. J. Rousseeuw. Finding Groups in Data: an Introduction to Cluster
Analysis. John Wiley & Sons, 1990.
 H. V. Jagadish, et al., Special Issue on Data Reduction Techniques. Bulletin of the Tech.
Committee on Data Eng., 20(4), Dec. 1997
 D. A. Keim. Information visualization and visual data mining, IEEE trans. on Visualization
and Computer Graphics, 8(1), 2002
 D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999
 S. Santini and R. Jain,” Similarity measures”, IEEE Trans. on Pattern Analysis and
Machine Intelligence, 21(9), 1999
 E. R. Tufte. The Visual Display of Quantitative Information, 2nd ed., Graphics Press,
2001
 C. Yu , et al., Visual data mining of multimedia data for social and behavioral studies,
Information Visualization, 8(1), 2009