Data Preprocessing

What is Data?

z Collection of data objects and Attributes

their attributes

z An attribute is a property or Tid Refund Marital

Status
Taxable
Income Cheat
characteristic of an object
1 Yes Single
g 125K No
– E
Examples:
l eye color
l off a
2 No Married 100K No
person, temperature, etc.
3 No Single 70K No
– Attribute is also known as
4 Yes Married 120K No
variable field
variable, field, characteristic
characteristic,
5 No Divorced 95K
or feature Objects
Yes
6 No Married 60K No
7 Yes Divorced 220K No
z A collection of attributes 8 No Single 85K Yes
describe an object 9 No Married 75K No
– Object is also known as 10 No Single 90K Yes
record point
record, point, case
case, sample
sample,
10

entity, or instance
Data Preprocessing
p g

z Why preprocess the data?

z Descriptive data summarization (covered!)
z Data cleaning
z D
Data iintegration
i and
d transformation
f i
z Data reduction
z Discretization and concept hierarchy generation
z Summary
Why
y Data Preprocessing?
p g
z Data in the real world is dirty
– iincomplete:
l t lacking
l ki attribute
tt ib t values,
l llacking
ki certain
t i
attributes of interest, or containing only aggregate
data
‹ e.g., occupation=“ ”
– noisy: containing errors or outliers
‹ e g Salary=“-10”
e.g., Salary 10
– inconsistent: containing discrepancies in codes or
names
‹ e.g., Age=“42”
A “42” Birthday=“03/07/1997”
Bi thd “03/07/1997”
‹ e.g., Was rating “1,2,3”, now rating “A, B, C”
‹ e.g., discrepancy between duplicate records
Why Is Data Dirty?

z Incomplete data may come from

– “Not
“N t applicable”
li bl ” ddata
t value
l when
h collected
ll t d
– Different considerations between the time when the data was collected
and when it is analyzed.
– Human/hardware/software
H /h d / ft problems
bl
z Noisy data (incorrect values) may come from
– Faulty data collection instruments
– Human or computer error at data entry
– Errors in data transmission
z Inconsistent data may come from
– Different data sources
– Functional dependency violation (e.g., modify some linked data)
z Duplicate records also need data cleaning
Why Is Data Preprocessing Important?

z No quality data, no quality mining results!

– Quality decisions must be based on quality data
‹ e.g., duplicate or missing data may cause incorrect or even
misleading statistics
statistics.
– Data warehouse needs consistent integration of quality data
z Data extraction,, cleaning,
g, and transformation comprises
p
the majority of the work of building a data mining system
Multi-Dimensional Measure of Data Quality

z A well-accepted
p multidimensional view:
– Accuracy
– Completeness
– Consistency
– Timeliness
– Believability
– V l added
Value dd d
– Interpretability
– Accessibility
 Major Tasks in Data Preprocessing

z Data cleaning
– Fill iin missing
i i values,
l smooth
th noisy
i d data,
t ididentify
tif or remove outliers,
tli
and resolve inconsistencies
z Data integration
– Integration of multiple databases or files
z Data transformation
– Normalization and aggregation
z Data reduction
– Obtains reduced representation in volume but produces the same or
similar
i il analytical
l ti l results
lt
z Data discretization
– Part of data reduction but with p
particular importance,
p , especially
p y for
numerical data
Forms of Data Preprocessing
 Data Preprocessing

z Why preprocess the data?

z Descriptive data summarization (covered!)
z Data cleaning
z D
Data iintegration
i and
d transformation
f i
z Data reduction
z Discretization and concept hierarchy generation
z Summary
Data Cleaning

z Importance
p
– garbage in garbage out principle (GIGO)

z Data cleaning tasks

– Fill in missing values

– Identify outliers and smooth out noisy data

– Correct inconsistent data

– Resolve redundancy caused by data integration
Missing Data

z Data is not always available

– E.g., many tuples have no recorded value for several attributes, such
as customer income in sales data
z Missing data may be due to
– equipment malfunction
– inconsistent with other recorded data and thus deleted
– data not entered due to misunderstanding
– certain data may not be considered important at the time of entry
– not register history or changes of the data
z Missing data may need to be inferred
How to Handle Missing Data?
z Ignore the tuple: usually done when class label is missing (assuming
the tasks in classification
classification—not
not effective when the percentage of
missing values per attribute varies considerably.

z Fill in the missing value manually: tedious + infeasible?

z Fill in it automatically with

– a global constant : e
e.g.,
g “unknown”
unknown , a new class?!

– the attribute mean

– the attribute mean for all data points belonging to the same class:
smarter

– the most probable value: inference-based such as Bayesian formula or

decision tree
Noisy Data

z Noise: random error or variance in a measured variable

z I
Incorrect
t attribute
tt ib t values
l may due
d to t
– faulty data collection instruments
– data entry problems
– data transmission problems
– technology limitation
– inconsistency in naming convention
z Class label noise is hard to deal with
– sometimes we don’t
don t know whether the class label is correct or it is
simply unexpected
z Noise demands robustness in training algorithms, that is,
t i i should
training h ld nott b
be sensitive
iti tto noise
i
 How to Handle Noisy Data?

z Binning
– first
fi t sortt d
data
t and
d partition
titi into
i t (equal-frequency)
( lf ) bins
bi
– then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
z Regression
– smooth by fitting the data into regression functions
z Cl t i
Clustering
– detect and remove outliers
z Combined computer and human inspection
– detect suspicious values and check by human (e.g., deal with
possible outliers)
Simple Discretization Methods: Binning

z Equal-width (distance) partitioning

– Divides the range into N intervals of equal size: uniform grid

– if A and B are the lowest and highest values of the attribute, the width of
intervals will be: W = (B –A)/N.
A)/N

– The most straightforward, but outliers may dominate presentation

– Skewed data is not handled well

z Equal-depth (frequency) partitioning

– Divides the range into N intervals,
intervals each containing approximately same
number of data points

– Good data scaling

g

– Managing categorical attributes can be tricky

Binning Methods for Data Smoothing

‰ Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28,
29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing
g by
y bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin
Bi 33: 29
29, 29
29, 29
29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Cluster Analysis as Binning
 Data Cleaning as a Process

z Data discrepancy detection

– Use metadata (e.g., domain, range, dependency, distribution)
– Check field overloading
– Check uniqueness rule, consecutive rule and null rule
– Use commercial tools
‹ Data scrubbing: use simple domain knowledge (e.g., postal code,
spell-check) to detect errors and make corrections
‹ Data auditing: by analyzing data to discover rules and relationship to
detect violators (e
(e.g.,
g correlation and clustering to find outliers)
z Data migration and integration
– Data migration tools: allow transformations to be specified
– ETL (Extraction/Transformation/Loading) tools: allow users to specify
transformations through a graphical user interface
z Integration of the two processes
– Iterative and interactive (e
(e.g.,
g Potter’s
Potter s Wheels)
Data Preprocessing

z Why preprocess the data?

z Data cleaning
z Data integration and transformation
z Data reduction
z Discretization and concept hierarchy generation
z Summary
Data Integration

z Data integration:
– Combines data from multiple sources into a coherent store
z Schema integration: e.g., A.cust-id ≡ B.cust-#
– Integrate metadata from different sources
z E tit id
Entity identification
tifi ti problem:
bl
– Identify real world entities from multiple data sources, e.g., Bill
Clinton = William Clinton
z Detecting and resolving data value conflicts
– For the same real world entity, attribute values from different
sources are different
– Possible reasons: different representations, different scales, e.g.,
metric vs. British units
Handling Redundancy in Data Integration

z Redundant data occur often when integration of multiple

databases
– Object identification: The same attribute or object may have
different names in different databases
– Derivable data: One attribute may be a “derived” attribute in
another table, e.g., annual revenue
z Redundant attributes may be able to be detected by
correlation analysis
z Careful integration of the data from multiple sources may
help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
Correlation Analysis (Numerical Data)

z Correlation coefficient (Pearson’s correlation coefficient)

rA , B =
∑ ( A − A )( B − B ) ∑ ( AB ) − n A B
=
( n − 1)σ A σ B ( n − 1)σ A σ B
where n is the number of tuples, A and B are the respective means of A
and B,
B σA and σB are the respective standard deviation of A and B,
B and
Σ(AB) is the sum of the AB cross-product.

z If rA,B > 0, A and B are positively correlated (A’s values

increase as B’s). The higher, the stronger correlation.
z rA,B = 0: uncorrelated; rA,B < 0: negatively correlated
Correlation Analysis (Categorical Data)

z Χ2 (chi-square) test

n
(Ob
Observed
d − E
Expected
d ) 2
χ n2−1 = ∑ i i

i =1 Expectedi
z n is
i th
the number
b off possible
ibl values
l
z The larger the Χ2 value, the more likely the variables are related
z The cells that contribute the most to the Χ2 value are those whose
actual count is very different from the expected count
z Correlation does not imply causality
– # of hospitals and # of car-theft in a city are correlated
– Both are causally linked to the third variable: population
Chi-Square Calculation: An Example

Play chess Not play chess Sum (row)

Like science fiction 250 200 450

Not like science fiction 50 1000 1050

Sum (col.)
(col ) 300 1200 1500

Probability to play chess: P(chess) = 300/1500 = 0.2

Probability to like science fiction: P(SciFi) = 450/1500 = 0.3

If science fiction and chess playing are independent attributes, then the
probability to like SciFi AND play chess is

P(SciFi, chess) = P(SciFi) · P(chess) = 0.06

That means, we expect 0.06 · 1500 = 90 such cases (if they are independent)
 Chi-Square Calculation: An Example

Play chess Not play chess Sum (row)

Like science fiction 250 (90) 200 (360) 450

Not like science fiction 50 (210) 1000 (840) 1050

Sum (col.)
(col ) 300 1200 1500

z Χ2 (chi-square) calculation (numbers in parenthesis are

expected counts calculated based on the data distribution
in the two categories)
(250 − 90) 2 (50 − 210) 2 (200 − 360) 2 (1000 − 840) 2
χ =
2
+ + + = 507.93
90 210 360 840
z It shows that like_science_fiction
like science fiction and play
play_chess
chess are
correlated in the group
Data Transformation

z Smoothing: remove noise from data

z Aggregation: summarization
z Generalization: concept hierarchy climbing
z Normalization: scaled to fall within a small
small, specified
range
– min-max normalization
– z-score normalization
– normalization by decimal scaling
z Attribute/feature construction
– New attributes constructed from the given ones
Aggregation

Variation of Precipitation in Australia

Sta da d Deviation
Standard e at o o of Average
e age Sta da d Deviation
Standard e at o o of Average
e age
Monthly Precipitation Yearly Precipitation
Attribute Transformation

z A function that maps the entire set of values of a

given attribute to a new set of replacement values
g
such that each old value can be identified with
one of the new values
– Simple functions: xk, log(x), ex, |x|
– Standardization and Normalization
Attribute Normalization

z Min-max normalization: to [new_minA, new_maxA]

v − min
i A
v' = (new _ maxA − new _ minA) + new _ minA
maxA − minA
– Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0].
Then $73,000 is mapped to
73,600 − 12,000
(1.0 − 0) + 0 = 0.716
98,000 − 12,000

z Z-score normalization (μ: mean, σ: standard deviation):

v − μA
v'=
σ A

– Ex. Let μ = 54,000, σ = 16,000. Then

73,600 − 54,000
= 1.225
16,000
Data Preprocessing

z Why preprocess the data?

z Data cleaning
z Data integration and transformation
z D
Data reduction
d i
z Discretization and concept
p hierarchy
yggeneration
z Summary
Data Reduction Strategies

z Why data reduction?

– A database may store terabytes of data
– Complex data analysis/mining may take a very long time to run on the
complete data set
z Data reduction
– Obtain a reduced representation of the data set that is much smaller in
volume but yet produce the same (or almost the same) analytical
results
z Data reduction strategies
– Data Compression
– Sampling
S li
– Discretization and concept hierarchy generation
– Dimensionality reduction — e.g. remove unimportant attributes
Data Compression

z String compression
– There
Th are extensive
t i theories
th i and
d well-tuned
ll t d algorithms
l ith
– Typically lossless
– But only limited manipulation is possible without expansion
z Audio/video compression
– Typically lossy compression, with progressive refinement
– Sometimes small fragments of signal can be reconstructed without
reconstructing the whole
z Time sequence
q is not audio
– Typically short and vary slowly with time
Data Compression

Original Data Compressed

Data
lossless

Original Data
Approximated
Data Compression (via PCA)

Dimensions
Dimensions==206
120
160
10
40
80
 Data Reduction Method: Sampling
p g
z Sampling: obtaining a small sample s to represent
the whole data set N
z Allow a mining algorithm to run in complexity that
is potentially sub-linear
sub linear to the size of the data
z Choose a representative subset of the data
– Simple
p random sampling
p g may y have very
yppoor
performance in the presence of skew
z Develop adaptive sampling methods
– St
Stratified
tifi d sampling:
li
‹ Approximate the percentage of each class (or
p p
subpopulation of interest)) in the overall database
‹ Used in conjunction with skewed data
Types of Sampling

z Simple Random Sampling

– There is an equal probability of selecting any particular item

z Sampling without replacement

– As each item is selected, it is removed from the population
p p

z Sampling with replacement

– Objects are not removed from the population as they are
selected for the sample.
‹ In sampling with replacement, the same object can be picked up
more than once

z Stratified sampling
– Split the data into several partitions; then draw random samples
from each partition
Sampling: with or without Replacement

Raw Data
Sample Size

8000 points 2000 Points 500 Points

Sample Size
z What sample size is necessary to get at least one
object from each of 10 groups.
Sampling: Cluster or Stratified Sampling

Raw Data Cluster/Stratified Sample

Feature Subset Selection

z Another way to reduce dimensionality of data

z Redundant features
– duplicate much or all of the information contained in
one or more other attributes
– Example: purchase price of a product and the amount
of sales tax paid

z Irrelevant features
– contain no information that is useful for the data
mining task at hand
– Example: students' ID is often irrelevant to the task of
predicting students' GPA
Feature Subset Selection

z Techniques:
– Brute
Brute-force
force approach:
‹ Try all possible feature subsets as input to data mining algorithm
– Embedded approaches:
‹ Feature selection occurs naturally as part of the data mining
algorithm
– Filter approaches:
‹ Features are selected before data mining algorithm is run
– Wrapper approaches:
‹ Use the data mining algorithm as a black box to find best subset
of attributes
Feature Creation

z Create new attributes that can capture the

important information in a data set much more
efficiently than the original attributes

z Methodologies:
– Mapping Data to New Space
‹ Feature construction by combining features
 Data Preprocessing
p g

z Why preprocess the data?

z Data cleaning
z D t integration
Data i t ti and
d ttransformation
f ti
z Data reduction
z Discretization and concept hierarchy generation
z Summary
Discretization

z Three types of attributes:

– Nominal — values from an unordered set, e.g., color, profession

– Ordinal — values from an ordered set, e.g., military or academic rank

– Continuous — real numbers, e.g.,

g integer
g or real numbers ((here we
aggregated interval and ratio attributes into continuous)

z Discretization:
– Divide the range of a continuous attribute into intervals

– Some classification algorithms only accept categorical attributes.

– Reduce data size by discretization

– Prepare for further analysis

Discretization Using Class Labels
z Entropy based approach

3 categories for both x and y 5 categories for both x and y

Discretization Without Using Class Labels

D t
Data E
Equal
l interval
i t l width
idth

Equal frequency K-means

Discretization and Concept Hierarchy

z Discretization
– Reduce the number of values for a given continuous attribute by dividing
the range of the attribute into intervals
– Interval labels can then be used to replace actual data values
– Supervised vs. unsupervised (use class or don’t use class variable)
– Split (top-down) vs. merge (bottom-up)

z Concept hierarchy formation

– Recursively reduce the data by collecting and replacing low level concepts
(such as numeric values for age) by higher level concepts (such as young,
middle-aged,
middle aged, or senior)
 Discretization and Concept Hierarchy
Generation for Numeric Data

z Typical methods: All the methods can be applied recursively

– Binning (covered earlier)

‹ Top-down split, unsupervised,

– Histogram analysis (covered earlier)

‹ Top-down split, unsupervised

– Clustering
Cl t i analysis
l i ((covered
d earlier
li and
d iin more d
detail
t il llater)
t )

‹ Either top-down split or bottom-up merge, unsupervised

– Entropy-based
Entropy based discretization: supervised,
supervised top
top-down
down split

– Interval merging by χ2 Analysis: unsupervised, bottom-up merge

– Segmentation by natural partitioning: top-down

top down split
split, unsupervised
Concept Hierarchy Generation for
Categorical Data

z Specification of a partial/total ordering of attributes

e plicitl at the schema le
explicitly level
el b
by users
sers or e
experts
perts
– street < city < state < country
z Specification of a hierarchy for a set of values by explicit
data grouping
– {Urbana, Champaign, Chicago} < Illinois
z Specification of only a partial set of attributes
– E.g., only street < city, not others
z Automatic generation of hierarchies (or attribute levels) by
the analysis of the number of distinct values
– E.g., for a set of attributes: {street, city, state, country}
Automatic Concept Hierarchy Generation
z Some hierarchies can be automatically generated
based on the analysis
y of the number of distinct values
per attribute in the data set
– The attribute with the most distinct values is placed at the
y
lowest level of the hierarchy
– Exceptions, e.g., weekday, month, quarter, year

country 15 di
distinct
ti t values
l

pprovince_or_
_ _ state 365 distinct values

city 3567 distinct values

street 674,339 distinct values

Data Preprocessing

z Why preprocess the data?

z Data cleaning
z Data integration and transformation
z Data reduction
z Discretization and concept hierarchy
generation
z Summary
Summary

z Data preparation or preprocessing is a big issue for data

mining
g
z Descriptive data summarization is need for quality data
p p
preprocessingg
z Data preparation includes
– Data cleaning
g and data integration
g
– Data reduction and feature selection
– Discretization

z A lot a methods have been developed but data

preprocessing still an active area of research