Based on slides from Han J., et. al.

2013

1
Chapter 3: Data Preprocessing
Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
2
2
Data Quality: Why Preprocess the Data?

Measures for data quality: A multidimensional view

Accuracy: correct or wrong, accurate or not

Completeness: not recorded, unavailable, …

Consistency: some modified but some not, dangling, …

Timeliness: timely update?

Believability: how trustable the data are correct?

Interpretability: how easily the data can be understood?

3
Major Tasks in Data Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data reduction
Dimensionality reduction
Numerosity reduction
Data compression
Data transformation and data discretization
Normalization
Concept hierarchy generation
4
Chapter 3: Data Preprocessing
Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
5
5
Data Cleaning
Data in the Real World Is Dirty: Lots of potentially incorrect data, e.g.,
instrument faulty, human or computer error, transmission error
 incomplete: lacking attribute values, lacking certain attributes of interest,
or containing only aggregate data
 e.g., Occupation = “ ” (missing data)

 noisy: containing noise, errors, or outliers

 e.g., Salary = “−10” (an error)

 inconsistent: containing discrepancies in codes or names, e.g.,

 Age = “42”, Birthday = “03/07/2010”

 Was rating “1, 2, 3”, now rating “A, B, C”

 discrepancy between duplicate records

 Intentional (e.g., disguised missing data)

 Jan. 1 as everyone’s birthday?

6
Incomplete (Missing) Data
Data is not always available
E.g., many tuples have no recorded value for several
attributes, such as customer income in sales data
Missing data may be due to
equipment malfunction
inconsistent with other recorded data and thus deleted
data not entered due to misunderstanding
certain data may not be considered important at the time of
entry
not register history or changes of the data
Missing data may need to be inferred
7
How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing (when
doing classification)—not effective when the % of missing values
per attribute varies considerably
Fill in the missing value manually: tedious + infeasible?
Fill in it automatically with
a global constant : e.g., “unknown”, a new class?!
the attribute mean
the attribute mean for all samples belonging to the same class:
smarter
the most probable value: inference-based such as Bayesian

8
formula or decision tree
Noisy Data
Noise: random error or variance in a measured variable
Incorrect attribute values may be due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
Other data problems which require data cleaning
duplicate records
incomplete data
inconsistent data

9
How to Handle Noisy Data?
Binning
first sort data and partition into (equal-frequency) bins
then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression functions
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human (e.g., deal with
possible outliers)

10
Data Cleaning as a Process
 Data discrepancy detection
 Use metadata (e.g., domain, range, dependency, distribution)
 Check field overloading
 Check uniqueness rule, consecutive rule and null rule
 Use commercial tools
 Data scrubbing: use simple domain knowledge (e.g., postal code, spell-
check) to detect errors and make corrections
 Data auditing: by analyzing data to discover rules and relationship to

detect violators (e.g., correlation and clustering to find outliers)

 Data migration and integration
 Data migration tools: allow transformations to be specified
 ETL (Extraction/Transformation/Loading) tools: allow users to specify
transformations through a graphical user interface
 Integration of the two processes
 Iterative and interactive (e.g., Potter’s Wheels)
11
Chapter 3: Data Preprocessing

Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
12
12
Data Integration
 Data integration:
 Combines data from multiple sources into a coherent store

 Schema integration: e.g., A.cust-id  B.cust-#

 Integrate metadata from different sources

 Entity identification problem:

 Identify real world entities from multiple data sources, e.g., Bill Clinton =

William Clinton
 Detecting and resolving data value conflicts
 For the same real world entity, attribute values from different sources are

different
 Possible reasons: different representations, different scales, e.g., metric vs.

British units
13
13
Handling Redundancy in Data Integration
Redundant data occur often when integration of multiple
databases
Object identification: The same attribute or object may have
different names in different databases
Derivable data: One attribute may be a “derived” attribute
in another table, e.g., annual revenue
Redundant attributes may be able to be detected by correlation
analysis and covariance analysis
Careful integration of the data from multiple sources may help
reduce/avoid redundancies and inconsistencies and improve
mining speed and quality
14
14
Correlation Analysis (Nominal Data)
Χ2 (chi-square) test
2
(Observed  Expected )
 2 
Expected
The larger the Χ2 value, the more likely the variables are related
The cells that contribute the most to the Χ2 value are those
whose actual count is very different from the expected count
Correlation does not imply causality
# of hospitals and # of car-theft in a city are correlated
Both are causally linked to the third variable: population

15
Chi-Square Calculation: An Example
Play Not play Sum
chess chess (row)
Like science fiction 250(90) 200(360) 450
Not like science 50(210) 1000(840) 1050
fiction
Sum(col.) 300 1200 1500

Χ2 (chi-square) calculation (numbers in parenthesis are expected

counts calculated based on the data distribution in the two
categories)
(250  90) 2 (50  210) 2 (200  360) 2 (1000  840) 2
 
2
   507.93
90 210 360 840
It shows that like_science_fiction and play_chess are correlated
in the group
16
Correlation Analysis (Numeric Data)
Correlation coefficient (also called Pearson’s product moment
coefficient)

i 1 (ai  A)(bi  B) 
n n
(ai bi )  n A B
rA, B   i 1
(n  1) A B (n  1) A B

where n is the number of tuples, Aand B are the respective

means of A and B, σA and σB are the respective standard
deviation of A and B, and Σ(aibi) is the sum of the AB cross-
product.
If rA,B > 0, A and B are positively correlated (A’s values increase
as B’s). The higher, the stronger correlation.
rA,B = 0: independent; rAB < 0: negatively correlated
17
Visually Evaluating Correlation

Scatter plots
showing the
similarity from
–1 to 1.

18
Correlation (viewed as linear relationship)
Correlation measures the linear relationship between
objects
To compute correlation, we standardize data objects, A
and B, and then take their dot product

a 'k (ak  mean( A)) / std ( A)

b'k (bk  mean( B )) / std ( B )

correlation( A, B)  A' B'

19
Covariance (Numeric Data)
 Covariance is similar to correlation

Correlation coefficient:

where n is the number of tuples, Aand Bare the respective mean or expected
values of A and B, σA and σB are the respective standard deviation of A and B
 Positive covariance: If CovA,B > 0, then A and B both tend to be larger than their
expected values
 Negative covariance: If CovA,B < 0 then if A is larger than its expected value, B is
likely to be smaller than its expected value
 Independence: Cov = 0 but the converse is not true:
A,B
 Some pairs of random variables may have a covariance of 0 but are not independent.
Only under some additional assumptions (e.g., the data follow multivariate normal
distributions) does a covariance of 0 imply independence
20
Co-Variance: An Example
 It can be simplified in computation as

 Suppose two stocks A and B have the following values in one week: (2, 5), (3,

8), (5, 10), (4, 11), (6, 14).

 Question: If the stocks are affected by the same industry trends, will their

prices rise or fall together?

 E(A) = (2 + 3 + 5 + 4 + 6)/ 5 = 20/5 = 4

 E(B) = (5 + 8 + 10 + 11 + 14) /5 = 48/5 = 9.6

 Cov(A,B) = (2×5+3×8+5×10+4×11+6×14)/5 − 4 × 9.6 = 4

 Thus, A and B rise together since Cov(A, B) > 0.

Chapter 3: Data Preprocessing
Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
22
22
Data Reduction Strategies
 Data reduction: Obtain a reduced representation of the data set that is much
smaller in volume but yet produces the same (or almost the same) analytical
results
 Why data reduction? — A database/data warehouse may store terabytes of
data. Complex data analysis may take a very long time to run on the complete
data set.
 Data reduction strategies
 Dimensionality reduction, e.g., remove unimportant attributes
 Wavelet transforms
 Principal Components Analysis (PCA)
 Feature subset selection, feature creation
 Numerosity reduction (some simply call it: Data Reduction)
 Regression and Log-Linear Models
 Histograms, clustering, sampling
 Data cube aggregation
 Data compression
23
Chapter 3: Data Preprocessing
Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
24
Data Transformation
 A function that maps the entire set of values of a given attribute to a new set
of replacement values s.t. each old value can be identified with one of the
new values
 Methods
 Smoothing: Remove noise from data
 Attribute/feature construction
 New attributes constructed from the given ones
 Aggregation: Summarization, data cube construction
 Normalization: Scaled to fall within a smaller, specified range
 min-max normalization
 z-score normalization
 normalization by decimal scaling
 Discretization: Concept hierarchy climbing
25
Normalization
 Min-max normalization: to [new_minA, new_maxA]
v  minA
v'  (new _ maxA  new _ minA)  new _ minA
maxA  minA
 Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0].
73,600  12,000
(1.0  0)  0 0.716
Then $73,000 is mapped to 98,000  12,000

 Z-score normalization (μ: mean, σ: standard deviation):

v  A
v' 
 A

73,600  54,000
1.225
 Ex. Let μ = 54,000, σ = 16,000. Then 16,000
 Normalization by decimal scaling
v
v'  j Where j is the smallest integer such that Max(|ν’|) < 1
10
26
 Discretization
Three types of attributes
 Nominal—values from an unordered set, e.g., color, profession
 Ordinal—values from an ordered set, e.g., military or academic rank
 Numeric—real numbers, e.g., integer or real numbers

Discretization: Divide the range of a continuous attribute into intervals

 Interval labels can then be used to replace actual data values
 Reduce data size by discretization
 Supervised vs. unsupervised
 Split (top-down) vs. merge (bottom-up)
 Discretization can be performed recursively on an attribute
 Prepare for further analysis, e.g., classification
27
Data Discretization Methods
Typical methods: All the methods can be applied recursively
Binning
 Top-down split, unsupervised
Histogram analysis
 Top-down split, unsupervised
Clustering analysis (unsupervised, top-down split or bottom-up
merge)
Decision-tree analysis (supervised, top-down split)
Correlation (e.g., 2) analysis (unsupervised, bottom-up merge)

28
Simple Discretization: Binning
 Equal-width (distance) partitioning
 Divides the range into N intervals of equal size: uniform grid

 if A and B are the lowest and highest values of the attribute, the width of

intervals will be: W = (B –A)/N.

 The most straightforward, but outliers may dominate presentation

 Skewed data is not handled well

 Equal-depth (frequency) partitioning

 Divides the range into N intervals, each containing approximately same

number of samples
 Good data scaling

 Managing categorical attributes can be tricky

29
Binning Methods for Data Smoothing
 Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34

30
Discretization Without Using Class Labels
(Binning vs. Clustering)

Data Equal interval width

(binning)

Equal frequency (binning) K-means clustering leads to better

results
31
Discretization by Classification & Correlation
Analysis
 Classification (e.g., decision tree analysis)

 Supervised: Given class labels, e.g., cancerous vs. benign

 Using entropy to determine split point (discretization point)

 Top-down, recursive split

 Details to be covered in Chapter “Classification”

 Correlation analysis (e.g., Chi-merge: χ2-based discretization)

 Supervised: use class information

 Bottom-up merge: find the best neighboring intervals (those having

similar distributions of classes, i.e., low χ2 values) to merge

 Merge performed recursively, until a predefined stopping condition

32
Concept Hierarchy Generation
Concept hierarchy organizes concepts (i.e., attribute values)
hierarchically and is usually associated with each dimension in a data
warehouse
Concept hierarchies facilitate drilling and rolling in data warehouses to
view data in multiple granularity
Concept hierarchy formation: Recursively reduce the data by collecting
and replacing low level concepts (such as numeric values for age) by
higher level concepts (such as youth, adult, or senior)
Concept hierarchies can be explicitly specified by domain experts
and/or data warehouse designers
Concept hierarchy can be automatically formed for both numeric and
nominal data—For numeric data, use discretization methods shown
33
Concept Hierarchy Generation for Nominal Data
Specification of a partial/total ordering of attributes explicitly at
the schema level by users or experts
street < city < state < country
Specification of a hierarchy for a set of values by explicit data
grouping
{Urbana, Champaign, Chicago} < Illinois
Specification of only a partial set of attributes
E.g., only street < city, not others
Automatic generation of hierarchies (or attribute levels) by the
analysis of the number of distinct values
E.g., for a set of attributes: {street, city, state, country}
34
Automatic Concept Hierarchy Generation
Some hierarchies can be automatically generated based on the
analysis of the number of distinct values per attribute in the
data set
The attribute with the most distinct values is placed at the
lowest level of the hierarchy
Exceptions, e.g., weekday, month, quarter, year

country 15 distinct values

province_or_ state 365 distinct values

city 3567 distinct values

street 674,339 distinct values

35
Chapter 3: Data Preprocessing
Data Preprocessing: An Overview

Data Quality

Major Tasks in Data Preprocessing

Data Cleaning

Data Integration

Data Reduction

Data Transformation and Data Discretization

Summary
36
Summary
Data quality: accuracy, completeness, consistency, timeliness,
believability, interpretability
Data cleaning: e.g. missing/noisy values, outliers
Data integration from multiple sources:
Entity identification problem; Remove redundancies; Detect
inconsistencies
Data reduction
Dimensionality reduction; Numerosity reduction; Data
compression
Data transformation and data discretization
Normalization; Concept hierarchy generation
37
References
 D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments.
Comm. of ACM, 42:73-78, 1999
 T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley,
2003
 T. Dasu, T. Johnson, S. Muthukrishnan, V. Shkapenyuk.
Mining Database Structure; Or, How to Build a Data Quality Browser. SIGMOD’02
 H. V. Jagadish et al., Special Issue on Data Reduction Techniques. Bulletin of the
Technical Committee on Data Engineering, 20(4), Dec. 1997
 D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999
 E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE
Bulletin of the Technical Committee on Data Engineering. Vol.23, No.4
 V. Raman and J. Hellerstein. Potters Wheel: An Interactive Framework for Data
Cleaning and Transformation, VLDB’2001
 T. Redman. Data Quality: Management and Technology. Bantam Books, 1992
 R. Wang, V. Storey, and C. Firth. A framework for analysis of data quality research.
IEEE Trans. Knowledge and Data Engineering, 7:623-640, 1995

38