DATA MINING

Data Preprocessing
Lec 3
Outlines
• Why preprocess the data?

• Data cleaning

• Data integration and transformation

• Data reduction

• Discretization and concept hierarchy generation

Why Data Preprocessing?

Low Quality Low Quality

Data Mining
Data Results
Why Data Preprocessing?
• Data in the real world is dirty
• incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate data
• e.g., occupation=“ ” (missin data)
• noisy: containing errors or outliers
• e.g., Salary=“-10”
• inconsistent: containing discrepancies in codes or
names
• e.g., Age=“42” Birthday=“03/07/1997”
• e.g., Was rating “1,2,3”, now rating “A, B, C”
• e.g., discrepancy between duplicate records
Why Is Data Dirty?
• Incomplete data may come from
• “Not applicable” data value when collected
• Different considerations between the time when the data was
collected and when it is analyzed.
• Human/hardware/software problems
• Noisy data (incorrect values) may come from
• Faulty data collection instruments
• Human or computer error at data entry
• Errors in data transmission
• Inconsistent data may come from
• Different data sources
• Functional dependency violation (e.g., modify some linked data)
• Duplicate records also need data cleaning
Why Is Data Preprocessing Important?
• No quality data, no quality mining results!
• Quality decisions must be based on quality data
• e.g., duplicate or missing data may cause incorrect or even
misleading statistics.
• Data warehouse needs consistent integration of quality
data
• Data extraction, cleaning, and transformation comprises
the majority of the work of building a data warehouse
Multi-Dimensional Measure of Data
Quality
• A well-accepted multidimensional view:
• Accuracy :correct or wrong
• Completeness : not recorded, unavailable
• Consistency : inonsistant naming, codring , format
• Timeliness : timely updated
• Believability : how much the data are trusted b users?
• Value added
• Interpretability : how easy the data are understood?
Major Tasks in Data Preprocessing
• Data cleaning
• Fill in missing values, smooth noisy data, identify or remove outliers,
and resolve inconsistencies
• Data integration
• Integration of multiple databases, data cubes, or files
• Data transformation
• Normalization and aggregation
• Data reduction
• Obtains reduced representation in volume but produces the same or
similar analytical results
• Data discretization
• Part of data reduction but with particular importance, especially for
numerical data e.g, age 1990 ~ 90 ,1980 ~ 80
Forms of Data Preprocessing
Data Preprocessing
• Why preprocess the data?

• Data cleaning

• Data integration and transformation

• Data reduction

• Discretization and concept hierarchy generation

Data Cleaning
• Importance
• “Data cleaning is one of the three biggest problems in
data warehousing”—Ralph Kimball
• “Data cleaning is the number one problem in data
warehousing”—DCI survey
• Data cleaning tasks

• Fill in missing values

• Identify outliers and smooth out noisy data

• Correct inconsistent data

• Resolve redundancy caused by data integration

Missing Data
• Data is not always available
• E.g., many tuples have no recorded value for several attributes,
such as customer income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• Missing data may need to be inferred.
How to Handle Missing Data?
• Ignore the tuple: usually done when class label is missing (assuming

the tasks in classification—not effective when the percentage of

missing values per attribute varies considerably.
• Fill in the missing value manually: tedious + infeasible?

• Fill in it automatically with

• a global constant : e.g., “unknown”, a new class?!

• the attribute mean

• the attribute mean for all samples belonging to the same class:

smarter
Noisy Data
• Noise: random error or variance in a measured variable
• Incorrect attribute values may due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
• inconsistency in naming convention
How to Handle Noisy Data?
• Binning
• first sort data and partition into (equal-frequency) bins
• then one can smooth by bin means, smooth by bin
median, smooth by bin boundaries, etc.
• Regression
• smooth by fitting the data into regression functions
• Clustering
• detect and remove outliers
• Combined computer and human inspection
• detect suspicious values and check by human (e.g.,
deal with possible outliers)
Simple Discretization Methods: Binning
• Equal-width (distance) partitioning

• Divides the range intoN intervals of equal size: uniform grid

• ifA andB are the lowest and highest values of the attribute, the

width of intervals will be:W =B( A N.

– )/
• The most straightforward, but outliers may dominate presentation

• Skewed data is not handled well

• Equal-depth (frequency) partitioning

• Divides the range intoN intervals, each containing approximately

same number of samples

• Good data scaling
Binning Methods for Data Smoothing
Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25,
26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Regression
Cluster Analysis
Data Preprocessing
• Why preprocess the data?

• Data cleaning

• Data integration and transformation

• Data reduction

• Discretization and concept hierarchy generation

Data Integration
• Data integration:
• Combines data from multiple sources into a coherent
store
• Schema integration: e.g., A.cust-id  B.cust-#
• Integrate metadata from different sources

• Challenges:
• Entity identification problem:
• How to match schemas and objects from different sources ?
• Redundancy and correlation analysis : are any attributes
correlated?
Entity identification problem
• Identify real world entities from multiple data sources are
tricky

• Same attribute or object may have different names in

different database.
• Attribute name : e.g, A.Student_id = B.Std_no
• Attribute values :e.g., Bill Clinton = William Clinton

• Detecting and resolving data value conflicts

• Possible reasons: different representations, different scales, e.g.,
metric vs. British units

• Metadata can be used to avoid errors in schema integration

• Example of metadata for attribute : name , data type, and range of
values permitted.
Handling Redundancy in Data
Integration
• Redundant data occur often when integration of multiple
databases
• Causes of redundancy :
• An attribute may be redundant if it can be derived from another
attribute or set of attributes.
• Inconsistencies in attribute naming can also cause redundancies in
the resulting dataset.
• Careful integration of the data from multiple sources may
help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
Correlation Analysis (Numerical Data)
• Correlation coefficient (also called Pearson’s product moment
coefficient)

rA,B 
 ( A  A )( B  B )

 ( AB )  n A B

(n  1)  
A B (n  1)  
A B

where n is the number of tuples, and are the respective means of A

and B, σA and σB are the respective standard deviation of A and B,
and Σ(AB) is the sum of the AB cross-product.
• If rA,B > 0, A and B are positively correlated
• (A’s values increase as B’s). The higher, the stronger correlation.
• rA,B = 0: independent;
• rA,B < 0: negatively correlated
Data Preprocess
• Why preprocess the data?

• Data cleaning

• Data integration

• Data transformation

• Data reduction
Data Transformation
• Data are transformed or consolidated into forms
appropriate for mining.
• The resulting mining process may be more efficient, and
the patterns found may be easier to understand
• Attribute Transformation: A function that maps
the entire set of values of a given attribute to a
new set of replacement values.
Data Transformation
• Data transformation strategies :
• Smoothing: remove noise from data
• Aggregation: summarization, data cube construction
• e.g.,daily sales data may be aggregated to monthly or annual sales

• Generalization: concept hierarchy climbing

• e.g.,streets Generalized to city or country

• Normalization: scaled to fall within a small, specified range

• min-max normalization
• z-score normalization
• normalization by decimal scaling
• Attribute/feature construction
• New attributes constructed from the given ones
Data Transformation: Normalization
• Min-max normalization: to [new_minA, new_maxA]

 min
v
v'  ( new _ max A  new _ min A )  new
A
_ min A

max  min
A A

• Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0].

73 , 600  12 , 000
(1 . 0  0 )  0  0 . 716
Then $73,000 is mapped to 98 , 000  12 , 000

• Z-score normalization (μ: mean, σ: standard deviation):

v 
v' 
A

 A

73 , 600  54 , 000
 1 . 225
• Ex. Let μ = 54,000, σ = 16,000. Then 16 , 000

• Normalization by decimal scaling

v
v'  Where j is the smallest integer such that Max(|ν’|) <
10
j
1
Data Preprocessing
• Why preprocess the data?

• Data cleaning

• Data integration

• Data transformation

• Data reduction
Data Reduction Strategies
• Why data reduction?
• A database/data warehouse may store terabytes of data
• Complex data analysis/mining may take a very long time to run on
the complete data set
• Data reduction
• Obtain a reduced representation of the data set that is much
smaller in volume but yet produce the same (or almost the same)
analytical results
• Data reduction strategies
• Data cube aggregation:
• Dimensionality reduction — e.g., remove unimportant attributes
• Sampling
• Data Compression
Data Reduction: Dimensionality
reduction
 Curse of dimensionality
 When dimensionality increases, data becomes increasingly
sparse
 Density and distance between points, which is critical to
clustering, outlier analysis, becomes less meaningful
 The possible combinations of subspaces will grow
exponentially
 Dimensionality reduction
 Avoid the curse of dimensionality
 Help eliminate irrelevant features and reduce noise
 Reduce time and space required in data mining
 Allow easier visualization
Data Reduction: Dimensionality
reduction
• Dimensionality reduction is the process of
reducing the number of attributes under
consideration including :
• Data compression techniques
• Attribute subset selection which removes irrelevant,
weakly relevant, or redundant attribute or dimentions.
• Why ?
• To improve quality and efficient of the mining process.
Data Reduction: sampling
• Sampling obtaining a small sample s to represent the
whole data set N.
• Choose a representative subset of the data.

• Statisticians sample because obtaining the entire set of data of

interest is too expensive or time consuming.
• Example: What is the average height of a person in Ioannina?
• We cannot measure the height of everybody

• Sampling is used in data mining because processing the entire

set of data of interest is too expensive or time consuming.
• Example: We have 1M documents. What fraction has at least 100 words in
common?
• Computing number of common words for all pairs requires 1012 comparisons
• Example: What fraction of tweets in a year contain the word “Greece”?
• 300M tweets per day, if 100 characters on average, 86.5TB to store all tweets
Data Reduction: sampling
• The key principle for effective sampling is the
following:
• using a sample will work almost as well as using the
entire data sets, if the sample is representative

• A sample is representative if it has approximately the

same property (of interest) as the original set of data
Types of Sampling
• Simple Random Sampling
• There is an equal probability of selecting any particular item

• Sampling without replacement

• As each item is selected, it is removed from the population

• Sampling with replacement

• Objects are not removed from the population as they are selected for the sample.

• Stratified sampling
• Split the data into several groups; then draw random samples from
each group.
• Ensures that both groups are represented.
Sample Size

8000 points 2000 Points 500 Points

Stratified sampling

Raw Data Cluster/Stratified Sample

Data Reduction: data cube aggregation
• data cube aggregation data can be aggregated so
that the resulting data summarize the data into
smaller volume, without loss of information
necessary for the analysis task.
Data Reduction: data compression
• Obtain a reduced or compressed representation
of the original data.
• Two types:
• Lossless :
• If the original data can be reconstructed from the compressed
data without any information loss.
• Lossy :
• If we can reconstruct only an approximation of the original data
Original Data Compressed
Data
lossless

ssy
lo
Original Data
Approximated
ANY QUESTIONS