Scribd
Upload
45 views

Integration and Normalization

This document discusses methods for collecting, integrating, and normalizing data from multiple sources. It describes how data can be collected manually or using computer peripherals, and classified as internal, personal, or external sources. The key challenges of data integration are entity identification, redundancy, data conflicts, and duplication. Redundancy can be detected using correlation analysis. Data normalization techniques include min-max normalization, z-score normalization, and decimal scaling to standardize data within a specified range.

Uploaded by

Mahmoud Elnahas
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
45 views

Integration and Normalization

This document discusses methods for collecting, integrating, and normalizing data from multiple sources. It describes how data can be collected manually or using computer peripherals, and classified as internal, personal, or external sources. The key challenges of data integration are entity identification, redundancy, data conflicts, and duplication. Redundancy can be detected using correlation analysis. Data normalization techniques include min-max normalization, z-score normalization, and decimal scaling to standardize data within a specified range.

Uploaded by

Mahmoud Elnahas
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 19

Data Integration and Normalization

Methods for Collecting Raw Data


2

◻Manually
⬜Using Surveys
⬜Using Observations
⬜Based on Contributions from experts
◻Using special computer peripherals equipments
(instruments and/or sensors)
⬜Bar-code readers
⬜Computer-based medical imaging system
⬜Remote sensing instruments
⬜Atmosphere sensing instruments
◻Web-based systems
⬜Becomes very popular to collect clients’ feedback via web
⬜Examples are online polls and questionnaires
Data Sources Classifications
3
⬜Internal data sources
■ Internal sources are data sources that are housed within a company
■ An organization’s internal data are about people, products, services,
and processes
⬜Personal data sources
■ Employees may document their own expertise by creating personal
data. These data are not necessarily just facts, but may include
concepts, thoughts, and opinions. They include, for example,
opinions about what competitors are likely to do, and certain rules and
formulas developed by the users
⬜External data sources
■ There are many sources for external data, ranging from commercial
databases to data collected by sensors and satellites
■ Internet and commercial database sources also considered as external
sources
Data integration
4

◻Data Integration means that we need to merge data


coming from multiple sources
◻This is normally happened in data warehousing and
data mining systems
Issues to be considered during data integration
5

◻Entity Identification Problem


◻Redundancy
◻Data Value Conflicts
◻Duplication
Entity Identification Problem
6

◻Means that we have attributes (coming from different


data sources) with different names that represent the
same entity
◻Example: we may have all the following attributes that
represent the same entity:
⬜Student_No
⬜Bench_No
⬜Student_Id
◻All of these attributes could represent the same entity
◻During data integration these attributes must be
identified to reduce data redundancy and inconsistency
Redundancy
7

◻An attribute is redundant if it can be derived from


another attribute
◻Some redundancy can be detected by correlation
analysis
Detecting Redundancies using Correlation
Analysis
8

◻Consider two attributes A and B coming from two


different data sources. The correlation between attributes
A and B can be measured by the correlation coefficient
rA,B:

Where

and

◻Where n is the number of data items in A or B (assuming


they have the same number of data items), and are
the mean values of A and B, and and are the
standard deviations of A and B
Detecting Redundancies using Correlation
Analysis
9

◻If the rA,B is greater than 0 then A and B are


positively correlated
◻If A and B are positively correlated then the values
of A increase as the values of B increase
◻The higher the value of rA,B the more each attribute
implies the other
◻So, a high value of rA,B implies that either A or B
may be removed as a redundancy but not both of
them
Detecting Redundancies using Correlation
Analysis
10

◻If the rA,B is 0, then A and B are independent, hence


there is no redundancy and we cannot remove any
of them
◻If rA,B is less than 0 then A and B are negatively
correlated
◻This implies that the values of one attribute
increase as the values of the other decrease and also
we cannot remove any of the attributes
Data Value Conflicts
11

◻ For the same entity, attribute values from different sources may
differ
◻ This may be due to differences in representation, scaling, or
encoding
◻ Example: a weight attribute may be stored in metric units (meter,
kilogram …) in one system and British units (Pound, inch …) in
another
◻ The price of different hotels may involve not only different
currencies but also different services and taxes
◻ Hence, these conflicts must be resolved before integrating data
sources
◻ Resolving these kinds of conflicts is done by converting one unit to
the other so that we use a unified unit for all integrated data sources
◻ Careful integration of data from multiple sources helps in reducing
and avoiding redundancy and inconsistency in the resulting data sets
Duplication
12

◻Duplication means there are two or more identical


records for a given unique data entry
◻It occurs when you have the same record coming
from different data sources
◻Resolving it can be done by discarding all
duplication and leaving single unique data entry for
each entity
Data Normalization
13

◻Data Normalization means that the attribute data


are scaled to fall within a small specified range
such as –1.0 to +1.0, or 0.0 to 1.0
◻Normalization is important for data classification
and analysis by data mining techniques
Min-Max Normalization
14

◻Min-Max normalization performs a linear


transformation on the original data
◻Suppose that minA and maxA are the minimum and
maximum values of an attribute A
◻Min-Max normalization maps any value v of A to v’ in
the range[new_minA, new_maxA] by computing:

◻Min-max normalization preserves the relationships


among the original data values
Min-Max Normalization: Example
15

◻Let the minimum and maximum values for the


attribute income be $12000 and $98000
◻To map income to the range [0.0, 1.0], we have:
minA = 12000 maxA = 98000
new_minA = 0.0 new_maxA = 1.0
◻By min-max normalization, a value of $73600 for
income is transformed to:
Z-Score Normalization
16

◻In Z-Score normalization, the values for an attribute A


are normalized based on the mean and standard
deviation of A
◻A value v of A is normalized to v’ by computing:

Where and

◻Where is the mean value of A and is the standard


deviation of A
◻The Z-Score normalization method is useful when the
actual minimum and maximum of attribute A are
unknown, or when there are outliers that dominate the
min-max normalization
Z-Score Normalization: Example
17

◻Suppose that the mean and standard deviation of the


values for the attribute income are $54000 and $16000,
respectively
◻With Z-Score normalization, a value of $73600 for
income is transformed to
Normalization by decimal scaling
18

◻This method normalizes by moving the decimal point


of values of attribute A
◻The number of decimal points moved depends on the
maximum absolute value of A
◻A value v of A is normalized to v’ by computing:

◻Where j is the smallest integer such that Max(|v'|) < 1


Normalization by decimal scaling:
Example
19

◻Let the values of A range from -986 to 917


◻The maximum absolute value of A is 986
◻To normalize by decimal scaling, we divide each value
by 1000, which means j = 3
◻Hence, -986 normalizes to -0.986

You might also like