Week 5:

Week 5: Data Mining: Exploring Data with R

CS ELEC 2: Data Mining

Jennifer L. Llovido, DIT
 Week 5

1. Exploratory Data Analysis

2. Basic Statistical Descriptions of Data
3. Data Visualization
4. Introduction to R
 Data Mining: Data

Lecture Notes for Chapter 2

Slides by Tan, Steinbach, Kumar adapted by Michael Hahsler

Look for accompanying R

code on the course web site.
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
What is Data?
▪ Collection of data objects Attributes
and their attributes

▪ An attribute (in Data Mining Tid Refund Marital Taxable Cheat

Status Income
and Machine learning often
"feature") is a property or 1 Yes Single 125K No
characteristic of an object 2 No Married 100K No
• Examples: eye color of a 3 No Single 70K No
person, temperature, etc. 4 Yes Married 120K No

Objects
• Attribute is also known as 5 No Divorced 95K Yes
variable, field, characteristic
6 No Married 60K No
▪ A collection of attributes 7 Yes Divorced 220K No
describe an object
8 No Single 85K Yes
• Object is also known as
record, point, case, sample, 9 No Married 75K No
entity, or instance 10 No Single 90K Yes
Attribute Values

▪ Attribute values are numbers or symbols assigned to an attribute

▪ Distinction between attributes and attribute values

—Same attribute can be mapped to different attribute values
• Example: height can be measured in feet or meters

—Different attributes can be mapped to the same set of values

• Example: Attribute values for ID and age are integers
• But properties of attribute values can be different
• ID has no limit but age has a maximum and minimum value
Types of Attributes - Scales

▪ There are different types of attributes

—Nominal
• Examples: ID numbers, eye color, zip codes Categorical,
—Ordinal Qualitative
• Examples: rankings (e.g., taste of potato chips on a scale
from 1-10), grades, height in {tall, medium, short}
—Interval
• Examples: calendar dates, temperatures in Celsius or
Fahrenheit.
Quantitative
—Ratio
• Examples: temperature in Kelvin, length, time, counts
Attribute Description Examples Operations
Type

Nominal The values of a nominal attribute are zip codes, employee mode, entropy,
just different names, i.e., nominal ID numbers, eye color, contingency
attributes provide only enough sex: {male, female} correlation, 2 test
information to distinguish one object
from another. (=, )

Ordinal The values of an ordinal attribute hardness of minerals, median, percentiles,

provide enough information to order {good, better, best}, rank correlation,
objects. (<, >) grades, street numbers run tests, sign tests

Interval For interval attributes, the calendar dates, mean, standard

differences between values are temperature in Celsius deviation, Pearson's
meaningful, i.e., a unit of or Fahrenheit correlation, t and F
measurement exists. tests
(+, - )

Ratio For ratio variables, both differences temperature in Kelvin, geometric mean,
and ratios are meaningful. (*, /) monetary quantities, harmonic mean,
counts, age, mass, percent variation
length, electrical
current
Attribute Transformation Comments
Level

Nominal Any permutation of values If all employee ID numbers

were reassigned, would it
make any difference?

Ordinal An order preserving change of An attribute encompassing

values, i.e., the notion of good, better
new_value = f(old_value) best can be represented
where f is a monotonic function. equally well by the values
{1, 2, 3} or by { 0.5, 1,
10}.
Interval new_value =a * old_value + b Thus, the Fahrenheit and
where a and b are constants Celsius temperature scales
differ in terms of where
their zero value is and the
size of a unit (degree).

Ratio new_value = a * old_value Length can be measured in

meters or feet.
Discrete and Continuous Attributes

▪ Discrete Attribute
—Has only a finite or countably infinite set of values
—Examples: zip codes, counts, or the set of words in a collection of documents
—Often represented as integer variables.
—Note: binary attributes are a special case of discrete attributes

▪ Continuous Attribute
—Has real numbers as attribute values
—Examples: temperature, height, or weight.
—Practically, real values can only be measured and represented using a finite
number of digits.
—Continuous attributes are typically represented as floating-point variables.
Examples

▪ What is the scale of measurement of:

—Number of cars per minute (count data)

—Age data grouped in:

0-4 years, 5-9, 10-14, …

—Age data grouped in:

<20 years, 21-30, 31-40, 41+
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Types of data sets

▪ Record
—Data Matrix
—Document Data
—Transaction Data
▪ Graph
—World Wide Web
—Molecular Structures
▪ Ordered
—Spatial Data
—Temporal Data
—Sequential Data
—Genetic Sequence Data
Important Characteristics of Structured Data

▪ Dimensionality
— Curse of Dimensionality

▪ Sparsity
— Only presence counts

▪ Resolution
— Patterns depend on the scale
Record Data

▪ Data that consists of a collection of records, each of which consists of

a fixed set of attributes (e.g., from a relational database)

Tid Refund Marital Taxable Cheat

Status Income
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
Data Matrix

▪ If data objects have the same fixed set of numeric attributes, then
the data objects can be thought of as points in a multi-dimensional
space, where each dimension represents a distinct attribute
▪ Such data set can be represented by an m by n matrix, where there
are m rows, one for each object, and n columns, one for each
attribute
n attributes
Sepal.Length Sepal.Width Petal.Length Petal.Width

5.6 2.7 4.2 1.3

m objects

6.5 3.0 5.8 2.2

6.8 2.8 4.8 1.4
5.7 3.8 1.7 0.3
5.5 2.5 4.0 1.3
4.8 3.0 1.4 0.1
5.2 4.1 1.5 0.1
Document Data

▪ Each document becomes a `term' vector,

—each term is a component (attribute) of the vector,
—the value of each component is the number of times the corresponding term
occurs in the document.
Terms

timeout

season
coach

game
score
team

ball

lost
pla

wi
n
y

Document 1 3 0 5 0 2 6 0 2 0 2

Document 2 0 7 0 2 1 0 0 3 0 0

Document 3 0 1 0 0 1 2 2 0 3 0
Transaction Data

▪ A special type of record data, where

—each record (transaction) involves a set of items.
—For example, consider a grocery store. The set of products purchased by a
customer during one shopping trip constitute a transaction, while the
individual products that were purchased are the items.

TID Items
1 Bread, Coke, Milk
2 Beer, Bread
3 Beer, Coke, Diaper, Milk
4 Beer, Bread, Diaper, Milk
5 Coke, Diaper, Milk
Graph Data
▪ Examples: Generic graph and HTML Links

<a href="papers/papers.html#bbbb">
Data Mining </a>
<li>
2 <a href="papers/papers.html#aaaa">
Graph Partitioning </a>
<li>
5 1 <a href="papers/papers.html#aaaa">
Parallel Solution of Sparse Linear System of Equations </a>
<li>
2 <a href="papers/papers.html#ffff">
N-Body Computation and Dense Linear System Solvers
5
 Graph Data

Examples: Generic graph, a molecule, and webpages

2
5 1
2
5

Benzene Molecule: C6H6

01/27/2021 Introduction to Data Mining, 2nd 21
Chemical Data

▪ Benzene Molecule: C6H6

Ordered Data

▪ Sequences of transactions

Items/Events

An element of
the sequence
Ordered Data

▪ Genomic sequence data

GGTTCCGCCTTCAGCCCCGCGCC
CGCAGGGCCCGCCCCGCGCCGTC
GAGAAGGGCCCGCCTGGCGGGCG
GGGGGAGGCGGGGCCGCCCGAGC
CCAACCGAGTCCGACCAGGTGCC Subsequences
CCCTCTGCTCGGCCTAGACCTGA
GCTCATTAGGCGGCAGCGGACAG
GCCAAGTAGAACACGCGAAGCGC
TGGGCTGCCTGCTGCGACCAGGG
Ordered Data: Time Series Data
Ordered Data: Spatio-Temporal
, Feb, Mar, …

Average Monthly
Temperature of
land and ocean
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Data Quality

▪ What kinds of data quality problems?

▪ How can we detect problems with the data?
▪ What can we do about these problems?

▪ Examples of data quality problems:

—Noise and outliers
—missing values
—duplicate data
Noise
▪ Noise refers to modification of original values
—Examples: distortion of a person’s voice when talking on a poor phone,
“snow” on television screen, measurement errors.

Two Sine Waves Two Sine Waves + Noise

▪ Find less noisy data

▪ De-noise (signal processing)
Outliers

▪ Outliers are data objects with characteristics that are considerably

different than most of the other data objects in the data set
—Case 1: Outliers are
noise that interferes
with data analysis

—Case 2: Outliers are

the goal of our analysis
• Credit card fraud
• Intrusion detection

▪ Causes?
Missing Values

▪ Reasons for missing values

—Information is not collected
(e.g., people decline to give their age and weight)
—Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)

▪ Handling missing values

—Eliminate data objects with missing value
—Eliminate feature with missing values
—Ignore the missing value during analysis
—Estimate missing values = Imputation
(e.g., replace with mean or weighted mean where all possible values are
weighted by their probabilities)
Duplicate Data

▪ Data set may include data objects that are duplicates, or "close
duplicates" of one another
—Major issue when merging data from heterogeneous sources

▪ Examples:
—Same person with multiple email addresses

▪ Data cleaning
—Process of dealing with duplicate data issues
—ETL tools typically support deduplication
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Data Preprocessing

▪ Aggregation
▪ Sampling
▪ Dimensionality Reduction
▪ Feature subset selection
▪ Feature creation
▪ Discretization and Binarization
▪ Attribute Transformation
Aggregation

▪ Combining two or more attributes (or objects) into a single attribute

(or object)

▪ Purpose
—Data reduction
• Reduce the number of attributes or objects
—Change of scale
• Cities aggregated into regions, states, countries, etc
—More “stable” data
• Aggregated data tends to have less variability (e.g., reduce seasonality by aggregation to
yearly data)
Aggregation
Variation of Precipitation in Australia

Histogram of Standard Deviation of Histogram of Standard Deviation of

Average Monthly Precipitation Average Yearly Precipitation
Sampling

▪ Sampling is the main technique employed for data selection.

—It is often used for both the preliminary investigation of the data and the
final data analysis.

▪ Statisticians sample because obtaining the entire set of data of

interest is too expensive or time consuming.

▪ Sampling is used in data mining because processing the entire set of

data of interest is too expensive (e.g., does not fit into memory or is
too slow).
Sampling …

▪ The key principle for effective sampling is the following:

—using a sample will work almost as well as using the entire data sets, if the
sample is representative.

—A sample is representative if it has approximately the same property (of

interest) as the original set of data.
Types of Sampling
Replacement?

▪ Sampling without replacement

—As each item is selected, it is removed from the population

▪ Sampling with replacement

—Objects are not removed from the population as they are selected
for the sample. Note: the same object can be picked up more than
once

▪ Simple Random Sampling

Selection?

—There is an equal probability of selecting any particular item

▪ Stratified sampling
—Split the data into several partitions; then draw random samples
from each partition
Sample Size

8000 points 2000 Points 500 Points

Sample Size

▪ What sample size is necessary to get at least one object from each of 10
groups.

▪ Sample size determination:

—Statistics: confidence interval for parameter estimate or desired statistical power
of test.
—Machine learning: often more is better, cross-validated accuracy.
Curse of
Dimensionality
▪ When dimensionality
increases, the size of the
data space grows
exponentially.
Points and space

▪ Definitions of density and Experiment: Randomly generate 500 points.

distance between points, Compute difference between max and min
which is critical for distance between any pair of points
clustering and outlier
detection, become less
meaningful
• Density → 0
• All points tend to have the
same Euclidean distance to
each other.
Dimensionality Reduction

▪ Purpose:
—Avoid curse of dimensionality
—Reduce amount of time and memory required by data mining algorithms
—Allow data to be more easily visualized
—May help to eliminate irrelevant features or reduce noise

▪ Techniques
—Principle Component Analysis
—Singular Value Decomposition
—Others: supervised and non-linear techniques
Dimensionality Reduction: Principal
Components Analysis (PCA)
▪ Goal: Map points to a lower dimensional space while preserving
distance information.

▪ Method: Find a projection (new axes) that captures the largest

amount of variation in data. This can be done using eigenvectors of
the covariance matrix or SVD (singular value decomposition).
Dimensionality Reduction: ISOMAP

▪ Goal: Unroll the “swiss roll!“ (i.e., preserve distances on the roll)
▪ Method: Use a non-metric space, i.e., distances are not measured by
Euclidean distance, but along the surface of the roll (geodesic distances).
1. Construct a neighbourhood graph (k-nearest neighbors or within a radius).
2. For each pair of points in the graph, compute the shortest path distances =
geodesic distances.
3. Create a lower dimensional embedding using the geodesic distances (multi-
dimensional scaling; MDS)
Low-dimensional Embedding
▪ General notion of representing objects described in one space (i.e., set of features) in
a different space using a map 𝑓 ∶ 𝑋 → 𝑌
▪ PCA is an example where Y is the space spanned by the principal components and
objects close in the original space X are embedded in space Y.
▪ Low-dimensional embeddings can be produced with various other methods:
—T-SNA: T-distributed Stochastic Neighbor Embedding; non-linear for visualization of
high-dimensional datasets.
—Autoencoders (deep learning): non-linear
—Word embedding: Word2vec, GloVe, BERT

Word Embedding

Autoencoder
Feature Subset Selection

▪ Another way to reduce dimensionality of data

▪ Redundant features
—duplicate much or all of the information contained in one or more other
attributes (are correlated)
—Example: purchase price of a product and the amount of sales tax paid

▪ Irrelevant features
—contain no information that is useful for the data mining task at hand
—Example: students' ID is often irrelevant to the task of predicting students'
GPA
Feature Subset Selection

▪ Embedded approaches:
— Feature selection occurs naturally as part of the data mining algorithm (e.g.,
regression, decision trees).
▪ Filter approaches:
— Features are selected before data mining algorithm is run
—(e.g., highly correlated features)
▪ Brute-force approach:
—Try all possible feature subsets as input to data mining algorithm and choose
the best.
▪ Wrapper approaches:
— Use the data mining algorithm as a black box to find best subset of
attributes (often using greedy search)
Feature Creation

▪ Create new attributes that can capture the important information in

a data set much more efficiently than the original attributes

▪ Three general methodologies:

—Feature Extraction
• Domain-specific (e.g., face recognition in image mining)
—Feature Construction / Feature Engineering
• combining features (interactions: multiply features)
—Mapping Data to New Space
Mapping Data to a New Space
▪ Fourier transform
▪ Wavelet transform

Two Sine Waves Two Sine Waves + Noise Frequency

Unsupervised Discretization

Data Equal interval width

Equal frequency K-means

Supervised Discretization
▪ Entropy based approach (entropy is a measure of unorder). Bisect
data to reduce entropy.

A
B

C
D
3 categories for both x and y 5 categories for both x and y
Attribute Transformation

▪ A function that maps the entire set of values of a given attribute to a

new set of replacement values such that each old value can be
identified with one of the new values
—Simple functions: 𝑥 𝑘 , log(𝑥), 𝑒 𝑥 , |𝑥|
—Standardization and Normalization
The z-score normalizes data roughly to an interval of [−3,3].

′
𝑥 − 𝑥ҧ
𝑥 =
𝑠𝑥
𝑥ҧ … column (attribute) mean
𝑠𝑥 … column (attribute) standard deviation
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Similarity and Dissimilarity

▪ Similarity
—Numerical measure of how alike two data objects are.
—Is higher when objects are more alike.
—Often falls in the range [0,1]
▪ Dissimilarity
—Numerical measure of how different are two data objects
—Lower when objects are more alike
—Minimum dissimilarity is often 0
—Upper limit varies
▪ Proximity refers to a similarity or dissimilarity
Similarity/Dissimilarity for Simple Attributes
p and q are the attribute values for two data objects.

𝑠=𝑓 𝑑
f can be any strictly
decreasing function.
 point x y
Euclidean Distance p 0 2
q 2 0

▪ Euclidean Distance (for quantitative attribute vectors)

𝑛
𝑑𝐸 = ෍ 𝑝𝑘 − 𝑞𝑘 2 = 𝒑−𝒒 2
𝑘=1

— Where 𝒑 and 𝒒 are two objects represented by vectors. n is the number of

dimensions (attributes) of the vectors and 𝑝𝑘 and 𝑞𝑘 are, respectively, the
kth attributes (components) or data objects p and q.
— ⋅ 2 is the 𝐿2 vector norm (i.e., length of a vector in Euclidean space).

▪ Note: If ranges differ between components of 𝒑 then standardization

(z-scores) is necessary to avoid one variable to dominate the
distance.
 Euclidean Distance
3
point x y
2 p1 p1 0 2
p3 p4 p2 2 0
1 p3 3 1
p2
0
p4 5 1
0 1 2 3 4 5 6

p1 p2 p3 p4
p1 0.00 2.83 3.16 5.10
p2 2.83 0.00 1.41 3.16
p3 3.16 1.41 0.00 2.00
p4 5.10 3.16 2.00 0.00
Distance Matrix
 point x y
Minkowski Distance p 0 2
q 2 0

▪ Minkowski Distance is a generalization of Euclidean Distance

1
𝑛 𝑟
𝑟
𝑑𝑀 = ෍ 𝑝𝑘 − 𝑞𝑘 = 𝒑−𝒒 𝑟
𝑘=1

— Where 𝒑 and 𝒒 are two objects represented by vectors. n is the number of

dimensions (attributes) of the vectors and and 𝑝𝑘 and 𝑞𝑘 are, respectively,
the kth attributes (components) or data objects p and q.

▪ Note: If ranges differ then standardization (z-scores) is necessary to

avoid one variable to dominate the distance.
Minkowski Distance: Examples

▪ 𝑟 = 1. City block (Manhattan, taxicab, 𝐿1 norm) distance.

—A common example of this is the Hamming distance, which is just the
number of bits that are different between two binary vectors

▪ 𝑟 = 2. Euclidean distance (𝐿2 norm)

▪ 𝑟 = ∞. “supremum” (maximum norm, 𝐿∞ norm) distance.

—This is the maximum difference between any component of the vectors

▪ Do not confuse r with n, i.e., all these distances are defined for all
numbers of dimensions.
Minkowski Distances Distance Matrix
𝑳𝟏 p1 p2 p3 p4
p1 0 4 4 6
p2 4 0 2 4
p3 4 2 0 2
p4 6 4 2 0
point x y
p1 0 2 𝑳𝟐 p1 p2 p3 p4
p1 0.00 2.83 3.16 5.10
p2 2 0
p2 2.83 0.00 1.41 3.16
p3 3 1
p3 3.16 1.41 0.00 2.00
p4 5 1
p4 5.10 3.16 2.00 0.00

𝑳∞ p1 p2 p3 p4
p1 0 2 3 5
p2 2 0 1 3
p3 3 1 0 2
p4 5 3 2 0
 Mahalanobis Distance
𝑑𝑚𝑎ℎ𝑎𝑙𝑎𝑛𝑜𝑏𝑖𝑠 𝒑, 𝒒 = 𝒑 − 𝒒 𝑇 𝑆 −1 𝒑 − 𝒒

𝑆 −1 is the inverse of the

covariance matrix of the input
data

Measures how many standard deviations two points are away from each other →
scale invariant measure
Example: For red points, the Euclidean distance is 14.7, Mahalanobis distance is 6.
Mahalanobis Distance
y Covariance Matrix:

.3 .2
𝑆=
.2 .3
C
A: (0.5, 0.5)

B B: (0, 1)
C: (1.5, 1.5)
A
𝑑𝑚𝑎ℎ𝑎𝑙 𝐴, 𝐵 = 5
𝑑𝑚𝑎ℎ𝑎𝑙 (𝐴, 𝐶) = 4

x
Data varies in direction A-C more than in A-B!
Cosine Similarity

For two vector A and B, the cosine

similarity is defined as

Example:
A= 3205000200
B= 1000000102
𝑨⋅𝑩 = 3∗1 + 2∗0 + 0∗0 + 5∗0 + 0∗0 + 0∗0 + 0∗0 + 2∗1 + 0∗0 + 0∗2 = 5
𝑨 = 3 ∗ 3 + 2 ∗ 2 + 0 ∗ 0 + 5 ∗ 5 + 0 ∗ 0 + 0 ∗ 0 + 0 ∗ 0 + 2 ∗ 2 + 0 ∗ 0 + 0 ∗ 0 0.5 = 6.481
𝑩 = (1 ∗ 1 + 0 ∗ 0 + 0 ∗ 0 + 0 ∗ 0 + 0 ∗ 0 + 0 ∗ 0 + 0 ∗ 0 + 1 ∗ 1 + 0 ∗ 0 + 2 ∗ 2) 0.5 = 2.245

𝑠𝑐𝑜𝑠𝑖𝑛𝑒 = .3150

Cosine similarity is often used for word count vectors to compare documents.
Similarity Between Binary Vectors

▪ Common situation is that objects, p and q, have only binary attributes

▪ Compute similarities using the following quantities

M01 = the number of attributes where p was 0 and q was 1
M10 = the number of attributes where p was 1 and q was 0
M00 = the number of attributes where p was 0 and q was 0
M11 = the number of attributes where p was 1 and q was 1

▪ Simple Matching and Jaccard Coefficients

𝑠𝑆𝑀𝐶 = number of matches / number of attributes
= (M11 + M00) / (M01 + M10 + M11 + M00)

𝑠𝐽 = number of 11 matches / number of not-both-zero attribute values

= (M11) / (M01 + M10 + M11)

Note: Jaccard ignores 0s!

SMC versus Jaccard: Example

p= 1000000000
q= 0000001001

M01 = 2 (the number of attributes where p was 0 and q was 1)

M10 = 1 (the number of attributes where p was 1 and q was 0)
M00 = 7 (the number of attributes where p was 0 and q was 0)
M11 = 0 (the number of attributes where p was 1 and q was 1)

𝑠𝑆𝑀𝐶 = (M11 + M00)/(M01 + M10 + M11 + M00) = (0+7) / (2+1+0+7) = 0.7

𝑠𝐽 = (M11) / (M01 + M10 + M11) = 0 / (2 + 1 + 0) = 0

Extended Jaccard Coefficient (Tanimoto)

▪ Variation of Jaccard for continuous or count attributes:

where ∙ is the dot product between two vectors and ||∙||2 is the
Euclidean norm (length of the vector).

▪ Reduces to Jaccard for binary attributes

Dis(similarities) With Mixed Types

▪ Sometimes attributes are of many different types (nominal, ordinal,

ratio, etc.), but an overall similarity is needed.
▪ Gower's (dis)similarity:
—Ignores missing values
—Final (dis)similarity is a weighted sum of variable-wise (dis)similarities
Common Properties of a Distance

▪ Distances, such as the Euclidean distance, have some well-known

properties.

1. 𝑑 𝑝, 𝑞 ≥ 0 for all p and q and 𝑑(𝑝, 𝑞) = 0 only if

p = q. (Positive definiteness)
2. 𝑑(𝑝, 𝑞) = 𝑑(𝑞, 𝑝) for all p and q. (Symmetry)
3. 𝑑 𝑝, 𝑟 ≤ 𝑑(𝑝, 𝑞) + 𝑑(𝑞, 𝑟) for all points p, q, and r.
(Triangle Inequality)
where 𝑑(𝑝, 𝑞) is the distance (dissimilarity) between points
(data objects), p and q.

▪ A distance that satisfies these properties is a metric and forms a

metric space.
Common Properties of a Similarity

▪ Similarities, also have some well-known properties.

—𝑠(𝑝, 𝑞) = 1 (or maximum similarity) only if p = q.

—𝑠(𝑝, 𝑞) = 𝑠(𝑞, 𝑝) for all p and q. (Symmetry)

where 𝑠(𝑝, 𝑞) is the similarity between points (data objects), p

and q.
Exercise

x y
A 2 1
B 4 3
C 1 1

▪ Calculate the Euclidean and the Manhattan distances between A and

C and A and B
▪ Calculate the Cosine similarity between A and C and A and B
Correlation

▪ Correlation measures the (linear) relationship between two variables.

▪ To compute Pearson correlation (Pearson's Product Moment
Correlation), we standardize data objects, p and q, and then take
their dot product
cov ( X , Y )
ρ=
sd ( X ) sd (Y )

▪ Estimation:

▪ Correlation is often used as a measure of similarity.

Visually Evaluating Correlation

Scatter plots showing the

similarity from –1 to 1.
Rank Correlation

▪ Measure the degree of similarity between two ratings (e.g., ordinal

data).
▪ Is more robust against outliers and does not assume normality of
data or linear relationship like Pearson Correlation.
▪ Measures (all are between -1 and 1)
—Spearman's Rho: Pearson correlation between ranked variables.
—Kendall's Tau
N s− N d N s ...concordant pair
τ= N d ... discordant pair
1
n(n− 1)
2
—Goodman and Kruskal's Gamma
N s− N d
γ=
N s+ N d
Topics

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Density

▪ Density-based clustering require a notion of density

▪ Examples:
—Probability density (function) = describes the likelihood of a random variable
taking a given value

—Euclidean density = number of points per unit volume

—Graph-based density = number of edges compared to a complete graph

—Density of a matrix = proportion of non-zero entries.

Kernel Density Estimation (KDE)

▪ KDE is a non-parametric way to estimate the probability density

function of a random variable.

▪ K is the kernel (a non-negative function that integrates to one) and h

> 0 is a smoothing parameter called the bandwidth. Often a Gaussian
kernel is used.
▪ Example:
Euclidean Density – Cell-based

▪ Simplest approach is to divide region into a number of rectangular

cells of equal volume and define density as # of points the cell
contains.
Euclidean Density – Center-based

▪ Euclidean density is the number of points within a specified radius of

the point
You should know
now about…

▪ Attributes/Features
▪ Types of Data Sets
▪ Data Quality
▪ Data Preprocessing
▪ Similarity and Dissimilarity
▪ Density
Examples of Classification Task

Task Attribute set, x Class label, y

Categorizing Features extracted from spam or non-spam

email email message header
messages and content
Identifying Features extracted from malignant or benign
tumor cells x-rays or MRI scans cells

Cataloging Features extracted from Elliptical, spiral, or

galaxies telescope images irregular-shaped
galaxies

2/1/2021 Introduction to Data Mining, 2nd Edition 82

Data Mining: Exploring Data

Lecture Notes for Chapter 3

Slides by Tan, Steinbach, Kumar adapted by Michael Hahsler

Look for accompanying R

code on the course web site.
Topics

▪ Exploratory Data Analysis

▪ Summary Statistics
▪ Visualization
What is Data Exploration?

“A preliminary exploration of the data to better

understand its characteristics.”

▪ Key motivations of data exploration include

—Helping to select the right tool for preprocessing or analysis
—Making use of humans’ abilities to recognize patterns.
People can recognize patterns not captured by data analysis tools

▪ Related to the area of Exploratory Data Analysis (EDA)

—Created by statistician John Tukey
—Seminal book is "Exploratory Data Analysis" by Tukey
—A nice online introduction can be found in Chapter 1 of the NIST Engineering
Statistics Handbook
—http://www.itl.nist.gov/div898/handbook/index.htm
Techniques Used In Data Exploration

▪ In EDA, as originally defined by Tukey

—The focus was on visualization
—Clustering and anomaly detection were viewed as exploratory techniques
—In data mining, clustering and anomaly detection are major areas of interest,
and not thought of as just exploratory

▪ In our discussion of data exploration, we focus on

—Summary statistics
—Visualization
Topics

▪ Exploratory Data Analysis

▪ Summary Statistics
▪ Visualization
Summary Statistics

Summary statistics are numbers that summarize properties of the data

▪ Summarized properties include location and spread for continuous

data

Examples: location - mean

spread - standard deviation

▪ Most summary statistics can be calculated in a single pass through

the data
Frequency and Mode

▪ The frequency of an attribute value is the percentage of time the

value occurs in the
data set
—For example, given the attribute ‘gender’ and a representative population of
people, the gender ‘female’ occurs about 50% of the time.
▪ The mode of an attribute is the most frequent attribute value
▪ The notions of frequency and mode are typically used with
categorical data.
Measures of Location: Mean and Median

▪ For quantitative features.

▪ The mean is the most common measure of the location of a set of
points.
▪ However, the mean is very sensitive to outliers.
▪ Thus, the median or a trimmed mean is also commonly used.
Measures of Spread: Range and Variance

▪ Range is the difference between the max and min

▪ The variance or standard deviation is the most common measure of
the spread of a set of points.

▪ However, this is also sensitive to outliers, so that other measures are

often used.
Percentiles

▪ Given an ordinal or continuous attribute x and a number p between 0

and 100, the 𝑝𝑡ℎ percentile is a value 𝑥𝑝% of x such that p% of the
observed values of x are less than 𝑥𝑝% .

▪ Example: the 50th percentile is the value 𝑥50% such that 50% of all
values of x are less than 𝑥50% .
 Median – 50% of the
Percentiles cases has a smaller value & 50%
are larger

xp
xp
Multivariate Summary Statistics

Object x1 x2 ▪ Covariance between features i and j

1 12 15 1 m

2 2 4
sij =
m− 1
∑ k= 1
( xki − x̄ i )(x kj − x̄ j )
... ... ...
m 18 4 ▪ Correlation
sij
r ij=
si s j
▪ 𝑠𝑖 is the variance of feature i
Topics

▪ Exploratory Data Analysis

▪ Summary Statistics
▪ Visualization
Visualization

▪ Visualization is the conversion of data into a visual or tabular format

so that the characteristics of the data and the relationships among
data items or attributes can be analyzed or reported.

▪ Visualization of data is one of the most powerful and appealing

techniques for data exploration.
—Humans have a well-developed ability to analyze large amounts of
information that is presented visually
—Can detect general patterns and trends
—Can detect outliers and unusual patterns
Example: Sea Surface Temperature

▪ The following shows the Sea Surface Temperature (SST) for July 1982
▪ Tens of thousands of data points are summarized in a single figure
Representation

▪ Is the mapping of information to a visual format

▪ Data objects, their attributes, and the relationships among data
objects are translated into graphical elements such as points, lines,
shapes, and colors.
▪ Example:
—Objects are often represented as points
—Their attribute values can be represented as the position of the points or the
characteristics of the points, e.g., color, size, and shape
—If position is used, then the relationships of points, i.e., whether they form
groups or a point is an outlier, is easily perceived.
Arrangement

▪ Is the placement of visual elements within a display

▪ Can make a large difference in how easy it is to understand the data
▪ Example:
Selection

▪ Is the elimination or the deemphasis of certain objects and attributes

▪ Selection may involve the choosing a subset of attributes

—Dimensionality reduction is often used to reduce the number of dimensions
to two or three
—Alternatively, pairs of attributes can be considered

▪ Selection may also involve choosing a subset of objects

— A region of the screen can only show so many points
—Can sample, but want to preserve points in sparse areas
 Iris Versicolor
Iris Virginica

The Iris Dataset

Many of the exploratory data
techniques are illustrated with the Iris
Plant data set.
▪ Included as a demo datasert in many
tools (R, scikit-learn, Rapidminer, …).
▪ Can be obtained from the UCI
Machine Learning Repository
http://www.ics.uci.edu/~mlearn/ML
Repository.html
▪ From the statistician R.A. Fisher Iris Setosa
▪ 150 flowers, three types (classes).
▪ Four (non-class) attributes

…
Distribution: Histograms

▪ Usually shows the distribution of values of a single variable

▪ Divide the values into bins and show a bar plot of the number of objects in each
bin.
▪ The height of each bar indicates the number of objects
▪ Shape of histogram depends on the number of bins

▪ Example: Petal Width (10 and 20 bins, respectively)

Empirical Cumulative Distribution Function (ECDF)

▪ Probability Density Function (PDF):

describes the relative likelihood for
PDF this random variable to take on a
given value. Histogram shows an
empirical PDF.

▪ Cumulative Distribution Function

(CDF): Shows the distribution of
CDF data as the fraction of points that
are less than this value.
Example: ECDF
Distribution Box Plots

▪ Invented by J. Tukey
▪ Simplified version of a PDF/histogram.

outlier

75th percentile +
1.5 IQR

75th percentile

50th percentile IQR

25th percentile

25th percentile –
1.5 IQR

PDF or histogram
Examples of Box Plots

▪ Box plots can be used to compare attributes or subgroups.

Two-Dimensional Histograms

▪ Show the joint distribution of the values of two attributes

▪ Example: petal width and petal length
—What does this tell us?
Scatter Plots

▪ Attributes values determine

the position
▪ Two-dimensional scatter
plots most common, but can
have three-dimensional
scatter plots
▪ Often additional attributes
can be displayed by using the
size, shape, and color of the
markers that represent the
objects
Scatter Plot Array of Iris Attributes
Contour Plots
- Useful when a continuous attribute is measured on a spatial grid
- They partition the plane into regions of similar values
- The contour lines that form the boundaries of these regions connect points with
equal values
- The most common example is contour maps of elevation
- Can also display temperature, rainfall, air pressure, etc.

Celsius
Matrix Plots

▪ Can plot a data matrix

▪ Can be useful when objects are sorted according to class
▪ Typically, the attributes are normalized to prevent one attribute from
dominating the plot
▪ Plots of similarity or distance matrices can also be useful for
visualizing the relationships between objects
Visualization of the Iris Data Matrix
Deviation form feature mean
Visualization of the Iris Correlation Matrix
Parallel Coordinates

▪ Used to plot the attribute values of high-dimensional data

▪ Instead of using perpendicular axes, use a set of parallel axes
▪ The attribute values of each object are plotted as a point on each
corresponding coordinate axis and the points are connected by a line

▪ Thus, each object is represented as a line

▪ Often, the lines representing a distinct class of objects group
together, at least for some attributes
▪ Ordering of attributes is important in seeing such groupings
Parallel Coordinates Plots for Iris Data

Reordered features
Other Visualization Techniques

▪ Star Plots
—Similar approach to parallel coordinates, but axes radiate from a central
point
—The line connecting the values of an object is a polygon

▪ Chernoff Faces
—Approach created by Herman Chernoff
—This approach associates each attribute with a characteristic of a face
—The values of each attribute determine the appearance of the corresponding
facial characteristic
—Each object becomes a separate face
—Relies on human’s ability to distinguish faces
Star Plots for Iris Data

▪ Setosa

▪ Versicolor

▪ Virginica
Chernoff Faces for Iris Data

▪ Setosa

▪ Versicolor

▪ Virginica
•Created by Ross Ihaka and Robert Gentleman at the
University of Auckland, New Zealand
•Currently developed by the R Development Core Team.
R vs. RStudio
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
R-Studio UI
Write your Global Environment-
code here See your datasets here

See your files,

graphs, help
documentation
and installed
packages here

Console - see
your code run
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
 Outline for this Session

1. Introduction to R
2. Data Post-processing
3. Graphics and Plotting
4. Regression Analysis using R
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing
https://phlconnect.ched.gov.ph/content/view/business-analytics-data-post-processing