Evolution of Database Technology

• 1960s and earlier:

• Data Collection and Database Creation
– Primitive file processing
 Evolution of Database Technology

• 1970s - early 1980s:

• Data Base Management Systems
– Hieratical and network database systems
– Relational database Systems
– Query languages: SQL
– Transactions, concurrency control and recovery.
– On-line transaction processing (OLTP)
 Evolution of Database Technology

• Mid -1980s - present:

– Advanced data models
• Extended relational, object-relational
– Advanced application-oriented DBMS
• spatial, scientific, engineering, temporal, multimedia,
active, stream and sensor, knowledge-based
 Evolution of Database Technology

• Late 1980s-present
– Advanced Data Analysis
• Data warehouse and OLAP
• Data mining and knowledge discovery
• Advanced data mining appliations
• Data mining and socity
• 1990s-present:
– XML-based database systems
– Integration with information retrieval
– Data and information integreation
 Evolution of Database Technology

• Present – future:
– New generation of integrated data and
information system.
What Is Data Mining?
 What Is Data Mining?

• Data mining refers to extracting or mining

knowledge from large amounts of data.
• Mining of gold from rocks or sand
• Knowledge mining from data, knowledge
extraction, data/pattern analysis, data
archeology, and data dreding.
• Knowledge Discovery from data, or KDD
 Data Mining: A KDD Process

Pattern Evaluation
– Data mining: the core of
knowledge discovery
process. Data Mining

Task-relevant Data

Data Warehouse Selection

Transformation

Data Cleaning

Data Integration

Databases
 Steps of a KDD Process
1. Data cleaning
2. Data integration
3. Data selection
4. Data transformation
5. Data mining
6. Pattern evaluation
7. Knowledge presentaion
 Steps of a KDD Process
• Learning the application domain:
– relevant prior knowledge and goals of
application
• Creating a target data set: data selection
• Data cleaning and preprocessing
• Data reduction and transformation:
– Find useful features, dimensionality/variable
reduction, invariant representation.
 Steps of a KDD Process
• Choosing functions of data mining
– summarization, classification, regression, association,
clustering.
• Choosing the mining algorithms
• Data mining: search for patterns of interest
• Pattern evaluation and knowledge presentation
– visualization, transformation, removing redundant
patterns, etc.
• Use of discovered knowledge
Architecture of a Typical Data
Mining System
Graphical user interface

Pattern evaluation

Data mining engine

Knowledge-base
Database or data
warehouse server
Data cleaning & data integration Filtering

Data
Databases Warehouse
Data Mining and Business Intelligence
Increasing potential
to support
business decisions End User
Making
Decisions

Data Presentation Business

Analyst
Visualization Techniques
Data Mining Data
Information Discovery Analyst

Data Exploration
Statistical Analysis, Querying and Reporting

Data Warehouses / Data Marts

OLAP, MDA DBA
Data Sources
Paper, Files, Information Providers, Database Systems, OLTP
Data Mining: On What Kind of Data?
• Answer : On any kind of data.
 Data Mining: On What Kind of Data?

• Relational databases
• Data warehouses
• Transactional databases
 Data Mining: On What Kind of Data?

• Advanced DB and information repositories

– Object-oriented and object-relational databases
– Spatial databases
– Time-series data and temporal data
– Text databases and multimedia databases
– Heterogeneous and legacy databases
– WWW
Data Mining Functionalities -What kind
of patterns can be mined?
 Data mining tasks generally classified into two
categories.

1.Descriptive :It characterize the general

properties of data in the database.
2.Predictive :It performs inference(interpretation)
on the current data in order to make
predictions.
 Data Mining Functionalities
• Concept description: Characterization and
discrimination
– Data can be associated with classes or concepts
– Ex. AllElectronics store classes of items for sale include
computer and printers.
– Description of class or concept is called class/concept
description.
– Data characterization : summarization of general features
of target class of data.
– Data discrimination : comparison of target class with one or
more contrasting classes.
 Data Mining Functionalities
• Association Analysis
– Multi-dimensional vs. single-dimensional
association
– age(X, “20..29”) ^ income(X, “20..29K”) => buys(X,
“PC”) [support = 2%, confidence = 60%]
– contains(T, “computer”) => contains(x,
“software”) [support=1%, confidence=75%]
 Data Mining Functionalities

• Classification and Prediction

– Finding models (functions) that describe and
distinguish data classes or concepts for predict the
class whose label is unknown
– E.g., classify countries based on climate, or classify
cars based on gas mileage
– Models: decision-tree, classification rules (if-then),
neural network
– Prediction: Predict some unknown or missing
numerical values
 Data Mining Functionalities

• Cluster analysis
– Analyze class-labeled data objects, clustering
analyze data objects without consulting a known
class label.
– Clustering based on the principle: maximizing the
intra-class similarity and minimizing the interclass
similarity
 Data Mining Functionalities
• Outlier analysis
– Outlier: a data object that does not comply(fulfill) with the general
behavior of the model of the data
– It can be considered as noise or exception but is quite useful in fraud
detection, rare events analysis

• Trend and evolution analysis

– Trend and deviation: regression analysis
– Sequential pattern mining, periodicity analysis
– Similarity-based analysis
Data Mining: Classification Schemes
 Data Mining: Confluence of Multiple
Disciplines
Database
Statistics
Technology

Information
Science Data Mining MachineLearnin
g

Visualization Other
Disciplines
Data Mining systems: Classification Schemes

• General functionality
– Descriptive data mining
– Predictive data mining
• Data mining various criteria's:
– Kinds of databases to be mined
– Kinds of knowledge to be discovered
– Kinds of techniques utilized
– Kinds of applications adopted
 Data Mining: Classification Schemes
• Databases to be mined
– Relational, transactional, object-oriented,
object-relational, active, spatial, time-series, text,
multi-media, heterogeneous, legacy, WWW, etc.
• Knowledge to be mined
– Characterization, discrimination, association,
classification, clustering, trend, deviation and outlier
analysis, etc.
– Multiple/integrated functions and mining at multiple
levels
 Data Mining: Classification Schemes

• Techniques utilized
– Database-oriented, data warehouse (OLAP),
machine learning, statistics, visualization,
neural network, etc.
• Applications adopted
– Retail, telecommunication, banking, fraud
analysis, DNA mining, stock market
Major Issues in Data Mining
 Major Issues in Data Mining

1. Mining methodology and user interaction issues

– Mining different kinds of knowledge in databases
– Interactive mining of knowledge at multiple levels of
abstraction
– Incorporation of background knowledge
– Data mining query languages and ad-hoc data mining
– Expression and visualization of data mining results
– Handling noise and incomplete data
– Pattern evaluation: the interestingness problem
 Major Issues in Data Mining
2. Performance issues
– Efficiency and scalability of data mining algorithms
– Parallel, distributed and incremental mining
methods
 Major Issues in Data Mining

3. Issues relating to the diversity of data types

– Handling relational(simple) and complex types of

data.

– Mining information from heterogeneous databases

and global information systems (WWW)
What is Data Warehouse?
 What is Data Warehouse?
• Defined in many different ways
– A decision support database that is maintained separately
from the organization’s operational database
– Support information processing by providing a solid
platform of consolidated, historical data for analysis.

• “A data warehouse is a subject-oriented, integrated,

time-variant, and nonvolatile collection of data in support of
management’s decision-making process.”—W. H. Inmon

• Data warehousing:
– The process of constructing and using data warehouses
 Data Warehouse—Subject-Oriented
• Organized around major subjects, such as customer, product,
sales.
• Focusing on the modeling and analysis of data for decision
makers, not on daily operations or transaction processing.
• Provide a simple and concise view around particular subject
issues by excluding data that are not useful in the decision
support process.
 Data Warehouse—Integrated
• Constructed by integrating multiple, heterogeneous
data sources
– relational databases, flat files, on-line transaction records
• Data cleaning and data integration techniques are
applied.
– Ensure consistency in naming conventions, encoding
structures, attribute measures, etc. among different data
sources
• E.g., Hotel price: currency, tax, breakfast covered, etc.
– When data is moved to the warehouse, it is converted.
 Data Warehouse—Time Variant
• The time horizon for the data warehouse is
significantly longer than that of operational systems.
– Operational database: current value data.
– Data warehouse data: provide information from a historical
perspective (e.g., past 5-10 years)
• Every key structure in the data warehouse
– Contains an element of time, explicitly or implicitly
– But the key of operational data may or may not contain
“time element”.
 Data Warehouse—Non-Volatile
• A physically separate store of data transformed from
the operational environment.
• Operational update of data does not occur in the
data warehouse environment.
– Does not require transaction processing, recovery, and
concurrency control mechanisms
– Requires only two operations in data accessing:
• initial loading of data and access of data.
 Data Warehouse vs. Operational DBMS
• Distinct features (OLTP vs. OLAP):
– User and system orientation: customer vs. market
– Data contents: current, detailed vs. historical, consolidated
– Database design: ER + application vs. star + subject
– View: current, local vs. evolutionary, integrated
– Access patterns: update vs. read-only but complex queries
 Data Warehouse vs. Operational DBMS
• OLTP (on-line transaction processing)
– Major task of traditional relational DBMS
– Day-to-day operations: purchasing, inventory, banking,
manufacturing, payroll, registration, accounting, etc.

• OLAP (on-line analytical processing)

– Major task of data warehouse system
– Data analysis and decision making
OLTP vs. OLAP
 Why Separate Data Warehouse?
• High performance for both systems

– DBMS— tuned for OLTP: access methods, indexing,

concurrency control, recovery

– Warehouse—tuned for OLAP: complex OLAP

queries, multidimensional view, consolidation.
 Why Separate Data Warehouse?
• Different functions and different data:
– missing data: Decision support requires historical
data which operational DBs do not typically
maintain
– data consolidation: DS requires consolidation
(aggregation, summarization) of data from
heterogeneous sources
– data quality: different sources typically use
inconsistent data representations, codes and
formats which have to be reconciled
 OLAP Operations in MDM
• Roll up (drill-up): summarize data
– by climbing up hierarchy or by
dimension reduction
• Roll down (Drill down): reverse of roll-up
– from higher level summary to lower
level summary or detailed data, or
introducing new dimensions
• Slice and dice:
– project and select
 OLAP Operations
• Pivot (rotate):
– reorient the cube, visualization, 3D to
series of 2D planes.
• Other operations
– drill across: involving (across) more
than one fact table
– drill through: through the bottom level
of the cube to its back-end relational
tables (using SQL)
Data warehouse architecture
 Steps for the Design and Construction of
Data Warehouse

• The design of a data warehouse: a business

analysis framework
• The process of data warehouse design
• A three-tier data ware house architecture
Design of a Data Warehouse: A Business Analysis
Framework

• Four views regarding the design of a data warehouse

– Top-down view
• allows selection of the relevant information
necessary for the data warehouse
Design of a Data Warehouse: A Business Analysis
Framework
– Data warehouse view
• consists of fact tables and dimension tables

– Data source view

• exposes the information being captured, stored, and
managed by operational systems

– Business query view

• sees the perspectives
 Data Warehouse Design Process

• Top-down, bottom-up approaches or a combination

of both
– Top-down: Starts with overall design and planning
(mature)
– Bottom-up: Starts with experiments and prototypes (rapid)
• From software engineering point of view
– Waterfall: structured and systematic analysis at each step
before proceeding to the next
– Spiral: rapid generation of increasingly functional systems,
short turn around time, quick turn around
 Data Warehouse Design Process

• Typical data warehouse design process

– Choose a business process to model, e.g., orders,
invoices, etc.
– Choose the grain (atomic level of data) of the
business process
– Choose the dimensions that will apply to each fact
table record
– Choose the measure that will populate each fact
table record
 Multi-Tiered Architecture
Monitor
& OLAP Server
other Metadata
Integrato
sources r
Analysis
Operational Extract Query
Transform Data Serve Reports
DBs
Load
Refresh
Warehouse Data mining

Data Marts

Data Sources Data OLAP Engine Front-End

 Metadata Repository
• Meta data is the data defining warehouse objects. It has the
following kinds
– Description of the structure of the warehouse
• schema, view, dimensions, hierarchies, derived data defn, data mart locations
and contents
– Operational meta-data
• data lineage (history of migrated data and transformation path), currency of
data (active, archived, or purged), monitoring information (warehouse usage
statistics, error reports, audit trails)
– The algorithms used for summarization
– The mapping from operational environment to the data warehouse
– Data related to system performance
• warehouse schema, view and derived data definitions
– Business data
• business terms and definitions, ownership of data, charging policies
Data Warehouse Back-End Tools and Utilities

• Data extraction:
– get data from multiple, heterogeneous, and external
sources
• Data cleaning:
– detect errors in the data and rectify them when possible
• Data transformation:
– convert data from legacy or host format to warehouse
format
• Load:
– sort, summarize, consolidate, compute views, check
integrity, and build indices and partitions
• Refresh
– propagate the updates from the data sources to the
warehouse
 Three Data Warehouse Models
• Enterprise warehouse
– collects all of the information about subjects spanning the entire
organization
• Data Mart
– a subset of corporate-wide data that is of value to a specific groups
of users. Its scope is confined to specific, selected groups, such as
marketing data mart
• Independent vs. dependent (directly from warehouse) data
mart
• Virtual warehouse
– A set of views over operational databases
– Only some of the possible summary views may be materialized
 Data Warehouse Development: A
Recommended Approach

Multi-Tier Data
Warehouse
Distributed
Data Marts

Enterprise
Data Data
Data
Mart Mart
Warehouse

Model refinement Model refinement

Define a high-level corporate data model

 Types of OLAP Servers
• Relational OLAP (ROLAP)
– Use relational or extended-relational DBMS to store and
manage warehouse data and OLAP middle ware to support
missing pieces
– Include optimization of DBMS backend, implementation of
aggregation navigation logic, and additional tools and
services
– greater scalability
• Multidimensional OLAP (MOLAP)
– Array-based multidimensional storage engine (sparse matrix
techniques)
– fast indexing to pre-computed summarized data
 Types of OLAP Servers
• Hybrid OLAP (HOLAP)
– User flexibility, e.g., low level: relational,
high-level: array
• Specialized SQL servers
– specialized support for SQL queries over
star/snowflake schemas