CHAPTER 10: DATA

WAREHOUSING & CACHING

PRINCIPLES OF

DATA INTEGRATION
ANHAI DOAN ALON HALEVY ZACHARY IVES

Data Warehousing and

Materialization
We have mostly focused on techniques for virtual
data integration (see Ch. 1)
Queries are composed with mappings on the fly and data
is fetched on demand
This represents one extreme point

In this chapter, we consider cases where data is

transformed and materialized in advance of the
queries
The main scenario: the data warehouse

What Is a Data Warehouse?

In many organizations, we want a central store of
all of our entities, concepts, metadata, and historical
information
For doing data validation, complex mining, analysis,
prediction,
This is the data warehouse

To this point weve focused on scenarios where the

data lives in the sources here we may have a
master version (and archival version) in a central
database
For performance reasons, availability reasons, archival
reasons,

In the Rest of this Chapter

The data warehouse as the master data instance

Data warehouse architectures, design, loading
Data exchange: declarative data warehousing
Hybrid models: caching and partial materialization
Querying externally archived data

Outline
The data warehouse
Motivation: Master data management
Physical design
Extract/transform/load

Data exchange
Caching & partial materialization
Operating on external data

Master Data Management

One of the modern uses of the data warehouse is
not only to support analytics but to serve as a
reference to all of the entities in the organization
A cleaned, validated repository of what we know
which can be linked to by data sources
which may help with data cleaning
and which may be the basis of data governance
(processes by which data is created and modified in a
systematic way, e.g., to comply with govt regulations)

There is an emerging field called master data

management out the process of creating these

Data Warehouse Architecture

At the top a centralized
database
Generally configured for
queries and appends
not transactions
Many indices,
materialized views, etc.

Data is loaded and

periodically updated via
Extract/Transform/Load
(ETL) tools

Data Warehouse

ETL pipeline
outputs

ETL
ETL

ETL

RDBMS1

ETL

ETL

RDBMS2
HTML1

XML1

ETL Tools
ETL tools are the equivalent of schema mappings in
virtual integration, but are more powerful
Arbitrary pieces of code to take data from a source,
convert it into data for the warehouse:
import filters read and convert from data sources
data transformations join, aggregate, filter, convert data
de-duplication finds multiple records referring to the
same entity, merges them
profiling builds tables, histograms, etc. to summarize
data
quality management test against master values, known
business rules, constraints, etc.

Example ETL Tool Chain

Item
records

Split
Date time

Filter
invalid

Join

Customer
records

Filter
invalid

Filter
non match

Group by
customer
Customer
balance

Invoice
line items
Invalid
dates /times

Invalid
items

Invalid
customers

This is an example for e-commerce loading

Note multiple stages of filtering (using selection or
join-like operations), logging bad records, before we
group and load

Basic Data Warehouse Summary

Two aspects:
A central DBMS optimized for appends and querying
The master data instance
Or the instance for doing mining, analytics, and prediction

A set of procedural ETL pipelines to fetch, transform,

filter, clean, and load data
Often these tools are more expressive than standard conjunctive
queries (as in Chapters 2-3)
But not always!

This raises a question can we do warehousing with declarative

mappings?

Outline
The data warehouse
Data exchange
Caching & partial materialization
Operating on external data

Data Exchange
Intuitively, a declarative setup for data warehousing
Declarative schema mappings as in Ch. 2-3
Materialized database as in the previous section

Also allow for unknown values when we map from

source to target (warehouse) instance
If we know a professor teaches a student, then there must
exist a course C that the student took and the professor
taught but we may not know which

Data Exchange Formulation

A data exchange setting (S,T,M,CT) has:
S, source schema representing all of the source tables
jointly
T, target schema
A set of mappings or tuple-generating dependencies

("X)s1 (X1 ), ..., sm (Xm ) ($Y) t1 (Y1 ), ..., t k (Yk )

relating S and T

A set of constraints (equality-generating dependencies)

(Y)t1 (Y1 ), ..., tl (Yl ) (Yi Yj )

An Example
Source S has

Target T has

Teaches(prof, student)
Adviser(adviser, student)

Advise(adviser, student)
TeachesCourse(prof, course)
Takes(course, student)

r1 : Teaches( prof , stud ) D. Advise( D, stud )

r2 : Teaches( prof , stud ) C.TeachesCourse ( prof , C ),Takes(C , stud )
r3 : Adviser( prof , stud ) Advise( prof , stud )
r4 : Adviser( prof , stud ) C , D.TeachesCourse ( D, C ),Takes(C , stud )

existential variables represent unknown

The Data Exchange Solution

The goal of data exchange is to compute an instance
of the target schema, given a data exchange setting
D = (S,T,M,CT) and an instance I(S)

An instance J of Schema T is a data exchange

solution for D and I if
1. the pair (I,J) satisfies schema mapping M, and
2. J satisfies constraints CT

Back to the Example, Now with Data

Instance I(S) has

Instance J(T) has

Teaches

Advise

prof

student

adviser student

Ann

Bob

Ellen

Bob

Felicia

David

Chloe David

TeachesCourse

Adviser

prof

course

adviser student

Ann

C1

Ellen

Bob

Chloe

C2

Felicia

David

variables or labeled nulls

represent unknown values

Takes
course

student

C1

Bob

C2

David

This Is also a Solution

Instance I(S) has
Teaches

Instance J(T) has

Advise

prof

student

adviser student

Ann

Bob

Ellen

Bob

Felicia

David

Chloe David

TeachesCourse

Adviser

prof

course

adviser student

Ann

C1

Ellen

Bob

Chloe

C1

Felicia

David

this time the labeled

nulls are all the same!

Takes
course

student

C1

Bob

C1

David

Universal Solutions
Intuitively, the first solution should be better than
the second
The first solution uses the same variable for the course
taught by Ann and by Chloe they are the same course
But this was not specified in the original schema!

We formalize that through the notion of the

universal solution, which must not lose any
information

Formalizing the Universal Solution

First we define instance homomorphism:
Let J1, J2 be two instances of schema T
A mapping h: J1 J2 is a homomorphism from J1 to J2 if
h(c) = c for every c C,
for every tuple R(a1,,an) J1 the tuple R(h(a1),,h(an)) J2

J1, J2 are homomorphically equivalent if there are

homomorphisms h: J1 J2 and h: J2 J1

Def: Universal solution for data exchange setting

D = (S,T,M,CT), where I is an instance of S.
A data exchange solution J for D and I is a universal
solution if, for every other data exchange solution J for D
and I, there exists a homomorphism h: J J

Computing Universal Solutions

The standard process is to use a procedure called the
chase
Informally:
Consider every formula r of M in turn:
If there is a variable substitution for the left-hand side (lhs) of r
where the right-hand side (rhs) is not in the solution add it
If we create a new tuple, for every existential variable in the rhs,
substitute a new fresh variable

See Chapter 10 Algorithm 10 for full pseudocode

Core Universal Solutions

Universal solutions may be of arbitrary size
The core universal solution is the minimal universal
solution

Data Exchange and Querying

As with the data warehouse, all queries are directly
posed over the target database no reformulation
necessary

However, we typically assume certain answers

semantics
To get the certain answers (which are the same as in the
virtual integration setting with GLAV/TGD mappings)
compute the query answers and then drop any tuples with
labeled nulls (variables)

Data Exchange vs. Warehousing

From an external perspective, exchange and
warehousing are essentially equivalent
But there are different trade-offs in procedural vs.
declarative mappings
Procedural more expressive
Declarative easier to reason about, compose, invert,
create matieralized views for, etc. (see Chapter 6)

Outline
The data warehouse
Data exchange
Caching & partial materialization
Operating on external data

The Spectrum of Materialization

Virtual integration
(EII)

sources materialized

Data exchange /
data warehouse

all mediated relations

materialized

caching or partial materialization

some views materialized

Many real EII systems compute and maintain

materialized views, or cache results
A hybrid point between the fully virtual and fully
materialized approaches

Possible Techniques for Choosing

What to Materialize
Cache results of prior queries
Take the results of each query, materialize them
Use answering queries using views to reuse
Expire using time-to-live May not always be fresh!

Administrator-selected views
Someone manually specifies views to compute and
maintain, as with a relational DBMS
System automatically maintains

Automatic view selection

Using query workload, update frequencies a view
materialization wizard chooses what to materialize

Outline
The data warehouse
Data exchange
Caching & partial materialization
Operating on external data

Many Integration-Like Scenarios

over Historical Data
Many Web scenarios where we have large logs of
data accesses, created by the server
Goal: put these together and query them!
Looks like a very simple data integration scenario
external data, but single schema
A common approach: use programming
environments like MapReduce (or SQL layers above)
to query the data on a cluster
MapReduce reliably runs large jobs across 100s or 1000s
of shared nothing nodes in a cluster

MapReduce Basics
MapReduce is essentially a template for writing
distributed programs corresponding to a single SQL
SELECT..FROM..WHERE..GROUP BY..HAVING block
with user-defined functions
The MapReduce runtime calls a set of functions:
map is given a tuple, outputs 0 or more tuples in response

roughly like the WHERE clause

shuffle is a stage for doing sort-based grouping on a key

(specified by the map)
reduce is an aggregate function called over the set of
tuples with the same grouping key

MapReduce Dataflow Template: Tuples

Map worker Shuffle Reduce worker
emit tuples
Map
Worker
Map
Worker
Map
Worker

Map
Worker

emit aggregate
results
Reduce
Worker
Reduce
Worker
Reduce
Worker
Reduce
Worker
Reduce
Worker
30

MapReduce as ETL
Some people use MapReduce to take data,
transform it, and load it into a warehouse
which is basically what ETL tools do!
The dividing line between DBMSs, EII, MapReduce is
blurring as of the development of this book

SQL MapReduce
MapReduce over SQL engines
Shared-nothing DBMSs
NoSQL

Warehousing & Materialization Wrap-up

There are benefits to centralizing & materializing data
Performance, especially for analytics / mining
Archival
Standardization / canonicalization

Data warehouses typically use procedural ETL tools to

extract, transform, load (and clean) data
Data exchange replaces ETL with declarative
mappings (where feasible)
Hybrid schemes exist for partial materialization
Increasingly we are integrating via MapReduce and its
cousins