Chapter 3

Data Warehouse Concepts

This chapter introduces the basic concepts of data warehouses. A data

warehouse is a particular database targeted toward decision support. It
takes data from various operational databases and other data sources and
transforms it into new structures that t better for the task of performing
business analysis. Data warehouses are based on a multidimensional model,
where data are represented as hypercubes, with dimensions corresponding to
the various business perspectives and cube cells containing the measures to be
analyzed. In Sect. 3.1, we study the multidimensional model and present its
main characteristics and components. Section 3.2 gives a detailed description
of the most common operations for manipulating data cubes. In Sect. 3.3, we
present the main characteristics of data warehouse systems and compare them
against operational databases. The architecture of data warehouse systems
is described in detail in Sect. 3.4. As we shall see, in addition to the data
warehouse itself, data warehouse systems are composed of back-end tools,
which extract data from the various sources to populate the warehouse, and
front-end tools, which are used to extract the information from the warehouse
and present it to users. In Sect. 3.5, we introduce the design methodology
we will use throughout the book. We nish by describing in Sect. 3.6 two
representative business intelligence suite of tools, SQL Server and Pentaho.

3.1

Multidimensional Model

The importance of data analysis has been steadily increasing from the early
1990s, as organizations in all sectors are being required to improve their
decision-making processes in order to maintain their competitive advantage.
Traditional database systems like the ones studied in Chap. 2 do not satisfy
the requirements of data analysis. They are designed and tuned to support
the daily operations of an organization, and their primary concern is to ensure
A. Vaisman and E. Zim
anyi, Data Warehouse Systems, Data-Centric
Systems and Applications, DOI 10.1007/978-3-642-54655-6 3,
Springer-Verlag Berlin Heidelberg 2014

53

54

Data Warehouse Concepts

fast, concurrent access to data. This requires transaction processing and

concurrency control capabilities, as well as recovery techniques that guarantee
data consistency. These systems are known as operational databases or
online transaction processing (OLTP) systems. The OLTP paradigm
is focused on transactions. In the Northwind database example, a simple
transaction could involve entering a new order, reserving the products
ordered, and, if the reorder point has been reached, issuing a purchase order
for the required products. Eventually, a user may want to know the status
of a given order. If a database is indexed following one of the techniques
described in the previous chapter, a typical OLTP query like the above would
require accessing only a few records of the database (and normally will return
a few tuples). Since OLTP systems must support heavy transaction loads,
their design should prevent update anomalies, and thus, OLTP databases
are highly normalized using the techniques studied in Chap. 2. Thus, they
perform poorly when executing complex queries that need to join many
relational tables together or to aggregate large volumes of data. Besides,
typical operational databases contain detailed data and do not include
historical data.
The above needs called for a new paradigm specically oriented to
analyze the data in organizational databases to support decision making.
This paradigm is called online analytical processing (OLAP). This
paradigm is focused on queries, in particular, analytical queries. OLAPoriented databases should support a heavy query load. Typical OLAP queries
over the Northwind database would ask, for example, for the total sales
amount by product and by customer or for the most ordered products by
customer. These kinds of queries involve aggregation, and thus, processing
them will require, most of the time, traversing all the records in a database
table. Indexing techniques aimed at OLTP are not ecient in this case: new
indexing and query optimization techniques are required for OLAP. It is easy
to see that normalization is not good for these queries, since it partitions the
database into many tables. Reconstructing the data would require a high
number of joins.
Therefore, the need for a dierent database model to support OLAP
was clear and led to the notion of data warehouses, which are (usually)
large repositories that consolidate data from dierent sources (internal and
external to the organization), are updated o-line (although as we will see,
this is not always the case in modern data warehouse systems), and follow
the multidimensional data model. Being dedicated analysis databases, data
warehouses can be designed and optimized to eciently support OLAP
queries. In addition, data warehouses are also used to support other kinds of
analysis tasks, like reporting, data mining, and statistical analysis.
Data warehouses and OLAP systems are based on the multidimensional
model, which views data in an n-dimensional space, usually called a data
cube or a hypercube. A data cube is dened by dimensions and facts.
Dimensions are perspectives used to analyze the data. For example, consider

55

Cu
s
(C tom
ity e
) r

3.1 Multidimensional Model

dimensions

Time (Quarter)

Kln
Berlin
Lyon
Paris
Q1

measure
values

Q2
Q3
Q4
Produce Seafood
Beverages Condiments
Product (Category)

Fig. 3.1 A three-dimensional cube for sales data with dimensions Product, Time,
and Customer, and a measure Quantity

the data cube in Fig. 3.1, based on a portion of the Northwind database. We
can use this cube to analyze sales gures. The cube has three dimensions:
Product, Time, and Customer. A dimension level represents the granularity,
or level of detail, at which measures are represented for each dimension of
the cube. In the example, sales gures are aggregated to the levels Category,
Quarter, and City, respectively. Instances of a dimension are called members.
For example, Seafood and Beverages are members of the Product dimension
at the Category level. Dimensions also have associated attributes describing
them. For example, the Product dimension could contain attributes such as
ProductNumber and UnitPrice, which are not shown in the gure.
On the other hand, the cells of a data cube, or facts, have associated
numeric values (we will see later that this is not always the case), called
measures. These measures are used to evaluate quantitatively various
aspects of the analysis at hand. For example, each number shown in a cell
of the data cube in Fig. 3.1 represents a measure Quantity, indicating the
number of units sold (in thousands) by category, quarter, and customers
city. A data cube typically contains several measures. For example, another
measure, not shown in the gure, could be Amount, indicating the total sales
amount.
A data cube may be sparse or dense depending on whether it has
measures associated with each combination of dimension values. In the case
of Fig. 3.1, this depends on whether all products are bought by all customers
during the period of time considered. For example, not all customers may have
ordered products of all categories during all quarters of the year. Actually, in
real-world applications, cubes are typically sparse.

56

3.1.1

Data Warehouse Concepts

Hierarchies

We have said that the granularity of a data cube is determined by the

combination of the levels corresponding to each axis of the cube. In Fig. 3.1,
the dimension levels are indicated between parentheses: Category for the
Product dimension, Quarter for the Time dimension, and City for the Customer
dimension.
In order to extract strategic knowledge from a cube, it is necessary to view
its data at several levels of detail. In our example, an analyst may want to
see the sales gures at a ner granularity, such as at the month level, or at
a coarser granularity, such as at the customers country level. Hierarchies
allow this possibility by dening a sequence of mappings relating lower-level,
detailed concepts to higher-level, more general concepts. Given two related
levels in a hierarchy, the lower level is called the child and the higher level
is called the parent. The hierarchical structure of a dimension is called the
dimension schema, while a dimension instance comprises the members at
all levels in a dimension. Figure 3.2 shows the simplied hierarchies for our
cube example. In the next chapter, we give full details of how dimension
hierarchies are modeled. In the Product dimension, products are grouped
in categories. For the Time dimension, the lowest granularity is Day, which
aggregates into Month, which in turn aggregates into Quarter, Semester,
and Year. Similarly, for the Customer dimension, the lowest granularity is
Customer, which aggregates into City, State, Country, and Continent. It is usual
to represent the top of the hierarchy with a distinguished level called All.
Product

Time

Customer

All

All

All

Category

Year

Continent

Product

Semester

Country

Quarter

State

Month

City

Day

Customer

Fig. 3.2 Hierarchies of the Product, Time, and Customer dimensions

At the instance level, Fig. 3.3 shows an example of the Product dimension.1
Each product at the lowest level of the hierarchy can be mapped to a
1

Note that, as indicated by the ellipses, not all nodes of the hierarchy are shown.

3.1 Multidimensional Model

57

corresponding category. All categories are grouped under a member called

all, which is the only member of the distinguished level All. This member is
used for obtaining the aggregation of measures for the whole hierarchy, that
is, for obtaining the total sales for all products.
All

all

Beverages

Category

Product

Chai

Chang

...

...

Ikura

Seafood

Konbu

...

Fig. 3.3 Members of a hierarchy Product Category

In real-world applications, there exist many kinds of hierarchies. For

example, the hierarchy depicted in Fig. 3.3 is balanced, since there is
the same number of levels from each individual product to the root of
the hierarchy. In Chaps. 4 and 5, we shall study these and other kinds of
hierarchies in detail, covering both their conceptual representation and their
implementation in current data warehouse and OLAP systems.

3.1.2

Measures

Each measure in a cube is associated with an aggregation function that

combines several measure values into a single one. Aggregation of measures
takes place when one changes the level of detail at which data in a cube are
visualized. This is performed by traversing the hierarchies of the dimensions.
For example, if we use the Customer hierarchy in Fig. 3.2 for changing the
granularity of the data cube in Fig. 3.1 from City to Country, then the sales
gures for all customers in the same country will be aggregated using, for
example, the SUM operation. Similarly, total sales gures will result in a cube
containing one cell with the total sum of the quantities of all products, that
is, this corresponds to visualizing the cube at the All level of all dimension
hierarchies.
Summarizability refers to the correct aggregation of cube measures
along dimension hierarchies, in order to obtain consistent aggregation results.
To ensure summarizability, a set of conditions may hold. Below, we list some
of these conditions
Disjointness of instances: The grouping of instances in a level with
respect to their parent in the next level must result in disjoint subsets.
For example, in the hierarchy of Fig. 3.3, a product cannot belong to two

58

Data Warehouse Concepts

categories. If this were the case, each product sales would be counted twice,
one for each category.
Completeness: All instances must be included in the hierarchy and each
instance must be related to one parent in the next level. For example,
the instances of the Time hierarchy in Fig. 3.2 must contain all days in
the period of interest, and each day must be assigned to a month. If
this condition were not satised, the aggregation of the results would be
incorrect, since there would be dates for which sales will not be counted.
Correctness: It refers to the correct use of the aggregation functions. As
explained next, measures can be of various types, and this determines the
kind of aggregation function that can be applied to them.
According to the way in which they can be aggregated, measures can be
classied as follows:
Additive measures can be meaningfully summarized along all the
dimensions, using addition. These are the most common type of measures.
For example, the measure Quantity in the cube of Fig. 3.1 is additive: it can
be summarized when the hierarchies in the Product, Time, and Customer
dimensions are traversed.
Semiadditive measures can be meaningfully summarized using addition
along some, but not all, dimensions. A typical example is that of
inventory quantities, which cannot be meaningfully aggregated in the Time
dimension, for instance, by adding the inventory quantities for two dierent
quarters.
Nonadditive measures cannot be meaningfully summarized using addition across any dimension. Typical examples are item price, cost per unit,
and exchange rate.
Thus, in order to dene a measure, it is necessary to determine the
aggregation functions that will be used in the various dimensions. This is
particularly important in the case of semiadditive and nonadditive measures.
For example, a semiadditive measure representing inventory quantities
can be aggregated computing the average along the Time dimension and
computing the sum along other dimensions. Averaging can also be used
for aggregating nonadditive measures such as item price or exchange rate.
However, depending on the semantics of the application, other functions such
as the minimum, maximum, or count could be used instead.
In order to allow users to interactively explore the cube data at dierent
granularities, optimization techniques based on aggregate precomputation are
used. To avoid computing the whole aggregation from scratch each time the
data warehouse is queried, OLAP tools implement incremental aggregation
mechanisms. However, incremental aggregation computation is not always
possible, since this depends on the kind of aggregation function used. This
leads to another classication of measures, which we explain next.

3.2 OLAP Operations

59

Distributive measures are dened by an aggregation function that can

be computed in a distributed way. Suppose that the data are partitioned
into n sets and that the aggregate function is applied to each set,
giving n aggregated values. The function is distributive if the result of
applying it to the whole data set is the same as the result of applying a
function (not necessarily the same) to the n aggregated values. The usual
aggregation functions such as the count, sum, minimum, and maximum are
distributive. However, the distinct count function is not. For instance, if we
partition the set of measure values {3, 3, 4, 5, 8, 4, 7, 3, 8} into the subsets
{3, 3, 4}, {5, 8, 4}, and {7, 3, 8}, summing up the result of the distinct count
function applied to each subset gives us a result of 8, while the answer over
the original set is 5.
Algebraic measures are dened by an aggregation function that can be
expressed as a scalar function of distributive ones. A typical example of an
algebraic aggregation function is the average, which can be computed by
dividing the sum by the count, the latter two functions being distributive.
Holistic measures are measures that cannot be computed from other
subaggregates. Typical examples include the median, the mode, and the
rank. Holistic measures are expensive to compute, especially when data
are modied, since they must be computed from scratch.

3.2

OLAP Operations

As already said, a fundamental characteristic of the multidimensional model

is that it allows one to view data from multiple perspectives and at several
levels of detail. The OLAP operations allow these perspectives and levels of
detail to be materialized by exploiting the dimensions and their hierarchies,
thus providing an interactive data analysis environment.
Figure 3.4 presents a possible scenario that shows how an end user can
operate over a data cube in order to analyze data in dierent ways. Later
in this section, we present the OLAP operations in detail. Our user starts
from Fig. 3.4a, a cube containing quarterly sales quantities (in thousands) by
product categories and customer cities for the year 2012.
The user rst wants to compute the sales quantities by country. For this,
she applies a roll-up operation to the Country level along the Customer
dimension. The result is shown in Fig. 3.4b. While the original cube contained
four values in the Customer dimension, one for each city, the new cube
contains two values, each one corresponding to one country. The remaining
dimensions are not aected. Thus, the values in cells pertaining to Paris and
Lyon in a given quarter and category contribute to the aggregation of the
corresponding values for France. The computation of the cells pertaining to
Germany proceeds analogously.

60

Data Warehouse Concepts

b
C
( C u s to
ou m
n tr e r
y)
Time (Quarter)

Cu
s
(C tom
i ty e
) r

Kln
Berlin
Lyon
Paris

Germany
France

Time (Quarter)

Q1
Q2
Q3
Q4

Q1
Q2
Q3
Q4

Produce Seafood

Produce Seafood

Beverages Condiments
Product (Category)

Beverages Condiments
Product (Category)

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Feb

Time (Quarter)

Time (Quarter)

Jan

Mar
...
Dec

Q1
Q2
Q3
Q4

Produce Seafood

Condiments Seafood
Beverages

Beverages Condiments
Product (Category)

Produce

Product (Category)

(C Pro
at du
eg c
or t
y)

Customer (City)

Seafood
Condiments
Produce
Beverages
Paris
Lyon
Berlin
Kln
Q1 Q2 Q3 Q4
Time (Quarter)

Fig. 3.4 OLAP operations. (a) Original cube. (b) Roll-up to the Country level.
(c) Drill-down to the Month level. (d) Sort product by name. (e) Pivot

3.2 OLAP Operations

61

g
Cu
s
Time
( to
(Quarter) City me
) r

f
Time (Quarter)

Q1

Lyon
Paris

Q2
Q3
Q4

Q1
Q2
Produce Seafood

Produce Seafood

Beverages Condiments

Beverages Condiments
Product (Category)

Product (Category)

Q1

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris
Q1

Time (Quarter)

Time (Quarter)

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Q2
Q3
Q4

Q2
Q3
Q4

Produce Seafood

Produce Seafood

Beverages Condiments
Product (Category)

Beverages Condiments
Product (Category)

Q1

k
Time (Quarter)

Time (Quarter)

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Q2
Q3
Q4
Produce Seafood
Beverages Condiments
Product (Category)

Q1

84

89 106 84

Q2

82

77

93

79

Q3 105 72

65

88

Q4 112 61

96 102

Lyon

Kln

Paris
Berlin
Customer (City)

Fig. 3.4 (continued) (f ) Slice on City='Paris'. (g) Dice on City='Paris' or 'Lyon' and
Quarter='Q1' or 'Q2'. (h) Cube for 2011. (i) Drill-across operation. (j) Percentage
change. (k) Total sales by quarter and city

62

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris
Q1

Time (Quarter)

Time (Quarter)

Q1
Q2
Q3
Q4

Q2
Q3
Q4
Produce Seafood

Produce Seafood

Beverages Condiments
Product (Category)

Beverages Condiments
Product (Category)

o
Kln
Berlin
Lyon
Paris

Cu
s
(C tom
ity e
) r

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris
Q1

Time (Quarter)

Time (Quarter)

Data Warehouse Concepts

Q2
Q3
Q4

Q1
Q2
Q3
Q4
Produce Seafood

Produce Seafood

Beverages Condiments
Product (Category)

Beverages Condiments
Product (Category)

Time (Quarter)

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris
Q1
Q2
Q3
Q4
Produce Seafood
Beverages Condiments
Product (Category)

Fig. 3.4 (continued) (l) Maximum sales by quarter and city. (m) Top two sales by
quarter and city. (n) Top 70% by city and category ordered by ascending quarter.
(o) Top 70% by city and category ordered by descending quantity. (p) Rank quarter
by category and city ordered by descending quantity

3.2 OLAP Operations

Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Kln
Berlin
Lyon
Paris

Cu
s
(C tom
ity e
) r

63

Jan
Time (Quarter)

Time (Quarter)

Jan
Feb
Mar
...
Dec

Feb
Mar
...
Dec

Produce Seafood

Produce Seafood

Beverages Condiments
Product (Category)

Beverages Condiments
Product (Category)

Q1

t
Cu
s
(C tom
ity e
) r

Kln
Berlin
Lyon
Paris

Time (Quarter)

Time (Quarter)

Cu
s
(C tom
ity e
) r

Madrid
Bilbao
Kln
Berlin
Lyon
Paris

Q2
Q3
Q4
Produce Seafood
Beverages Condiments
Product (Category)

Q1
Q2
Q3
Q4
Produce Seafood
Beverages Condiments
Product (Category)

Fig. 3.4 (continued) (q) Three-month moving average. (r) Year-to-date computation. (s) Union of the original cube and another cube with data from Spain.
(t) Dierence of the original cube and the cube in (m)

Our user then notices that sales of the category Seafood in France are
signicantly higher in the rst quarter compared to the other ones. Thus,
she rst takes the cube back to the City aggregation level and then applies a
drill-down along the Time dimension to the Month level to nd out whether
this high value occurred during a particular month. In this way, she discovers
that, for some reason yet unknown, sales in January soared both in Paris and
in Lyon, as shown in Fig. 3.4c.

64

Data Warehouse Concepts

Our user now wants to explore alternative visualizations of the cube to

better understand the data contained in it. For this, she sorts the products
by name, as shown in Fig. 3.4d. Then, she wants to see the cube with the
Time dimension on the x-axis. Therefore, she takes the original cube and
rotates the axes of the cube without changing granularities. This is called
pivoting and is shown in Fig. 3.4e.
Continuing her browsing of the original cube, the user then wants to
visualize the data only for Paris. For this, she applies a slice operation
that results in the subcube depicted in Fig. 3.4f. Here, she obtained a twodimensional matrix, where each column represents the evolution of the sales
quantity by category and quarter, that is, a collection of time series.
As her next operation, our user goes back to the original cube and builds
a subcube, with the same dimensionality, but only containing sales gures
for the rst two quarters and for the cities Lyon and Paris. This is done with
a dice operation, as shown in Fig. 3.4g.
Our user now wants to compare the sales quantities in 2012 with those in
2011. For this, she needs the cube in Fig. 3.4h, which has the same structure
as the one for 2012 given in Fig. 3.4a. She wants to have the measures in
the two cubes consolidated in a single one. Thus, she uses the drill-across
operation that, given two cubes, builds a new one with the measures of both
in each cell. This is shown in Fig. 3.4i.
The user now wants to compute the percentage change of sales between the
2 years. For this, she takes the cube resulting from the drill-across operation
above, and applies to it the add measure operation, which computes a new
value for each cell from the values in the original cube. The new measure is
shown in Fig. 3.4j.
After all these manipulations, the user wants to aggregate data in various
ways. Given the original cube in Fig. 3.4a, she rst wants to compute to
total sales by quarter and city. This is obtained by the sum aggregation
operation, whose result is given in Fig. 3.4k. Then, the user wants to obtain
the maximum sales by quarter and city, and for this, she uses the max
operation to obtain the cube in Fig. 3.4l. After seeing the result, she decides
that she needs more information; thus, she computes the top two sales by
quarter and city, which is also obtained with the max operation yielding the
cube in Fig. 3.4m.
In the next step, the user goes back to the original cube in Fig. 3.4a and
computes the quarterly sales that amount to 70% of the total sales by city and
category. She explores this in two possible ways: according to the ascending
order of quarters, as shown in Fig. 3.4n, and according to the descending
order of quantity, as shown in Fig. 3.4o. In both cases, she applies the top
percent aggregation operation. She also wants to rank the quarterly sales
by category and city in descending order of quantity, which is obtained in
Fig. 3.4p.

3.2 OLAP Operations

65

Now, the user wants to apply window operations to the cube in Fig. 3.4c
in order to see how monthly sales behave. She starts by requesting a 3-month
moving average to obtain the result in Fig. 3.4q. Then, she asks the year-todate computation whose result is given in Fig. 3.4r.
Finally, the user wants to add to the original cube data from Spain, which
are contained in another cube. She obtains this by performing a union of the
two cubes, whose result is given in Fig. 3.4s. As another operation, she also
wants to remove from the original cube all sales measures except the top two
sales by quarter and city. For this, she performs the dierence of the original
cube in Fig. 3.4a and the cube in Fig. 3.4m, yielding the result in Fig. 3.4t.
The OLAP operations illustrated in Fig. 3.4 can be dened in a way
analogous to the relational algebra operations introduced in Chap. 2.
The roll-up operation aggregates measures along a dimension hierarchy
(using an aggregate function) to obtain measures at a coarser granularity.
The syntax for the roll-up operation is:
ROLLUP(CubeName, (Dimension Level)*, AggFunction(Measure)*)

where Dimension Level indicates to which level in a dimension the roll-up

is to be performed and function AggFunction is applied to summarize the
measure. When there is more than one measure in a cube, we must specify
an aggregate function for each measure that will be kept in the cube. All the
measures for which the aggregation is not specied will be removed from the
cube. In the example given in Fig. 3.4b, we applied the operation:
ROLLUP(Sales, Customer Country, SUM(Quantity))

When querying a cube, a usual operation is to roll up a few dimensions

to particular levels and to remove the other dimensions through a roll-up
to the All level. In a cube with n dimensions, this can be obtained by
applying n successive ROLLUP operations. The ROLLUP* operation provides
a shorthand notation for this sequence of operations. The syntax is as follows:
ROLLUP*(CubeName, [(Dimension Level)*], AggFunction(Measure)*)

For example, the total quantity by quarter can be obtained as follows:

ROLLUP*(Sales, Time Quarter, SUM(Quantity))

which performs a roll-up along the Time dimension to the Quarter level and
the other dimensions (in this case Customer and Product) to the All level. On
the other hand, if the dimensions are not specied as in
ROLLUP*(Sales, SUM(Quantity))

all the dimensions of the cube will be rolled up to the All level, yielding a
single cell containing the overall sum of the Quantity measure.

66

Data Warehouse Concepts

A usual need when applying a roll-up operation is to count the number of

members in one of the dimensions removed from the cube. For example, the
following query obtains the number of distinct products sold by quarter:
ROLLUP*(Sales, Time Quarter, COUNT(Product) AS ProdCount)

In this case, a new measure ProdCount will be added to the cube. We will see
below other ways to add measures to a cube.
In many real-world situations, hierarchies are recursive, that is, they
contain a level that rolls up to itself. A typical example is a supervision
hierarchy over employees. Such hierarchies are discussed in detail in Chap. 4.
The particularity of such hierarchies is that the number of levels of the
hierarchy is not xed at the schema level, but it depends on its members.
The RECROLLUP operation is used to aggregate measures over recursive
hierarchies by iteratively performing roll-ups over the hierarchy until the top
level is reached. The syntax of this operation is as follows:
RECROLLUP(CubeName, Dimension Level, Hierarchy, AggFct(Measure)*)

We will show an example of such an operation in Sect. 4.4.

The drill-down operation performs the inverse of the roll-up operation,
that is, it goes from a more general level to a more detailed level in a hierarchy.
The syntax of the drill-down operation is as follows:
DRILLDOWN(CubeName, (Dimension Level)*)

where Dimension Level indicates to which level in a dimension we want to

drill down to. In our example given in Fig. 3.4c, we applied the operation
DRILLDOWN(Sales, Time Month)

The sort operation returns a cube where the members of a dimension have
been sorted. The syntax of the operation is as follows:
SORT(CubeName, Dimension, (Expression [ {ASC | DESC | BASC | BDESC} ])*)

where the members of Dimension are sorted according to the value of

Expression either in ascending or descending order. In the case of ASC
or DESC, members are sorted within their parent (i.e., respecting the
hierarchies), whereas in the case of BASC or BDESC, the sorting is performed
across all members (i.e., irrespective of the hierarchies). The ASC is the
default option. For example, the following expression
SORT(Sales, Product, ProductName)

sorts the members of the Product dimension in ascending order of their name,
as shown in Fig. 3.4d. Here, ProductName is supposed to be an attribute of
products. When the cube contains only one dimension, the members can
be sorted based on its measures. For example, if SalesByQuarter is obtained
from the original cube by aggregating sales by quarter for all cities and all
categories, the following expression

3.2 OLAP Operations

67

SORT(SalesByQuarter, Time, Quantity DESC)

sorts the members of the Time dimension on descending order of the Quantity
measure.
The pivot (or rotate) operation rotates the axes of a cube to provide an
alternative presentation of the data. The syntax of the operation is as follows:
PIVOT(CubeName, (Dimension Axis)*)

where the axes are specied as {X, Y, Z, X1, Y1, Z1, . . . }. Thus, the example
illustrated in Fig. 3.4e is expressed by:
PIVOT(Sales, Time X, Customer Y, Product Z)

The slice operation removes a dimension in a cube, that is, a cube of

n 1 dimensions is obtained from a cube of n dimensions. The syntax of this
operation is:
SLICE(CubeName, Dimension, Level = Value)

where the Dimension will be dropped by xing a single Value in the Level.
The other dimensions remain unchanged. The example illustrated in Fig. 3.4f
is expressed by:
SLICE(Sales, Customer, City = 'Paris')

The slice operation assumes that the granularity of the cube is at the specied
level of the dimension (in the example above, at the city level). Thus, a
granularity change by means of a ROLLUP or DRILLDOWN operation is often
needed prior to applying the slice operation.
The dice operation keeps the cells in a cube that satisfy a Boolean
condition . The syntax for this operation is
DICE(CubeName, )

where is a Boolean condition over dimension levels, attributes, and

measures. The DICE operation is analogous to the relational algebra selection
(R), where the argument is a cube instead of a relation. The example
illustrated in Fig. 3.4g is expressed by:
DICE(Sales, (Customer.City = 'Paris' OR Customer.City = 'Lyon') AND
(Time.Quarter = 'Q1' OR Time.Quarter = 'Q2'))

The rename operation returns a cube where some schema elements or

members have been renamed. The syntax is:
RENAME(CubeName, ({SchemaElement | Member} NewName)*)

For example, the following expression

RENAME(Sales, Sales Sales2012, Quantity Quantity2012)

renames the cube in Fig. 3.4a and its measure. As another example,

68

Data Warehouse Concepts

RENAME(Sales, Customer.all AllCustomers)

will rename the all member of the customer dimension.

The drill-across operation combines cells from two data cubes that have
the same schema and instances. The syntax of the operation is:
DRILLACROSS(CubeName1, CubeName2, [Condition])

The DRILLACROSS operation is analogous to a full outer join in the relational

algebra. If the condition is not stated, it corresponds to an outer equijoin.
Given the cubes in Fig. 3.4a, h, the cube in Fig. 3.4i is expressed by:
Sales2011-2012 DRILLACROSS(Sales2011, Sales2012)

Notice that a renaming of the cube and the measure, as stated above, is
necessary prior to applying the drill-across operation. Notice also that the
resulting cube is named Sales2011-2012. On the other hand, if in the Sales
cube of Fig. 3.4c we want to compare the sales of a month with those of the
previous month, this can be expressed in two steps as follows:
Sales1 RENAME(Sales, Quantity PrevMonthQuantity)
Result DRILLACROSS(Sales1, Sales, Sales1.Time.Month+1 = Sales.Time.Month)

In the rst step, we create a temporary cube Sales1 by renaming the measure.
In the second step, we perform the drill across of the two cubes by combining
a cell in Sales1 with the cell in Sales corresponding to the subsequent month.
As already stated, the join condition above corresponds to an outer join.
Notice that the Sales cube in Fig. 3.4a contains measures for a single year.
Thus, in the result above, the cells corresponding to January and December
will contain a null value in one of the two measures. As we will see in
Sect. 4.4, when the cube contains measures for several years, the join condition
must take into account that measures of January must be joined with those
of December of the preceding year. Notice also that the cube has three
dimensions and the join condition in the query above pertains to only one
dimension. For the other dimensions, it is supposed that an outer equijoin is
performed.
The add measure operation adds new measures to the cube computed
from other measures or dimensions. The syntax for this operation is as follows:
ADDMEASURE(CubeName, (NewMeasure = Expression, [AggFct])* )

where Expression is a formula over measures and dimension members and

AggFct is the default aggregation function for the measure, SUM being the
default. For example, given the cube in Fig. 3.4i, the measure shown in
Fig. 3.4j is expressed by:
ADDMEASURE(Sales2011-2012, PercentageChange =
(Quantity2011-Quantity2012)/Quantity2011)

The drop measure operation removes one or several measures from a

cube. The syntax is as follows:

3.2 OLAP Operations

69

DROPMEASURE(CubeName, Measure*)

For example, given the result of the add measure above, the cube illustrated
in Fig. 3.4j is expressed by:
DROPMEASURE(Sales2011-2012, Quantity2011, Quantity2012)

We have seen that the roll-up operation aggregates measures when

displaying the cube at coarser level. On the other hand, we also need to
aggregate measures of a cube at the current granularity, that is, without
performing a roll-up operation. The syntax for this is as follows:
AggFunction(CubeName, Measure) [BY Dimension*]

Usual aggregation operations are SUM, AVG, COUNT, MIN, and MAX.
In addition to these, we use extended versions of MIN and MAX, which
have an additional argument that is used to obtain the n minimum or
maximum values. Further, TOPPERCENT and BOTTOMPERCENT select
the members of a dimension that cumulatively account for x percent of a
measure. Analogously, RANK and DENSERANK are used to rank the members
of a dimension according to a measure. We show next examples of these
operations.
For example, the cube in Fig. 3.4a is at the Quarter and City levels. The
total sales by quarter and city can be obtained by
SUM(Sales, Quantity) BY Time, Customer

This will yield the two-dimensional cube in Fig. 3.4k. On the other hand, to
obtain the total sales by quarter, we can write
SUM(Sales, Quantity) BY Time

which returns a one-dimensional cube with values for each quarter. Notice
that in the query above, a roll-up along the Customer dimension up to the
All level is performed before applying the aggregation operation. Finally, to
obtain the overall sales, we can write
SUM(Sales, Quantity)

which will result in a single cell.

Aggregation functions in OLAP can be classied in two types. Cumulative aggregation functions compute the measure value of a cell from
several other cells. Examples of cumulative functions are SUM, COUNT,
and AVG. On the other hand, ltering aggregation functions lter the
members of a dimension that appears in the result. Examples of these
functions are MIN and MAX. The distinction between these two types
of aggregation functions is important in OLAP since ltering aggregation
functions must compute not only the aggregated value but must also
determine the dimension members that belong to the result. As an example,

70

Data Warehouse Concepts

when asking for the best-selling employee, we must compute the maximum
sales amount but also identify who is the employee that performed best.
Therefore, when applying an aggregation operation, the resulting cube will
have dierent dimension members depending on the type of the aggregation
function. For example, given the cube in Fig. 3.4a, the total overall quantity
can be obtained by the expression
SUM(Sales, Quantity)

This will yield a single cell, whose coordinates for the three dimensions will
be all equal to all. On the other hand, when computing the overall maximum
quantity as follows
MAX(Sales, Quantity)

we will obtain the cell with value 47 and coordinates Q4, Condiments, and
Paris (we suppose that cells that are hidden in Fig. 3.4a contain a smaller
value for this measure). Similarly, the following expression
SUM(Sales, Quantity) BY Time, Customer

returns the total sales by quarter and customer, resulting in the cube given
in Fig. 3.4k. This cube has three dimensions, where the Product dimension
only contains the member all. On the other hand,
MAX(Sales, Quantity) BY Time, Customer

will yield the cube in Fig. 3.4l, where only the cells containing the maximum
by time and customer will have values, while the other ones will be lled with
null values. Similarly, the two maximum quantities by product and customer
as shown in Fig. 3.4m can be obtained as follows:
MAX(Sales, Quantity, 2) BY Time, Customer

Notice that in the example above, we requested the two maximum quantities
by time and customer. If in the cube there are two or more cells that tie for
the last place in the limited result set, then the number of cells in the result
could be greater than two. For example, this is the case in Fig. 3.4m for Berlin
and Q1, where there are three values in the result, that is, 33, 25, and 25.
To compute top or bottom percentages, the order of the cells must be
specied. For example, to compute the top 70% of the measure quantity by
city and category ordered by quarter, as shown in Fig. 3.4n, we can write
TOPPERCENT(Sales, Quantity, 70) BY City, Category ORDER BY Quarter ASC

The operation computes the running sum of the sales by city and category
starting with the rst quarter and continues until the target percentage is
reached. In the example above, the sales by city and category for the rst
three quarters covers 70% of the sales. Similarly, the top 70% of the measure
quantity by city and category ordered by quantity, as shown in Fig. 3.4o, can
be obtained by

3.2 OLAP Operations

71

TOPPERCENT(Sales, Quantity, 70) BY City, Category ORDER BY Quantity DESC

The rank operation also requires the specication of the order of the cells.
As an example, to rank quarters by category and city order by descending
quantity, as shown in Fig. 3.4p, we can write
RANK(Sales, Time) BY Category, City ORDER BY Quantity DESC

The rank and the dense rank operations dier in the case of ties. The former
assigns the same rank to ties, with the next ranking(s) skipped. For example,
in Fig. 3.4p, there is a tie in the quarters for Seafood and Koln, where Q2 and
Q4 are in the rst rank and Q3 and Q1 are in the third and fourth ranks,
respectively. If the dense rank is used, then Q3 and Q1 would be in the second
and third ranks, respectively.
In the examples above, the new measure value in a cell is computed from
the values of other measures in the same cell. However, we often need to
compute measures where the value of a cell is obtained by aggregating the
measures of several nearby cells. Examples of these include moving average
and year-to-date computations. For this, we need to dene a subcube that
is associated with each cell and perform the aggregation over this subcube.
These functions correspond to the window functions in SQL that will be
described in Chap. 5. For example, given the cube in Fig. 3.4c, the 3-month
moving average in Fig. 3.4q can be obtained by
ADDMEASURE(Sales, MovAvg = AVG(Quantity) OVER
Time 2 CELLS PRECEDING)

Here, the moving average for January is equal to the measure in January,
since there are no previous cells. Analogously, the measure for February is
the average of the values of January and February. Finally, the average for
the remaining months is computed from the measure value of the current
month and the two preceding ones. Notice that in the window functions,
it is supposed that the members of the dimension over which the window is
constructed are already sorted. For this, a sort operation can be applied prior
to the application of the window aggregate function.
Similarly, to compute the year-to-date sum in Fig. 3.4r, we can write
ADDMEASURE(Sales, YTDQuantity = SUM(Quantity) OVER
Time ALL CELLS PRECEDING)

Here, the window over which the aggregation function is applied contains
the current cell and all the previous ones, as indicated by ALL CELLS
PRECEDING.
The union operation merges two cubes that have the same schema
but disjoint instances. For example, if CubeSpain is a cube having the
same schema as our original cube but containing only the sales to Spanish
customers, the cube in Fig. 3.4s is obtained by
UNION(Sales, SalesSpain)

72

Data Warehouse Concepts

The union operation is also used to display dierent granularities on the

same dimension. For example, if SalesCountry is the cube in Fig. 3.4b, then
the following operation
UNION(Sales, SalesCountry)

results in a cube containing sales measures summarized by city and by

country.
The dierence operation removes the cells in a cube that exist in another
one. Obviously, the two cubes must have the same schema. For example, if
TopTwoSales is the cube in Fig. 3.4m, then the following operation
DIFFERENCE(Sales, TopTwoSales)

will result in the cube in Fig. 3.4t, which contains all sales measures except
for the top two sales by quarter and city.
Finally, the drill-through operation allows one to move from data at the
bottom level in a cube to data in the operational systems from which the cube
was derived. This could be used, for example, if one were trying to determine
the reason for outlier values in a data cube.
Table 3.1 summarizes the OLAP operations we have presented in this
section. In addition to the basic operations described above, OLAP tools
provide a great variety of mathematical, statistical, and nancial operations
for computing ratios, variances, interest, depreciation, currency conversions,
etc.

3.3

Data Warehouses

A data warehouse is a repository of integrated data obtained from several

sources for the specic purpose of multidimensional data analysis. More
technically, a data warehouse is dened as a collection of subject-oriented,
integrated, nonvolatile, and time-varying data to support management
decisions. We explain next these characteristics:
Subject oriented means that data warehouses focus on the analytical
needs of dierent areas of an organization. These areas vary depending
on the kind of activities performed by the organization. For example, in
the case of a retail company, the analysis may focus on product sales
or inventory management. In operational databases, on the contrary, the
focus is on specic functions that applications must perform, for example,
registering sales of products or inventory replenishment.
Integrated means that data obtained from several operational and
external systems must be joined together, which implies solving problems
due to dierences in data denition and content, such as dierences
in data format and data codication, synonyms (elds with dierent

3.3 Data Warehouses

73

Table 3.1 Summary of the OLAP operations

Operation Purpose
Add measure Adds new measures to a cube computed from other measures or
dimensions.
Aggregation Aggregate the cells of a cube, possibly after performing a groupoperations
ing of cells.
Dice
Keeps the cells of a cube that satisfy a Boolean condition over
dimension levels, attributes, and measures.
Dierence
Removes the cells of a cube that are in another cube. Both cubes
must have the same schema.
Drill-across
Merges two cubes that have the same schema and instances using
a join condition.
Drill-down
Disaggregates measures along a hierarchy to obtain data at a
ner granularity. It is the opposite of the roll-up operation.
Drill-through Shows data in the operational systems from which the cube was
derived.
Drop measure Removes measures from a cube.
Pivot
Rotates the axes of a cube to provide an alternative presentation
of its data.
Recursive
Performs an iteration of roll-ups over a recursive hierarchy until
roll-up
the top level is reached.
Rename
Renames one or several schema elements of a cube.
Roll-up
Aggregates measures along a hierarchy to obtain data at a coarser
granularity. It is the opposite of the drill-down operation.
Roll-up*
Shorthand notation for a sequence of roll-up operations.
Slice
Removes a dimension from a cube by xing a single value in a
level of the dimension.
Sort
Orders the members of a dimension according to an expression.
Union
Combines the cells of two cubes that have the same schema but
disjoint members.

names but the same data), homonyms (elds with the same name but
dierent meanings), multiplicity of occurrences of data, and many others.
In operational databases these problems are typically solved in the design
phase.
Nonvolatile means that durability of data is ensured by disallowing data
modication and removal, thus expanding the scope of the data to a longer
period of time than operational systems usually oer. A data warehouse
gathers data encompassing several years, typically 510 years or beyond,
while data in operational databases is often kept for only a short period
of time, for example, from 2 to 6 months, as required for daily operations,
and it may be overwritten when necessary.
Time varying indicates the possibility of retaining dierent values for
the same information, as well as the time when changes to these values
occurred. For example, a data warehouse in a bank might store information

74

Data Warehouse Concepts

about the average monthly balance of clients accounts for a period

covering several years. In contrast, an operational database may not have
explicit temporal support, since sometimes it is not necessary for day-today operations and it is also dicult to implement.
A data warehouse is aimed at analyzing the data of an entire organization.
It is often the case that particular departments or divisions of an organization
only require a portion of the organizational data warehouse specialized for
their needs. For example, a sales department may only need sales data, while
a human resources department may need demographic data and data about
the employees. These departmental data warehouses are called data marts.
However, these data marts are not necessarily private to a department; they
may be shared with other interested parts of the organization.
A data warehouse can be seen as a collection of data marts. This view
represents a bottom-up approach in which a data warehouse is built by
rst building the smaller data marts and then merging these to obtain the
data warehouse. This can be a good approach for organizations not willing
to take the risk of building a large data warehouse, which may take a long
time to complete, or organizations that need fast results. On the other hand,
in the classic data warehouse view, data marts are obtained from the data
warehouse in a top-down fashion. In this approach, a data mart is sometimes
just a logical view of a data warehouse.
Table 3.2 shows several aspects that dierentiate operational database (or
OLTP) systems from data warehouse (or OLAP) systems. We analyze next
in detail some of these dierences.
Typically, the users of OLTP systems are operations and employees who
perform predened operations through transactional applications, like payroll
systems or ticket reservation systems. Data warehouse users, on the other
hand, are usually located higher in the organizational hierarchy and use
interactive OLAP tools to perform data analysis, for example, to detect
salary inconsistencies or most frequently chosen tourist destinations (lines
12). Therefore, it is clear that data for OLTP systems should be current
and detailed, while data analytics require historical, summarized data (line
3). The dierence on data organization (line 4) follows from the type of use
of OLTP and OLAP systems.
From a more technical viewpoint, data structures for OLTP are optimized
for rather small and simple transactions, which will be carried out frequently
and repeatedly. In addition, data access for OLTP requires reading and
writing data les. For example, in the Northwind database application, a
user may be able to frequently insert new orders, modify old ones, and delete
orders if customers cancel them. Thus, the number of records accessed by an
OLTP transaction is usually small (e.g., the records involved in a particular
sales order). On the other hand, data structures for OLAP must support
complex aggregation queries, thus requiring access to all the records in one or
more tables, which will translate in long, complex SQL queries. Furthermore,

3.3 Data Warehouses

75

Table 3.2 Comparison between operational databases and data warehouses

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Aspect
User type
Usage
Data content
Data organization

Operational databases
Operators, oce employees
Predictable, repetitive
Current, detailed data
According to operational
needs
Data structures
Optimized for small
transactions
Access frequency High
Access type
Read, insert, update, delete
Number of records Few
per access
Response time
Short
Concurrency level High
Lock utilization
Needed
Update frequency High
Data redundancy Low (normalized tables)
Data modeling
UML, ER model

Data warehouses
Managers, executives
Ad hoc, nonstructured
Historical, summarized data
According to analysis needs
Optimized for complex
queries
From medium to low
Read, append only
Many
Can be long
Low
Not needed
None
High (denormalized tables)
Multidimensional model

OLAP systems are not so frequently accessed as OLTP systems. For example,
a system handling purchase orders is frequently accessed, while performing
analysis of orders may not be that frequent. Also, data warehouse records
are usually accessed in read mode (lines 58). From the above, it follows
that OLTP systems usually have a short query response time, provided the
appropriate indexing structures are dened, while complex OLAP queries can
take a longer time to complete (line 9).
OLTP systems have normally a high number of concurrent accesses and
therefore require locking or other concurrency management mechanisms to
ensure safe transaction processing (lines 1011). On the other hand, OLAP
systems are read only, and therefore queries can be submitted and computed
concurrently, with no locking or complex transaction processing requirements.
Further, the number of concurrent users in an OLAP system is usually low.
Finally, OLTP systems are constantly being updated online through transactional applications, while OLAP systems are updated o-line periodically.
This leads to dierent modeling choices. OLTP systems are modeled using
UML or some variation of the ER model studied in Chap. 2. Such models
lead to a highly normalized schema, adequate for databases that support
frequent transactions, to guarantee consistency and reduce redundancy.
OLAP designers use the multidimensional model, which, at the logical level
(as we will see in Chap. 5), leads in general to a denormalized database
schema, with a high level of redundancy, which favors query processing (lines
1214).

76

3.4

Data Warehouse Concepts

Data Warehouse Architecture

We are now ready to present a general data warehouse architecture that will
be used throughout the book. This architecture, depicted in Fig. 3.5, consists
of several tiers:
The back-end tier is composed of extraction, transformation, and
loading (ETL) tools, used to feed data into the data warehouse from
operational databases and other data sources, which can be internal or
external from the organization, and a data staging area, which is an
intermediate database where all the data integration and transformation
processes are run prior to the loading of the data into the data warehouse.
The data warehouse tier is composed of an enterprise data warehouse and/or several data marts and a metadata repository storing
information about the data warehouse and its contents.
The OLAP tier is composed of an OLAP server, which provides a
multidimensional view of the data, regardless of the actual way in which
data are stored in the underlying system.
The front-end tier is used for data analysis and visualization. It contains
client tools such as OLAP tools, reporting tools, statistical tools,
and data mining tools.
We now describe in detail the various components of the above architecture.

3.4.1

Back-End Tier

In the back-end tier, the process commonly known as extraction, transformation, and loading is performed. As the name indicates, it is a three-step
process as follows:
Extraction gathers data from multiple, heterogeneous data sources.
These sources may be operational databases but may also be les in various
formats; they may be internal to the organization or external to it.
In order to solve interoperability problems, data are extracted whenever
possible using application programming interfaces (APIs) such as ODBC
(Open Database Connectivity) and JDBC (Java Database Connectivity).
Transformation modies the data from the format of the data sources
to the warehouse format. This includes several aspects: cleaning, which
removes errors and inconsistencies in the data and converts it into a
standardized format; integration, which reconciles data from dierent data
sources, both at the schema and at the data level; and aggregation, which
summarizes the data obtained from data sources according to the level of
detail, or granularity, of the data warehouse.

3.4 Data Warehouse Architecture

Internal
sources

Data staging

77

Metadata
OLAP tools

Operational
databases

ETL
process

Enterprise
data
warehouse

OLAP
server

Reporting
tools

Statistical
tools
External
sources

Data marts
Data mining
tools

Data
sources

Back-end
tier

Data warehouse
tier

OLAP
tier

Front-end
tier

Fig. 3.5 Typical data warehouse architecture

Loading feeds the data warehouse with the transformed data. This also
includes refreshing the data warehouse, that is, propagating updates from
the data sources to the data warehouse at a specied frequency in order
to provide up-to-date data for the decision-making process. Depending on
organizational policies, the refresh frequency may vary from monthly to
several times a day or even near to real time.
ETL processes usually require a data staging area, that is, a database in
which the data extracted from the sources undergoes successive modications
to eventually be ready to be loaded into the data warehouse. Such a database
is usually called operational data store.

3.4.2

Data Warehouse Tier

The data warehouse tier in Fig. 3.5 depicts an enterprise data warehouse
and several data marts. As we have explained, while an enterprise data
warehouse is centralized and encompasses an entire organization, a data

78

Data Warehouse Concepts

mart is a specialized data warehouse targeted toward a particular functional

or departmental area in an organization. A data mart can be seen as a small,
local data warehouse. Data in a data mart can be either derived from an
enterprise data warehouse or collected directly from data sources.
Another component of the data warehouse tier is the metadata repository.
Metadata can be dened as data about data. Metadata has been traditionally classied into technical and business metadata. Business metadata
describes the meaning (or semantics) of the data and organizational rules,
policies, and constraints related to the data. On the other hand, technical
metadata describes how data are structured and stored in a computer
system and the applications and processes that manipulate such data.
In the data warehouse context, technical metadata can be of various
natures, describing the data warehouse system, the source systems, and the
ETL process. In particular, the metadata repository may contain information
such as the following:
Metadata describing the structure of the data warehouse and the data
marts, both at the conceptual/logical level (which includes the facts,
dimensions, hierarchies, derived data denitions) and at the physical level
(such as indexes, partitions, and replication). In addition, these metadata
contain security information (user authorization and access control) and
monitoring information (such as usage statistics, error reports, and audit
trails).
Metadata describing the data sources, including their schemas (at the
conceptual, logical, and/or physical levels), and descriptive information
such as ownership, update frequencies, legal limitations, and access
methods.
Metadata describing the ETL process, including data lineage (i.e., tracing
warehouse data back to the source data from which it was derived), data
extraction, cleaning, transformation rules and defaults, data refresh and
purging rules, and algorithms for summarization.

3.4.3

OLAP Tier

The OLAP tier in the architecture of Fig. 3.5 is composed of an OLAP

server, which presents business users with multidimensional data from data
warehouses or data marts.
Most database products provide OLAP extensions and related tools
allowing the construction and querying of cubes, as well as navigation,
analysis, and reporting. However, there is not yet a standardized language
for dening and manipulating data cubes, and the underlying technology
diers between the available systems. In this respect, several languages are
worth mentioning. XMLA (XML for Analysis) aims at providing a common

3.4 Data Warehouse Architecture

79

language for exchanging multidimensional data between client applications

and OLAP servers. Further, MDX (MultiDimensional eXpressions) is a query
language for OLAP databases. As it is supported by a number of OLAP
vendors, MDX became a de facto standard for querying OLAP systems. The
SQL standard has also been extended for providing analytical capabilities;
this extension is referred to as SQL/OLAP. In Chap. 6, we present a detailed
study of both MDX and SQL/OLAP.

3.4.4

Front-End Tier

The front-end tier in Fig. 3.5 contains client tools that allow users to
exploit the contents of the data warehouse. Typical client tools include the
following:
OLAP tools allow interactive exploration and manipulation of the
warehouse data. They facilitate the formulation of complex queries that
may involve large amounts of data. These queries are called ad hoc
queries, since the system has no prior knowledge about them.
Reporting tools enable the production, delivery, and management of
reports, which can be paper-based reports or interactive, web-based
reports. Reports use predened queries, that is, queries asking for
specic information in a specic format that are performed on a regular
basis. Modern reporting techniques include key performance indicators and
dashboards.
Statistical tools are used to analyze and visualize the cube data using
statistical methods.
Data mining tools allow users to analyze data in order to discover
valuable knowledge such as patterns and trends; they also allow predictions
to be made on the basis of current data.
In Chap. 9, we show some of the tools used to exploit the data warehouse,
like data mining tools, key performance indicators, and dashboards.

3.4.5

Variations of the Architecture

Some of the components in Fig. 3.5 can be missing in a real environment.

In some situations, there is only an enterprise data warehouse without data
marts, or alternatively, an enterprise data warehouse does not exist. Building
an enterprise data warehouse is a complex task that is very costly in time
and resources. In contrast, a data mart is typically easier to build than an
enterprise warehouse. However, this advantage may turn into a problem when

80

Data Warehouse Concepts

several data marts that were independently created need to be integrated into
a data warehouse for the entire enterprise.
In some other situations, an OLAP server does not exist and/or the client
tools directly access the data warehouse. This is indicated by the arrow
connecting the data warehouse tier to the front-end tier. This situation
is illustrated in Chap. 6, where the same queries for the Northwind case
study are expressed both in MDX (targeting the OLAP server) and in
SQL (targeting the data warehouse). An extreme situation is where there is
neither a data warehouse nor an OLAP server. This is called a virtual data
warehouse, which denes a set of views over operational databases that are
materialized for ecient access. The arrow connecting the data sources to
the front-end tier depicts this situation. Although a virtual data warehouse
is easy to build, it does not provide a real data warehouse solution, since it
does not contain historical data, does not contain centralized metadata, and
does not have the ability to clean and transform the data. Furthermore, a
virtual data warehouse can severely impact the performance of operational
databases.
Finally, a data staging area may not be needed when the data in the source
systems conforms very closely to the data in the warehouse. This situation
typically arises when there is one data source (or only a few) having highquality data. However, this is rarely the case in real-world situations.

3.5

Data Warehouse Design

Like in operational databases (studied in Sect. 2.1), there are two major
methods for the design of data warehouses and data marts. In the topdown approach, the requirements of users at dierent organizational levels
are merged before the design process starts, and one schema for the entire
data warehouse is built, from which data marts can be obtained. In the
bottom-up approach, a schema is built for each data mart, according to
the requirements of the users of each business area. The data mart schemas
produced are then merged in a global warehouse schema. The choice between
the top-down and the bottom-up approach depends on many factors that will
be studied in Chap. 10 in this book.

Requirements
specification

Conceptual
design

Fig. 3.6 Phases in data warehouse design

Logical design

Physical design

3.6 Business Intelligence Tools

81

There is still no consensus on the phases that should be followed for

data warehouse design. Most of the books in the data warehouse literature
follow a bottom-up, practitioners approach to design based on the relational
model, using the star, snowake, and constellation schemas, which we will
study in detail in Chap. 5. In this book, we follow a dierent, model-based
approach for data warehouse design, which follows the traditional phases for
designing operational databases described in Chap. 2, that is, requirements
specication, conceptual design, logical design, and physical design. These
phases are shown in Fig. 3.6. In Chap. 10, which studies the design method
in detail, we will see that there are important dierences between the design
phases for databases and data warehouses, arising from their dierent nature.
Also note that although, for simplicity, the phases in Fig. 3.6 are depicted
consecutively, actually there are multiple interactions between them. Finally,
we remark that the phases in Fig. 3.6 may be applied to dene either the
schema of the overall schema of the organizational data warehouse or the
schemas of individual data marts.
A distinctive characteristic of the method presented in this book is the
importance given to the requirements specication and conceptual design
phases. For these phases, we can follow two approaches, which we explain
in detail in Chap. 10. In the analysis-driven approach, key users from
dierent organizational levels provide useful input about the analysis needs.
On the other hand, in the source-driven approach, the data warehouse
schema is obtained by analyzing the data source systems. In this approach,
normally, the participation of users is only required to conrm the correctness
of the data structures that are obtained from the source systems or to identify
some facts and measures as a starting point for the design of multidimensional
schemas. Finally, the analysis/source-driven approach is a combination
of the analysis- and source-driven approaches, aimed at matching the users
analysis needs with the information that the source systems can provide. This
is why this approach is also called top-down/bottom-up analysis.

3.6

Business Intelligence Tools

Nowadays, the oer in business intelligence tools is quite large. The major
database providers, such as Microsoft, Oracle, IBM, and Teradata, have their
own suite of business intelligence tools. Other popular tools include SAP,
MicroStrategy, and Targit. In addition to the above commercial tools, there
are also open-source tools, of which Pentaho is the most popular one.
In this book, we have chosen two representative suites of tools for
illustrating the topics presented: Microsofts SQL Server tools and Pentaho
Business Analytics. In this section, we briey describe these tools, while the
bibliographic notes section at the end of this chapter provides references to
other well-known business intelligence tools.

82

3.6.1

Data Warehouse Concepts

Overview of Microsoft SQL Server Tools

Microsoft SQL Server provides an integrated platform for building analytical

applications. It is composed of three main components, described below:
Analysis Services is an OLAP tool that provides analytical and data
mining capabilities. It is used to dene, query, update, and manage
OLAP databases. The MDX (MultiDimensional eXpressions) language is
used to retrieve data. Users may work with OLAP data via client tools
(Excel or other OLAP clients) that interact with Analysis Services server
component. We will study these in Chaps. 5 and 6 when we dene and
query the data cube for the Northwind case study. Further, Analysis
Services provides several data mining algorithms and uses the DMX (Data
Mining eXtensions) language for creating and querying data mining models
and obtaining predictions. We will study these in Chap. 9 when we exploit
the Northwind data warehouse for data analytics.
Integration Services supports ETL processes, which are used for loading
and refreshing data warehouses on a periodic basis. Integration Services
is used to extract data from a variety of data sources; to combine, clean,
and summarize this data; and, nally, to populate a data warehouse with
the resulting data. We will explain in detail Integration Services when we
describe the ETL for the Northwind case study in Chap. 8.
Reporting Services is used to dene, generate, store, and manage
reports. Reports can be built from various types of data sources, including
data warehouses and OLAP cubes. Reports can be personalized and
delivered in a variety of formats. Users can view reports with a variety
of clients, such as web browsers or other reporting clients. Clients
access reports via Reporting Services server component. We will explain
Reporting Services when we build dashboards for the Northwind case study
in Chap. 9.
SQL Server provides two tools for developing and managing these components. SQL Server Data Tools (SSDT) is a development platform
integrated with Microsoft Visual Studio. SQL Server Data Tools supports
Analysis Services, Reporting Services, and Integration Services projects.
On the other hand, SQL Server Management Studio (SSMS) provides
integrated management of all SQL Server components.
The underlying model across these tools is called the Business Intelligence Semantic Model (BISM). This model comes in two modes, the
multidimensional and tabular modes, where, as their name suggest, the dierence among them stems from their underlying paradigm (multidimensional or
relational). From the data model perspective, the multidimensional mode has
powerful capabilities for building advanced business intelligence applications
and supports large data volumes. On the other hand, the tabular mode is
simpler to understand and quicker to build than the multidimensional data

3.6 Business Intelligence Tools

83

mode. Further, the data volumes supported by the tabular mode are smaller
than those of the multidimensional mode in Analysis Services. From the query
language perspective, each of these modes has an associated query language,
MDX and DAX (Data Analysis Expressions), respectively. Finally, from the
data access perspective, the multidimensional mode supports data access in
MOLAP (multidimensional OLAP), ROLAP (relational OLAP), or HOLAP
(hybrid OLAP), which will be described in Chap. 5. On the other hand,
the tabular mode accesses data through xVelocity, an in-memory, columnoriented database engine with compression capabilities. We will cover such
databases in Chap. 13. The tabular mode also allows the data to be retrieved
directly from relational data sources.
In this book, we cover only the multidimensional mode of BISM as well as
the MDX language.

3.6.2

Overview of Pentaho Business Analytics

Pentaho Business Analytics is a suite of business intelligence products.

It comes in two versions: an enterprise edition that is commercial and
a community edition that is open source. The main components are the
following:
Pentaho Business Analytics Platform serves as the connection
point for all other components. It enables a unied, end-to-end solution
from data integration to visualization and consumption of data. It also
includes a set of tools for development, deployment, and management of
applications.
Pentaho Analysis Services, also known as Mondrian, is a relational
OLAP server. It supports the MDX (multidimensional expressions) query
language and the XML for Analysis and olap4j interface specications. It
reads from SQL and other data sources and aggregates data in a memory
cache.
Pentaho Data Mining uses the Waikato Environment for Knowledge
Analysis (Weka) to search data for patterns. Weka consists of machine
learning algorithms for a broad set of data mining tasks. It contains
functions for data processing, regression analysis, classication methods,
cluster analysis, and visualization.
Pentaho Data Integration, also known as Kettle, consists of a data
integration (ETL) engine and GUI (graphical user interface) applications
that allow users to dene data integration jobs and transformations. It
supports deployment on single node computers as well as on a cloud or a
cluster.
Pentaho Report Designer is a visual report writer that can query and
use data from many sources. It consists of a core reporting engine, capable
of generating reports in several formats based on an XML denition le.

84

Data Warehouse Concepts

In addition, several design tools are provided, which are described next:
Pentaho Schema Workbench provides a graphical interface for designing OLAP cubes for Mondrian. The schema created is stored as an XML
le on disk.
Pentaho Aggregation Designer operates on Mondrian XML schema
les and the database with the underlying tables described by the schema
to generate precalculated, aggregated answers to speed up analysis work
and MDX queries executed against Mondrian.
Pentaho Metadata Editor is a tool that simplies the creation of
reports and allows users to build metadata domains and relational data
models. It acts as an abstraction layer from the underlying data sources.

3.7

Summary

In this chapter, we introduced the multidimensional model, which is the

basis for data warehouse systems. We dened the notion of online analytical
processing (OLAP) systems as opposite to online transaction processing
(OLTP) systems. We then studied the data cube concept and its components:
dimensions, hierarchies, and measures. In particular, we presented several
classications of measures and dened the notions of measure aggregation
and summarizability. Then, we dened a set of OLAP operations, like rollup and drill-down, that are used to interactively manipulate a data cube.
We then described data warehouse systems and highlighted their dierences
with respect to traditional database systems. As data warehouse systems
include many dierent components, we discussed the basic architecture of
data warehouse systems and several variants of it that may be considered.
We nished this chapter by giving an overview of two representative sets of
tools: Microsoft SQL Server tools and Pentaho Business Analytics.

3.8

Bibliographic Notes

Basic data warehouse concepts can be found in the classic books by

Kimball [103] and by Inmon [90, 91]. In particular, the denition of data
warehouses we gave in Sect. 3.3 is from Inmon.
The notion of hypercube underlying the multidimensional model was
studied in [72], where the ROLLUP and the CUBE operations were dened
for SQL. Hierarchies in OLAP are studied, among other works, in [22, 123].
The notion of summarizability of measures was dened in [115] and has been
studied, for example, in [8486]. Other classication of measures are given in

3.9 Review Questions

85

[72, 103]. More details on these concepts are given in Chap. 5, where we also
give further references.
There is not yet a standard denition of the OLAP operations, in a similar
way as the relational algebra operations are dened for the relational algebra.
Many dierent algebras for OLAP have been proposed in the literature,
each one dening dierent sets of operations. A comparison of these OLAP
algebras is given in [181], where the authors advocate the need for a reference
algebra for OLAP. The denition of the operations we presented in this
chapter was inspired from [32].
There are many books that describe the various business intelligence tools.
We next give some references for commercial and open-source tools. For SQL
Server, the series of books devoted to Analysis Services [79], Integration Services [105], and Reporting Services [209] cover extensively these components.
The business intelligence tools from Oracle are covered in [175, 218], while
those of IBM are covered in [147, 225]. SAP BusinessObjects is presented in
[81, 83], while MicroStrategy is covered in [50, 139]. For Pentaho, the book
[18] gives an overall description of the various components of the Pentaho
Business Intelligence Suite, while Mondrian is covered in the book [10], Kettle
in [26, 179], Reporting in [57], and Weka in [228]. The book [157] is devoted
to Big Data Analytics using Pentaho. On the academic side, a survey of
open-source tools for business intelligence is given in [199].

3.9

Review Questions

3.1 What is the meaning of the acronyms OLAP and OLTP?

3.2 Using an example of an application domain that you are familiar with,
describe the various components of the multidimensional model, that
is, facts, measures, dimensions, and hierarchies.
3.3 Why are hierarchies important in data warehouses? Give examples of
various hierarchies.
3.4 Discuss the role of measure aggregation in a data warehouse. How can
measures be characterized?
3.5 Give an example of a problem that may occur when summarizability is
not veried in a data warehouse.
3.6 Describe the various OLAP operations using the example you dened
in Question 3.2.
3.7 What is an operational database system? What is a data warehouse
system? Explain several aspects that dierentiate these systems.
3.8 Give some essential characteristics of a data warehouse. How do a data
warehouse and a data mart dier? Describe two approaches for building
a data warehouse and its associated data marts.

86

Data Warehouse Concepts

3.9 Describe the various components of a typical data warehouse architecture. Identify variants of this architecture and specify in what situations
they are used.
3.10 Briey describe the multidimensional model implemented in Analysis
Services.

3.10

Exercises

3.1 A data warehouse of a telephone provider consists of ve dimensions:

caller customer, callee customer, time, call type, and call program and
three measures: number of calls, duration, and amount.
Dene the OLAP operations to be performed in order to answer the
following queries. Propose the dimension hierarchies when needed.
(a) Total amount collected by each call program in 2012.
(b) Total duration of calls made by customers from Brussels in 2012.
(c) Total number of weekend calls made by customers from Brussels to
customers in Antwerp in 2012.
(d) Total duration of international calls started by customers in Belgium
in 2012.
(e) Total amount collected from customers in Brussels who are enrolled
in the corporate program in 2012.
3.2 A data warehouse of a train company contains information about train
segments. It consists of six dimensions, namely, departure station, arrival
station, trip, train, arrival time, and departure time, and three measures,
namely, number of passengers, duration, and number of kilometers.
Dene the OLAP operations to be performed in order to answer the
following queries. Propose the dimension hierarchies when needed.
(a) Total number of kilometers made by Alstom trains during 2012
departing from French or Belgian stations.
(b) Total duration of international trips during 2012, that is, trips
departing from a station located in a country and arriving at a station
located in another country.
(c) Total number of trips that departed from or arrived at Paris during
July 2012.
(d) Average duration of train segments in Belgium in 2012.
(e) For each trip, average number of passengers per segment, which
means take all the segments of each trip and average the number
of passengers.

3.10 Exercises

87

3.3 Consider the data warehouse of a university that contains information

about teaching and research activities. On the one hand, the information
about teaching activities is related to dimensions department, professor,
course, and time, the latter at a granularity of academic semester.
Measures for teaching activities are number of hours and number of
credits. On the other hand, the information about research activities is
related to dimensions professor, funding agency, project, and time, the
latter twice for the start date and the end date, both at a granularity of
day. In this case, professors are related to the department to which they
are aliated. Measures for research activities are the number of person
months and amount.
Dene the OLAP operations to be performed in order to answer the
following queries. For this, propose the necessary dimension hierarchies.
(a) By department the total number of teaching hours
academic year 20122013.
(b) By department the total amount of research projects
calendar year 2012.
(c) By department the total number of professors involved
projects during the calendar year 2012.
(d) By professor the total number of courses delivered
academic year 20122013.
(e) By department and funding agency the total number
started in 2012.

during the
during the
in research
during the
of projects