Scribd
Upload
675 views

SCD Docs

This document discusses different types of slowly changing dimensions (SCDs) in data warehousing and how to implement each type in IBM Datastage: - SCD Type 1 overwrites old data with new data and does not track history. SCD Type 2 stores all history by adding new records. SCD Type 3 tracks the immediate previous value in an "old" column. SCD Type 4 stores historical changes in a separate table. - The examples show jobs that lookup new data, compare to old, and either update existing records (Type 1) or insert new records while flagging old as inactive (Type 2). Type 3 would update the "old" column and Type 4 would insert to a separate historical

Uploaded by

smadanan9063
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
675 views

SCD Docs

This document discusses different types of slowly changing dimensions (SCDs) in data warehousing and how to implement each type in IBM Datastage: - SCD Type 1 overwrites old data with new data and does not track history. SCD Type 2 stores all history by adding new records. SCD Type 3 tracks the immediate previous value in an "old" column. SCD Type 4 stores historical changes in a separate table. - The examples show jobs that lookup new data, compare to old, and either update existing records (Type 1) or insert new records while flagging old as inactive (Type 2). Type 3 would update the "old" column and Type 4 would insert to a separate historical

Uploaded by

smadanan9063
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 13

SCD Type 1

Type 1 Slowly Changing Dimension data warehouse architecture applies when no history is kept in the database. The new, changed
data simply overwrites old entries. This approach is used quite often with data which change over the time and it is caused by
correcting data quality errors (misspells, data consolidations, trimming spaces, language specific characters).
Type 1 SCD is easy to maintain and used mainly when losing the ability to track the old history is not an issue.

SCD 1 implementation in Datastage


The job described and depicted below shows how to implement SCD Type 1 in Datastage. It is one of many possible designs which
can implement this dimension. The example is based on the customers load into a data warehouse
The most important facts and stages of the CUST_SCD2 job processing:
 There is a hashed file (Hash_NewCust) which handles a lookup of the new data coming from the text file.
 A T001_Lookups transformer does a lookup into a hashed file and maps new and old values to separate columns.
A T002 transformer updates old values with new ones without concerning about the overwritten data.
SCD1 Transformer update old entries

The database is updated in a target ODBC stage (with the 'update existing rows' update action)
SCD Type 2
Slowly changing dimension Type 2 is a model where the whole history is stored in the database. An additional dimension record is
created and the segmenting between the old record values and the new (current) value is easy to extract and the history is clear.
The fields 'effective date' and 'current indicator' are very often used in that dimension and the fact table usually stores dimension key
and version number.

SCD 2 implementation in Datastage


The job described and depicted below shows how to implement SCD Type 2 in Datastage. It is one of many possible designs which
can implement this dimension.
For this example, we will use a table with customers data (it's name is D_CUSTOMER_SCD2) which has the following structure and
data:

CUST_ CUST_ CUST_ CUST_ REC_ REC_ REC_


GROUP_I TYPE_I COUNTRY_I VERSIO CURRENT_IN
CUST_ID NAME D D D N EFFDT D
DRBOUA Dream 10/1/200
7 Basket EL S PL 1 6 Y
ETL tools 9/29/200
ETIMAA5 info BI C FI 1 6 Y
FAMMFA 9/27/200
0 Fajatso FD S CD 1 6 Y
First 9/25/200
FICILA0 Pactonic FD C IT 1 6 Y
9/23/200
FRDXXA2 Frasir EL C SK 1 6 Y
GAMOPA Ganpa 9/21/200
9 LTD. FD C US 1 6 Y
GGMOPA GG 9/19/200
9 electronics EL S RU 1 6 Y
GLMFIA6 Glasithkli FD S PL 1 9/17/200 Y
ni 6
Globitelec 9/15/200
GLMPEA9 o TC S FI 1 6 Y
GONDWA Goli 9/13/200
5 Airlines BN S GB 1 6 Y
The most important facts and stages of the CUST_SCD2 job processing:
 The dimension table with customers is refreshed daily and one of the data sources is a text file. For the purpose of this example the
CUST_ID=ETIMAA5 differs from the one stored in the database and it is the only record with changed data. It has the following
structure and data:
SCD 2 - Customers file extract:

 There is a hashed file (Hash_NewCust) which handles a lookup of the new data coming from the text file.
 A T001_Lookups transformer does a lookup into a hashed file and maps new and old values to separate columns.
SCD 2 lookup transformer
A T002_Check_Discrepacies_exist transformer compares old and new values of records and passes through only records that differ.
SCD 2 check discrepancies transformer
A T003 transformer handles the UPDATE and INSERT actions of a record. The old record is updated with current indictator flag set
to no and the new record is inserted with current indictator flag set to yes, increased record version by 1 and the current date.
SCD 2 insert-update record transformer
 ODBC Update stage (O_DW_Customers_SCD2_Upd) - update action 'Update existing rows only' and the selected key columns
are CUST_ID and REC_VERSION so they will appear in the constructed where part of an SQL statement.
 ODBC Insert stage (O_DW_Customers_SCD2_Ins) - insert action 'insert rows without clearing' and the key column is CUST_ID.
CUST_ CUST_ CUST_ CUST_ REC_ REC_ REC_
CUST_ID
NAME GROUP_ID TYPE_ID COUNTRY_ID VERSION EFFDT CURRENT_IND
DRBOUA7 Dream Basket EL S PL 1 10/1/2006 Y
ETIMAA5 ETL tools info BI C FI 1 9/29/2006 N
FAMMFA0 Fajatso FD S CD 1 9/27/2006 Y
FICILA0 First Pactonic FD C IT 1 9/25/2006 Y
FRDXXA2 Frasir EL C SK 1 9/23/2006 Y
GAMOPA9 Ganpa LTD. FD C US 1 9/21/2006 Y
GG
GGMOPA9 electronics EL S RU 1 9/19/2006 Y
GLMFIA6 Glasithklini FD S PL 1 9/17/2006 Y
GLMPEA9 Globiteleco TC S FI 1 9/15/2006 Y
GONDWA5 Goli Airlines BN S GB 1 9/13/2006 Y
ETIMAA5 ETL-Tools.info BI C ES 2 12/2/2006 Y

SCD Type 3
In the Type 3 Slowly Changing Dimension only the information about a previous value of a dimension is written into the database.
An 'old 'or 'previous' column is created which stores the immediate previous attribute. In Type 3 SCD users are able to describe history
immediately and can report both forward and backward from the change.
However, that model can't track all historical changes, such as when a dimension changes twice or more. It would require creating
next columns to store historical data and could make the whole data warehouse schema very complex.

To implement SCD Type 3 in Datastage use the same processing as in the SCD-2 example, only changing the destination stages to
update the old value with a new one and update the previous value field.

SCD Type 4
The Type 4 SCD idea is to store all historical changes in a separate historical data table for each of the dimensions.

To implement SCD Type 4 in Datastage use the same processing as in the SCD-2 example, only changing the destination stages to
insert an old value into the destionation stage connected to the historical data table (D_CUSTOMER_HIST for example) and update
the old value with a new one.

You might also like