Customizing Incremental ETL for Operational Reporting

for HP Project and Portfolio Management Center version 9.12 Content Pack 1

About this Document ...................................................................................................................... 2

Operational Reporting ETL Architecture ......................................................................................... 2
ETL Batch Job ............................................................................................................................ 2
PPM Center Tables Published for Change Data Capture ................................................................... 3
COMMON ETL Module .......................................................................................................... 6
TM ETL Module ...................................................................................................................... 7
RM ETL Module ...................................................................................................................... 8
FM ETL Module ...................................................................................................................... 8
PM ETL Module ...................................................................................................................... 9
Guidelines for Custom Code ........................................................................................................... 11
CDC Operations ....................................................................................................................... 11
Adding PPM Center Tables for Incremental Change............................................................................ 12
Creating a New ETL Module....................................................................................................... 12
About this Document
This paper provides information about how to customize the incremental ETL process for the
Operational Reporting solution for PPM Center. It is written for PPM Center administrators,
configurators, and DBAs. Readers are assumed to be knowledgeable about PPM Center and SAP
BusinessObjects Enterprise.
The Operational Reporting Administrator’s Guide provides information about the incremental ETL job
(PPM_ETL_BATCH_JOB) that covers the day-to-day updates for PPM Center data tables. This paper
describes how you can bring data that are not part of the default ETL process into the Operational
Reporting system from the PPM Center database.

Operational Reporting ETL Architecture

The incremental ETL tool (extract, transform and load for Operational Reporting in PPM Center is
designed using Oracle’s Change Data Capture (CDC) and job infrastructure. It uses synchronous
CDC. Tables are published in the PPM Center database and subscribed in multiple modules in PPM
Center databases.
The five PPM Center modules that participate in the incremental ETL process for Operational Reporting
are designed to handle specific tables in the PPM Center database based on functionality. For
example, the TM module loads and converts data for tables related to HP Time Management.
The five functional modules are:
• CM (Common)
• TM (Time Management)
• RM (Resource Management)
• FM (Financial Management)
• PM (Project Management)

ETL Batch Job

RPT_ETL_BATCH_JOB is a scheduled job that calls RPT_ETL_BATCH_JOB_CHAIN, which is illustrated
in Figure 1. This ETL job chain has seven blocks, with one block for each of the five ETL modules, one
for the ETL_START_PROGRAM, and one for ETL_END_PROGRAM.
Incremental ETL for each functional module in PPM Center is called from each block of the ETL job
chain. The Oracle job scheduler schedules the ETL job chain, which is illustrated in Figure 1.
Each block calls a function from the RPT_ETL_JOB_UTIL package. In the event of a failure at any point
along the chain, the function throws an exception and the process jumps to the ETL_END block,
instead of the next block in the job chain.

2
Figure 1. ETL job chain

PPM Center Tables Published for Change Data Capture

The framework uses the Oracle publish and subscribe model to publish a list of PPM Center tables for
change. Each module subscribes to a list of tables. (The subscription is created during installation or
upgrade.) To load changes in the Operational Reporting database, each module calls the Oracle API
using the subscription name, and then calls a module-specific API to process changes within that
specific window or time frame. For example, the CM (common) module calls the
RPT_INCREMENTAL_ETL.DO_COMMON_INCREMENTAL_ETL function. The
RPT_ETL_JOB_UTIL.RUN_ETL function maintains the CDC window and also the transition
COMMIT or ROLLBACK.

3
Figure 2.

Each module executes generic code to copy the change capture data (CDC) from the PPM Center
database CDC table to the Operational Reporting database CDC table using a database link. During
Operational Reporting deployment or upgrade, this CDC table is created in the Operational
Reporting database. It (?) executes another generic method to merge those changes in the PPM
Center tables into the Operational Reporting database. After the changes are copied and merged,
PPM Center tables in the Operational Reporting schema are the same as the tables in the PPM Center
schema and each module executes module-specific fact calculation logic. (See Figure 3 below)

Figure 3.

4
The following PPM Center tables are published for change data capture (CDC):
• FM_APPROVED_BUDGETS
• FM_FINANCIAL_LINE_CELLS
• FM_FINANCIAL_SUMMARY
• FM_FORECAST_ACTUALS
• FM_FORECAST_ACTUAL_LINES
• FM_USER_DATA
• ITG_ACTIVITIES_NLS
• ITG_AUDIT_EVENTS
• ITG_SETTINGS_ATTRIBUTES
• ITG_SETTINGS_CONTAINERS
• ITG_SETTINGS_CONTAINER_CHLDRN
• ITG_SETTINGS_CONT_ATTRS
• KCRT_CONTACTS
• KCRT_FG_MASTER_PROJ_REF
• KCRT_FG_PFM_ASSET
• KCRT_FG_PFM_PROJECT
• KCRT_FG_PFM_PROPOSAL
• KCRT_FG_PROJ_ISSUE
• KCRT_FG_PROJ_RISK
• KCRT_FG_PROJ_SCOPE_CHANGE
• KCRT_REQUESTS
• KCRT_REQUEST_DETAILS
• KCRT_REQUEST_TYPES_NLS
• KCRT_REQ_HEADER_DETAILS
• KCRT_STATUSES_NLS
• KCST_CURRENCIES_NLS
• KDLV_PACKAGES
• KDRV_CALENDARS
• KDRV_CALENDAR_EXCEPTIONS
• KDRV_CALENDAR_SETUPS
• KDRV_WORKING_DAYS
• KNTA_LOOKUPS_NLS
• KNTA_REGIONS_NLS
• KNTA_SECURITY_GROUPS_NLS
• KNTA_USERS
• KPMO_BUSINESS_OBJECTIVES_NLS
• KRSC_ORG_UNITS_NLS
• KRSC_ORG_UNIT_MEMBERS
• KTMG_CHARGE_CODES_NLS
• KTMG_PERIODS
• KTMG_PERIOD_TYPES_NLS
• KTMG_POLICIES
• KWFL_WORKFLOWS_NLS
• KWFL_WORKFLOW_INSTANCES

5
• KWFL_WORKFLOW_INSTANCE_STEPS
• KWFL_WORKFLOW_STEPS_NLS
• PFM_LIFECYCLE_PARENT_ENTITY
• PGM_PROGRAMS
• PGM_PROGRAM_CONTENT
• PM_PROGRAM_BUSINESS_OBJECTIVES
• PM_PROJECTSPM_PROJECT_ROLLUP
• PM_PROJECT_TYPES
• PM_WORK_PLANS
• PPM_FISCAL_PERIODS_NLS
• RSC_POSITIONS
• RSC_POSITION_FORECAST
• RSC_RESOURCES
• RSC_RESOURCE_ASSIGNMENTS
• RSC_RESOURCE_POOLS
• RSC_RESOURCE_POOL_MANAGERS
• RSC_ROLES_NLS
• RSC_RP_DISTRIBUTION_ENTRIES
• RSC_STAFFING_PROFILES
• RSC_STAFF_PROF_ALLOCATION
• TM_ACTUALS
• TM_ACTUALS_EFFORT
• TM_LINE_CHARGE_CODES
• TM_TIME_SHEETS
• TM_TIME_SHEET_LINES
• WP_MILESTONES
• WP_TASKS
• WP_TASK_ACTUALS
• WP_TASK_COSTS
• WP_TASK_INFO
• WP_TASK_SCHEDULE
• WP_USER_DATA
• WP_WORK_UNITS
The different modules subscribe to different tables to get the changes for their areas of functionality.

COMMON ETL Module

The CM module loads all common tables and transforms requests, fiscal periods, and custom
parameter data to fact. The CM module subscribes to the following tables. (This list of tables is
defined in the RPT_PPM_OBJECTS.COMMON_CDC_TABLES object.)
• KCRT_CONTACTS
• KCRT_FG_PFM_ASSET
• KCRT_FG_PFM_PROJECT
• KCRT_FG_PFM_PROPOSAL
• KCRT_REQUESTS
• KCRT_REQUEST_TYPES_NLS
• KCRT_REQ_HEADER_DETAILS

6
• KCRT_REQUEST_DETAILS
• KCRT_STATUSES_NLS
• KCST_CURRENCIES_NLS
• KDRV_CALENDAR_SETUPS
• KDRV_CALENDARS
• KDRV_WORKING_DAYS
• KNTA_LOOKUPS_NLS
• KNTA_REGIONS_NLS
• KNTA_USERS
• KPMO_BUSINESS_OBJECTIVES_NLS
• KRSC_ORG_UNIT_MEMBERS
• KRSC_ORG_UNITS_NLS
• FM_FINANCIAL_SUMMARY
• RSC_STAFFING_PROFILES
• KWFL_WORKFLOW_INSTANCES
• KWFL_WORKFLOW_INSTANCE_STEPS
• KWFL_WORKFLOW_STEPS_NLS
• KWFL_WORKFLOWS_NLS
• PFM_LIFECYCLE_PARENT_ENTITY
• PGM_PROGRAM_CONTENT
• PGM_PROGRAMS
• PM_PROGRAM_BUSINESS_OBJECTIVES
• PM_PROJECTS
• PM_PROJECT_TYPES
• PM_PROJECT_ROLLUP
• PM_WORK_PLANS
• WP_TASKS
• WP_TASK_ACTUALS
• PPM_FISCAL_PERIODS_NLS
• RSC_RESOURCES
• RSC_RESOURCE_POOL_MANAGERS
• RSC_RESOURCE_POOLS
• RSC_ROLES_NLS
• RSC_RP_DISTRIBUTION_ENTRIES
• TM_TIME_SHEETS
• TM_TIME_SHEET_LINES
Fact transformation logic for the Common ETL is defined in the following packages and is called from
the RPT_INCREMENTAL_ETL.DO_COMMON_INCREMENTAL_ETL function:
• RPT_POPULATE_DIMENSION.RPT_POPULATE_DIM_FISC_PERIODS();
• rpt_request_update_dimension.do_req_custom_params_incrm_etl('H');
• rpt_request_update_dimension.do_req_custom_params_incrm_etl('D');
• rpt_request_update_dimension.do_request_incremental_etl(start_year_boundry);

TM ETL Module
The TM module works with tables for HP Time Management. The TM module subscribes to the
following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.TM_CDC_TABLES object.)

7
• ITG_AUDIT_EVENTS
• KTMG_PERIODS
• KTMG_POLICIES
• TM_ACTUALS
• TM_ACTUALS_EFFORT
• TM_TIME_SHEETS
• TM_TIME_SHEET_LINES
• WP_TASKS
• WP_TASK_INFO
• TM_LINE_CHARGE_CODES
• KTMG_CHARGE_CODES_NLS
• KTMG_PERIOD_TYPES_NLS
• KDLV_PACKAGES
• ITG_ACTIVITIES_NLS
Fact transformation logic for HP Time Management ETL is defined in the following packages/functions
and is called from the RPT_INCREMENTAL_ETL.DO_TM_INCREMENTAL_ETL function:
• rpt_tm_refresh.rpt_tm_refresh_all
• RPT_RM_UPDATE_EFFORT_FACT.DO_INCREMENTAL_ACTUALS;

RM ETL Module
The RM module works with tables for HP Resource Management. The RM module subscribes to the
following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.RM_CDC_TABLES object.
• RSC_POSITIONS
• RSC_RESOURCE_ASSIGNMENTS
• RSC_STAFF_PROF_ALLOCATION
• RSC_POSITION_FORECAST
• RSC_RESOURCE_POOLS
• RSC_STAFFING_PROFILES
• KNTA_REGIONS_NLS
• KDRV_WORKING_DAYS
• KDRV_CALENDAR_SETUPS
• RSC_RP_DISTRIBUTION_ENTRIES
• RSC_RESOURCES
• KNTA_USERS
• KDRV_CALENDAR_EXCEPTIONS
• KNTA_SECURITY_GROUPS_NLS
Fact transformation logic for the HP Resource Management ETL is defined in the following packages/
functions and is called from the RPT_INCREMENTAL_ETL.DO_RM_INCREMENTAL_ETL function:
• rpt_rm_update_facts.do_incremental_capacity(start_year_boundry, end_year_boundry);
• rpt_rm_update_facts.do_incremental_demand(start_year_boundry, end_year_boundry);

FM ETL Module
The FM module works with tables for HP Financial Management. The FM module subscribes to the
following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.FM_CDC_TABLES object.)
• PM_PROGRAM_BUSINESS_OBJECTIVES
• PM_PROJECTS

8
• KPMO_BUSINESS_OBJECTIVES_NLS
• PFM_LIFECYCLE_PARENT_ENTITY
• KCRT_FG_PFM_PROPOSAL
• KCRT_FG_PFM_PROJECT
• KCRT_FG_PFM_ASSET
• KCST_CURRENCIES_NLS
• FM_FORECAST_ACTUAL_LINES
• FM_FINANCIAL_LINE_CELLS
• FM_APPROVED_BUDGETS
• FM_USER_DATA
• FM_FORECAST_ACTUALS
• FM_FINANCIAL_SUMMARY
• PGM_PROGRAM_CONTENT
• PGM_PROGRAMS
• KRSC_ORG_UNITS_NLS
• ITG_SETTINGS_ATTRIBUTES
• ITG_SETTINGS_CONT_ATTRS
• ITG_SETTINGS_CONTAINER_CHLDRN
• ITG_SETTINGS_CONTAINERS
Fact transformation logic for the HP Financial Management ETL is defined in the
frpt_fm_update_facts.do_fm_full_etl; function, and is called from the
RPT_INCREMENTAL_ETL.DO_FM_INCREMENTAL_ETL function.

PM ETL Module
The PM module works with tables for HP Project Management. The PM module subscribes to the
following tables. (This list is of tables is defined in the RPT_PPM_OBJECTS.PM_CDC_TABLES object.)
• KCRT_FG_PFM_PROJECT
• KCRT_FG_PROJ_ISSUE
• KCRT_FG_PROJ_RISK
• KCRT_FG_PROJ_SCOPE_CHANGE
• KCRT_FG_MASTER_PROJ_REF
• WP_TASK_SCHEDULE
• WP_TASK_ACTUALS
• WP_WORK_UNITS
• WP_TASK_COSTS
• WP_TASK_INFO
• WP_TASKS
• RSC_RESOURCES
• KNTA_USERS
• PM_WORK_PLANS
• PM_PROJECTS
• WP_MILESTONES
• WP_USER_DATA
• KNTA_LOOKUPS_NLS
• KCRT_REQ_HEADER_DETAILS
• KCRT_REQUEST_DETAILS

9
Fact transformation logic for the HP Project Management ETL is defined in following packages/
functions, and is called from the RPT_INCREMENTAL_ETL.DO_PM_INCREMENTAL_ETL function:
• RPT_PM_UPDATE_FACTS.update_task_dim;
• RPT_PM_UPDATE_FACTS.RPT_PM_UPDATE_TASK_COST_FCT;
• RPT_PM_UPDATE_FACTS.UPDATE_ASSIGNMENT_FACTS;
• RPT_PM_UPDATE_FACTS.update_aggr_effort_fact;
• RPT_PM_UPDATE_FACTS.UPDATE_PROJECT_MGR;

Note:
For a complete list of data tables and views created in PPM Center for the
Operational Reporting schema, see the Data Model Guide.

10
Guidelines for Custom Code
You can add custom code to the functions called during ETL. You can leverage custom code using the
CDC infrastructure, logging mechanisms, scheduled job, and so on.
The custom code that you add to functions must comply with the following restrictions:
• Place all custom code into a separate package. Custom code must not update/modify any existing
fact transformation logic.
• Custom code must select the right PPM Center module. For example, if the code uses tables
included in the common list, then it would be called from the common module.
• Custom code must not call any commit or rollback. Transaction is managed in the upper layer.
• Custom code must throw an exception for any failure. The upper layer can then break the job
chain execution.
• Custom code must not use DBMS_OUTPUT.PUT_LINE for logging, but instead use the following
rpt_event_util functions:
• Use rpt_event_util.LOG_BEGIN_FUNC_EVENT to mark the start of the function.
• Use rpt_event_util.LOG_INFO_EVENT to log debug information.
• Use rpt_event_util.LOG_END_FUNC_EVENT to mark the end of the function.
• Use rpt_event_util.LOG_ERROR_EVENT to log error information.
• Custom code can include events in the CDC table in its logic.
A CDC table exists for each table in a module. For example, for the KNTA_USERS table, the CM
module contains a KNTA_USERS_CDC_CM_W table for CDC changes. Similarly, the RM module
contains a KNTA_USERS_CDC_RM_W table exists for the RM module.

CDC Operations
The following operation types occur in CDC:
• I (insert)
• UU (before update)
• UN (after update)
• D (delete)
For any update in PPM Center, CDC adds UU(with old values) and UN (with new values). For any
data created in PPM Center, CDC may add insert and multiple update events. However, in the
Operational Reporting schema, the infrastructure only copies the last event, and if last event happens
to be an update, then it brings in data of the UN operation type. For the RM module, CDC copies
data of the UU and UN operation types. These CDC data are deleted before the next copy and
merge are performed during next incremental ETL run.

11
Adding PPM Center Tables for Incremental Change
Custom code can use additional PPM Center tables for incremental change. There is no
straightforward way to implement it, but it can be done in different ways. The approach described
here requires minimal change to existing code.
Suppose, that custom code must bring in changes for the following portfolio hierarchy tables (for HP
Portfolio Management):
• KPFM_PORTFOLIO_LIFE_CYCLE
• PFM_PORTFOLIOS
• PFM_PORTFOLIO_CONTENTS
• PFM_PORTFOLIO_MANAGERS
This requires a new ETL module.

Creating a New ETL Module

In this approach, you build custom code using the API that the framework provides. Because this
involves minimal interaction with the existing code base, the risk of overriding changes during an
upgrade is very small.
To create a new ETL module with the example codes provided:
1 Unzip the example codes package and, and open the rpt_ex_etl_objects.pls file in a text
editor.
2 Update the following parameters:
• ETL_OBJECTS: Add one or more PPM Center data table names to participate in the ETL for the
new module.
• ETL_CHANGE_SET_NAME: Add a change set name for Oracle Change Data Capture. All
ETL_OBJECTS tables are to be published under this change set.
• ETL_SUBS_NAME: Add a subscription name for Oracle Change Data Capture. Changes in
published tables can be used to this subscription.
• ETL_VIEW_PREFIX: Add a two-letter preview name for the Oracle Change Data Capture
subscription. This is used to uniquely identify views in the subscription.
3 If any table name that you specified for the ETL_OBJECTS parameter is more than 15 characters
long, open the latest rpt_constants.plb output file from your reporting schema, and then
update the two functions below (see examples in rpt_constants.plb):
• GET_SHORTNAME: This function must return the shortened form (fewer than 15 characters) of the
table name.
• GET_FULLNAME: This function must return the full form of the table name.
4 If any of the tables that you specified for the ETL_OBJECTS parameter are missing a primary key
or have composite primary keys, you must open the latest rpt_sync_objects.plb file for your
reporting schema and update the following function (see examples in rpt_sync_objects.plb):
• GET_PRIMARY_KEYS(table_name IN VARCHAR2): This function must return the primary key
for the table.
5 Open the rpt_fct_ex_table.sql file and update the fact table structure. The
rpt_fct_ex_table.sql file creates the RPT_FCT_EXTEND table. (You can either reuse the
RPT_FCT_EXTEND table, or create a new fact table to use.)
6 Open rpt_ex_populate_facts.pls file and update logic for initial load data in
populate_facts function. The populate_facts function does not contain any code, you may
need to add code to populate the fact table.

12
7 Open rpt_ex_update_facts.pls file and update logic for incremental data changes in
update_facts function. The update_facts function does not contain any code, you may need to
add code to update fact tale with delta changes.
To apply the above changes in PPM Center and Operational Reporting databases, execute the scripts
described in the following table.

Script Description

update_ppm_schema.sql Top-level script. Run this script in PPM database using schema username
and password. It requires two parameters:
• PPM Schema name, and
• Index table space name for PPM schema.
This script publishes new tables, creates subscription, and index for newly
created CDC tables.

update_reporting_schema.sql Top-level script. To apply changes in the Operational Reporting database

and load initial data, run this script in the Operational Reporting
database using the schema username and password. The script requires
the following two parameters:
• PPM Center schema name
• Data tablespace name for the Operational Reporting database
schema

run_ex_inc_etl.sql Run this script to bring incremental changes from PPM Center to the
Operational Reporting database. The script copies and merges changes,
and then executes code to calculate fact.

13
© 2012 Hewlett-Packard Development Company, L.P. The information contained herein is
subject to change without notice. The only warranties for HP products and services are set forth
in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical
or editorial errors or omissions contained herein.
Business Objects and the Business Objects logo, BusinessObjects, Web Intelligence, and other
Business Objects products and services mentioned herein as well as their respective logos are
trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an
SAP company.
Adobe is a trademark of Adobe Systems Incorporated.
Oracle and Javaô are registered trademarks of Oracle Corporation and/or its affiliates.

March 2012