No Silver Bullets : OBIEE Performance

in the Real World

Robin Moffatt, Principal Consultant
May 2014
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

The Requirement

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

The Problem
Lots of moving parts and complexity

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

Overall solution crosses disciplines and job roles

OBIEE developer
Database developer
DBA
Server Admin
Network
SAN
LDAP
etc

E : info@[Link]
W : [Link]

The Additional Problem

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

The Only Answer

Evidence-based DESIGN and DIAGNOSTICS

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Performance Beyond the Best Practices

Tear down the reliance on

Best Practice, but with a
viable, better, alternative
instead.

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Two-pronged approach
Do It Right, First Time
What Makes A Performant OBIEE System?
!

If Its Not Done Right, Know How To Figure Out What Is Broke
Practical elements of diagnosing the causes of performance issues
Methodical analysis - nose to tail

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Is your problem Performance or Capacity?

Response time

Performance: Response time is slow for one user

Capacity: Response time degrades as user concurrency increases

Number of concurrent users

Adding Capacity alone will not fix your performance problems

Performance improvements can alleviate Capacity problems

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Nose to Tail Diagnostics

Request
DWH
Managed Server
Web Browser

BI Plug-in

Presentation
Services

[ ... ]

BI Server

WebLogic Server

OBIEE system components

Managed Server
Web Browser

BI Plug-in

Network

User

Network

[ ... ]
Data Source(s)
DWH
Presentation
Services

JavaHost

BI Server
[ ... ]

[ ... ]

Response
Not all components listed, eg cluster controller

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Time Profile
To understand why is it slow, we first must understand where is it slow
!

Approach championed by Cary Millsap / Method-R

Thinking Clearly About Performance (2010)

Performance improvement is proportional to how much a

program uses the thing you improved.
Amdahls Law

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

 End-user reports a response time of 40 seconds.

How can we make it faster?

DB Query 1
25 seconds

Where did the time get spent?

Data
Processing
Page Generation

10 seconds

5 seconds

DB Query 2
5 seconds

Component
User

WebLogic Server

OBIEE system components

Managed Server
Web Browser

BI Plug-in

Presentation
Services

BI Server

Response

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Data Source(s)
DWH

DWH

Time, seconds

Time Profile in Action

Time Profile in Action

DB Query 1

DB Query 2
Data Processing

Page Generation

Step

Action

Response
Time (s)

Physical SQL 1 execute on DB

25.00

62.5

Physical SQL 2 execute on DB

[5.00]

BI Server combines dataset from DB results

10.00

25.0

Presentation Services generates page

5.00

12.5

Time profile shows clearly :

[Link] performance of Query 1
[Link] work into single query

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Total 40.00

100.0

Deep Dive into BI Server Time from [Link]

Elapsed

Logical Query Summary Stats: Elapsed time e

Response

Logical Query Summary Stats: [...] Response time r

Physical

DB Connect

Compilation

Physical Query 1

Logical Query Summary

Stats: [...] Compilation
time c (seconds)

Physical query response

time p1 (seconds)
Physical Query x

BI Server
processing
(federation,
calculation,
aggregation)

Send to client, wait for

acknowledgement

Eg. Presentation Services

generating pivot tables,
table scrolling/paging, etc.

Physical query response

time px (seconds)
Physical Query Summary Stats:
[...] DB-connect time d (seconds)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

r - d - c - max(p)

e - r

[Link] or Usage Tracking?

Step
Logical Query Compilation
Database connection
Physical Query 1 response
Physical Query 2 response
Physical Query n response
Logical Query Response
Logical Query Elapsed
BI Server processing (federation,
calculation, aggregation)
Send to client, wait for
acknowledgement (eg sawserver
generating pivot tables etc)

[Link] source

Usage Tracking

c
d
p
p
p
r
e
r-d-c-max(p)

Logical Query Summary Stats: [...] Compilation time

S_NQ_ACCT.COMPILE_TIME_SEC

Physical Query Summary Stats: [...] DB-connect time

n/a

Physical query response time

S_NQ_DB_ACCT.TIME_SEC

Physical query response time

S_NQ_DB_ACCT.TIME_SEC

Physical query response time

S_NQ_DB_ACCT.TIME_SEC

Logical Query Summary Stats: [...] Response time

S_NQ_ACCT.TOTAL_TIME_SEC

Logical Query Summary Stats: Elapsed time

n/a

[derived]

[derived]

e-r

[derived]

n/a

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Usage Trackings blind spot

Usage Tracking is good, but it omits the Total Elapsed Time
!
!
!
!
!
!
!

Elapsed
[Link]:
Logical Query Summary Stats: Elapsed time e
Usage Tracking: [Not Available]
Response
[Link]:
Logical Query Summary Stats: [...] Response time r
Usage Tracking: S_NQ_ACCT.TOTAL_TIME_SEC

uh oh!

This can hide problems:

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

Send to client, wait for

acknowledgement

E : info@[Link]
W : [Link]

Iterative Approach
Be sure to account for all time, end-to-end (nose to tail)
Expand time profile to focus on where the time is going in particular
In reality, time profile based on BI Server alone will help in a lot of cases

Browser tools

WLS logs

[Link]

[Link]

DB Explain plans, V$SQL, etc

Request
OBIEE system components

Managed Server
BI Plug-in

Presentation
Services

BI Server

Response

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Data Source(s)
Network

Web Browser

WebLogic Server
Network

User

DWH

DWH

Iterative Approach
Most components have extended logging
levels
Presentation Services
WebLogic Server
Browser Tools

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Digging Deeper with DMS Metrics

OBIEE supports Dynamic Monitoring System (DMS)
!

Hundreds of low-level metrics

!

Ranging from the obvious

(BI Server Total Sessions)
to the less obvious
(Peak_DXEParameterizedImpl_Count)
!

Once we have found WHERE the time has gone,

DMS metrics are one way to help us find out WHY

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Accessing DMS Metrics

DMS Spy
opmn
EM
obi-metrics-agent
+ graphite
+ grafana

[Link]
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

DMS Case Study

Query is running slowly, but its not clear why

Step
[Link] source
Logical Query Summary Stats: Elapsed time

Logical Query Elapsed

e
Logical Query Summary Stats: [...] Response time
Logical Query Response
r

Logical Query Summary Stats: [...] Compilation time

Logical Query Compilation
c

Physical Query Summary Stats: [...] DB-connect time

Database connection
d
Physical query response time
Physical Query 1 response
p

BI Server processing
r-d-c-max(p) [derived]

[derived]
Send to client, wait for cursor close
e-r
1

CPU is only at 50% utilisation

Only a handful of rows returned from the DB

What is the BI Server up to for this time?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Time %
14
13
0
0%
0
0%
4
29%
9
64%
1
7%

DMS Provides the Proof

The Connection Pool has a fixed Capacity
Connections will start to Queue once Capacity is reached

Physical connections limit is reached for

connection pool "Star_orcl_Connection Pool 01".
Start delaying the queries

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Performance Diagnostics Approach

Capacity
Examine capacity metrics (OS, DMS) over time
Correlate with reported problems

Performance
Build a time profile
Dont know where to start? Try Usage Tracking or [Link]
- Long running queries
- Logical queries returning lots of data to the client
- Logical queries returning lots of data from the DB but small % to the client
- Logical queries requiring lots of physical queries

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

So What Does Good Look Like?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

This Is Not Best Practice

There is no one right way : It

Depends

c.f. ExtremeBI

The only best practice you should be using all the

time is Use Your Brain.
Steven Robbins / Tom Kyte

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

The Bucket List of Performance Gains

Dont Do It
Do It Only Once
Do It Less Often
Do It More Efficiently
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

(h/t Greg Rahn)

Database Pushdown (Do It More Efficiently)

Reduce the amount of work/data handled further up the stack
Federation across sources can be a challenge
c.f. BI Server caching

BI Plug-in

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

OBIEE system components

Presentation Services

E : info@[Link]
W : [Link]

BI Server

Data Source(s)
Network

Web Browser

WebLogic Server
Managed Server
Network

User

DWH

[ ... ]

Do You Need Database Pushdown?

BI Servers temp directory shows lots of work being done by BI Server
Long time in BI Server processing phase
Usage Tracking / Query logs :
How much data the BI Server pulls back from the database as a percentage of rows
returned to the user (low % is bad).
How many database queries one report triggers - higher is generally less efficient
Step
Logical Query Elapsed
Logical Query Response
Logical Query Compilation
Database connection
Physical Query 1 response
BI Server processing
Send to client, wait for cursor close

[Link] source
Logical Query Summary Stats: Elapsed time
e
Logical Query Summary Stats: [...] Response time
r
Logical Query Summary Stats: [...] Compilation time
c
Physical Query Summary Stats: [...] DB-connect time
d
Physical query response time
p1
r-d-c-max(p) [derived]
[derived]
e-r

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Feeding the Excel Monster (Do It More Efficiently / Dont Do it)

Whats being done with the data once its in Excel?
Could it be done in OBIEE instead?
!

Alternatives to Export from Analysis/Dashboard:

Dump direct from the database
Use Logical SQL against BI Servers ODBC/JDBC
interface directly
Oracle have specific recommendations (DocID
1558070.1 p.13)
- Favour CSV export over Excel
- Favour BI Publisher export over OBIEE Analysis Export

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Dont Do It / Do It Less Often

Filter Early
Table Prompt != Filter
Report by Exception
!

Make sure Dashboards have

default/mandatory prompts
([Link]

Yes, please
do.

Be smart about Dashboard and Analysis design - dont cram everything into one page
Less clutter : Better user experience & Better performance

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

Not forgetting
Do It More Efficiently
Aggregation
Do It Less Often
BI Server caching
!

Database optimisation (partitioning, indexing, parallelism, statistics, etc)

Balanced Hardware configuration

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

OBIEE Performance in the Real World

Evidence-based DESIGN and DIAGNOSTICS
!

Do It Right, First Time

!

If Its Not Done Right, Know How To Figure Out What Is Broke
Methodical analysis - nose to tail

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

#EOF
email:
[Link]@[Link]
web:
[Link]
twitter: @rmoff

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]

References & Further Reading

Cary Millsap - Thinking Clearly About Performance
[Link]
Blog series: OBIEE and Performance
[Link]
obi-metrics-agent
[Link]
Greg Rahn - The Core Performance Fundamentals Of Oracle Data Warehousing Balanced
Hardware Configuration
[Link]
Oracle documentation - Oracle Database 2 Day + Data Warehousing Guide - Balanced Hardware Configuration
E25555-03

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : info@[Link]
W : [Link]