<Insert Picture Here>

Back to Basics:
DB Time Performance Tuning: Theory and Practice
John Beresniewicz, Graham Wood
Oracle America ST Partners June, 2011
 The following is intended to outline our
general product direction. It is intended for
information purposes only, and may not be
incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in
making purchasing decisions.
The development, release, and timing of any
features or functionality described for
Oracle‟s products remains at the sole
discretion of Oracle.
 Agenda

• Time
• Database Time
• Average Active Sessions

• Techniques
• The DB Time Method

• Tools
• ADDM
• EM User Interface
• Active Reports
 Oracle Tuning Methods: A History

• Prehistoric (v5)
• Debug code
• Dark Ages (v6)
• Counters/Ratios
• BSTAT/ESTAT
• SQL*Trace
• Renaissance (v7/v8)
• Introduction of Wait Event instrumentation
• Move from counters to timers
• STATSPACK
• Modernity (v10)
• DB Time Tuning – Tuning using fundamental notion of time spent in database
• Multiple scoping levels
• Always on, non-intrusive
• Built into infrastructure: instrumentation, ASH, AWR, ADDM, EM
 Why Do We Care About Time?

• Human time is critical to the enterprise

• Systems performance affects business goals

• Human time + technology resource time

• “Time is money”

• Performance improvement means doing things faster

Performance is always and only about time

 <Insert Picture Here>

Database Time and

Average Active Sessions
 Database Time (DB Time)

• Total time in database calls by foreground sessions

• Includes CPU time, IO time and non-idle wait time

• DB Time <> response time

• Common currency for Oracle performance analysis

Database time is total time spent by user

processes either actively working or actively
waiting in a database call.
A Single Session
Single session with Database Black Box server

Browse Read Reviews

Add to Cart Checkout
Books For One Book

= time spent in database TIME

Fundamental Concepts

Database Time (DB Time) =

Total time session spent in all database calls
Active Session =
Session currently spending time in a database call
Average Activity of the Session (% Activity) =
The ratio of time active to total wall clock time

Browse Read Reviews

Add to Cart Checkout
Books For One Book

= time spent in database TIME

Multiple Sessions
DB Time = Sum of DB Time Over All Sessions

Avg. Active Sessions = Sum of Avg. Activity Over All Sessions

At time t we have 2 active sessions

User 1

User 2

User 3

User n
TIME
t
= time spent in database
Visualizing DB Time

Total Database Time

Avg. Active Sessions =
Wall Clock (Elapsed) Time

User 1
User 2
User 3
User n

t0 t1

4
3
2
1

Active Sessions over time TIME

 EM Performance Page

• Active Sessions by wait class over time

• Colored area = amount of DB time
• “Click on the big stuff”
 Average active sessions

= DB time / elapsed time

• Time-normalized DB time
• Time units in numerator and denominator must
synchronize to produce the proper metric
 Average Active Sessions

• Full-time equivalent sessions

• Not whole sessions
• How many full-time virtual sessions
to do the work?

• Comparable
• Across systems
• Across time periods
 <Insert Picture Here>

DB Time and
System Performance
 System Load and DB Time

• More users
• => More calls
• => DB time increases

• Larger transactions
• => Longer calls
• => DB time increases

DB time increases as system load increases.

 System Performance and DB Time

• IO performance degrades
• => IO time increases
• => DB time increases

• Application performance degrades

• => Wait time increases
• => DB time increases

DB time increases when performance degrades.

 Host Performance and DB Time

• Host is CPU-bound
• => foregrounds accumulate active run-queue time
• => wait event times are artificially inflated
• => DB time increases

Tune for CPU before waits when CPU constrained

 CPU Run-queue and DB Time

Recorded wait time Recorded wait time

User 1

Run-queue Run-queue
Db file sequential read On CPU Db file sequential read On CPU

Actual wait time Actual wait time

DB time is inflated when host is CPU-bound

System performance and DB time

CPU or I/O problem?

 Instrumentation: Where to find DB
Time?
• V$SYS_TIME_MODEL, V$SESS_TIME_MODEL
• STAT_NAME = „DB time‟

• V$SYSMETRIC_HISTORY
• “Database Time Per Second”, “CPU Usage Per Sec”
• 10g units = centi-secs/sec (100xAvg. Active Sessions)
• 11g new metric “Average Active Sessions”

• V$SQL
• ELAPSED_TIME and CPU_TIME
• Wait class times:
APPLICATION, CONCURRENCY, CLUSTER, USER_IO

• V$ACTIVE_SESSION_HISTORY
 <Insert Picture Here>

Active Session History

 Active Session History (ASH)

• All „Active‟ sessions captured every second

• Foregrounds and backgrounds are sampled
• Active foregrounds contribute to DB Time

• In-memory: V$ACTIVE_SESSION_HISTORY
• Sampling interval = 1 second

• On-disk: DBA_HIST_ACTIVE_SESS_HISTORY
• Sampling interval = 10 second

• ASH is a system-wide record of database activity

ASH Math

COUNT(*) = DB Time
GROUP BY ?
 ASH Math: COUNT(*)=DB Time

• ASH is a big fact table

• Each row represents 1-second of active session time

• V$ACTIVE_SESSION_HISTORY
• COUNT(*) = DB time in seconds

• DBA_HIST_ACTIVE_SESS_HISTORY
• COUNT(*) * 10 = DB time in seconds
 Estimating DB Time with ASH

• ASH sample counts = DB Time in seconds

• Low sample counts are less reliable

• Enables DB Time analysis over many dimensions

• Sqlid, session id, instance, service, module, action
• 10gR2
• Blocking_sid (10gR2)
• XID
• 11g
• Row source
• Execution ID
• Operation type
• Connect
• Java/SQL/PLSQL
• parse, bind, execute/fetch, close
Example: DB Time by SQL ID
select sql_id
, count(*) DBTime
, round(count(*)*100/sum(count(*))
over (), 2) pctload
from v$active_session_history
where sample_time > sysdate - 1/24/60
and session_type <> 'BACKGROUND‘
group by sql_id
order by count(*) desc;
Example: DB Time by SQL ID
select sql_id
, count(*) DBTime
, round(count(*)*100/sum(count(*))
over (), 2) pctload
from v$active_session_history
where sample_time > sysdate - 1/24/60
and session_type <> 'BACKGROUND'
group by sql_id
order by count(*) desc;

SQL_ID DBTIME PCTLOAD

------------- --------- -------
6bmxrabnwwsxd 60 63.83
azzsynmz43nrr 8 8.51
28pb73sbwhmm8 5 5.32
58psyvgau23s2 3 3.19
amrq8hk767tuz 2 2.13
2r5qhb3fb63vm 1 1.06
f3919usqp5wj2 1 1.06
 The calculus of DB time

• The number of active sessions at any time is the rate

of change of the DB time function at that time.

DBtime / t  ActiveSess ions

• DB time is the integral of the Active Session function.

t1
DBtime   ActiveSess ions
t0
 Avg Active Sessions and DB Time
ASH sample count is value
Active sessions of active sessions function
at sample times

DB time is area
under curve t = 1 sec

DB Time

t0 time t1
DB Time: ASH vs Time Model
 ASH Timing for Nano-Operations

• Some important operations are still too frequent and

short-lived for timing
• No “bind” wait event
• A session-level bit vector is updated in binary fashion
before/after an operation
• Cheaper than timer call
• The bit vector is sampled into ASH
• ASH math allows us to estimate time spent in these
un-timed transient operations
 <Insert Picture Here>

Techniques:
The DB Time Method
 Where is DB Time used?

• ADDM

• EM Performance page and drill downs

• ASH report

• AWR and AWR compare periods reports

• SYSMETRICS and Server-generated Alerts

The DB Time Method: Short Course

Determine where
or database
time isjust
spent, and reduce
ask ADDM it!
 The DB Time Method: Process

1. Identify performance issue

2. Scope the issue
3. Set goals
4. Data capture (NO OP)
5. Investigate DB time distribution
• Identify the largest potential for improvement
6. Modify system to tune for largest gain
7. Evaluate against goals
• Repeat from step 4 if goals not met

Performance tuning by removing excess DB time

 Investigate DB Time Distribution

• Identify uneven distributions of DB time (skew)

• => Largest potential improvement within scope

• System scope:
• Resource limits – is problem outside the DB?

• Application scope:
• Service, module, action
• Resource contention (e.g. latches)
• SQLID, rowsource

• Session scope:
• Long running SQL
• Resource contention (e.g. enqueues)
DB time
 Identify Potential Solutions

• Session contention issues

• Kill session
• Fix application

• SQL issues
• SQL Tuning Advisor => Indexes, SQL profile
• Re-write SQL

• Design issues
• Access Advisor => Indexes, physical layout

• System issues
• Initialization parameters
• Add resources
 Modify System

• Start with the largest DB time issues first

• Address root causes, not symptoms

• Match solution scope to problem scope

• Don‟t tweak optimizer parameters before tuning SQL

• Proceed iteratively one fix at a time

• Concurrent fixes should be orthogonal

• Measure and validate results at each successive step

• Stop when goals are met

 The DB Time Method: Advantages

• Tunes the one thing that affects users: Time

• Data capture scoping not necessary

• „Always on‟ data collection
• No requirement to reproduce problem

• Works for concurrency problems such as locking

• Combines best of current methods

• Less intrusive, more inclusive
 Method Summary

• DB time is the fundamental performance metric

• The method allows DB time analysis at many scopes

• Properly scoped problems and solutions are critical to success

• DB time based diagnosis removes value judgments

• Scientific method, not sorcerer‟s magic

• Performance improvement means doing the same work in

less DB Time
 <Insert Picture Here>

Tools:
ADDM
Enterprise Manager
Active Reports
 Tools for Applying DB Time Method

Two use-cases, one method:

1. Tuning steady-state performance

• Improve overall workload throughput or response time
• Best practice: use ADDM

2. Diagnosing transient performance problems

• Confirm and investigate reported performance issues
• Best practice: use EM real-time screens
 Best Practice: Use ADDM

• Embedded expert system using the DB time method

• Identifies root causes behind the symptoms

• Variably scoped:
• Host to instance to SQL and even database block
• Scoped to database for RAC (new in 11g)

• Findings prioritized by impact on DB time

• Finding history allows flexible time scoping
• Directives can filter findings

• Recommendations by benefit (reduction) to DB time

 Best Practice: EM Real-time Interface

• Transient (sub-hour) or immediate time scope

• Requires interactivity of UI

• „Click on the big stuff‟

• Data visualizations display skew directly

• Takes some expertise to separate symptoms from

root causes
Grid Control DB Loadmap
Grid Control DB Loadmap