CASE STUDY

Using 11g Real Application Testing to analyse the impact of making the following
changes:

1) Upgrade the database from 10g to 11g

2) Convert from single instance database to a two node RAC database
3) Convert from file system based storage to 11g ASM

Overview of Real Application Testing

One of the best new features in Oracle 11g is the Real Application Testing option
which enables us to test the result of a change without actually implementing the same
in a production environment. Changes such as database version upgrades, optimizer
changes, hardware and storage changes all carry with them an element of risk and
uncertainty. Using this feature much of that uncertainty and guesswork can be
avoided.

Real Application Testing comprises of two components:

a) Database Replay
b) SQL Performance Analyzer (SPA)

SPA enables us to test the impact of a change at the level of a particular SQL
statement to see if any improvements or regressions occurred in the execution of the
SQL statement because of the change.

Database Replay enables us to capture workload real time in a production

environment and then replay the same in a clone or copy of the changed production
environment with the exact timings, concurrency and transaction characteristics found
in the original workload. The AWR comparison and Database Replay reports can
easily help us pinpoint any potential problems and errors as well any divergence in
performance.

This feature will afford businesses increased confidence and surety when faced with a
system change as well as significantly reducing the costs as well as resources
associated with the testing phase.

Database Capture is now supported in versions 9.2.0.8 as well as 10.2.0.3 and 10.2.0.4
enabling us to test the impact of upgrades from both 9i as well as 10g to 11g. Note,
however, that replay can only be performed in a Oracle 11g database.

At a very high level, the database capture and replay will involve the following steps:

1) Prepare the source database for capture

2) Capture the workload
3) Copy the workload capture files to the target machine
4) Prepare the target database for replay and process captured workload files
5) Start the workload replay clients
 6) Once the workload replay has finished, examine the generated reports and
make the analysis

Scenario

We will be generating some workload in an Oracle 10.2.0.4 database (TESTDB)

hosted on AIX with JFS using the SwingBench load generation tool. The tool will
simulate a Order Entry and Shipping application where customers place orders and
those orders are processed and then shipped. The tool will generate 25 user
connections and will run for about 35 minutes going through the entire transaction
cycle. The database capture process is initiated before the load generation starts and
then the workload is replayed in a two node RAC database (BWDBA) running on
ASM storage.

Prepare the source 10.2.0.4 database for capture

We will create a directory to hold the captured workload files and then also create a
filter so that only database operations of the user ‘SOE’ are captured.

SQL> create directory capture_dir as '/u02/oradata/capture/';

Directory created.

SQL> grant all on directory capture_dir to public;

Grant succeeded.

SQL> exec
dbms_workload_capture.add_filter('swingbench_filter','USER','SOE');

PL/SQL procedure successfully completed.

We then also need to set the parameter PRE_11G_ENABLE_CAPTURE to TRUE

otherwise we will get the error as shown below when we try to start
the capture:

SQL> exec
dbms_workload_capture.start_capture('10204_TO_11g,'CAPTURE_DIR',NULL,
'EXCLUDE');
BEGIN
dbms_workload_capture.start_capture('10204_TO_11g','CAPTURE_DIR',NULL
, 'EXCLUDE'); END;

*
ERROR at line 1:
ORA-15591: cannot start capture because parameter
"PRE_11G_ENABLE_CAPTURE" is
not enabled
ORA-06512: at "SYS.DBMS_WORKLOAD_CAPTURE", line 799
ORA-06512: at line 1
SQL> show parameter PRE_11G_ENABLE_CAPTURE

NAME TYPE VALUE

------------------------------------ -----------
------------------------------
pre_11g_enable_capture boolean FALSE

SQL> alter system set pre_11g_enable_capture=TRUE;

System altered.

Start the Capture process

SQL> exec
dbms_workload_capture.start_capture('10204_TO_11g','CAPTURE_DIR',NULL
, 'EXCLUDE');

PL/SQL procedure successfully completed.

If we go to the /u02/oradata/capture location we will find some files

as shown below

testdb:/u02/oradata/capture> ls -l
total 16
-rw-r--r-- 1 oracle dba 995 Sep 11 08:56
wcr_4pad09400280u.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:56
wcr_cap_0001n.start
-rw-r--r-- 1 oracle dba 130 Sep 11 08:56
wcr_scapture.wmd

Start the workload generation using Swingbench

testdb:/u01/oracle> cd /u01/oracle/loadgen/swingbench/bin
testdb:/u01/oracle/loadgen/swingbench/bin>

testdb:/u01/oracle/loadgen/swingbench/bin> export
DISPLAY=10.107.105.179:0.0

testdb:/u01/oracle/loadgen/swingbench/bin> ./swingbench
Started Swingbench, Version 2.3.0.422. Using config file
swingconfig.xml
BI Beans Graph version [2.7.5.32] Engine version [2.7.5.32]
If we query the database, we can see an number of PL/SQL calls are
being made by the 25 connected user sessions
SU TXT
----------
---------------------------------------------------------------------
-----------
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.processorders(:2,:3); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.processorders(:2,:3); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.processorders(:2,:3); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseandupdateorders(:2,:3,:4);
END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.browseproducts(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;
SOE BEGIN :1 := orderentry.neworder(:2,:3,:4); END;

25 rows selected.

If we go to the CAPTURE_DIR location we can see a number of *.rec

files are also generated. These are the binary workload capture
files.

testdb:/u02/oradata/capture> ls -lrt
total 16
-rw-r--r-- 1 oracle dba 130 Sep 11 08:56
wcr_scapture.wmd
-rw-r--r-- 1 oracle dba 0 Sep 11 08:56
wcr_cap_0001n.start
-rw-r--r-- 1 oracle dba 995 Sep 11 08:56
wcr_4pad09400280u.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003w2h.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003s3q.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003h66.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003h5f.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00347u.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00300u.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002w0w.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002s6c.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00284n.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002812.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003s34.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003s00.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003n5k.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003854.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00383f.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00342n.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00341k.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002n5n.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002h0h.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002c6k.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002846.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00243q.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs00201q.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs003h3c.rec
-rw-r--r-- 1 oracle dba 0 Sep 11 08:57
wcr_4pad0fs002w3n.rec

The Swingbench tool has been configured to run for 35 minutes and
will automatically logoff the 25 SOE user sessions once that time
period has elapsed.

Stop the Workload Capture and Generate the AWR export

SQL> exec dbms_workload_capture.finish_capture;

PL/SQL procedure successfully completed.

SQL> declare
capture_id number;
begin
select max(id) into capture_id
from dba_workload_captures
where status = 'COMPLETED';
dbms_workload_capture.export_awr(capture_id);
end;
/

PL/SQL procedure successfully completed.

testdb:/u02/oradata/capture> ls -lrt
total 25544
-rw-r--r-- 1 oracle dba 130 Sep 11 08:56
wcr_scapture.wmd
-rw-r--r-- 1 oracle dba 995 Sep 11 08:56
wcr_4pad09400280u.rec
-rw-r--r-- 1 oracle dba 71 Sep 11 08:58
wcr_4pad0qc003c2k.rec
-rw-r--r-- 1 oracle dba 71 Sep 11 08:58
wcr_4pad0qc002n00.rec
-rw-r--r-- 1 oracle dba 71 Sep 11 09:21
wcr_4pad60n002s70.rec
-rw-r--r-- 1 oracle dba 981 Sep 11 09:23
wcr_4pad1t800202f.rec
-rw-r--r-- 1 oracle dba 254839 Sep 11 09:27
wcr_4pad0fs003s34.rec
-rw-r--r-- 1 oracle dba 234486 Sep 11 09:27
wcr_4pad0fs00300u.rec
-rw-r--r-- 1 oracle dba 208884 Sep 11 09:27
wcr_4pad0fs002s6c.rec
-rw-r--r-- 1 oracle dba 216954 Sep 11 09:27
wcr_4pad0fs002h0h.rec
-rw-r--r-- 1 oracle dba 210586 Sep 11 09:27
wcr_4pad0fs002c6k.rec
-rw-r--r-- 1 oracle dba 222085 Sep 11 09:27
wcr_4pad0fs003s00.rec
-rw-r--r-- 1 oracle dba 218768 Sep 11 09:27
wcr_4pad0fs00347u.rec
-rw-r--r-- 1 oracle dba 236026 Sep 11 09:27
wcr_4pad0fs002w3n.rec
-rw-r--r-- 1 oracle dba 222652 Sep 11 09:27
wcr_4pad0fs00284n.rec
-rw-r--r-- 1 oracle dba 238417 Sep 11 09:27
wcr_4pad0fs002846.rec
-rw-r--r-- 1 oracle dba 239365 Sep 11 09:27
wcr_4pad0fs003n5k.rec
-rw-r--r-- 1 oracle dba 224863 Sep 11 09:27
wcr_4pad0fs003h3c.rec
-rw-r--r-- 1 oracle dba 230934 Sep 11 09:27
wcr_4pad0fs00243q.rec
-rw-r--r-- 1 oracle dba 227368 Sep 11 09:27
wcr_4pad0fs00201q.rec
-rw-r--r-- 1 oracle dba 231523 Sep 11 09:27
wcr_4pad0fs00342n.rec
-rw-r--r-- 1 oracle dba 226836 Sep 11 09:27
wcr_4pad0fs00341k.rec
-rw-r--r-- 1 oracle dba 227811 Sep 11 09:27
wcr_4pad0fs003s3q.rec
-rw-r--r-- 1 oracle dba 222510 Sep 11 09:27
wcr_4pad0fs003h5f.rec
-rw-r--r-- 1 oracle dba 223408 Sep 11 09:27
wcr_4pad0fs003854.rec
-rw-r--r-- 1 oracle dba 221885 Sep 11 09:27
wcr_4pad0fs00383f.rec
-rw-r--r-- 1 oracle dba 239250 Sep 11 09:27
wcr_4pad0fs002n5n.rec
-rw-r--r-- 1 oracle dba 229125 Sep 11 09:27
wcr_4pad0fs002w0w.rec
-rw-r--r-- 1 oracle dba 242049 Sep 11 09:27
wcr_4pad0fs002812.rec
-rw-r--r-- 1 oracle dba 220770 Sep 11 09:27
wcr_4pad0fs003w2h.rec
-rw-r--r-- 1 oracle dba 222513 Sep 11 09:27
wcr_4pad0fs003h66.rec
-rw-r--r-- 1 oracle dba 983 Sep 11 09:33
wcr_4pad8uc00385n.rec
-rw-r--r-- 1 oracle dba 200 Sep 11 09:33
wcr_fcapture.wmd
-rw-r--r-- 1 oracle dba 47916 Sep 11 09:33 wcr_cr.html
-rw-r--r-- 1 oracle dba 23826 Sep 11 09:33 wcr_cr.text
-rw-r--r-- 1 oracle dba 20667 Sep 11 09:36 wcr_ca.log
-rw-r----- 1 oracle dba 7200768 Sep 11 09:36 wcr_ca.dmp

Prepare the target database for Workload Replay

In normal circumstances, we would have taken a backup of the source

database before we started the load generation and then restored the
backup on a test server to run the replay. But in this case, we are
using a pre-created 11g RAC database with the required SOE schema
objects so that the replay can be run on this. Note that the database
is having the ASM storage as well.

We will create the Replay directory and then move all the captured
workload files from the source machine (devu007) to the target
machine (10.1.210.21). This is one of the nodes in the cluster, the
other node being 10.1.210.22.

SQL> create directory replay_dir as '/u01/oracle/replay/';

Directory created.

SQL> grant read on directory replay_dir to public;

Grant succeeded.

testdb:/u02/oradata/capture> scp -rp *.*

10.1.210.21:/u01/oracle/replay
[email protected]'s password:
wcr_4pad09400280u.rec
100% 995 1.0KB/s 00:00
wcr_4pad0fs00201q.rec
100% 222KB 222.0KB/s 00:00
wcr_4pad0fs00243q.rec
100% 226KB 225.5KB/s 00:00
wcr_4pad0fs002812.rec
100% 236KB 236.4KB/s 00:00
wcr_4pad0fs002846.rec
100% 233KB 232.8KB/s 00:00
wcr_4pad0fs00284n.rec
100% 217KB 217.4KB/s 00:00
wcr_4pad0fs002c6k.rec
100% 206KB 205.7KB/s 00:00
wcr_4pad0fs002h0h.rec
100% 212KB 211.9KB/s 00:00
wcr_4pad0fs002n5n.rec
100% 234KB 233.6KB/s 00:00
wcr_4pad0fs002s6c.rec
100% 204KB 204.0KB/s 00:00
wcr_4pad0fs002w0w.rec
100% 224KB 223.8KB/s 00:01
wcr_4pad0fs002w3n.rec
100% 230KB 230.5KB/s 00:00
wcr_4pad0fs00300u.rec
100% 229KB 229.0KB/s 00:00
wcr_4pad0fs00341k.rec
100% 222KB 221.5KB/s 00:00
wcr_4pad0fs00342n.rec
100% 226KB 226.1KB/s 00:00
wcr_4pad0fs00347u.rec
100% 214KB 213.6KB/s 00:00
wcr_4pad0fs00383f.rec
100% 217KB 216.7KB/s 00:00
wcr_4pad0fs003854.rec
100% 218KB 218.2KB/s 00:00
wcr_4pad0fs003h3c.rec
100% 220KB 219.6KB/s 00:00
wcr_4pad0fs003h5f.rec
100% 217KB 217.3KB/s 00:00
wcr_4pad0fs003h66.rec
100% 217KB 217.3KB/s 00:00
wcr_4pad0fs003n5k.rec
100% 234KB 233.8KB/s 00:00
wcr_4pad0fs003s00.rec
100% 217KB 216.9KB/s 00:01
wcr_4pad0fs003s34.rec
100% 249KB 248.9KB/s 00:00
wcr_4pad0fs003s3q.rec
100% 222KB 222.5KB/s 00:00
wcr_4pad0fs003w2h.rec
100% 216KB 215.6KB/s 00:00
wcr_4pad0qc002n00.rec
100% 71 0.1KB/s 00:00
wcr_4pad0qc003c2k.rec
100% 71 0.1KB/s 00:00
wcr_4pad1t800202f.rec
100% 981 1.0KB/s 00:00
wcr_4pad60n002s70.rec
100% 71 0.1KB/s 00:00
wcr_4pad8uc00385n.rec
100% 983 1.0KB/s 00:00
wcr_ca.dmp
100% 7032KB 6.9MB/s 00:00
wcr_ca.log
100% 20KB 20.2KB/s 00:00
wcr_cr.html
100% 47KB 46.8KB/s 00:00
wcr_cr.text
100% 23KB 23.3KB/s 00:00
wcr_fcapture.wmd
100% 200 0.2KB/s 00:00
wcr_scapture.wmd
100% 130 0.1KB/s 00:00

Process the workload replay files on target

On the 11g target database we need to process the captured workload

files. We need to do this always on the 11g side of things and this
needs to be done only once. In other words, we process once but can
replay many times.

SQL> exec dbms_workload_replay.process_capture('REPLAY_DIR');

PL/SQL procedure successfully completed.

Database Replay

While all the steps mentioned here can be performed using the
DBMS_WORKLOAD_REPLAY API, we will demonstrate the same using the GUI
11g Enterprise Manager (Database Control and not Grid Control)

We need to click on the ‘Software and Support’ tab and then go to the
‘Database Replay’ link.
Since we have already finished the Capture Workload and Preprocess
Captured Workload tasks, we need to click on the Replay Workload Go
to Task
Once we select the ‘REPLAY_DIR’ in the directory object drop down list, we find
that the page is then populated with the relevant capture details
We provide a name for the Workload Replay ……
We can customise certain options in the workload replay like the database connection
strings and the speed at which the replay is done among others. Note in this case the
tns connect alias is the one that has been enabled for the RAC database providing for
load balancing.using the service ‘bwdba’.
We then need to configure and start the Workload Replay Clients (WRC) on the
server using the wrc command.

The workload replay clients and responsible for parsing and sending the workload to
the server. The number of clients that need to be started depends on the amount of
workload that wil be run and the by running the wrc command in calibration mode, it
will help us determine how many clients we need for the job on hand.

In our case, it shows that we need the single workload client to replay the workload
that has been generated.
middba1:/u01/oracle/product/11.1.0/db/bin> wrc system/oracle mode=calibrate
replaydir=/u01/oracle/replay

Workload Replay Client: Release 11.1.0.6.0 - Production on Fri Sep 11 10:11:51 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

Report for Workload in: /u01/oracle/replay

-----------------------

Recommendation:
Consider using at least 1 clients divided among 1 CPU(s).

Workload Characteristics:
- max concurrency: 25 sessions
- total number of sessions: 25

Assumptions:
- 1 client process per 50 concurrent sessions
- 4 client process per CPU
- think time scale = 100
- connect time scale = 100
- synchronization = TRUE
We then start the Workload Replay Clients by using the wrc command . We see that
the workload replay client after it starts has spawned a database session and an OS
process as well
middba1:/u01/oracle/product/11.1.0/db/bin> wrc system/oracle
replaydir=/u01/oracle/replay

Workload Replay Client: Release 11.1.0.6.0 - Production on Fri Sep 11 10:13:58 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

Wait for the replay to start (10:13:58)

If we now find that the Workload Replay has now actually started ……..
middba1:/u01/oracle/product/11.1.0/db/bin> wrc system/oracle
replaydir=/u01/oracle/replay

Workload Replay Client: Release 11.1.0.6.0 - Production on Fri Sep 11 10:13:58 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

Wait for the replay to start (10:13:58)

Replay started (10:15:54)

We can now go back to the Enterprise Manager session and view the progress of the
Workload Replay. We can see a graphical representation of the time that is being
taken for the replay as compared to the capture as well as the total work that is
remaining to be completed.
If we check after about 10 minutes we find that the replay on the RAC database is
happening almost 50% faster than the original capture time and by querying the
database we find that the load has been distributed between the two instances as well.
Of the 25 SOE sessions, 11 are connected to instance bwdba1 and 14 are connected
to instance bwdba2

BWDBA1

SQL> select count(*) from v$session where username='SOE';

COUNT(*)
----------
11

BWDBA2

SQL> select count(*) from v$session where username='SOE';

COUNT(*)
----------
14

Once the replay is fully completed, the workload replay clients will exit and in our
case we find just by using time taken as factor, the performance improvement on the
11g RAC database has been greater than 50%. Note that the replay clients ran for
about 21 minutes while the original workload capture took about 36 minutes.
middba1:/u01/oracle/product/11.1.0/db/bin> wrc system/oracle
replaydir=/u01/oracle/replay

Workload Replay Client: Release 11.1.0.6.0 - Production on Fri Sep 11 10:13:58 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.

Wait for the replay to start (10:13:58)

Replay started (10:15:54)
Replay finished (10:36:47)
If we go to the ‘Reports’ tab, we will find a number of report options. The most
important one in my opinion is the AWR comparison report.
Before we run the AWR Compare Periods report, we need to import the AWR data
that we exported when we completed the capture process on the source. We specify
the schema in which the imported AWR data will be temporarily staged and then a
Database Scheduler job is commenced to import the AWR data.
Once the job is completed, we can go back to the Database Replay page via the
Software and Support tab of the Database Control. Once we select the directory object
REPLAY_DIR from the drop down list, we will find that the
10204_TO_11g_REPLAY task has the status ‘Completed’. By clicking on the Name
link, we can go to the Reports tab and view the AWR Compare Periods report.

What stands out t he most is the % IO wait time of the second snapshot set which
pertains to the 11g RAC database is significantly lower than the first run. The RAC
database is hosted on ASM storage and this could clearly support the fact that IO
distribution and throughput in a well configured ASM system could outweigh other
conventional file systems.

A look at the Wait Events section will also indicate that we had I/O issues in the first
run on the 10g source system which is found in the fact that db file sequential reads
and log file sync and log file parallel writes events are the main wait events. These are
not found in the second run on the 11g RAC on ASM environment.

Based on just this workload report, we can safely conclude that the upgrade
from 10g to 11g on a cluster database hosted on an ASM storage environment
will not cause any real performance digressions – in fact, on the contrary there is
evidence to show that significant performance benefits will be attained by
making the change.