UAT database observations during functional test & EOD

Details of all DB Parameters changed in UAT to enhance performance:

1. _lm_drm_disable has been disabled.

2. SGA_MAX_SIZE & SGA_TARGET set to 600G.
3. shared_pool_size set to 30G as minimum.
4. db_cache_size set to 90G.
5. gcs_server_processes set to 8.
6. parallel_max_servers set to 1600.
7. parallel_min_servers set to 160.
8. parallel_servers_target set to 1600.
9. vm.nr_hugepages=320000

EOD Observations and recommendations –

1) Flash Cache Hit # shows negative value.

Negative value of flash cache hit is noticed in few AWR reports. This seems to be a bug to us. Please
raise an issue with Oracle support to get the fix. Also check with Oracle support if below metalink ID
is referring to this bug.
2) Indexes of table ACTB_DAILY_LOG, ACTB_HISTORY,
ICTB_BACK_DATED_EVENTS, ICTB_ACC_PR, ICTB_ITM_TOV has degree
set as DEFAULT

Observation: It seems some activity was performed last night on table

(ACTB_DAILY_LOG) and the degree was set as DEFAULT.

Action: Need to change the degree of these indexes from DEFAULT to 1.

3) Table ACTB_DAILY_LOG partition move and Indexes rebuild while

EOD is in progress.

No maintenance activity should be performed during EOD run.

4) Ges Generic Event – This wait event is observed in some AWR’s.

Refer doc ID - High Wait Counts For ges generic event wait Caused by
RMV Processes (Doc ID 2638402.1)
Observation – The high wait counts due to "ges generic event" wait event for RMV*
processes can be ignored when seen in an AWR.
Action – Check with oracle support if any action is required on this to
supress this wait event.

5) SQL_ID - cc9x1qm1js2su
INSERT INTO fbtb_txnlog_details_hist (BranchCode, FunctionId, XrefId, UserId, TxnStageId, TxnStatus,
Timestamp,CheckerId, ErrorCode, ReqXML, RespXML, OnlineStatus, STAGESTARTDATE, STAGEENDDATE, ADVICE,
ADVICEXML, STAGESTATUS, SEQUENCE_NO) (SELECT BranchCode, FunctionId, XrefId, UserId, TxnStageId,
TxnStatus, Timestamp, CheckerId, ErrorCode, ReqXML, RespXML, OnlineStatus, STAGESTARTDATE,
STAGEENDDATE, ADVICE, ADVICEXML, STAGESTATUS, SEQUENCE_NO FROM fbtb_txnlog_details WHERE
BRANCHCODE = :1 and TO_DATE(to_char(timestamp, :"SYS_B_0"), :"SYS_B_1") <= :2 )

SQL statement with SQL_ID "cc9x1qm1js2su" was executed 10 times and had
an average elapsed time of 22 seconds.

Action: Gather stats for table FBTB_TXNLOG_DETAILS_HIST,

FBTB_TXNLOG_DETAILS and rebuild the index IDX01_FBTB_TXNLOG_DETAILS
online.

6) SQL_ID - 2b8731qgfmzc4
UPDATE ACTBS_DAILY_LOG A SET A.VDBAL_UPDATE_FLAG = 'I' WHERE ROWID = :B1

SQL statement with SQL_ID "2b8731qgfmzc4" was executed 371 times

and had an average elapsed time of 15 seconds.

Action: Gather stats for table ACTB_DAILY_LOG and rebuild all indexes
online. This should be done when volume is highest in this table. Then
lock the statistics.

7) SQL_ID - gjbaqh71p95ay

INSERT INTO ICTBS_ENTRIES_HISTORY_ACCR (BRN, ACC, PROD, FRM_NO,

ENT_DT, AMT, ACCRUED_AMT, CUR_RUN_ACCR, AMT_TO_ACCRUE, RUN_DATE,
ENTRY_TYPE, CCY, LCY_AMT, CUR_RUN_ACCR_LCY, DRCR ,ENTRY_PASSED
,PROCESS ) VALUES (:B1 , :B2 , :B3 , :B4 , :B5 , :B6 , :B7 , :B8 ,
:B9 , :B10 , :B11 , :B12 , :B13 , :B14 , :B15 ,:B16 ,:B17 )

SQL statement with SQL_ID "gjbaqh71p95ay" was executed 745 times and
had an average elapsed time of 1.6 seconds.

Action: Gather stats for table ICTB_ENTRIES_HISTORY_ACCR and rebuild all

indexes online.

8) SQL_ID - 4417kf5xa92dh

INSERT INTO ICTB_ENTRIES_HIST_ZEROACCR (BRN, ACC, PROD, FRM_NO,

ENT_DT, AMT, ACCRUED_AMT, CUR_RUN_ACCR, AMT_TO_ACCRUE, RUN_DATE,
ENTRY_TYPE, CCY, LCY_AMT, CUR_RUN_ACCR_LCY,
LAST_CUR_RUN_ACCR,DRCR,ENTRY_PASSED ,PROCESS )
VALUES (:B1 , :B2 , :B3 , :B4 , :B5 , :B6 , :B7 , :B8 , :B9 ,
:B10 , :B11 , :B12 , :B13 , :B14 , :B15 , :B16 ,:B17 , :B18 )

SQL statement with SQL_ID "4417kf5xa92dh" was executed 1197 times and
had an average elapsed time of 4.7 seconds.

Action: Gather stats for table ICTB_ENTRIES_HIST_ZEROACCR and rebuild

index online.
9) SQL_ID - cdkwffa8jjafa

SELECT DISTINCT A.RELATED_CUSTOMER, A.TXN_CCY, A.DRCR_IND FROM

STVW_IR_CASH_TXN_LIST A WHERE EXISTS (SELECT 1 FROM STTM_CUSTOMER
WHERE CUSTOMER_NO = RELATED_CUSTOMER AND LOCAL_BRANCH = :B1 )

SQL statement with SQL_ID "cdkwffa8jjafa" was executed 39 times and had
an average elapsed time of 56 seconds.

Observation: STVW_IR_CASH_TXN_LIST is the view which contains

the table ACTB_DAILY_LOG, we identified some maintenance activity
happened on this table.

Action: Gather stats for table ACTB_DAILY_LOG and rebuild all indexes
online. This should be done when volume is highest in this table. Then
lock the statistics.

10) SQL_ID - gf923sxqjmv7t

SELECT A.*,ROWID RECORD_ROWID FROM CLTBS_ACCOUNT_EVENTS_DIARY A WHERE

A.EVENT_CODE IN ('TLIQ', 'TROL', 'TBOK','TSRL') AND A.CUTOFF_STATUS =
'U' AND A.BRANCH_CODE = :B3 AND A.PROCESS_NO = NVL(:B2 ,
A.PROCESS_NO) AND A.CONTRACT_STATUS ='A' AND EXECUTION_DATE <= :B1
ORDER BY A.ACCOUNT_NUMBER, A.EVENT_SEQ_NO

SQL statement with SQL_ID "gf923sxqjmv7t" was executed 39 times and had
an average elapsed time of 2.7 seconds.

Action: Use below command to set profile.

execute dbms_sqltune.accept_sql_profile(task_name
=>'sql_tuning_task_gf923sxqjmv7t', task_owner => 'SYS', replace =>TRUE);

11) SQL_ID - a32jn7kgk891t

INSERT INTO ACTBS_HISTORY (TRN_REF_NO, EVENT_SR_NO, EVENT, AC_BRANCH,

AC_NO, AC_CCY, DRCR_IND, TRN_CODE, AMOUNT_TAG, FCY_AMOUNT, \
EXCH_RATE, LCY_AMOUNT, RELATED_CUSTOMER, TRN_DT,
VALUE_DT,TXN_INIT_DATE,..

SQL statement with SQL_ID "a32jn7kgk891t" was executed 161 times and had
an average elapsed time of 6.1 seconds.

Action: Gather stats for table ACTB_HISTORY and rebuild the indexes.

12) SQL_ID - 3498fun4v64d7

DELETE FROM ACTB_DAILY_LOG WHERE ROWID = :B1

SQL statement with SQL_ID "3498fun4v64d7" was executed 171 times and had
an average elapsed time of 1.9 seconds.

Action: Gather stats for table ACTB_DAILY_LOG and rebuild all indexes
online. This should be done when volume is highest in this table. Then
lock the statistics.
13) SQL_ID - 7yd1csaj61tsb

INSERT INTO STTM_WITHDRAWABLE_HISTORY VALUES (:B1 ,:B2 ,:B3 ,:B4 ,:B5

,:B6 ,:B7 ,:B8 ,:B9 ,:B10 ,:B11 ,:B12 ,:B13 ,:B14 ,:B15 )

SQL statement with SQL_ID "7yd1csaj61tsb" was executed 2824 times and
had an average elapsed time of 1.5 seconds.

Action: Gather stats of table STTM_WITHDRAWABLE_HISTORY and lock the

stats.

14) SQL_ID - 1q78sym88q5d3

SELECT /*+ ORDERED */ A.* FROM ACVWS_ALL_ENTRIES_SEL A, (SELECT

BRANCH_CODE, CUST_AC_NO FROM CSTB_CUSTACSEQ B WHERE
BRANCH_CODE = :B2 AND B.SEQ_NO = :B1 ) B
WHERE NVL(A.VDBAL_UPDATE_FLAG, 'N') = 'N' AND A.BALANCE_UPD = 'U'
AND A.AUTH_STAT = 'A' AND A.CUST_GL = 'A' AND
A.AC_BRANCH = :B2 AND A.AC_BRANCH = B.BRANCH_CODE AND A.AC_NO =
B.CUST_AC_NO ORDER BY A.AC_NO, A.AC_CCY, A.VALUE_DT, A.TRN_CODE

SQL statement with SQL_ID "1q78sym88q5d3" was executed 1106 times and
had an average elapsed time of 0.94 seconds.

Observations: Index IX01_ACTB_DAILY_LOG is getting used in

the SQL execution plan.

Action: Degree needs to be changed from DEFAULT to 1.

15) SQL_ID - cd6pgs8hxjg59

SELECT /*+ ALL_ROWS SHARED (2) FULL ("AC") */
"ACCOUNT","DT","MAKER","AMAKER","OBJ" FROM
"XAPP"."CASA_MANAGEMENT_LOG" "AC" WHERE :1="ACCOUNT" AND
"OBJ"=:"SYS_B_0" AND "ACCOUNT" IS NOT NULL AND (:2=TO_DATE("DT")
OR :3=TO_DATE("DT"))

Action: Accept the below profile for better performance of the query.

execute dbms_sqltune.accept_sql_profile(task_name =>

'sql_tuning_task_cd6pgs8hxjg59', task_owner => 'SYS', replace => TRUE);

16) _lm_drm_disable , _gc_policy_time : As per Oracle support the recommended value for this
parameter should be 7 for disabling. We can see the current value set by Bank for
_lm_drm_disable , _gc_policy_time as 0 and 4. We are still seeing many wait events related
drm in multiple reports.
 Action: Please change below value as per Oracle support recommendation -
_lm_drm_disable = 7
_gc_policy_time = 0

17) enq: IV – contention: This wait event is appearing in some reports ---- lmd process is
same on both nodes and # of cpus are also same on both nodes.....

Action: Check with MOS if this is a bug or hidden parameter _ges_server_processes needs to
be set

# of sessions, memory consumption, CPU consumption, IO details were captured at

different time and screenshots are attached -

No of ACTIVE sessions in the database

4th Aug

Node 1 at 9:00 AM:

Node 1 at 11:00 AM

Node 1 at 2:20 PM
Node 1 at 3:05 PM

Node 1 at 4:00 PM

Node 1 at 5:00 PM

Node 2 at 9:00 AM
 Node 2 at 11:00 AM

Node 2 at 2:45 PM

Node 2 at 3:45 PM

Node 2 at 4:50 PM

7th Aug.

Node 1 at 9:30 AM
 Node 1 at 11:30 AM

Node 1 at 1:20 PM

Node 1 at 3:45 PM

Node 1 at 5:15 PM

Node 2 at 10:00 AM

Node 2 at 12:15 PM
Node 2 at 2:40 PM

Node 2 at 4:30 PM

8th Aug
CPU run queue is fine on Node 1 and Node2.

Node 1 server usage on Aug 4th.

Memory, CPU, Swap and IO usage looks normal at 8:30AM.

Memory, CPU, Swap and IO usage looks normal at 11:30 AM

Memory, CPU, Swap and IO usage looks normal at 02:30 PM

Node 2 server usage on Aug 4th.

Memory, CPU, Swap and IO usage looks normal at 8:10AM.

Memory, CPU, Swap and IO usage looks normal at 12:40 PM.

Memory, CPU, Swap and IO usage looks normal at 04:43 PM.

Node 1 server usage on 7th Aug.

Memory, CPU, Swap and IO usage looks normal at 08:12 AM.

Memory, CPU, Swap and IO usage looks normal at 11:54 AM.

Memory, CPU, Swap and IO usage looks normal at 5:25 PM.

Node 2 server usage on 7th Aug.

Memory, CPU, Swap and IO usage looks normal at 09:02 AM.

Memory, CPU, Swap and IO usage looks normal at 12:33 PM.

Memory, CPU, Swap and IO usage looks normal at 5:34 PM.

Node 1 server usage on 8th Aug

At 9:00 AM

At 12 PM
At 4:33 PM

Node 2 server usage on 8th Aug

At 9:20 AM

At 12:20 PM
At 4:51 PM

Top processes details and load average are captured in below screenshot. Load average is fine on
Node 1 and Node2.

Node 1 Top process on 4th Aug at 09:02 AM.

At 11:37 AM on 4th Aug

At 3:46 PM on 4th Aug

Node 1 Top process on 7th Aug.

At 9:55 AM on 7th Aug

At 1:32 PM on 7th Aug

At 5:41 PM on 7th Aug

Node 1 Top process

At 9:07 AM on 8th Aug

At 12:14 PM on 8th Aug

At 4:22 PM on 8th Aug

Node 2 Top process

At 9:48 AM on 4th Aug

At 1:25 PM on 4th Aug

At 5:02 AM on 4th Aug

At 9:35 AM on 7th Aug

At 1:44 PM on 7th Aug

At 5:21 PM on 7th Aug

At 9:29 AM on 8th Aug

At 12:35 PM on 8th Aug

At 4:43 PM on 8th Aug

IO Stat are fine on Node 1 and Node2.

IO Stats Node 1 at 10:41 AM on 4th Aug.

IO Stats Node 1 at 03:31 PM on 4th Aug.

IO Stats Node 1 at 09:23 AM on 7th Aug.

IO Stats Node 1 at 09:53 AM on 7th Aug.

IO Stats Node 1 at 05:23 PM on 7th Aug.

IO Stats Node 1 at 10:00 AM on 8th Aug.

IO Stats Node 1 at 4:30 PM on 8th Aug.

IO Stats Node 2 at 9:12AM on 4th Aug.

IO Stats Node 2 at 04:40 PM on 4th Aug.

IO Stats Node 2 at 09:01 PM on 7th Aug.

IO Stats Node 2 at 04:31 PM on 7th Aug.

IO Stats Node 2 at 05:32 PM on 7th Aug.

IO Stats Node 2 at 10:18 AM on 8th Aug.

IO Stats Node 2 at 4:18 PM on 8th Aug.