12c Data guard Switchover Best Practices using SQLPLUS

I. Pre-Switchover check
Verify that the target standby is ready for switchover.
Checks ORL status on target standby.
What to do?
Verify Primary and Standby tempfiles match
NOTE : For multiple Standby environment make sure all the standby in SYNC with primary.
Check if log_archive_Dest(remote redo transport) set on Standby side
If switchover_status(v$database) shows UNRESOLVABLE GAP in RAC/non RAC
environment
II SWITCHOVER
Enable tracing on both primary and standby to diagnose in case of any issue.
Tail Alert Logs (optional) on all instances
Switchover the standby database to a primary
Open the new primary database
Restart the new standby
III. Post - Switchover Check
Set Trace to Prior Value
Verify that the new primary shipping logs to standby and standby is applying.

For the purposes of this document, the following fictitious environment is used as an
example to describe the procedure:

Primary database: boston

Standby database: chicago

**********

I. Pre-Switchover check

Verify that the target standby is ready for switchover.

NOTE : In CDB environment switchover happen at CDB level only.

Verify that the target standby is synchronous with primary and Redo shipping working fine from
primary.

SQL>alter database switchover to <target standby db_unique_name> verify;

The above command will verify following things,

a. Verifies that the release version is minimum 12.1

b. Primary REDO shipping.

c. Standby MRP recovery running and its synchronous with primary, else it will through below
error in SQLPROMPT and alert log as well.

For example,

SQL>alter database switchover to chicago verify;

ORA-16470: Redo Apply is not running on switchover target

------Primary alert log------

SQL>alter database switchover to chicago verify;

ORA-16470 signaled during: alter database switchover to chicago verify...

Checks ORL status on target standby.

If it is dirty then it will though the warning messages like

SQL> alter database switchover to chicago verify;

ERROR at line 1:

ORA-16475: succeeded with warnings, check alert log for more details

-----Primary Alert log-----------

SWITCHOVER VERIFY WARNING: switchover target has dirty online redo logfiles that require
clearing.

It takes time to clear online redo logfiles. This may slow down switchover process.

What to do?

Make sure log_file_name_convert set properly on target standby.

SQL>show parameter log_file_name_convert;

NOTE: If not set set the value for example if directory structure changing only in
db_unique_name. for example,

Primary ORL location is /oradata/boston/<path>

Standby logfile location is /oradata/chicago/<logfiles>

Set the parameter like,

alter system set LOG_FILE_NAME_CONVERT=’boston’,’chicago’ scope=spfile;

NOTE: Restart the standby and start the MRP. When you restart standby and start MRP then it
clears all the ORLs at standby provided log_file_name_convert set properly.

If Standby is synchronous with primary, we will get below messages,

SQL> alter database switchover to chicago verify;

Database altered.

--------- Primary Alert log -------

SWITCHOVER VERIFY: Send VERIFY request to switchover target CHICAGO

SWITCHOVER VERIFY COMPLETE

Completed: alter database switchover to chicago verify

Verify Primary and Standby tempfiles match

Tempfiles created after standby creation will not be propagated to standby. Check the TEMP files
using below query and add new TEMP files at standby is required.

SQL> col name for a45

SQL> select ts#,name,ts#,status from v$tempfile;

NOTE : For multiple Standby environment make sure all the standby in SYNC with primary.

Check if log_archive_Dest(remote redo transport) set on Standby side

SWITCHOVER VERIFY command returns ORA-16475 when log_archive_dest_n is not set on

Standby as follows.

SQL> ALTER DATABASE SWITCHOVER TO chicago VERIFY;

ALTER DATABASE SWITCHOVER TO chicago VERIFY
*
ERROR at line 1:
ORA-16475: succeeded with warnings, check alert log for more details

In alert.log, following messages appear.

SWITCHOVER VERIFY: Send VERIFY request to switchover target chicago

SWITCHOVER VERIFY COMPLETE
SWITCHOVER VERIFY WARNING: switchover target has no standby database definedin
LOG_ARCHIVE_DEST_n parameter. If the switchover target is converted to
a primary database, the new primary database will not be protected.ORA-16475 signalled during:
ALTER DATABASE SWITCHOVER TO chicago VERIFY...

What to do? set log_archive_dest_state_n on Standby.

If switchover_status(v$database) shows UNRESOLVABLE GAP in RAC/non RAC

environment

1. Check if any closed thread exists and disable (if any)

SQL>SELECT thread#, instance, status FROM v$thread;

Disable the CLOSED thread using,

SQL> ALTER DATABASE DISABLE THREAD <n>;

2. Check any LAD(log_archive_destination) pointing to invalid DESTINATION

SQL> select
status,DEST_ID,TYPE,ERROR,GAP_STATUS,SYNCHRONIZED,SYNCHRONIZATION_STATUS,RECOV
ERY_MODE from V$ARCHIVE_DEST_STATUS where STatus <> 'INACTIVE';
SQL>show parameter log_Archive_dest

II SWITCHOVER

Enable tracing on both primary and standby to diagnose in case of any issue.

SQL>alter system set log_archive_trace=8191 sid=’*’;

Tail Alert Logs (optional) on all instances

@primary and standby,

SQL>show parameter background_dump_dest

$tail -600f background_dump_dest/alert*

* NOTE : In RAC environment no need to shutdown all instances apart from one instanc.
Switchover(SQLPLUS) comand itself shutdown all the nodes.

Switchover the standby database to a primary

Primary-Boston,

SQL>alter database switchover to chicago;

Database altered.
.

Sample output from primary (BOSTON) and standby (CHICAGO),

--------------Primary Alert log "alert_boston.log"--------------

Fri Aug 23 11:05:23 2013

ALTER SYSTEM SET log_archive_trace=8191 SCOPE=BOTH;

alter database switchover to chicago

Fri Aug 23 11:05:43 2013

Starting switchover [Process ID: 3340]

Fri Aug 23 11:05:43 2013

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY [Process Id: 3340]

(boston)

Fri Aug 23 11:05:44 2013

Active, synchronized Physical Standby switchover target has been identified

Preventing updates and queries at the Primary

Generating and shipping final logs to target standby

Switchover End-Of-Redo Log thread 1 sequence 11 has been fixed <---------Sequence 11 is

End-Of-Redo

Switchover: Primary highest seen SCN set to 0x0.0x229306

ARCH: Noswitch archival of thread 1, sequence 11

Switchover: Primary controlfile converted to standby controlfile succesfully.

Switchover: Complete - Database shutdown required

Sending request (convert to primary database) to switchover target CHICAGO

OCISessionBegin with PasswordVerifier succeeded

Switchover complete. Database shutdown required

USER (ospid: 3340): terminating the instance

Fri Aug 23 11:05:51 2013

Instance terminated by USER, pid = 3340

Completed: alter database switchover to chicago

-------------------alert log "alert_chicago.log"------------------

Fri Aug 23 11:05:47 2013

SWITCHOVER: received request 'ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY' from

primary database.

Fri Aug 23 11:05:47 2013

ALTER DATABASE SWITCHOVER TO PRIMARY (chicago)

Maximum wait for role transition is 15 minutes.

Standby became primary SCN: 2265860

Switchover: Complete - Database mounted as primary

SWITCHOVER: completed request from primary database.

NOTE: At standby side we wait only for 15 minutes for the standby to synchronous with primary.

Open the new primary database

SQL>alter database open;

Restart the new standby

SQL>shut abort

SQL>startup

SQL>alter database recover managed standby database disconnect;

The scenario worked, but after switchover completed, the Database Role in the OCR is not
changed automatically.
We need use [srvctl modify database] to change Database Role in the OCR according to real
database role.
for new primary:

#sqlplus / as sysdba
SQL> select DB_UNIQUE_NAME,DATABASE_ROLE from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
------------------------------
chicago PRIMARY

#srvctl config database -db chicago

...
Database Role: PHYSICAL_STANDBY

#srvctl modify database -db chicago -role PRIMARY

for new standby:

#sqlplus / as sysdba
SQL> select DB_UNIQUE_NAME,DATABASE_ROLE from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
------------------------------
boston PHYSICAL STANDBY

#srvctl config database -db boston

...
Database Role: PRIMARY

#srvctl modify database -db boston -role PHYSICAL_STANDBY

III. Post - Switchover Check

Set Trace to Prior Value

SQL> alter system set log_archive_trace=0;

System altered.

Verify that the new primary shipping logs to standby and standby is applying.

@primary,

SQL>alter system switch logfile;

SQL>select dest_id,error,status from v$archive_dest where dest_id=<your

remote log_archive_dest_<n>>;

SQL>select max(sequence#),thread# from v$log_history group by thread#;

If remote log_Archive_destination is 2 i.e log_archive_dest_2.

SQL>select max(sequence#) from v$archived_log where applied='YES' and
dest_id=2;

@standby,

SQL>select thread#,sequence#,process,status from gv$managed_standby;

SQL>select max(sequence#),thread# from v$archived_log group by thread#;

NOTE : For Oracle version 12.2 use v$dataguard_process instead of v$managed_standby which is
deprecated in Oracle 12.2

SQL> select name,role,instance,thread#,sequence#,action from

gv$dataguard_process;

Ref: Doc ID 1578787.1