Disaster Recovery System

A knowledge transfer session

December 2008 (Updated March 2009)

Nikos Plevris
Principal Service Delivery Manager
 Agenda

• Purpose & Scope <Insert Picture Here>

• Standby Database Environment
• Compatibility and Operational Requirements
• Standby modes
• Setting up the standby database for managed recovery
• Troubleshooting
• Standby Db Configuration
• Gap Sequence before initiating Managed Recovery
• Standby Db Administration & Maintenance
• The SDR Activation/Operation procedure
• Graceful Switchover and Switchback
• Summary
• Q&A
 Purpose & Scope

• Disaster protection for Information System

• Ensure Business Continuity
• Faster recovery and lower MTTR
• Potentially no data loss (if online redo logs are applied)
• Protection against data corruption
• Supplemental reporting
• Based on:
• Standby database operation
• Failover and Switchback functions
 Standby Database Environment

• Managed standby environment

• Automatic shipment and application of archived logs from
primary
• Procedural graceful switchover and switchback
functions in effect
• Considerations on Administration Costs :
• Network connection maintenance
• Additional system resources and storage space
• Ensure mirroring of structural operations performed on
primary
• Extra loss of time due to graceful procedure
 Standby Database Environment

• Managed standby environment

• Automatic shipment and application of archived logs from
primary
• Procedural graceful switchover and switchback
functions in effect
• Considerations on Administration Costs :
• Network connection maintenance
• Additional system resources and storage space
• Ensure mirroring of structural operations performed on
primary
• Extra loss of time due to graceful procedure
 Compatibility and Operational
Requirements
• The primary db must run in ARCHIVELOG mode
• The redo logs that you apply to the standby db must
be either archived or noncurrent online redo logs
• Same version and release of the operating system
and database on the primary and standby hosts
• The standby host can use a different directory
structure
• Primary and Standby db cannot share the same
control file
• An activated standby db becomes a normal primary
database
Standby modes

Managed
Recovery
mode

Read-only
mode

Manual
Recovery
mode
 Setting up the Standby Database –
Managed Recovery1
• Backup primary database
• Connect to the primary database and create the
standby control file

SQL>ALTER DATABASE CREATE

STANDBY CONTROLFILE AS
’/oracle/dbs/stbycf.f’;
 Setting up the Standby Database –
Managed Recovery2
• Copy the backup datafiles and control file from the
primary site to the standby site
• On the primary site Create a service name for the
standby database
• tnsnames.ora standby_service_name =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)
(PORT=port_number)
(HOST=host_name))
(CONNECT_DATA=(SID=standby_sid)))
 Setting up the Standby Database –
Managed Recovery3
• Configure the listener on the standby site so that it
can receive the archived redo logs from the primary
site
• $TNS_ADMIN/network/admin/listener.ora
STANDBY_LISTENER = (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)
(PORT=port_number)
(HOST=host_name))
)
SID_LIST_STANDBY_LISTENER =
(SID_LIST=
(SID_DESC=
(SID_NAME=standby_sid_name)
(ORACLE_HOME=/oracle_home))
)
 Setting up the Standby Database –
Managed Recovery4
• Set the initialization parameters for the primary
database - add new archiving destinations
• LOG_ARCHIVE_DEST_n
• OPTIONAL or MANDATORY
• LOG_ARCHIVE_MIN_SUCCEED_DEST=m
# first local archiving destination
#
• LOG_ARCHIVE_DEST_STATE_n
LOG_ARCHIVE_DEST_1 = LOCATION=/oracle/arc/
MANDATORY’
#
• REOPEN=s # second local archiving destination
LOG_ARCHIVE_DEST_STATE_2 = ENABLE
LOG_ARCHIVE_DEST_2 = ’LOCATION=/oracle/arc2/’
#
# third standby archiving destination
LOG_ARCHIVE_DEST_3 = ’SERVICE=
standby_service_name OPTIONAL REPOEN=2’
 Setting up the Standby Database –
Managed Recovery5
• Set the initialization parameters for the standby
database
• most initialization parameters at the primary and
standby databases should be identical
• Parameters in standby init.ora
• DB_NAME = Primary db’s DB_NAME
• CONTROL_FILES <> Primary db’s DB_NAME
• DB_FILE_NAME_CONVERT Distinguishes standby’s datafiles
• LOCK_NAME_SPACE When the standby and primary databases share a host
• LOG_ARCHIVE_DEST Location of archives during manual recovery
• LOG_ARCHIVE_TRACE Audit trail of archived log progression
• LOG_FILE_NAME_CONVERT Distinguishes standby’s logfiles
• STANDBY_ARCHIVE_DEST Location of archives during managed recovery
Managed Recovery Mode
Primary Db Online redo Archived redo

Transactions
Remote archiving
Via Oracle Net

PRIMARY_HOST

SDR Standby Db

Archived redo Managed

Recovery
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
RECOVER MANAGED STANDBY DATABASE;
Read-Only mode
Primary Db Online redo Archived redo

Transactions
Remote archiving
Via Oracle Net

PRIMARY_HOST

SDR Standby Db

Archived redo Managed

Recovery
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE OPEN READ ONLY;
Queries
Failover to standby database
Primary Db

ALTER DATABASE ACTIVATE STANDBY DATABASE;

PRIMARY_HOST

Charybdis
Transactions Online redo Archived redo

Standby database
activated, becomes
primary database
 Troubleshooting the Standby Database
Configuration1
• The Standby Site Does Not Receive Logs Archived by
the Primary Database
• Immediate Action: Query V$ARCHIVE_DEST

SQL> SELECT dest_id "ID",

•At the primary site Check: 2> status "DB_status",
3> destination "Archive_dest",
tnsnames.ora, 4> error "Error"
5> FROM v$archive_dest;
LOG_ARCHIVE_DEST_n,
LOG_ARCHIVE_DEST_STATE_n
•At the standby site Check:
listener.ora,
STANDBY_ARCHIVE_DEST
 Troubleshooting the Standby Database
Configuration2
• Cannot Mount the Standby Database
• The standby instance is not started in NOMOUNT mode
• The standby control file was not created with the ALTER
DATABASE CREATE STANDBY CONTROLFILE ... Statement
• Unacceptable control file backups:
• RMAN-created backup
• An operating system-created backup
• A backup created using an ALTER DATABASE statement
without the STANDBY option
 Resolving a Gap Sequence Before
Initiating Managed Recovery1
• A gap sequence is a range of archived redo logs
needed by the standby database
• When standby db is created from an old backup
• Standby db is generated from a hot backup
• Shutdown of the Standby Database When the
Primary Database Is Open
• Primary db’s Controlfile change, ie MAXDATAFILES
• ALTER DATABASE CLEAR UNARCHIVED LOGFILES
• a RESETLOGS operation on the primary database
invalidates the standby database
 Resolving a Gap Sequence Before
Initiating Managed Recovery2
• To avoid creating gap sequences:
• Start the standby databases and listeners before starting the
primary database
• Shut down the primary database before shutting down the
standby database
• dynamically change MANDATORY to OPTIONAL before
shutting down the standby database
• Network Failure Preventing the Transfer of Logs to
the Standby Site
• To prevent set LOG_ARCHIVE_DEST_2 =
’SERVICE=standby1 MANDATORY’
• WARNING: Primary might stall as it cannot switch into an
unarchived online redo log
 Resolving a Gap Sequence Before
Initiating Managed Recovery3
• To determine if Gap Sequence Exists (Standby site)
SELECT high.thread#, "LowGap#", "HighGap#"
FROM
(
SELECT thread#, MIN(sequence#)-1 "HighGap#"
FROM
(
SELECT a.thread#, a.sequence#
FROM
(
SELECT *
FROM v$archived_log
) a,
(
SELECT thread#, MAX(next_change#)gap1
FROM v$log_history
GROUP BY thread#
) b
WHERE a.thread# = b.thread#
AND a.next_change# > gap1
)
GROUP BY thread#
) high,
(
SELECT thread#, MIN(sequence#) "LowGap#"
FROM
(
SELECT thread#, sequence#
FROM v$log_history, v$datafile
WHERE checkpoint_change# <= next_change#
AND checkpoint_change# >= first_change#
)
GROUP BY thread#
) low
WHERE low.thread# = high.thread#;
 Resolving a Gap Sequence Before
Initiating Managed Recovery4
• Transmit the Logs in the Gap Sequence to the
Standby Site
•At the primary site determine
filenames
•At the standby site review the SQL> CONNECT sys/sys_pwd@primary
SQL> SELECT name
settings for 2> FROM v$archived_log
3> WHERE sequence# in (460, 202);
STANDBY_ARCHIVE_DEST and
LOG_ARCHIVE_FORMAT to determine
archived log filenames
•Copy archives to standby site
 Resolving a Gap Sequence Before
Initiating Managed Recovery5
• Apply the Logs in the Gap Sequence to the
Standby Database

SQL> CONNECT sys/sys_pwd@standby1 AS SYSDBA

SQL> STARTUP NOMOUNT

pfile=/oracle/admin/pfile/initSTBY.ora

SQL> ALTER DATABASE MOUNT STANDBY DATABASE;

SQL> RECOVER AUTOMATIC STANDBY DATABASE;

 Resolving a Gap Sequence Before
Initiating Managed Recovery6
• Determine which archived logs have been
received by the Standby Site
• Tracing ALTER SYSTEM SET LOG_ARCHIVE_TRACE = 32
SQL> SELECT thread#, max(sequence#) AS
"LAST_APPLIED_LOG"
2> FROM v$log_history
3> GROUP BY thread#;

SQL>SELECT thread#, sequence# AS "SEQ#",

name, first_change# AS "FIRSTSCN",
next_change# AS "NEXTSCN",archived,
deleted,completion_time AS "TIME"
FROM v$archived_log;
Administration & Maintenance1
 Adding Tablespaces or Datafiles to the
Primary Database
• CREATE TABLESPACE redo adds the new filename
to the standby control file
• Manually create datafile to standby

SQL> RECOVER MANAGED STANDBY DATABASE

CANCEL;

SQL>ALTER DATABASE CREATE DATAFILE

’/private1/stby/t_db2.f’
AS ’/private1/stby/t_db2.f’;

SQL> RECOVER MANAGED STANDBY DATABASE;

 Refreshing the Standby Database
Control File

SQL> ALTER DATABASE CREATE STANDBY Transfer the standby control file and archived log files
CONTROLFILE AS ’filename’;

SQL> RECOVER MANAGED STANDBY DATABASE

CANCEL;

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP NOMOUNT;

SQL> ALTER DATABASE MOUNT STANDBY
DATABASE;
SQL> RECOVER MANAGED STANDBY DATABASE;

Primary db

Standby db
 PRIMARY_HOST

SDR

Secondary Disaster Recovery system (SDR)

 The SDR Activation and Operation
internal procedure
• Graceful shutdown of the production information
system
• Change of the configuration files on user site
• Interfaces to external systems transition
• Rolling forward to SDR system
• Switch back to production system (PRIMARY_HOST)
 Graceful Switchover and Switchback

• Avoid opening the targeted db with resetlogs

• Switchover
• From Primary (Source) -> Standby db (Target)
• Switchback
• From Standby (Source) -> Primary db (Target)
PRIMARY_HOST

Primary db Standby db

SDR
 PRIMARY_HOST
Graceful Switchover and
Switchback Prerequisites
SDR

• Primary db is shutdown normal/immediate

• No loss of any unapplied archived logs
• All archives are applied to standby db
• Standby db is shutdown normal/immediate
• Source db’s online redo logs are available
• Target db’s files, controlfile intact – Same resetlogs
version as Source db
• Loss of unrecoverable transactions (direct loads) is
acceptable
 PRIMARY_HOST
What prevents a Graceful
Switchover and Switchback
SDR

• Database undergoes a resetlogs version change:

• ALTER DATABASE OPEN RESETLOGS;
• ALTER DATABASE ACTIVATE STANDBY DATABASE;
• Above commands are required after:
• RECOVER DATABASE UNTIL [Cancel/Time/Change];
• RECOVER STANDBY DATABASE;
• RECOVER MANAGED STANDBY DATABASE;
Interfaces to external systems
transition
Production system External system Interface Disaster recovery site failover
type
ABS Aeronautical Billing system Incoming Auto TNS
AIANET AIA Intranet Outgoing Auto
ATC-PALLAS Pallas System Both Change of the /etc/hosts file Manual
BHS Baggage Handling System Outgoing Auto
COSMOTE COSMOTE Outgoing Auto
CUTE SITA-CUTE system Incoming Change of the /etc/hosts file Manual
DOAS Differential Optical Absorption Outgoing Auto
Spectroscopy
DYSI DYSI System Outgoing Auto
FB/LB FB/LB Displays Incoming Change of the /etc/hosts file Manual
FIDS / Conrac Flight Information Displays Outgoing Auto
HTML Information HTML / information desk Outgoing Auto
Desk Staff Pages
IDS ATC Tower IDS System Outgoing Auto
LCD boards/Matrix LCD Boards/Matrix LCD Outgoing Auto
MAIN UFIS AS GmbH Help Desk Both Changes to VPN (e-mail to be send to
Germany UFIS AS)
MCL Master Clock Outgoing Auto
MIS Management Information Incoming Auto TNS
System
NMS Noise Monitoring System Outgoing Auto (config file should be changed)
RMS Resource Management System Outgoing Auto
SAFCO file interface SAFCO Fueling Co Outgoing Auto
OIC file interface OIC Fueling Co Outgoing Auto
VB Pages applications Outgoing Auto (config file should be changed)
OTENET file interface OTENET internet site Outgoing
SITA SITA Messaging System Incoming Activate (tick) new service Manual
PUBLIC S.A file interface Public S.A ftp server Outgoing Auto
Collaborative Decision Ourania (AIA server) Outgoing
Making (CDM) interface
Information Desk Pages Outgoing Auto
file interface
 The Switchover1

1 4 7
Online redo
shutdown Logs shipped Mount NEW
To Standby STANDBY

PRIMARY_HOS

Charybdis

2 3 5 6
Standby Standby •Create standby
•Create controlfile
apply shutdown •Startup
controlfile
•Ship to NEW
archives NEW Primary
STANDBY

Timeline
Graceful Switchover2 steps
Primary site

SQL>ALTER SYSTEM ARCHIVELOG

CURRENT;
SQL>ALTER DATABASE BACKUP CONTROLFILE
TO TRACE NORESETLOGS;
SQL> SELECT sequence# from v$loghist;
SQL>SHUTDOWN IMMEDIATE;
•Online redo logs are shipped to Standby site -
•CREATE CONTROLFILE script sent to Standby site
Graceful Switchover3 steps
Standby site

SQL>SELECT MAX(fhrba_seq) from

x$kcvfh;`
SQL>RECOVER MANAGED STANDBY CANCEL;
SQL>SHUTDOWN IMMEDIATE;
SQL>STARTUP NOMOUNT PFILE=init.ora
Run CREATE CONTROLFILE script from primary
SQL>RECOVER DATABASE;
SQL>ALTER DATABASE OPEN;
SQL>ALTER SYSTEM ARCHIVE LOG ALL;
SQL>ALTER DATABASE CREATE STANDBY
CONTROLFILE AS ‘standby.ctl’;
CREATE CONTROLFILE script is sent to Standby site
 Graceful Switchover4 Technical aspects Machine: i686
Machine: i686 Instance name: LNX817 Primary
Instance name: STDBY Redo thread mounted by this instance: 1
Redo thread mounted by this instance: 1
Standby Oracle process number: 9
Oracle process number: 9 Unix process pid: 1960, image: [email protected] (TNS V1-V3)
Unix process pid: 1500, image: [email protected] (TNS V1-V3)
*** 2008-12-16 10:46:02.484
*** 2008-12-16 10:38:22.717 *** SESSION ID:(8.1) 2008-12-16 10:46:02.483
*** SESSION ID:(8.1) 2008-12-16 10:38:22.717 DUMP OF CONTROL FILES, Seq # 608 = 0x260
SQL>ALTER SESSION SET EVENTS
DUMP OF CONTROL FILES, Seq # 588 = 0x24c FILE HEADER:
FILE HEADER: ‘immediate trace name controlf Software vsn=135266304=0x8100000, Compatibility
Software vsn=135266304=0x8100000, Compatibility Vsn=134217728=0x8000000
level
Vsn=134217728=0x8000000
10’; Db Id=51525935=0x312392f, Db Name='LNX817'
Db Id=51525935=0x312392f, Db Name='LNX817' Control Seq=608=0x260, File size=556=0x22c
Control Seq=588=0x24c, File size=556=0x22c File Number=0, Blksiz=8192, File Type=1 CONTROL
File Number=0, Blksiz=8192, File Type=4 BACKUP CONTROL
********************************************************************
************************************************************************ DATABASE ENTRY
DATABASE ENTRY ********************************************************************
************************************************************************ (blkno = 0x1, size = 192, max = 1, in-use = 1, last-recid= 0)
(blkno = 0x1, size = 192, max = 1, in-use = 1, last-recid= 0) DF Version: creation=0x8100000 compatible=0x8000000, Date
DF Version: creation=0x8100000 compatible=0x8000000, Date 06/02/2008 06/02/2008 19:36:28
19:36:28 DB Name "LNX817"
DB Name "LNX817" Database flags = 0x00004001
Database flags = 0x00005447 Controlfile Creation Timestamp 06/02/2008 19:36:29
Controlfile Creation Timestamp 06/02/2008 19:36:29 Incmplt recovery scn: 0x0000.00000000
Incmplt recovery scn: 0x0000.0071e3f2 Resetlogs scn: 0x0000.005cefe8 Resetlogs Timestamp 06/02/2008
Resetlogs scn: 0x0000.005cefe8 Resetlogs Timestamp 06/02/2008 19:38:50 19:38:50
Prior resetlogs scn: 0x0000.00000001 Prior resetlogs Timestamp Prior resetlogs scn: 0x0000.00000001 Prior resetlogs Timestamp
07/19/2005 18:18:23 07/19/2005 18:18:23
Redo Version: creation=0x8100000 compatable=0x8100000 Redo Version: creation=0x8100000 compatable=0x8100000
#Data files = 10, #Online files = 10 #Data files = 10, #Online files = 10
Database checkpoint: Thread=1 scn: 0x0000.0071e3b5 Database checkpoint: Thread=1 scn: 0x0000.0071e40d
Threads: #Enabled=1, #Open=1, Head=1, Tail=1 Threads: #Enabled=1, #Open=1, Head=1, Tail=1
enabled threads: 01000000 00000000 00000000 00000000 00000000 enabled threads: 01000000 00000000 00000000 00000000 00000000
00000000 00000000
00000000 00000000 00000000 00000000
Max log members = 2, Max data members = 1 Max log members = 2, Max data members = 1
Arch list: Head=1, Tail=1, Force scn: 0x0000.0071e3bd Arch list: Head=4, Tail=4, Force scn: 0x0000.0071e3f0
Controlfile Checkpointed at scn: 0x0000.0071e3f2 12/15/2008 17:07:36 Controlfile Checkpointed at scn: 0x0000.0071e40d 12/16/2008
thread:1 rba:(0x3b.2.0) 10:45:41
enabled threads: 01000000 00000000 00000000 00000000 00000000 thread:0 rba:(0x0.0.0)
00000000 enabled threads: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
00000000 00000000
 Scenario: Standby db has been
activated
• ALTER DATABASE ACTIVATE STANDBY DATABASE;
• SHUTDOWN IMMEDIATE
• STARTUP
• Prerequisites:
• $ORACLE_HOME/dbs/Init.ora files exist in both sites
• All expected init.ora parameters have been set
 Scenario: Fallback to Primary db –
Steps1
• Hot backup of current primary db (ex-standby db):
• ALTER TABLESPACE <> BEGIN BACKUP;
• host cp <source_datafile> <dest_datafile>
• ...
• ...
• ALTER TABLESPACE <> END BACKUP;
• Backup current redo:
• ALTER SYSTEM ARCHIVE LOG ALL;
• ALTER SYSTEM ARCHIVE CURRENT LOG;
 Scenario: Fallback to Primary db –
Steps2
• Copy archive logs to primary db
• Create controlfile creation script:
• ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
• Shutdown immediate current primary db (ex-standby
db)
• Startup primary db and recreate controlfile...
• STARTUP NOMOUNT
• CREATE CONTROLFILE REUSE DATABASE "<>"
RESETLOGS ARCHIVELOG
•…
•…
 Scenario: Fallback to Primary db –
Steps3
• Apply the archivelogs and CANCEL upon the non-
existent archivelog:
• RECOVER DATABASE USING BACKUP CONTROLFILE
UNTIL CANCEL
• ALTER DATABASE OPEN RESETLOGS
 Scenario: Recreate Standby db –
Steps1
• Hot backup of primary db:
• ALTER TABLESPACE <> BEGIN BACKUP;
• host cp <source_datafile> <dest_datafile>
• ...
• ...
• ALTER TABLESPACE <> END BACKUP;
• ALTER DATABASE CREATE STANDBY
CONTROLFILE AS '<>';
• Move standby controlfile to standby site (where
init.ora control_files expects)
 Scenario: Recreate Standby db –
Steps2
• ALTER SYSTEM ARCHIVE LOG ALL;
• ALTER SYSTEM ARCHIVE CURRENT LOG;
• Copy archive logs to standby db
• Check if any sequence gap exists
• Apply the archivelogs and CANCEL upon the non-
existent archivelog:
• RECOVER STANDBY DATABASE;
• Set standby db to managed mode:
• RECOVER MANAGED STANDBY DATABASE;
 Hot backup datafiles

AL
TE Archived redo logs
R
DA
TA
B AS
E
CR
EA
Primary Db TE StandBy Db
ST
AN
DB
Y
CO
NT
Init.ora RO Init.ora
LF
IL
E
AS
…

CREATE CONTROLFILE Control_file Control_file

script
 Summary

Be thankful for problems. If they were less difficult, someone

else with less ability might have your job.
James A. Lovell.
Q&
A