Hacmp generic manual takeover

IBM GLOBAL SERVICES

HACMP generic manual takeover

Version 2.0

-1-
Hacmp generic manual takeover

Document Management

Changes history
Version Date Changes
1.0 05/10/2005 Initial version – DRAFT (not tested)
1.1 06/10/2005 Initial Version with following updates :
 Available for all hacmp versions
 Mac address management option
 Refresh –s inetd
2.0 18/10/2005 Tested and validated Version (change 401100 )

Reviewers list
Name Job responsability
Alain Lentini Admin. AIX HACMP

Approvers list
Name Job responsability Date

-2-
Hacmp generic manual takeover

1. Introduction.........................................................................................................................4
2. Resource releasing actions..................................................................................................4
2.1. Applications stop........................................................................................................4
2.2. Unmount the crossmount NFS filesystems on secondary node..................................4
2.3. Unmount all the NFS local filesystems on primary node...........................................4
2.4. Unexport all NFS filesystems on primary node..........................................................4
2.5. Unmount all the application filesystems.....................................................................5
2.6. Varyoffvg all the volume groups................................................................................5
2.7. LVM synchronisation.................................................................................................5
2.8. Save route table on primary HACMP server..............................................................5
2.9. Change the IP address of the service interface...........................................................5
2.10. Save the MAC address if needed (for IP replacement only).................................6
2.11. Delete the MAC address on the server if needed (for IP replacement only)........6
2.12. Reboot if necessary.................................................................................................6
3. Process to acquire the resources on backup server.............................................................6
3.1. process to acquire MAC address (if needed)..............................................................7
3.2. define routes saved from primary server....................................................................7
3.3. Acquiring IP address ( for IP replacement only)........................................................7
3.4. Acquiring volume groups...........................................................................................7
3.5. Filesystems mount......................................................................................................8
3.6. NFS crossmount..........................................................................................................8
3.7. Normal NFS mount.....................................................................................................8
3.8. Arp table......................................................................................................................8
3.9. Hostname takeover if required....................................................................................8
3.10. Applications start....................................................................................................8

-3-
Hacmp generic manual takeover

1. Introduction

This document describes how to do a takeover on servers with HACMP or HACMP/ES

installed and configured on both nodes of a cluster but not running. For example after forced
stop
Some commands depend on the HACMP version. Commands are different with HACMP
versions < 4.5.

Warning
The commands used in this document are available only if the following fileset is installed:
 HACMP or HACMP/ES version > 4.4: cluster.es.server.utils
 HACMP version <= 4.4: cluster.base.server.utils

2. Resource releasing actions

2.1. Applications stop

If the primary server is still up, stop all the applications before the takeover.

First you can try to stop them with the HACMP application server stop scripts.
To find the scripts use the following command:
/usr/sbin/cluster/utilities/cllsserv –h or /usr/es/sbin/cluster/utilities/cllsserv –h

If it doesn’t work, try to use the application commands directly.

Have a look in the HACMP application server stop scripts to see which commands to use and
try running them manually.

Check which processes are still alive and kill the rest as needed.
Determine the application filesystems with the following commands
/usr/sbin/cluster/sbin/cl_lsvg
For HACMP versions < 4.5: /usr/sbin/cluster/utilities/cllsvg
lsvg –l <vg>
For each application filesystem:
fuser –cuxk <filesystem> or /usr/sbin/cluster/events/utils/cl_NFSkill or kill command

2.2. Unmount the crossmount NFS filesystems on secondary node

On the backup node, unmount all the NFS filesystems mounted from the primary node only.
Use the df | grep <primary node> command to see the crossmount NFS list.
Use the following command to try to force them to unmount if it doesn’t work:
fuser –cuxk <filesystem> or /usr/sbin/cluster/events/utils/cl_nfskill
( this command just kills the processes…doesn’t unmount the filesystem )

2.3. Unmount all the NFS local filesystems on primary node

2.4. Unexport all NFS filesystems on primary node

-4-
Hacmp generic manual takeover

exportfs –a –u

2.5. Unmount all the application filesystems

All the filesystems defined in the volume groups of the primary cluster node must be
unmounted.
Use lsvg –l on all the volume groups shown by the following command:
/usr/sbin/cluster/sbin/cl_lsvg
For HACMP versions < 4.5: /usr/sbin/cluster/utilities/cllsvg
and unmount all of them on the primary node.

2.6. Varyoffvg all the volume groups

varyoffvg <vg> on all the HACMP volume groups (see lsvg command described before).

2.7. LVM synchronisation

If you think the LVM is not synchronised between the HACMP nodes, you’d better do it
before activating the volume groups.
To do it
 Be sure all the shared volume groups are varyoff on the primary node
 Save all the disk PVIDs on the primary node: lspv > /tmp/lspv.save
 Save all the volume groups major numbers on the backup node before exportvg
 Exportvg all the shared volume groups on the backup node
 Find a disk in the shared volume group on the backup node with the same PVID as on
the primary node
 Importvg the volume group using the same major number with this disk:
importvg –V<major nb> -y <vg> <disk>
 Update quorum and auto varyon parameters: chvg -Qn –an <vg>
 Leave the volume groups varyon as you will be using them for the takeover

2.8. Save route table on primary HACMP server

Netstat -r

2.9. Change the IP address of the service interface

If the cluster use IP Adress Takeover via IP replacement

Put the Boot IP address of the server instead of the service IP address.
To find the service IP address and the standby interface use the following commands:
/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif
For HACMP versions < 4.5 use netstat –i to find the devices

Warning: To change the interface IP address you must be able to use another network
interface for your working shell (for example use the HMC terminal console on a P5).

To change the IP address use smitty tcpip -> Further Configuration

After the change, check all the network addresses of the primary node in /etc/hosts.

-5-
Hacmp generic manual takeover

If the cluster use IP Adress Takeover via IP aliasing

Move manually the IP service address to one of the network interface of the backup server.

2.10. Save the MAC address if needed (for IP replacement only)

If there are MAC addresses configured on the cluster (HWAT) you must use them as well
after the takeover (Delete on primary node and add on the backup one).

To check if HACMP MAC addresses are defined:

/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif

Use netstat –i to see the active MAC address on the service interfaces.
Compare the active MAC address of the service interfaces with the value in the cllsif
“Hardware Address” field.
If the HACMP MAC address is not being used, there must be the real network address active
on each network interface: lscfg –vl entX | grep “Network Address”; in this case you don’t
have to takeover the MAC address on the backup node.

2.11. Delete the MAC address on the server if needed (for IP

replacement only)

On the service interface, delete the alternate MAC Addresses if there are ones configured

To find the service interface use the following commands:

/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif
For HACMP versions < 4.5 use netstat –i to find the devices

Detach the IP interface before deleting the MAC address with the following command:
ifconfig <interface> detach

To delete the MAC address use the following command:

chdev -l 'interface' -a alt_addr='''' ,

or use smitty dev->Communication->Network Type-> Adapter->

Change / Show Characteristics of an Ethernet Adapter->Select Adapter->
Empty the following line and validate:
ALTERNATE ETHERNET address []

Enable the interface again with the following command:

ifconfig <interface> up
refresh –s inetd

2.12. Reboot if necessary

If one of the stop steps doesn’t work you can just reboot the primary node to release all the
resources and check all the steps again.

3. Process to acquire the resources on backup server

-6-
Hacmp generic manual takeover

3.1. process to acquire MAC address (if needed)

If needed, put the service MAC address of the primary node on the standby interface of the
backup node

Warning: To change the MAC address you must be able to use another network interface for
your working shell (for example use the HMC terminal console on a P5).

To find the standby interface and the service MAC address of the primary node use the
following commands:
/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif
For HACMP versions < 4.5 use netstat –i to find the interfaces

Detach the standby IP interface before setting the MAC address with the following
command:
ifconfig <interface> detach

Set the MAC address with the following command

chdev -l 'ent1' -a alt_addr='mac address'
where interface = standby interface on the backup server, for example, entX for Ethernet.

Activate the interface again with the following command:

ifconfig <interface> up
refresh –s inetd

3.2. define routes saved from primary server

3.3. Acquiring IP address ( for IP replacement only)

Replace the standby IP address of the backup node with the service IP address of the primary
node.

To find the service IP address and the standby interface use the following commands:
/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif
For HACMP versions < 4.5 use netstat –i to find the interfaces

To change the IP address use smitty tcpip -> Further Configuration

After the change, check the line of the IP service address in /etc/hosts file.

3.4. Acquiring volume groups

To find the list of the volume groups use the following command:
/usr/sbin/cluster/sbin/cl_lsvg
For HACMP versions < 4.5: /usr/sbin/cluster/utilities/cllsvg

Run varyonvg on all the server volume groups (the ones not yet varied on) : varyonvg <vg>

-7-
Hacmp generic manual takeover

3.5. Filesystems mount

To find the list of all the filesystems to mount run lsvg –l on all the volume groups of the
HACMP server.
Be careful with the mount order of the filesystems.
You can use mount –t <type> if a type is defined for the filesystems.
Check that all filesystems are mounted with lsvg –l <vg> | grep closed.

3.6. NFS crossmount

exportfs –a –f /usr/sbin/cluster/etc/exports

Mount all the /local filesystems (crossmount) exported on their NFS mountpoint. See
/usr/sbin/cluster/etc/exports file for the list of /local filesystems (crossmount).
e.g. sapprddb01:/local/sapmnt/P01 on /sapmnt/P01

If the file /usr/sbin/cluster/etc/exports is not available or not complete, just export all the /local
filesystems to all the network interfaces.
To find all the crossmounts you can use the following command:
df |grep -i local

To check the mountpoint of the NFS crossmounts you can use the smitty hacmp menu of the
resource group parameters.

3.7. Normal NFS mount

Mount all the NFS filesystems defined on the host.

Check the list with the following command: lsnfsmnt

3.8. Arp table

If you have any problem with the NFS crossmounts or normal NFS you can check the arp
table for all the IP addresses of the server with the following command:
arp –a
check that the service IP address is mapped on the HACMP logical MAC address defined in
the following menu:
/usr/sbin/cluster/utilities/cllsif or /usr/es/sbin/cluster/utilities/cllsif

3.9. Hostname takeover if required

Put the primary server hostname on the backup one

Hostname <host>
Uname –S <host>

3.10. Applications start

To find the start scripts use the following command:

/usr/sbin/cluster/utilities/cllsserv –h or /usr/es/sbin/cluster/utilities/cllsserv –h

-8-