clmgr: A Technical Reference

PowerHA SystemMirror 7.1 introduces a robust CLI utility

Figure 1
Figure 2
In its more than 20-year history, PowerHA SystemMirror (HACMP) has lacked a consistent, well-
documented, thoroughly supported command-line interface (CLI). Configuration and control of
PowerHA SystemMirror has been left to SMIT. Attempts were made at graphical utilities, which were
mostly wrappers for the SMIT functions to perform the task at hand. For the command line-oriented
administrator, looking to automate operations, including configuration tasks, the best that could be
done was to find the command in SMIT and hope that all of the right environment variables were set.
In 2006, HACMP 5.4 introduced a utility called clvt that provided a glimmer of hope. Developed as a
tool Smart Assists use to perform configuration tasks at the command level, its capabilities were
limited to those the Smart Assists needed. In later releases, very little was done to update the clvt
utility beyond that needed for the Smart Assists.
This lack of CLI changes with the PowerHA SystemMirror 7.1 plugin for IBM System Director. Now all
of the PowerHA SystemMirror tasks need to be executable using a command-level utility. A substantial
amount of development effort was expended to polish the interface and complete the capabilities.
Although a huge improvement has been made in 7.1, be sure to check the man page for the task you
wish to perform because not all tasks are there, yet. And of course, be sure to test thoroughly prior to
final deployment.
The cluster used to develop this paper, which will be outlined later, is running on a pair of LPARs, one
on each of two Power 570 systems. The storage and network are provided via virtual I/O (VIO), that
is, the network traffic is bridged and the disks are virtual SCSI (vSCSI). One Ethernet adapter will be
used for the cluster. The repository disk has been identified. See the Figures 1 and 2 for more details.
Figure 1. Cluster Infrastructure
Figure 2. Resource Group Configuration
Basics
There is online help. The man page for clmgr as of the writing of this paper is available using man
clvt. If you know what you want to do, but you arent quite sure what options are available or their
format, simply type as much of the command as you know, followed by a -h. This will produce only
the help relevant to the typed command. For example, if you want to activate a node, you can
typeclmgr on node -h, which produces the following output:
# clmgr start node -h
# Available options for "clvt start node":
start_node
<_NAME>
WHEN
MANAGE
BROADCAST
CLINFO
FORCE
FIX
TIMEOUT
The output above provides all of the options for starting a node.
The format of the command is: clmgr <action> <class> [<name>] [<attributes>]. The
action is what you want to do to the class (like node or resource group). The name parameter
identifies the specific class object. The most common actions are:
add, query, delete
online, offline
move
sync
The most common classes are:
cluster
node
resource_group
service_ip
application_controller
Specify the -a parameter when using the query action to filter out only the desired attribute. Specify
the -v parameter when using the query action to produce verbose output. This will also produce
verbose output for all of the objects in the specified class. Verbose output is the default when
specifying a named object on the query; that is, no -v is necessary.
Step-by-Step Guide
Next, Ill outline the tasks that can be executed using only clmgr. Any of these tasks can be executed
individually, but if you do them in the suggested order, you create, test and back up a simple, two-
node cluster.
1. Build a two-node PowerHA SystemMirror 7.1 cluster
2. Define persistent addresses for each node
3. Define two service addresses for each resource group
4. Define two application controllers, one for each resource group
5. Define two resource groups, with resources, each having a different home node
6. Synchronize the cluster definition
7. Start cluster services on each node
8. Verify resource groups are online as expected
9. Move a resource group from one node to another
10. Stop cluster services, moving resource groups to the other node
11. Restart cluster services on down node
12. Move resource groups back to home nodes
13. Generate a snapshot of this validated cluster configuration
Step 1Build the cluster. The command is:
# clmgr add cluster pha_cluster repository=hdisk2 nodes=node1,node2
You should see the following output:
# cltopinfo
Cluster Name: pha_cluster
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
Repository Disk: hdisk2
Cluster IP Address:
There are 2 node(s) and 1 network(s) defined
NODE node1:
Network net_ether_01
node1 10.6.51.121
NODE node2:
Network net_ether_01
node2 10.6.51.221

No resource groups defined
Step 2Define persistent addresses for each node. The command:
# clmgr add persistent_ip 192.168.3.1 network=net_ether_01 node=node1
# clmgr a pe 192.168.3.2 network=net_ether_01 node=node2
The result:
# clmgr -v q pe
NAME="node1admin"
IPADDR="192.168.3.1"
NODE="node1"
SITE=""
NETWORK="net_ether_01"
INTERFACE=""
NETTYPE="ether"
TYPE="persistent"
ATTR="public"
GLOBALNAME=""
HADDR=""

NAME="node2admin"
IPADDR="192.168.3.2"
NODE="node2"
SITE=""
NETWORK="net_ether_01"
INTERFACE=""
NETTYPE="ether"
TYPE="persistent"
ATTR="public"
GLOBALNAME=""
HADDR=""
Step 3Define two service addresses, one for each nodes resource group. The command:
# clmgr add service_ip appAsvc network=net_ether_01 \ netmask=255.255.255.0
# clmgr a se appBsvc network=net_ether_01 netmask=255.255.255.0
The result:
# clmgr -v q se
NAME="appAsvc"

IPADDR="192.168.3.10"
NODE=""
SITE="ignore"
NETWORK="net_ether_01"
INTERFACE=""
NETTYPE="ether"
TYPE="service"
ATTR="public"
GLOBALNAME=""
HWADDR=""

NAME="appBsvc"
IPADDR="192.168.3.20"
NODE=""
SITE="ignore"
NETWORK="net_ether_01"
INTERFACE=""
NETTYPE="ether"
TYPE="service"
ATTR="public"
GLOBALNAME=""
HWADDR=""
Step 4Define two application controllers, one for each nodes resource group. The command:
# clmgr add application_controller appActrl \ startscript=/lab/config/startA
stopscript=/lab/config/stopA
# clmgr a ac appBctrl startscript=/lab/config/startB \
stopscript=/lab/config/stopB
The result:
# clmgr -v q ac
NAME="appActrl"
MONITORS=""
STARTSCRIPT="/lab/config/startA"
STOPSCRIPT="/lab/config/stopA"

NAME="appBctrl"
MONITORS=""
STARTSCRIPT="/lab/config/startB"
STOPSCRIPT="/lab/config/stopB"
Step 5Define two resource groups with resources, one for each node. The command is as follows.
Note: The policy parameters must be uppercase.
# clmgr add resource_group appAgroup nodes=node1,node2 startup=OHN \
fallover=FNPN fallback=NFB service_label=appAsvc \ applications=appActrl
volume_group=vgA fs_before_ipaddr=true

# clmgr add resource_group appBgroup nodes=node2,node1 startup=OHN \
fallover=FNPN fallback=NFB service_label=appBsvc \ applications=appBctrl
volume_group=vgB
The result:
# cltopinfo
Cluster Name: pha_cluster
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
Repository Disk: hdisk2
Cluster IP Address:
There are 2 node(s) and 1 network(s) defined
NODE node1:
Network net_ether_01
appBsvc 192.168.3.20
appAsvc 192.168.3.10
node1 10.6.51.121
NODE node2:
Network net_ether_01
appBsvc 192.168.3.20
appAsvc 192.168.3.10
node2 10.6.51.221

Resource Group appAgroup
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes node1 node2
Service IP Label appAsvc

Resource Group appBgroup
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes node2 node1
Service IP Label appBsvc
Step 6Synchronize the cluster definitions with the command:
# clmgr sync cluster
The result (some output was pruned):
# clmgr sync cluster
Verification to be performed on the following:
Cluster Topology
Cluster Resources

Retrieving data from available cluster nodes. This could take a few minutes.

Start data collection on node node1
Start data collection on node node2

Verifying Cluster Topology...

Completed 10 percent of the verification checks

node1 net_ether_01
node2 net_ether_01

Verifying Cluster Resources...

Completed 40 percent of the verification checks

appActrl appAgroup
appBctrl appBgroup
Completed 100 percent of the verification checks

Remember to redo automatic error notification if configuration has changed.

Verification has completed normally.

Committing any changes, as required, to all available nodes...
Adding any necessary PowerHA SystemMirror for AIX entries to /etc/inittab and
/etc/rc.net for IP Address Takeover on node node1.
Adding any necessary PowerHA SystemMirror for AIX entries to /etc/inittab and
/etc/rc.net for IP Address Takeover on node node2.

Verification has completed normally.

WARNING: Multiple communication interfaces are recommended for networks that
use IP aliasing in order to prevent the communication interface from
becoming a single point of failure. There are fewer than the recommended
number of communication interfaces defined on the following node(s) for
the given network(s):

Node: Network:
---------------------------------- ------------------------------
WARNING: Network option "nonlocsrcroute" is set to 0 and will be set to 1 on
during HACMP startup on the following nodes:

node1
node2

WARNING: Application monitors are required for detecting application failures
in order for HACMP to recover from them. Application monitors are started
by HACMP when the resource group in which they participate is activated.
The following application(s), shown with their associated resource group,
do not have an application monitor configured:
Step 7Check state of cluster services. The command:
# clmgr -a state query cluster
# clmgr -cv -a name,state query node
The result:
# clmgr -a state query cluster
STATE="OFFLINE"
# clmgr -cv -a name,state query node
# NAME:STATE
node1:OFFLINE
node2:OFFLINE
Step 8Start cluster services on both nodes with the command:
# clmgr start cluster
The result:
# clmgr -a state q cluster
STATE="STABLE"
# clmgr -cv -a name,state,raw_state q node
# NAME:STATE:RAW_STATE
node1:NORMAL:ST_STABLE
node2:NORMAL:ST_STABLE
Step 9Check the status of the resource groups with the command:
# clmgr -cv -a name,state,current_node q rg
The result:
# clmgr -cv -a name,state,current_node q rg
# NAME:STATE:CURRENT_NODE
appAgroup:ONLINE:node1
appBgroup:ONLINE:node2
At this point, checking the resources in the resource group must be done using AIX commands.
Although theres state information for many of the cluster objects via clmgr, it doesnt provide state
information for objects like the service address or application-controller scripts. Lets move on to some
testing.
Step 10Move the appAgroup resource group to node2 with the command:
# clmgr mv rg appAgroup node=node2
The result:
# clmgr mv rg appAgroup node=node2
Attempting to move resource group appAgroup to node node2.

Waiting for the cluster to process the resource group movement request....

Waiting for the cluster to stabilize....

Broadcast message from root@node1 (tty) at 16:29:04 ...

appA stopping

...........

Resource group movement successful.
Resource group appAgroup is online on node node2.

Cluster Name: pha_cluster

Resource Group Name: appAgroup
Primary instance(s):
The following node temporarily has the highest priority for this instance:
node2, user-requested rg_move performed on Wed Dec 1 16:28:56 2010

Node Group State
---------------------------- ---------------
node1 OFFLINE
node2 ONLINE

Resource Group Name: appBgroup
Node Group State
---------------------------- ---------------
node2 ONLINE
node1 OFFLINE
Step 11Stop cluster services on node2 with the Move Resource Groups option. The command:
# clmgr stop node node2 manage=move
The result:
# clmgr -cv -a name,state,raw_state q node
# NAME:STATE:RAW_STATE
node1:NORMAL:ST_STABLE
node2:OFFLINE:ST_INIT

# clmgr -cv -a name,state,current_node q rg
# NAME:STATE:CURRENT_NODE
appAgroup:ONLINE:node1
appBgroup:ONLINE:node1
Step 12Restart cluster services on node2. The command (using online alias instead of start) is:
# clmgr on node node2
The result:
# clmgr -cv -a name,state,raw_state q node
# NAME:STATE:RAW_STATE
node1:NORMAL:ST_STABLE
node2:NORMAL:ST_STABLE

# clmgr -cv -a name,state,current_node q rg
# NAME:STATE:CURRENT_NODE
appAgroup:ONLINE:node1
appBgroup:ONLINE:node1
Step 13Move the appBgroup resource group back to node2 using the command:
# clmgr mv rg appBgroup node=node2
The result:
# clmgr -cv -a name,state,current_node q rg
# NAME:STATE:CURRENT_NODE
appAgroup:ONLINE:node1
appBgroup:ONLINE:node2
Step 14To make a snapshot of all of this good work, use the command:
# clmgr mk sn clmgr_snap description="Snap of clmgr example cluster"
The result:
# clmgr mk sn clmgr_snap description="Snap of clmgr example cluster"

clsnapshot: Creating file /usr/es/sbin/cluster/snapshots/clmgr_snap.odm.

clsnapshot: Creating file /usr/es/sbin/cluster/snapshots/clmgr_snap.info.

clsnapshot: Executing clsnapshotinfo command on node: node1...

clsnapshot: Executing clsnapshotinfo command on node: node2...

clsnapshot: Succeeded creating Cluster Snapshot: clmgr_snap
Usability features
As shown above, many aliases or shortcuts are available in the utility. For a complete list, visit the
man page. Misspell something or get a parameter wrong, and clmgr tells you clearly what is wrong.
For example:
# clmgr add service_ip appAsvc network=net_ether01 netmask=255.255.255.0

ERROR: failed to create "appAsvc".

Network net_ether01 cannot be found in the configuration.
Do you want to do a query but youre not sure what object names were used in PowerHA
SystemMirror for a given class? Provide any name and consult the very user-friendly error message.
# clmgr q rg what_group

ERROR: "what_group" does not appear to exist!

Available Resource Groups:

appAgroup
appBgroup
Notice, the problem value is clearly shown, and all possible correct values are also shown. To find all
off the resource groups that have the option to mount filesystems before the IP addresses are
acquired:
# clmgr q rg fs_before_ipaddr=true
appAgroup
Troubleshooting and Logging
Troubleshooting a utility like clmgr is arguably not all that important. If it doesnt work, switch to SMIT
and get the task done. But, if you are trying to get clmgr to do something and you are planning to use
it extensively in the future, it does produce some very good log information. The logging is done to
/var/hacmp/log/clutils.log (or wherever clutils.log is kept on your system). Further, the clutils.log
contains the output of the action, so if the output has overrun the scroll buffer, its all safely in the log.
An excerpt of the log is shown here. This is the result of starting cluster services on node2.
CLMGR STARTED (9153:10354698:5177392): Wed Dec 1 16:56:43 CET 2010
CLMGR USER (9153:10354698:5177392): ::root:system
CLMGR COMMAND (9153:7471358:10354698): clmgr -T 9153 online node node2
CLMGR ACTUAL (9153:7471358:10354698): start_node node2
CLMGR RETURN (9153:7471358:10354698): 0
CLMGR STDERR -- BEGIN (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010
CLMGR STDERR -- END (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010
CLMGR ENDED (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010
Some useful bits of information are contained in the output, including the user executing the
command, the command entered by the user (CLMGR COMMAND) and the actual command executed.
he return code is also displayed, which may be helpful if a failure had occurred.
This is just the beginning of troubleshooting. If the problem is with the executed command, not with
clmgr, then other logs or the AIX configuration will have to be consulted. For example, the failure to
configure the repository disk during the initial synchronization will not be solved by looking in
clutils.log. That will most likely be solved using syslog.
clmgr is a Viable CLI
While earlier versions of clvt were limited, the updated clmgr is ready for mainstream use. As for the
common administrative tasks in PowerHA SystemMirror, I think Ive shown that this utility covers
them all. As you can see, theres now a viable CLI utility for PowerHA SystemMirror administration.
Many more features are available. For example, to execute a DARE operation, you could change an
attribute (like adding a volume group to a resource group) using the Modify action and then
synchronize as shown earlier. Additionally, since this is the utility that the IBM IBM System Director
plugin for PowerHA SystemMirror uses, youre assured the underlying infrastructure for the plugin
functions well.
While this might not be the right tool for every administrative situation nor will it replace SMIT for
PowerHA SystemMirror administration, tasks like starting or stopping services or performing a
synchronization can be simplified requiring only a few keystrokes.
Disclaimer: All of the commands shown in this paper worked as shown during the development of the
paper. Neither IBM nor the author make any guarantee that the same or similar results will be seen in
other cluster environments. Use this document as a reference and guide to using the clmgr utility.