General Parallel File System

Version 4 Release 1.0.5

Documentation Update:
GPFS Version 4 Release 1.0.5 for Linux
on System z
(Applied to GPFS Version 4 Release
1.0.4 Information Units)


General Parallel File System
Version 4 Release 1.0.5

Documentation Update:
GPFS Version 4 Release 1.0.5 for Linux
on System z
(Applied to GPFS Version 4 Release
1.0.4 Information Units)


ii GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Contents
Figures . . . . . . . . . . . . . . . v Assign mount command options . . . . . . 29
Enabling quotas . . . . . . . . . . . . 29
Tables . . . . . . . . . . . . . . . vii Enable DMAPI . . . . . . . . . . . . 30
Verifying disk usage . . . . . . . . . . 30
Changing the file system format to the latest
December 2014 . . . . . . . . . . . 1 level . . . . . . . . . . . . . . . . 31
Enabling file system features . . . . . . . 31
Planning for GPFS . . . . . . . . . . 3 Specifying whether the df command will report
Hardware requirements. . . . . . . . . . . 3 numbers based on quotas for the fileset . . . . 31
Software requirements . . . . . . . . . . . 4 Specifying the maximum number of files that can
GPFS product structure . . . . . . . . . . . 4 be created . . . . . . . . . . . . . . 31
Recoverability considerations . . . . . . . . . 5 Controlling the order in which file systems are
Node failure . . . . . . . . . . . . . 5 mounted . . . . . . . . . . . . . . 32
Network Shared Disk server and disk failure . . 8 A sample file system creation . . . . . . . 32
Reduced recovery time using Persistent Reserve 10
GPFS cluster creation considerations . . . . . . 11 Installing GPFS on Linux nodes . . . . 35
GPFS node adapter interface names . . . . . 11 Preparing the environment on Linux nodes. . . . 35
Nodes in your GPFS cluster . . . . . . . . 12 Installing the GPFS software on Linux nodes . . . 36
GPFS cluster configuration servers. . . . . . 13 Accepting the electronic license agreement on
Remote shell command . . . . . . . . . 13 Linux nodes . . . . . . . . . . . . . 36
Remote file copy command . . . . . . . . 14 Extracting the GPFS software on Linux nodes . . 36
Cluster name . . . . . . . . . . . . . 14 Extracting GPFS patches (update SLES or Red
User ID domain for the cluster . . . . . . . 14 Hat Enterprise Linux RPMs or Debian Linux
Starting GPFS automatically . . . . . . . . 14 packages) . . . . . . . . . . . . . . 38
Cluster configuration file . . . . . . . . . 15 Installing the GPFS man pages on Linux nodes 38
GPFS license designation . . . . . . . . . . 15 Installing the GPFS software packages on Linux
Disk considerations . . . . . . . . . . . . 15 nodes . . . . . . . . . . . . . . . 38
Network Shared Disk (NSD) creation Verifying the GPFS installation on SLES and Red
considerations . . . . . . . . . . . . 17 Hat Enterprise Linux nodes . . . . . . . . 39
NSD server considerations . . . . . . . . 18 Verifying the GPFS installation on Debian Linux
File system descriptor quorum . . . . . . . 19 nodes . . . . . . . . . . . . . . . 40
Preparing direct access storage devices (DASD) Building the GPFS portability layer on Linux nodes 40
for NSDs . . . . . . . . . . . . . . 19 Using the automatic configuration tool to build
File system creation considerations . . . . . . 20 the GPFS portability layer on Linux nodes . . . 40
Device name of the file system . . . . . . . 24 For Linux on System z: Changing the kernel settings 41
NFS V4 deny-write open lock . . . . . . . 24
Disks for your file system . . . . . . . . 24
Accessibility features for GPFS . . . . 43
Deciding how the file system is mounted . . . 24
Accessibility features . . . . . . . . . . . 43
Block size . . . . . . . . . . . . . . 25
Keyboard navigation . . . . . . . . . . . 43
atime values . . . . . . . . . . . . . 26
IBM and accessibility . . . . . . . . . . . 43
mtime values . . . . . . . . . . . . . 26
Block allocation map . . . . . . . . . . 26
File system authorization . . . . . . . . . 27 Notices . . . . . . . . . . . . . . 45
Strict replication . . . . . . . . . . . . 27 Trademarks . . . . . . . . . . . . . . 46
Internal log file . . . . . . . . . . . . 27
File system replication parameters . . . . . . 27 Glossary . . . . . . . . . . . . . . 49
Number of nodes mounting the file system. . . 28
Windows drive letter . . . . . . . . . . 28 Index . . . . . . . . . . . . . . . 55
Mountpoint directory . . . . . . . . . . 29

© Copyright IBM Corp. 2014 iii

iv GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Figures
1. GPFS configuration using node quorum . . . 6 4. Configuration using GPFS replication for
2. GPFS configuration using node quorum with improved availability. . . . . . . . . . 9
tiebreaker disks . . . . . . . . . . . 8
3. An example of a highly available SAN
configuration for a GPFS file system . . . . 9

© Copyright IBM Corp. 2014 v

vi GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Tables
1. GPFS cluster creation options . . . . . . 11 2. File system creation options . . . . . . . 21

© Copyright IBM Corp. 2014 vii

viii GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
December 2014
This Documentation Update provides information about IBM® GPFS™ Version 4 Release 1.0.5 for Linux on
System z (applied to GPFS Version 4 Release 1.0.4 information units).

Within this documentation update, a vertical line (|) to the left of the text indicates technical changes or
additions made to the previous edition of the information.

Changes and additions to GPFS: Concepts, Planning, and Installation Guide

(GA76-0441-01)

Specifically, this Documentation Update provides new and changed information for the following chapters
of the previously-published GPFS Version 4 Release 1.0.4 Concepts, Planning, and Installation Guide:
v “Planning for GPFS” on page 3
v “Installing GPFS on Linux nodes” on page 35

© Copyright IBM Corp. 2014 1

2 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Planning for GPFS
Although you can modify your GPFS configuration after it has been set, a little consideration before
installation and initial setup will reward you with a more efficient and immediately useful file system.

During configuration, GPFS requires you to specify several operational parameters that reflect your
hardware resources and operating environment. During file system creation, you can specify parameters
that are based on the expected size of the files or you can let the default values take effect.

Planning for GPFS includes:

v “Hardware requirements”
v “Software requirements” on page 4
v “GPFS product structure” on page 4
v “Recoverability considerations” on page 5
v “GPFS cluster creation considerations” on page 11
v “GPFS license designation” on page 15
v “Disk considerations” on page 15
v “File system creation considerations” on page 20

Hardware requirements
You can validate that your hardware meets GPFS requirements by taking the steps outlined in this topic.
1. Consult the GPFS FAQ in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/
SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html) for the latest list of:
v Supported server hardware
v Tested disk configurations
v Maximum cluster size
2. Provide enough disks to contain the file system. Disks can be:
v SAN-attached to each node in the cluster
v Attached to one or more NSD servers
v A mixture of directly-attached disks and disks that are attached to NSD servers
Refer to “Network Shared Disk (NSD) creation considerations” on page 17 for additional information.
3. When doing network-based NSD I/O, GPFS passes a large amount of data between its daemons. For
NSD server-to-client traffic, it is suggested that you configure a dedicated high-speed network solely
for GPFS communications when the following are true:
v There are NSD disks configured with servers providing remote disk capability.
v Multiple GPFS clusters are sharing data using NSD network I/O.
Refer to the GPFS: Advanced Administration Guide for additional information.

GPFS communications require static IP addresses for each GPFS node. IP address takeover operations that
transfer the address to another computer are not allowed for the GPFS network. Other IP addresses
within the same computer that are not used by GPFS can participate in IP takeover. To provide
availability or additional performance, GPFS can use virtual IP addresses created by aggregating several
network adapters using techniques such as EtherChannel or channel bonding.

© Copyright IBM Corp. 2014 3

Software requirements
GPFS planning includes understanding the latest software requirements.
v GPFS is supported on AIX®, Linux, and Windows.
v For existing GPFS 3.5 clusters, OpenSSL is required for remote cluster access.
v Kernel development files and compiler utilities are required to build the GPFS portability layer on
Linux nodes. The required RPMs or packages for each supported Linux distribution are:
SLES Linux RPMs
kernel-default-devel, cpp, gcc, gcc-c++, binutils
RedHat Linux RPMs
kernel-devel, cpp, gcc, gcc-c++, binutils
Debian Linux Packages
linux-headers, cpp, gcc, gcc-c++, binutils
v To use active file management (AFM), the following is required:
nfs-utils
v To use CNFS, the following are required:
ethtool
nfs-utils
rpcbind
v To use the mmchconfig numaMemoryInterleave parameter, the following is required:
numactl

Consult the GPFS FAQ in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/SSFKCN/

com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html) for the latest list of:
v AIX environments
v Linux distributions
v Linux kernel versions
v Windows environments

GPFS product structure

GPFS has three different editions based on functional levels. Each edition can be licensed by GPFS Client,
GPFS Server, and FPO.
GPFS Express Edition
Available on AIX, Linux, and Windows. Provides the base GPFS functions.
GPFS Standard Edition
Available on AIX, Linux, and Windows. Provides extended features in addition to the base GPFS
functions provided in the GPFS Express Edition. On AIX and Linux, the extended features
include Information Lifecycle Management (ILM), Active File Management (AFM), and Clustered
NFS (CNFS). On Windows, the extended features include limited Information Lifecycle
Management (ILM).
GPFS Advanced Edition
Available on AIX and Linux. Provides high-level data protection using the GPFS cryptographic
subsystem. For additional information, refer to the Encryption topic in the GPFS: Advanced
Administration Guide.

Note: All nodes in a cluster must have the same edition installed.

Consult the GPFS FAQ in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/SSFKCN/

com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html) for the latest features included in each edition.

4 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Recoverability considerations
Sound file system planning requires several decisions about recoverability. After you make these
decisions, GPFS parameters enable you to create a highly-available file system with rapid recovery from
failures.
v At the disk level, consider preparing disks for use with your file system by specifying failure groups
that are associated with each disk. With this configuration, information is not vulnerable to a single
point of failure. See “Network Shared Disk (NSD) creation considerations” on page 17.
v At the file system level, consider replication through the metadata and data replication parameters. See
“File system replication parameters” on page 27.

Additionally, GPFS provides several layers of protection against failures of various types:
1. “Node failure”
2. “Network Shared Disk server and disk failure” on page 8
3. “Reduced recovery time using Persistent Reserve” on page 10

Node failure
In the event of a node failure, GPFS:
v Prevents the continuation of I/O from the failing node
v Replays the file system metadata log for the failed node

GPFS prevents the continuation of I/O from a failing node through a GPFS-specific fencing mechanism
called disk leasing. When a node has access to file systems, it obtains disk leases that allow it to submit
I/O. However, when a node fails, that node cannot obtain or renew a disk lease. When GPFS selects
another node to perform recovery for the failing node, it first waits until the disk lease for the failing
node expires. This allows for the completion of previously submitted I/O and provides for a consistent
file system metadata log. Waiting for the disk lease to expire also avoids data corruption in the
subsequent recovery step.

To reduce the amount of time it takes for disk leases to expire, you can use Persistent Reserve (SCSI-3
protocol). If Persistent Reserve (configuration parameter: usePersistentReserve) is enabled, GPFS prevents
the continuation of I/O from a failing node by fencing the failed node using a feature of the disk
subsystem called Persistent Reserve. Persistent Reserve allows the failing node to recover faster because
GPFS does not need to wait for the disk lease on the failing node to expire. For additional information,
refer to “Reduced recovery time using Persistent Reserve” on page 10. For further information about
recovery from node failure, see the GPFS: Problem Determination Guide.

File system recovery from node failure should not be noticeable to applications running on other nodes.
The only noticeable effect may be a delay in accessing objects that were being modified on the failing
node when it failed. Recovery involves rebuilding metadata structures which may have been under
modification at the time of the failure. If the failing node is acting as the file system manager when it
fails, the delay will be longer and proportional to the level of activity on the file system at the time of
failure. In this case, the failover file system management task happens automatically to a surviving node.

Quorum
GPFS uses a cluster mechanism called quorum to maintain data consistency in the event of a node
failure.

Quorum operates on the principle of majority rule. This means that a majority of the nodes in the cluster
must be successfully communicating before any node can mount and access a file system. This keeps any
nodes that are cut off from the cluster (for example, by a network failure) from writing data to the file
system.

Planning for GPFS 5

During node failure situations, quorum needs to be maintained in order for the cluster to remain online.
If quorum is not maintained due to node failure, GPFS unmounts local file systems on the remaining
nodes and attempts to reestablish quorum, at which point file system recovery occurs. For this reason it is
important that the set of quorum nodes be carefully considered (refer to “Selecting quorum nodes” on
page 8 for additional information).

GPFS quorum must be maintained within the cluster for GPFS to remain active. If the quorum semantics
are broken, GPFS performs recovery in an attempt to achieve quorum again. GPFS can use one of two
methods for determining quorum:
v Node quorum
v Node quorum with tiebreaker disks

Node quorum: Node quorum is the default quorum algorithm for GPFS. With node quorum:
v Quorum is defined as one plus half of the explicitly defined quorum nodes in the GPFS cluster.
v There are no default quorum nodes; you must specify which nodes have this role.

For example, in Figure 1, there are three quorum nodes. In this configuration, GPFS remains active as
long as there are two quorum nodes available.

Quorum nodes

/gpfs1

GPFS cluster

Figure 1. GPFS configuration using node quorum

Node quorum with tiebreaker disks: When running on small GPFS clusters, you might want to have
the cluster remain online with only one surviving node. To achieve this, you need to add a tiebreaker
disk to the quorum configuration. Node quorum with tiebreaker disks allows you to run with as little as
one quorum node available as long as you have access to a majority of the quorum disks (refer to
Figure 2 on page 8). Enabling node quorum with tiebreaker disks starts by designating one or more nodes
as quorum nodes. Then one to three disks are defined as tiebreaker disks using the tiebreakerDisks
parameter on the mmchconfig command. You can designate any disk to be a tiebreaker.

When utilizing node quorum with tiebreaker disks, there are specific rules for cluster nodes and for
tiebreaker disks.

Cluster node rules:

1. There is a maximum of eight quorum nodes.
2. All quorum nodes need to have access to all of the tiebreaker disks.
3. When using the traditional server-based (non-CCR) configuration repository, you should include the
primary and secondary cluster configuration servers as quorum nodes.
4. You may have an unlimited number of non-quorum nodes.

6 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
5. If a network connection fails, which causes the loss of quorum, and quorum is maintained by
tiebreaker disks, the following rationale is used to re-establish quorum. If a group has the cluster
manager, it is the “survivor”. The cluster manager can give up its role if it communicates with fewer
than the minimum number of quorum nodes as defined by the minQuorumNodes configuration
parameter. In this case, other groups with the minimum number of quorum nodes (if they exist) can
choose a new cluster manager.

Changing quorum semantics:

When using the cluster configuration repository (CCR) to store configuration files, the total number of
quorum nodes is limited to eight, regardless of quorum semantics, but the use of tiebreaker disks can be
enabled or disabled at any time by issuing an mmchconfig tiebreakerDisks command. The change will
take effect immediately, and it is not necessary to shut down GPFS when making this change.

When using the traditional server-based (non-CCR) configuration repository, it is possible to define more
than eight quorum nodes, but only when no tiebreaker disks are defined:
1. To configure more than eight quorum nodes under the server-based (non-CCR) configuration
repository, you must disable node quorum with tiebreaker disks and restart the GPFS daemon. To
disable node quorum with tiebreaker disks:
a. Issue the mmshutdown -a command to shut down GPFS on all nodes.
b. Change quorum semantics by issuing mmchconfig tiebreakerdisks=no.
c. Add additional quorum nodes.
d. Issue the mmstartup -a command to restart GPFS on all nodes.
2. If you remove quorum nodes and the new configuration has less than eight quorum nodes, you can
change the configuration to node quorum with tiebreaker disks. To enable quorum with tiebreaker
disks:
a. Issue the mmshutdown -a command to shut down GPFS on all nodes.
b. Delete the appropriate quorum nodes or run mmchnode --nonquorum to drop them to a client.
c. Change quorum semantics by issuing the mmchconfig tiebreakerdisks="diskList" command.
v The diskList contains the names of the tiebreaker disks.
v The list contains the NSD names of the disks, preferably one or three disks, separated by a
semicolon (;) and enclosed by quotes.
d. Issue the mmstartup -a command to restart GPFS on all nodes.

Tiebreaker disk rules:

v You can have one, two, or three tiebreaker disks. However, you should use an odd number of
tiebreaker disks.
v Among the quorum node groups that appear after an interconnect failure, only those having access to
a majority of tiebreaker disks can be candidates to be the survivor group.
v Tiebreaker disks must be connected to all quorum nodes.

In Figure 2 on page 8 GPFS remains active with the minimum of a single available quorum node and two
available tiebreaker disks.

Planning for GPFS 7

 Quorum nodes

3 NSDs defined as
tiebreaker disks
/gpfs1

GPFS cluster

Figure 2. GPFS configuration using node quorum with tiebreaker disks

When a quorum node detects loss of network connectivity, but before GPFS runs the algorithm that
decides if the node will remain in the cluster, the tiebreakerCheck event is triggered. This event is
generated only in configurations that use quorum nodes with tiebreaker disks. It is also triggered on the
cluster manager periodically by a challenge-response thread to verify that the node can still continue as
cluster manager.

Selecting quorum nodes

To configure a system with efficient quorum nodes, follow these rules:
v Select nodes that are likely to remain active
– If a node is likely to be rebooted or require maintenance, do not select that node as a quorum node.
v Select nodes that have different failure points such as:
– Nodes located in different racks
– Nodes connected to different power panels
v You should select nodes that GPFS administrative and serving functions rely on such as:
– Primary configuration servers
– Secondary configuration servers
– Network Shared Disk servers
v Select an odd number of nodes as quorum nodes
– The suggested maximum is seven quorum nodes.
v Having a large number of quorum nodes may increase the time required for startup and failure
recovery.
– Having more than seven quorum nodes does not guarantee higher availability.

Network Shared Disk server and disk failure

The three most common reasons why data becomes unavailable are:
v Disk failure
v Disk server failure with no redundancy
v Failure of a path to the disk

In the event of a disk failure in which GPFS can no longer read or write to the disk, GPFS will
discontinue use of the disk until it returns to an available state. You can guard against loss of data
availability from disk failure by:

8 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
v Utilizing hardware data protection as provided by a Redundant Array of Independent Disks (RAID)
device (see Figure 3)

SAN switch 1 SAN switch 2

/gpfs1
Storage with dual
RAID controllers

Figure 3. An example of a highly available SAN configuration for a GPFS file system

v Utilizing the GPFS data and metadata replication features (see Increased data availability) along with
the designation of failure groups (see “Network Shared Disk (NSD) creation considerations” on page
17 and Figure 4)

SAN switch 1 SAN switch 2

/gpfs1

GPFS replication keeps a copy of

data on each storage controller

Figure 4. Configuration using GPFS replication for improved availability

It is suggested that you consider RAID as the first level of redundancy for your data and add GPFS
replication if you desire additional protection.

In the event of an NSD server failure in which a GPFS client can no longer contact the node that provides
remote access to a disk, GPFS discontinues the use of the disk. You can guard against loss of an NSD
server availability by using common disk connectivity on multiple NSD server nodes and specifying
multiple Network Shared Disk servers for each common disk.

Note: In the event that a path to a disk fails, GPFS reports a disk failure and marks the disk down. To
bring the disk back online, first follow the directions supplied by your storage vendor to determine and
repair the failure.

Guarding against loss of data availability due to path failure

You can guard against loss of data availability from failure of a path to a disk by doing the following:
v Creating multiple NSD servers for each disk
Planning for GPFS 9
 As GPFS determines the available connections to disks in the file system, it is recommended that you
always define more than one NSD server for each disk. GPFS allows you to define up to eight NSD
servers for each NSD. In a SAN configuration where NSD servers have also been defined, if the
physical connection is broken, GPFS dynamically switches to the next available NSD server (as defined
on the server list) and continues to provide data. When GPFS discovers that the path has been
repaired, it moves back to local disk access. This is the default behavior, which can be changed by
designating file system mount options. For example, if you never want a node to use the NSD server
path to a disk, even if the local path fails, you can set the -o useNSDserver mount option to never.
You can set the mount option using the mmchfs, mmmount, mmremotefs, and mount commands.

Important: In Linux on System z®, it is mandatory to have multiple paths to one SCSI disk (LUN) to
avoid single path of failure. The coalescing of the paths to one disk is done by the kernel (via the
device-mapper component). As soon as the paths are coalesced, a new logical, multipathed device is
created, which is used for any further (administration) task. (The single paths can no longer be used.)

The multipath device interface name depends on the distribution and is configurable:
SUSE /dev/mapper/Unique_WW_Identifier
For example: /dev/mapper/36005076303ffc56200000000000010cc
Red Hat
/dev/mapper/mpath*

To obtain information about a multipathed device, use the multipath tool as shown in the following
example:
# multipath -ll

The system displays output similar to this:

36005076303ffc56200000000000010cc dm-0 IBM,2107900
[size=5.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ 1:0:0:0 sdb 8:16 [active][ready]
\_ 0:0:0:0 sda 8:0 [active][ready]
v Using an I/O driver that provides multiple paths to the disks for failover purposes
Failover is a path-management algorithm that improves the reliability and availability of a device
because the system automatically detects when one I/O path fails and reroutes I/O through an
alternate path.

Reduced recovery time using Persistent Reserve

Persistent Reserve (PR) provides a mechanism for reducing recovery times from node failures. To enable
PR and to obtain recovery performance improvements, your cluster requires a specific environment:
v All disks must be PR-capable.
v On AIX, all disks must be hdisks. Starting with 3.5.0.16, it is also possible to use a logical volume as a
descOnly disk without disabling the use of Persistent Reserve. See the GPFS FAQ in IBM Knowledge
Center (www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/
gpfsclustersfaq.html).
On Linux, typically all disks must be generic (/dev/sd*) or DM-MP (/dev/dm-*) disks.
However, for Linux on System z, multipath device names are required for SCSI disks, and the names
are dependent on the distribution and are configurable. For details, see “Guarding against loss of data
availability due to path failure” on page 9.
v If the disks have defined NSD servers, all NSD server nodes must be running the same operating
system (AIX or Linux).
v If the disks are SAN-attached to all nodes, all nodes in the cluster must be running the same operating
system (AIX or Linux).

10 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
You must explicitly enable PR using the usePersistentReserve option of the mmchconfig command. If
you set usePersistentReserve=yes, GPFS attempts to set up PR on all of the PR capable disks. All
subsequent NSDs are created with PR enabled if they are PR capable. However, PR is only supported in
the home cluster. Therefore, access to PR-enabled disks from another cluster must be through an NSD
server that is in the home cluster and not directly to the disk (for example, through a SAN).

GPFS cluster creation considerations

A GPFS cluster is created using the mmcrcluster command.

Table 1 details the cluster creation options, how to change the options, and the default values for each
option.
Table 1. GPFS cluster creation options
Options Command to change the option Default value
“Nodes in your GPFS cluster” on page 12 mmaddnode
None
mmdelnode
Node designation: manager or client, see
mmchnode client
“Nodes in your GPFS cluster” on page 12
Node designation: quorum or nonquorum,
see “Nodes in your GPFS cluster” on page mmchnode nonquorum
12
Primary cluster configuration server, see
“GPFS cluster configuration servers” on mmchcluster None
page 13
Secondary cluster configuration server, see
“GPFS cluster configuration servers” on mmchcluster None
page 13
“Remote shell command” on page 13 mmchcluster /usr/bin/rsh
“Remote file copy command” on page 14 mmchcluster /usr/bin/rcp
“Cluster name” on page 14 The node name of the primary
mmchcluster
GPFS cluster configuration server
GPFS administration adapter port name, Same as the GPFS
see “GPFS node adapter interface names” mmchnode communications adapter port
name
GPFS communications adapter port name,
mmchnode None
see “GPFS node adapter interface names”
“User ID domain for the cluster” on page
mmchconfig The name of the GPFS cluster
14
“Starting GPFS automatically” on page 14 mmchconfig no
“Cluster configuration file” on page 15 Not applicable None

GPFS node adapter interface names

An adapter interface name refers to the hostname or IP address that GPFS uses to communicate with a
node. Specifically, the hostname or IP address identifies the communications adapter over which the
GPFS daemons or GPFS administration commands communicate. The administrator can specify two node
adapter interface names for each node in the cluster:
GPFS node name
Specifies the name of the node adapter interface to be used by the GPFS daemons for internode
communication.

Planning for GPFS 11

GPFS admin node name
Specifies the name of the node adapter interface to be used by GPFS administration commands
when communicating between nodes. If not specified, the GPFS administration commands use
the same node adapter interface used by the GPFS daemons.

These names can be specified by means of the node descriptors passed to the mmaddnode or
mmcrcluster command and can later be changed with the mmchnode command.

If multiple adapters are available on a node, this information can be communicated to GPFS by means of
the subnets parameter on the mmchconfig command.

Nodes in your GPFS cluster

When you create your GPFS cluster, you must provide a file containing a list of node descriptors, for
each node to be included in the cluster. The node descriptors can be included in the command line, or
they can be contained in a separate node descriptor file with one node definition per line. Each descriptor
must be specified in this form:
NodeName:NodeDesignations:AdminNodeName

NodeName is a required parameter. NodeDesignations and AdminNodeName are optional parameters.

NodeName
The host name or IP address of the node for GPFS daemon-to-daemon communication.
The host name or IP address that is used for a node must refer to the communication adapter over
which the GPFS daemons communicate. Alias names are not allowed. You can specify an IP address
at NSD creation, but it will be converted to a host name that must match the GPFS node name. You
can specify a node using any of these forms:
v Short hostname (for example, h135n01)
v Long hostname (for example, h135n01.frf.ibm.com)
v IP address (for example, 7.111.12.102)
Whichever form you specify, the other two forms must be defined correctly in DNS or the hosts file.
NodeDesignations
An optional, "-" separated list of node roles.
v manager | client – Indicates whether a node is part of the node pool from which file system
managers and token managers can be selected. The default is client, which means do not include
the node in the pool of manager nodes. For detailed information on the manager node functions,
see The file system manager.
In general, it is a good idea to define more than one node as a manager node. How many nodes
you designate as manager depends on the workload and the number of GPFS server licenses you
have. If you are running large parallel jobs, you may need more manager nodes than in a
four-node cluster supporting a Web application. As a guide, in a large system there should be a
different file system manager node for each GPFS file system.
v quorum | nonquorum – This designation specifies whether or not the node should be included in
the pool of nodes from which quorum is derived. The default is nonquorum. You need to
designate at least one node as a quorum node. It is recommended that you designate at least the
primary and secondary cluster configuration servers and NSD servers as quorum nodes.
How many quorum nodes you designate depends upon whether you use node quorum or node
quorum with tiebreaker disks. See “Quorum” on page 5.
AdminNodeName
Specifies an optional field that consists of a node name to be used by the administration commands
to communicate between nodes.
If AdminNodeName is not specified, the NodeName value is used.

12 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Follow these rules when adding nodes to your GPFS cluster:
v While a node may mount file systems from multiple clusters, the node itself may only reside in a
single cluster. Nodes are added to a cluster using the mmcrcluster or mmaddnode command.
v The nodes must be available when they are added to a cluster. If any of the nodes listed are not
available when the command is issued, a message listing those nodes is displayed. You must correct
the problem on each node and then issue the mmaddnode command to add those nodes.
v Designate at least one but not more than seven nodes as quorum nodes. When not using tiebreaker
disks, you can designate more quorum nodes, but it is recommended to use fewer than eight if
possible. It is recommended that you designate the cluster configuration servers as quorum nodes.
How many quorum nodes altogether you will have depends on whether you intend to use the node
quorum with tiebreaker algorithm or the regular node based quorum algorithm. For more details, see
“Quorum” on page 5.

GPFS cluster configuration servers

You must designate one of the nodes in your GPFS cluster as the primary GPFS cluster configuration
server, where GPFS configuration information is maintained. It is strongly suggested that you also specify
a secondary GPFS cluster configuration server.

If you do not specify a secondary cluster configuration server:

1. If your primary server fails, the GPFS cluster configuration data files are inaccessible, which results in
the failure of any GPFS administration commands that are issued. Similarly, when the GPFS daemon
starts up, at least one of the two GPFS cluster configuration servers must be accessible. See Cluster
configuration data files.
2. If the primary server fails, you can use the mmchcluster command with the -p option to designate
another node as the primary cluster configuration server. Similarly, you can use the mmchcluster
command with the -s option to define a secondary cluster configuration server. See the GPFS:
Administration and Programming Reference for more information about the mmchcluster command.

Remote shell command

GPFS commands need to be able to communicate across all nodes in the cluster. To achieve this, the GPFS
commands use the remote shell command that you specify on the mmcrcluster command or the
mmchcluster command.

The default remote shell command is rsh. You can designate the use of a different remote shell command
by specifying its fully-qualified path name on the mmcrcluster command or the mmchcluster command.
The remote shell command must adhere to the same syntax as rsh, but it can implement an alternate
authentication mechanism.

Clusters that include both UNIX and Windows nodes must use ssh for the remote shell command. For
more information, see Installing and configuring OpenSSH.

Clusters that only include Windows nodes may use the mmwinrsh utility that comes with GPFS. The
fully-qualified path name is /usr/lpp/mmfs/bin/mmwinrsh. For more information about configuring
Windows GPFS clusters, see the topic that discusses the mmwinservctl command in the GPFS:
Administration and Programming Reference.

By default, you can issue GPFS administration commands from any node in the cluster. Optionally, you
can choose a subset of the nodes that are capable of running administrative commands. In either case, the
nodes that you plan to use for administering GPFS must be able to run remote shell commands on any
other node in the cluster as user root without the use of a password and without producing any
extraneous messages.

For additional information, see the topic that discusses requirements for administering a GPFS file system
in the GPFS: Administration and Programming Reference.

Planning for GPFS 13

Remote file copy command
The GPFS commands must maintain a number of configuration files across all nodes in the cluster. To
achieve this, the GPFS commands use the remote file copy command that you specify on the mmcrcluster
command or the mmchcluster command.

The default remote file copy program is rcp. You can designate the use of a different remote file copy
command by specifying its fully-qualified path name on the mmcrcluster command or the mmchcluster
command. The remote file copy command must adhere to the same syntax as rcp, but it can implement
an alternate authentication mechanism. Many clusters use scp instead of rcp, as rcp cannot be used in a
cluster that contains Windows Server nodes.

Clusters that include both UNIX and Windows nodes must use scp for the remote copy command. For
more information, see Installing and configuring OpenSSH.

Clusters that only include Windows nodes may use the mmwinrcp utility that comes with GPFS. The
fully-qualified path name is /usr/lpp/mmfs/bin/mmwinrcp. For more information about configuring
Windows GPFS clusters, see the topic that discusses the mmwinservctl command in the GPFS:
Administration and Programming Reference.

The nodes that you plan to use for administering GPFS must be able to copy files using the remote file
copy command to and from any other node in the cluster without the use of a password and without
producing any extraneous messages.

For additional information, see “Requirements for administering a GPFS file system” in the GPFS:
Administration and Programming Reference.

Cluster name
Provide a name for the cluster by issuing the -C option on the mmcrcluster command. If the
user-provided name contains dots, it is assumed to be a fully qualified domain name. Otherwise, to make
the cluster name unique in a multiple cluster environment, GPFS appends the domain name of the
primary cluster configuration server. If the -C option is not specified, the cluster name defaults to the
hostname of the primary cluster configuration server. The name of the cluster may be changed at a later
time by issuing the -C option on the mmchcluster command.

The cluster name is applicable when GPFS file systems are mounted by nodes belonging to other GPFS
clusters. See the mmauth and the mmremotecluster commands.

User ID domain for the cluster

The user ID domain for a cluster when accessing a file system remotely. This option is further explained
in the GPFS: Advanced Administration Guide and the IBM white paper entitled UID Mapping for GPFS in a
Multi-cluster Environment in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/SSFKCN/
com.ibm.cluster.gpfs.doc/gpfs_uid/uid_gpfs.html).

Starting GPFS automatically

You can specify whether to start the GPFS daemon automatically on a node when it is started.

Whether or not GPFS automatically starts is determined using the autoload parameter of the
mmchconfig command. The default is not to automatically start GPFS on all nodes. You may change this
by specifying autoload=yes using the mmchconfig command. This eliminates the need to start GPFS by
issuing the mmstartup command when a node is booted.

The autoload parameter can be set the same or differently for each node in the cluster. For example, it
may be useful to set autoload=no on a node that is undergoing maintenance since operating system
upgrades and other software can often require multiple reboots to be completed.

14 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Cluster configuration file
GPFS provides default configuration values, so a cluster configuration file is not required to create a
cluster.

This optional file can be useful if you already know the correct parameter values based on previous
testing or if you are restoring a cluster and have a backup copy of configuration values that apply to
most systems. Typically, however, this option is not used at cluster creation time, and configuration
parameters are modified after the cluster is created (using the mmchconfig command).

GPFS license designation

According to the GPFS Licensing Agreement, each node in the cluster must be designated as possessing a
GPFS Client license, a GPFS FPO license, or a GPFS Server license.

The full text of the Licensing Agreement is provided with the installation media and can be found at the
IBM Software license agreements website (www.ibm.com/software/sla/sladb.nsf).

The type of license that is associated with any one node depends on the functional roles that the node
has been designated to perform.
GPFS Client license
The GPFS Client license permits exchange of data between nodes that locally mount the same
GPFS file system. No other export of the data is permitted. The GPFS Client may not be used for
nodes to share GPFS data directly through any application, service, protocol or method, such as
Network File System (NFS), Common Internet File System (CIFS), File Transfer Protocol (FTP), or
Hypertext Transfer Protocol (HTTP). For these functions, a GPFS Server license would be
required.
GPFS FPO license
The GPFS FPO license permits the licensed node to perform NSD server functions for sharing
GPFS data with other nodes that have a GPFS FPO or GPFS server license. This license cannot be
used to share data with nodes that have a GPFS client license or non-GPFS nodes.
GPFS server license
The GPFS server license permits the licensed node to perform GPFS management functions such
as cluster configuration manager, quorum node, manager node, and Network Shared Disk (NSD)
server. In addition, the GPFS Server license permits the licensed node to share GPFS data directly
through any application, service protocol or method such as NFS, CIFS, FTP, or HTTP.

These licenses are all valid for use in the GPFS Express Edition, GPFS Standard Edition, and GPFS
Advanced Edition.

The GPFS license designation is achieved by issuing the appropriate mmchlicense command. The
number and type of licenses currently in effect for the cluster can be viewed using the mmlslicense
command. See the GPFS: Administration and Programming Reference for more information about these
commands.

Disk considerations
Designing a proper storage infrastructure for your GPFS file systems is key to achieving performance and
reliability goals. When deciding what disk configuration to use, you should consider three key areas:
infrastructure, performance, and disk access method.
Infrastructure
v Ensure that you have sufficient disks to meet the expected I/O load. In GPFS terminology, a
disk may be a physical disk or a RAID device.

Planning for GPFS 15

 v Ensure that you have sufficient connectivity (adapters and buses) between disks and network
shared disk servers.
v Determine whether you are within GPFS limits. Starting with GPFS 3.1, the structural limit on
the maximum number of disks in a file system increased from 2048 to 4096; however, the
current version of GPFS still enforces the original limit of 2048. Should your environment
require support for more than 2048 disks, contact the IBM Support Center to discuss increasing
the enforced limit. (However, the number of disks in your system is often constrained by
products other than GPFS.)
v For a list of storage devices tested with GPFS, see the GPFS FAQ in IBM Knowledge Center
(www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/
gpfsclustersfaq.html).
v For Linux on System z: See the “Storage” topics “DASD device driver” and “SCSI-over-Fibre
Channel device driver” in Device Drivers, Features, and Commands (www.ibm.com/support/
knowledgecenter/api/content/linuxonibm/liaaf/lnz_r_dd.html) in the Linux on System z
library overview.
Disk access method
v Decide how your disks will be connected. Supported types of disk connectivity include the
following configurations:
1. All disks SAN-attached to all nodes in all clusters that access the file system
In this configuration, every node sees the same disk simultaneously and has a
corresponding disk device entry.
2. Each disk connected to multiple NSD server nodes (up to eight servers), as specified on the
server list
In this configuration, a single node with connectivity to a disk performs data shipping to all
other nodes. This node is the first NSD server specified on the NSD server list. You can
define additional NSD servers on the server list. Having multiple NSD servers guards
against the loss of a single NSD server. When using multiple NSD servers, all NSD servers
must have connectivity to the same disks. In this configuration, all nodes that are not NSD
servers will receive their data over the local area network from the first NSD server on the
server list. If the first NSD server fails, the next available NSD server on the list will control
data distribution.
3. A combination of SAN-attached and an NSD server configuration.
Configuration consideration:
– If the node has a physical attachment to the disk and that connection fails, the
node switches to using a specified NSD server to perform I/O. For this reason, it
is recommended that you define NSDs with multiple servers, even if all nodes
have physical attachments to the disk.
– Configuring GPFS disks without an NSD server stops the serving of data when
the direct path to the disk is lost. This may be a preferable option for nodes
requiring a higher speed data connection provided through a SAN as opposed to
a lower speed network NSD server connection. Parallel jobs using MPI often
have this characteristic.
– The -o useNSDserver file system mount option on the mmmount, mount,
mmchfs, and mmremotefs commands can be used to specify the disk discovery,
and limit or eliminate switching from local access to NSD server access, or the
other way around.
v Decide whether you will use storage pools to manage your disks.
Storage pools allow you to manage your file system's storage in groups. You can partition your
storage based on such factors as performance, locality, and reliability. Files are assigned to a
storage pool based on defined policies.
Policies provide for the following:

16 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
 – Placing files in a specific storage pool when the files are created
– Migrating files from one storage pool to another
– File deletion based on file characteristics
See the GPFS: Advanced Administration Guide for more information.

Disk considerations include:

1. “Network Shared Disk (NSD) creation considerations”
2. “NSD server considerations” on page 18
3. “File system descriptor quorum” on page 19
4. “Preparing direct access storage devices (DASD) for NSDs” on page 19

Network Shared Disk (NSD) creation considerations

You must prepare each physical disk you intend to use with GPFS by first defining it as a Network
Shared Disk (NSD) using the mmcrnsd command.

On Windows, GPFS will only create NSDs from empty disk drives. mmcrnsd accepts Windows Basic disk
or Unknown/Not Initialized disks. It always re-initializes these disks so that they become Basic GPT Disks
with a single GPFS partition. NSD data is stored in GPFS partitions. This allows other operating system
components to recognize that the disks are used. mmdelnsd deletes the partition tables created by
mmcrnsd.

A new NSD format was introduced with GPFS 4.1. The new format is referred to as NSD v2, and the old
format is referred to as NSD v1. The NSD v1 format is compatible with GPFS releases prior to 4.1. The
latest GPFS release recognizes both NSD v1 and NSD v2 formatted disks.

The NSD v2 format provides the following benefits:

v On Linux, includes a partition table so that the disk is easily recognized as a GPFS device
v Adjusts data alignment to support disks with a 4 KB physical block size
v Adds backup copies of some key GPFS data structures
v Expands some reserved areas to allow for future growth

Administrators do not need to select one format or the other when managing NSDs. GPFS will always
create and use the correct format based on the minReleaseLevel for the cluster and the file system
version. When minReleaseLevel (as reported by mmlsconfig) is less than 4.1.0.0, mmcrnsd will only
create NSD v1 formatted disks. When minReleaseLevel is at least 4.1.0.0, mmcrnsd will only create NSD
v2 formatted disks. In this second case, however, the NSD format may change dynamically when the
NSD is added to a file system so that the NSD is compatible with the file system version.

On Linux, NSD v2 formatted disks include a GUID Partition Table (GPT) with a single partition. The GPT
allows other operating system utilities to recognize when a disk is owned by GPFS, which helps prevent
inadvertent data corruption. After running mmcrnsd, Linux utilities like parted can show the partition
table. When an NSD v2 formatted disk is added to a 3.5 or older file system, its format is changed to
NSD v1 and the partition table is converted to an MBR (MS-DOS compatible) type.

The mmcrnsd command expects as input a stanza file. For details, see the following GPFS: Administration
and Programming Reference topics:
v “Stanza files”
v “mmchdisk command”
v “mmchnsd command”
v “mmcrfs command”
v “mmcrnsd command”

Planning for GPFS 17

NSD server considerations
If you plan to use NSD servers to remotely serve disk data to other nodes, as opposed to having disks
SAN-attached to all nodes, you should consider the total computing and I/O load on these nodes:
v Will your Network Shared Disk servers be dedicated servers or will you also be using them to run
applications? If you will have non-dedicated servers, consider running less time-critical applications on
these nodes. If you run time-critical applications on a Network Shared Disk server, servicing disk
requests from other nodes might conflict with the demands of these applications.
v The special functions of the file system manager consume extra processing time. If possible, avoid
using a Network Shared Disk server as the file system manager. The Network Shared Disk server
consumes both memory and processor cycles that could impact the operation of the file system
manager. See The file system manager.
v The actual processing capability required for Network Shared Disk service is a function of the
application I/O access patterns, the type of node, the type of disk, and the disk connection. You can
later run iostat on the server to determine how much of a load your access pattern will place on a
Network Shared Disk server.
v Providing sufficient disks and adapters on the system to yield the required I/O bandwidth. Dedicated
Network Shared Disk servers should have sufficient disks and adapters to drive the I/O load you
expect them to handle.
v Knowing approximately how much storage capacity you will need for your data.

You should consider what you want as the default behavior for switching between local access and NSD
server access in the event of a failure. To set this configuration, use the -o useNSDserver file system
mount option of the mmmount, mount, mmchfs, and mmremotefs commands to:
v Specify the disk discovery behavior
v Limit or eliminate switching from either:
– Local access to NSD server access
– NSD server access to local access

You should consider specifying how long to wait for an NSD server to come online before allowing a file
system mount to fail because the server is not available. The mmchconfig command has these options:
nsdServerWaitTimeForMount
When a node is trying to mount a file system whose disks depend on NSD servers, this option
specifies the number of seconds to wait for those servers to come up. If a server recovery is
taking place, the wait time you are specifying with this option starts after recovery completes.

Note: The decision to wait for servers is controlled by the nsdServerWaitTimeWindowOnMount

option.
nsdServerWaitTimeWindowOnMount
Specifies a window of time (in seconds) during which a mount can wait for NSD servers as
described for the nsdServerWaitTimeForMount option. The window begins when quorum is
established (at cluster startup or subsequently), or at the last known failure times of the NSD
servers required to perform the mount.

Notes:
1. When a node rejoins a cluster, it resets all the failure times it knew about within that cluster.
2. Because a node that rejoins a cluster resets its failure times within that cluster, the NSD server
failure times are also reset.
3. When a node attempts to mount a file system, GPFS checks the cluster formation criteria first.
If that check falls outside the window, it will then check for NSD server fail times being in the
window.

18 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
File system descriptor quorum
A GPFS structure called the file system descriptor is initially written to every disk in the file system and is
replicated on a subset of the disks as changes to the file system occur, such as the adding or deleting of
disks. Based on the number of failure groups and disks, GPFS creates one to five replicas of the
descriptor:
v If there are at least five different failure groups, five replicas are created.
v If there are at least three different disks, three replicas are created.
v If there are only one or two disks, a replica is created on each disk.

Once it decides how many replicas to create, GPFS picks disks to hold the replicas, so that all replicas are
in different failure groups, if possible, to reduce the risk of multiple failures. In picking replica locations,
the current state of the disks is taken into account. Stopped or suspended disks are avoided. Similarly,
when a failed disk is brought back online, GPFS might rebalance the file system descriptors in order to
assure reliability across the failure groups. The disks used to hold the file system descriptor replicas can
be seen by running the mmlsdisk fsname -L command and looking for the string desc in the Remarks
column.

GPFS requires that a majority of the replicas on the subset of disks remain available to sustain file system
operations:
v If there are at least five different replicas, GPFS can tolerate a loss of two of the five replicas.
v If there are at least three replicas, GPFS can tolerate a loss of one of the three replicas.
v If there are fewer than three replicas, a loss of one replica might make the descriptor inaccessible.

The loss of all disks in a disk failure group might cause a majority of file systems descriptors to become
unavailable and inhibit further file system operations. For example, if your file system is backed up by
three or more disks that are assigned to two separate disk failure groups, one of the failure groups will
be assigned two of the file system descriptor replicas, while the other failure group will be assigned only
one replica. If all of the disks in the disk failure group that contains the two replicas were to become
unavailable, the file system would also become unavailable. To avoid this particular scenario, you might
want to introduce a third disk failure group consisting of a single disk that is designated as a descOnly
disk. This disk would exist solely to contain a replica of the file system descriptor (that is, it would not
contain any file system metadata or data). This disk should be at least 128MB in size.

For more information on this topic, see “Network Shared Disk (NSD) creation considerations” on page 17
and the topic "Establishing disaster recovery for your GPFS cluster" in the GPFS: Advanced Administration
Guide.

Preparing direct access storage devices (DASD) for NSDs

When preparing direct access storage devices (DASD) for NSDs, see the table “Disk hardware tested with
GPFS for Linux on System z” in the GPFS FAQ in IBM Knowledge Center (www.ibm.com/support/
knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html).

Preparing your environment for use of extended count key data (ECKD) devices
If your GPFS cluster includes Linux on System z instances, do not use virtual reserve/release. Instead,
follow the process described in Sharing DASD without Using Virtual Reserve/Release
(www.ibm.com/support/knowledgecenter/SSB27U_6.3.0/com.ibm.zvm.v630.hcpa5/hcpa5259.htm). Data
integrity is handled by GPFS itself.

Preparing an ECKD™ device for GPFS

To prepare an ECKD device for GPFS, complete these steps on a single node:
1. Ensure that the ECKD device is online. To set it online, issue the following command:
chccwdev -e device_bus_id

Planning for GPFS 19

 where device_bus_id identifies the device to be configured. device_bus_id is a device number with a
leading 0.n, where n is the subchannel set ID. For example:
chccwdev -e 0.0.3352
2. Low-level format the ECKD using one of the following commands.

Note: GPFS supports ECKD disks in either compatible disk layout (CDL) format or Linux disk layout
(LDL) format. The DASD must be formatted with a block size of 4096.
v To specify CDL format, issue the following command:
dasdfmt -d cdl device
There is no need to specify a block size value, as the default value is 4096.
v To specify LDL format, issue the following command:
dasdfmt -d ldl device
There is no need to specify a block size value, as the default value is 4096.
In both of these commands, device is the node of the device. For example:
dasdfmt -d cdl /dev/dasda
| 3. This step is for CDL disks only. It is an optional step because partitioning is optional for CDL disks.
| If you wish to partition the ECKD and create a single partition that spans the entire device, use the
| following command:
| fdasd -a device

| Notes:
| v This step is not required for LDL disks because the dasdfmt -d ldl command issued in the previous
| step automatically creates a single Linux partition on the disk.
| v If a CDL disk is partitioned, the partition name should be specified in the stanza input file for
| mmcrnsd. If a CDL disk is not partitioned, the disk name should be specified in the stanza input
| file.

| For more information about all of these commands, see the following:
v “Commands for Linux on System z” topic in Device Drivers, Features, and Commands
(www.ibm.com/support/knowledgecenter/api/content/linuxonibm/liaaf/lnz_r_dd.html) in the Linux
on System z library overview.
| v “Getting started with Elastic Storage for Linux on System z based on GPFS technology” white paper,
| available on the Welcome Page forGPFS in IBM Knowledge Center (www.ibm.com/support/
| knowledgecenter/SSFKCN/gpfs_welcome.html)

Repeat these steps for each ECKD to be used with GPFS.

After preparing the environment, set the ECKD devices online on the other nodes.

Note: In the case of an ECKD device that is attached to two servers and is online on both, if you format
or partition the disk on one of the servers, the other server does not automatically know about that. You
must refresh the information about the disk on the other server. One way to do this is to set the disk
offline and then online again.

Always ensure that the ECKD devices are online before starting GPFS. To automatically set ECKD
devices online at system start, see the documentation for your Linux distribution.

File system creation considerations

File system creation involves anticipating usage within the file system and considering your hardware
configurations. Before creating a file system, consider how much data will be stored and how great the
demand for the files in the system will be.

20 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
 Each of these factors can help you to determine how much disk resource to devote to the file system,
which block size to choose, where to store data and metadata, and how many replicas to maintain. For
the latest supported file system size, see the GPFS FAQ in IBM Knowledge Center (www.ibm.com/
support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html).

Your GPFS file system is created by issuing the mmcrfs command. Table 2 details the file system creation
options specified on the mmcrfs command, which options can be changed later with the mmchfs
command, and what the default values are.

To move an existing file system into a new GPFS cluster, see Exporting file system definitions between
clusters in the GPFS: Advanced Administration Guide.
Table 2. File system creation options
Options mmcrfs mmchfs Default value
Device name of the file system.
X X none
See “Device name of the file system”
on page 24.
DiskDesc for each disk in your file Issue the mmadddisk or
system. mmdeldisk command to add
Note: The use of disk descriptors is or delete disks from the file
discouraged. X system. none

See “Disks for your file system” on

page 24.
-F StanzaFile specifies a file that Issue the mmadddisk or
contains a list of NSD stanzas. mmdeldisk command to add
or delete disks as indicated in
See the GPFS: Administration and X none
the stanza file.
Programming Reference topic “Stanza
files”.
-A {yes | no | automount} to
determine when to mount the file
system.
X X yes
See “Deciding how the file system is
mounted” on page 24.
-B BlockSize to set the data block size: This value cannot be changed
64K, 128K, 256K, 512K, 1M, 2M, 4M, without re-creating the file
8M or 16M. X system. 256K

See “Block size” on page 25.

-D {posix | nfs4} semantics for a
deny-write open lock
X X nfs4
See “NFS V4 deny-write open lock” on
page 24.
-E {yes | no} to report exact mtime
values. X X yes
See “mtime values” on page 26.
-i InodeSize to set the size of inodes: This value cannot be
| X 4096
512, 1024, or 4096 bytes. changed.

Planning for GPFS 21

Table 2. File system creation options (continued)
Options mmcrfs mmchfs Default value
-j {cluster | scatter} to determine the
block allocation map type. See “Block allocation map”
X NA
See “Block allocation map” on page on page 26.
26.
-k {posix | nfs4 |all} to determine the
authorization types supported by the
file system. X X all
See “File system authorization” on
page 27.
-K {no | whenpossible | always} to
enforce strict replication. X X whenpossible
See “Strict replication” on page 27.
-L LogFileSize to specify the size of the
The default is 4 MB or the
internal log files. X X metadata block size,
whichever is larger.
See GPFS recovery logs.
-m DefaultMetadataReplicas

See “File system replication X X 1

parameters” on page 27.
-M MaxMetadataReplicas
This value cannot be
X 2
See “File system replication changed.
parameters” on page 27.
-n NumNodes that will mount the file
system.
X X 32
See “Number of nodes mounting the
file system” on page 28.
-o MountOptions to be passed to the
mount command.
NA X none
See “Assign mount command options”
on page 29.
-Q {yes | no} to activate quota.
X X no
See “Enabling quotas” on page 29.
-r DefaultDataReplicas

See “File system replication X X 1

parameters” on page 27.
-R MaxDataReplicas
This value cannot be
See “File system replication X 2
changed.
parameters” on page 27.
-S {yes | no | relatime} to control
how the atime value is updated.
X X no
See “atime values” on page 26.

22 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Table 2. File system creation options (continued)
Options mmcrfs mmchfs Default value
-t DriveLetter

See “Windows drive letter” on page X X none

28.
-T Mountpoint

See “Mountpoint directory” on page X X /gpfs/DeviceName

29.
-V {full | compat} to change the file
system format to the latest level.
NA X none
See “Changing the file system format
to the latest level” on page 31
-v {yes | no} to verify disk usage.
X NA yes
See “Verifying disk usage” on page 30.
-W NewDeviceName to assign a new
NA X none
device name to the file system.
-z {yes | no} to enable DMAPI
X X no
See “Enable DMAPI” on page 30.
--filesetdf to specify (when quotas are
enforced for a fileset) whether the df
command will report numbers based X X --nofilesetdf
on the quotas for the fileset and not
for the total file system.
--inode-limit MaxNumInodes
[:NumInodesToPreallocate] to determine
the maximum number of files in the
file system. X X file system size/1 MB
See “Specifying the maximum number
of files that can be created” on page
31.
--log-replicas LogReplicas to specify the
X X none
number of recovery log replicas.
--metadata-block-size
MetadataBlockSize to specify the block The default is the same as the
size for the system storage pool. X NA
value set for -B BlockSize.
See “Block size” on page 25.
--mount-priority Priority to control the
order in which the individual file
systems are mounted at daemon
X X 0
startup or when one of the all
keywords is specified on the
mmmount command.
--perfileset-quota to set the scope of
user and group quota limit checks to X X --noperfileset-quota
the individual fileset level.

Planning for GPFS 23

Table 2. File system creation options (continued)
Options mmcrfs mmchfs Default value
--rapid-repair to keep track of
incomplete replication on an
NA X none
individual file block basis (as opposed
to the entire file).
--version VersionString to enable only
the file system features that are
compatible with the specified release.
X NA 4.1.0.0
See “Enabling file system features” on
page 31.
Notes:
1. X – indicates that the option is available on the command.
2. NA (not applicable) – indicates that the option is not available on the command.

Device name of the file system

File system names must be unique within a GPFS cluster. However, two different clusters can have two
distinct file systems with the same name. The device name of the file system does not need to be fully
qualified. fs0 is as acceptable as /dev/fs0. The name cannot be the same as an existing entry in /dev.

Note: If your cluster includes Windows nodes, the file system name should be no longer than 31
characters.

NFS V4 deny-write open lock

You can specify whether a deny-write open lock blocks writes, which is expected and required by NFS
V4, Samba, and Windows. See Managing GPFS access control lists and NFS export in the GPFS:
Administration and Programming Reference.
nfs4 Must be specified for file systems supporting NFS V4 and file systems mounted on Windows.
This is the default.
posix Specified for file systems supporting NFS V3 or ones which are not NFS exported.
posix allows NFS writes even in the presence of a deny-write open lock.

Disks for your file system

Disks must be defined as NSDs before they can be added to a GPFS file system. NSDs are created using
the mmcrnsd command. You can use the mmlsnsd -F command to display a list of available NSDs.

See “Disk considerations” on page 15.

Deciding how the file system is mounted

Specify when the file system is to be mounted:
yes When the GPFS daemon starts. This is the default.
no Manual mount.
automount
When the file system is first accessed.

This can be changed at a later time by using the -A option on the mmchfs command.

Considerations:

24 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
1. GPFS mount traffic may be lessened by using the automount feature to mount the file system when it
is first accessed instead of at GPFS startup. Automatic mounts only produce additional control traffic
at the point that the file system is first used by an application or user. Mounts at GPFS startup on the
other hand produce additional control traffic at every GPFS startup. Thus startup of hundreds of
nodes at once may be better served by using automatic mounts.
2. Automatic mounts will fail if the node does not have the operating systems automount support
enabled for the file system.
3. When exporting file systems for NFS mounts, it may be useful to mount the file system when GPFS
starts.

Block size
The size of data blocks in a file system can be specified at file system creation by using the -B option on
the mmcrfs command or allowed to default to 256 KB. This value cannot be changed without re-creating
the file system.

GPFS supports these block sizes for file systems: 64 KB, 128 KB, 256 KB, 512 KB, 1 MB, 2 MB, 4 MB, 8
MB and 16 MB (for GPFS Native RAID only). This value should be specified with the character K or M
as appropriate, for example: 512K or 4M. You should choose the block size based on the application set
that you plan to support and whether you are using RAID hardware.

The --metadata-block-size option on the mmcrfs command allows a different block size to be specified
for the system storage pool, provided its usage is set to metadataOnly. This can be especially beneficial if
the default block size is larger than 1 MB. Valid values are the same as those listed for the -B option.

If you plan to use RAID devices in your file system, a larger block size may be more effective and help
avoid the penalties involved in small block write operations to RAID devices. For example, in a RAID
configuration using 4 data disks and 1 parity disk (a 4+P RAID 5 configuration), which uses a 64 KB
stripe size, the optimal file system block size would be an integral multiple of 256 KB (4 data disks × 64
KB stripe size = 256 KB). A block size of an integral multiple of 256 KB results in a single data write that
encompasses the 4 data disks and a parity-write to the parity disk. If a block size smaller than 256 KB,
such as 64 KB, is used with the same RAID configuration, write performance is degraded by the
read-modify-write behavior. A 64 KB block size results in a single disk writing 64 KB and a subsequent
read from the three remaining disks in order to compute the parity that is then written to the parity disk.
The extra read degrades performance.

The choice of block size also affects the performance of certain metadata operations, in particular, block
allocation performance. The GPFS block allocation map is stored in blocks, similar to regular files. When
the block size is small:
v It takes more blocks to store a given amount of data resulting in additional work to allocate those
blocks
v One block of allocation map data contains less information

Note: The choice of block size is particularly important for large file systems. For file systems larger than
100 TB, you should use a block size of at least 256 KB.

Fragments and subblocks

GPFS divides each block into 32 subblocks. Files smaller than one block size are stored in fragments, which
are made up of one or more subblocks. Large files are stored in a number of full blocks plus zero or more
subblocks to hold the data at the end of the file.

The block size is the largest contiguous amount of disk space allocated to a file and therefore the largest
amount of data that can be accessed in a single I/O operation. The subblock is the smallest unit of disk
space that can be allocated. For a block size of 256 KB, GPFS reads as much as 256 KB of data in a single
I/O operation and small files can occupy as little as 8 KB of disk space.

Planning for GPFS 25

atime values
atime is a standard file attribute that represents the time when the file was last accessed. The -S file
system configuration parameter controls how the atime value is updated. The default is -S no, which
results in updating atime locally in memory whenever a file is read, but the value is not visible to other
nodes until after the file is closed. If an accurate atime value is needed, the application must use the
GPFS calls gpfs_stat() and gpfs_fstat() functions. When -S yes is specified, or the file system is mounted
read-only, the updating of the atime value is suppressed. This means that the atime value is no longer
updated. This can be an issue if you have policies that use the ACCESS_TIME file attribute for file
management. For more information, see the topic, Exceptions to the Open Group technical standards in the
GPFS: Administration and Programming Reference.

When -S relatime is specified, the file access time is updated only if the existing access time is older than
the value of the atimeDeferredSeconds configuration attribute or the existing file modification time is
greater than the existing access time.

mtime values
mtime is a standard file attribute that represents the time when the file was last modified. The -E
parameter controls when the mtime is updated. The default is -E yes, which results in standard interfaces
including the stat() and fstat() calls reporting exact mtime values. Specifying -E no results in the stat()
and fstat() calls reporting the mtime value available at the completion of the last sync period. This may
result in the calls not always reporting the exact mtime. Setting -E no can affect backup operations that
rely on last modified time or the operation of policies using the MODIFICATION_TIME file attribute.

For more information, see the topic, Exceptions to the Open Group technical standards in the GPFS:
Administration and Programming Reference.

Block allocation map

GPFS has two different methods of allocating space in a file system. The -j parameter specifies the block
allocation map type to use when creating a file system. The block allocation map type cannot be changed
once the file system is created.

When allocating blocks for a given file, GPFS first uses a round-robin algorithm to spread the data across
all of the disks in the file system. After a disk is selected, the location of the data block on the disk is
determined by the block allocation map type.

The two types of allocation methods are cluster and scatter:

cluster
GPFS attempts to allocate blocks in clusters. Blocks that belong to a given file are kept next to
each other within each cluster.
This allocation method provides better disk performance for some disk subsystems in relatively
small installations. The benefits of clustered block allocation diminish when the number of nodes
in the cluster or the number of disks in a file system increases, or when the file system free space
becomes fragmented. The cluster allocation method is the default for GPFS clusters with eight or
fewer nodes and for files systems with eight or fewer disks.
scatter GPFS chooses the location of the blocks randomly.
This allocation method provides more consistent file system performance by averaging out
performance variations due to block location (for many disk subsystems, the location of the data
relative to the disk edge has a substantial effect on performance). This allocation method is
appropriate in most cases and is the default for GPFS clusters with more than eight nodes or file
systems with more than eight disks.

26 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
This parameter for a given file system is specified at file system creation by using the -j option on the
mmcrfs command, or allowing it to default. This value cannot be changed after the file system has been
created.

File system authorization

The type of authorization for the file system is specified on the -k option on the mmcrfs command or
changed at a later time by using the -k option on the mmchfs command:
posix Traditional GPFS access control lists (ACLs) only (NFS V4 and Windows ACLs are not allowed).
nfs4 Support for NFS V4 and Windows ACLs only. Users are not allowed to assign traditional ACLs to
any file system objects.
all Allows for the coexistence of POSIX, NFS V4, and Windows ACLs within a file system. This is
the default.

Avoid specifying nfs4 or all unless files will be exported to NFS V4 clients or the file system will be
mounted on Windows.

Strict replication
Strict replication means that data or metadata replication is performed at all times, according to the
replication parameters specified for the file system. If GPFS cannot perform the file system's replication,
an error is returned. These are the choices:
no Strict replication is not enforced. GPFS tries to create the needed number of replicas, but returns
an errno of EOK if it can allocate at least one replica.
whenpossible
Strict replication is enforced if the disk configuration allows it. If the number of failure groups is
insufficient, strict replication is not enforced. This is the default value.
always
Indicates that strict replication is enforced.

The use of strict replication can be specified at file system creation by using the -K option on the mmcrfs
command. The default is whenpossible. This value can be changed using the mmchfs command.

Internal log file

You can specify the internal log file size. Refer to GPFS recovery logs for additional information.

File system replication parameters

The metadata (inodes, directories, and indirect blocks) and data replication parameters are set at the file
system level and apply to all files. They are initially set for the file system when issuing the mmcrfs
command. They can be changed for an existing file system using the mmchfs command. When the
replication parameters are changed, files created after the change are affected. To apply the new
replication values to existing files in a file system, issue the mmrestripefs command.

Metadata and data replication are specified independently. Each has a default replication factor of 1 (no
replication) and a maximum replication factor with a default of 2. Although replication of metadata is
less costly in terms of disk space than replication of file data, excessive replication of metadata also
affects GPFS efficiency because all metadata replicas must be written. In general, more replication uses
more space.

Default metadata replicas

The default number of copies of metadata for all files in the file system may be specified at file system
creation by using the -m option on the mmcrfs command or changed at a later time by using the -m

Planning for GPFS 27

option on the mmchfs command. This value must be equal to or less than MaxMetadataReplicas, and
cannot exceed the number of failure groups with disks that can store metadata. The allowable values are
1, 2, or 3, with a default of 1.

Maximum metadata replicas

The maximum number of copies of metadata for all files in the file system can be specified at file system
creation by using the -M option on the mmcrfs command. The default is 2. The allowable values are 1, 2,
or 3, but it cannot be less than the value of DefaultMetadataReplicas. This value cannot be changed.

Default data replicas

The default replication factor for data blocks may be specified at file system creation by using the -r
option on the mmcrfs command or changed at a later time by using the -r option on the mmchfs
command. This value must be equal to or less than MaxDataReplicas, and the value cannot exceed the
number of failure groups with disks that can store data. The allowable values are 1, 2, and 3, with a
default of 1.

If you want to change the data replication factor for the entire file system, the data disk in each storage
pool must have a number of failure groups equal to or greater than the replication factor. For example,
you will get a failure with error messages if you try to change the replication factor for a file system to 2
but the storage pool has only one failure group.

Maximum data replicas

The maximum number of copies of data blocks for a file can be specified at file system creation by using
the -R option on the mmcrfs command. The default is 2. The allowable values are 1, 2, and 3, but cannot
be less than the value of DefaultDataReplicas. This value cannot be changed.

Number of nodes mounting the file system

The estimated number of nodes that will mount the file system may be specified at file system creation
by using the -n option on the mmcrfs command or allowed to default to 32.

When creating a GPFS file system, over-estimate the number of nodes that will mount the file system.
This input is used in the creation of GPFS data structures that are essential for achieving the maximum
degree of parallelism in file system operations (see GPFS architecture). Although a larger estimate
consumes a bit more memory, insufficient allocation of these data structures can limit the ability to
process certain parallel requests efficiently, such as the allotment of disk space to a file. If you cannot
predict the number of nodes, allow the default value to be applied. Specify a larger number if you expect
to add nodes, but avoid wildly overestimating as this can affect buffer operations.

You can change the number of nodes using the mmchfs command. Changing this value affects storage
pools created after the value was set; so, for example, if you need to increase this value on a storage pool,
you could change the value, create a new storage pool, and migrate the data from one pool to the other.

Windows drive letter

In a Windows environment, you must associate a drive letter with a file system before it can be mounted.
The drive letter can be specified and changed with the -t option of the mmcrfs and mmchfs commands.
GPFS does not assign a default drive letter when one is not specified.

The number of available drive letters restricts the number of file systems that can be mounted on
Windows.

Note: Certain applications give special meaning to drive letters A:, B:, and C:, which could cause
problems if they are assigned to a GPFS file system.

28 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Mountpoint directory
Every GPFS file system has a default mount point associated with it. This mount point can be specified
and changed with the -T option of the mmcrfs and mmchfs commands. If you do not specify a mount
point when you create the file system, GPFS will set the default mount point to /gpfs/DeviceName.

Assign mount command options

Options may be passed to the file system mount command using the -o option on the mmchfs command.
In particular, you can choose the option to perform quota activation automatically when a file system is
mounted.

Enabling quotas
The GPFS quota system can help you control file system usage. Quotas can be defined for individual
users, groups of users, or filesets. Quotas can be set on the total number of files and the total amount of
data space consumed. When setting quota limits for a file system, the system administrator should
consider the replication factors of the file system. Quota management takes replication into account when
reporting on and determining if quota limits have been exceeded for both block and file usage. In a file
system that has either data replication or metadata replication set to a value of two, the values reported
on by both the mmlsquota and mmrepquota commands are double the value reported by the ls
command.

Whether or not to enable quotas when a file system is mounted may be specified at file system creation
by using the -Q option on the mmcrfs command or changed at a later time by using the -Q option on the
mmchfs command. After the file system has been mounted, quota values are established by issuing the
mmedquota command and activated by issuing the mmquotaon command. The default is to not have
quotas activated.

GPFS levels are defined at three limits that you can explicitly set using the mmedquota and
mmdefedquota commands:
Soft limit
Defines levels of disk space and files below which the user, group of users, or fileset can safely
operate.
Specified in units of kilobytes (k or K), megabytes (m or M), or gigabytes (g or G). If no suffix is
provided, the number is assumed to be in bytes.
Hard limit
Defines the maximum amount of disk space and number of files the user, group of users, or
fileset can accumulate.
Specified in units of kilobytes (k or K), megabytes (m or M), or gigabytes (g or G). If no suffix is
provided, the number is assumed to be in bytes.
Grace period
Allows the user, group of users, or fileset to exceed the soft limit for a specified period of time.
The default period is one week. If usage is not reduced to a level below the soft limit during that
time, the quota system interprets the soft limit as the hard limit and no further allocation is
allowed. The user, group of users, or fileset can reset this condition by reducing usage enough to
fall below the soft limit; or the administrator can increase the quota levels using the mmedquota
or mmdefedquota.

Default quotas
Applying default quotas provides all new users, groups of users, or filesets with established minimum
quota limits. If default quota values are not enabled, new users, new groups, or new filesets have a quota
value of zero, which establishes no limit to the amount of space that can be used.

Planning for GPFS 29

Default quota limits can be set or changed only if the -Q yes option is in effect for the file system. To set
default quotas at the fileset level, the --perfileset-quota option must also be in effect. The -Q yes and
--perfileset-quota options are specified when creating a file system with the mmcrfs command or
changing file system attributes with the mmchfs command. Use the mmlsfs command to display the
current settings of these quota options. Default quotas may then be enabled by issuing the
mmdefquotaon command. Default values are established by issuing the mmdefedquota command.

Quota system files

The GPFS quota system maintains three separate files that contain data about usage and limits. These
files reside in the root directory of the GPFS file systems when quotas are enabled:
v user.quota
v group.quota
v fileset.quota

All three .quota files are:

v Built with the information provided in the mmedquota and mmdefedquota commands.
v Updated through normal allocation operations throughout the file system and when the
mmcheckquota command is issued.
v Readable by the mmlsquota and mmrepquota commands.

The .quota files are read from the root directory when mounting a file system with quotas enabled. When
these files are read, one of three possible actions take place:
v The files contain quota information and the user wants these files to be used.
v The files contain quota information, however, the user wants different files to be used.
To specify the use of different files, the mmcheckquota command must be issued prior to the mount of
the file system.
v The files do not contain quota information. In this case the mount fails and appropriate error messages
are issued. See the GPFS: Problem Determination Guide for further information regarding mount failures.

Enable DMAPI
Whether or not the file system can be monitored and managed by the GPFS Data Management API
(DMAPI) may be specified at file system creation by using the -z option on the mmcrfs command or
changed at a later time by using the -z option on the mmchfs command. The default is not to enable
DMAPI for the file system.

For further information about DMAPI for GPFS, see the GPFS: Data Management API Guide.

Verifying disk usage

The -v option controls whether the mmcrfs command checks whether the specified disks can safely be
added to the file system. The default (-v yes) is to perform the check and fail the command if any of the
disks appear to belong to some other GPFS file system. You should override the default behavior and
specify -v no only if the mmcrfs command rejects the disks and you are certain that all of the disks
indeed do not belong to an active GPFS file system. An example for an appropriate use of -v no is the
case where an mmcrfs command is interrupted for some reason and you are reissuing the command.
Another example would be if you are reusing disks from an old GPFS file system that was not formally
destroyed with the mmdelfs command.

Important: Using mmcrfs -v no on a disk that already belongs to a file system will corrupt that file
system.

30 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Changing the file system format to the latest level
You can change the file system format to the latest format supported by the currently-installed level of
GPFS by issuing the mmchfs command with the -V full option or the -V compat option. The full option
enables all new functionality that requires different on-disk data structures. This may cause the file
system to become permanently incompatible with earlier releases of GPFS. The compat option enables
only changes that are backward compatible with the previous GPFS release. If all GPFS nodes that are
accessing a file system (both local and remote) are running the latest level, then it is safe to use full
option. Certain features may require you to run the mmmigratefs command to enable them.

Enabling file system features

By default, new file systems are created with all currently available features enabled. Since this may
prevent clusters that are running earlier GPFS releases from accessing the file system, you can enable
only the file system features that are compatible with the specified release by issuing the mmcrfs
command with the --version Version option. See the GPFS: Administration and Programming Reference for
more information.

Specifying whether the df command will report numbers based on

quotas for the fileset
You can specify (when quotas are enforced for a fileset) whether the df command will report numbers
based on the quotas for the fileset and not for the total file system. To do so, use the --filesetdf |
--nofilesetdf option on either the mmchfs command or the mmcrfs command.

Specifying the maximum number of files that can be created

The maximum number of files that can be created can be specified by using the --inode-limit option on
the mmchfs command. Allowable values, which range from the current number of created inodes
(determined by issuing the mmdf command with the -F option) through the maximum number of files
that are supported, are constrained by the formula:
maximum number of files = (total file system space) / (inode size + subblock size)

You can determine the inode size (-i) and subblock size (value of the -B parameter / 32) of a file system
by running the mmlsfs command. The maximum number of files in a file system may be specified at file
system creation by using the --inode-limit option on the mmcrfs command, or it may be increased at a
later time by using --inode-limit on the mmchfs command. This value defaults to the size of the file
system at creation divided by 1 MB and cannot exceed the architectural limit. When a file system is
created, 4084 inodes are used by default; these inodes are used by GPFS for internal system files.

The --inode-limit option applies only to the root fileset. When there are multiple inode spaces, use the
--inode-space option of the mmchfileset command to alter the inode limits of independent filesets. The
mmchfileset command can also be used to modify the root inode space. The --inode-space option of the
mmlsfs command shows the sum of all inode spaces.

Inodes are allocated when they are used. When a file is deleted, the inode is reused, but inodes are never
deallocated. When setting the maximum number of inodes in a file system, there is the option to
preallocate inodes. However, in most cases there is no need to preallocate inodes because, by default,
inodes are allocated in sets as needed. If you do decide to preallocate inodes, be careful not to preallocate
more inodes than will be used; otherwise, the allocated inodes will unnecessarily consume metadata
space that cannot be reclaimed.

These options limit the maximum number of files that may actively exist within a file system. However,
the maximum number of files in the file system may be restricted further by GPFS so the control
structures associated with each file do not consume all of the file system space.

Planning for GPFS 31

Further considerations when managing inodes:
1. For file systems that are supporting parallel file creates, as the total number of free inodes drops
below 5% of the total number of inodes, there is the potential for slowdown in file system access.
Take this into consideration when creating or changing your file system. Use the mmdf command to
display the number of free inodes.
2. Excessively increasing the value for the maximum number of files may cause the allocation of too
much disk space for control structures.

Controlling the order in which file systems are mounted

You can control the order in which the individual file systems are mounted at daemon startup (if
mmlsconfig autoload shows yes) or when one of the all keywords is specified. To do so, use the
--mount-priority Priority option on the mmcrfs, mmchfs, or mmremotefs command.

A sample file system creation

To create a file system called gpfs2 with the properties:
v The disks for the file system listed in the file /tmp/gpfs2dsk
v Automatically mount the file system when the GPFS daemon starts (-A yes)
v A block size of 256 KB (-B 256K)
v Mount it on 32 nodes (-n 32)
v Both default replication and the maximum replication for metadata set to two (-m 2 -M 2)
v Default replication for data set to one and the maximum replication for data set to two (-r 1 -R 2)
v Default mount point (-T /gpfs2)

Enter:
mmcrfs /dev/gpfs2 -F /tmp/gpfs2dsk -A yes -B 256K -n 32 -m 2 -M 2 -r 1 -R 2 -T /gpfs2

The system displays information similar to:

The following disks of gpfs2 will be formatted on node k194p03.tes.nnn.com:
hd25n09: size 17796014 KB
hd24n09: size 17796014 KB
hd23n09: size 17796014 KB
Formatting file system ...
Disks up to size 59 GB can be added to storage pool system.
Creating Inode File
56 % complete on Mon Mar 3 15:10:08 2014
100 % complete on Mon Mar 3 15:10:11 2014
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
44 % complete on Mon Mar 3 15:11:32 2014
90 % complete on Mon Mar 3 15:11:37 2014
100 % complete on Mon Mar 3 15:11:38 2014
Completed creation of file system /dev/gpfs2.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

To confirm the file system configuration, issue the command:

mmlsfs gpfs2

The system displays information similar to:

flag value description
------------------- ------------------------ -----------------------------------
-f 262144 Minimum fragment size in bytes
-i 512 Inode size in bytes
-I 32768 Indirect block size in bytes

32 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
-m 2 Default number of metadata replicas
-M 2 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 2 Maximum number of data replicas
-j scatter Block allocation type
-D nfs4 File locking semantics in effect
-k all ACL semantics in effect
-n 32 Estimated number of nodes that will mount file system
-B 262144 Block size
-Q none Quotas accounting enabled
none Quotas enforced
none Default quotas enabled
--perfileset-quota yes Per-fileset quota enforcement
--filesetdf yes Fileset df enabled?
-V 14.10 (4.1.0.4) File system version
--create-time Fri Aug 8 18:39:47 2014 File system creation time
-z no Is DMAPI enabled?
-L 262144 Logfile size
-E yes Exact mtime mount option
-S yes Suppress atime mount option
-K whenpossible Strict replica allocation option
--fastea yes Fast external attributes enabled?
--encryption no Encryption enabled?
--inode-limit 2015232 Maximum number of inodes
--log-replicas 0 Number of log replicas (max 2)
--is4KAligned yes is4KAligned?
--rapid-repair yes rapidRepair enabled?
-P system Disk storage pools in file system
-d gpfs1001nsd;gpfs1002nsd Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /gpfs2 Default mount point
--mount-priority 0 Mount priority

Planning for GPFS 33

34 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Installing GPFS on Linux nodes
There are three steps to installing GPFS on Linux nodes: Preparing the environment; installing the GPFS
software; and building the GPFS portability layer. The information in this topic points you to the detailed
steps.

Before installing GPFS, you should review “Planning for GPFS” on page 3 and the GPFS FAQ in IBM
Knowledge Center (www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/
gpfs_faqs/gpfsclustersfaq.html).

Installing GPFS without ensuring that the prerequisites listed in “Hardware requirements” on page 3 and
“Software requirements” on page 4 are satisfied can lead to undesired results.

The installation process includes:

1. “Preparing the environment on Linux nodes”
2. “Installing the GPFS software on Linux nodes” on page 36
3. “Building the GPFS portability layer on Linux nodes” on page 40
4. “For Linux on System z: Changing the kernel settings” on page 41

Preparing the environment on Linux nodes

Before proceeding with installation, prepare your environment by following the suggestions in the
following sections.

Add the GPFS bin directory to your shell PATH

Ensure that the PATH environment variable for the root user on each node includes /usr/lpp/mmfs/bin.
(This is not required for the operation of GPFS, but it can simplify administration.)

Other suggestions for cluster administration

GPFS commands operate on all nodes required to perform tasks. When you are administering a cluster, it
may be useful to have a more general form of running commands on all of the nodes. One suggested
way to do this is to use an OS utility like dsh or pdsh that can execute commands on all nodes in the
cluster. For example, you can use dsh to check the kernel version of each node in your cluster:
# dsh uname -opr
Node01: 2.6.18-128.1.14.e15 x86_64 GNU/Linux
Node02: 2.6.18-128.1.14.e15 x86_64 GNU/Linux

Once you have dsh set up, you can use it to install GPFS on a large cluster. For details about setting up
dsh or a similar utility, review the documentation for the utility.

Verify that prerequisite software is installed

Before installing GPFS, it is necessary to verify that you have the correct levels of the prerequisite
software installed on each node in the cluster. If the correct level of prerequisite software is not installed,
see the appropriate installation manual before proceeding with your GPFS installation.

For the most up-to-date list of prerequisite software, see the GPFS FAQ in IBM Knowledge Center
(www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/
gpfsclustersfaq.html).

© Copyright IBM Corp. 2014 35

The FAQ contains the latest information about the following:
v Supported Linux distributions and kernel levels
v Recommended or required RPM levels
v Software recommendations
v Configuration information

Before proceeding, see also GPFS and network communication.

Installing the GPFS software on Linux nodes

Follow the steps in this topic in the specified order to install the GPFS software.

This procedure installs GPFS on one node at a time:

1. “Accepting the electronic license agreement on Linux nodes”
2. “Extracting the GPFS software on Linux nodes”
3. “Extracting GPFS patches (update SLES or Red Hat Enterprise Linux RPMs or Debian Linux
packages)” on page 38
4. “Installing the GPFS man pages on Linux nodes” on page 38
5. “Installing the GPFS software packages on Linux nodes” on page 38
6. “Verifying the GPFS installation on SLES and Red Hat Enterprise Linux nodes” on page 39
or
“Verifying the GPFS installation on Debian Linux nodes” on page 40

Accepting the electronic license agreement on Linux nodes

The GPFS software license agreement is shipped with the GPFS software and is viewable electronically.
When you extract the GPFS software, you are asked whether or not you accept the license. The electronic
license agreement must be accepted before software installation can continue. Read the software
agreement carefully before accepting the license. See “Extracting the GPFS software on Linux nodes.”

Extracting the GPFS software on Linux nodes

The GPFS software is delivered in a self-extracting archive. The self-extracting image contains the
following:
v The GPFS product installation SLES and Red Hat Enterprise Linux RPMs and Debian Linux packages

Note: Debian Linux packages are not available for System z.

v The License Acceptance Process (LAP) Tool
The LAP Tool is invoked for acceptance of the GPFS license agreements. The license agreements must
be accepted to obtain access to the GPFS product installation images.
v A version of the Java™ Runtime Environment (JRE) necessary to run the LAP Tool

Before installing GPFS, you need to extract the RPMs from the archive.
1. Copy the self-extracting product image, gpfs_install-4.1*, from the CD-ROM to a local directory
(where * is the correct version of the product for your hardware platform and Linux distribution). For
example:
cp /media/cdrom/gpfs_install-4.1.0-0_x86_64 /tmp/gpfs_install-4.1.0-0_x86_64
2. Verify that the self-extracting program has executable permissions, for example:
# ls -l /tmp/gpfs_install-4.1.0-0_x86_64

The system displays information similar to the following:

-rwxr-xr-x l root root 110885866 Apr 27 15:52 /tmp/gpfs_install-4.1.0-0_x86_64

36 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
3. Invoke the self-extracting image that you copied from the CD-ROM and accept the license agreement:
a. By default, the LAP Tool, JRE, and GPFS installation images are extracted to the target directory
/usr/lpp/mmfs/4.1.
b. The license agreement files on the media can be viewed in graphics mode or text-only mode:
v Graphics mode is the default behavior. To view the files in graphics mode, invoke
gpfs_install-4.1*. Using the graphics-mode installation requires a window manager to be
configured.
v To view the files in text-only mode, add the --text-only option. When run in text-only mode, the
output explains how to accept the agreement:
<...Last few lines of output...>
Press Enter to continue viewing the license agreement, or
enter "1" to accept the agreement, "2" to decline it, "3"
to print it, "4" to read non-IBM terms, or "99" to go back
to the previous screen.
c. You can use the --silent option to accept the license agreement automatically.
d. Use the --help option to obtain usage information from the self-extracting archive.
The following is an example of how to extract the software using text mode:
/tmp/gpfs_install_4.1.0-0_x86_64 --text-only

Upon license agreement acceptance, the GPFS product installation images are placed in the extraction
target directory (/usr/lpp/mmfs/4.1). This directory contains the following GPFS SLES and Red Hat
Enterprise Linux RPMs:
gpfs.base-4.1.*.rpm
gpfs.gpl-4.1.*.noarch.rpm
gpfs.docs-4.1.*.noarch.rpm
gpfs.msg.en_US-4.1.*.noarch.rpm
gpfs.gskit-8.0.50-16.*.rpm
gpfs.ext-4.1.*.rpm (GPFS Standard Edition and GPFS Advanced Edition only)
gpfs.crypto-4.1.*.rpm (GPFS Advanced Edition only)

and the following GPFS Debian Linux Packages:

gpfs.base_4.1.*_amd64.deb
gpfs.docs_4.1.*_all.deb
gpfs.gpl_4.1.*_all.deb
gpfs.msg.en-us_4.1.*_all.deb
gpfs.gskit-8.0.50-16.*.deb
gpfs.ext-4.1.*.deb (GPFS Standard Edition and GPFS Advanced Edition only)
gpfs.crypto-4.1.*.deb (GPFS Advanced Edition only)

In this directory there is a license subdirectory that contains license agreements in multiple languages. To
view which languages are provided, issue the following command:
# ls /usr/lpp/mmfs/4.1/license

The system displays information similar to the following:

Chinese_TW.txt French.txt Japanse.txt Notices.txt Slovenian.txt
Chinese.txt German.txt Korean.txt Polish.txt Spanish.txt
Czech.txt Greek.txt Lithuanian.txt Portguese.txt Status.dat
English.txt Italian.txt non_ibm_license.txt Russian.txt Turkish.txt

Installing GPFS on Linux nodes 37

The license agreement remains available in the extraction target directory under the license subdirectory
for future access. The license files are written using operating system-specific code pages. This enables
you to view the license in English and in the local language configured on your machine. The other
languages are not guaranteed to be viewable.

Extracting GPFS patches (update SLES or Red Hat Enterprise Linux

RPMs or Debian Linux packages)
Typically when you install a GPFS system there are patches available. It is recommended that you always
look for the latest patches when installing or updating a GPFS node.

Note: For information about restrictions pertaining to Debian Linux, see the GPFS FAQ in IBM
Knowledge Center (www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/
gpfs_faqs/gpfsclustersfaq.html).

GPFS patches (update SLES and Red Hat Enterprise Linux RPMs and Debian Linux packages) are
available from the IBM Support Portal: Downloads for General Parallel File System (www.ibm.com/
support/entry/portal/Downloads/Software/Cluster_software/General_Parallel_File_System).

The update SLES and Red Hat Enterprise Linux RPMs and Debian Linux packages are distributed in a
different form from that of the base software; they are stored in a tar file. Use the tar command to extract
the update SLES and Red Hat Enterprise Linux RPMs or Debian Linux packages into a local directory.

Recommendation: Because you need to install the base software SLES or Red Hat Enterprise Linux
RPMs or Debian Linux packages completely before installing a patch level, it is recommended that you
place update SLES or Red Hat Enterprise Linux RPMs or Debian Linux packages in a separate directory
from the base SLES or Red Hat Enterprise Linux RPMs or Debian Linux packages. This enables you to
simplify the installation process by using the rpm or dpkg command with wildcards (rpm -ivh *.rpm or
dpkg -i *.deb), first on the directory containing the base SLES or Red Hat Enterprise Linux RPMs or
Debian Linux packages, then on the directory containing the update SLES or Red Hat Enterprise Linux
RPMs or Debian Linux packages. (There is no license acceptance required on patches, so once you have
extracted the contents of the tar file, the update SLES or Red Hat Enterprise Linux RPMs or Debian Linux
packages are available for installation.)

If you are applying a patch during the initial installation of GPFS on a node, you only need to build the
portability layer once after the base and update SLES or Red Hat Enterprise Linux RPMs or Debian Linux
packages are installed.

Installing the GPFS man pages on Linux nodes

In order to use the GPFS man pages, the gpfs.docs RPM must be installed. Once you have installed the
gpfs.docs RPM, the GPFS man pages are located at /usr/share/man/.

You do not need to install the gpfs.docs RPM on all nodes if man pages are not desired (for example, if
local file system space on the node is minimal).

Installing the GPFS software packages on Linux nodes

The GPFS software is installed using the rpm command (for SLES and Red Hat Enterprise Linux) or the
dpkg command (for Debian Linux).
Required packages
The following packages are required for SLES and Red Hat Enterprise Linux:
gpfs.base-4.1.*.rpm
gpfs.gpl-4.1.*.noarch.rpm
gpfs.msg.en_US-4.1.*.noarch.rpm
gpfs.gskit-8.0.50-16.*.rpm

38 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
 gpfs.ext-4.1.*.rpm (GPFS Standard Edition and GPFS Advanced Edition only)
gpfs.crypto-4.1.*.rpm (GPFS Advanced Edition only)
The following packages are required for Debian Linux:
gpfs.base-4.1.*.deb
gpfs.gpl-4.1.*all.deb
gpfs.msg.en_US-4.1.*all.deb
gpfs.gskit-8.0.50-16.*.deb
gpfs.ext-4.1.*.deb (GPFS Standard Edition and GPFS Advanced Edition only)
gpfs.crypto-4.1.*.deb (GPFS Advanced Edition only)
Optional packages
The following package is optional for SLES and Red Hat Enterprise Linux:
gpfs.docs-4.1.*noarch.rpm
The following package is optional for Debian Linux:
gpfs.docs-4.1.*all.deb
The following package is required only if GPFS Native RAID on Power® 775 will be used:
gpfs.gnr-4.1.*.ppc64.rpm
The following packages are required (and provided) only on the IBM System x® GPFS Storage
Server (GSS):
gpfs.gnr-4.1.*.x86_64.rpm
gpfs.platform-4.1.*.x86_64.rpm
gpfs.gss.firmware-4.1.*.x86_64.rpm

To install all of the GPFS SLES or Red Hat Enterprise Linux RPMs, issue the following command:
rpm -ivh /usr/lpp/mmfs/4.1/gpfs*.rpm

To install all of the GPFS Debian Linux packages, issue the following command:
dpkg -i /usr/lpp/mmfs/4.1/gpfs*.deb

Verifying the GPFS installation on SLES and Red Hat Enterprise Linux
nodes
You can verify the installation of the GPFS SLES or Red Hat Enterprise Linux RPMs on each node. To
check that the software has been successfully installed, use the rpm command:
rpm -qa | grep gpfs

The system should return output similar to the following:

gpfs.docs-4.1.0-0
gpfs.base-4.1.0-0
gpfs.msg.en_US-4.1.0-0
gpfs.gpl-4.1.0-0
gpfs.gskit-8.0.50-16

If you have the GPFS Standard Edition or the GPFS Advanced Edition installed, you should also see the
following line in the output:
gpfs.ext-4.1.0-0

If you have the GPFS Advanced Edition installed, you should also see the following line in the output:
gpfs.crypto-4.1.0-0

For installations that include GPFS Native RAID, you should also see the following line in the output:

Installing GPFS on Linux nodes 39

 gpfs.gnr-4.1.0-0

Verifying the GPFS installation on Debian Linux nodes

You can verify the installation of the GPFS Debian Linux packages on each node. To check that the
software has been successfully installed, use the dpkg command:
dpkg -l | grep gpfs

The system should return output similar to the following:

ii gpfs.base 4.1.0-0 GPFS File Manager
ii gpfs.docs 4.1.0-0 GPFS Server Manpages and Documentation
ii gpfs.gpl 4.1.0-0 GPFS Open Source Modules
ii gpfs.gskit 8.0.6-50 GPFS GSKit Cryptography Runtime
ii gpfs.msg.en_US 4.1.0-0 GPFS Server Messages - U.S. English

If you have the GPFS Standard Edition or the GPFS Advanced Edition installed, you should also see the
following line in the output:
ii gpfs.ext 4.1.0-0 GPFS Extended Features

If you have the GPFS Advanced Edition installed, you should also see the following line in the output:
ii gpfs.crypto 4.1.0-0 GPFS Cryptographic Subsystem

Building the GPFS portability layer on Linux nodes

Before starting GPFS, you must build and install the GPFS portability layer.

The GPFS portability layer is a loadable kernel module that allows the GPFS daemon to interact with the
operating system.

Note: The GPFS kernel module should be updated any time the Linux kernel is updated. Updating the
GPFS kernel module after a Linux kernel update requires rebuilding and installing a new version of the
module.

To build the GPFS portability layer on Linux nodes, do the following:

1. Check for the following before building the portability layer:
v Updates to the portability layer at the IBM Support Portal: Downloads for General Parallel File
System (www.ibm.com/support/entry/portal/Downloads/Software/Cluster_software/
General_Parallel_File_System).
v The latest kernel levels supported in the GPFS FAQ in IBM Knowledge Center
(www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/
gpfsclustersfaq.html).
2. Build your GPFS portability layer in one of the following ways:
v Using the mmbuildgpl command (recommended). See the GPFS: Administration and Programming
Reference for more information.
v Using the Autoconfig tool.
v Using the directions in /usr/lpp/mmfs/src/README.

Using the automatic configuration tool to build the GPFS portability

layer on Linux nodes
To simplify the build process, GPFS provides an automatic configuration tool.

| To build the GPFS portability layer using this tool, enter the following command:
| /usr/lpp/mmfs/bin/mmbuildgpl

40 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
| Each kernel module is specific to a Linux version and platform. If you have multiple nodes running
| exactly the same operating system level on the same platform, and only some of these nodes have a
| compiler available, you can build the kernel module on one node, then create an RPM that contains the
| binary module for ease of distribution.

If you choose to generate an RPM package for portability layer binaries, perform the following additional
step:
| /usr/lpp/mmfs/bin/mmbuildgpl --buildrpm

When the command finishes, it displays the location of the generated RPM:
<...Last line of output...>
Wrote: /usr/src/redhat/RPMS/x86_64/gpfs.gplbin-2.6.18-128.1.14.e15-3.3.0-1.x86_64.rpm

You can then copy the generated RPM package to other machines for deployment. The generated RPM
can only be deployed to machines with identical architecture, distribution level, Linux kernel, and GPFS
maintenance level.

Note: During the package generation, temporary files are written to the /tmp/rpm directory, so be sure
there is sufficient space available. By default, the generated RPM goes to /usr/src/packages/RPMS/<arch>
for SUSE Linux Enterprise Server and /usr/src/redhat/RPMS/<arch> for Red Hat Enterprise Linux.

For Linux on System z: Changing the kernel settings

In order for GPFS to run on Linux on System z, the kernel settings need to be changed.

Before starting GPFS, perform the following steps on each Linux on System z node:
1. In the /etc/zipl.conf file, add vmalloc=4096G user_mode=home as shown in the following example:
(10:25:41) dvtc1a:~ # cat /etc/zipl.conf
# Modified by YaST2. Last modification on Mon May 19 09:39:04 EDT 2014
[defaultboot]
defaultmenu = menu

###Don’t change this comment - YaST2 identifier: Original name: linux###

[SLES11_SP3_2]
image = /boot/image-3.0.101-0.15-default
target = /boot/zipl
ramdisk = /boot/initrd-3.0.101-0.15-default,0x2000000
parameters = "root=/dev/mapper/mpatha_part2 hvc_iucv=8 TERM=dumb
resume=/dev/mapper/mpatha_part1 crashkernel=258M-:129M vmalloc=4096G
user_mode=home"

Note: For SUSE Linux Enterprise Server (SLES) 11 and Red Hat Enterprise Linux 6, user_mode=home is
optional.
2. Run the zipl command.

Note: For information about the zipl command, see the “Initial program loader for System z -zipl”
topic in Device Drivers, Features, and Commands (www.ibm.com/support/knowledgecenter/api/
content/linuxonibm/liaaf/lnz_r_dd.html) in the Linux on System z library overview.
3. Reboot the node.

| Note: For more detailed information about installation and startup of GPFS on System z, see the “Getting
| started with Elastic Storage for Linux on System z based on GPFS technology” white paper, available on
| the Welcome Page forGPFS in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/
| SSFKCN/gpfs_welcome.html).

Installing GPFS on Linux nodes 41

42 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Accessibility features for GPFS
Accessibility features help users who have a disability, such as restricted mobility or limited vision, to use
information technology products successfully.

Accessibility features
The following list includes the major accessibility features in GPFS:
v Keyboard-only operation
v Interfaces that are commonly used by screen readers
v Keys that are discernible by touch but do not activate just by touching them
v Industry-standard devices for ports and connectors
v The attachment of alternative input and output devices

IBM Knowledge Center, and its related publications, are accessibility-enabled. The accessibility features
are described in IBM Knowledge Center (www.ibm.com/support/knowledgecenter).

Keyboard navigation
This product uses standard Microsoft Windows navigation keys.

IBM and accessibility

See the IBM Human Ability and Accessibility Center (www.ibm.com/able) for more information about
the commitment that IBM has to accessibility.

© Copyright IBM Corp. 2014 43

44 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Notices
This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.
Consult your local IBM representative for information on the products and services currently available in
your area. Any reference to an IBM product, program, or service is not intended to state or imply that
only that IBM product, program, or service may be used. Any functionally equivalent product, program,
or service that does not infringe any IBM intellectual property right may be used instead. However, it is
the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or
service.

IBM may have patents or pending patent applications covering subject matter described in this
document. The furnishing of this document does not grant you any license to these patents. You can send
license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property
Department in your country or send inquiries, in writing, to:

Intellectual Property Licensing

Legal and Intellectual Property Law
IBM Japan Ltd.
19-21, Nihonbashi-Hakozakicho, Chuo-ku
Tokyo 103-8510, Japan

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"

WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied
warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically
made to the information herein; these changes will be incorporated in new editions of the publication.
IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this
publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in
any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of
the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.

© Copyright IBM Corp. 2014 45

Licensees of this program who wish to have information about it for the purpose of enabling: (i) the
exchange of information between independently created programs and other programs (including this
one) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Corporation
Dept. 30ZA/Building 707
Mail Station P300
2455 South Road,
Poughkeepsie, NY 12601-5400
U.S.A.

Such information may be available, subject to appropriate terms and conditions, including in some cases,
payment or a fee.

The licensed program described in this document and all licensed material available for it are provided
by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or
any equivalent agreement between us.

Any performance data contained herein was determined in a controlled environment. Therefore, the
results obtained in other operating environments may vary significantly. Some measurements may have
been made on development-level systems and there is no guarantee that these measurements will be the
same on generally available systems. Furthermore, some measurements may have been estimated through
extrapolation. Actual results may vary. Users of this document should verify the applicable data for their
specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their
published announcements or other publicly available sources. IBM has not tested those products and
cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of
those products.

This information contains examples of data and reports used in daily business operations. To illustrate
them as completely as possible, the examples include the names of individuals, companies, brands, and
products. All of these names are fictitious and any similarity to the names and addresses used by an
actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs
in any form without payment to IBM, for the purposes of developing, using, marketing or distributing
application programs conforming to the application programming interface for the operating platform for
which the sample programs are written. These examples have not been thoroughly tested under all
conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these
programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not be
liable for any damages arising out of your use of the sample programs.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

46 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Intel is a trademark of Intel Corporation or its subsidiaries in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or
its affiliates.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States,
other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Notices 47
48 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Glossary
This glossary provides terms and definitions for D
the GPFS product.
Data Management Application Program
Interface (DMAPI)
The following cross-references are used in this
The interface defined by the Open
glossary:
Group's XDSM standard as described in
v See refers you from a nonpreferred term to the the publication System Management: Data
preferred term or from an abbreviation to the Storage Management (XDSM) API Common
spelled-out form. Application Environment (CAE) Specification
v See also refers you to a related or contrasting C429, The Open Group ISBN
term. 1-85912-190-X.
deadman switch timer
For other terms and definitions, see the IBM
A kernel timer that works on a node that
Terminology website (www.ibm.com/software/
has lost its disk lease and has outstanding
globalization/terminology) (opens in new
I/O requests. This timer ensures that the
window).
node cannot complete the outstanding
I/O requests (which would risk causing
B
file system corruption), by causing a
block utilization panic in the kernel.
The measurement of the percentage of
dependent fileset
used subblocks per allocated blocks.
A fileset that shares the inode space of an
existing independent fileset.
C
disk descriptor
cluster
A definition of the type of data that the
A loosely-coupled collection of
disk contains and the failure group to
independent systems (nodes) organized
which this disk belongs. See also failure
into a network for the purpose of sharing
group.
resources and communicating with each
other. See also GPFS cluster. disk leasing
A method for controlling access to storage
cluster configuration data
devices from multiple host systems. Any
The configuration data that is stored on
host that wants to access a storage device
the cluster configuration servers.
configured to use disk leasing registers
cluster manager for a lease; in the event of a perceived
The node that monitors node status using failure, a host system can deny access,
disk leases, detects failures, drives preventing I/O operations with the
recovery, and selects file system storage device until the preempted system
managers. The cluster manager is the has reregistered.
node with the lowest node number
disposition
among the quorum nodes that are
The session to which a data management
operating at a particular time.
event is delivered. An individual
control data structures disposition is set for each type of event
Data structures needed to manage file from each file system.
data and metadata cached in memory.
domain
Control data structures include hash
A logical grouping of resources in a
tables and link pointers for finding
network for the purpose of common
cached data; lock states and tokens to
management and administration.
implement distributed locking; and
various flags and sequence numbers to
keep track of updates to the cached data.

© Copyright IBM Corp. 2014 49

E fileset snapshot
A snapshot of an independent fileset plus
ECKD See extended count key data (ECKD).
all dependent filesets.
ECKD device
file clone
See extended count key data device (ECKD
A writable snapshot of an individual file.
device).
file encryption key (FEK)
encryption key
A key used to encrypt sectors of an
A mathematical value that allows
individual file. See also encryption key.
components to verify that they are in
communication with the expected server. file-management policy
Encryption keys are based on a public or A set of rules defined in a policy file that
private key pair that is created during the GPFS uses to manage file migration and
installation process. See also file encryption file deletion. See also policy.
key, master encryption key.
file-placement policy
extended count key data (ECKD) A set of rules defined in a policy file that
An extension of the count-key-data (CKD) GPFS uses to manage the initial
architecture. It includes additional placement of a newly created file. See also
commands that can be used to improve policy.
performance.
file system descriptor
extended count key data device (ECKD device) A data structure containing key
A disk storage device that has a data information about a file system. This
transfer rate faster than some processors information includes the disks assigned to
can utilize and that is connected to the the file system (stripe group), the current
processor through use of a speed state of the file system, and pointers to
matching buffer. A specialized channel key files such as quota files and log files.
program is needed to communicate with
file system descriptor quorum
such a device. See also fixed-block
The number of disks needed in order to
architecture disk device.
write the file system descriptor correctly.
F file system manager
The provider of services for all the nodes
failback
using a single file system. A file system
Cluster recovery from failover following
manager processes changes to the state or
repair. See also failover.
description of the file system, controls the
failover regions of disks that are allocated to each
(1) The assumption of file system duties node, and controls token management
by another node when a node fails. (2) and quota management.
The process of transferring all control of
fixed-block architecture disk device (FBA disk
the ESS to a single cluster in the ESS
device)
when the other clusters in the ESS fails.
A disk device that stores data in blocks of
See also cluster. (3) The routing of all
fixed size. These blocks are addressed by
transactions to a second controller when
block number relative to the beginning of
the first controller fails. See also cluster.
the file. See also extended count key data
failure group device.
A collection of disks that share common
fragment
access paths or adapter connection, and
The space allocated for an amount of data
could all become unavailable through a
too small to require a full block. A
single hardware failure.
fragment consists of one or more
FEK See file encryption key. subblocks.
fileset A hierarchical grouping of files managed
G
as a unit for balancing workload across a
cluster. See also dependent fileset, global snapshot
independent fileset. A snapshot of an entire GPFS file system.

50 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
GPFS cluster which are important for running intranet
A cluster of nodes defined as being and other high-performance e-business
available for use by GPFS file systems. file servers.
GPFS portability layer junction
The interface module that each A special directory entry that connects a
installation must build for its specific name in a directory of one fileset to the
hardware platform and Linux root directory of another fileset.
distribution.
K
GPFS recovery log
A file that contains a record of metadata kernel The part of an operating system that
activity, and exists for each node of a contains programs for such tasks as
cluster. In the event of a node failure, the input/output, management and control of
recovery log for the failed node is hardware, and the scheduling of user
replayed, restoring the file system to a tasks.
consistent state and allowing other nodes
to continue working. M
master encryption key (MEK)
I
A key used to encrypt other keys. See also
ill-placed file encryption key.
A file assigned to one storage pool, but
MEK See master encryption key.
having some or all of its data in a
different storage pool. metadata
A data structures that contain access
ill-replicated file
information about file data. These include:
A file with contents that are not correctly
inodes, indirect blocks, and directories.
replicated according to the desired setting
These data structures are not accessible to
for that file. This situation occurs in the
user applications.
interval between a change in the file's
replication settings or suspending one of metanode
its disks, and the restripe of the file. The one node per open file that is
responsible for maintaining file metadata
independent fileset
integrity. In most cases, the node that has
A fileset that has its own inode space.
had the file open for the longest period of
indirect block continuous time is the metanode.
A block containing pointers to other
mirroring
blocks.
The process of writing the same data to
inode The internal structure that describes the multiple disks at the same time. The
individual files in the file system. There is mirroring of data protects it against data
one inode for each file. loss within the database or within the
recovery log.
inode space
A collection of inode number ranges multi-tailed
reserved for an independent fileset, which A disk connected to multiple nodes.
enables more efficient per-fileset
functions. N
ISKLM namespace
IBM Security Key Lifecycle Manager. For Space reserved by a file system to contain
GPFS encryption, the ISKLM is used as an the names of its objects.
RKM server to store MEKs.
Network File System (NFS)
A protocol, developed by Sun
J
Microsystems, Incorporated, that allows
journaled file system (JFS) any host in a network to gain access to
A technology designed for another host or netgroup and their file
high-throughput server environments, directories.

Glossary 51
Network Shared Disk (NSD) operating systems without requiring
A component for cluster-wide disk changes to the source code.
naming and access.
primary GPFS cluster configuration server
NSD volume ID In a GPFS cluster, the node chosen to
A unique 16 digit hex number that is maintain the GPFS cluster configuration
used to identify and access all NSDs. data.
node An individual operating-system image private IP address
within a cluster. Depending on the way in A IP address used to communicate on a
which the computer system is partitioned, private network.
it may contain one or more nodes.
public IP address
node descriptor A IP address used to communicate on a
A definition that indicates how GPFS uses public network.
a node. Possible functions include:
manager node, client node, quorum node, Q
and nonquorum node.
quorum node
node number A node in the cluster that is counted to
A number that is generated and determine whether a quorum exists.
maintained by GPFS as the cluster is
quota The amount of disk space and number of
created, and as nodes are added to or
inodes assigned as upper limits for a
deleted from the cluster.
specified user, group of users, or fileset.
node quorum
quota management
The minimum number of nodes that must
The allocation of disk blocks to the other
be running in order for the daemon to
nodes writing to the file system, and
start.
comparison of the allocated space to
node quorum with tiebreaker disks quota limits at regular intervals.
A form of quorum that allows GPFS to
run with as little as one quorum node R
available, as long as there is access to a
Redundant Array of Independent Disks (RAID)
majority of the quorum disks.
A collection of two or more disk physical
non-quorum node drives that present to the host an image
A node in a cluster that is not counted for of one or more logical disk drives. In the
the purposes of quorum determination. event of a single physical device failure,
the data can be read or regenerated from
P the other disk drives in the array due to
data redundancy.
policy A list of file-placement, service-class, and
encryption rules that define characteristics recovery
and placement of files. Several policies The process of restoring access to file
can be defined within the configuration, system data when a failure has occurred.
but only one policy set is active at one Recovery can involve reconstructing data
time. or providing alternative routing through a
different server.
policy rule
A programming statement within a policy remote key management server (RKM server)
that defines a specific action to be A server that is used to store master
performed. encryption keys.
pool A group of resources with similar replication
characteristics and attributes. The process of maintaining a defined set
of data in more than one location.
portability
Replication involves copying designated
The ability of a programming language to
changes for one location (a source) to
compile successfully on different
another (a target), and synchronizing the
data in both locations.

52 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
RGD Recovery group data. source node
The node on which a data management
RKM server
event is generated.
See remote key management server.
stand-alone client
rule A list of conditions and actions that are
The node in a one-node cluster.
triggered when certain conditions are met.
Conditions include attributes about an storage area network (SAN)
object (file name, type or extension, dates, A dedicated storage network tailored to a
owner, and groups), the requesting client, specific environment, combining servers,
and the container name associated with storage products, networking products,
the object. software, and services.
storage pool
S A grouping of storage space consisting of
SAN-attached volumes, logical unit numbers (LUNs), or
Disks that are physically attached to all addresses that share a common set of
nodes in the cluster using Serial Storage administrative characteristics.
Architecture (SSA) connections or using
stripe group
Fibre Channel switches.
The set of disks comprising the storage
Scale Out Backup and Restore (SOBAR) assigned to a file system.
A specialized mechanism for data
striping
protection against disaster only for GPFS
A storage process in which information is
file systems that are managed by Tivoli®
split into blocks (a fixed amount of data)
Storage Manager (TSM) Hierarchical
and the blocks are written to (or read
Storage Management (HSM).
from) a series of disks in parallel.
secondary GPFS cluster configuration server
subblock
In a GPFS cluster, the node chosen to
The smallest unit of data accessible in an
maintain the GPFS cluster configuration
I/O operation, equal to one thirty-second
data in the event that the primary GPFS
of a data block.
cluster configuration server fails or
becomes unavailable. system storage pool
A storage pool containing file system
Secure Hash Algorithm digest (SHA digest)
control structures, reserved files,
A character string used to identify a GPFS
directories, symbolic links, special devices,
security key.
as well as the metadata associated with
session failure regular files, including indirect blocks and
The loss of all resources of a data extended attributes The system storage
management session due to the failure of pool can also contain user data.
the daemon on the session node.
T
session node
The node on which a data management token management
session was created. A system for controlling file access in
which each application performing a read
Small Computer System Interface (SCSI)
or write operation is granted some form
An ANSI-standard electronic interface
of access to a specific block of file data.
that allows personal computers to
Token management provides data
communicate with peripheral hardware,
consistency and controls conflicts. Token
such as disk drives, tape drives, CD-ROM
management has two components: the
drives, printers, and scanners faster and
token management server, and the token
more flexibly than previous interfaces.
management function.
snapshot
token management function
An exact copy of changed data in the
A component of token management that
active files and directories of a file system
requests tokens from the token
or fileset at a single point in time. See also
fileset snapshot, global snapshot.

Glossary 53
 management server. The token
management function is located on each
cluster node.
token management server
A component of token management that
controls tokens relating to the operation
of the file system. The token management
server is located at the file system
manager node.
twin-tailed
A disk connected to two nodes.

U
user storage pool
A storage pool containing the blocks of
data that make up user files.

V
VCD See vdisk configuration data.
VFS See virtual file system.
vdisk configuration data (VCD)
Configuration data associated with a disk.
virtual file system (VFS)
A remote file system that has been
mounted so that it is accessible to the
local user.
virtual node (vnode)
The structure that contains information
about a file system object in a virtual file
system (VFS).

54 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Index
A DASD for NSDs, preparing 19
data
access control lists (ACLs) guarding against failure of a path to a disk 9
file system authorization 27 recoverability 5
accessibility features for the GPFS product 43 replication 27
adapter Data Management API (DMAPI)
invariant address requirement 3 enabling 30
atime value 26 Debian Linux packages (update), extracting 38
autoload attribute 14 default quotas
description 29
dfcommand (specifying whether it will report numbers based
B on quotas for the fileset) 31
bin directory 35 direct access storage devices (DASD) for NSDs, preparing 19
block directory, bin 35
size 25 disk descriptor replica 19
block allocation map 26 disk usage
verifying 30
disks
C considerations 15
failure 8
commands mmcrfs command 24
failure of 13 stanza files 17
mmchcluster 13, 14 storage area network 15
mmcheckquota 30 documentation
mmchfs 21, 27, 28, 29 installing man pages on Linux nodes 38
mmcrcluster 11, 13, 14
mmcrfs 21, 27, 28, 29
mmcrnsd 17
mmdefedquota 29 E
mmdefquotaon 29 ECKD devices, preparing environment for 19
mmdelnsd 17 electronic license agreement
mmedquota 29, 30 Linux nodes 36
mmlsdisk 19 enabling file system features 31
mmlsquota 29, 30 environment for ECKD devices, preparing 19
mmrepquota 29, 30 environment, preparing 35
mmstartup 14 estimated node count 28
mmwinservctl 13, 14 extracting GPFS patches 38
remote file copy extracting the GPFS software 36
rcp 14 extracting update Debian Linux packages 38
scp 14 extracting update SUSE Linux Enterprise Server and Red Hat
remote shell Enterprise Linux RPMs 38
rsh 13
ssh 13
communication F
GPFS daemon to daemon 12 failure
invariant address requirement 3 disk 8
configuration Network Shared Disk server 8
of a GPFS cluster 11 node 5
configuration and tuning settings failure groups 19
configuration file 15 loss of 19
default values 15 preventing loss of data access 8
controlling the order in which file systems are mounted 32 use of 19
created files (maximum number) 31 file system descriptor 19
creating GPFS directory failure groups 19
/tmp/gpfslpp on Linux nodes 36 inaccessible 19
quorum 19
file system features
D enabling 31
daemon file system manager
communication 12 internal log file 27
starting 14 NSD creation considerations 18

© Copyright IBM Corp. 2014 55

file system manager (continued) installing GPFS on Linux nodes (continued)
windows drive letter 28 creating the GPFS directory 36
file systems electronic license agreement 36
authorization 27 installing the software packages 38
block size 25 License Acceptance Process (LAP) Tool 36
creating 21 man pages 38
device name 24 procedure for 36
disk descriptor 24 verifying the GPFS installation 39
enabling DMAPI 30 what to do before you install GPFS 35
internal log file 27 invariant address adapter
last time accessed 26 requirement 3
list of disk descriptors 27
maximum number of files 31
mount options 29
mounting 24, 30
L
latest level of file system
mountpoint 29
migrating 31
number of nodes mounted by 28
License Acceptance Process (LAP) Tool 36
quotas 29
license designation 15
recoverability parameters 27
license inquiries 45
sample creation 32
Linux
sizing 21
installation instructions for GPFS 35
time last modified 26
installing GPFS 36
windows drive letter 28
kernel requirement 4
file systems (controlling the order in which they are
prerequisite software 35
mounted) 32
Linux on System z, DASD tested with 19
files
Linux on System z, running GPFS 41
maximum number of 31
files that can be created, maximum number of 31
filesetdf option 31
fragments, storage of files 25 M
man pages
installing on Linux nodes 38
G maximum number of files 31
maximum number of files that can be created 31
GPFS administrative adapter port name 11
metadata
GPFS clusters
replication 27
administration adapter port name 11
migrating file system format to the latest level 31
configuration file 15
mmchcluster command 13, 14
configuration servers 13
mmcheckquota command 30
creating 11
mmchfs command 21, 27, 28, 29
daemon
mmcrcluster command 11, 13, 14
starting 14
mmcrfs command 21, 27, 28, 29
naming 14
mmcrnsd command 17
nodes in the cluster 12
mmdefedquota command 29
planning nodes 11
mmdefquotaon command 29
portability layer 40
mmdelnsd command 17
server nodes 11
mmedquota command 29, 30
starting the GPFS daemon 14
mmlsdisk command 19
user ID domain 14
mmlsquota command 29, 30
GPFS communications adapter port name 11
mmrepquota command 29, 30
GPFS for Linux on System z, running 41
mmstartup command 14
GPFS license designation 15
mmwinservctl command 13, 14
GPFS patches, extracting 38
mount options 29
GPFS product structure 4
mount-priority option 32
GPFS, planning for 3
mounting a file system 24, 29, 30
mounting of file systems, controlling the order of the 32
mountpoint 29
H mtime values 26
hard limit, quotas 29 Multiple Path I/O (MPIO)
hardware requirements 3 utilizing 9

I N
installing GPFS (Debian) Network File System (NFS)
verifying the GPFS installation 40 access control lists 27
installing GPFS on Linux nodes deny-write open lock 24
building your GPFS portability layer 40

56 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)
Network Shared Disk (NSD) quotas (continued)
creation of 17 values reported in a replicated file system 29
server disk considerations 15
server failure 8
server node considerations 18
node quorum
R
rcp command 14
definition of 6
recoverability
selecting nodes 8
disk failure 8
node quorum with tiebreaker disks
node failure 5
definition of 6
parameters 5
selecting nodes 8
recovery time
nodes
reducing with Persistent Reserve 10
descriptor form 12
reduced recovery time using Persistent Reserve 10
designation as manager or client 12
Redundant Array of Independent Disks (RAID)
estimating the number of 28
block size considerations 25
failure 5
preventing loss of data access 8
in a GPFS cluster 11
remote command environment
in the GPFS cluster 12
rcp 14
quorum 12
rsh 13
nofilesetdf option 31
scp 14
notices 45
ssh 13
number of files that can be created, maximum 31
replication
affect on quotas 29
preventing loss of data access 8
O reporting numbers based on quotas for the fileset 31
order in which file systems are mounted, controlling the 32 requirements
hardware 3
software 4
P RPMs (update), extracting SUSE Linux Enterprise Server and
Linux 38
patches (GPFS), extracting 38
rsh command 13
patches (GPFS), extracting SUSE Linux Enterprise Server and
running GPFS for Linux on System z 41
Red Hat Enterprise Linux 38
patent information 45
Persistent Reserve
reduced recovery time 10 S
planning considerations 3 scp command 14
cluster creation 11 shell PATH 35
disks 15 sizing file systems 21
file system creation 21 soft limit, quotas 29
GPFS license designation 15 softcopy documentation 38
GPFS product structure 4 software requirements 4
hardware requirements 3 Specifying whether the dfcommand will report numbers based
recoverability 5 on quotas for the fileset 31
software requirements 4 ssh command 13
portability layer starting GPFS 14
building 40 Storage Area Network (SAN)
preparing direct access storage devices (DASD) for NSDs 19 disk considerations 15
preparing environment for ECKD devices 19 strict replication 27
preparing the environment 35 subblocks, use of 25
programming specifications SUSE Linux Enterprise Server and Linux RPMs (update),
Linux prerequisite software 35 extracting 38
verifying prerequisite software 35 System z, DASD tested with Linux on 19
System z, running GPFS for Linux on 41

Q
quorum T
definition of 6 tiebreaker disks 6
during node failure 5 trademarks 46
file system descriptor 19
selecting nodes 8
quotas
default quotas 29
U
update Debian Linux packages, extracting 38
description 29
update SUSE Linux Enterprise Server and Red hat Enterprise
in a replicated system 29
Linux RPMs, extracting 38
mounting a file system with quotas enabled 30
system files 30

Index 57
V
verifying
GPFS for Linux installation 39
GPFS installation (Debian) 40
prerequisite software for Linux nodes 35
verifying disk usage 30

W
windows drive letter 28

58 GPFS Documentation Update: GPFS 4.1.0.5 Information (Applied to GPFS 4.1.0.4 Information Units)



Product Number: 5725-Q01

5641-GPF
5641-GP6
5641-GP7
5641-GP8
5641-S28

Printed in USA