Platform MPI™ 5.6.

5 User’s Guide

Version 5.6.5
September 29, 2008 11:58 am
Comments to: [email protected]
Support: [email protected]
Copyright © 1994-2008, Platform Computing Inc.
Although the information in this document has been carefully reviewed, Platform Computing Inc.
(“Platform”) does not warrant it to be free of errors or omissions. Platform reserves the right to make
corrections, updates, revisions or changes to the information in this document.
UNLESS OTHERWISE EXPRESSLY STATED BY PLATFORM, THE PROGRAM DESCRIBED
IN THIS DOCUMENT IS PROVIDED “AS IS” AND WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN
NO EVENT WILL PLATFORM COMPUTING BE LIABLE TO ANYONE FOR SPECIAL,
COLLATERAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING WITHOUT
LIMITATION ANY LOST PROFITS, DATA, OR SAVINGS, ARISING OUT OF THE USE OF OR
INABILITY TO USE THIS PROGRAM.
We’d like to hear from you You can help us make this document better by telling us what you think of the content, organization,
and usefulness of the information. If you find an error, or just want to make a suggestion for
improving this document, please address your comments to [email protected].
Your comments should pertain only to Platform documentation. For product support, contact
[email protected].
Document redistribution This document is protected by copyright and you may not redistribute or translate it into another
and translation language, in part or in whole.
Internal redistribution You may only redistribute this document internally within your organization (for example, on an
intranet) provided that you continue to check the Platform Web site for updates and update your
version of the documentation. You may not make it available to your organization over the Internet.
Trademarks LSF is a registered trademark of Platform Computing Inc. in the United States and in other
jurisdictions.
ACCELERATING INTELLIGENCE, PLATFORM COMPUTING, PLATFORM SYMPHONY,
PLATFORM JOBSCHEDULER, PLATFORM ENTERPRISE GRID ORCHESTRATOR,
PLATFORM EGO, and the PLATFORM and PLATFORM LSF logos are trademarks of Platform
Computing Inc. in the United States and in other jurisdictions.
UNIX is a registered trademark of The Open Group in the United States and in other jurisdictions.
Microsoft is either a registered trademark or a trademark of Microsoft Corporation in the United
States and/or other countries.
Windows is a registered trademark of Microsoft Corporation in the United States and other countries.
Other products or services mentioned in this document are identified by the trademarks or service
marks of their respective owners.
Third-party license http://www.platform.com/Company/third.part.license.htm
agreements
Third-party copyright http://www.platform.com/Company/Third.Party.Copyright.htm
notices
Table of Contents
Chapter 1 Introduction ..............................................................................................1
1.1 Platform MPI product context ................................................................................. 1
1.2 Support .............................................................................................................. 2
1.2.1 Platform MPI FAQ ............................................................................................ 2
1.2.2 Platform MPI release documents ........................................................................ 2
1.2.3 Problem reports .............................................................................................. 2
1.2.4 Platforms supported ......................................................................................... 3
1.2.5 Licensing ........................................................................................................ 3
1.2.6 Feedback ........................................................................................................ 3
1.3 How to read this guide .......................................................................................... 3
1.3.1 Acronyms and abbreviations ............................................................................ 4
1.3.2 Terms ............................................................................................................ 5
1.3.3 Typographic conventions .................................................................................. 6

Chapter 2 Using Platform MPI with Unix ....................................................................7

2.1 Setting up a Platform MPI environment ................................................................... 7
2.2 Compiling and linking ............................................................................................ 7
2.2.1 Using mpicc / mpic++ / mpif77 ......................................................................... 7
2.2.2 Using gnu compiler .......................................................................................... 8
2.2.3 Notes on compiling and linking on various plattforms ............................................ 9
2.2.3.1 Compiling and linking on AMD64 and EM64T ................................................... 9
2.2.3.2 Compiling and linking on Power series ........................................................... 9
2.2.4 Compiling with Platform MPI-IO ........................................................................10
2.3 Running Platform MPI programs ............................................................................10
2.3.1 Naming conventions ........................................................................................10
2.3.2 mpimon - monitor program ..............................................................................11
2.3.2.1 Basic usage ...............................................................................................11
2.3.2.2 Identity of parallel processes .......................................................................12
2.3.2.3 Controlling options to mpimon .....................................................................12
2.3.2.4 Standard input ..........................................................................................13
2.3.2.5 Standard output ........................................................................................13
2.3.2.6 How to provide options to mpimon ...............................................................14
2.3.2.7 Network options ........................................................................................14
2.3.3 mpirun - wrapper script ...................................................................................16
2.3.3.1 mpirun usage ............................................................................................16
2.3.4 Suspending and resuming jobs .........................................................................17
2.3.5 Running with dynamic interconnect failover capabilities .......................................18
2.3.6 Debugging and profiling ..................................................................................18
2.3.7 Debugging with a sequential debugger ..............................................................18
2.3.8 Built-in-tools for debugging ..............................................................................19
2.3.9 Using built-in segment protect violation handler, .................................................20
2.3.10 Assistance for external profiling ......................................................................20
2.3.11 Debugging with Etnus Totalview .....................................................................20
2.3.12 Controlling communication resources ...............................................................21
2.3.12.1 Communication resources on DAT-devices ...................................................21
2.3.13 Good programming practice with Platform MPI ..................................................22
2.3.13.1 Matching MPI_Recv() with MPI_Probe() .......................................................22
2.3.13.2 MPI_Isend(), MPI_Irecv() ..........................................................................22
2.3.13.3 MPI_Bsend() ...........................................................................................22

Platform MPI 5.6.5 User’s Guide Page iii

 2.3.13.4 Avoid starving MPI-processes - fairness ......................................................23
2.3.13.5 Unsafe MPI programs ...............................................................................23
2.3.13.6 Name space pollution ...............................................................................23
2.3.14 Error and warning messages ..........................................................................24
2.3.14.1 User interface errors and warnings .............................................................24
2.3.14.2 Fatal errors .............................................................................................24
2.3.14.3 mpimon options .......................................................................................24
2.3.14.4 Giving numeric values to mpimon ...............................................................26

Chapter 3 Using Platform MPI in Windows ...............................................................27

3.1 Setting up a Platform MPI environment ..................................................................27
3.1.1 Platform MPI environment variables ..................................................................27
3.2 Compiling and linking ...........................................................................................27
3.2.1 Compiler support ............................................................................................27
3.2.2 Linker flags ....................................................................................................28
3.2.2.1 Notes on compiling with MPI-2 features ........................................................28
3.2.2.2 Testing the compilation ...............................................................................28
3.3 Running Platform MPI programs ............................................................................29
3.3.1 mpimon - monitor program ..............................................................................29
3.3.1.1 Basic usage ...............................................................................................29
3.3.1.2 Identity of parallel processes .......................................................................30
3.3.1.3 Controlling options to mpimon .....................................................................30
3.3.1.4 Standard input ..........................................................................................31
3.3.1.5 Standard output ........................................................................................31
3.3.1.6 How to provide options to mpimon ...............................................................31
3.3.1.7 Network options ........................................................................................32
3.4 Debugging and profiling .......................................................................................32
3.4.1 Debugging with a sequential debugger ..............................................................32
3.4.1.1 Built-in-tools for debugging .........................................................................33
3.4.1.2 Using built-in segment protect violation handler .............................................33
3.4.2 Assistance for external profiling ........................................................................33
3.5 Controlling communication resources .....................................................................34
3.5.1 Communication resources on DAT-devices .........................................................34
3.6 Good programming practice with Platform MPI .......................................................35
3.6.1 Matching MPI_Recv() with MPI_Probe() .............................................................35
3.6.2 MPI_Isend(), MPI_Irecv() ................................................................................35
3.6.3 MPI_Bsend() ..................................................................................................36
3.6.4 Avoid starving MPI-processes - fairness .............................................................36
3.6.5 Unsafe MPI programs ......................................................................................36
3.6.6 Name space pollution ......................................................................................36
3.7 Error and warning messages .................................................................................37
3.7.1 User interface errors and warnings ....................................................................37
3.7.2 Fatal errors ....................................................................................................37
3.7.3 mpimon options .............................................................................................37
3.7.4 Giving numeric values to mpimon .....................................................................39

Chapter 4 Operation of Platform MPI .......................................................................40

4.1 Platform MPI components .....................................................................................40
4.1.1 user_authentication ........................................................................................40
4.1.2 disable_toss_page_cache ................................................................................40
4.2 Application startup. .............................................................................................41
4.3 Network devices ..................................................................................................42

Platform MPI 5.6.5 User’s Guide Page iv

 4.3.1 Shared Memory Device ....................................................................................43
4.3.2 Ethernet Devices ............................................................................................43
4.3.2.1 TCP ..........................................................................................................44
4.3.2.2 Myrinet GM ...............................................................................................44
4.3.2.3 Infiniband .................................................................................................44
4.3.2.4 Multi-rail Infiniband ....................................................................................45
4.3.3 Communication protocols .................................................................................45
4.3.3.1 Channel buffer ...........................................................................................46
4.3.3.2 Inline protocol ...........................................................................................47
4.3.3.3 Eager buffering protocol .............................................................................48
4.3.3.4 Transporter protocol ...................................................................................49
4.3.3.5 Zerocopy protocol ......................................................................................50

Chapter 5 Profiling with Platform MPI .....................................................................51

5.1 Tracing ..............................................................................................................53
5.1.1 Using Platform MPI built-in trace .......................................................................53
5.1.1.1 Example: using the test application .............................................................55
5.1.2 Features ........................................................................................................56
5.2 Timing ...............................................................................................................57
5.2.1 Using Platform MPI built-in timing .....................................................................57
5.3 Using scanalyze ..................................................................................................60
5.3.1 Analyzing all2all .............................................................................................60
5.3.2 Using Platform MPI's built-in CPU-usage functionality ..........................................61

Chapter 6 Tuning Platform MPI to your application .................................................64

6.1 Tuning communication resources ...........................................................................64
6.1.1 Communication buffer adaption ........................................................................64
6.1.1.1 Example: Application is sending only 900 bytes messages. ..............................64
6.1.1.2 Example: Controlling shared memory usage .................................................64
6.1.2 Automatic buffer management .........................................................................64
6.1.3 How to optimize MPI performance .....................................................................65
6.1.3.1 Performance analysis .................................................................................65
6.1.3.2 Using processor-power to poll ......................................................................65
6.1.3.3 Reorder network traffic to avoid conflicts ......................................................66
6.1.3.4 Multicore Awareness ...................................................................................66
6.1.3.5 Granularity ...............................................................................................66
6.1.3.6 Policy .......................................................................................................66
6.1.3.7 Benchmarking ...........................................................................................66
6.1.3.7.1 How to get expected performance ...........................................................66
6.1.3.7.2 Memory consumption increase after warm-up ...........................................67
6.1.3.7.3 Collective operations .............................................................................67
6.1.3.7.4 Finding the best algorithm ......................................................................69

Appendix A Licensing Management .......................................................................... 77

A.1 Product key overview .......................................................................................... 77
A.2 Licensing Tool - lmutil.......................................................................................... 77
A.2.1 Product Activation .......................................................................................... 78
A.2.1.1 Automatic activation ................................................................................. 78
A.2.1.2 Manual activation ...................................................................................... 78
A.2.2 Managing product keys ................................................................................... 78
A.2.2.1 Listing product keys using the lmutil tool ...................................................... 78
A.2.2.2 Adding product keys .................................................................................. 79

Platform MPI 5.6.5 User’s Guide Page v

 A.2.2.3 Removing product keys .............................................................................. 79

Appendix B Install Platform MPI in Unix .................................................................. 80

B.0.1 Installation of license-server ............................................................................ 80
B.0.2 Per node installation of Platform MPI ................................................................. 80
B.1 Setting default networks ...................................................................................... 81
B.1.1 Install Platform MPI for TCP/IP ......................................................................... 81
B.1.2 Install Platform MPI for Myrinet or Infiniband ..................................................... 82
B.1.2.1 Example ................................................................................................... 82
B.2 License server options ......................................................................................... 82
B.3 Uninstalling Platform MPI ..................................................................................... 82
B.4 Troubleshooting Network providers........................................................................ 82
B.4.1 Troubleshooting the IB provider ....................................................................... 82
B.4.2 Troubleshooting the GM provider ...................................................................... 83

Appendix C Installing Platform MPI In Windows ...................................................... 84

C.1 Installing Platform MPI......................................................................................... 84
C.2 Uninstalling Platform MPI ..................................................................................... 85

Appendix D Bracket expansion and grouping ........................................................... 86

D.1 Bracket expansion .............................................................................................. 86
D.2 Grouping ........................................................................................................... 86
D.3 File format ......................................................................................................... 86

Appendix E The ScaMPItst package .......................................................................... 88

E.1 Image contrast enhancement................................................................................ 89
E.2 File format ......................................................................................................... 91

Appendix F Multi-rail Support in Platform MPI ......................................................... 92

F.1 Software Requirements ........................................................................................ 92
F.2 Hardware Requirements ....................................................................................... 92
F.3 Implementation Overview..................................................................................... 92
F.4 Data-striping ...................................................................................................... 92
F.5 Load-balancing ................................................................................................... 92
F.6 Multirail Configuration .......................................................................................... 93

Appendix G Checkpoint and restart ......................................................................... 94

G.1 MPI application architecture ................................................................................. 94
G.1.1 Controlling Checkpoint/Restart......................................................................... 95
G.1.2 The Checkpoint Process .................................................................................. 96
G.1.2.1 Drainage stage.......................................................................................... 97
G.1.2.2 Closing stage ............................................................................................ 98
G.1.2.3 Checkpoint & Continue/Abort stage............................................................ 100
G.1.2.4 Restart .................................................................................................. 101
G.2 Using checkpoint/restart .................................................................................... 101
G.2.1 Starting jobs ............................................................................................... 101
G.2.2 Taking a checkpoint...................................................................................... 101
G.2.3 Restarting from a checkpoint ......................................................................... 101
G.2.4 List taken checkpoints .................................................................................. 102
G.2.5 Delete checkpointfiles ................................................................................... 102

Appendix H Troubleshooting .................................................................................. 103

Platform MPI 5.6.5 User’s Guide Page vi

 H.1 Why does not my program in Unix start to run? .................................................... 103
H.2 Why can I not start mpid in Unix? ....................................................................... 103
H.3 Space overflow ................................................................................................. 104
H.4 Why does my program terminate abnormally? ...................................................... 104
H.5 General problems.............................................................................................. 104

Appendix I Related documentation ........................................................................ 105

Appendix J List of Figures ...................................................................................... 106

Appendix K List of Tables ....................................................................................... 107

Platform MPI 5.6.5 User’s Guide Page vii

Chapter 1 - Introduction
This manual describes Platform MPI in detail. Platform MPI is sold as a separate stand-alone
product with an Platform MPI distribution.

This manual is written for users who have a basic programming knowledge of C or Fortran, as
well as an understanding of MPI.

1.1 Platform MPI product context

Figure 1.1 Platform MPI Simplified Topology

Figure 1.1 shows a simplified view of the underlying architecture of clusters using Platform MPI:
A number of compute nodes are connected together in a Ethernet network through which a
front-end interfaces the cluster with the corporate network. A high performance interconnect can
be attached to service communication requirements of key applications.

The front-end imports services like file systems from the corporate network to allow users to run
applications and access their data.

Platform MPI implements the MPI standard for a number of popular high performance
interconnects, such as Gigiabit Ethernet, Infiniband and Myrinet.

While the high performance interconnect is optional, the networking infrastructure is mandatory.
Without it the nodes in the cluster will have no way of sharing resources. TCP/IP functionality
implemented by the Ethernet network enables the front-end to issue commands to the nodes,
provide them with data and application images, and collect results from the processing the
nodes perform.

Platform MPI 5.6.5 User’s Guide Page 1

Platformtargets its users involved in High Performance Computing, also known as
supercomputing, which typically includes CPU-intensive parallel applications. Platform produces
software tools that maximize the power and ease the use of computing hardware.

CPU-intensive parallel applications use a programming library called MPI (Message Passing
Interface), the state-of-the-art library for high performance computing. Note that the MPI library
is NOT described within this manual; MPI is defined by a standards committee, and the API, and
its user guides are available free of charge on the Internet. A link to the MPI Standard and other
MPI resources can be found in chapter 7, "Related documentation".

Platform MPI consists of Platform's implementation of the MPI programming library and the
necessary support programs to launch and run MPI applications. This manual often uses the
term Platform MPI to refer to the specifics of the MPI itself, and not the support applications.

1.2 Support
1.2.1 Platform MPI FAQ
An updated list of Frequently Asked Questions is posted on
http://www.platform.com In addition, for users who have installed Platform MPI, the
version of the FAQ that was current when Platform MPI was installed is available as a text file

in Unix

${MPI_HOME}/doc/ScaMPI/FAQ

in Windows

%MPI_HOME%\doc\FAQ.TXT

1.2.2 Platform MPI release documents

When Platform MPI has been installed, a number of smaller documents such as the FAQ,
RELEASE NOTES, README, SUPPORT, LICENSE_TERMS, INSTALL are available as text files in the
default directory

in Windows:

%MPI_HOME%\doc

in UNIX:

${MPI_HOME}/doc/ScaMPI

1.2.3 Problem reports

Problem reports should, whenever possible, include both a description of the problem, the
software version(s), the computer architecture, a code example, and a record of the sequence of
events causing the problem. In particular, any information that you can include about what
triggered the error will be helpful. The report should be sent by e-mail to
[email protected].

Platform MPI 5.6.5 User’s Guide Page 2

1.2.4 Platforms supported
Platform MPI is available for a number of platforms. For up-to-date information, please see the
Platform MPI section of http://www.platform.com. For additional information, please
contact Platform at [email protected].

1.2.5 Licensing
Platform MPI is licensed using Platform license manager system. In order to run Platform MPI
a valid demo or a permanent license must be obtained. Customers with valid software
maintenance contracts with Platform may request this directly from [email protected].
All other requests, including DEMO licenses, should be directed to [email protected].

1.2.6 Feedback
Platform appreciates any suggestions users may have for improving both this Platform MPI
User’s Guide and the software described herein. Please send your comments by e-mail to
[email protected].

Users of parallel tools software using Platform MPI on a Platform System are also encouraged
to provide feedback to the National HPCC Software Exchange (NHSE) - Parallel Tools Library [7].
The Parallel Tools Library provides information about parallel system software and tools, and also
provides for communication between software authors and users.

1.3 How to read this guide

This guide is written for skilled computer users and professionals. It is assumed that the reader
is familiar with the basic concepts and terminology of computer hardware and software since
none of these will be explained in any detail. Depending on your user profile, some chapters are
more relevant than others.

Platform MPI 5.6.5 User’s Guide Page 3

1.3.1 Acronyms and abbreviations

Acronym/
Abbreviation Meaning

The 64 bit Instruction set architecture (ISA) that is the 64 bit

extension to the Intel x86 ISA. Also known as x86-64. The Opteron
AMD64 and Athlon64 from AMD are the first implementations of this ISA.

Direct Access Provider Library DAT Instantiation for a given

DAPL interconnect

Direct Access Transport - Transport-independent,

platform-independent Application Programming Interfaces that exploit
DAT RDMA

Direct Ethernet Transport - Platform's DAT implementation for

DET Ethernet-like devices, including channel aggregation

The Intel implementation of the 64 bit extension to the x86 ISA. Also
EM64T See AMD64.

A software interface provided by Myricom for their Myrinet

interconnect
GM hardware.

Hardware Channel Adapter. Term used by Infiniband vendors

HCA referencing to the hardware adapter.

HPC High Performance Computer

IA32 Instruction set Architecture 32 Intel x86 architecture

IA64 Instruction set Architecture 64 Intel 64-bit architecture, Itanium, EPIC

MPI Message Passing Interface - De-facto standard for message passing

NIC Network Interface Card

OEM Original Equipment Manufacturer

Abbreviation for PowerPC 64, which is the common 64 bit instruction

set architecture(ISA) name used in Linux for the PowerPC and POWER
processor families. These processors have a common core ISA that
allow one single Linux version to be made for all three processor
PPC64 families.

Remote DMA Read or Write Data in a remote memory at a given

RDMA address

Scali MPI - First generation MPI Connect product, replaced by

ScaMPI Platform MPI

Table 1.1 Abbreviations

Platform MPI 5.6.5 User’s Guide Page 4

 Acronym/
Abbreviation Meaning

Platform MPI Platform MPI - Platform's second generation MPI

VAR Value Added Reseller

x86-64 see AMD64 and EM64T

Table 1.1 Abbreviations

1.3.2 Terms
Unless explicitly specified otherwise, gcc (gnu c-compiler) and bash (gnu Bourne-Again-SHell)
are used in all examples.

Term Description

A cluster is a set of interconnected nodes with the aim to act as one

cluster single unit

A computer outside the cluster nodes dedicated to run configuration,

front end monitoring and licensing software

A high speed interconnect standard available from a number of

Infiniband vendors

Instance of application program with unique rank within

MPI process MPI_COMM_WORLD

An interconnect developed by Myricom. Myrinet is the product name

Myrinet™ for the hardware. (See GM).

A single computer in an interconnected system consisting of more

Node than one computer

Platform system A cluster consisting of Platform components

A generic term that cover the PowerPC and POWER processor families.
These processors are both 32 and 64 bit capable. The common case is
to have a 64 bit OS that support both 32 and 64 bit executables. See
Power also PPC64

PowerPC The IBM/Motorola PowerPC processor family. See PPC64

The IBM POWER processor family. Platform support the 4 and 5

POWER versions. See PPC64

Table 1.2 Terms

Platform MPI 5.6.5 User’s Guide Page 5

 Term Description

A cluster is a set of interconnected nodes with the aim to act as one

cluster single unit

greek word for ring, used in Platform documents in the context of 2-

torus and 3-dimensional interconnect topologies

Refers to all UNIX and look-alike Operating Systems supported, i.e.

UNIX Solaris and Linux.

Windows Refers to Microsoft Windows XP/Server2003/Vista

Table 1.2 Terms

1.3.3 Typographic conventions

Typography Description

Bold Program names, options and default values

Italics User input

mono spaced Computer related: Shell commands, examples, environment variables,

file locations (directories) and contents

# Command prompt in shell with super user privileges

% Command prompt in shell with normal user privileges

Table 1.3 Typographical conventions

Platform MPI 5.6.5 User’s Guide Page 6

Chapter 2 - Using Platform MPI with Unix
This chapter describes how to setup, compile, link and run a program using Platform MPI, and
briefly discusses some useful tools for debugging and profiling.

• “Setting up a Platform MPI environment” on page 7

• “Compiling and linking” on page 7
• “Running Platform MPI programs” on page 10
• “Suspending and resuming jobs” on page 17
• “Running with dynamic interconnect failover capabilities” on page 18
• “Debugging and profiling” on page 18
• “Controlling communication resources” on page 21
• “Good programming practice with Platform MPI” on page 22
• “Error and warning messages” on page 24

2.1 Setting up a Platform MPI environment

The use of Platform MPI requires that some environment variables are defined.We have
provided a script (<install-dir>/etc/scampivars.{sh,csh}) that can be sourced for easy
setup of these variables. <install-dir> is the base-directory where Platform MPI is installed
(default /opt/Platform). The environment can also be set in startup shell scripts or defined
manually.

2.2 Compiling and linking

MPI is an “Application Programming Interface” (API) and not an “Application Binary Interface”
(ABI). This means that in general applications should be recompiled and linked when used with
Platform MPI. Since the MPICH-implementation is widely used Platform has made Platform
MPI ABI-compatible, depending on the versions of MPICH and Platform MPI used. Please
check the “Platform MPI Release Notes” for details. For applications that are dynamically
linked with MPICH it should only be necessary to change the library-path (LD_LIBRARY_PATH).
For applications with the necessary object files, only a relinking is needed.

Platform MPI is a C library built using the GNU compiler. Applications can however be compiled
with most compilers, as long as they are linked with the GNU runtime library.

The details of the process of linking with the Platform MPI libraries vary depending on which
compiler is used. Check the “Platform MPI Release Notes” for information on supported
compilers and how linking is done.

2.2.1 Using mpicc / mpic++ / mpif77

We provide wrappers that encapsulates all details of compiler-settings.

Platform MPI 5.6.5 User’s Guide Page 7

mpicc should be used when compiling c-code, mpic++ for c++-code and mpif77 for
fortran-code.

Usage: <wrapper> [options] [compiler args]

(<wrapper> == mpicc / mpic++ / mpif77)

Option Description

-c Compile only

-v Verbose output

-h Show this help message

-ccl <compiler> Use the specified compiler rather than gcc

Table 2.1 mpicc/mpic++/mpif77 options

2.2.2 Using gnu compiler

When compiling the following string must be included as compiler flags (bash syntax):

-I${MPI_HOME}/include

The pattern for compiling is:

user% gcc -c -I${MPI_HOME}/include hello-world.c

user% g77 -c -I${MPI_HOME}/include hello-world.f

The following string outlines the setup for the necessary linker flags (bash syntax):

-L${MPI_HOME}/lib -lmpi
-L${MPI_HOME}/lib64 -lmpi

The following versions of MPI libraries are available:

• libmpi - Standard library containing the C API.

• libfmpi - Library containing the Fortran API wrappers.
The pattern for linking is:

user% gcc hello-world.o -L${MPI_HOME}/lib -lmpi -o hello-world

user% g77 hello-world.o -L${MPI_HOME}/lib -lfmpi -lmpi -o hello-world

Start the hello-world program on the three nodes called nodeA, nodeB and nodeC.

% mpimon hello-world -- nodeA 1 nodeB 1 nodeC 1

Platform MPI 5.6.5 User’s Guide Page 8

The hello-world program should produce the following output:

Hello-world, I'm rank 0; Size is 3

Hello-world, I'm rank 1; Size is 3
Hello-world, I'm rank 2; Size is 3

2.2.3 Notes on compiling and linking on various plattforms

2.2.3.1 Compiling and linking on AMD64 and EM64T
AMD’s AMD64 and Intel’s EM64T (also known as x86-64) are instruction set architectures (ISA)
that add 64 bit extensions to Intel x86 (ia32) ISA.

These processors are capable of running 32 bit programs at full speed while running a 64 bit OS.
For this reason Platform supports running both 32 bit and 64 bit MPI programs while running 64
bit OS.

Having both 32 bit and 64 bit libraries installed at the same time require some tweaks to the
compiler and linker flags.

Please note:

• All compilers for x86-64 generate 64 bit code by default, but have flags for 32 bit code
generation. For gcc/g77 these are -m32 and -m64 for making 32 and 64 bit code
respectively. For Portland Group Compilers these are -tp k8-32 and -tp k8-64. For
other compilers please check the compiler documentation.
• It is not possible to link 32 and 64 bit object code into one executable, (no cross
dynamic linking either) so there must be double set of libraries.
• It is common convention on x86-64 systems that all 32 bit libraries are placed in lib
directories (for compatibility with x86 OS’s) and all 64 bit libraries in lib64. This
means that when linking a 64 bit application with Platform MPI, you must use the
-L${MPI_HOME}/lib64 argument instead of the normal -L${MPI_HOME}/lib
2.2.3.2 Compiling and linking on Power series
The Power series processors (PowerPC, POWER4 and POWER5) are both 32 and 64 bit capable.
There are only 64 bit versions of Linux provided by SUSE and RedHat, and only a 64 bit OS is
supported by Platform. However the Power families are capable of running 32 bit programs at
full speed while running a 64 bit OS. For this reason Platform supports running both 32 bit and
64 bit MPI programs.

Please note:

• The gcc default compiles 32 bit on Power, use the gcc/g77 flags -m32 and -m64
explicitly to select code generation.
• It is not possible to link 32 and 64 bit object code into one executable, (no cross
dynamic linking, either) so there must be double set of libraries.
• It is common convention on ppc64 systems that all 32 bit libraries are placed in lib
directories and all 64 bit libraries in lib64. This means that when linking a 64 bit
application with Platform MPI, you must use the -L${MPI_HOME}/lib64 argument
instead of the normal -L${MPI_HOME}/lib.

Platform MPI 5.6.5 User’s Guide Page 9

The PowerPC and POWER4/5 have a common core instruction set but different extensions, be
sure to read the specifics in the documentation on the compilers code generations flags for
optimal performance.

2.2.4 Compiling with Platform MPI-IO

To compile and link with the Platform MPI-IO features you need to do the following depending on
whether it is a C or a Fortran program:

For C programs mpio.h must be included in your program and you must link with the libmpio
shared library in addition to the Platform MPI 1.2 C shared library (libmpi):

<CC> <program>.o -I${MPI_HOME}/include -L${MPI_HOME}/lib -lmpio -lmpi -o

<program>

For Fortran programs you will need to include mpiof.h in your program and link with the
libmpio shared library in addition to the Platform MPI 1.2 C and Fortran shared libraries:

<F77> <program>.o -I${MPI_HOME}/include -L${MPI_HOME}/lib -lmpio -lfmpi -lmpi

-o <program>

2.3 Running Platform MPI programs

Note that executables issuing Platform MPI calls cannot be started directly from a shell prompt.
Platform MPI programs can either be started using the MPI monitor program mpimon, the
wrapper script mpirun, or from the Platform Manage GUI [See Platform Manage User Guide for
details].

2.3.1 Naming conventions

When an application program is started, Platform MPI modifies the program name argv[0] to
identify the instances. Use the following convention for the executable reported on the command
line using the Unix utility ps:

Argument <userprogram> -<rank_number>(mpi:<pid>@<nodename>)

<userprogram> is the name of the application program.

<rank_number> is the application’s MPI-process rank number.

<pid> is the Unix process identifier of the monitor program mpimon.

<nodename> is the name of the node where mpimon is running.

Table 2.2 MPI Connect Options

Attention
Note: Platform MPI requires a homogenous file system image, i.e. a file system
providing the same path and program names on all nodes of the cluster on which
Platform MPI is installed.

Platform MPI 5.6.5 User’s Guide Page 10

Attention
Note: To turn off this naming convention feature, you will have to set the environment
variable SCAMPI_DISABLE_ARGV0_MOD. You do this by entering:

# export SCAMPI_DISABLE_ARGV0_MOD=1

for ksh/bash, or:

# setenv SCAMPI_DISABLE_ARGV0_MOD 1

for csh type shells.

2.3.2 mpimon - monitor program

The control and start-up of an Platform MPI application are monitored by mpimon. A complete
listing of mpimon options can be found in “mpimon options” on page 24.

2.3.2.1 Basic usage

Normally mpimon is invoked as:

mpimon <userprogram> <program_options> -- <node name> [<count>][<nodename>

[<count>]]...

<userprogram> is name of application

<program_options> are options to the application

"--" is the separator ending the application options

<nodename>[<count>] is name of node and the number of MPI-processes to run on that

node. The option can occur several times in the list. Mpi-processes
will be given ranks sequentially according to the list of
node-number pairs.
<count> is optional and defaults to 1.

Table 2.3 mpimon options

Example
Starting the program “${MPI_HOME}/examples/bin/hello” on a node called “hugin”:
mpimon ${MPI_HOME}/examples/bin/hello -- hugin

Example
Starting the same program with two processes on the same node:
mpimon ${MPI_HOME}/examples/bin/hello -- hugin 2

Platform MPI 5.6.5 User’s Guide Page 11

Example
Starting the same program on two different nodes with 4 processes on each:
mpimon ${MPI_HOME}/examples/bin/hello -- hugin 4 munin 4

Example
Starting the same program on two different nodes, “hugin” and “munin”:
mpimon ${MPI_HOME}/examples/bin/hello -- hugin munin

Example
Using bracket expansion and grouping (if configured):
mpimon ${MPI_HOME}/examples/bin/hello -- node[1-16] 2 node[17-32] 1

For more information regarding bracket expansion and grouping, refer to “Bracket expansion
and grouping” on page 86.

2.3.2.2 Identity of parallel processes

The identification of nodes and the number of processes to run on each particular node
translates directly into the rank of the MPI processes. For example, specifying n1 2 n2 2 will
place process 0 and 1 on node n1 and process 2 and 3 on node n2. On the other hand, specifying
n1 1 n2 1 n1 1 n2 1 will place process 0 and 2 on node n1 while process 1 and 3 are placed on
node n2.

This control over placement of processes can be very valuable when application performance
depends on all the nodes having the same amount of work to do.

2.3.2.3 Controlling options to mpimon

The program mpimon has a multitude of options which can be used for optimising Platform
MPI performance. Normally it should not be necessary to use any of these options. However,
unsafe MPI programs might need buffer adjustments to solve deadlocks. Running multiple
applications in one run may also be facilitated with some of the “advanced options”.

mpimon also has an advanced tracing option to enable pinpointing of communication

bottlenecks.

The complete syntax for the program is as follows:

mpimon [<mpimon option>]... <program & node-spec> [-- <program & node-spec>]...
<mpimon options> are options to mpimon (“See mpimon options” and the Platform
MPI Release Notes for a complete list of options),
-- is the separator that signals end of user program options.
<program_&_node-spec> is an application and node specification consisting of <program
spec> -- <node spec> [<node spec>]...,
Table 2.4 mpimon option syntax

Platform MPI 5.6.5 User’s Guide Page 12

mpimon [<mpimon option>]... <program & node-spec> [-- <program & node-spec>]...
<program spec> is an application and application-options specification
(<userprogram>[<programoptions>])
<node spec> specifies which node and how many instances (<nodename>
[<count>]). If <count> is omitted, one MPI-process is started on
each node specified.
Table 2.4 mpimon option syntax

Example
Starting the program “${MPI_HOME}/examples/bin/hello” on a node called “hugin” and the
program “${MPI_HOME}/examples/bandwidth” with 2 processes on “munin”:
mpimon ${MPI_HOME}/examples/bin/hello -- hugin --
${MPI_HOME}/examples/bin/bandwidth -- munin 2

Example
Changing one of the mpimon-parameters:
mpimon -channel_entry_count 32 ${MPI_HOME}/examples/bin/hello -- hugin 2

2.3.2.4 Standard input

The -stdin option specifies which MPI-process rank should receive the input. You can in fact
send stdin to all the MPI-processes with the all argument, but this requires that all
MPI-processes read the exact same amount of input. The most common way of doing it is to
send all data on stdin to rank 0:

mpimon -stdin 0 myprogram -- node1 node2... < input_file>

Attention
Note: The default for -stdin is none.

2.3.2.5 Standard output

By default the processes’ output to stdout all appear in the stdout of mpimon, where they are
merged in some random order. It is however possible to keep the outputs apart by directing
them to files that have unique names for each process. This is accomplished by giving mpimon
the option -separate_output <seletion>, e.g., -separate_output all to have each process deposit
its stdout in a file. The files are named according to the following template:
ScaMPIoutput_<host>_<pid>_<rank>, where <host> and <pid> identify the particular
invocation of mpimon on the host, and <rank> identifies the process.

It is possible to control the placement of the output file by setting the environment variable
SCAMPI_OUTPUT_FILE=prefix. The prefix will replace the ScaMPIoutput portion of the filenames
used for output. For example, by setting SCAMPI_OUTPUT_FILE=/tmp/MyOutput together with
the -separate_output option, the output files will be placed in the /tmp directory and be named
MyOutput_<host>_<pid>_<rank>.

Platform MPI 5.6.5 User’s Guide Page 13

2.3.2.6 How to provide options to mpimon
There are three different ways to provide options to mpimon. The most common way is to
specify options on the command line invoking mpimon. Another way is to define environment
variables, and the third way is to define options in configuration file(s).

• Command line options: Options for mpimon must be placed after mpimon, but before the
program name
• Environment-variable options: Setting an mpimon-option with environment variables
requires that variables are defined as SCAMPI_<uppercase-option> where SCAMPI_ is a
fixed prefix followed by the option converted to upper case. For example
SCAMPI_CHANNEL_SIZE=64K means setting -channel_size to 64K
• Configuration-files options: mpimon reads up to three different configuration files when
starting. First the system-wide configuration (${MPI_HOME}/etc/ScaMPI.conf) is read.
If the user has a file on his/her home-directory, that file(${HOME}/ScaMPI.conf) is then
read. Finally if there is a configuration file in the current directory, that file(./ScaMPI.conf)
is then read. The files should contain one option per line, given as for command line options.
The options described either on the command line, as environment variables or in configuration
files are prioritized the following way (ranked from lowest to highest):

1 System-wide configuration-file(${MPI_HOME}/etc/ScaMPI.conf)
2 Configuration-file on home-directory(${HOME}/ScaMPI.conf)
3 Configuration-file on current directory(./ScaMPI.conf)
4 Environment-variables
5 Command line-options
2.3.2.7 Network options
Platform MPI is designed to handle several networks in one run. There are two types of
networks, built-in standard-devices and DAT-devices. The devices are selected by giving the
option “-networks <net-list>” to mpimon. <net-list> is a comma-separated list of device
specifications. Platform MPI uses the list when setting up connections to other MPI-processes.
It starts off with the first device specification in the list and sets up all possible connections with
that device. If this fails the next on list is tried and so on until all connections are live or all
adapters in <net-list> have been tried.We have introduced a new syntax for
device-specifications but we also support the old syntax for backwards compatability(marked as
deprecated below).

-net <spec>[,<spec>]
<spec> = <nspec> | <dspec>
New syntax:
<nspec> = <type>[<delimit><typespec>][<delimit><parname>=<parvalue>]
<type> = ibverbs | vapi | ibta |
dapl |
gm
tcp |
smp |

Platform MPI 5.6.5 User’s Guide Page 14

 file |
mem |
infinipath |
liquidio
<delimit> = :
ibverbs | vapi | ibta:
<typespec> = <hca><delimit><port>
dapl:
<typespec> = <dapl-name> /* dapl-name defined in /etc/dat.conf */
gm:
<typespec> = <unit><delim><port>
tcp:
<typespec> = <device>
smp:
<typespec> = "smp" | "pms"
file:
<typespec> = <user-defined-name>
/* User-defined-network-name defined in the file(s) scampi-networks.conf */
Optional: (zero or more)
<parname> = Predefined name for parameter
<parvalue>= Value for named parameter

<typespec> will be expanded with scaBracketExpand allowing specifications like mthca[0,1]

(See documentation of scabracket for more info)
Deprecated syntax:
<dspec> = tcp<n> |
ib[<n>] |
vapi[<n>] |
gm<n> |
myri[<n>] |
sci
/* <n> is a one digit "unit-number" */

Example, Preferred syntax

-networks ibverbs:ne0:1 /* hca: ne0 port: 1 */
-networks ibverbs:ne1: /* hca: ne1 port: default */
-networks ibverbs::1 /* hca: default port: 1 */
-networks ibverbs /* hca: default port: default */
-networks ibverbs::[1,2] /* two specs; hca: default port: 1 AND hca: default port: 2 */

-networks tcp:eth0 /* Device: eth0 */

-networks tcp /* Device depending on nodenames */

-networks file:hwr

file "scampi-networks.conf":
hwr ibverbs:mthca0:2
/* File format
<user-def-name> <spec>
<spec> = <type>[<delimit><typespec>]
NB. no bracket-expandable expression! one name == one net-spec */

Platform MPI 5.6.5 User’s Guide Page 15

Example, Deprecated syntax
-networks ib /* hca: default port: default */
-networks ib0 /* hca: 1 port: default */

A list of preferred devices can be defined in several ways;(ranked from lowest to highest):

1 System-wide configuration-file(${MPI_HOME}/etc/scampi_networks.conf)
2 Configuration-file on home-directory(${HOME}/scampi_networks.conf)
3 Configuration-file on current directory(./scampi_networks.conf)
4 Environment-variables
5 Command line-options
The values should be provided in a comma-separated list of device names.For each MPI process
Platform MPI will try to establish contact with each other MPI process, in the order listed. This
enables mixed interconnect systems, and provides a means for working around failed hardware.

In a system interconnect where the primary interconnect is Myrinet, if one node has a faulty
card, using the device list in the example, all communication to and from the faulty node will
happen over TCP/IP while the remaining nodes will use Myrinet. This offers the unique ability to
continue running applications over the full set of nodes even when there are interconnect faults.

2.3.3 mpirun - wrapper script

mpirun is a wrapper script for mpimon, providing legacy MPICH style startup for Platform
MPI applications. Instead of the mpimon syntax, where a list of pairs of node name and
number of MPI-processes is used as startup specification, mpirun uses only the total number of
MPI-processes.

mpirun uses the file ${MPI_HOME}/etc/nodelist (optionally ${HOME}/.nodelist) for

selecting the list of operational notes. In the generated list of nodes, mpirun evenly divides the
MPI-processes among the nodes.

2.3.3.1 mpirun usage

Normally mpirun is invoked as:

mpirun <mpirunoptions> <mpimonoptions> <userprogram> [<programoptions>]

<mpirunoptions> mpirun options
<mpimonoptions> options passed on to mpimon
<userprogram> name of application program to run.
Table 2.5 mpirun usage

Platform MPI 5.6.5 User’s Guide Page 16

The following mpirun options exist:

Option Description
-cpu <time> Limit runtime to <time> minutes.
-np <count> Total number of MPI-processes to be started, default 2.
-npn <count> Maximum number of MPI-processes pr. node, default np <count>/
nodes.
-pbs Submit job to PBS queue system
-pbsparams <“params”> Specify PBS scasub parameters
-p4pg <pgfile> Use mpich compatible pgfile for program, MPI-process and node
specification. pgfile entry: <nodename> <#procs> <progname>
The program name given at command line is additionally started
with one MPI-process at first node.
-v Verbose
-gdb Debug all MPI-processes using the GNU debugger gdb.
-maxtime <time> Limit runtime to <time> minutes.
-machinefile <filename> Take the list of possible nodes from <filename>
-mstdin <proc> Distribute stdin to MPI-process(es).
<proc> The choices are all (default), none, or MPI-process number(s).
-part <part> Use nodes from partition <part>
-q Keep quiet, no mpimon printout.
-t test mode, no MPI program is started
<params> Parameters not recognized are passed on to mpimon.
Table 2.6 mpirun options

2.3.4 Suspending and resuming jobs

From time to time it is convenient to be able to suspend regular jobs running on a cluster in
order to allow a critical, maybe real-time job to be use the cluster. When using Platform MPI to
run parallel applications suspending jobs to yield the cluster to other jobs can be achieved by
sending a SIGUSR1 or SIGTSTP signal to the mpimon representing the job.

Assuming that the process identifier for this mpimon is <PID>, the user interface for this is:

user% kill -USR1 <PID>

or

user% kill -TSTP <PID>

Similarly the suspended job can be resumed by sending it a SIGUSR2 or SIGCONT signal, i.e.,

user% kill -USR2 <PID>

Platform MPI 5.6.5 User’s Guide Page 17

or

user% kill -CONT <PID>

2.3.5 Running with dynamic interconnect failover capabilities

If a runtime failure on a high speed interconnect occurs, Platform MPI has the ability to do an
interconnect failover and continue running on a secondary network device. This high availability
feature is part of the Platform MPI/HA product, which requires a separately priced license.
Once this license is installed, you may enable the failover functionality by setting the
environment variable SCAMPI_FAILOVER_MODE to 1, or by using the mpimon command line
argument -failover_mode.

Currently the Platform MPI Infiniband (ib0), Myrinet (gm0) and all DAT-based interconnects
are supported.

Some failures will not result in a explicit error value propagating to Platform MPI. Platform
MPI handles this by treating a lack of progress within a specified time as a failure. You may alter
this time by setting the environment variable SCAMPI_FAILOVER_TIMEOUT to the desired
number of seconds.

Failures will be logged using the standard syslog mechanism.

2.3.6 Debugging and profiling

The complexity of debugging programs can grow dramatically when going from serial to parallel
programs. So to assist in debugging MPI programs Platform MPI has a number of features
built-in, like starting processes directly in a debugger and tracing processes’ traffic patterns.

2.3.7 Debugging with a sequential debugger

Platform MPI applications can be debugged using a sequential debugger. By default, the GNU
debugger gdb is invoked by mpimon. If another debugger is to be used, specify the debugger
using the mpimon option -debugger <debugger>.

To set debug-mode for one or more MPI-processes, specify the MPI-process(es) to debug using
the mpimon option -debug <select>. In addition, note that the mpimon option -display
<display> should be used to set the display for the xterm terminal emulator. An xterm
terminal emulator, and one debugger, is started for each of the MPI-processes being debugged.

For example, to debug an application using the default gdb debugger do:

user% mpimon -debug all <application +parameters> -- <node specification>

Initially, for both MPI-process 0 and MPI-process 1, an xterm window is opened. Next, in the
upper left hand corner of each xterm window, a message containing the application program’s
run parameter(s) is displayed. Typically, the first line reads Run parameters: run
<programoptions>. The information following the colon, i.e. run <programoptions>, is

Platform MPI 5.6.5 User’s Guide Page 18

needed by both the debugger and the Platform MPI application being debugged. Finally, one
debugger is started for each session. In each debugger's xterm window, first input the
appropriate debugging action before the MPI-process is started. Then, when ready to run the
MPI-process, paste <programoptions> into the debugger to start running.

Figure 2.1 Example of debug windows

2.3.8 Built-in-tools for debugging

Built-in tools for debugging in Platform MPI covers discovery of the MPI calls used through
tracing and timing, and an attachment point to processes that fault with segmentation violation.
The tracing (see “Tracing” on page 53) and timing (see “Timing” on page 57) are covered
in Chapter 4.

Platform MPI 5.6.5 User’s Guide Page 19

2.3.9 Using built-in segment protect violation handler,
When running applications that terminate with a SIGSEGV-signal it is often useful to be able to
freeze the situation instead of exiting, the default behavior. In previous versions of Platform
MPI you could use SCAMPI_INSTALL_SIGSEGV_HANDLER.
SCAMPI_INSTALL_SIGNAL_HANDLER has replaced this environment variable. This allows
for Platform MPI to be backwardly compatible with earlier versions of MPI. The values for the
signals are put into a comma separated list.

SCAMPI_INSTALL_SIGNAL_HANDLER=11

will install a handler for signal number 11 (which is the same as SIGSEGV)

To install a handler for signal 11 (SIGSEGV) and signal 4 (SIGILL) enter

SCAMPI_INSTALL_SIGNAL_HANDLER=11,4

The default action has the handler dump all registers and starts processes looping. Attaching
with a debugger will then make it possible to examine the situation which resulted in the
segment protect violation.

As an alternative the handler dumps all registers but all processes will exit afterwards.

To attach to process <pid> on a machine with the GNU debugger (gdb) use the following
command:

user% gdb /proc/<pid>/exe <pid>

In general, this will allow gdb to inspect the stack trace and identify the functions active when
the sigsegv occurred, and disassemble the functions. If the application is compiled with debug
info (-g) and the source code is available, then source level debugging can be carried out.

2.3.10 Assistance for external profiling

Profiling parallel applications is complicated by having multiple processes at the same time. But
Platform MPI comes to assistance; through the SCAMPI_PROFILE_APPLICATION environment
variable together with the -separate_output option (SCAMPI_SEPARATE_OUTPUT) the output
from the application runs is directed at one file per process for easier use.

The environment variables SCAMPI_PROFILE_APPLICATION_START and

SCAMPI_PROFILE_APPLICATION_END are also available for steering the range of memory
addresses applicable to profiling.

2.3.11 Debugging with Etnus Totalview

Platform MPI applications can be debugged using the Etnus Totalview, see

http://www.etnus.com for more information about this product.

Platform MPI 5.6.5 User’s Guide Page 20

To start the Etnus Totalview debugger with a Platform MPI application, use the tvmpimon
wrapper script. This wrapper script accepts the regular options that mpimon accepts, and sets
up the environment for Totalview. The totalview binary must be in the search path when
launching tvmpimon. If the mpirun script is the preferred way of starting jobs, it accepts the
“standard” -tv option, however the same rules applies with regards to the search path.

2.3.12 Controlling communication resources

You can set buffer parameters when running applications for performance reasons, but it isn’t
normally necessary. Platform MPI automatically adjusts communication resources based on
the number of processes in each node and based on pool_size and chunk_size.

The built-in devices SMP and TCP/IP use a simplified protocol based on serial transfers. This can
be visualized as data being written into one end of a pipe and read from the other end. Messages
arriving out-of-order are buffered by the reader. The names of these standard devices are SMP
for intra-node communication and TCP for node-to-node-communication.

The size of the buffer inside the pipe can be adjusted by setting the following environment
variables:

Environment variable Description

SCAFUN_TCP_TXBUFSZ Sets the size of the transmit buffer.
SCAFUN_TCP_RXBUFSZ Sets the size of the receive buffer.
SCAFUN_SMP_BUFSZ Sets the size of the buffer for intra-node communication.

The ring buffers are divided into equally sized entries. The size varies differs for different
architectures and networks; see “Platform MPI Release Notes” for details. An entry in the ring
buffer, which is used to hold the information forming the message envelope, is reserved each
time a message is being sent, and is used by the inline protocol, the eagerbuffering protocol, and
the transporter protocol. In addition, one or more entries are used by the inline protocol for
application data being transmitted.

mpimon has the following interface for the eagerbuffer and channel thresholds:

Variable Description
channel_inline_threshold <size> to set threshold for inline buffering
eager_threshold <size> to set threshold for eager buffering

2.3.12.1 Communication resources on DAT-devices

All resources (buffers) used by Platform MPI reside in shared memory in the nodes. This way
multiple processes (typically when a node has multiple CPU’s) can share the communication
resources.

Platform MPI operates on a buffer pool. The pool is divided into equally sized parts called
chunks. Platform MPI uses one chunk per connection to other processes. The mpimon option
“pool_ size” limits the total size of the pool and the “chunk_size” limits the block of memory that
can be allocated for a single connection.

Platform MPI 5.6.5 User’s Guide Page 21

To set the pool_size and chunk_size:

Variable Description
pool_size <size> to set the buffer pool size
chunk_size <size> to set the chunk size.

2.3.13 Good programming practice with Platform MPI

2.3.13.1 Matching MPI_Recv() with MPI_Probe()
During development and testing of Platform MPI, Platform has come across several application
programs with the following code sequence:

while (...)
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
if (sts->MPI_TAG == SOME_VALUE)
MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm,sts);
doStuff();
doOtherStuff();
}

For MPI implementations that have one, and only one, receive-queue for all senders, the
program’s code sequence works as desired. However, the code will not work as expected with
Platform MPI. Platform MPI uses one receive-queue per sender (inside each MPI-process).
Thus, a message from one sender can bypass the message from another sender. In the time-gap
between the completion of MPI_Probe() and before MPI_Recv() matches a message, another
new message from a different MPI-process could arrive, i.e. it is not certain that the message
found by MPI_Probe() is identical to one that MPI_Recv() matches.

The revised code sequence should be:

while (...) {
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
if (sts->MPI_TAG == SOME_VALUE) {
MPI_Recv(buf, cnt, dtype, sts->MPI_SOURCE, sts->MPI_TAG, comm, sts);
doStuff();
}
doOtherStuff();
}

2.3.13.2 MPI_Isend(), MPI_Irecv()

If communication and calculations do not overlap, using immediate calls like MPI_Isend() and
MPI_Irecv() are performance ineffective.

2.3.13.3 MPI_Bsend()
Using buffered send, e.g., MPI_Bsend(), usually degrade performance significantly in
comparison with their unbuffered relatives.

Platform MPI 5.6.5 User’s Guide Page 22

2.3.13.4 Avoid starving MPI-processes - fairness
MPI programs may, if not special care is taken, be unfair and may starve MPI-processes, e.g., by
using MPI_Waitany(), as illustrated for a client-server application in examples 3.15 and 3.16 in
the MPI 1.1 standard [1]. Fairness can also be enforced, e.g. through the use of several tags or
separate communicators.

2.3.13.5 Unsafe MPI programs

Because of different buffering behavior, some programs may run with MPICH, but not with
Platform MPI. Unsafe MPI programs may require resources that are not always guaranteed by
Platform MPI, and deadlock might occur (since Platform MPI uses spin locks, these may
appear to be live locks). If you want to know more about how to write portable MPI programs,
see for example MPI: The complete reference: vol. 1, the MPI core [2].

A typical example that will not work with Platform MPI (for long messages):

while (...) {
MPI_Send(buf, cnt, dtype, partner, tag, comm);
MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
doStuff();
}

This code tries to see the same buffer for both sending and receiving. Such logic can be found,
e.g., where processes from a ring where they communicate with their neighbors. Unfortunately
writing the code this way leads to deadlock, and to make it work the MPI_Send() must be
replaced with MPI_Isend() and MPI_Wait(), or the whole construction should be replaced with
MPI_Sendrecv() or MPI_Sendrecv_replace().

2.3.13.6 Name space pollution

The Platform MPI library is written in C and all of its C names are prefixed with scampi_.
Depending on the compiler used, the user may run into problems if he/she has C code using the
same scampi_ prefix. In addition, there are a few global variables that may cause problems.
All of these functions and variables are listed in the include files mpi.h and mpif.h. Normally,
these files are installed in ${MPI_HOME}/include.

Given that Platform MPI has not fixed its OS routines to specific libraries, it is good
programming practice to avoid using OS functions or standard C-lib functions as application
function names.

STOP!
Warning: Naming routines or global variables as send, recv, open, close, yield,
internal_error, failure, service or other OS reserved names may result in an unpredictable
and undesirable behavior.

Platform MPI 5.6.5 User’s Guide Page 23

2.3.14 Error and warning messages
2.3.14.1 User interface errors and warnings
User interface errors usually result from problems where the setup of the environment causes
difficulties for mpimon when starting an MPI program. mpimon will not start before the
environment is properly defined. These problems are usually easy to fix, by giving mpimon the
correct location of the necessary executable. The error message provides a straight forward
indication of what to do. Thus, only particularly troublesome user interface errors will be listed
here.

Using the -verbose option enables mpimon to print more detailed warnings.

2.3.14.2 Fatal errors

When a fatal error occurs, Platform MPI prints an error message before calling MPI_Abort()
to shut down all MPI-processes.

2.3.14.3 mpimon options

The full list of options accepted by mpimon is listed below. To obtain the actual values used for a
particular run include the -verbose option when starting the application.

Option Description
-affinity_mode <mode> Set affinity-mode for process(es).
-'automatic'<selection> Set automatic-mode for process(es).
-backoff_enable <selection> Set backoff-mode for process(es).
-ccp_mode <mode> Set checkpoint/restart mode.
-channel_entry_count <count> Set number of entries per channel.
-channel_entry_size <size> Set entry_size (in bytes) per channel.
-channel_inline_threshold <size> Set threshold for in-lining (in bytes) per channel.
-channel_size <size> Set buffer size (in bytes) per channel.
-chunk_size <size> Set chunk-size for communication.
-debug <selection> Set debug-mode for process(es).
-debugger <debugger> Set debugger to start in debug-mode.
-disable-timeout Disable process timeout.
disable_toss_page_cache Please see “Platform MPI components” on page 40
<true|false> for more details.
-display <display> Set display to use in debug-/manual-mode.
-dryrun <mode> Set dryrun-mode. The default value is 'none'.
-eager_count <count> Set number of buffers for eager protocol.
-eager_factor <factor> Set factor for subdivision of eagerbuffers.
-eager_size <size> Set buffer size (in bytes) for eager protocol.
-eager_threshold <size> Set threshold (in bytes) for eager protocol.
Table 2.7 mpimon options

Platform MPI 5.6.5 User’s Guide Page 24

 Option Description
-environment <value> Define how to export environment. The default value is
'export'. This is only present for the purpose of being
backwardly compatible. Changing this value will have no
effect in Platform MPI.
-exact_match Set exact-match-mode.
-execpath <execpath> Set path to internal executables.
-failover_mode Set failover-mode
-help Display available options.
-home <directory> Set installation-directory
-immediate_handling <selection> Handling of immediates. The default value is 'lazy'.
-inherit_limits Inherit user-definable limits to processes.
-manual <selection> Set manual-mode for process(es).
-- Separator, marks end of user program options.
-networks <networklist> Define priority order when searching a network.
-pool_size <size> Set buffer-pool-size for communication.
-pr_trace <selection> Enable debug trace for process(es).
-privatekey_path <path> Set path to private key to use when user authentication is
required.
-rsh_mode <remoteshell-binary> Start processes with rsh or equivalent.
-separate_output <selection> Enable separate output for process(es). Filename:
ScaMPIoutput_host_pid_rank.
-shmem_commcount <count> Set number of max communicators for shmem
optimization.
-shmem_chunkcount <size> Set number of chunks for shmem optimization.
-shmem_chunksize <size> Set size of shmem optimization chunk.
-sm_debug <selection> Set debug-mode for submonitor(s).
-sm_manual <selection> Set manual-mode for submonitor(s).
-sm_trace <selection> Enable debug trace for submonitor(s).
-statistics Enable statistics.
-stdin <selection> Distribute standard in to process(es).
-timeout <timeout> Set timeout (elapsed time in seconds) for run.
-timing <timing-spec.> Enable built-in-timing trace
-transporter_count <count> Set number of buffers for transporter-protocol.
-transporter_size <size> Set buffer size (in bytes) for transporter-protocol.
-trace <trace-spec.> Enable built-in trace
-user_authentication Please see “user_authentication” on page 40 for
details on this option
Table 2.7 mpimon options

Platform MPI 5.6.5 User’s Guide Page 25

 Option Description
-verbose Display values for user-options.
-Version Display version of monitor.
-working_directory <directory> Set working directory.
-xterm <xterm> Set xterm to use in debug-/manual-mode.
-zerocopy_count <count> Set number of buffers for zerocopy-protocol.
-zerocopy_size <size> Set buffer size (in bytes) for zerocopy-protocol.
Table 2.7 mpimon options

2.3.14.4 Giving numeric values to mpimon

Numeric values can be given as mpimon options in the following way:

[<prefix>]<numeric value>[<postfix>]

<prefix> selects numeric base when interpreting the value“0x” indicates hex-number (base =
16), “0” indicates octal-number (base = 8). If <prefix> is omitted, decimal-number (base = 10)
is assumed and <postfix> selects a multiplication factor.

K means a multiplication with 1024.

M means a multiplication with 1024 * 1024.

Example
Input: Value as interpreted by mpimon (in decimal):
123 123
0x10 16
0200 128
1K 1 024
2M 2 097 152

Platform MPI 5.6.5 User’s Guide Page 26

Chapter 3 - Using Platform MPI in Windows
This chapter describes how to setup, compile, link and run a program using Platform MPI, and
briefly discusses some useful tools for debugging and profiling.

• “Setting up a Platform MPI environment” on page 27

• “Compiling and linking” on page 27
• “Running Platform MPI programs” on page 29
• “Debugging and profiling” on page 32
• “Controlling communication resources” on page 34
• “Good programming practice with Platform MPI” on page 35
• “Error and warning messages” on page 37
3.1 Setting up a Platform MPI environment
3.1.1 Platform MPI environment variables
The installer automatically adds an environment variable MPI_HOME. It also adds the variables
that point to the paths of binaries.

MPI_HOME is the installation directory.

3.2 Compiling and linking

MPI is an “Application Programming Interface” (API) and not an “Application Binary Interface”
(ABI). This means that in general applications should be recompiled and linked when used with
Platform MPI.

3.2.1 Compiler support

Platform MPI is a C library which uses the cdecl calling convention. This should be compatible
with most compilers.

The details of the process of linking with the Platform MPI libraries vary depending on which
compiler is used. Check the “Platform MPI Release Notes” for information on supported
compilers and how linking is done.

All examples below have been compiled with the Microsoft Visual C compiler and Intel Visual
Fortran. See the documentation for your compiler for the proper syntax and further information.

The pattern for compiling is:

cmd> cl /c /I%MPI_HOME%\include hello-world.c

cmd> ifort /c /I%MPI_HOME%\include hello-world.f

Platform MPI 5.6.5 User’s Guide Page 27

3.2.2 Linker flags
The following versions of MPI libraries are available:

• scampi.lib - Standard library containing the C API.

• scampif.lib - contains the Fortran API wrappers for the standard library.
• scampio.lib - Standard MPI- IO library containing the C API.
• scampiof.lib - the Fortran API wrappers to the MPI-IO library

The pattern for linking C code is:

cmd> link hello-world.obj /libpath:%MPI_HOME%\lib scampi.lib /out:

hello-world.exe

The pattern for linking Fortran code is:

cmd> link hello-world.obj /libpath:%MPI_HOME%\lib scampif.lib /out:

hello-world.exe

3.2.2.1 Notes on compiling with MPI-2 features

To compile and link with the Platform MPI-IO features you need to do the following depending on
whether it is a C or a Fortran program:

For C programs mpio.h must be included in your program and you must link with the
smcmpio.lib library in addition to the Platform MPI 1.2 C library (scampi.lib):

The pattern for compiling and linking C code is:

cmd> cl hello-world.c /I%MPI_HOME%\include /link /libpath:%MPI_HOME%\lib

scampi.lib scampio.lib /out:hello-world.exe

For Fortran programs you will need to include mpiof.h in your program and link with the
scampio.lib library in addition to the Platform MPI 1.2 C and Fortran shared libraries:

The pattern for compiling and linking Fortran code is:

cmd> iforth hello-world.f /I%MPI_HOME%\include /link /libpath:%MPI_HOME%\lib

scampif.lib scampiof.lib /out:hello-world.exe

3.2.2.2 Testing the compilation

Start the hello-world program on the three nodes called nodeA, nodeB and nodeC.

cmd> mpimon hello-world.exe -- nodeA 1 nodeB 1 nodeC 1

Platform MPI 5.6.5 User’s Guide Page 28

The hello-world program should produce the following output:

Hello-world, I'm rank 0; Size is 3

Hello-world, I'm rank 1; Size is 3
Hello-world, I'm rank 2; Size is 3

3.3 Running Platform MPI programs

Note that executables linked to the Platform MPI library cannot be started directly from a shell
prompt. Platform MPI programs must be started using the MPI monitor program mpimon.

Attention
Note: Platform MPI requires a homogenous file system image, i.e. a file system
providing the same path and program names on all nodes of the cluster on which
Platform MPI is installed.

3.3.1 mpimon - monitor program

The control and start-up of an Platform MPI application are monitored by mpimon. A complete
listing of mpimon options can be found in “mpimon options” on page 37.

3.3.1.1 Basic usage

Normally mpimon is invoked as:

<program options> -- <node name> [<count>][<nodename> [<count>]]...

<userprogram> is name of application
<program options> are options to the application
"--" is the separator ending the application options
<nodename>[<count>] is name of node and the number of MPI-processes to run on that node.
The option can occur several times in the list. Mpi-processes will be
given ranks sequentially according to the list of node-number pairs.
<count> is optional and defaults to 1.
Table 3.1 mpimon usage (windows)

Example
Starting the program “opt\Platform\examples\bin\hello” on a node called “hugin”:
mpimon C:\Program Files\Platform\Platform MPI\examples\bin\hello -- hugin

For more information regarding bracket expansion and grouping, refer to “Bracket expansion
and grouping” on page 86.

Platform MPI 5.6.5 User’s Guide Page 29

3.3.1.2 Identity of parallel processes

The identification of nodes and the number of processes to run on each particular node
translates directly into the rank of the MPI processes. For example, specifying n1 2 n2 2 will
place process 0 and 1 on node n1 and process 2 and 3 on node n2. On the other hand, specifying
n1 1 n2 1 n1 1 n2 1 will place process 0 and 2 on node n1 while process 1 and 3 are placed on
node n2.

This control over placement of processes can be very valuable when application performance
depends on all the nodes having the same amount of work to do.

3.3.1.3 Controlling options to mpimon

The program mpimon has a multitude of options which can be used for optimising Platform
MPI performance. Normally it should not be necessary to use any of these options. However,
unsafe MPI programs might need buffer adjustments to solve deadlocks. Running multiple
applications in one run may also be facilitated with some of the “advanced options”.

mpimon also has an advanced tracing option to enable pinpointing of communication

bottlenecks.

The complete syntax for the program is as follows:

mpimon [<mpimon option>]... <program_&_node-spec> [--

Syntax <program_&_node-spec>]...
<mpimon_options> are options to mpimon (“See mpimon options” and the Platform
MPI Release Notes for a complete list of options),
-- is the separator that signals end of user program options.
<program_&_node-spec> is an application and node specification consisting of <program
spec> -- <node spec> [<node spec>]...,
<program spec> is an application and application-options specification
(<userprogram>[<programoptions>])
<node spec> specifies which node and how many instances (<nodename>
[<count>]). If <count> is omitted, one MPI-process is started on
each node specified.
Table 3.2 mpimon options (windows)

Example
Starting the program “\opt\Platform\examples\bin\hello” on a node called “hugin” and the
program “\opt\Platform\examples\bandwidth” with 2 processes on “munin”:
mpimon C:\Program Files\Platform\Platform MPI\examples\bin\hello -- hugin --
C:\Program Files\Platform\Platform MPI\examples\bin\bandwidth -- munin 2

Example
Changing one of the mpimon-parameters:

Platform MPI 5.6.5 User’s Guide Page 30

Example
mpimon -channel_entry_count 32 C:\Program Files\Platform\Platform
MPI\examples\bin\hello -- hugin

3.3.1.4 Standard input

The -stdin option specifies which MPI-process rank should receive the input. You can in fact
send stdin to all the MPI-processes with the all argument, but this requires that all
MPI-processes read the exact same amount of input. The most common way of doing it is to
send all data on stdin to rank 0:

mpimon -stdin 0 myprogram -- node1 node2... <input_file>

Note: The default for -stdin is none.

3.3.1.5 Standard output

By default the processes’ output to stdout all appear in the stdout of mpimon, where they are
merged in some random order. It is however possible to keep the outputs apart by directing
them to files that have unique names for each process. This is accomplished by giving mpimon
the option -separate_output <seletion>, e.g., -separate_output all to have each process deposit
its stdout in a file. The files are named according to the following template:
ScaMPIoutput_<host>_<pid>_<rank>, where <host> and <pid> identify the particular
invocation of mpimon on the host, and <rank> identifies the process.

It is possible to control the placement of the output file by setting the environment variable
SCAMPI_OUTPUT_FILE=prefix. The prefix will replace the ScaMPIoutput portion of the filenames
used for output. For example, by setting SCAMPI_OUTPUT_FILE=\tmp\MyOutput together with
the -separate_output option, the output files will be placed in the \tmp directory and be named
MyOutput_<host>_<pid>_<rank>.

3.3.1.6 How to provide options to mpimon

There are three different ways to provide options to mpimon. The most common way is to
specify options on the command line invoking mpimon. Another way is to define environment
variables, and the third way is to define options in configuration file(s).

• Command line options: Options for mpimon must be placed after mpimon, but before the
program name
• Environment-variable options: Setting an mpimon-option with environment variables
requires that variables are defined as SCAMPI_<uppercase-option> where SCAMPI_ is a
fixed prefix followed by the option converted to upper case. For example
SCAMPI_CHANNEL_SIZE=64K means setting -channel_size to 64K
• Configuration-files options: mpimon reads up to three different configuration files when
starting. First the system-wide configuration
(C:\Program Files\Platform\Platform MPI\etc\ScaMPI.conf) is read. If the user has
a file on his\her home-directory, that file(${HOME}\ScaMPI.conf) is then read. Finally if
there is a configuration file in the current directory, that file(.\ScaMPI.conf) is then read.
The files should contain one option per line, given as for command line options.

Platform MPI 5.6.5 User’s Guide Page 31

The options described either on the command line, as environment variables or in configuration
files are prioritized the following way (ranked from lowest to highest):

1 System-wide configuration-file
(C:\Program Files\Platform\Platform MPI\etc\ScaMPI.conf)
2 Configuration-file on home-directory(${HOME}\ScaMPI.conf)
3 Configuration-file on current directory(.\ScaMPI.conf)
4 Environment-variables
5 Command line-options
3.3.1.7 Network options

With Platform MPI you can utilize multiple networks in the same invocation.

There are two types of networks, built-in standard-devices and DAT-devices. The devices are
selected by giving the option “-networks <net-list>” to mpimon. <net-list> is a
comma-separated list of device names. Platform MPI uses the list when setting up connections
to other MPI processes. Each point-to-point connection between MPI processes starts off with
the first device in the list and sets up all possible connections with that device. If this fails the
next on list is tried and so on until all connections are live or all adapters in <net-list> have been
tried.

Example
For systems installed with the Platform Manage installer, a list of preferred devices is provided in
ScaMPI.conf. An explicit list of devices may be set either in a private ScaMPI.conf, through
the SCAMPI_NETWORKS environment variable, or by the -networks parameter to mpimon. The
values should be provided in a comma-separated list of device names.
mpimon -networks smp,gm0,tcp...

For each MPI process Platform MPI will try to establish contact with each other MPI process, in
the order listed. This enables mixed interconnect systems, and provides a means for working
around failed hardware.

In a system interconnect where the primary interconnect is Myrinet, if one node has a faulty
card, using the device list in the example, all communication to and from the faulty node will
happen over TCP\IP while the remaining nodes will use Myrinet. This offers the unique ability to
continue running applications over the full set of nodes even when there are interconnect faults.

3.4 Debugging and profiling

The complexity of debugging programs can grow dramatically when going from serial to parallel
programs. So to assist in debugging MPI programs Platform MPI has a number of features
built-in like tracing processes.

3.4.1 Debugging with a sequential debugger

Platform MPI applications can be debugged using a sequential debugger such as the Microsoft
Visual Studio.

Platform MPI 5.6.5 User’s Guide Page 32

3.4.1.1 Built-in-tools for debugging

Built-in tools for debugging in Platform MPI covers discovery of the MPI calls used through
tracing and timing, and an attachment point to processes that fault with segmentation violation.
The tracing (see “Tracing” on page 53) and timing (see “Timing” on page 57) are covered
in Chapter 4.

3.4.1.2 Using built-in segment protect violation handler

When running applications that terminate with a SIGSEGV-signal it is often useful to be able to
freeze the situation instead of exiting, the default behavior. In previous versions of Platform
MPI you could use SCAMPI_INSTALL_SIGSEGV_HANDLER.
SCAMPI_INSTALL_SIGNAL_HANDLER has replaced this environment variable. This allows for
Platform MPI to be backwardly compatible with earlier versions of MPI. The values for the
signals are put into a comma separated list.

SCAMPI_INSTALL_SIGNAL_HANDLER=11

will install a handler for signal number 11 (which is the same as SIGSEGV).

To install a handler for signal 11 (SIGSEGV) and signal 4 (SIGILL) enter:

SCAMPI_INSTALL_SIGNAL_HANDLER=11,4

The default action has the handler dump all registers and starts processes looping. Attaching
with a debugger will then make it possible to examine the situation which resulted in the
segment protect violation.

As an alternative the handler dumps all registers but all processes will exit afterwards.

You may attach a debugger to process <pid> on a machine with your preferred debugger. In
general, this will allow the debugger to inspect the stack trace and identify the functions active
when the sigsegv occurred, and disassemble the functions. If the application is compiled with
debug info and the source code is available, then source level debugging can be carried out.

3.4.2 Assistance for external profiling

Profiling parallel applications is complicated by having multiple processes at the same time. But
Platform MPI comes to assistance; through the SCAMPI_PROFILE_APPLICATION environment
variable together with the -separate_output option (SCAMPI_SEPARATE_OUTPUT) the output
from the application runs is directed at one file per process for easier use.

The environment variables SCAMPI_PROFILE_APPLICATION_START and

SCAMPI_PROFILE_APPLICATION_END are also available for steering the range of memory
addresses applicable to profiling.

Platform MPI 5.6.5 User’s Guide Page 33

3.5 Controlling communication resources
You can set buffer parameters when running applications for performance reasons, but it isn’t
normally necessary. Platform MPI automatically adjusts communication resources based on
the number of processes in each node and based on pool_size and chunk_size.

The built-in devices SMP and TCP\IP use a simplified protocol based on serial transfers. This can
be visualized as data being written into one end of a pipe and read from the other end. Messages
arriving out-of-order are buffered by the reader. The names of these standard devices are SMP
for intra-node communication and TCP for node-to-node-communication.

The size of the buffer inside the pipe can be adjusted by setting the following environment
variables:

Environment variable Description

SCAFUN_TCP_TXBUFSZ Sets the size of the transmit buffer.
SCAFUN_TCP_RXBUFSZ Sets the size of the receive buffer.
SCAFUN_SMP_BUFSZ Sets the size of the buffer for intra-node communication.

The ring buffers are divided into equally sized entries. The size varies differs for different
architectures and networks; see “Platform MPI Release Notes” for details. An entry in the ring
buffer, which is used to hold the information forming the message envelope, is reserved each
time a message is being sent, and is used by the inline protocol, the eagerbuffering protocol, and
the transporter protocol. In addition, one or more entries are used by the in-line protocol for
application data being transmitted.

mpimon has the following interface for the eagerbuffer and channel thresholds:

Variable Description
channel_inline_threshold <size> to set threshold for inline buffering
eager_threshold <size> to set threshold for eager buffering

3.5.1 Communication resources on DAT-devices

All resources (buffers) used by Platform MPI reside in shared memory in the nodes. This way
multiple processes (typically when a node has multiple CPU’s) can share the communication
resources.

Platform MPI operates on a buffer pool. The pool is divided into equally sized parts called
chunks. Platform MPI uses one chunk per connection to other processes. The mpimon option
pool_ size” limits the total size of the pool and the “chunk_size” limits the block of memory that
can be allocated for a single connection.

Platform MPI 5.6.5 User’s Guide Page 34

To set the pool size and the chunk size, specify:

Variable Description
pool_size <size> to set the buffer pool size
chunk_size <size> to set the chunk size.

3.6 Good programming practice with Platform MPI

3.6.1 Matching MPI_Recv() with MPI_Probe()
During development and testing of Platform MPI, Platform has come across several application
programs with the following code sequence:

while (...)
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
if (sts->MPI_TAG == SOME_VALUE)
MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm,sts);
doStuff();
doOtherStuff();
}

For MPI implementations that have one, and only one, receive-queue for all senders, the
program’s code sequence works as desired. However, the code will not work as expected with
Platform MPI. Platform MPI uses one receive-queue per sender (inside each MPI-process).
Thus, a message from one sender can bypass the message from another sender. In the time-gap
between the completion of MPI_Probe() and before MPI_Recv() matches a message, another
new message from a different MPI-process could arrive, i.e. it is not certain that the message
found by MPI_Probe() is identical to one that MPI_Recv() matches.

The revised code sequence should be:

while (...) {
MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
if (sts->MPI_TAG == SOME_VALUE) {
MPI_Recv(buf, cnt, dtype, sts->MPI_SOURCE, sts->MPI_TAG, comm, sts);
doStuff();
}
doOtherStuff();
}

3.6.2 MPI_Isend(), MPI_Irecv()

If communication and calculations do not overlap, using immediate calls like MPI_Isend() and
MPI_Irecv() are performance ineffective.

Platform MPI 5.6.5 User’s Guide Page 35

3.6.3 MPI_Bsend()
Using buffered send, e.g., MPI_Bsend(), usually degrade performance significantly in
comparison with their unbuffered relatives.

3.6.4 Avoid starving MPI-processes - fairness

MPI programs may, if not special care is taken, be unfair and may starve MPI-processes, e.g., by
using MPI_Waitany(), as illustrated for a client-server application in examples 3.15 and 3.16 in
the MPI 1.1 standard [1]. Fairness can also be enforced, e.g. through the use of several tags or
separate communicators.

3.6.5 Unsafe MPI programs

Because of different buffering behavior, some programs may run with MPICH, but not with
Platform MPI. Unsafe MPI programs may require resources that are not always guaranteed by
Platform MPI, and deadlock might occur (since Platform MPI uses spin locks, these may
appear to be live locks). If you want to know more about how to write portable MPI programs,
see for example MPI: The complete reference: vol. 1, the MPI core [2].

A typical example that will not work with Platform MPI (for long messages):

while (...) {
MPI_Send(buf, cnt, dtype, partner, tag, comm);
MPI_Recv(buf, cnt, dtype, MPI_ANY_SOURCE, MPI_ANY_TAG, comm, sts);
doStuff();
}

This code tries to see the same buffer for both sending and receiving. Such logic can be found,
e.g., where processes from a ring where they communicate with their neighbors. Unfortunately
writing the code this way leads to deadlock, and to make it work the MPI_Send() must be
replaced with MPI_Isend() and MPI_Wait(), or the whole construction should be replaced with
MPI_Sendrecv() or MPI_Sendrecv_replace().

3.6.6 Name space pollution

The Platform MPI library is written in C and all of its C names are prefixed with scampi_.
Depending on the compiler used, the user may run into problems if he\she has C code using the
same scampi_ prefix. In addition, there are a few global variables that may cause problems.
All of these functions and variables are listed in the include files mpi.h and mpif.h. Normally,
these files are installed in C:\Program Files\Platform\Platform MPI\include.

Platform MPI 5.6.5 User’s Guide Page 36

Given that Platform MPI has not fixed its OS routines to specific libraries, it is good
programming practice to avoid using OS functions or standard C-lib functions as application
function names.

STOP!
Warning: Naming routines or global variables as send, recv, open, close, yield, internal_error,
failure, service or other OS reserved names may result in an unpredictable and undesirable
behavior.

3.7 Error and warning messages

3.7.1 User interface errors and warnings
User interface errors usually result from problems where the setup of the environment causes
difficulties for mpimon when starting an MPI program. mpimon will not start before the
environment is properly defined. These problems are usually easy to fix, by giving mpimon the
correct location of the necessary executable. The error message provides a straight forward
indication of what to do. Thus, only particularly troublesome user interface errors will be listed
here.

Using the -verbose option enables mpimon to print more detailed warnings.

3.7.2 Fatal errors

When a fatal error occurs, Platform MPI prints an error message before calling MPI_Abort()
to shut down all MPI-processes.

3.7.3 mpimon options

The full list of options accepted by mpimon is listed below. To obtain the actual values used for a
particular run include the -verbose option when starting the application.

Option Description
-affinity_mode <mode> Set affinity-mode for process(es). The default value is
'automatic'. The legal choices are 'automatic' and 'none'.
-'automatic'<selection> Set automatic-mode for process(es).
-backoff_enable <selection> Set backoff-mode for process(es). The default value is 'none'.
-channel_entry_count Set number of entries per channel. The default value is
<count> '512,smp:32'.
-channel_entry_size <size> Set entry_size (in bytes) per channel.
-channel_inline_threshold Set threshold for in-lining (in bytes) per channel.
<size>
-channel_size <size> Set buffer size (in bytes) per channel.
-chunk_size <size> Set chunk-size for communication.
Table 3.3 mpimon options (windows)

Platform MPI 5.6.5 User’s Guide Page 37

 Option Description
-disable-timeout Disable process timeout.
disable_toss_page_cache Please see “Platform MPI components” on page 40 for
<true|false> more details.
-dryrun <mode> Set dryrun-mode. The default value is 'none'.
-eager_count <count> Set number of buffers for eager protocol.
-eager_factor <factor> Set factor for subdivision of eagerbuffers.
-eager_size <size> Set buffer size (in bytes) for eager protocol.
-eager_threshold <size> Set threshold (in bytes) for eager protocol.
-environment <value> Define how to export environment. The default value is
'export'. This is only present for the purpose of being
backwardly compatible. Changing this value will have no effect
in Platform MPI.
-exact_match Set exact-match-mode.
-execpath <execpath> Set path to internal executables.
-help Display available options.
-home <directory> Set installation-directory
-immediate_handling Handling of immediates. The default value is 'lazy'.
<selection>
-manual <selection> Set manual-mode for process(es).
-- Separator, marks end of user program options.
-networks <networklist> Define priority order when searching a network.
-pool_size <size> Set buffer-pool-size for communication.
-pr_trace <selection> Enable debug trace for process(es).
-privatekey_path <path> Set path to private key to use when user authentication is
required.
-rsh_mode <binary for remote Start processes with rsh or equivalent.
shell>
-separate_output <selection> Enable separate output for process(es). Filename:
ScaMPIoutput_host_pid_rank.
-shmem_commcount <count> Set number of max communicators for shmem optimization.
-shmem_chunkcount <size> Set number of chunks for shmem optimization.
-shmem_chunksize <size> Set size of shmem optimization chunk.
-sm_debug <selection> Set debug-mode for submonitor(s).
-sm_manual <selection> Set manual-mode for submonitor(s).
-sm_trace <selection> Enable debug trace for submonitor(s).
-statistics Enable statistics.
-stdin <selection> Distribute standard in to process(es).
Table 3.3 mpimon options (windows)

Platform MPI 5.6.5 User’s Guide Page 38

 Option Description
-timeout <timeout> Set timeout (elapsed time in seconds) for run.
-timing <timing-spec.> Enable built-in-timing trace
-transporter_count <count> Set number of buffers for transporter-protocol.
-transporter_size <size> Set buffer size (in bytes) for transporter-protocol.
-trace <trace-spec.> Enable built-in trace
-user_authentication Please see “user_authentication” on page 40 for details
on this option
-verbose Display values for user-options.
-Version Display version of monitor.
-working_directory Set working directory.
<directory>
-xterm <xterm> Set xterm to use in debug manual-mode.
-zerocopy_count <count> Set number of buffers for zerpcopy-protocol.
-zerocopy_size <size> Set buffer size (in bytes) for zerocopy-protocol.
Table 3.3 mpimon options (windows)

3.7.4 Giving numeric values to mpimon

Numeric values can be given as mpimon options in the following way:

[<prefix>]<numeric value>[<postfix>]

<prefix> selects numeric base when interpreting the value“0x” indicates hex-number (base =
16), “0” indicates octal-number (base = 8). If <prefix> is omitted, decimal-number (base = 10)
is assumed and <postfix> selects a multiplication factor.

K means a multiplication with 1024.

M means a multiplication with 1024 * 1024.

Example
Input: Value as interpreted by mpimon (in decimal):
123 123
0x10 16
0200 128
1K 1 024
2M 2 097 152

Platform MPI 5.6.5 User’s Guide Page 39

Chapter 4 - Operation of Platform MPI
This chapter gives the details of the operation of Platform MPI (Platform MPI). Platform MPI
consists of libraries to be linked and loaded with user application program(s), and a set of
executable files which control the start-up and execution of the user application program(s). The
relationship between these components and their interfaces are described in this chapter. It is
necessary to understand this chapter in order to control the execution of parallel processes and
be able to tune Platform MPI for optimal application performance.

4.1 Platform MPI components

Platform MPI consists of a number of programs, daemons, libraries, include and configuration
files that together implements the MPI functionality needed by applications.

Starting applications rely on the following daemons and launchers:

• mpimon is a monitor program which is the user’s interface for running the application program.
• mpisubmon is a submonitor program which controls the execution of application programs. One
submonitor program is started on each node per run.
• mpiboot is a bootstrap program used when running in manual-/debug-mode.
• Platformservices is a service responsible for starting daemons.
• mpid is a daemon program running on all nodes that are able to run Platform MPI. mpid is used for
starting the mpisubmon programs (to avoid using Unix facilities like the remote shell rsh). mpid is
started automatically when a node boots, and must run at all times. There are two options that are for
mpid use only:
• user_authentication <true|false>
• disable_toss_page_cache <true|false>

These options can be set in the global ${MPI_HOME}/etc/ScaMPI.conf (in Windows:

%MPI_HOME%\etc\ScaMPI.conf) only and is read by the mpi daemon (mpid) when it
starts up.

4.1.1 user_authentication
When set to “true” all users are required to generate public/private key pairs in order to launch
mpi applications. Key pairs can be generated by running ìmpikeygenî. Key pairs should be
located in the users home directory in the Unix subdirectory ${HOME}/.scampi
(in the Windows subdirectory: %APPDATA%/.scampi).

4.1.2 disable_toss_page_cache
When set to “true”, mpid will no longer toss the page cache prior to starting the application.
Tossing of the page cache is useful on NUMA machines to ensure memory locality and thus most
optimal performance. If you are experiencing kernel issues or weird behavior you might consider
disabling this feature

Platform MPI 5.6.5 User’s Guide Page 40

4.2 Application startup.

Figure 4.1 Application startup

Figure 4.1 illustrates how applications started with mpimon have their communication system
established by a system of daemons on the nodes. This process uses TCP/IP communication over
the networking Ethernet, whereas optional high performance interconnects are used for
communication between processes.

Parameter control is performed by mpimon to check as many of the specified options and
parameters as possible. The user program names are checked for validity, and the nodes are
contacted (using sockets) to ensure they are responding and that mpid is running.

Via mpid mpimon establishes contact with the nodes and transfers basic information to
enable mpid to start the submonitor mpisubmon on each node. Each submonitor establishes a
connection to mpimon for exchange of control information between each mpisubmon and
mpimon to enable mpisubmon to start the specified userprograms (MPI-processes).

As mpisubmon starts all the MPI-processes to be executed they MPI_Init(). Once inside here
the user processes wait for all the mpisubmons involved to coordinate via mpimon. Once all
processes are ready mpimon will return a “start running” message to the processes. They will
then return from MPI_Init() and start executing the user code.

Stopping MPI application programs is requested by the user processes as they enter the
MPI_Finalize() call. The local mpisubmon will signal mpimon and wait for mpimon to return a “all
stopped message”. This comes when all processes are waiting in MPI_Finalize(). As the user
processes return from the MPI_Finalize() they release their resources and terminates. Then the
local mpisubmon terminates and eventual mpimon terminates.

Platform MPI 5.6.5 User’s Guide Page 41

Platform MPI offers generic support for interconnects. This does not yet mean that every
interconnect is supported out of the box, since Platform MPI still requires a driver for each
interconnect. But from Platform MPI's point of view, a driver is just a library, which in turn may
(e.g. Myrinet) or may not require a kernel driver (e.g. TCP/IP).

These libraries provide a network device for Platform MPI.

4.3 Network devices

There are two basic types of network devices in Platform MPI, native and DAT. The native
devices are built-in and are neither replaceable nor upgradable without replacing the Platform
MPI package. There are currently five built in devices, SMP, TCP, IB, and GM InfiniPath.

The TCP/IP stack can run on any network supported by the socket stream. That includes
Ethernet, Myrinet, Infiniband, and SCI. For more details please refer to the Release Notes
included with the Platform MPI package.

To find out what network device is used between two processes, set the environment variable
SCAMPI_NETWORKS_VERBOSE=2. The MPI library will print out during startup a table of processes
and devices in use.

Figure 4.2 Network devices part 1

The other type of devices use the DAT µDAPL API in order to have an open API for generic third
party vendors. µDAPL is an abbreviation for User DAT Provider library. This is a shared library
that Platform MPI loads at runtime through the static DAT registry. These libraries are normally
listed in /etc/dat.conf (in Windows: C:\dat\dat.conf). For clusters using ‘exotic’ interconnects

Platform MPI 5.6.5 User’s Guide Page 42

whose vendor provides a µDAPL shared object, these can be added to this file (if this isn’t done
automatically by the vendor). The device name is given by the µDAPL, and the interconnect
vendor must provide it. Please see the release notes for updated lists of supported interconnects
for platforms.

Figure 4.3 Network devices part 2

Please note that Platform has a certification program, and may not provide support for unknown
third party vendors.

The DAT header files and registry library conforming to the µDAPL v1.2 specification, is provided
by the dat-registry package.

For more information on DAT, please refer to

http://www.datcollaborative.org.

4.3.1 Shared Memory Device

The SMP device is a shared memory device that is used exclusively for intra-node
communication and use SYS V IPC shared memory. Multiple CPU nodes are frequent in clusters,
and SMP provide optimal communication between the CPU’s. In cases where only one processor
per node is used, SMP is not used.

4.3.2 Ethernet Devices

An Ethernet for networking is a basic requirement for a cluster. For some uses this also has
enough performance for carrying application communication. To serve this Platform MPI has a
TCP device.

Platform MPI 5.6.5 User’s Guide Page 43

4.3.2.1 TCP

The TCP device is really a generic device that works over any TCP/IP network, even WAN’s. This
network device requires only that the node names given to mpimon map correctly to the nodes
IP address. TCP/IP connectivity is required for Platform MPI operation, and for this reason the
TCP device is always operational.

Attention
Note: Users should always append the TCP device at the end of a device list as the device of last
resort. This way communication will fall back to the management ethernet that anyway has to be
present for the cluster to work.

4.3.2.2 Myrinet GM

This is a RDMA capable device that uses the Myricom GM driver and library. A GM release above
2.0 is required. This device is straight forward and requires no configuration other than the
presence of the libgm.so library in the library path (see /etc/ld.so.conf).

Attention
Note: Myricom GM software is not provided by Platform. If you have purchased a
Myrinet interconnect you have the right to use the GM source, and a source tar ball is
available from Myricom. It is necessary to obtain the GM source since it must be
compiled per kernel version.
Your management software should provide tools for generating binary RPM’s to ease
installing and management.
If you used Platform Manage to install your compute nodes, and supplied it with the GM
source tar ball, the installation is already complete.

4.3.2.3 Infiniband

Infiniband is an interconnect that has been available since 2002 and has gained popularity in
recent years. There are several flavors to choose from, each having different performance
profiles. Infiniband vendors provide different hardware and software environments. Platform
MPI supports 3 different Infiniband software stack types:

• OFED (libibverbs.so.1) is the new OpenIB software stack which all vendors are
embracing (though they still ship their proprietary stacks. Both OFED 1.1 and
1.2 work with Platform MPI 5.5.
• IBT (/dev/SysIbt) is Silverstorm specific only and is only available when running the
Silverstorm Infiniband stack.
• VAPI (libvapi.so) is the Mellanox type stack used by Cisco/Topspin and Voltaire.

Platform MPI 5.6.5 User’s Guide Page 44

Check with your proprietary hardware vendor driver documentation.

Attention
Note: When you specify “-networks ib”, Platform MPI will try OFED first, then
Silverstorm and finally VAPI. If the first two aren't available you will see “Mutable
errors” in the output when running with the -verbose flag. If none of them are available,
you will get the error message “Unable to complete network setup”. See release notes
on the exact versions of software stack that Platform MPI supports.

Your management software should provide a utility that automates installation of some of these
stacks.

The different vendors’ InfiniBand switches vary in feature sets, but the most important
difference is whether they have a built in subnet manager or not. An InfiniBand network must
have a subnet manager (SM) and if the switches don't come with a built-in SM, one has to be
started on a node attached to the IB network. The SM’s of choice for software SM’s are OpenSM
or minism. If you have SM-less switches your vendor will provide one as part of their software
bundle.

Platform MPI uses either the µDAPL (User DAT Provider Library) supplied by the IB vendor, or
the low level VAPI/IBA layer. DAT is an established standard and is guaranteed to work with
Platform MPI. However better performance is usually achieved with the VAPI/IBT interfaces.
However, VAPI is an API that is in flux and Platform MPI is not guaranteed to work with all
(current nor future) versions of VAPI.

4.3.2.4 Multi-rail Infiniband

Multirailing is the ability to stripe data across the multiple Infiniband HCA’s in a system as well as
to balance HCA loads between processes on the same system. See “Multi-rail Support in
Platform MPI” on page 92 for more details.

4.3.3 Communication protocols

In Platform MPI, the communication protocol used to transfer data between a sender and a
receiver depends on the size of the message to transmit, excepting TCP and PMS, as illustrated
in Figure 4.4.

Platform MPI 5.6.5 User’s Guide Page 45

l

Figure 4.4 Thresholds for different communication protocols

The default thresholds that control whether a message belongs to the in-line, eager buffering or
transporter protocols can be controlled from the application launch program (mpimon) described
in chapter 3.

Figure 4.4 illustrates the node resources associated with communication and mechanisms
implemented in Platform MPI for handling messages of different sizes. The communication
protocols from Figure 4.4 rely on buffers located in the main memory of the nodes. This memory
is allocated as shared, i.e., it is not private to a particular process in the node. Each process has
one set of receiving buffers for of the processes it communicates with.

As the figure shows all communication relies on the sending process depositing messages
directly into the communication buffers of the receiver. For In-line and eager buffering the
management of the buffer resources does not require participation from the receiving process,
because of their designs as ring buffers.

4.3.3.1 Channel buffer

The Channel ring buffer is divided into equally sized entries. The size varies differs for different
architectures and networks; see “Platform MPI Release Notes” for details. An entry in the
ring buffer, which is used to hold the information forming the message envelope, is reserved
each time a message is being sent, and is used by the In-line protocol, the eager buffering
protocol, and the transporter protocol. In addition, one ore more entries are used by the In-line
protocol for application data being transmitted.

Platform MPI 5.6.5 User’s Guide Page 46

4.3.3.2 Inline protocol

If the message data is small enough, the sender uses the in-line protocol to write the data in the
message header, using one or more of the receiver's channel ring buffer entries.

Figure 4.5 Inline protocol

Platform MPI 5.6.5 User’s Guide Page 47

4.3.3.3 Eager buffering protocol

The sender uses the eager buffering protocol when sending medium-sized messages to the
receiver's channel ring buffer, identifying the data's position in the eager buffer.

The sender allocates buffer resources, which are released by the receiver, without any explicit
communication between the two communicating partners.

The eager buffering protocol uses one channel ring buffer entry for the message header, and one
eager buffer entry for the application data being sent.

Figure 4.6 Eager Buffering Protocol

Platform MPI 5.6.5 User’s Guide Page 48

4.3.3.4 Transporter protocol

When large messages are transferred the transporter protocol is used. The sender
communicates with the receiver to negotiate which resources to use for data transport. The
receiver's runtime system processes the message and returns a response.

The transporter protocol utilizes one channel ring buffer entry for the message header, and
transporter buffers for the application data being sent. The sender's runtime system
de-fragments and reassembles large messages whose size is larger than the size of the
transporter ring buffer entry (transporter_size). The data is then written to the agreed upon
position in the transporter ring buffer.

Figure 4.7 Transporter Protocol

Platform MPI 5.6.5 User’s Guide Page 49

4.3.3.5 Zerocopy protocol
The zerocopy protocol is a special case of the transporter protocol. The zerocopy protocol sends
data directly to the receiver instead of being buffered anywhere.

Figure 4.8 Zero Copy Protocol

Platform recommends that you select the zerocopy protocol if your underlying hardware can
support it. To disable the zerocopy protocol, set the zerocopy_count or the zerocopy_size
parameters to 0.

Platform MPI 5.6.5 User’s Guide Page 50

Chapter 5 - Profiling with Platform MPI
The Platform MPI communication library has a number of built-in timing and trace facilities.
These features are built into the run time version of the library, so no extra recompiling or linking
of libraries is needed. All MPI calls can be timed and/or traced. A number of different
environment variables control this functionality. In addition an implied barrier call can be
automatically inserted before all collective MPI calls. All of this can give detailed insights into
application performance.

The trace and timing facilities are initiated by environment variables that either can be set and
exported or set at the command line just before running mpimon.

There are different tools available that can be useful to detect and analyze the cause of
performance bottlenecks:

• Built-in proprietary trace and profiling tools provided with Platform MPI
• Commercial tools that collect information during run and post processes and present
results afterwards such as Vampir from Pallas GmbH.
See http://www.pallas.de for more information.
The main difference between these tools is that the Platform MPI built-in tools can be used
with an existing binary while the other tools require reloading with extra libraries.

The powerful run time facilities Platform MPI trace and Platform MPI timing can be used to
monitor and keep track of MPI calls and their characteristics. The various trace and timing
options can yield many different views of an application's usage of MPI. Common to most of
these logs are the massive amount of data which can sometimes be overwhelming, especially
when run with many processes and using both trace and timing concurrently.

The second part shows the timing of these different MPI calls. The timing is a sum of the timing
for all MPI calls for all MPI processes and since there are many MPI processes the timing can look
unrealistically high. However, it reflects the total time spent in all MPI calls. For situations in
which benchmarking focuses primarily on timing rather than tracing MPI calls, the timing
functionality is more appropriate. The trace functionality introduces some overhead and the total
wall clock run time of the application goes up. The timing functionality is relatively light and can
be used to time the application for performance benchmarking.

Example
To illustrate the potential of tracing and timing with Platform MPI consider the code fragment
below (full source reproduced in E.1).

Platform MPI 5.6.5 User’s Guide Page 51

Example
int main( int argc, char** argv )
{
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Comm_size( MPI_COMM_WORLD, &size );
/* read image from file */

/* broadcast to all nodes */

MPI_Bcast( &my_count, 1, MPI_INT, 0, MPI_COMM_WORLD );
/* scatter the image */
MPI_Scatter( pixels, my_count, MPI_UNSIGNED_CHAR, recvbuf,
my_count, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD );
/* sum the squares of the pixels in the sub-image */

/* find the global sum of the squares */

MPI_Reduce( &my_sum, &sum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD );
/* let rank 0 compute the root mean square */

/* rank 0 broadcasts the RMS to the other nodes */

MPI_Bcast( &rms, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD );
/* perform filtering operation (contrast enhancement) */

/* gather back to rank 0 */

MPI_Gather( recvbuf, my_count, MPI_UNSIGNED_CHAR, pixels,
my_count, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD );
/* write image to file */
MPI_Finalize();
}

This code uses collective operations (broadcast, scatter, reduce and gather) to employ multiple processes to perform
operations on a image. For example, the figure below () shows the result of processing an ultrasonic image of a fetus.

Figure 5.1 A ultrasound image before and after contrast enhancement

Platform MPI 5.6.5 User’s Guide Page 52

5.1 Tracing
Tracing enables features in Platform MPI’s implementataion of MPI that report detail about the
MPI calls. By capturing the printout of the tracing information the user can monitor the
development of the application run, perform analysis of the application run, figure out how the
application uses the communication mechanisms, and discover details that can be used to
improve performance.

5.1.1 Using Platform MPI built-in trace

To use built-in trace-facility you need to set the mpimon-option -trace “<options>” specifying
what options you want to apply. The following options can be specified: (<...-list> is a
semicolon-separated list of Posix-regular-expressions.)

option Description
-b Trace beginning and end of each MPI_call
-s <seconds> Start trace after <seconds> seconds
-S <seconds> End trace after <seconds> seconds
-c <calls> Start trace after <calls>MPI_calls
-C <calls> End trace after <calls>MPI_calls
-m <mode> Special modes for trace

<mode> = “sync”: Synchronize with MPI_Barrier before starting

the collective call
-p <selection> Enable for process(es): 'n,m,o..' = (list) or

'n-m' = (range) or 'all'

-t <call-list> Enable for MPI_calls in <call-list>.

MPI_call = 'MPI_call' | 'call'

-x <call-list> Disable for MPI_calls in <call-list>.

MPI_call = 'MPI_call' | 'call'

-f <format-list> Define format: 'timing', 'arguments', 'rate'
-v Verbose. By default only one line is written per MPI-call.
Table 5.1 mpimon trace-options

This format can be extended by using the "-f"-option. Adding "-f arguments" will provide
some additional information concerning message length. If "-f timing" is given some timing
information between the <absRank>and <MPIcall>-fields is provided.

Calls may be specified with or without the "MPI_"-prefix, and in upper- or lower-case. The
default format of the output has the following parts:

The extra field has the following format:

Platform MPI 5.6.5 User’s Guide Page 53

 <absRank>:
Syntax <MPIcall><commName>_<rank><call-dependant-parameters>
<absRank> is the rank within MPI_COMM_WORLD
<MPIcall> is the name of the MPI-call
<commName> is the name of the communicator
<rank> is the rank within the communicator used

Table 5.2 Default trace-output

Syntax <relSecs> S <eTime>

is the elapsed time in seconds since returning to the application
<relSecs> from MPI_Init
<eTime> is the elapsed execution time for the current call

Table 5.3 Extended trace-output

"-f rate" will add some rate-related information. The rate is calculated by dividing the number
of bytes transferred by the elapsed time to execute the call. All parameters to -f can be
abbreviated and can occur in any mix.

Normally no error messages are provided concerning the options which have been selected. But
if -verbose is added as a command-line option to mpimon, errors will be printed.

Trace provides information about which MPI routines were called and possibly information about
parameters and timing.

Platform MPI 5.6.5 User’s Guide Page 54

5.1.1.1 Example: using the test application

mpimon -trace "-p all" ./kollektive-8 ./uf256-8.pgm -- r1 r2

This will print a trace of all MPI calls for this relatively simple run:

0: MPI_Init
0: MPI_Comm_rank Rank: 0
0: MPI_Comm_size Size: 2
0: MPI_Bcast root: 0 Id: 0
my_count = 32768
0: MPI_Scatter Id: 1
1: MPI_Init
1: MPI_Comm_rank Rank: 1
1: MPI_Comm_size Size: 2
1: MPI_Bcast root: 0 Id: 0
my_count = 32768
1: MPI_Scatter Id: 1
1: MPI_Reduce Sum root: 0 Id: 2
1: MPI_Bcast root: 0 Id: 3
0: MPI_Reduce Sum root: 0 Id: 2
0: MPI_Bcast root: 0 Id: 3
1: MPI_Gather Id: 4
1: MPI_Keyval_free
0: MPI_Gather Id: 4
0: MPI_Keyval_free

If more information is needed the arguments to SCAMPI_TRACE can be enhanced to request

more information. The option “-f arg;timing” requests a list of the arguments given to each MPI
call, including message size ( useful information when evaluating interconnect performance).

Example
mpimon -trace "-f arg;timing" ./kollektive-8 ./uf256-8.pgm -- n1 2
0: +-0.951585 s 951.6ms MPI_Init
0: +0.000104 s 3.2us MPI_Comm_rank Rank: 0
0: +0.000130 s 1.7us MPI_Comm_size Size: 2
0: +0.038491 s 66.3us MPI_Bcast root: 0 sz: 1 x 4 = 4 Id: 0
my_count = 32768
0: +0.038634 s 390.0us MPI_Scatter Id: 1
1: +-1.011783 s 1.0s MPI_Init
1: +0.000100 s 3.8us MPI_Comm_rank Rank: 1
1: +0.000129 s 1.7us MPI_Comm_size Size: 2
1: +0.000157 s 69.6us MPI_Bcast root: 0 sz: 1 x 4 = 4 Id: 0
my_count = 32768
1: +0.000300 s 118.7us MPI_Scatter Id: 1

Platform MPI 5.6.5 User’s Guide Page 55

Example
0: +0.039267 s 38.8ms MPI_Reduce Sum root: 0 sz: 1 x 4 = 4 Id: 2
0: +0.078089 s 18.8us MPI_Bcast root: 0 sz: 1 x 8 = 8 Id: 3
1: +0.000641 s 56.2us MPI_Reduce Sum root: 0 sz: 1 x 4 = 4 Id: 2
1: +0.000726 s 113.6us MPI_Bcast root: 0 sz: 1 x 8 = 8 Id: 3
1: +0.002547 s 99.0us MPI_Gather Id: 4
0: +0.079834 s 579.5us MPI_Gather Id: 4
1: +0.002758 s 26.4us MPI_Keyval_free
0: +0.113244 s 1.4us MPI_Keyval_free

There are a number of parameters for selecting only a subset, either by limiting the number of
calls and intervals as described above under ‘Timing’ , or selecting or excluding just some MPI
calls.

5.1.2 Features
The "-b" option is useful when trying to pinpoint which MPI-call has been started but not
completed (i.e. are deadlocked). The "-s/-S/-c/-C" -options also offer useful support for an
application that runs well for a longer period and then stop, or for examining some part of the
execution of the application.

From time to time it may be desirable or feasible to trace only one or a few of the processes.
Specifying the "-p" options offers the ability to pick the processes to be traced.

All MPI-calls are enabled for tracing by default. To view only a few calls, specify a "-t
<call-list>" option; to exclude some calls, add a "-x <call-list>" option. The "-t" will disable
all tracing and then enable those calls that match the <call-list>. The matching is done using
"regular-posix-expression"-syntax. "-x" will lead to the opposite; first enable all tracing and
then disable those call matching <call-list>.

Example
"-t MPI_[b,r,s]*send" : Trace only send-calls (MPI_Send, MPI_Bsend, MPI_Rsend,
MPI_Ssend)

Example
"-t MPI_Irecv" : Trace only immediate recv (MPI_Irecv)

Example
"-t isend;irecv;wait" :Trace only MPI_Isend, MPI_Irecv and MPI_Wait

Example
“-t i[a-z]*" : Trace only calls beginning with MPI_I

Platform MPI 5.6.5 User’s Guide Page 56

5.2 Timing
Timing will give you information about which MPI routines were called and how long the MPI calls
took. This information is printed at intervals set by the user with the “-s n” option, where n if the
number of seconds.

5.2.1 Using Platform MPI built-in timing

To use the built-in timing functionality in Platform MPI, the mpimon-option -timing
"<options>" must be set, specifying which options are to be applied.

The following options can be specified:

Option Description

<...-list> is a semicolon-separated list of Posix-regular-expressions.

legal values often include ‘none, a comma separated list a range

<selection> e.g. ‘m-n’, or ‘all’
-s <seconds> print for intervals of <seconds>seconds

-c <calls> print for intervals of <calls>MPI_calls

special mode for timing

<mode> = ”sync”: Synchronize with MPI_Barrier before starting
-m <mode> collective call
-p <selection> enable for process(es) 'n,m,o..' = (list) or 'n-m' = (range) or 'all'

-f <call-list> Print after MPI_calls in <call-list>: MPI_call = 'MPI_call' | 'call'

-v verbose

-h print this list of options

Table 5.4 mpimon timing options

Printing of timing-information can be either at a fixed time-interval, if "-s <seconds>" is

specified, or for a fixed number-of-calls-interval, if "-c <calls>" ia used. It is also possible to
obtain output after specific MPI-calls by using "-f <call-list>"; see above for details on how to
write <call-list>.

The output has two parts: a timing-part and a buffer-statistics-part.

The first part has the following layout:

All lines start with <rank>: where <rank>: is rank within MPI_COMM_WORLD. This part is
included to facilitate separation of output (grep).

Example
mpimon -timing ”-s 1” ./kollektive-8 ./uf256-8.pgm -- r1 r2
where '<seconds>' is the number of seconds per printout from Platform MPI
produces:

Platform MPI 5.6.5 User’s Guide Page 57

1: 13.26.10 -------------Delta---------------------------------------
-Total----------------------------
1: Init+0.002659 s #calls time tim/cal #calls time tim/cal
1: MPI_Bcast 2 169.0us 84.5us 2 169.0us 84.5us
1: MPI_Comm_rank 1 3.1us 3.1us 1 3.1us 3.1us
1: MPI_Comm_size 1 1.5us 1.5us 1 1.5us 1.5us
1: MPI_Gather 1 109.9us 109.9us 1 109.9us 109.9us
1: MPI_Init 1 1.0s 1.0s 1 1.0s 1.0s
1: MPI_Keyval_free 1 1.2us 1.2us 1 1.2us 1.2us
1: MPI_Reduce 1 51.5us 51.5us 1 51.5us 51.5us
1: MPI_Scatter 1 138.7us 138.7us 1 138.7us 138.7us
1: Sum 9 1.0s 112.8ms 9 1.0s 112.8ms
1: Overhead 0 0.0ns 9 27.2us 3.0us
1: ========================================================
0: 13.26.40 ------ -Delta-- -------- ------ -Total--- ---------
0: Init+0.111598 s #calls time tim/cal #calls time tim/cal
0: MPI_Bcast 2 79.6us 39.8us 2 79.6us 39.8us
0: MPI_Comm_rank 1 3.3us 3.3us 1 3.3us 3.3us
0: MPI_Comm_size 1 1.4us 1.4us 1 1.4us 1.4us
0: MPI_Gather 1 648.8us 648.8us 1 648.8us 648.8us
0: MPI_Init 1 965.9ms 965.9ms 1 965.9ms 965.9ms
0: MPI_Keyval_free 1 1.1us 1.1us 1 1.1us 1.1us
0: MPI_Reduce 1 37.6ms 37.6ms 1 37.6ms 37.6ms
0: MPI_Scatter 1 258.1us 258.1us 1 258.1us 258.1us
0: Sum 9 1.0s 111.6ms 9 1.0s 111.6ms
0: Overhead 0 0.0ns 9 35.6us 4.0us
0: ========================================================

The <seconds> field can be set to a large number in order to collect only final statistics.

We see that the output gives statistics about which MPI calls are used and their frequency and
timing. Both delta numbers since last printout and the total accumulated statistics. By setting
the interval timing (in -s <seconds>) to a large number, only the cumulative statistics at the end
are printed. The timings are presented for each process, and with many processes this can yield
a huge amount of output. There are many options for modifying SCAMPI_TIMING to reduce this
output. The selection parameter can time only those MPI processes to be monitored. There are
also other ways to minimize the output, by screening away selected MPI calls either before or
after a certain number of calls or between an interval of calls. Some examples are:

Platform MPI 5.6.5 User’s Guide Page 58

The rest of the format has the following fields:

<MPIcall><Dcalls><Dtime><Dfreq> <Tcalls><Ttime><Tfreq>

where

• <MPIcall>is the name of the MPI-call

• <Dcalls> is the number of calls to <MPIcall> since the last printout
• <Dtime> is the sum of the execution-time for calls to <MPIcall> since the last printout
• <Dfreq> is the average time-per-call for calls to <MPIcall> since the last printout
• <Tcalls> is the number of calls to <MPIcall>
• <Ttime> is the sum of the execution-time for calls to <MPIcall>
• <Tfreq> is the average time-per-call for calls to <MPIcall>
After all detail-lines (one per MPI-call which has been called since last printout), there will be a
line with the sum of all calls followed by a line giving the overhead introduced when obtaining
the timing measurements.

The second part containing the buffer-statistics has two types of lines, one for receives and one
for sends.

"Receive lines" has the following fields:

<Comm><rank> recv from <from>(<worldFrom>):<commonFields>

where:

• <Comm> is the communicator being used

• <rank> is the rank within <Comm>
• <from> is the rank within <Comm>
• <worldFrom> is the rank within MPI_COMM_WORLD
"Send-lines" has the following fields:

<Comm><rank> send to <to>(<worldTo>):<commonFields>

where :

• <Comm> is the communicator being used

• <rank> is the rank within <Comm>
• <to> is the rank within <Comm>
• <worldTo> is the rank within MPI_COMM_WORLD
The <commonFields> are as follows:

!<count>!<avrLen>!<zroLen>!<inline>!<eager>!<transporter>!

where:

• <count> is the number of sends/receives

Platform MPI 5.6.5 User’s Guide Page 59

 •
<avrLen>is the average length of messages in bytes
•
<zroLen> is the number of messages sent/received using the zero-bytes mechanism
•
<inline> is the number of messages sent/received using the inline mechanism
•
<eager> is the number of messages sent/received using the eagerbuffer mechanism
•
<transporter> is the number of messages sent/received using the transporter
mechanism
More details on the different mechanisms can be found in “Platform MPI components” on
page 40.

5.3 Using scanalyze

Tracing and timing the image processing example above produced little data, and interpreting
the data posed little problem. However, most applications run for longer time, with
correspondingly larger logs as a result; output from tracing and timing with Platform MPI easily
amounts to megabytes of data.

In order to extract information from the huge amount of data, Platform has developed a simple
analysis tool called scanalyze. This analysis tool accept output from Platform MPI applications
run with certain predefined trace and timing variables set.

5.3.1 Analyzing all2all

The all2all program in ${MPI_HOME}/examples/bin is a simple communication benchmark, but
tracing and timing it produces massive log files. For example running

mpimon -trace ”-f arg;timing” ./all2all -- r1 r2

on a particular system produced a 2354159 byte log file, while running

mpimon -timing ”-s 10” ./all2all -- r1 r2

produced a 158642 byte file.

Digesting the massive information in these files is a challenge, but scanalyze produces the
following summaries for tracing:

Count Total < 128 < 1k < 8k < 256k < 1M

MPI_Alltoall 24795 5127 3078 4104 10260 2226 0
MPI_Barrier 52 0 0 0 0 0 0
MPI_Comm_rank 2 0 0 0 0 0 0
MPI_Comm_size 2 0 0 0 0 0 0
MPI_Init 2 0 0 0 0 0 0
MPI_Keyval_free 2 0 0 0 0 0 0
Count Total < 128 < 1k < 8k < 256k < 1M
MPI_Alltoall 21.20 0.21 0.15 0.41 9.35 11.08 0.00
MPI_Barrier 0.01 0.00 0.00 0.00 0.00 0.00 0.00

Platform MPI 5.6.5 User’s Guide Page 60

MPI_Comm_rank 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Comm_size 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Init 2.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Keyval_free 0.00 0.00 0.00 0.00 0.00 0.00 0.00

and for timing:

==========================================================
#call time tim/cal #call time tim/cal
s s
0: MPI_Alltoall 0 0.0ns 12399 10.6s 855.1us
0: MPI_Barrier 0 0.0ns 26 1.2ms 45.8us
0: MPI_Comm_rank 0 0.0ns 1 3.2us 3.2us
0: MPI_Comm_size 0 0.0ns 1 1.4us 1.4us
0: MPI_Init 0 0.0ns 1 1.0s 1.0s
0: MPI_Keyval_free 1 27.9us 27.9us 1 27.9us 27.9us
0: SUM 2 29.0us 14.5us 12481 11.7s 933.5us
0: Overhead 0 0.0ns 12481 12.6ms 1.0us
==========================================================
#call time tim/cal #call time tim/cal
s s
1: MPI_Alltoall 0 0.0ns 12399 10.6s 854.9us
1: MPI_Barrier 0 0.0ns 26 2.9ms 109.6us
1: MPI_Comm_rank 0 0.0ns 1 3.5us 3.5us
1: MPI_Comm_size 0 0.0ns 1 1.5us 1.5us
1: MPI_Init 0 0.0ns 1 1.0s 1.0s
1: MPI_Keyval_free 1 10.8us 10.8us 1 10.8us 10.8us
1: SUM 2 29.0us 14.5us 12481 11.7s 933.5us
1: Overhead 0 0.0ns 12479 12.7ms 1.0us

5.3.2 Using Platform MPI's built-in CPU-usage functionality

Platform MPI has the capability to report wall clock time, and user and system CPU time on all
processes with a built-in CPU timing facility. To use Platform MPI's built-in CPU-usage-timing it
is necessary first to set the environment variable SCAMPI_CPU_USAGE.

The information displayed is collected with the system-call "times"; see man-pages for more
information.

Platform MPI 5.6.5 User’s Guide Page 61

The output has two different blocks. The first block contains CPU-usage by the sub monitors on
the different nodes. One line is printed for each sub monitor followed by a sum-line and an
average-line. The second block consists of one line per process followed by a sum-line and an
average-line.

For example, to get the CPU usage when running the image enhancement program in Unix enter
this:

user% SCAMPI_CPU_USAGE=1 mpirun -np 4 ./kollektive-8 ./uf256-8.pgm

When using Windows enter:

cmd> set SCAMPI_CPU_USAGE=1

cmd> mpirun -np 4 ./kollektive-8 ./uf256-8.pgm

These produce the following report:

----------- Own ------------- ---Own + Children ---

Elapsed User System Sum User System Sum
2.970 0.000 0.000 0.000 0.090 0.030 0.120
3.250 0.000 0.000 0.000 0.060 0.040 0.100
Submonitor-3@r7 3.180 0.000 0.000 0.000 0.050 0.030 0.080
Submonitor-4@r6 3.190 0.010 0.000 0.010 0.090 0.020 0.110
Total for submonitors 12.590 0.010 0.000 0.010 0.290 0.120 0.410
Average per submonitor 3.147 0.003 0.000 0.003 0.073 0.030 0.103

---------- Own
-------------------------------
--
Process timing stat. in Elapsed User System Sum
secs

kollektive-8-0@r9 0.080 0.070 0.030 0.100

kollektive-8-1@r8 0.050 0.020 0.040 0.060
kollektive-8-2@r7 0.050 0.020 0.030 0.050
kollektive-8-3@r6 0.010 0.020 0.020 0.040
Sum for processes 0.190 0.130 0.120 0.250
Average per process 0.048 0.033 0.030 0.062

Elapsed is walltime used by user-process/submonitor.

User is cpu-time used in user-process/submonitor.

Platform MPI 5.6.5 User’s Guide Page 62

System is cpu-time used in system-calls.

Sum is total cpu-time used by user-process/submonitor.

Platform MPI 5.6.5 User’s Guide Page 63

Chapter 6 - Tuning Platform MPI to your
application
Platform MPI allows the user to exercise control over the communication mechanisms through
adjustment of the thresholds that steer which mechanism to use for a particular message. This is
one technique that can be used to improve performance of parallel applications on a cluster.

Forcing size parameters to mpimon is usually not necessary. This is only a means of optimising
Platform MPI to a particular application, based on knowledge of communication patterns. For
unsafe MPI programs it may be necessary to adjust buffering to allow the program to complete.

6.1 Tuning communication resources

The communication resources allocated by Platform MPI are shared among the MPI processes
in the node.

6.1.1 Communication buffer adaption

If the communication behaviour of the application is known, explicitly providing buffersize
settings to mpimon, to match the requirement of the application, will in most cases improve
performance.

6.1.1.1 Example: Application is sending only 900 bytes messages.

1 Set channel_inline_threshold 964 (64 added for alignment) and increase the
channel-size significantly (32-128 k).
2 Set eager_size 1k and eager_count high (16 or more). If all messages can be
buffered, the transporter-size/-count can be set to low values to reduce shared
memory consumption.
6.1.1.2 Example: Controlling shared memory usage
Solution: Adjust Platform MPI buffer sizes.
The buffer space required by a communication channel is approximately:
chunk-size = roundup(channel-entry-size * channel-entry-count)
+ roundup(transporter-size * transporter-count)
+ roundup(eager-size * eager-count)
+ roundup(rb_count * 256)
+ extrapage
(where roundup is a funtion that round the the demand up to a multiplum of pages)

Total-usage = chunk-size * no-of-processes

6.1.2 Automatic buffer management
The pool-size is a limit for the total amount of shared memory. The automatic buffer size
computations is based on full connectivity, i.e. all communicating with all others. Given a total
pool of memory dedicated to communication, each communication channel will be restricted to
use a partition of only (P = number of processes):

chunk = inter_pool_size / P

Platform MPI 5.6.5 User’s Guide Page 64

The automatic approach is to downsize all buffers associated with a communication channel until
it fits in its part of the pool. The automatic chunk size is calculated to wrap a complete
communication channel.

6.1.3 How to optimize MPI performance

There is no universal recipe for getting good performance out of a message passing program.
Here are some do’s and don’t’s for Platform MPI.

6.1.3.1 Performance analysis

Learn about the performance behaviour of your particular MPI applications on a Platform System
by using a performance analysis tool.

6.1.3.2 Using processor-power to poll

To maximize performance, Platform MPI is using poll when waiting for communication to
terminate, instead of using interrupts. Polling means that the CPU is performing busy-wait
(looping) when waiting for data over the interconnect. All exotic interconnects require polling.

Some applications create threads which may end up having more active threads than you have
CPUs. This will have huge impact on MPI performance. In threaded application with irregular
communication patterns you probably have other threads that could make use of the processor.
To increase performance in this case, Platform has provided a “backoff” feature in Platform
MPI. The backoff feature will still poll when waiting for data, but will start to enter sleep states
on intervals when no data is coming. The algorithm is as follows: Platform MPI polls for a short
time (idle time), then stops for a periode, and polls again.

The sleep periode starts a parameter controlled minimum and is doubled every time until it
reaches the maximum value. The following environment variables set the parameters:

SCAMPI_BACKOFF_ENABLE turns the mechanism on. Legal values are:

'n,m,o..' = (list) or
'n-m' = (range) or
'all'.
SCAMPI_BACKOFF_IDLE=n defines idle-period as n ms [Default = 20 ms]
SCAMPI_BACKOFF_MIN=n defines minimum backoff-time in ms [Default = 10 ms]
SCAMPI_BACKOFF_MAX=n defines maximum backoff-time in ms [Default = 100 ms]

Examples of how to set parameter SCAMPI_BACKOFF_ENABLE

# set all ranks

SCAMPI_BACKOFF_ENABLE=all

# set ranks 0 through 4

SCAMPI_BACKOFF_ENABLE=0-4

# set ranks 0, 1 and 8

SCAMPI_BACKOFF_ENABLE=0,1,8

Platform MPI 5.6.5 User’s Guide Page 65

6.1.3.3 Reorder network traffic to avoid conflicts

Many-to-one communication may introduce bottlenecks. Zero-byte messages are low-cost. In a

many-to-one communication, performance may improve if the receiver sends ready-to-receive
tokens (in the shape of a zero-byte message) to the MPI-process wanting to send data.

6.1.3.4 Multicore Awareness

Platform MPI allows you to assign which cpu will run a particular process through the use of bit
masks. This is known as "hard CPU affinity”. Hard affinity cuts down on overhead by optimizing
cache performance.

mpimon -affinity_mode manual:<mask0>:...:<maskN> | none |

automatic[:<policy>[:<granularity>]]

6.1.3.5 Granularity

You have a choice of three settings for the bitmasks in Platform MPI: execunit, core and
socket. The smallest processing unit in a node is called an “execution unit”. Using the
granularity setting of execunit allows the process to run on a specified single execunit. Using the
granularity setting of core allows the process to run on all execunits in a single core. Using the
granularity setting of socket allows the process to run on all execution units in a single socket.

6.1.3.6 Policy

A policy specifies how processes are placed in relation to each other on one node. Platform MPI
has four policies for binding processes on a node automatically and one for binding manually:

leastload - Processes can run on the least loaded socket, core or hyperthread.
latency - The processes run on as many cores as possible within one socket. If the number of
processes is higher than the number of cores, the next socket is used as well.
bandwidth - Only one process is run on a socket, as long as there are enough sockets. If the
number of processes excedes the number of sockets, then one socket will run the addditional
"excess process".
none - No limitations are set on the CPU affinity. The processes can run on all execunits in all cores
in all sockets.
manual - You can specify bind values for processes from rank 0 to rank n-1, using a
colon-separated list of hexadecimal values. If the number of the processes is greater than the
number of hexadecimal values, the list is begun again so that more than one process share a
hexidecimal value.
6.1.3.7 Benchmarking

Benchmarking is that part of performance evaluation that deals with the measurement and
analysis of computer performance using various kinds of test programs. Benchmark figures
should always be handled with special care when making comparisons with similar results.

6.1.4.8 How to get expected performance

• Caching the application program on the nodes.

Platform MPI 5.6.5 User’s Guide Page 66

 • For benchmarks with short execution time, total execution time may be reduced when
running the process repetitively. For large configurations, copying the
application to the local file system on each node will reduce startup latency
and improve disk I/O bandwidth.
• The first iteration is (very) slow.
• This may happen because the MPI-processes in an application are not started
simultaneously. Inserting an MPI_Barrier() before the timing loop will
eliminate this.

6.1.5.9 Memory consumption increase after warm-up

Remember that group operations (MPI_Comm_{create, dup, split}) may involve creating
new communication buffers. If this is a problem, decreasing chunk_size may help.

6.1.6.10 Collective operations

A collective communication is a communication operation in which a group of processes works

together to distribute or gather together a set of one or more values. Platform MPI uses a
number of different approaches to implement collective operations. Through environment
variables the user can control which algorithm the application uses.

Consider the Integer Sort (IS) benchmark in NPB (NAS Parallel Benchmarks). When running on
ten processes on 5 nodes over Gigabit Ethernet (mpimon -net smp,tcp bin/is.A.16.scampi
-- r1 2 r2 2 r3 2 r4 2 r5 2) the resulting performance is:

• Mop/s total = 34.05

• Mop/s/process = 2.13

Extracting the MPI profile of the run can be done as follows:

user% mpimon -trace "-f arg;timing" bin/is.A.16.scampi -- $ALL2 > trace.out

And running the output through scanalyze yields the following:

MPI Call <128 128-1k 1-8k 8-32k 32-256k 256k-1M >1M

MPI_Send 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Irecv 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Wait 0.69 0.14 0.00 0.00 0.00 0.00 0.00
MPI_Alltoall 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Alltoallv 11.20 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Reduce 1.04 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Allreduce 0.00 0.00 15.63 0.00 0.00 0.00 0.00
MPI_Comm_size 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Comm_rank 0.00 0.00 0.00 0.00 0.00 0.00 0.00
MPI_Keyval_free 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Platform MPI 5.6.5 User’s Guide Page 67

The MPI_Alltoallv uses a high fraction of the total execution time. The communication time is the
sum of all used algorithms and the total timing may depend on more than one type of
communication. If one type or a few operations dominate the time consumption, they are
promising candidates for tuning/optimization.

Attention
Note: Please note that the run time selectable algorithms and their values may vary on
different Platform MPI release versions.

For information on which algorithms that are selectable at run time and their valid values, set
the environment variable SCAMPI_ALGORITHM and run an example application:

For Windows:

cmd> set SCAMPI_ALGORITHM=1

cmd> mpimon %MPI_HOME%\examples\bin\hello.exe -- localhost

For Unix:

# SCAMPI_ALGORITHM=1 mpimon ${MPI_HOME}/examples/bin/hello -- localhost

This will produce a listing of the different implementations of particular collective MPI calls.

For each collective operation a listing consisting of a number and a short description of the
algoritmn is produced, e.g., for MPI_Alltoallv() the following:

SCAMPI_ALLTOALLV_ALGORITHM alternatives
0 pair0
1 pair1
2 pair2
3 pair3
4 pair4
5 pipe0
6 pipe1
7 safe
def 8 smp

Platform MPI 5.6.5 User’s Guide Page 68

 By looping through these alternatives the performance of IS varies:

algorithm 0: Mop/s total = 95.60

algorithm 1: Mop/s total = 78.37
algorithm 2: Mop/s total = 34.44
algorithm 3: Mop/s total = 61.77
algorithm 4: Mop/s total = 41.00
algorithm 5: Mop/s total = 49.14
algorithm 6: Mop/s total = 85.17
algorithm 7: Mop/s total = 60.22
algorithm 8: Mop/s total = 48.61

algorithm 1: Mop/s total = 78.37

algorithm 2: Mop/s total = 34.44
algorithm 3: Mop/s total = 61.77
algorithm 4: Mop/s total = 41.00
algorithm 5: Mop/s total = 49.14
algorithm 6: Mop/s total = 85.17
algorithm 7: Mop/s total = 60.22
algorithm 8: Mop/s total = 48.61

For this particular combination of Alltoallv-algorithm and application (IS) the performance varies
significantly, with algorithm 0 close to doubling the performance over the default.

6.1.7.11 Finding the best algorithm

Consider the image processing example from “Profiling with Platform MPI” on page 51 which
contains four collective operations. All of these can be tuned with respect to algorithm according
to the following pseudo-language pattern:

user% for a in <range>; do

set SCAMPI_<MPI-function>_ALGORITHM=$a
mpimon <application> -- <nodes> ><application>.out.$a; \
done

For example, trying out the alternative algorithms for MPI_Reduce with two processes can be
done as follows (assuming Bourne Again Shell [bash]:

user% for a in 0 1 2 3 4 5 6 7 8; do
export SCAMPI_REDUCE_ALGORITHM=$a
mpimon ./kollektive-8 ./uf256-8.pgm -- r1 r2;
done

Given that the application then reports the timing of the relevant parts of the code a best choice
can be made. Note however that with multiple collective operations working in the same
program there may be interference between the algorithms. Also, the performance of the
implementations is interconnect dependent.

Platform MPI 5.6.5 User’s Guide Page 69

Appendix A - Licensing Management
Platform software products are now licensed using product keys. This chapter explains the
Platform Licensing system. The following topics are included:

• “Product key overview” on page 77

• “Product Activation” on page 78
• “Managing product keys” on page 78

A.1 Product key overview

There is a one-to-one relationship between the product key and software product, so you will
have an unique product key for each of your Platform software products.

There are two types of product keys: demo keys and permanent product keys. Demo keys have
an expiry date and cannot be activated. Both types of keys have the following format:

XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX
You will be requested to enter your product key. The product keys can be found on the activation
form that came with your Platform software.

The product keys allow Platform software to work right out of the box and immediately give you
full enjoyment of your Platform software. For the software to remain operational it must be
activated within 30 days of intallation. Activation is simple and described in further detail in
the next section.

A.2 Licensing Tool - lmutil

The lmutil is a product license management utility, accessed through the console.

Syntax: lmutil <command>[<parameter>]<

Command Parameter Description

add <productkey> Add productkey to list of product-keys
remove <productkey> Remove productkey from list of product-keys
list none Show list of product-keys
auto <productkey> Automatically activate productkey by requesting
<company> activation-key from server
<contactemail>
get <productkey> Get the activationkey for this productkey.
<company>
<contactemail>
manual <productkey> Manually activate productkey
<activationkey>
show [lmode] Show status for products
Table A.1 lmutil commands and options

Platform MPI 5.6.5 User’s Guide Page 77

A.2.1 Product Activation
Product activation may be either automatic, or manual.

A.2.1.1 Automatic activation

This requires that you have internet access to the Platform activation server from the computer
running the activation process. This will usually be the License Server.

Please run the following command:

lmutil auto <productkey> <company> <contactemail>

A.2.1.2 Manual activation

If your computer running the activation process does not have internet access, you will have to
activate your copy manually, using the information found with your software copy. You will find
this information in the Platform Product Activation Form that came with your Platform installation
CD.

To use the Platform product activation web service at

http://www.platform.com/Products/platform-manager/product-activation

you will need one or another computer with internet access.

1 Use lmhostid utility to obtain the host ID of the license server:

lmhostid

2 Go to the Platform product activation web server, and enter your product details,
including the host ID. You will be given the activation key for your product.
3 Add the activation code for the product with cli as follows:

lmutil manual <productkey> <activationkey>

The product will change state to "activated" immediately.

A.2.2 Managing product keys

Product keys may be managed using the lmutil tool.

A.2.2.1 Listing product keys using the lmutil tool

Use the lmutil show command to list the installed product keys:

lmutil show
Product Version Used Free Expire date Status
-------------------------------------------------------------------
Platform MPI 5.2 0 10 18-jun-2006 Unknown
manage 5.000 1 9 18-jun-2006 Unknown....

Platform MPI 5.6.5 User’s Guide Page 78

A.2.2.2 Adding product keys
Use the lmutil to add a product key like this:

lmutil add XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

A.2.2.3 Removing product keys

Use the lmutil to remove an installed product key:

lmutil remove XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

Platform MPI 5.6.5 User’s Guide Page 79

Appendix B - Install Platform MPI in Unix
Platform MPI (Platform MPI) is made available by installing it in a 2-phase installation. The
first phase is to install the license-server on the node selected to be the license-server.
Afterwards one have to install Platform MPI on each particular node in the cluster. In the
following the steps needed to install Platform MPI are detailed.

There are some of the cluster-administation-tools that have support to install Platform MPI
automatically; check with the supplier.

B.0.1 Installation of license-server

When running lminstall you should give arguments to specify license-key(s).

The -h option gives you details on the installation command and shows you which options you
need to specify in order to install the software components you want :

Usage: lminstall -p | -u [-aixVh?][-FfU]

Option Description

-p <productkey> Specify productkey to register with license server.

-u <licensefile> Specify license file to use with license server.

-a Automatically accept license terms

-i Ignore SSP check.

-x Ignore errors.

-h/-? Show this help message

-V Print version.

-F Use alternative versioned destination directory *)

-f <prefix> Specify destination directory to install packages *)

-U Use unpack method instead of rpm install. *)

*) are only available on Linux

Table B.1 lminstall license options

B.0.2 Per node installation of Platform MPI

Platform MPI must be installed on every node in the cluster. When running smcinstall you
should give arguments to specify your interconnects.

Platform MPI 5.6.5 User’s Guide Page 80

The -h option gives you details on the installation command and shows you which options you
need to specify in order to install the software components you want :.

Usage: smcinstall -n [-tmb] [-aixVh?] [-FfU]

Option Description

-t Install Platform MPI with TCP/IP as default network.

-m Install Platform MPI with Myrinet GM as default network.

-b Install Platform MPI with Infiniband as default network.

-n <hostname> Specify hostname of Platform license server.

-a Automatically accept license terms

-i Ignore SSP check.

-x Ignore errors.

-h/-? Show this help message

-V Print version.

-F Use alternate versioned destination directory *)

-f <prefix> Specify destination directory to install packages *)

-U Use unpack method instead of rpm install. *)

*) are only available on Linux

Table B.2 smcinstall options

Note: You must have root privileges to install Platform MPI

Note: One or more of the product selection options (-t, -m, -o, or -b) must be specified
to install a working MPI environment.
Note: Installing Platform MPI will not set up the necessary environment variables. You
can use the environment script (scampivars.sh / scampivars.csh) to setup your
environment.

B.1 Setting default networks

B.1.1 Install Platform MPI for TCP/IP

To install Platform MPI for TCP/IP, please specify the -t option to smcinstall. No further
configuration is needed.

Platform MPI 5.6.5 User’s Guide Page 81

B.1.2 Install Platform MPI for Myrinet or Infiniband
To install Platform MPI for Myrinet or Infiniband, please specify the -m or -b option to
smcinstall.

B.1.2.1 Example

root# ./smcinstall -m

When -m or -b option is selected Platform MPI will default to Myrinet or Infiniband respectively
as the default transport devices. If this is not desired, modify the networks line in the global
${MPI_HOME}/etc/ScaMPI.conf configuration file. See the section “Network devices” on
page 42 for more information regarding network device selection.

B.2 License server options

Platform MPI is licensed software. You need a valid product key from Platform to be able to run
an MPI application. See “Product key overview” on page 77. for more information.

B.3 Uninstalling Platform MPI

To remove Platform MPI, use the script :

root# ${MPI_HOME}/sbin/smcunistall

B.4 Troubleshooting Network providers

B.4.1 Troubleshooting the IB provider

The IB provider provides a network device for each Infiniband port available on the node, named
ib0, ib1, etc. If you have a dual port HCA, ib0 is Port1 and ib1 is Port2.

In order for Platform MPI to locate the Infiniband libraries, it is important that /etc/
ld.so.conf is correctly set up with paths to the library location. Please verify with your
Infiniband vendor on how to troubleshoot Infiniband network issues.

Platform MPI comes with a predefined list of InfiniBand device names. If you are using a OFED
based InfiniBand stack you may override the default device name and other parameters by
creating an iba_params.conf file in one of the following locations:

• ${MPI_HOME}/etc/iba_params.conf
• ${HOME}/iba_params.conf
• ./iba_params.conf

The format of the file is one parameter=value pair per line. Recognized parameters are:

• "hcadevice"
• "mtu"

Platform MPI 5.6.5 User’s Guide Page 82

 • "static_rate"
• "time_out"
• "retry_count"
• "service_level"
• "pkey_ix"
• "max_pending"
• "max_inline"

Please refer to the OFED stack documentation for descriptions on each of the parameters. An
example configuration file would could look like this:

$ cat ${MPI_HOME}/etc/iba_params.conf
hcadevice=ibhca0

B.4.2 Troubleshooting the GM provider

The GM provider provides a network device for each Myrinet card installed on the node, named
gm0, gm1... etc.

To verify that the gm0 device is operational, run an MPI test job on two, or more, nodes in
question:

user% mpimon -networks gm0 ${MPI_HOME}/examples/bin/bandwidth -- node1 node2

If the gm0 devices fails, the MPI job should fail with a "[ 1] No valid network connection from 1
to 0" message.

First of all, keep in mind that the GM source must be obtained from Myricom, and compiled on
your nodes.

Verify that the "gm" kernel module is loaded by running lsmod(8) on the compute node in
question.

To verify that GM is operational run:

root# /opt/gm/bin/gm_board_info

You should see all the nodes on your GM network listed. (This command must be run on a node
with a Myrinet card installed!)

A simple cause of failure is that /opt/gm/lib is not in /etc/ld.so.conf and/or ldconfig is not run,
you will get a unable to find libgm.so error message is this is the case.

Platform MPI 5.6.5 User’s Guide Page 83

Appendix C - Installing Platform MPI In Win-
dows
Platform MPI is distributed as an MSI executable. Platform MPI must be installed on all nodes
in your cluster(s).

Note: You must have Administrator privileges to install Platform MPI

C.1 Installing Platform MPI

1 To start the installation wizard just click on the file or use the msiexec installation utility,
specifying the name of the installer package file. The file has the following
nomenclature: PlatformMPIConnect-<x.y.z>.msi
2 Click on next to continue to the welcome screen.Click on next to continue to the readme
screen.

After reading the text, click on next to continue to the License agreement screen.Platform MPI
is a licensed product. On each cluster you must designate a license server. This server validates
licenses through the product key. You must obtain a valid product key to continue successfully.
This product key remains valid for a maximum of 30 days. In that time you must activate the
license. Please see the chapter on Licensing for details.

There are two types of product keys: demo keys and permanent product keys. Demo keys have
an expiry date and cannot be activated. Both types of keys have the following format. There are
eight groups of four alpha-numeric values thus:

XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

Note: Note: Only enter the product key and activate it on the license server.

3 Do NOT activate the product key on the compute nodes. If you install the product key
on one of the compute nodes first, you will activate the license for a license
server on it. Activation is irreversible. You can install/activate the product
key on ONLY ONE server. Install the product key ONLY on the designated
license server.On all compute nodes enter the License server node name and
IP address.Please accept the settings as is and click on Next.Please confirm
the installation now by clicking Next. If a window with rolling text appears:
DON’T PANIC. This is merely a set-up tool which is configuring your
installation.When the "Installation Complete" screen comes up, please click on
Close to close the wizard.

Platform MPI 5.6.5 User’s Guide Page 84

C.2 Uninstalling Platform MPI
To remove Platform MPI, use the software manager in the Windows Control Panel, or
double-click on the MSI executable or use the msiexec installation utility, specifying the name
of the installer package file. You will be able to re-pair or uninstall the previous installation of
Platform MPI.

Platform MPI 5.6.5 User’s Guide Page 85

Appendix D - Bracket expansion and grouping
To ease usage of Platform software on large cluster configuration, many of the command line
utilities have bracket expansion and grouping functionality.

D.1 Bracket expansion

The following syntax applies:

<bracket>== "["<number_or_range>[,<number_or_range>]*"]"
<number_or_range>== <number> | <from>-<to>[:<stride>]
<number>== <digit>+
<from>== <digit>+
<to> == <digit>+
<stride>== <digit>+
<digit>== 0|1|2|3|4|5|6|7|8|9
This is typically used to expand nodenames from a range, using from 1 to multi-dimensional
numbering, or an explicit list.

If <to> or <from> contains leading zeros, then the expansion will contain leading zeros such
that the width is constant and equal to the larger of the widths of <to> and <from>.

The syntax does not allow for negative numbers. <from> does not have to be less that <to>.

Examples:

n[0-2]
is equivalent to

n0 n1 n2
n[00-10:3]

is equivalent to

n00 n03 n06 n09

D.2 Grouping
Utilities that use scagroup will accept a group alias wherever a host name of hostlist i expected.
The group alias will be resolved to a list of hostnames as specified in the file scagroup config
file. If there exists a file ${HOME}/scagroup.conf in the users home directory, this will be used.
Otherwise, the system default file
${MPI_HOME}/etc/scagroup.conf
will be used.

D.3 File format

Each group has the keyword group at the beginning of a line followed by a group alias and a list
of hostnames included in the group. The list may itself contain previously defined group aliases
which will be recursivly resolved. The host list may use bracket expressions which will be
resolved as specified above.

Platform MPI 5.6.5 User’s Guide Page 86

Lines starting with “#” contain comments.

Examples:

group master n00

group slaves n[01-32]
group all master slaves

Platform MPI 5.6.5 User’s Guide Page 87

Appendix E - The ScaMPItst package
The ScaMPItst package is installed together with installation of Platform MPI. The package
contains a number of programs in ${MPI_HOME}/examples with executable code in bin/ and
source code in src/. A description of the programs can be found in the README file, located in
the ${MPI_HOME}/doc/ScaMPItst directory. These programs can be used to experiment with the
features of Platform MPI.

Platform MPI 5.6.5 User’s Guide Page 88

E.1 Image contrast enhancement
Adopted from "MPI Tutorial", by Puri Bangalore, Anthony Skjellum and Shane Herbert, High
Performance Computing Lab, Dept. of Computer Science and NSF Engineering Research Center,
Mississippi State University, Feb 2000.

#include <mpi.h>
#include <stdio.h>
#include <math.h>
int main( int argc, char** argv )
{
int width, height, rank, size, sum, my_sum;
int numpixels, my_count, i, val;
unsigned char pixels[65536], recvbuf[65536];
unsigned int buffer;
double rms;
FILE *infile;
FILE *outfile;
char line[80];

MPI_Init( &argc, &argv );

MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Comm_size( MPI_COMM_WORLD, &size );
if ( rank == 0 )
{
/* assume valid file name in argv[1] */
infile = fopen( argv[1], "r" );
if ( ! infile )
{
printf("%s :: can't open file\n", argv[1]);
MPI_Finalize();
exit(-1);
}
/* valid file available */
fscanf( infile, "%s", line );
fscanf( infile, "%d", &height );
fscanf( infile, "%d", &width );
fscanf( infile, "%u", &buffer );
numpixels = width * height;
/* read the image */
for ( i = 0; i < numpixels; i ++ )
{
fscanf( infile, "%u", &buffer );
pixels[i] = (unsigned char)buffer;
}
fclose( infile );
/* calculate number of pixels for each node */
my_count = numpixels / size;
}
/* broadcast to all nodes */
MPI_Bcast( &my_count, 1, MPI_INT, 0, MPI_COMM_WORLD );
/* scatter the image */

Platform MPI 5.6.5 User’s Guide Page 89

 MPI_Scatter( pixels, my_count, MPI_UNSIGNED_CHAR, recvbuf,
my_count, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD );
/* sum the squares of the pixels in the sub-image */
my_sum = 0;
for ( i = 0; i < my_count; i ++ )
my_sum += recvbuf[i] * recvbuf[i];
/* find the global sum of the squares */
MPI_Reduce( &my_sum, &sum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD );
/* let rank 0 compute the root mean square */
if ( rank == 0 )
{
rms = sqrt( (double)sum / (double)numpixels );
}
/* rank 0 broadcasts the RMS to the other nodes */
MPI_Bcast( &rms, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD );
/* perform filtering operation (contrast enhancement) */
for ( i = 0; i < my_count; i ++ )
{
val = 2 * recvbuf[i] - rms;
if ( val < 0 ) recvbuf[i] = 0;
else if ( val > 255 ) recvbuf[i] = 255;
else recvbuf[i] = val;
}
/* gather back to rank 0 */
MPI_Gather( recvbuf, my_count, MPI_UNSIGNED_CHAR, pixels,
my_count, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD );
/* dump the image from rank 0 */
if ( rank == 0 )
{
outfile = fopen( "try.pgm", "w" );
if ( ! outfile )
{
printf("unable to open try.ppm\n");
}
else
{
fprintf( outfile, "%s\n", (unsigned char*)line );
fprintf( outfile, "%d %d\n", height, width );
fprintf( outfile, "255\n");
/* numpixels = height * width; */
for ( i = 0; i < numpixels; i ++ )
{
fprintf( outfile, "%d\n", (int)pixels[i] );
}
fflush( outfile );
fclose ( outfile );
}
}
MPI_Finalize();
return 0;
}

Platform MPI 5.6.5 User’s Guide Page 90

E.2 File format
The code contains the logic to read and write images in .pgm format. This “Portable Gray Map”
format uses ASCII characters for encoding pixel intensities, as illustrated by the example below:
P2
8 8
255
160 160 160 137 137 160 170 160
160 160 160 137 160 160 160 108
160 160 160 137 160 137 160 160
160 160 137 160 150 137 160 106
160 160 137 160 140 137 160 160
160 137 160 160 120 137 137 137
160 137 160 90 160 137 160 160
160 160 160 160 160 160 160 130

original

P2
8 8
255
168 168 168 122 122 168 188 168
168 168 168 122 168 168 168 64
168 168 168 122 168 122 168 168
168 168 122 168 148 122 168 60
168 168 122 168 128 122 168 168
168 122 168 168 88 122 122 122
168 122 168 28 168 122 168 168
168 168 168 168 168 168 168 108
enhanced contrast

Figure E.1 Gray map

Platform MPI 5.6.5 User’s Guide Page 91

Appendix F - Multi-rail Support in Platform MPI
This document describes the multi-rail Open Fabrics interconnect support in Platform MPI.
Multirailing is the ability to stripe data across the multiple Infiniband HCA’s in a system as well as
to balance HCA loads between processes on the same system.

F.1 Software Requirements

The multi-rail support in Platform MPI depends on OpenFabrics 1.1 (or higher) software being
available on the system. This includes both the native verbs libraries and drivers, as well as the
uDAPL static registry (libdat.so). Multi-rail support is available from Platform MPI 5.6.0.

F.2 Hardware Requirements

Platforms that are certified with OpenFabrics and Platform MPI (currently i386 and x86_64
architecture) support the multi-rail functionality. The actual multi-rail functionality requires more
than one HCA in the system, however the software runs on systems with only a single HCA. Data
striping and load-balancing will in these cases be disabled.

F.3 Implementation Overview

The multi-rail capable provider is implemented using an uDAPL 1.2 compatible provider library.
It is this library that will manage the multiple HCA devices and hence, the MPI library itself
remains unchanged. Device names used for the multi-rail capability are controlled through the
standard DAT static registry (/etc/dat.conf). The DAT static registry will also define the behavior
of the multi rail device(s). For example: you may have one device for data-striping capability,
and a different device for load-balancing.

F.4 Data-striping
Data-striping is the ability to split the data that is sent from the MPI layer into multiple “chunks”
and send them across the configured underlying HCA devices. This can increase the available
bandwidth to the MPI process by a factor of N where N is the number of HCAs configured for the
multi-rail device in use. The stripe-size as well as which HCAs should be used for the multi-rail
device is configured through the static DAT registry.

Although this technique in theory provides N times higher interconnect bandwidth, its actual
benefit to application performance depends on several issues:

• How much time does the application spend in communication?

• Is the communication time bandwidth or latency sensitive?
• Does the switching fabric sustain the accumulated traffic without incurring throttling or
reduced throughput?
• Do the nodes in the system have sufficient I/O and memory bandwidth to sustain the
accelerated traffic?

F.5 Load-balancing
The purpose of load balancing is to distribute the available HCAs in the system among the MPI
processes to avoid that multiple processes will congest a single HCA. This will increase the
aggregated system bandwidth available to all MPI processes compared to a single HCA, and
hence has the potential of improving the MPI application performance.

Platform MPI 5.6.5 User’s Guide Page 92

F.6 Multirail Configuration
Multi rail devices in Platform MPI are DAPL 1.2 devices and must be configured in the DAPL static
registry (/etc/dat.conf). The configuration format follows the standard DAPL 1.2 format and uses
the “instance_data” field to configure the Multi rail parameters. An example device could look
like this :

iam1 u1.2 nonthreadsafe default libibmultirail.so ri.1.2 "1 2 8192;

mlx4_0:1,2" ""

The first field must match the desired name that will be used with the “-networks” option to
mpirun/mpimon for network selection. Please note that this name cannot match any of the
built-in names in Platform MPI (such as “smp”, “tcp” and “ib”). The next 5 fields should always be
as the example line above, look to the DAPL 1.2 API document for more information about the
static registry configuration if desired.

The 7th field, also known as the “instance data” field, in the configuration line is what controls
the multi rail device. It should always be a quoted string and has the following format :

<num_hcas> <num_ports> <stripe_size>; <hca_name>:<port_num>,<port_num>

As you can see from the example above, the “iam1” device would use 1 HCA (mlx4_0) and 2
ports (port1 and port2).

Another example using 2 HCAs, with only 1 port on each, is shown below :

iam2 u1.2 nonthreadsafe default libibmultirail.so ri.1.2 "2 1 8192;

mlx4_0:1; mlx4_1:1" ""

In both cases, data larger than 8192 bytes would be striped across the multiple rails using 8192
bytes per rail in a round robin fashion.

Platform MPI 5.6.5 User’s Guide Page 93

Appendix G - Checkpoint and restart
G.1 MPI application architecture
The figure below illustrates the main architecture of a running MPI application using Platform
MPI:

Figure G.1 An MPI-based parallel application running under Platform MPI

The application code itself (“application”) is linked together with functions from the Platform
MPI library (“libmpi.so”). That unit constitutes one of the parallel processes executing the work
of a running job. The running processes in every processing node are started by an independent
Platform MPI process (“mpisubmon”), which subsequently provides local MPI monitoring
services.

Each “mpisubmon” process, in turn, is started from a single top process from which the entire
parallel job is initially submitted (“mpimon”). The top process ‘mpimon’ will thereafter
communicate with and monitor all ‘mpisubmon’ processes running under it.

Platform MPI 5.6.5 User’s Guide Page 94

The “application” has direct access to files located on a shared or private file system, but it
communicates with other running processes through the MPI functions implemented in
“libmpi.so”, which establish and control the “MPI fabric” using the HPC interconnect available
infrastructure in the environment.

G.1.1 Controlling Checkpoint/Restart

In order to use the checkpoint/restart feature, two additional components are needed in the
architecture, as illustrated below:

Figure G.2 The MPI architecture including additional Checkpoint/Restart components

The “libvendorcpr.so” library is a plugin component. Currently, Platform MPI supports Berkeley
Labs open source version, BLCR. The library contains the necessary functions to interface with
BLCR-librarires to create an image of the running process and store its state during a checkpoint,
as well as to restore that image during a restart (“libvendorcpr.so”).

Platform MPI 5.6.5 User’s Guide Page 95

The final component is the an utility (“mpictl”). This is the component that controls the process
of taking checkpoints and restarting.

G.1.2 The Checkpoint Process

The checkpoint process is performed in three stages, namely:

1. The “Drainage” stage.

2. The “Closing” stage.

3. The “Checkpoint & Continue/Abort” stage.

These stages are described below.

Platform MPI 5.6.5 User’s Guide Page 96

G.1.2.1 Drainage stage
This stage comprises the initial preparation phase for the checkpoint, and it is a process of
“draining” the MPI fabric from ongoing messages. The illustration below depicts this stage:

Figure G.3 "Drainage" stage illustrated

mpimon halts all ongoing communication and the existing MPI fabric are “drained” of messages.

1 CP/R client sends req_drain.

2 mpimon requests drain from instances of mpisubmon.
3 Instances of mpisubmon request drain from processes.
4 libmpi.so drains HPC interconnect.
5 libmpi.so reports back to mpisubmon when HPC network is drained.

Platform MPI 5.6.5 User’s Guide Page 97

 6 mpisubmon reports back to mpimon when all processes have drained network(s).
7 mpimon reports back to cpr client when all instances of mpisubmon report back.
Result: At the end of this stage there will be no outstanding messages in the MPI fabric and no
running processes will be allowed to issue new MPI messages.

G.1.2.2 Closing stage

This stage is the final preparation phase for the checkpoint, and in it the entire “MPI fabric”
established over the available HPC Interconnect will be stopped and closed down. The illustration
below depicts this stage:

Figure G.4 The "Closing" stage illustrated

mpimon closes down the current MPI fabric established upon the available HPC Interconnect
environment.

Platform MPI 5.6.5 User’s Guide Page 98

 1 CP/R client sends req_close_fds.
2 mpimon requests req_close_fds from instances of mpisubmon.
3 Instances of mpisubmon request req_close_fds from processes.
4 libmpi.so closes HPC interconnect.
5 libmpi.so reports back to mpisubmon when HPC network is closed.
6 Instances of mpisubmon report back to mpimon when all processes have closed
network(s).
7 mpimon reports back to CP/R client after all instances of mpisubmon report back.
Result: When the stage is complete there will be no open MPI fabric connecting the running
processes, and each running process will be effectively a stand-alone process, whose image can
be checkpointed on an individual basis.

Platform MPI 5.6.5 User’s Guide Page 99

G.1.2.3 Checkpoint & Continue/Abort stage
This is the final stage and it is when the checkpoint actually happens. Having prepared the
environment through the drainage and closing down of the MPI fabric, each individual process
can now undergo a checkpoint. The figure below illustrates this stage:

Figure G.5 The "Checkpoint & Continue" stage illustrated

The components from Platform MPI will then invoke the necessary functions in “libvendorcpr.so”
to perform the checkpoint (i.e., save the state of all running processes onto disks).

1 CP/R client sends req_checkpoint_and_continue.

2 mpimon requests req_checkpoint_and_continue from instances of mpisubmon.
3 The instances of mpisubmon request req_checkpoint_and_continue from
processes.

Platform MPI 5.6.5 User’s Guide Page 100

 4 libmpi.so invokes checkpoint function in CP/R.so which saves the state of the process
in a storage system.
5 CP/R.so saves the state of the process in a storage system. On return, the state is
saved.
6 When CP/R.so returns libmpi.so will reopen and restore all HPC connections. The
processes continue. The checkpoint process is complete.
Result: When the processes report back after having saved their state through a checkpoint,
Platform MPI could re-establish and re-open the MPI fabric over the HPC Interconnect and the
processes will continue their work or abort the processes if "abort" is requested.

G.1.2.4 Restart
Applications can be restarted from a checkpoint. A restart resembles the continue-part of the
"checkpoint & continue". The processes are loaded into memory by the
libvendorcpr.so-functionality and control is transferred to the process which re-establishes /
re-opens the MPI fabric before continuing.

G.2 Using checkpoint/restart

G.2.1 Starting jobs

An application should be started in a normal manner except that "-ccp_mode Platform" should
be given as option to mpimon/mpirun.

G.2.2 Taking a checkpoint

Use mpictl to have mpimon take a checkpoint.
mpictl checkpointJob [--jobid <job-id>]

Depending on whether the --abort is given the job will continue or stop after the checkpoint is
done. If the jobid-option is omitted the newest job for the user is checkpointed

G.2.3 Restarting from a checkpoint

The job can be restarted from a checkpoint using:
mpictl restartJob [<cp-spec>]

The <cp-spec> could be omitted and if soo the newest job/checkpoint will be restarted. The job
can be indentified by given "--jobid <jobid>" and/or "--checkpoint-number <cpn>". (<jobid>
and <cpn> can be found by "mpictl listJobsAndCheckpoints)

Start newest checkpoint for newest job:

mpictl restartJob

Start newest checkpoint for specified job:

mpictl restartJob --jobid 2dfa0b7a-31e9-4692-81c1-60765753e25a

Start given checkpoint for newest job:

mpictl restartJob -checkpoint-number 4

Platform MPI 5.6.5 User’s Guide Page 101

Start given checkpoint for specified job:
mpictl restartJob --jobid 2dfa0b7a-31e9-4692-81c1-60765753e25a -checkpoint-number
4

It is possible to make changes to the environment for the restarted job; node-list, network and
other parameters to mpimon can be changed. Examples:

Start with new nodelist:

mpictl restartJob [<cp-spec>] --nodelist "n1 n2 n3 n4"

Start with new network-spec:

mpictl restartJob [<cp-spec>] --mpimon-opt "-net ib,myri,tcp"

G.2.4 List taken checkpoints

mpictl listJobsAndCheckpoints

will give a list of currently executing jobs, and checkpoints taken with checkpoint-numbers.

G.2.5 Delete checkpointfiles

mpictl deleteCheckpoints

Delete checkpoint-files for a jobid or if no jobid is given, all checkpoint-files will be deleted.

Platform MPI 5.6.5 User’s Guide Page 102

Appendix H - Troubleshooting
This appendix offers initial suggestions for what to do when something goes wrong with
applications running together with Platform MPI and is intended to serve as a starting point to
help with software and hardware debugging. The main focus is on locating and repairing faulty
hardware and software setup, but can also be helpful in getting started after installing a new
system.

When problems occur, first check the list of common errors and their solutions; an updated list of
Platform MPI-related Frequently Asked Questions (FAQ) is posted in the Support section
of the Platform website (http://www.platform.com). If you are unable to find a solution to
the problem(s) there, please read this chapter before contacting [email protected].

Problems and fixes reported to Platform will eventually be included in the appropriate sections of
this manual. Please send relevant remarks by e-mail to [email protected].

Many problems find their origin in not using the right application code, daemons that Platform
MPI rely on are stopped, and incomplete specification of network drivers. Below some typical
problems and their solutions are described.

H.1 Why does not my program in Unix start to run?

Error Message: mpimon: command not found.

Error Message: mpimon can’t find mpisubmon.

Error Message: The application has problems loading libraries (libsca*).

Solution: Run the scampivars.sh/csh script to setup environment-variables correctly.

Error Message: Incompatible MPI versions.

mpid, mpimon, mpisubmon and the libraries all have version variables that are checked at
start-up. To insure that these are correct, try the following:

1. Set the environment variable MPI_HOME correctly

2. Restart mpid, because a new version of Platform MPI has been installed without
restarting mpid
3. Reinstall Platform MPI, because a new version of Platform MPI was not cleanly
installed on all nodes.

Error Message: Set working directory failed

Solution: Platform MPI assumes that there is a homogenous file-structure. If you start
mpimon from a directory that is not available on all nodes you must set
SCAMPI_WORKING_DIRECTORY to point to a directory that is available on all nodes.

H.2 Why can I not start mpid in Unix?

Problem:mpid opens a socket and assigns a predefined mpid port number (see /etc/services for more
information), to the end point. If mpid is terminated abnormally, the mpid port number cannot be
re-used until a system defined timer has expired.

Platform MPI 5.6.5 User’s Guide Page 103

Solution: Use netstat -a | grep mpid to observe when the socket is released. When the socket
is released, restart mpid again.

H.3 Space overflow

Problem:The application has required too many shared memory resources.
Solution: The mpimon pool-size specifications are too large, and must be reduced.

H.4 Why does my program terminate abnormally?

Problem:The application core dumps.
Solution: Use a debugger to locate the point of violation. The application may need to be
recompiled to include symbolic debug information (-g for most compilers).

Solution: Define SCAMPI_INSTALL_SIGSEGV_HANDLER=1 and attach to the failing process

with the debugger.

H.5 General problems

Problem:Are you reasonably certain that your algorithms are MPI safe?
Solution: Check if every send has a matching receive.

Problem:The program just hangs

Solution: If the application has a large degree of asynchronicity, try to increase the
channel-size. Further information is available on page 64.

Problem:The program terminates without an error message

Solution: Investigate the core file, or rerun the program in a debugger.

Platform MPI 5.6.5 User’s Guide Page 104

Appendix I - Related documentation
[1] MPI: A Message-Passing Interface Standard
The Message Passing Interface Forum, Version 1.1, June 12, 1995,
Message Passing Interface Forum, http://www.mpi-forum.org
[2] MPI: The complete Reference: Volume 1, The MPI Core
Marc Snir, Steve W. Otto, Steven Huss-Lederman, David W. Walker, Jack Dongarra. 2e, 1998, The
MIT Press, http://www.mitpress.com
[3] MPI: The complete Reference: Volume 2, The MPI Extension
William Grop, Steven Huss-Lederman, Ewing Lusk, Bill Nitzberg, W. Saphir, Marc Snir, 1998, The MIT
Press, http://www.mitpress.com
[4] Dat Collaborative: User-level API-spesification(µDAPL)
http://www.datcollaborative.org
[5] Review of Performance Analysis Tools for MPI Parallel Programs
UTK Computer Science Department,
http://www.cs.utk.edu/~browne/perftools-review/
[6] Debugging Tools and Standards
HPDF - High Performance Debugger Forum, http://www.ptools.org/hpdf/
[7] Parallel Systems Software and Tools
NHSE - National HPCC Software Exchange, http://www.nhse.org/ptlib
[8] MPICH - A Portable Implementation of MPI
The MPICH home page, http://www.mcs.anl.gov/mpi/mpich/index.html
[9] MPI Test Suites freely available
Argonne National Laboratory, http://www-unix.mcs.anl.gov/mpi/mpi-test/tsuite.html
[10] ROMIO - A high-performance, portable implemetation of MPI-IO
The I/O chapter in MPI-2. Homepage: http://www-unix.mcs.anl.gov/romio/

Platform MPI 5.6.5 User’s Guide Page 105

Appendix J - List of Figures
Platform MPI Simplified Topology 1
Example of debug windows 19
Application startup 41
Network devices part 1 42
Network devices part 2 43
Thresholds for different communication protocols 46
Inline protocol 47
Eager Buffering Protocol 48
Transporter Protocol 49
Zero Copy Protocol 50
A ultrasound image before and after contrast enhancement 52
Gray map 91
An MPI-based parallel application running under Platform MPI 94
The MPI architecture including additional Checkpoint/Restart components 95
"Drainage" stage illustrated 97
The "Closing" stage illustrated 98
The "Checkpoint & Continue" stage illustrated 100

Scali MPI Connect 5.6 User’s Guide Page 106

Appendix K - List of Tables
Table 1.1 Abbreviations ............................................................................................ 4
Table 1.2 Terms ...................................................................................................... 5
Table 1.3 Typographical conventions .......................................................................... 6
Table 2.1 mpicc/mpic++/mpif77 options .................................................................... 8
Table 2.2 MPI Connect Options .................................................................................10
Table 2.3 mpimon options ........................................................................................11
Table 2.4 mpimon option syntax ...............................................................................12
Table 2.5 mpirun usage ...........................................................................................16
Table 2.6 mpirun options .........................................................................................17
Table 2.7 mpimon options ........................................................................................24
Table 3.1 mpimon usage (windows) ..........................................................................29
Table 3.2 mpimon options (windows) ........................................................................30
Table 3.3 mpimon options (windows) ........................................................................37
Table 5.1 mpimon trace-options ...............................................................................53
Table 5.2 Default trace-output ..................................................................................54
Table 5.3 Extended trace-output ...............................................................................54
Table 5.4 mpimon timing options ..............................................................................57
Table A.1 lmutil commands and options .....................................................................77
Table B.1 lminstall license options .............................................................................80
Table B.2 smcinstall options .....................................................................................81

Scali MPI Connect 5.6 User’s Guide Page 107

 Index
Symbols
...................................................................................................... 12, 13, 16, 20, 30, 33
# ............................................................................................................................. 6
% ............................................................................................................................. 6

A
activation code ..........................................................................................................78

B
Benchmarking Platform MPI ........................................................................................66

C
-chunk_size ......................................................................................................... 21, 34
Command line options ........................................................................................... 14, 31
Communication protocols in ScaMPI
Eagerbuffering protocol ...........................................................................................48
Inline protocol ........................................................................................................47
Transporter protocol........................................................................................... 49, 50
Zerocopy protocol ...................................................................................................50
Communication resources in Platform MPI ................................................................ 21, 34

D
-debug .....................................................................................................................18
-debugger ................................................................................................................18
-display ....................................................................................................................18

E
eagerbuffering protocol .................................................................................... 21, 34, 48
Environment .............................................................................................................27
Environment-variable options ................................................................................. 14, 31

F
-failover_mode ..........................................................................................................18

G
global variables .................................................................................................... 23, 36
GM ..........................................................................................................................44

I
inline protocol ................................................................................................. 21, 34, 47

K
Key Activation ...........................................................................................................77

L
libfmpi ..................................................................................................................8, 28
libmpi ....................................................................................................................... 8
Licensing ..................................................................................................................77
lmhostid ...................................................................................................................78

Platform MPI 5.6.5 User’s Guide Page 108

 lmutil .......................................................................................................................78

M
Managing Product keys...............................................................................................78
manual activation ......................................................................................................78
MPI........................................................................................................................ 105
MPI_Abort() ......................................................................................................... 24, 37
MPI_Bsend() ........................................................................................................ 22, 36
MPI_COMM_WORLD .................................................................................................... 5
MPI_Irecv() ......................................................................................................... 22, 35
MPI_Isend()......................................................................................................... 22, 35
MPI_Probe()......................................................................................................... 22, 35
MPI_Recv().......................................................................................................... 22, 35
MPI_SOURCE ....................................................................................................... 22, 35
MPI_TAG ............................................................................................................. 22, 35
MPI_Waitany() ..................................................................................................... 23, 36
mpi.h .................................................................................................................. 23, 36
mpiboot ...................................................................................................................40
mpic++..................................................................................................................... 8
mpicc........................................................................................................................ 8
MPICH.................................................................................................................... 105
mpid ........................................................................................................................40
mpif.h ................................................................................................................. 23, 36
mpif77 ...................................................................................................................... 8
mpimon......................................................... 11, 14, 17, 18, 21, 24, 26, 29, 31, 34, 37, 39, 40
mpimon-parameters ..................................................................................................30
mpirun ................................................................................................................ 16, 21
mpirun options ..........................................................................................................17
mpisubmon...............................................................................................................40
Myrinet ....................................................................................................................44

N
-networks ............................................................................................................ 14, 32

O
Optimize Platform MPI performance .............................................................................65

P
Platform MPI Environment ........................................................................................... 7
Platform product activation web service ........................................................................78
-pool_size............................................................................................................ 21, 34
Product Activation......................................................................................................78
product key ..............................................................................................................77
demo keys ........................................................................................................ 77, 84
permanent product key ...................................................................................... 77, 84
Profiling
ScaMPI..................................................................................................................51

R
Run parameters.........................................................................................................18

Platform MPI 5.6.5 User’s Guide Page 109

S
SCAFUN_SMP_BUFSZ ............................................................................................ 21, 34
SCAFUN_TCP_RXBUFSZ......................................................................................... 21, 34
SCAFUN_TCP_TXBUFSZ ......................................................................................... 21, 34
ScaMPI
Builtin-timing .........................................................................................................57
Builtin-trace ...........................................................................................................53
Executables ...........................................................................................................40
SCAMPI_ALGORITHM .................................................................................................68
SCAMPI_BACKOFF_ENABLE, backoff-mechanism............................................................65
SCAMPI_BACKOFF_IDLE,backoff-mechanism .................................................................65
SCAMPI_BACKOFF_MAX, backoff-mechanism ................................................................65
SCAMPI_BACKOFF_MIN, backoff-mechanism .................................................................65
SCAMPI_CPU_USAGE .................................................................................................61
SCAMPI_INSTALL_SIGSEGV_HANDLER, builtin SIGSEGV handler ................................... 104
SCAMPI_PROFILE_APPLICATION ............................................................................. 20, 33
SCAMPI_PROFILE_APPLICATION_END ..................................................................... 20, 33
SCAMPI_PROFILE_APPLICATION_START .................................................................. 20, 33
SCAMPI_REDUCE_ALGORITHM ....................................................................................69
SCAMPI_SEPARATE_OUTPUT .................................................................................. 20, 33
SCAMPI_TIMING, builtin timing-facility .........................................................................57
SCAMPI_TRACE, builtin trace-facility ............................................................................53
SCAMPI_WORKING_DIRECTORY, set working directory ................................................. 103
ScaMPI.conf ..............................................................................................................32
segment protect violation handler ........................................................................... 20, 33
SIGSEGV ............................................................................................................. 20, 33
SIGTSTP...................................................................................................................17
SIGUSR1 ..................................................................................................................17
SMP .................................................................................................................... 21, 34
stdin ................................................................................................................... 13, 31
stdout ................................................................................................................. 13, 31

T
TCP .................................................................................................................... 21, 34
totalview ..................................................................................................................21
transporter protocol ......................................................................................... 21, 34, 49
tvmpimon .................................................................................................................21

U
Unsafe MPI programs ............................................................................................ 23, 36
User interface errors and warnings.......................................................................... 24, 37

V
-verbose.............................................................................................................. 24, 37

X
xterm.......................................................................................................................18

Z
zerocopy protocol ......................................................................................................50

Platform MPI 5.6.5 User’s Guide Page 110