Another Architecture: PVM on Windows 95/NT

Markus Fischer y Jack Dongarra z

October 4, 1996

Abstract
This paper describes the implementation of PVM in the new WIN32-bit world. There are
no restrictions to existing applications which are using PVM since it is fully compatible to the
existing PVM3 release. We discuss the limits and provide some benchmarking results. The
software package is freely available at netlib: http://www.netlib.org/pvm3/index.html

1 Introduction
1.1 The PVM System
PVM (Parallel Virtual Machine) is a de facto standard message passing interface. It is an inte-
grated set of software tools and libraries that emulates a general-purpose, exible, heterogeneous
concurrent computing framework on interconnected computers of varied architectures. The overall
objective of the PVM system is to enable such a collection of computers to be used cooperatively
for concurrent or parallel computation.
1.1.1 Current Architectures
So far, PVM is available for 40 dierent architectures combining Unix- workstations, shared memory
machines and MPP's to one single parallel virtual machine. Obviously, the established architectures
take place in the more scientic area.
1.1.2 New Architecture: Computers running WINDOWS 95/NT
However, since software companies like MS provide multiuser (WINDOWS NT) and multitasking
(WINDOWS 95/NT) operating systems, lots of personal computers in smaller companies could be
used for parallel processing if this processing power could be put together. As a matter of fact,
Department of Computer Science, University of Tennessee, TN 37996
y Department of Computer Science, University of Paderborn, 33100 Paderborn, Germany
z
Mathematical Science Section, Oak Ridge National Laboratory, Oak Ridge, TN 37831

0
there are Unix -like operating systems (Linux for example) for PC's available, but they are not
used by companies or individual people. It is Microsoft which leads the market with almost 90
percent. But not only improved operating systems and development tools lead to a growing market
for scientic computing with PC's.
Figure 1 shows the growing performance of PC's compared to Unix workstations PDS] (SUN
workstations for example) which will make the use of more aordable PC's more attractive.

MIPS
200
PC Generation
SUN Workstations Sun Ultra1
Pentium Pro

150
Sparc 20
Pentium 90
100
Pentium 60

Sparc 10
50

Intel 486/33
Sparc2
Sparc 1
Intel 386

Generations

Figure 1: Growing Performance of PC's

Therefore, PVM now oers a version to this so called WIN32 - bit world. Using existing equipment
and operating system, its application could be to
solve large industry problems like parallelized combinatorical optimization algorithms ( Job
Scheduling , Cutting Stock ) at 'home'.
use computers in High Schools for teaching purposes. Basic steps in the more and more
important becoming parallel processing area can be made.
combine new WIN32 - compiler function calls (like built-in 3D rendering) with parallel pro-
cessing power.
PVM's most powerful feature is that it provides the message passing interface which lets the
application assume to run on one single machine.

PVM - Parallel Virtuell Machine

Windows
MPP’s Alpha’s
Unix 32-bit generation
Workstations 95/NT

Figure 2: The PVM Model

1
2 Developing Applications in the WIN32-bit World
2.1 Towards scientic computing
A bit of history can let us understand our diculties in programming for a parallel world for PC's.
Microsoft's rst operating system was its command-line MS-DOS. Because of the competition with
Apple Macintosh the rst version of Windows was released. It oered the rst graphical user
interface for PC's to ease working with computers. The more and more growing need of connected
computers in companies leaded to Windows for Workgroups, followed by Windows 95/NT today.
The newest version of these operating systems oer the same surface and are making use of the
32-bit availability.
Developing applications for both operating systems requires only one compiler. However lots of
functions from the more professional Windows NT are not implemented in Windows 95. Basically
it can be said that Windows 95 is a subset of Windows NT and is just on its way for becoming a
real operating system.
2.2 Dierences Between Windows 95 and Windows NT
The major dierence is that Windows 95 is designed to provide downgrade compatibility. Programs
developed under the early DOS 3.3 still run with Windows 95. Windows NT does not provide this
MS-DOS shell. This is the reason why the PVM WIN32 executable itself switches to the running
operating system.
The reason is the concept of using device drivers. User's code in Windows NT is not allowed to
access hardware directly.
In former versions of Windows direct access was possible and caused unmeant shutdowns when
conicts arose. In Windows NT, function calls to device drivers, which actually access the hardware,
guarantee that Windows NT runs stable (Process runstate level).
Another contrast to Windows NT is that Windows 95 allows only one user to be logged in at one
time. It therefore does not provide a function which lets the system give dierent users dierent
access privileges. Therefore every process actually has 'root' privileges. Performance is also better
on NT. It oers a Virtual Memory Space resulting in faster switching of multiple processes. Fur-
thermore NT is designed to run as a multiple processor machine. Its scalability is dependant on
the motherboard. Unfortunately we have no experience within that environment, however, it can
be assumed that the operating system takes care of dynamic started processes and performs the
mapping. This will probably lead to an investigation of a shared memory implementation on those
systems.
2.2.1 Security Aspects
The new generation oers sucient security only for the Windows NT side. Security is set to
objects, where objects can be declared as multiple things. It is possible to handle a socket as an
object, but also a process or a le, standard input, -output and -error can be seen as an object.
Finally, a pointer to this object can modify the security status of this object. This modication
is done by the Security Identier, which exists for every user and in which the privileges of the
specic users are set. In PVM an important security aspect is a secured le system in which users
can deny access to this le to other users.

2
To the contrast of Windows NT's NTFS, Windows 95 does not oer a le system which restricts
access. The FAT is readable for everyone. The reason was mentioned above, once logged in, the
process has root privileges. A process which likes to enroll into the PVM reads a socket address out
of the %PVM_TMP%\pvmd.username le. Since this le is readable to every process dierent users
could get access to the pvmd daemon ! It is recommended to run NT as the operating system,
however people are using Windows 95 more often than NT because of the downgrade capability.

3 The Implementation
Like the original PVM this version needs a daemon process called pvmd, which keeps track of the
entire task management. The rst pvmd process becomes a master pvmd. In the original version
this process is also used to start up slave daemons on other hosts. This version however uses a
separate hoster process. This is kept invisible to the user, so that he will not notice any dierence.
The hoster process is started automatically and is running as a task of the virtual machine. He
also will be started again automatically, once a user has executed a reset.
processes
time

pvm-console
master pvmd daemon
"spawn"

provide socket
port

connect
to localdaemon

no entry for hoster:

"pvm_addhost" reminder for

message to hoster
hoster process
"spawn"
Blocked

connect
to local daemon

enroll as hoster
"pvm_reg_hoster"

send out
reminder message slave pvmd daemon
rsh / rexec
new hosts

slave_config

host table update

host startup result

Figure 3: PVM Startup Protocol

The main functions of these daemons are to start or delete new tasks, to route messages between
tasks, but also to establish direct connections for better performance. For more detailed information
the interested reader may refer to the book about PVM PVM].

3
3.1 The Communication Layer
Messages between processes are exchanged using Windows Winsockets. They oer TCP and UDP
on top of the IP layer. The specication is close to the BSD standard, however it is not possible to
handle them as le or stream descriptors. It diers also from the standard by initializing a socket
structure, where the version of the socket layer must be specied. A socket failure is reported
otherwise.
Using a heterogeneous system with dierent architectures requires encoding and decoding of mes-
sages. The WIN32 version oers XDR -en/decoding possibilities. Like the existing PVM, message
buers can be created in three dierent manners: PvmDataInPlace (fastest method, only pointer
to the message data is stored and packed for message transfer without encoding), PvmDataRaw
(a copy of the data is made, sent out without encoding), PvmDataDefault (a encoded copy is sent
out).
Messages sent inbetween one architecture should use PvmDataRaw or better PvmDataInPlace, if
possible. XDR en- /decoding is expensive and slows down the performance (dependend on CPU
power).

pvm hoster

TCP TCP pvm

UDP group server
WinSock Compliant Application
TCP TCP
UDP UDP Windows Socket DLL

User task Protocol Stack (TCP/IP)

User task
Hardware Driver
TCP User task
TCP
connection Network (Hardware) Interface
pvm daemons
User task
Network

Figure 4: Communication and Layout of PVM

3.2 Dierences Between WIN32 and UNIX

In the Unix environment a user has his unique user-id. Furthermore all users have the same rights
concerning executing processes. The easy method of the Unix call chmod secures les from being
read by other users. It can be allowed / denied to all users or the access may be restricted to
a specic group. In WIN32 each user has to be included into a security object if he wants to
have access to it. If a new object is created, a security attribute is generated to this object and
if not specied otherwise, access is granted to everyone. Users even have special rights in this
environment. More specic the user's SID, the Security Identier, which is a binary structure

4
stores the rights for the user which are set by the administrator. The SID is invisible to other users
and information can only be obtained by using specic function calls. Therefore dierent PVM's in
the WIN32 environment are distinguished by there user name, which is also stored in the SID and
is connected to the running process. Creating a new process can be done by several spawn calls
which will generally take a lename as its argument. Dierent avors provide environment setting
or startup parameters. It is possible to start a new process but it is not possible to split up a task
in the way the Unix 'fork' call does.
This leads to the need of a hoster process which is responsible for starting up new daemons on
other hosts. If the master daemon would perform this startup, the process would block, waiting for
a response or sending initial data to the new hosts. Consequently, other input would have to wait
like the request of starting up new hosts, which is sent to the master daemon, routing messages or
startin up new tasks, for example. The hoster method keeps the master daemon free to response
to other requests.

4 Using PVM
Before we describe the software handling of this version we point to the additional requirements.
4.1 Settings
The correct use of PVM for WIN32 needs the following environment variables,
PVM TMP which species the location of the temporary les (PVM TMP=c:\temp)
PVM ROOT points to the installation of pvm (PVM ROOT=c:\pvm\pvm3)
PVM RSH locates the rsh-command (On NT: %winntsystem%\system32\rsh.exe)
PVM ARCH has to be set to WIN32
As well as Windows 95, Windows NT is designed for networking. Nevertheless they do not provide
convenient tools for remote process handling. As a matter of fact at least one additional daemon
has to run on each host. Users have to look out for a remote shell daemon (rshd), which will allow
to add other hosts to the machine. If you have a dierent account on the other machine, you will
probably need a remote execute daemon (rexecd). Note that Unix does provide those. They are
only required on WIN32 computers. It is also possible to perform a manual startup. This, however,
is very inconvenient and takes time.
4.2 Setting Up a Virtual Machine
To provide a convenient way for the user to interact with the pvmd, the pvm-console process can
be used. Here new hosts can be added to the virtual machine and passwords can be typed in. The
user can reset his parallel virtual machine if tasks hang. Last not least in provides the possibility
of a proper shutdown. Figure 5 shows a typical startup of a virtual machine. After starting up
the master pvm daemon via pvm (gure 5) the user can add other hosts to his virtual machine.
(host 'rudolph' and 'shenzi' respectively). Tasks can be started within the console and the 'ps -a'
command gives information about running tasks on every machine in the VM.

5
 c:> pvm hostfile
hoster() 2 to start
0. t80000 rudolph so=""
1. tc0000
:
thud so=""
Password
3.3.10
t40002
pvm> conf
3 hosts, 2 data formats
HOST DTID ARCH SPEED
ed 40000WIN32 1000
rudolph 80000 SUN4 1000
thud c0000 SUN4 1000
pvm> ps
HOST TID FLAG 0x COMMAND
ed 40001 204/H,c c:/pvm/pvm3/bin/WIN32/hoster.exe
pvm> spawn -> spmd
1 successful
t80001
libpvm [t80001]: token ring done
pvm>

Figure 5: PVM Session

4.3 Creating a PVM Application: FORTRAN and C

Sequential code can be parallelized using PVM's message passing interface. The algorithm has
to be changed that processes can divide up work and gather the solution. These functions can
be found in the library of PVM. A linking with $(PVM_ROOT)/lib/$(PVM_ARCH)/libpvm3.lib is
necessary. The new PVM version also oers group functionality. Application which are using the
group server have to link with $(PVM_ROOT)/lib/$(PVM_ARCH)/libgpvm3.lib, too. It is also possible
to bring existing FORTRAN applications to the new environment. They have to be linked with
$(PVM_ROOT)/libfpvm/$(PVM_ARCH)/libfpvm3.lib.

4.4 Debugging in PVM on WIN32

The common way in the existing PVM version was to start a new task under a debugger, which was
specied in the $(PVM_ROOT)/lib/debugger le. This debugger was displayed at the users machine.
Debugging is also possible in the WIN32 version. Dierent to the existing Unix version the new
task is started on the remote machine, but waits for the connection to a debugger. Users then
have to start their debugger manually on the local machine and then choose the option for remote
connection. For developing, we advise to test the application on the local machine. The operating
system allows local socket connections without restrictions. (See gure 6)

5 Results
We ran a benchmark program which performs a ping pong test between two processes. Each size
of a message was sent out and received 100 times. Based on these round trip values, an average
6
round trip time is computed. Every message content was double checked, on the receiver and on
the initiator. The tests did not drop any message and they were received correctly by 100 percent.
The result for the WIN32 environment lacks good performance. We optimized the code and will
TCP Over Ethernet Local TCP
0.7 4.5
"sun.sun.tcp.inp" "sparc1.local.tcp.inp"
"win32.win32.tcp.inp" "win32.local.tcp.inp"
"win32.sun.tcp.def" 4 "sparc5.local.tcp.inp"
0.6

3.5

0.5
3

0.4
2.5
MB/s

MB/s
2
0.3

1.5
0.2

0.1
0.5

0 0
0 100000 200000 300000 400000 500000 600000 0 100000 200000 300000 400000 500000 600000
Message Sizes in Bytes Message Sizes in Bytes

Figure 6: PVM Performance

include the new winsocket library (Winsocket 2 specication), which will lead to better values.

6 Conclusion
As tests have shown, the PVM package on top of MS Windows NT/95 brings reliable parallel
computing possibilities. Crashed applications do not interfere with the operating system and users
do not have to to be afraid of causing unmeant shutdowns. It is also possible to have multiple,
communicating processes running on one machine.
The version to the WIN32 world was done by using MS VC++ compiler but the library can also
be linked by other brands compilers.
Also a cross compiling of this package with fortran compilers was done successfully. Users can even
take their existing PVM - application and run it on the new architecture. There is no need of long
lasting modications.
The only restriction is the possibility of using the pvm_sendsig call, which is usually provided by
the PVM interface.

References
PVM] Al Geist, Adam Beguelin, Jack Dongarra, Weicheng Jiang, Robert Manchek, Vaidy Sun-
deram PVM: Parallel Virtual Machine A Users' Guide and Tutorial for Networked Par-
allel Computing 1994. Published by MIT Press, Boston.
http://www.netlib.org/pvm3/book/pvm-book.html
PDS] Alfred Aburto PDS: The Performance Database Server November 26, 1995, Naval Ocean
Systems Center, San Diego
http://performance.netlib.org/performance/html/PDStop.html
7