CENG479 PARALLEL COMPUTING

Lec.8: POSIX Threads

Dr. Hüseyin TEMUÇİN

Gazi Üniversitesi Bilgisayar Mühendisliği Bölümü
BM5351 PARALEL HESAPLAMA YÖNTEMLERİ

What about memory structure ?

● Shared Memory System

● Distributed Memory System

These slides are adopted from Prof Dr Zahran's Parallel Computing lecture notes.
BM5351 PARALEL HESAPLAMA YÖNTEMLERİ

Shared Memory System

● A collection of autonomous processors is connected to a memory system via an

interconnection network.
● Each processor can access each memory location.
● The processors usually communicate implicitly by accessing shared data
structures.

These slides are adopted from Prof Dr Zahran's Parallel Computing lecture notes.
BM5351 PARALEL HESAPLAMA YÖNTEMLERİ

Shared Memory System

These slides are adopted from Prof Dr Zahran's Parallel Computing lecture notes.
 Process

A process is an instance of a running (or suspended) program. In addition to its executable, it consists of the
following:

● A block of memory for the stack

● A block of memory for the heap
● A Descriptors of resources that the that the system has allocated for the process—for example, file
descriptors
● Security information—for example, information about which hardware and software resources the process
can access
● Information about the state of the process, such as whether the process is ready to run or is waiting on a
resource, the content of the registers including the program counter, and so on

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Process

● A process’ memory blocks are private: another process can’t directly access the memory of a process unless
the operating system intervenes.
○ In most systems, a default and expected behaviours
● However, this isn’t what we want when we’re running shared-memory programs.
○ At a minimum, we’d like certain variables to be available to multiple processes
○ So shared-memory “processes” typically allow much easier access to each others’ memory.
○ They also often share things such as access to stdout .
● In fact, it’s conceivable that they share pretty much everything that’s process specific, Except
○ Their stacks
○ Their program counters.

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Thread

● This can be relatively easily arranged by starting a single process and then having the process start
these “lighter-weight” processes.
● The more commonly used term, thread, comes from the concept of “thread of control.”
○ A thread of control is just a sequence of statements in a program.
● The term suggests a stream of control in a single process, and in a shared-memory program a single
process may have multiple threads of control.

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Posix Threads

● Pthreads is not a programming language (such as C or Java). Rather, like MPI, Pthreads specifies a
library that can be linked with C programs.
○ POSIX is a standard for Unix-like operating systems—for example, Linux and Mac OS X
● Unlike MPI, the Pthreads API is only available on POSIX systems—Linux, Mac OS X, Solaris, HPUX,
and so on
● Unlike MPI, there are a number of other widely used specifications for multithreaded
programming:
○ Java threads,
○ Windows threads,
○ Solaris threads.
● Since Pthreads is a C library, it can, in principle, be used in C ++ programs

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Posix Threads

● Execution
○ gcc − g − Wall − o pth hello pth hello.c − lpthread
● The − lpthread tells the compiler that we want to link in the Pthreads library.
● To run the program, we just type
○ ./pth hello <number of threads>

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Posix Threads - Hello World

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Posix Threads - Hello World

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Posix Threads - Hello World

● In Line 6 we define a global variable thread count .

○ In Pthreads programs, global variables are shared by all the threads.
○ Local variables and function arguments—that is, variables declared in functions—are (ordinarily) private to
the thread executing the function.
● If several threads are executing the same function, each thread will have its own private copies of
the local variables and function arguments.
○ This makes sense if you recall that each thread has its own stack

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
 Starting the threads

● Unlike MPI programs, in which the processes are usually started by a script,
○ The threads are started by the program executable.
○ This introduces a bit of additional complexity, as we need to include code in our program to explicitly start the
threads
● The pthread t objects are examples of opaque objects.
○ The actual data that they store is system specific, and their data members aren’t directly accessible
○ However, the Pthreads standard guarantees that a pthread t object does store enough information to
uniquely identify the thread with which it’s associated.
● The most Pthreads functions, its name starts with the string pthread_

These slides are based on Peter Pacheco, An Introduction to Parallel Programming Book, 2012
pthread_create

● The first argument is a pointer to the appropriate pthread_t object. Note that the object is not allocated by the
call to pthread create
○ it must be allocated before the call.
● We won’t be using the second argument, so we just pass the argument NULL in our function call
● The third argument is the function that the thread is to run
● The last argument is a pointer to the argument that should be passed to the function start routine
● The return value for most Pthreads functions indicates if there’s been an error in the function call
Running the thread

● The thread that’s running the main function is sometimes called the main thread.
○ Hence, after starting the threads, it prints the message
● In Pthreads, the programmer doesn’t directly control where the threads are run.
● There’s no argument in pthread create saying which core should run which thread.
○ Thread placement is controlled by the operating system.
● If a program starts more threads than cores, we should expect multiple threads to be run on a
single core.
Stopping the thread
Dynamic Thread Management

● Imagine a Web server that handles requests for information about highway traffic in the San
Francisco Bay Area.
○ Suppose that the main thread receives the requests and subsidiary threads actually fulfill the requests. At 1
o’clock on a typical
○ Tuesday morning, there will probably be very few requests, while at 5 o’clock on a typical Tuesday evening,
there will probably be thousands of requests.
● Thus, a natural approach to the design of this Web server is to have the main thread start
subsidiary threads when it receives requests.
Dynamic Thread Management

● Thread startup necessarily involves some overhead.

○ The time required to start a thread will be much greater than, say, a floating point arithmetic operation,
● So in applications that need maximum performance the “start threads as needed” approach may
not be ideal.
● In such a case, it may make sense to use a somewhat more complicated scheme—a scheme that has
characteristics of both approaches.
● Our main thread can start all the threads it anticipates needing at the beginning of the program.
● However, when a thread has no work, instead of terminating, it can sit idle until more work is
available.
MATRIX-VECTOR MULTIPLICATION
Serial code for Matrix-Vector Multiplication
Matrix - Vector Multiplication - PThread
Matrix-vector multiplication - MPI vs PThread
Critical Section

● Matrix-vector multiplication was very easy to code because the shared-memory locations were
accessed in a highly desirable way.
● After initialization, all of the variables—except y —are only read by the threads.
○ That is, except for y , none of the shared variables are changed after they’ve been initialized by the main
thread.
○ Furthermore, although the threads do make changes to y , only one thread makes changes to any individual
component, so there are no attempts by two (or more) threads to modify any single component.
● What happens if this isn’t the case? That is, what happens when multiple threads update a single
memory location?
Critical Section
Critical Section
Critical Section Example : Sum
Critical Section
Critical Section
Busy - Waiting

● The while loop is an example of busy-waiting. In busy-waiting, a thread repeatedly tests a

condition, but, effectively, does no useful work until the condition has the appropriate
value (false in our example).
● Compiler optimization problem !
Pthreads global sum with busy-waiting - 1
Pthreads global sum with busy-waiting - final
BBM101- Bilgisayar Programlamaya Giriş-I

Questions