MPI & OpenMP Examples !

TeraGrid Outreach Workshop ! Marcela Madrid! Mahidhar Tatineni !

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Parallel Programming Models !

" Parallel programming models are an abstraction above the hardware and memory architectures.! " Several approaches!
" " " " Shared Memory! Threads! Message Passing! Hybrid, GPU/CUDA etc!

" These are mature programming models and standards have been developed.!
" OpenMP! " MPI !
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

OpenMP !
" Application Program Interface (API) which allows multi-threaded, shared memory parallelism.! " Implemented by all C/C++, Fortran compilers. Also available on most major platforms: Unix/ Linux/Windows.! " Three parts:!
" Compiler Directives! " Runtime library routines ! " Environment variables!

" Several very large shared memory resources on TeraGrid: Blacklight (PSC), Nautilus (NICS), Ember (NCSA), Dash (SDSC).!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

OpenMP !
" OpenMP enables parallelism by use of multiple threads in the shared memory programming paradigm.! " Uses fork-join model of parallel execution.! " All OpenMP codes begin as a single process (master thread) until the rst parallel region => FORK multiple threads.! " Once parallel section is complete, threads synchronize and leave only master thread (JOIN).! " You can have thread private and shared variables.! " Compiling OpenMP codes is easy: one compiler ag!!
" Intel: ifort openmp, icc openmp ! " PGI: pgf90 mp, pgcc mp !

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple OpenMP Program Compute PI !

" Find the number of tasks and taskids (omp_get_num_threads, omp_get_thread_num)! PI is calculated using an integral. The number of intervals used for the integration is xed at 128000.! Use OpenMP loop parallelization to divide up the compute work.! Introduce concept of private and shared variables.! OpenMP reduction operation used to compute the sum for the nal integral.!

"

" " "

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

OpenMP Program to Compute PI !

#include <omp.h>! #include <stdio.h>! #include <stdlib.h>! int main (int argc, char *argv[]) ! {! int nthreads, tid;! int i, INTERVALS; ! double n_1, x, pi = 0.0; ! INTERVALS=128000;! /* Fork a team of threads giving them their own copies of variables */! #pragma omp parallel private(nthreads, tid)! {! /* Obtain thread number */! tid = omp_get_thread_num();! printf("Hello from thread = %d\n", tid);! /* Only master thread does this */! if (tid == 0) ! {! nthreads = omp_get_num_threads();! printf("Number of threads = %d\n", nthreads);! }! } /* All threads join master thread and disband */! n_1 = 1.0 / (double)INTERVALS; ! /* Parallel loop with reduction for calculating PI */ ! #pragma omp parallel for private(i,x) shared (n_1,INTERVALS) reduction(+:pi) ! for (i = 0; i < INTERVALS; i++)! {! x = n_1 * ((double)i - 0.5);! pi += 4.0 / (1.0 + x * x);! }! pi *= n_1; ! printf ("Pi = %.12lf\n", pi); ! }!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Examples !
" Today"s examples available in train01 account on Pople. Copy them into your directory:!
cp -r ~train01/workshop/PR_EXAMPLES .!

" Compile the OpenMP code to compute Pi:!

cd PR_EXAMPLES! icc -o pi_omp -openmp pi_openmp.c! pi_openmp.c(14): (col. 1) remark: OpenMP DEFINED REGION WAS PARALLELIZED.! pi_openmp.c(33): (col. 1) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.!

" Next: We look at the job submission script included in the directory: pople_pi_omp.sub!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Pople Job Submission Script!

#!/bin/csh! #PBS -l ncpus=4! #ncpus must be a multiple of 4! #PBS -l walltime=5:00 ! #PBS -j oe! set echo! ja! #move to my $SCRATCH directory! cd $SCRATCH! #copy executable to $SCRATCH! cp $HOME/PR_EXAMPLES/pi_omp .! #run my executable! setenv OMP_NUM_THREADS 4! ./pi_omp! ja -chlst!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Pople Job Submission & Results !

" qsub q training pople_pi_omp.sub! " Output le tagged with jobid. For example in my run : pople_pi_omp.sub.o156619! " Output from code:!
Hello from thread = 0! Number of threads = 4! Hello from thread = 3! Hello from thread = 2! Hello from thread = 1! Pi = 3.141608278534!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Message Passing Interface (MPI) !

" Set of tasks use local memory during computation.! " Multiple tasks can reside on the same physical node but will not share memory. ! " Exchange data through communication between tasks sending and receiving data.! " Standardized (MPI, MPI-2), vendors typically have optimized implementations for their machines. ! " Open source versions MPICH, MVAPICH, OpenMPI etc.!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

MPI Programming !
" Extensive set of routines (115 in MPI-1) to provide range of functionality. ! " Don"t worry you can write parallel codes with far fewer !.! " Communication routines!
" Point to point! " Collective!
" Synchronization! " Data movement! " Collective computation reduction operations (collecting data from every task and summing for example).!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple MPI Program Compute PI !

" " " Initialize MPI (MPI_Init function)! Find the number of tasks and taskids (MPI_Comm_size, MPI_Comm_rank)! PI is calculated using an integral. The number of intervals used for the integration is xed at 128000.! Computes the sums for a different sections of the intervals in each MPI task. ! At the end of the code, the sums from all the tasks are added together to evaluate the nal integral. This is accomplished through a reduction operation (MPI_Reduce function).! Simple code illustrates decomposition of problem into parallel components.!

" "

"

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

MPI Program to Compute PI !

#include <stdio.h>! #include <mpi.h>! int main(int argc, char *argv[]) ! {! int numprocs, rank;! int i, iglob, INTERVALS, INTLOC;! double n_1, x;! double pi, piloc;! MPI_Init(&argc, &argv);! MPI_Comm_size(MPI_COMM_WORLD, &numprocs);! MPI_Comm_rank(MPI_COMM_WORLD, &rank);! INTERVALS=128000;! printf("Hello from MPI task= %d\n", rank);! MPI_Barrier(MPI_COMM_WORLD);! if (rank == 0) ! {!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

printf("Number of MPI tasks = %d\n", numprocs);! }! INTLOC=INTERVALS/numprocs;! piloc=0.0;! n_1=1.0/(double)INTERVALS;! for (i = 0; i < INTLOC; i++)! {! iglob = INTLOC*rank+i;! x = n_1 * ((double)iglob - 0.5);! piloc += 4.0 / (1.0 + x * x);! }! MPI_Reduce(&piloc,&pi,1,MPI_DOUBLE,MPI_SUM, 0,MPI_COMM_WORLD);! if (rank == 0)! {! pi *= n_1; ! printf ("Pi = %.12lf\n", pi); ! }! MPI_Finalize();! } !

MPI Examples [on Steele] !

" Copy examples over:!
" cp r ~mtatinen/PR_EXAMPLES .!

" Load modules to pick up mpich2 and intel compiler:!

" module load mpich2-intel!

" Compile Code:!

" mpicc -o pi_mpi pi_mpi.c!

" Job Submission Script: pi.sub!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

MPI Job Submission Script!

#PBS -l select=1:ncpus=4! #PBS -l walltime=00:10:00! #PBS -N pi_mpi! #PBS -A TG-TRA900337N ! module load mpich2-intel! cd $PBS_O_WORKDIR! mpiexec -n 4 ./pi_mpi!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Output from MPI Job !

" Submit job using:!
qsub q R7160158 pi.sub!

" Output le tagged with jobid. For example I have pi_mpi.o7171148.! " Sample output:!
Hello from MPI task= 0! Hello from MPI task= 1! Hello from MPI task= 2! Number of MPI tasks = 4! Hello from MPI task= 3! Pi = 3.141608278534!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Break!!
" Questions!

" More detailed example to follow!!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple Application using MPI: 1-D Heat Equation !

" #T/#t = $(#2T/#x2); T(0) = 0; T(1) = 0; (0 %x%1)! T(x,0) is know as an initial condition.! " Discretizing for numerical solution we get: ! T(n+1)i T(n)i = ($&t/&x2)(T(n)i-1-2T(n)i+T(n)i+1)! (n is the index in time and i is the index in space)! " In this example we solve the problem using 11 points and we distribute this problem over 3 processors shown graphically below:!
Processor 0 Processor 2

5
Processor 1

10

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple Application using MPI: 1-D Heat Equation !

Processor 0 Processor 2

5
Processor 1

10

Processor 0: Value at 0 known; Get 4 from processor 1; Solve the equation at (1,2,3); Send 3 to processor 1 Local Data Index : ilocal = 0 , 1, 2, 3, 4 Global Data Index: iglobal = 0, 1, 2, 3, 4 Processor 1: Get 3 from processor 0; Get 7 from processor 2; Solve the equation at (4,5,6); Send 4 to processor 0 and send 6 to processor 2 Local Data Index : ilocal = 0, 1, 2, 3, 4 Global Data Index : iglobal = 3, 4, 5, 6, 7 Processor 2: Get 6 from processor 1; Solve the equation at (7,8,9); Value at 10 known; Send 7 to processor 1 Local Data Index : ilocal = 0, 1, 2, 3, 4 Global Data Index : iglobal = 6, 7, 8, 9, 10

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

FORTRAN MPI CODE: 1-D Heat Equation !

PROGRAM HEATEQN! include "mpif.h"! implicit none! integer :: iglobal, ilocal, itime! integer :: ierr, nnodes, my_id! integer :: IIM, IIK1, IIK2, IIK3, IIK4! integer :: dest, from, status(MPI_STATUS_SIZE),tag! integer :: msg_size! real*8 :: xalp,delx,delt,pi! real*8 :: T(0:100,0:5), TG(0:10)! CHARACTER(20) :: FILEN! delx = 0.1d0! delt = 1d-4! xalp = 2.0d0! call MPI_INIT(ierr)! call MPI_COMM_SIZE(MPI_COMM_WORLD, nnodes, ierr)! call MPI_COMM_RANK(MPI_COMM_WORLD, my_id, ierr)! print *, "Process ", my_id, "of", nnodes ,"has started"! ************** Initial Conditions *********************************! pi = 4d0*datan(1d0)! do ilocal = 0, 4! iglobal = 3*my_id+ilocal! T(0,ilocal) = dsin(pi*delx*doat(iglobal))! enddo! write(*,*)"Processor", my_id, "has nished ! +setting initial conditions"! ************** Iterations ****************************************! do itime = 1 , 3 ! if (my_id.eq.0) then! write(*,*)"Running Iteration Number ", itime! endif! do ilocal = 1, 3! T(itime,ilocal)=T(itime-1,ilocal)+! + xalp*delt/delx/delx*! + (T(itime-1,ilocal-1)-2*T(itime-1,ilocal)+T(itime-1,ilocal+1))! enddo! if (my_id.eq.0) then! write(*,*)"Sending and receiving overlap points"! dest = 1! msg_size = 1!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Fortran MPI Code: 1-D Heat Equation (Contd.) !

call MPI_SEND(T(itime,3),msg_size,MPI_DOUBLE_PRECISION,dest,! + tag,MPI_COMM_WORLD,ierr)! endif ! if (my_id.eq.1) then! from = 0! dest = 2! msg_size = 1! call MPI_RECV(T(itime,0),msg_size,MPI_DOUBLE_PRECISION,from,! + tag,MPI_COMM_WORLD,status,ierr)! call MPI_SEND(T(itime,3),msg_size,MPI_DOUBLE_PRECISION,dest,! + tag,MPI_COMM_WORLD,ierr)! endif! if (my_id.eq.2) then! from = 1 ! dest = 1 ! msg_size = 1! call MPI_RECV(T(itime,0),msg_size,MPI_DOUBLE_PRECISION,from,! + tag,MPI_COMM_WORLD,status,ierr)! call MPI_SEND(T(itime,1),msg_size,MPI_DOUBLE_PRECISION,dest,! + tag,MPI_COMM_WORLD,ierr)! endif! if (my_id.eq.1) then! from = 2 ! dest = 0 ! msg_size = 1! call MPI_RECV(T(itime,4),msg_size,MPI_DOUBLE_PRECISION,from,! + tag,MPI_COMM_WORLD,status,ierr)! call MPI_SEND(T(itime,1),msg_size,MPI_DOUBLE_PRECISION,dest,! + tag,MPI_COMM_WORLD,ierr)! endif! if (my_id.eq.0) then! from = 1 ! msg_size = 1! call MPI_RECV(T(itime,4),msg_size,MPI_DOUBLE_PRECISION,from,! + tag,MPI_COMM_WORLD,status,ierr)! endif! enddo! if (my_id.eq.0) then! write(*,*)"SOLUTION SENT TO FILE AFTER 3 TIMESTEPS:"! endif! FILEN = 'data'//char(my_id+48)//'.dat'! open (5, le=FILEN) ! write(5,*)"Processor ",my_id! do ilocal = 0 , 4! iglobal = 3*my_id + ilocal! write(5,*)"ilocal=",ilocal,";iglobal=",iglobal,";T=",T(3,ilocal)! enddo ! close(5)! call MPI_FINALIZE(ierr)! END! at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

SAN DIEGO SUPERCOMPUTER CENTER

Simple Application using MPI: 1-D Heat Equation !

Processor 0 Processor 2

5
Processor 1

10

"Compilation of MPI Version (on Steele):! "module load mpich2-intel! "mpif90 -o heat_mpi xed heat_mpi.f90! "Run script (on Steele):! "Script name : heat.sub! "Submit using:! qsub q R7160158 heat.sub!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple Application using MPI: 1-D Heat Equation !

Processor 0 Processor 2

5
Processor 1

10

OUTPUT FROM SAMPLE PROGRAM!

Process 0 of 3 has started! Processor 0 has nished setting initial conditions! Process 1 of 3 has started! Processor 1 has nished setting initial conditions! Process 2 of 3 has started! Processor 2 has nished setting initial conditions! Running Iteration Number 1! Sending and receiving overlap points! Running Iteration Number 2! Sending and receiving overlap points! Running Iteration Number 3! Sending and receiving overlap points! SOLUTION SENT TO FILE AFTER 3 TIMESTEPS:!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Simple Application using MPI: 1-D Heat Equation !

Processor 0 Processor 2

5
Processor 1

10

ds100 % more data0.dat ! Processor 0! ilocal= 0 ;iglobal= 0 ;T= 0.000000000000000000E+00! ilocal= 1 ;iglobal= 1 ;T= 0.307205621017284991! ilocal= 2 ;iglobal= 2 ;T= 0.584339815421976549! ilocal= 3 ;iglobal= 3 ;T= 0.804274757358271253! ilocal= 4 ;iglobal= 4 ;T= 0.945481682332597884!

ds100 % more data2.dat! Processor 2! ilocal= 0 ;iglobal= 6 ;T= 0.945481682332597995! ilocal= 1 ;iglobal= 7 ;T= 0.804274757358271253! ilocal= 2 ;iglobal= 8 ;T= 0.584339815421976660! ilocal= 3 ;iglobal= 9 ;T= 0.307205621017285102! ilocal= 4 ;iglobal= 10 ;T= 0.000000000000000000E+00!

ds100 % more data1.dat! Processor 1! ilocal= 0 ;iglobal= 3 ;T= 0.804274757358271253! ilocal= 1 ;iglobal= 4 ;T= 0.945481682332597884! ilocal= 2 ;iglobal= 5 ;T= 0.994138272681972301! ilocal= 3 ;iglobal= 6 ;T= 0.945481682332597995! ilocal= 4 ;iglobal= 7 ;T= 0.804274757358271253!
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

Fortran OpenMP Code: 1-D Heat Equation !

PROGRAM HEATEQN! implicit none! integer :: iglobal, itime, nthreads! real*8 :: xalp,delx,delt,pi! real*8 :: T(0:100,0:10)! integer:: id! integer:: OMP_GET_THREAD_NUM, OMP_GET_NUM_THREADS! !$OMP PARALLEL SHARED(nthreads)! !$OMP MASTER! nthreads = omp_get_num_threads()! write (*,*) 'There are', nthreads, 'threads'! !$OMP END MASTER! !$OMP END PARALLEL! if (nthreads.ne.3) then! write(*,*)"Use exactly 3 threads for this case"! stop! endif! delx = 0.1d0! delt = 1d-4! xalp = 2.0d0! ************** Initial Conditions *********************************! pi = 4d0*datan(1d0)! do iglobal = 0, 10! T(0,iglobal) = dsin(pi*delx*doat(iglobal))! enddo! ************** Iterations ****************************************! do itime = 1 , 3 ! write(*,*)"Running Iteration Number ", itime! !$OMP PARALLEL DO PRIVATE(iglobal) SHARED (T,xalp,delx,delt,itime)! do iglobal = 1, 9 ! T(itime,iglobal)=T(itime-1,iglobal)+! + xalp*delt/delx/delx*! + (T(itime-1,iglobal-1)-2*T(itime-1,iglobal)+T (itime-1,iglobal+1))! enddo! !$OMP BARRIER! enddo! do iglobal = 0, 10! write(*,*)iglobal,T(3, iglobal)! enddo! END!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

OpenMP result: 1-D Heat Equation !

setenv OMP_NUM_THREADS 3! ./heat_omp! There are 3 threads! Running Iteration Number 1! Running Iteration Number 2! Running Iteration Number 3! 0 0.000000000000000E+000! 1 0.307205621017285 ! 2 0.584339815421977 ! 3 0.804274757358271 ! 4 0.945481682332598 ! 5 0.994138272681972 ! 6 0.945481682332598 ! 7 0.804274757358271 ! 8 0.584339815421977 ! 9 0.307205621017285 ! 10 0.000000000000000E+000!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO

References !
" TeraGrid user support website!
" https://www.teragrid.org/web/user-support/home!

" OpenMP Tutorials:!

" https://computing.llnl.gov/tutorials/openMP/! " http://www.nersc.gov/nusers/help/tutorials/openmp/!

" MPI Tutorials:!

" https://computing.llnl.gov/tutorials/mpi/!

" Lot of info available online Explore!!

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO