Parallel Algorithms and Optimization: Introduction

Mobile Robotics Reading Group Timothy Chung and Vijay Gupta 08 April 2004

Parallel Algorithms

With the increase in computation capability and the complexity of the problems of interest, we need to come up with methods that will take advantage of existing structure in the problems. Parallel algorithms are based on the principle that a problem can be decomposed into independent parts, so that each part can be processed in parallel.

1.1

What is a Parallel Algorithm?

A Parallel algorithm is any algorithm which can be implemented in a parallel manner which results in an increase in performance as compared to an implementation of a serial or sequential algorithm for the solution of the same problem. 1.1.1 Implementation in a parallel manner

It is important to understand what kinds of algorithms can be implemented in a parallel manner. An algorithm may be parallelizable due to its mathematical structure or due to the specic characteristics of the problem being solved. Classication of parallel algorithms fall along these lines. Some problems, such as solving a system of equations, possess iterative structure: xi (t + 1) = fi (x1 (t), . . . , xn (t)), i = 1, . . . , n

We let each of n processors examine each component of x, and iterate until we reach a xed point of the system f , namely, where x = f (x ). Note that this assumes that all nodes have information about all components at time t. NOTE: Gauss-Seidel Iterations Other problems depend on the structure, such as block-separable problems. A function F is block separable if it can be written as the sum of functions of subsets of its variables. In other 1

words, for a vector x = (x1 )T , (x2 )T , . . . , (xK )T written as

, F : RnK R is block separable if it can be fk : R n R

F (x) =
k=1

fk (xk ),

NOTE: Separation principle in control theory between control and estimation.

1.1.2

Increase in performance

This begs the question of how we measure performance to begin with, and what it means to improve it. There are many dierent metrics for evaluating the performance of Parallel algorithms: FLOPS : oating-point operations per second executed Speedup : ratio of the solution time for a problem with the fastest serial code on a single processor to the solution time with a parallel algorithm on multiple processors Relative Speedup : ratio of solution time of a parallel algorithm running on a single processor with the solution time of it running on multiple processors Absolute Speedup : ratio of solution time of fastest serial code on fastest serial computer to the solution time of the parallel algorithm on a parallel computer. Eciency : ratio of relative speedup to the number of processors Scalability : the ability of a parallel algorithm to maintain a constant level of eciency as the number of processors increases as the size of the problem also increases Mostly, were interested in speedup, eciency and scalability of our parallel algorithms. For example, given a sensor network with N sensing nodes, and we want to fuse the measurements taken at these nodes. Some questions: 1. Does our parallel algorithm work better than sending all our measurements to a central node, which fuses all of the measurements using a centralized algorithm? 2. What happens when we lose some of the nodes? How robust is our algorithm to the eects of the number of processors? 3. How do the results of the algorithm vary as we increase the number of sensor nodes?

1.2
1.2.1

Why wouldnt we do everything with parallel algorithms?

Now that we have multiple nodes carrying out the computation instead of just a single central node, obviously there needs to be a communication network among the multiple processors. 2

1.2.2

Task Allocation Issues

The problem of deciding which node does what component of the task is another complicating issue for parallel computation. Some sort of coordinated partitioning of the task into independent components must take place to ensure that each processor can proceed independently and concurrently. 1.2.3 Synchronization Issues

Firstly, for synchronous operations, there must exist a common global clock. Since the behavior of the multiple processors is easier to predict/control if they are synchronized, the design of a parallel algorithm may be simplied. Further, properties of algorithms such as convergence are more easily proved for synchronous algorithms. However, sometimes the task of synchronizing causes signicant overhead, and other times, synchronization is virtually impossible (e.g. data communication networks). 1.2.4 Complexity Issues

Obviously, all of the above issues increase the complexity of the problem. Needing to account for communication, task allocation/scheduling, and synchronization of the multiple processors adds an extra layer on top of what may already be a complex task. The question is that of trade-o. Are the advantages of parallel computation worth the penalties it will impose? This question is the essence of why were investigating this topic, to see whether or not there are applications where the advantages far outweigh the disadvantages that are relevant to our research.

Next Time...

Well investigate some more distributed algorithms, using a more thorough examination of an example.

References
[1] D.P. Bertsekas and J.N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. [2] Y. Censor and S.A. Zenios. Parallel Optimization: Theory, Algorithms, and Applications. [3] M. Raynal. Distributed Algorithms and Protocols.