See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/242691691

Sparse Matrix Technology

Article · January 1984

CITATIONS READS
244 2,268

1 author:

Sergio Pissanetzky
University of Houston - Clear Lake
78 PUBLICATIONS   688 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Artificial Intelligence View project

All content following this page was uploaded by Sergio Pissanetzky on 10 August 2014.

The user has requested enhancement of the downloaded file.

SPARSE MATRIX
TECHNOLOGY
electronic edition

Sergio Pissanetzky
Sparse Matrix Technology
electronic edition

Sergio Pissanetzky
Copyright c 2007 by Sergio Pissanetzky and SciControls.com. All rights reserved. No
part of the contents of this book can be reproduced without the written permission of the
publisher.

Professionally typeset by LATEX. This work is in compliance with the mathematical typeset-
ting conventions established by the International Organization for Standardization (ISO).

Dr. Pissanetzky retired after a rewarding career as an Entrepreneur, Professor, Research Scientist
and Consultant. He was the founder of Magnus Software Corporation, where he focused on develop-
ment of specialized applications for the Magnetic Resonance Imaging (MRI) and the High Energy
Particle Accelerator industries. He has served as Member of the International Editorial Board of
the “International Journal for Computation in Electrical and Electronic Engineering”, as a Member
of the International Advisory Committee of the International Journal “Métodos Numéricos para
Cálculo y Diseño en Ingenierı́a”, and as a member of the International Committee for Nuclear Res-
onance Spectroscopy, Tokyo, Japan. Dr. Pissanetzky has held professorships in Physics at Texas
A&M University and the Universities of Buenos Aires, Córdoba and Cuyo, Argentina. He has also
held positions as a Research Scientist with the Houston Advanced Research Center, as Chairman of
the Computer Center of the Atomic Energy Commission, San Carlos de Bariloche, Argentina, and
as a Scientific Consultant at Brookhaven National Laboratory. Dr. Pissanetzky holds several US
and European patents and is the author of two books and numerous peer reviewed technical papers.
Dr. Pissanetzky earned his Ph.D. in Physics at the Balseiro Institute, University of Cuyo, in 1965.
Dr. Pissanetzky has 35 years of teaching experience and 30 years of programming experience in
languages such as Fortran, Basic, C and C++. Dr. Pissanetzky now lives in a quite suburban
neighborhood in Texas.

Website: http://www.SciControls.com

ISBN 978-0-9762775-3-8
iv
Contents

Preface xiii

Introduction 1

1 Fundamentals 5
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Storage of arrays, lists, stacks and queues . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Storage of lists of integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Representation and storage of graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Diagonal storage of band matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 Envelope storage of symmetric matrices . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.7 Linked sparse storage schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.8 The sparse row-wise format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.9 Ordered and unordered representations . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.10 Sherman’s compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.11 Storage of block-partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.12 Symbolic processing and dynamic storage schemes . . . . . . . . . . . . . . . . . . . 26
1.13 Merging sparse lists of integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.14 The multiple switch technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.15 Addition of sparse vectors with the help of an expanded real accumulator . . . . . . 30
1.16 Addition of sparse vectors with the help of an expanded integer array of pointers . . 32
1.17 Scalar product of two sparse vectors with the help of an array of pointers . . . . . . 33

2 Linear Algebraic Equations 35

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2 Some definitions and properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3 Elementary matrices and triangular matrices . . . . . . . . . . . . . . . . . . . . . . 40
2.4 Some properties of elementary matrices . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.5 Some properties of triangular matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.6 Permutation matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

v
vi

2.7 Gauss elimination by columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.8 Gauss elimination by rows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.9 Gauss-Jordan elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.10 Relation between the elimination form of the inverse and the product form of the
inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.11 Cholesky factorization of a symmetric positive definite matrix . . . . . . . . . . . . . 53
2.12 Practical implementation of Cholesky factorization . . . . . . . . . . . . . . . . . . . 55
2.13 Forward and backward substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.14 Cost considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.15 Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

3 Numerical Errors in Gauss Elimination 63

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2 Numerical errors in floating point operations . . . . . . . . . . . . . . . . . . . . . . 65
3.3 Numerical errors in sparse factorization . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4 Numerical errors in sparse substitution . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5 The control of numerical errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6 Numerical stability and pivot selection . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.7 Monitoring or estimating element growth . . . . . . . . . . . . . . . . . . . . . . . . 82
3.8 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4 Ordering for Gauss Elimination: Symmetric Matrices 85

4.1 Introduction: Statement of the problem . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2 Basic notions of graph theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3 Breadth-first search and adjacency level structures . . . . . . . . . . . . . . . . . . . 93
4.4 Finding a pseudoperipheral vertex and a narrow level structure of a graph . . . . . . 95
4.5 Reducing the bandwidth of a symmetric matrix . . . . . . . . . . . . . . . . . . . . . 96
4.6 Reducing the profile of a symmetric matrix . . . . . . . . . . . . . . . . . . . . . . . 98
4.7 Graph-theoretical background of symmetric Gauss elimination . . . . . . . . . . . . 101
4.8 The minimum degree algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.9 Tree partitioning of a symmetric sparse matrix . . . . . . . . . . . . . . . . . . . . . 109
4.10 Nested dissection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.11 Properties of nested dissection orderings . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.12 Generalized nested dissection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.13 One-way dissection of finite element problems . . . . . . . . . . . . . . . . . . . . . . 122
4.14 Orderings for the finite element method . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.15 Depth-first search of an undirected graph . . . . . . . . . . . . . . . . . . . . . . . . 132
4.16 Lexicographic search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.17 Symmetric indefinite matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
 vii

5 Ordering for Gauss Elimination: General Matrices 143

5.1 Introduction: Statement of the problem . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.2 Graph theory for unsymmetric matrices . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.3 The strong components of a digraph . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
5.4 Depth-first search of a digraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.5 Breadth-first search of a digraph and directed
adjacency level structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.6 Finding a maximal set of vertex disjoint paths in an acyclic digraph . . . . . . . . . 157
5.7 Finding a transversal: the algorithm of Hall . . . . . . . . . . . . . . . . . . . . . . . 158
5.8 Finding a transversal: the algorithm of Hopcroft and Karp . . . . . . . . . . . . . . 161
5.9 The algorithm of Sargent and Westerberg
for finding the strong components of a digraph . . . . . . . . . . . . . . . . . . . . . 167
5.10 The algorithm of Tarjan for finding the strong components of a digraph . . . . . . . 168
5.11 Pivoting strategies for unsymmetric matrices . . . . . . . . . . . . . . . . . . . . . . 172
5.12 Other methods and available software . . . . . . . . . . . . . . . . . . . . . . . . . . 175

6 Sparse Eigenanalysis 177

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.2 The Rayleigh quotient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
6.3 Bounds for eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
6.4 The bisection method for eigenvalue calculations . . . . . . . . . . . . . . . . . . . . 184
6.5 Reduction of a general matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.6 Reduction of a symmetric band matrix to tridiagonal form . . . . . . . . . . . . . . . 188
6.7 Eigenanalysis of tridiagonal and Hessenberg matrices . . . . . . . . . . . . . . . . . . 189
6.8 Direct and inverse iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.9 Subspaces and invariant subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
6.10 Simultaneous iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.11 Lanczos algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
6.12 Lanczos algorithm in practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
6.13 Block Lanczos and band Lanczos algorithms . . . . . . . . . . . . . . . . . . . . . . . 206
6.14 Trace minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
6.15 Eigenanalysis of hermitian matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.16 Unsymmetric eigenproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

7 Sparse Matrix Algebra 211

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
7.2 Transposition of a sparse matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.3 Algorithm for the transposition of a general sparse matrix . . . . . . . . . . . . . . . 215
7.4 Ordering a sparse representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.5 Permutation of rows or columns of a sparse matrix: First procedure . . . . . . . . . 217
viii

7.6 Permutation of rows or columns of a sparse matrix: Second procedure . . . . . . . . 218

7.7 Ordering of the upper representation of a sparse symmetric matrix . . . . . . . . . . 218
7.8 Addition of sparse matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
7.9 Example of addition of two sparse matrices . . . . . . . . . . . . . . . . . . . . . . . 220
7.10 Algorithm for the symbolic addition of two sparse matrices with N rows and M columns222
7.11 Algorithm for the numerical addition of two sparse matrices with N rows . . . . . . 223
7.12 Product of a general sparse matrix by a column vector . . . . . . . . . . . . . . . . . 224
7.13 Algorithm for the product of a general sparse matrix by a full column vector . . . . 225
7.14 Product of a row vector by a general sparse matrix. . . . . . . . . . . . . . . . . . . 226
7.15 Example of product of a full row vector by a general sparse matrix. . . . . . . . . . . 226
7.16 Algorithm for the product of a full row vector by a general sparse matrix. . . . . . . 227
7.17 Product of a symmetric sparse matrix by a column vector. . . . . . . . . . . . . . . . 228
7.18 Algorithm for the product of a symmetric sparse matrix by a full column vector . . . 229
7.19 Multiplication of sparse matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
7.20 Example of product of two matrices which are stored by rows. . . . . . . . . . . . . . 231
7.21 Algorithm for the symbolic multiplication of two sparse matrices given in row-wise
format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7.22 Algorithm for the numerical multiplication of two sparse matrices given in row-wise
format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
7.23 Triangular factorization of a sparse symmetric matrix given in row-wise format . . . 235
7.24 Numerical triangular factorization of a sparse symmetric matrix given in row-wise
format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
7.25 Algorithm for the symbolic triangular factorization of a symmetric sparse matrix A . 240
7.26 Algorithm for the numerical triangular factorization of a symmetric positive definite
sparse matrix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
7.27 Example of forward and backward substitution . . . . . . . . . . . . . . . . . . . . . 245
7.28 Algorithm for the solution of the system UT DUx = b . . . . . . . . . . . . . . . . . 246

8 Connectivity and Nodal Assembly 249

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
8.2 Boundary conditions for scalar problems . . . . . . . . . . . . . . . . . . . . . . . . . 251
8.3 Boundary conditions for vector problems . . . . . . . . . . . . . . . . . . . . . . . . . 252
8.4 Example of a connectivity matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
8.5 Example of a nodal assembly matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.6 Algorithm for the symbolic assembly of a symmetric nodal assembly matrix . . . . . 259
8.7 Algorithm for the numerical assembly of an element matrix and vector into the nodal
assembly matrix A and right-hand vector b: Symmetric case . . . . . . . . . . . . . . 261
8.8 Algorithm for the numerical assembly of an element matrix and vector into the nodal
assembly matrix A and right-hand vector b: General case . . . . . . . . . . . . . . . 264
 ix

9 General Purpose Algorithms 267

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
9.2 Multiplication of the inverse of a lower triangular matrix by a general matrix . . . . 268
9.3 Algorithm for the symbolic multiplication of the inverse of a lower triangular matrix
U−T by a general matrix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
9.4 Algorithm for the numerical multiplication of the inverse of a lower triangular matrix
U−T by a general matrix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.5 Algorithm for the multiplication of the inverse of an upper triangular unit diagonal
matrix U by a full vector x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
9.6 Algorithm for the multiplication of the transpose inverse of an upper triangular unit
diagonal matrix U by a full vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
9.7 Solution of linear equations by the Gauss-Seidel iterative method . . . . . . . . . . . 274
9.8 Algorithm for the iterative solution of linear equations by the Gauss-Seidel method 275
9.9 Checking the representation of a sparse matrix . . . . . . . . . . . . . . . . . . . . . 276
9.10 Printing and displaying a sparse matrix . . . . . . . . . . . . . . . . . . . . . . . . . 277
9.11 Algorithm for transforming a RR(C)U of a symmetric matrix into a RR(U)U of the
same matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.12 Algorithm for the pre-multiplication of a sparse matrix A by a diagonal matrix D . . 279
9.13 Algorithm for copying a sparse matrix from IA, JA, AN to IB, JB, BN . . . . . . . 279

Bibliography and Index 281

x
Preface to the Electronic Edition

This is an electronic edition of the classic book Sparse Matrix Technology by Sergio Pissanetzky,
originally published in English by Academic Press, London, in 1984, and later translated into
Russian and published by MIR, Moscow, in 1988. The electronic edition has been typed from the
original, with only minor changes of format where dictated by electronics.

xi
xii
Preface

As computers grow in power and speed, matrices grow in size. In 1968, practical production
calculations with linear algebraic systems of order 5,000 were commonplace, while a “large” system
was one of order 10,000 or more.a
In 1978, an over determined problem with 2.5 million equations in 400,000 unknowns was
reported;b in 1981, the magnitude of the same problem had grown: it had 6,000,000 equations, still
in 400,000 unknowns.c The matrix of coefficients had 2.4 × 1012 entries, most of which were zero: it
was a sparse matrix. A similar trend toward increasing size is observed in eigenvalue calculations,
where a “large” matrix is one of order 4,900 or 12,000.d Will matrix problems continue to grow
even further? Will our ability to solve them increase at a sufficiently high rate?
But this is only one side of the question. The other side concerns the microcomputer explosion.
Microcomputers now have about the same power as large computers had two decades ago. Are
users constrained to solving matrix problems of the same size as those of twenty years ago?
The owner of a microcomputer may not care too much about the cost of computation; the main
difficulty is storage. On a large machine, the cost of solving a matrix problem increases rapidly
if the size of the problem does, because both storage and labor grow. The overall cost becomes a
primary consideration. How can such cost be minimized for a given problem and installation?
Answers to these and other related questions are given in this book for the following classes of
matrix problems: direct solution of sparse linear algebraic equations, solution of sparse standard
and generalized eigenvalue problems, and sparse matrix algebra. Methods are described which range
from very simple yet surprisingly effective ideas to highly sophisticated algorithms. Sparse matrix
technology is now a well established discipline, which was defined as “the art of handling sparse
matrices”.e It is composed of a beautiful blend of theoretical developments, numerical experience
and practical considerations. It is not only an important computational tool in a broad spectrum

a
In the Preface, the pertinent references are given as footnotes, because this enhances clarity. The full
list of references is given at the end of the book. Tinney, 1969,237 p.28; Willoughby, 1971,250 p.271.
b
Kolata, 1978.144
c
Golub and Plemmons, 1981, 106 p.3.
d
Cullum and Willoughby, 1981,42 p. 329; Parlett, 1980,175 p. XIII.
e
Harary, 1971.124

xiii
xiv

of computational areas,f but also is in itself a valuable contribution to the general development
of computer software. The new ideas developed during the last fifteen years were used to devise
nearly optimum algorithms for a variety of matrix problems. Research in the field is currently very
active and the spectrum of applications broadens continuously. Sparse matrix technology is here
and will stay.
The concept expressing the nature of our concern is contained in the title of the book. Tech-
nology is applied science, the science or study of the practical or industrial arts.g The phrase
“sparse matrix technology” was an everyday saying in the early nineteen seventies at the IBM T. J.
Watson Research Center.h Nowadays it seems to be in desuetude. The material for the book was
selected from the several Symposia and Congresses on large matrix problems regularly held since
1968.i Major sources of inspiration were: an advanced course with four review articles,j excellent
survey articlesk and books,l a collection of papers,m and many publications which are cited where
pertinent. Several basic ideas can be found in the literature published before 1973.n No attempt is
made, however, to cover such an important amount of material. Rather, the fundamental methods
and procedures are introduced and described in detail, the discussion reaching the point where the
reader can understand the specialized literature on each subject. A unified treatment is provided
whenever possible, although, like any field of human knowledge which grows fast, sparse matrix
technology has grown unevenly. Some areas are well developed, while other areas lack further
research. We have not included proofs of all the theorems, except when they are closely related to
practical techniques which are used subsequently. The concepts and methods are introduced at an
elementary level, in many cases with the help of simple examples. Many fundamental algorithms
are described and carefully discussed. Ready-to-use very efficient and professional algorithms are
given in Fortran. The reader is assumed to be familiar with this popular language. The algorithms,
however, are explained so clearly that even a person with a limited knowledge of Fortran can under-
stand them and eventually translate them into other languages. Linear algebra and graph theory
are used extensively in the book. No particular acquaintance with these subjects is necessary be-
cause all definitions and properties are introduced from the beginning, although some preparation
f
Rose and Willoughby, 1972.198
g
Webster’s Dictionary, second edition, 1957.
h
Willoughby, 1971;250 Rose and Willoughby, 1972,198 Preface; Willoughby, 1972;251 Hachtel, 1976,117 p.
349.
i
Willoughby, 1969;249 Reid, 1971a;187 Rose and Willoughby, 1972;198 Bunch and Rose, 1976;28 Duff and
Stewart, 1979;68 Duff, 1981b.61 The Proceedings of the Symposium held at Fairfield Glade, Tennessee, in
1982, will be published as a special issue of the SIAM Journal on Scientific and Statistical Computing, and
possibly other SIAM journals, to appear in 1983. The Software Catalog prepared in conjunction with the
Symposium is available (Heath, 1982.126)
j
Barker, 1977.10
k
Duff, 1977,55 1982.62
l
Wilkinson, 1965;247 Parlett, 1980;175 George and Liu, 1981.97
m
Björck et al. 1981 16
n
Brayton et al. 1970; 19 Willoughby, 1972;251 Tewarson, 1973.235
 xv

may be helpful. An extensive bibliography and a survey of the relevant literature are included in
many sections. The book fills the gap between books on the design of computer algorithms and
specialized literature on sparse matrix techniques, on the one side, and user needs and application
oriented requirements on the other.
The purpose of the book is to bring sparse matrix technology within reach of engineers, pro-
grammers, analysts, teachers and students. This book will be found helpful by everyone who wishes
to develop his own sparse matrix software, or who is using it and wishes to understand better how
it operates, or who is planning to acquire a sparse matrix package and wishes to improve his under-
standing of the subject. Teachers who need an elementary presentation of sparse matrix methods
and ideas and many examples of application at a professional level, will find such material in this
book.
Chapter 1 covers all fundamental material such as storage schemes, basic definitions and com-
putational techniques needed for sparse matrix technology. It is very convenient to read at least
Sections 1 to 9 and Section 12 of Chapter 1 first. The first reading may, however, be superficial.
The reader will feel motivated to examine this material in more detail while reading other chapters
of the book, where numerous references to sections of Chapter 1 are found.
Chapters 2 to 5 deal with the solution of linear algebraic equations. They are not independent.
The material in Chapter 2 is rather elementary, but its form of presentation serves as an introduction
for Chapters 4 and 5, which contain the important material. Chapter 3 deals with numerical errors
in the case where the linear system is sparse, and also serves as an introduction to Chapters 4
and 5. This material is not standard in the literature. Sparse matrix methods and algorithms for
the direct solution of linear equations are presented in Chapters 4 and 5. Chapter 4 deals with
symmetric matrices, and Chapter 5 with general matrices.
The calculation of eigenvalues and eigenvectors of a sparse matrix, or of a pair of sparse matrices
in the case of a generalized eigenvalue problem, is discussed in Chapter 6. Chapter 6 can be read
independently, except that some references are made to material in Chapters 1 and 7.
Chapters 7, 8 and 9 deal with sparse matrices stored in row-wise format. Algorithms for
algebraic operations, triangular factorization and back substitution are explicitly given in Fortran
and carefully discussed in Chapter 7. The material in Chapter 1 is a prerequisite, particularly
Sections 8, 9 and 10 and 12 to 17. In addition, Chapter 2 is a prerequisite for Sections 23 to
28 of Chapter 7. Chapter 8 covers the sparse matrix techniques associated with mesh problems,
in particular with the finite element method, and in Chapter 9 we present some general purpose
Fortran algorithms.
Sparse matrix technology has been applied to almost every area where matrices are employed.
Anyone interested in a particular application may find it helpful to read the literature where
the application is described in detail, in addition to the relevant chapters of this book. A list
of bibliographical references sorted by application was publishedo and many papers describing a

o
Duff, 1977,55 p. 501.
xvi

variety of applications can be found in the Proceedings of the 1980 IMA Conferencep and in other
publicationsq
Good, robust sparse matrix software is now commercially available. The Sparse Matrix Software
Catalogr lists more than 120 programs. Many subroutines are described in the Harwell Catalogues
and two surveys have also been published.t Producing a good piece of sparse matrix software
is not an easy task. It requires expert programming skills. As in any field of engineering, the
software designer must build a prototype, test it carefullyu and improve it before the final product
is obtained and mass production starts. In software engineering, mass production is equivalent
to obtaining multiple copies of a program and implementing them in many different installations.
This requires transportability. From the point of view of the user, the software engineer must
assume responsibility for choosing the right program and file structures and installing them into
the computer. For the user, the product is not the program but the result. The desirable attributes
of a good program are not easily achieved.v In this book, the characteristics and availability of
software for each particular application are discussed in the corresponding sections.
I would like to acknowledge the collaboration of Neil Callwood. He has read the manuscript
several times, correcting many of my grammatical infelicities, and is responsible for the “British
flavour” that the reader may find in some passages. I would also like to acknowledge the patience
and dedication Mrs. Carlota R. Glücklich while typing the manuscript and coping with our revi-
sions.

Sergio Pissanetzky January 1984

[email protected]

p
Duff, 1981b.61
q
Bunch and Rose, 1976;28 Duff and Stewart, 1979. 68
r
Heath, 1982.126
s
Hopper, 1980.130
t
Duff, 1982;62 Parlett, 1983. 176
u
Duff, 1979;57 Eisenstat et al. 1979;75 Duff et al. 1982.69
v
Gentleman and George, 1976;87 Silvester, 1980.215
FUNDAMENTALS 19
1 2 3 4 5
| A11
1 A13 A14 |
| 2 A22 A25 |
A = 3| A33 A35 |
4| A44 |
5 | symmetric A55 |

head 1 A11 0 • A13 0 • A14 0 0

head 2 A22 0 • A25 0 0

head 3 A33 • • A35 • 0

head 4 A44 • 0

head 5 A55 • 0

Figure 1.4: Larcombe’s version of Knuth’s storage scheme for symmetric matrices with
no zero elements on the diagonal.

1.8 The sparse row-wise format

The sparse row-wise format (Chang, 1969;29 Curtis and Reid, 1971b;46 Gustavson, 1972112) to
be described here is one of the most commonly used storage schemes for sparse matrices. The
scheme has minimal storage requirements and at the same time it has proved to be very convenient
for several important operations such as addition, multiplication, permutation and transposition
of sparse matrices, the solution of linear equations with a sparse matrix of coefficients by either
direct or iterative methods, etc. In this scheme, the values of the nonzero elements of the matrix
are stored by rows, along with their corresponding column indices, in two arrays, say AN and JA,
respectively. An array of pointers, say IA, is also provided to indicate the locations in AN and
JA where the description of each row begins. An extra entry in IA contains a pointer to the first
empty position in JA and AN. An example is convenient at this point. Consider the matrix:

1 2 3 4 5 6 7 8 9 10
1 | 0 0 1. 3. 0 0 0 5. 0 0 |
A= 2 | 0 0 0 0 0 0 0 0 0 0 |
3 | 0 0 0 0 0 7. 0 1. 0 0 |
20 SPARSE MATRIX TECHNOLOGY

A is represented as follows:
1 2 3 4 5 6
IA = 1 4 4 6
JA = 3 4 8 6 8 RR(C)O
AN = 1. 3. 5. 7. 1.

The description of row 1 of A begins at the position IA(1) = 1 of AN and JA. Since the description
of row 2 begins at IA(2) = 4, this means that row 1 of A is described in positions 1, 2 and 3 of AN
and JA. In this example:
IA(1) = 1 first row begins at JA(1) and AN(1).
IA(2) = 4 second row begins at JA(4) and AN(4)
IA(3) = 4 third row begins at JA(4) and AN(4). Since this is the same position at
which row 2 begins, this means that row 2 is empty.
IA(4) = 6 this is the first empty location in JA and AN. The description of row 3
thus ends at position 6 − 1 = 5 of JA and AN.
In general, row r of A is described in positions IA(r) to IA(r + 1) − 1 of JA and AN, except when
IA(r + 1) = IA(r) in which case row r is empty. If matrix A has m rows, then IA has m + 1
positions.
This representation is said to be complete because the entire matrix A is represented, and
ordered because the elements of each row are stored in the ascending order of their column indices.
It is thus a Row-wise Representation Complete and Ordered, or RR(C)O.
The arrays IA and JA represent the structure of A, given as the set of the adjacency lists of
the graph associated with A. If an algorithm is divided into a symbolic section and a numerical
section (Section 1.12), the arrays IA and JA are computed by the symbolic section, and the array
AN by the numerical section.
Gustavson (1972)112 also proposed a variant of row-wise storage, suitable for applications re-
quiring both row and column operations. A is stored row-wise as described, and in addition the
structure of AT is computed and also stored row-wise. A row-wise representation of the structure
of AT is identical to a column-wise representation of the structure of A. It can be obtained by
transposition of the row-wise structure of A (Chapter 7). This scheme has been used, for example,
for linear programming applications (Reid, 1976).189
A much simpler row-oriented scheme was proposed by Key (1973)141 for unsymmetric matrices.
The nonzeros are held in a two-dimensional array of size n by m, where n is the order of the matrix
and m the maximum number of nonzeros in a row. This scheme is easy to manipulate but has the
disadvantage that m may not be predictable and may turn out to be large.

1.9 Ordered and unordered representations

Sparse matrix representations do not necessarily have to be ordered, in the sense that the elements
of each row can be stored in any order while still preserving the order of the rows. The matrix A
LINEAR ALGEBRAIC EQUATIONS 51

The kth step consists of the elimination of the nonzeros on column k of A(k) both above and below
the diagonal. Row k is first normalized by dividing all its elements by the diagonal element. Then,
convenient multiples of the normalized row k are subtracted from all those rows which have a
nonzero in column k either above or below the diagonal. The matrix A(k+1) is thus obtained with
zeros in its k initial columns. This process is continued until, at the end of step n, the identity
matrix A(n+1) ≡ I is obtained. The kth step of Gauss-Jordan elimination by columns is equivalent
to pre-multiplication of A(k) by D−1
k and by the complete column elementary matrix (Tk ) :
C −1

−1 −1 (k)
A(k+1) = (TC
k ) Dk A (2.37)

where A(1) ≡ A and:

(k)
(Dk )kk = Akk (2.38)
(k)
(TkC )ik = Aik for all i 6= k

Thus, we have:
−1 −1 C −1 −1 C −1 −1
(TC
n ) Dn . . . (T2 ) D2 (T1 ) D1 A = I. (2.39)

The factorized form of A is:

A = D1 TC C C
1 D2 T2 . . . Dn Tn , (2.40)

and the product form of the inverse in terms of column matrices is:
C −1 −1 C −1 −1
A−1 = (TC −1 −1
n ) Dn . . . (T2 ) D2 (T1 ) D1 . (2.41)

The close relationship between this expression and the elimination form of the inverse, Expression
(2.24), will be discussed in Section 2.10. The results of the elimination are usually recorded as a
table of factors:

(D1 )−1
11 (T2C )12 (T3C )13 . . .
(T1C )21 (D2 )−1
22 (T3C )23 (2.42)
(T1C )31 (T2C )32 (D3 )−1
33
.. .. ..
. . .
(k)
By Equation (2.38), this table is formed simply by leaving each off-diagonal Aik where it is
obtained. The diagonal is obtained, as in Gauss elimination, by storing the reciprocals of the
diagonal elements used to normalize each row. The lower triangle and diagonal of this table are
thus identical to those of the Gauss table. Expressions (2.40) and (2.41) indicate how to use the
table (2.42). When solving linear equations by means of x = A−1 b, Equation (2.41) is used, with
the matrices (TC −1 obtained from the table by reversing the signs of the off-diagonal elements of
k)
52 SPARSE MATRIX TECHNOLOGY

column k (Property 2.4(d)). The matrices D−1 k are directly available from the table. The product
of A with any matrix or vector can also be computed using the table, as indicated by Equation
(2.40).
Gauss-Jordan elimination can also be performed by rows. The version by columns requires
the addition of multiples of row k to all other rows in order to cancel the off-diagonal elements
of column k. This process can be understood conceptually as the construction of new equations
which are linear combinations of the original ones. On the other hand, in Gauss-Jordan elimination
by rows, we add multiples of column k to all other columns, in such a way that the off-diagonal
elements of row k become zero. This process can be viewed as the construction of new unknowns
which are linear combinations of the original ones and which satisfy linear equations with some
zero coefficients. Alternatively, we can forget about the system of linear equations and view the
row algorithm as the triangularization of AT , the transpose of A, by columns. Doing this, we obtain
the equivalent of Expression (2.41):
(AT )−1 = (T0C −1 0 −1
n ) (Dn ) . . . (T0C −1 0 −1 0C −1 0 −1
2 ) (D2 ) (T1 ) (D1 ) , (2.43)
which by transposition and using (AT )−1 = (A−1 )T yields:
A−1 = (D01 )−1 (T0R −1 0 −1 0R −1
1 ) (D2 ) (T2 ) . . . (D0n )−1 (T0R
n )
−1
(2.44)
Equation (2.44) is the product form of the inverse in terms of row matrices. The elimination by
rows is equivalent to multiplying A from the right by Expression (2.44). The nontrivial elements
of the matrices of Expression (2.44) are recorded as a table of factors in the usual way, and the
table can be used to solve linear equations or to multiply either A or A−1 by any matrix or vector.

2.10 Relation between the elimination form of the in-

verse and the product form of the inverse
From the preceding section it should be clear that Gauss-Jordan elimination by columns can be
performed equally well if we first eliminate all nonzeros from the lower triangle of A, and then all
nonzeros from the upper triangle of A. In fact, when we start at the upper left-hand corner of
A, we can eliminate lower and upper portions of columns in any order, provided only that upper
portions are eliminated in order, lower portions are also eliminated in order, and the upper portion
of any column k is eliminated after the lower portion of the preceding column. This statement
holds true due to the fact that a row k + 1 is obtained in final form immediately after the lower
portions of columns 1 to k have been eliminated and row k + 1 has been normalized; row k + 1
can then be used either immediately or at any later stage to eliminate the upper portion of column
k + 1, provided that the upper portions of columns 1 to k have been previously eliminated. These
facts can be stated formally using the properties of the elementary matrices (Section 2.4). We use
Property 2.4(c) to express TCk as follows:

TC C C
k = Lk Uk , (2.45)
74 SPARSE MATRIX TECHNOLOGY

Table 3.1. Bounds for the norms of L, expression for nij (see Equation 2.16), and
bounds for the norm of the error matrix E for the factorization LU = A + E, where all
matrices are of order n. The bandwidth of band matrices is assumed not to exceed n.

A Bounds for L nij Error bounds for factorization

m (k) Pn
maxj cL
P
Sparse kLk1 ≤ aM j +1 k=1 nij kEk1 ≤ 3.01εM aM maxj i=1 nij

Pn
kLk∞ ≤ aM maxi riL + 1 m = min(i, j) kEk∞ ≤ 3.01εM aM maxi j=1 nij
3.01
Full kLk1 ≤ aM n min(i, j) kEk1 , kEk∞ ≤ 2 εM aM n(n + 1)

kLk∞ ≤ aM n

Band kLk1 ≤ aM (β + 1) max[0, min(i, j, i − j + β + 1, kEk1 , kEk∞ ≤ 3.01εM aM (β + 1)2

|\β\β\| kLk∞ ≤ aM (β + 1) j − i + β + 1)]

Band kLk1 ≤ aM (β + 1) max[0, min(i, j, i − j + 2β + 1, kEk1 , kEk∞ ≤ 3.01εM aM (β + 1)

|\β\2β\| kLk∞ ≤ aM (β + 1) j − i + β + 1, β + 1)] ×(2β + 1)

Then, the computed result w satisfies the exact relation:

Lw = b + δb (3.52)

where, from Equations 3.47 and 3.50, the following bounds hold for the components of δb:

|δbi | ≤ 3.01εM bM i (riL + 1). (3.53)

A less tight but simpler bound is obtained if bM is the absolute value of the largest element of all
the vectors b(k) , so that bM i ≤ bM and:
(k)
|bi | ≤ bM ; i = 1, 2, . . . , n; k ≤ i. (3.54)

Then:

|δbi | ≤ 3.01εM bM (riL + 1). (3.55)

Backward substitution is the solution of Ux = w. It can be viewed as an algorithm with n steps,

where the sequence of vectors w(n) ≡ w, w(n−1) , . . . , w(2) , w(1) is computed, with w(k) and w(k−1)
having their components k to n identical. Step k, k = n, n − 1, . . . , 1, is:
(k)
xk = wk
(k−1) (k) (k)
wi = wi − Uik xk + gi ; i = 1, . . . , k − 1, (3.56)
NUMERICAL ERRORS IN GAUSS ELIMINATION 75

(k)
where gi is the error introduced by the floating point computation. The operations performed on
an element wi , i < n, are:
(n) (n) (n−1) (i+1)
wi − Uin xn + gi + gi − Ui,n−1 xn−1 + gi − . . . − Ui,i+1 xi+1 + gi = xi (3.57)

or:
n n
X (k) X
wi + gi = Uik xk ; i < n, (3.58)
k=i+1 k=i

Thus, if we define the error vector δw:

n
X (k)
δwi = gi ; i<n
k=i+1
δwn = 0, (3.59)

we have the following exact relation between the computed numbers:

Ux = w + δw. (3.60)
(k)
In order to obtain bounds for δw, we let wM i = maxk |wi |, so that:
(k)
|wi | ≤ wM i ; 1 ≤ i ≤ n; i ≤ k ≤ n. (3.61)
(i)
In particular, for k = i, wi = xi , so that |xi | ≤ wM i . We also let wM be the largest wM i ; therefore:
(k)
|wi | ≤ wM ; i = 1, 2, . . . , n; k ≥ i. (3.62)

Then, using Equation 3.22:

(k)
|gi | ≤ 3.01εM wM i ; k>i (3.63)

and

|δwi | ≤ 3.01εM wM i riU , (3.64)

where riU is the number of off-diagonal nonzeros in row i of U. Alternatively, using Equation 3.62:

|δwi | ≤ 3.01εM wM riU . (3.65)

Finally, we consider the residual

r = Ax − b (3.66)
76 SPARSE MATRIX TECHNOLOGY

obtained when the solution x of System 3.1 is computed using floating point arithmetic. Using
Equations 3.41, 3.52 and 3.60, we obtain:

r = −Ex + Lδw + δb. (3.67)

Taking the 1-norm or the ∞-norm, we have:

krk ≤ kEkkxk + kLkkδwk + kδbk. (3.68)

From Equation 3.62 we obtain bounds for the norms of x:

kxk1 ≤ nwM
kxk∞ ≤ wM . (3.69)

Bounds for the norms of E and L are given in table 3.1. Bounds for the norms of δw and δb were
obtained from Equations 3.65 and 3.55, respectively, and are listed in Table 3.2. Thus, a bound for
krk can be computed using Equation 3.68.
The residual r has another interpretation. Let x̃ be the exact solution of Equation 3.1; then
Ax̃ = b and

r = A(x − x̃), (3.70)

Table 3.2. Values of some parameters and bounds for the norms δb and δw for
forward and backward substitution.

A Parameters of Section 3.4 Forward substitution Backward substitution

Sparse See Section 3.4 kδbk1 ≤ 3.01εM bM nL kδwk1 ≤ 3.01εM wM n0U


kδbk∞ ≤ 3.01εM bM maxi riL + 1 kδwk∞ ≤ 3.01εM wM maxi riU
Full riL = i − 1 kδbk1 ≤ (3.01/2)εM bM n(n + 1) kδwk1 ≤ (3.01/2)εM wM n(n − 1)
riU = n − i kδbk∞ ≤ 3.01εM bM n kδwk∞ ≤ 3.01εM wM (n − 1)
nL = n(n + 1)/2
n0U = n(n − 1)/2
Band riL = min(i − 1, β) kδbk1 ≤ 3.01εM bM (n − β/2)(β + 1) kδwk1 ≤ 3.01εM wM (n − β/2 − 1/2)β
U
|\β\β\| ri = min(n − i, β) kδbk∞ ≤ 3.01εM bM (β + 1) kδwk∞ ≤ 3.01εM wM β
nL = (n − β/2)(β + 1)
n0U = (n − β/2 − 1/2)β
Band riL = min(i − 1, β) kδbk1 ≤ 3.01εM bM (n − β/2)(β + 1) kδwk1 ≤ 3.01εM wM (2n − 2β − 1)β
|\β\2β\| riU = min(n − i, 2β) kδbk∞ ≤ 3.01εM bM (β + 1) kδwk∞ ≤ 6.02εM wM β
nL = (n − β/2)(β + 1)
n0U = (2n − 2β − 1)β
102 SPARSE MATRIX TECHNOLOGY

spondence between fill-ins and new edges added to the graph is evident. The reader can finish the
exercise.
1 2 3 4 5 6 7 8 9 10 11
1 |× × × | 1 10
2 | × × × |
3 | × × × ×|
4 | × × × | 6 8
A(2)
= 5 | × × × | (a)
6 | × ×⊗ |
7 | ×× × ×| 9 11 5
8 | × × ×|
9 | × × × ×|
10 | ⊗ ××| 2 4 7 3
11 | × ×××××|
G2
1 2 3 4 5 6 7 8 9 10 11
1 |× × × | 1 10
2 | × × × |
3 | × × × ×|
4 | × × ⊗ | 6 8
A(3) = 5 | × × × | (b)
6 | × ×⊗ |
7 | ×× × ×| 9 11 5
8 | × × ×|
9 | ⊗ × × ×|
10 | ⊗ ××| 2 4 7 3
11 | × ×××××|
G3
1 2 3 4 5 6 7 8 9 10 11
1 |× × × | 1 10
2 | × × × |
3 | × × × ×|
4 | × × ⊗ | 6 8
A(4)
= 5 | × ⊗× ⊗| (c)
6 | × ×⊗ |
7 | ×⊗ × ×| 9 11 5
8 | × × ×|
9 | ⊗ × × ×|
10 | ⊗ ××| 2 4 7 3
11 | ⊗ ×××××|
G4
Figure 4.8: The three initial elimination steps and the corresponding elimi-
nation graphs for the matrix of Fig. 4.1(a). Fill-ins are encircled.

In terms of graph theory, Parter’s rule says that the adjacent set of vertex k becomes a clique
when vertex k is eliminated. Thus, Gauss elimination generates cliques systematically. Later, as
elimination progresses, cliques grow or sets of cliques join to form larger cliques, a process known
112 SPARSE MATRIX TECHNOLOGY

Y = {14, 16, 1, 7} and finds that Y has adjacent vertices in L5 which have not yet been placed
in any partition. Thus S = {7} is pushed onto the stack and the algorithm branches to Step 5,
where, picking v5 = 13, it is found that the path can not be prolonged any longer, so t = 1. Letting
S = {13}, the algorithm continues with Step 1, where S is not modified, and with Step 2, where Y
is determined to be {13, 15}, which becomes the third partition member.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1 |× × × |
2 | × ×× ×× |
3 | × × × |
4 | × × ×× |
5 | ××× ×|
6 | ×× ×|
7 |× × × × ×|
8 | × ×× × |
9 | × ××× × |
10 | × ××× ×× | (a)
11 |× × ×× × × × |
12 | × × × |
13 | ××× |
14 | × ×××× |
15 | ××× |
16 |× × × × |
17 | × × × ×× |
18 | × × ×× ×× |
19 | × ××× × |
20 | ××× ×|

6 20 5 15 13 16 1 14 7 12 4 11 17 18 2 10 3 9 19 8
6 |××× |
20 |××× × |
5 |××× × |
15 | ×× × |
13 | ×× × |
16 | ××× × |
1 | ×× × × |
14 | ××× × × |
7 | ×× × × × |
12 | ×× × | (b)
4 | ×× ×× |
11 | ×××× × × × |
17 | ×× ××× |
18 | ×××××× |
2 | ×××× × |
10 | × ××× ×× |
3 | × ××|
9 | ×× ×××|
19 | ×××××|
8 | ××××|
Figure 4.12: The Refined Quotient Tree algorithm. (a) Structure of the
matrix corresponding to the graph of Fig. 4.2(a). (b) The permuted block
matrix corresponding to the quotient tree of Fig. 4.2(c).
ORDERING FOR GAUSS ELIMINATION: SYMMETRIC MATRICES 123

Table 4.1

Bound for multipli-

Class of graphs Bound for fill-in Observations and references
cation count
Ordering time is O[(m + n) log n]
if separators can be found in
Any, such that σ = 1/2 c3 n log2 n + O(n) c7 n3/2 + O[n(log n)2 ] O(m + n) time. c3 and c7 given
by Eq. 4.23 (Lipton et al.,
1977152
c3 ≤ 129, c7 ≤ 4002. Ordering
Planar graphs (in this
√ case σ = c3 n log n + O(n) time is O(n log n) (Lipton and
c7 n3/2 + O[n(log n)2 ]
1/2, α = 2/3, β = 2 2) Tarjan, 1979151; Lipton et al.,
1979153)
k is the maximum number of
Two-dimensional finite element
boundary nodes of the elements.
graphs (in this case σ = 1/2, α = O(k 2 n log n) O(k 3 n3/2 )
Ordering time is O(n log n) (Lip-
2/3, β = 4bk/2c)
ton et al., 1979153)
31 829 3/2
Regular planar grid 8 n log2 n + O(n) 84 n + O(n log2 n) (George and Liu, 198197)

Any such that σ > 1/2 O(n2σ ) O(n3σ ) (Lipton et al., 1979153)
Three-dimensional grid graphs
(in this case σ = 2/3)
O(n4/3 ) O(n2 ) (Lipton et al., 1979153)

Any, such that 1/3 < σ < 1/2 O(n) O(n3σ ) (Lipton et al., 1979153)

Any, such that σ = 1/3 O(n) O(n log2 n) (Lipton et al., 1979153)

Any, such that σ < 1/3 O(n) O(n) (Lipton et al., 1979153)

The idea is illustrated in Fig. 4.19(a), where the rectangle represents the set of nodes of a
two-dimensional finite element grid. Choose σ small separators (σ = 3 in the figure) which consist
of grid lines and dissect the grid into σ +1 blocks R1 , R2 , . . . of comparable size. If all separators are
considered to form another single block, a tree partitioning is obtained as shown by the quotient tree
of Fig. 4.19(b). The advantages of tree partitioning regarding the reduction of fill-in and operation
count were discussed in Section 4.9. Now, let us number the nodes of each R-set sequentially,
following lines from left to right as closely as possible, and starting at the bottom left as indicated
by the arrows. When all R-sets have been numbered, the separators are also numbered sequentially,
as the arrows show. The numbering corresponds to a monotone ordering of the tree. The matrix
associated with the finite element grid is partitioned into blocks as shown in Fig. 4.19(c), where
all nonzeros are confined to the cross-hatched areas. If Gauss elimination is performed on this
matrix, fill-in will result only inside the cross-hatched areas and in the dotted areas. Besides, the
hatched blocks are not completely full. For example, the four leading diagonal blocks are banded.
ORDERING FOR GAUSS ELIMINATION: SYMMETRIC MATRICES 135

13 1513 6 20 5 7 1 161412 4 17 3 8 19 9 2 181011

3 15 | ×× × |
13 | ×× × |
14 6 | ××× |
8 20 | ×××× |
5 | ×××× |
15 7 | ×××× ×|
10 3 1 | ××× ×|
12 19 6 16 | ××× ×|
16
14 | ×× ×× ×|
11 1 4 12 | ××× |
4 9 15 20 4 | ××× × |
17 | ××× ×× |
12 17
2 5 3 | ××× |
17 2 13 5 8 | ×××× |
19 | ×××× × |
18 19 9 | ×××× × |
9 8 76 2 | × ×××× |
18 10 14 16 1 7 18 | ×× ×××× |
20 10 | ×××××× |
11 | ×××× ××× |
11

(a) (b)

Figure 4.25: Reverse depth-first ordering, short frond strategy, for the graph of Fig. 4.2(a).

in favor of vertex 19, which is adjacent to two visited vertices: 9 and 10. The reader may continue
the search and verify that the spanning tree and reverse depth-first ordering shown in Fig. 4.25(a)
may be obtained. The separators (11), (10, 18, 2) and (14) can be immediately identified. The
corresponding permuted matrix is shown in Fig. 4.25(b). No fill-in at all is produced by elimination
on this matrix, a result obtained at a very low computational cost. The reason why an ordering
with no fill-in exists for the graph of Fig. 4.2(a) is that this graph is triangulated (Rose, 1970194),
see Section 4.16.
Now consider the application of the long frond strategy to the same graph. Again 11 is the
starting vertex. Vertices 10 and 18 are the next candidates, both of degree 5. We arbitrarily select
vertex 10. At this point Vv = {11, 10}, and vertices 18, 2, 9 and 19 all have three edges leading
to vertices not in Vv . Vertex 18 is discarded because it is adjacent to both visited vertices, while
2, 9 and 19 are adjacent to only one of the visited vertices. Let us choose vertex 2 to be the next
vertex to visit.
At this point Vv = {11, 10, 2} and |Adj(w) − Vv | is equal to 3, 2 and 2 for vertices 17, 18 and 9,
respectively. Thus, we select vertex 17. Next is vertex 4, which introduces two new edges (while 12
or 18 would have introduced only one), and finally vertex 12, which is adjacent to only two visited
vertices (while 18 is adjacent to five). On backtracking to vertex 4 we find the tree arc (4, 18).
Figure 4.26(a) shows one possible ordering obtained in this way. The four separators (11), (10, 2),
(17, 4) and (14) can be identified. As expected, this strategy has produced more separators than
the short frond strategy. The corresponding permuted matrix is shown in Fig. 4.26(b). Elimination
would produce 10 fill-ins in this matrix.
136 SPARSE MATRIX TECHNOLOGY

20 6 5 7 1 1615131419 3 8 9 1812 4 17 2 1011

20 | ×××× |
10 15 6 | ××× |
19 12 5 3 6 2 20 1 5 | ×××× |
7 | × ××× ×|
11 16 14 4 1 | ××× ×|
3 4 18 7 16 | ×× × ×|
15 | ××× |
12 17 5 13 | ××× |
14 | ×××× ×|
8 17 1 19 | ×××× × |
13 18 6
3 | ××× |
8 | ×××× |
9 2 16 9 | × ×× ×× |
18 | × ××××× |
9 8 7 12 | ××× |
10 19 14 13 15 4 | ×××× |
17 | ××××× |
2 | ×× ××× |
20 11 10 | × ×× ××× |
11 | ××× × × ×× |

(a) (b)
Figure 4.26: Reverse depth-first ordering, long frond strategy, for the graph of Fig. 4.2(a).

When the user is dealing with a large problem, a sophisticated ordering algorithm may be
convenient, and may even determine whether the problem is tractable or not. For a medium-size
problem, a simple ordering technique may often produce a large improvement as compared with
no ordering at all, at a low programming cost.

4.16 Lexicographic search

In this section we continue the analysis of low fill orderings for symmetric matrices, but now from
a different point of view. We consider a special class of matrices which can be ordered in such a
way that Gauss elimination would cause no fill-in. Then we take advantage of the properties of
such matrices to give a procedure which finds a low fill ordering for any symmetric matrix. As
usual, we discuss the ideas in terms of graph theory. Let GA = (V, E) be the undirected graph
associated with a symmetric matrix A, and let GF = (V, E ∪ F ) be the corresponding filled graph
associated with U + UT , where A = UT DU is the factorization of A and F is the set of new edges
(or nonzeros of U) introduced during factorization. If the graph GA has an elimination ordering for
which F = ∅, i.e., no fill-in is produced if elimination is carried out in that order, we say that GA is
a perfect elimination graph. The ordering itself is called a perfect elimination ordering. Note
that fill-in may result if we eliminate in a different order, even when GA is a perfect elimination
graph. Note also that every elimination graph GF is a perfect elimination graph since no fill-in
would result if elimination were performed again in the same order.
Chapter 6

Sparse Eigenanalysis

6.1 Introduction
The standard eigenvalue problem is defined by

Ax = λx (6.1)

where A is the given n by n matrix. It is desired to find the eigenpairs (λ, x) of A, where λ is an
eigenvalue and x is the corresponding eigenvector. The generalized eigenvalue problem is
Ax = λBx (6.2)

where A and B are given n by n matrices and again we wish to determine λ and x. For historical
reasons the pair A, B is called a pencil (Gantmacher, 195983). When B = I the generalized problem
reduces to the standard one.
Both for simplicity and to follow the general trend imposed by most of the literature and existing
software, we restrict the analysis to the case where A is real symmetric and B is real symmetric and
positive definite, except when stated otherwise. Almost all the results become valid for hermitian
matrices when the conjugate transpose superscript H is written in place of the transpose superscript
T . On the other hand, an eigenvalue problem where A or A and B, are hermitian, can be solved
using software for real matrices (Section 6.15).
Equation 6.1 has a nonzero solution x when

Det(A − λ I) = 0. (6.3)

This is a polynomial equation of the nth degree in λ, which has n roots λ1 , λ2 , . . . , λn . The roots
are the eigenvalues of A, and they may be either all different or there may be multiple roots with
any multiplicity. When A is real symmetric, the eigenvalues are all real. The simplest example is
the identity matrix I, which has an eigenvalue equal to 1 with multiplicity n. To each eigenvalue

177
SPARSE MATRIX ALGEBRA 223

array of pointers IC at lines 5 and 24. The multiple switch array IX is initialized to 0 at lines 2 and
3. I, defined at line 4, identifies each row. The DO 20 loop scans row I of the first given matrix:
the column indices, if any, are stored in JC at line 11 and the row index is stored in IX at line 13,
thus turning “on” the corresponding switch. The DO 40 loop runs over row I of the second matrix.
For each column index J, defined at line 18, the multiple switch is tested at line 19: if the value
of IX(J) is I, then the switch is on, which means that J has already been added to the list JC
and should not be added again. Otherwise, J is added to JC at line 20. The reader may expect
that the sentence IX(J)=I should appear between lines 21 and 22 in order to record the fact that
the column index J has been added to the list JC. However, such a record is now not necessary
because, during the processing of row I, the same value of J will never be found again: there are
no repeated column indices in the representation of row I in the array JB.

7.11 Algorithm for the numerical addition of two sparse

matrices with N rows
Input: IA, JA, AN first given matrix in RR(C)U.

IB, JB, BN second given matrix in RR(C)U.

IC, JC structure of the resulting matrix in RR(C)U.

N number of rows of the matrices.

Output: CN numerical values of the nonzeros of the resulting matrix.

Workspace: X expanded array used to accumulate the nonzeros; the dimension of X

is M, the number of columns of the matrices.

1. DO 70 I=1,N
2. IH=I+1
3. ICA=IC(I)
4. ICB=IC(IH)-1
5. IF(ICB.LT.ICA)GO TO 70
6. DO 10 IP=ICA,ICB
7. 10 X(JC(IP))=0.
8. IAA=IA(I)
9. IAB=IA(IH)-1
10. IF(IAB.LT.IAA)GO TO 30
11. DO 20 IP=IAA,IAB
12. 20 X(JA(IP))=AN(IP)
292 REFERENCES

213 Sherman, A. H. (1975). On the efficient solution of sparse systems of linear and nonlinear
equations. Ph.D. Thesis, Department of Computer Science, Yale University, New Haven, CT.
Report No. 46.
214 Shirey, R. W. (1969). “Implementation and analysis of efficient graph planarity testing algo-
rithms.” Ph.D. Dissertation, University of Wisconsin, Madison, WI.
215 Silvester, P. P. (1980). Software engineering aspects of finite elements. In Chari abd Silvester
(1980) 30, pp. 69-85.
216 Skeel, R. D. (1981). Effect of equilibration on residual size for partial pivoting. SIAM J.
Numer. Anal. 18, 449-454.
217 Smith, B. T., Boyle, J. M., Dongarra, J. J., Garbow, B. S., Ikebe, Y., Klema, V. C. and Moler,
C. B. (1976). Matrix Eigensystem Routines – EISPACK Guide. Lecture Notes in Computer
Science, Vol. 6, 2nd. edn. Springer-Verlag: Berlin.
218 Speelpenning, B. (1978). “The generalized element method.” Department of Computer Sci-
ence, University of Illinois, Urbana, Champaign, Il. Report UIUCDCS-R-78.
219 Stewart, G. W. (1973). Introduction to Matrix Computations. Academic Press; New York.
220 Stewart, G. W. (1974). Modifying pivot elements in Gaussian elimination. Math. Comput.
28, 537-542.
221 Stewart, G. W. (1976a). A bibliographical tour of the large sparse generalized eigenvalue
problem. In Bunch and Rose (1976) 28, pp. 113-130.
222 Stewart, G. W. (1976b). Simultaneous iteration for computing invariant subspaces of non-
hermitian matrices. Numer. Math. 25, 123-136.
223 Stewart, W. J. and Jennings, A. (1981). Algorithm 570.LOPSI: A simultaneous iteration
algorithm for real matrices. ACM Trans. Math. Software 7, 230-232.
224 Strassen, V. (1969). Gaussian elimination is not optimal. Numer. Math. 13, 354-356.
225 Swift, G. (1960). A comment on matrix inversion by partitioning. SIAM Rev. 2, 132-133.
226 Szyld, D. B. and Vishnepolsky, O. (1982). “Some operations on sparse matrices.” Institute
for Economic Analysis, New York University. Private communication.
227 Takahashi, H. and Natori, M. (1972). Eigenvalue problem of large sparse matrices. Rep.
Comput. Cent. Univ. Tokyo 4, 129-148.
228 Tarjan, R. E. (1971). “An efficient planarity algorithm.” Computer Science Department,
Stanford University, Stanford, CA. Technical Report 244.
229 Tarjan, R. E. (1972). Depth first search and linear graph algorithms. SIAM J. Comput. 1,
146-160.
230 Tarjan, R. E. (1975). Efficiency of a good but not linear set union algorithm. J. Assoc.
Comput. Mach. 22, 215-225.
231 Tarjan, R. E. (1976). Graph theory and Gaussian elimination. In Bunch and Rose (1976) 28,
pp. 3-22.
232 Tewarson, R. P. (1967). On the product form of inverses of sparse matrices and graph theory.
SIAM Rev. 9, 91-99.
233 Tewarson, R. P. (1967). Row-column permutation of sparse matrices. Comput. J. 10, 300-305.
Index

n-dimensional space, 185 Array, 5

data structure, 8
Accidental cancellation, 58, 69, 85, 143
storage, 5
Accumulator
switch, 29
expanded, 30, 230, 234, 244
Assembly
Active column, 15
nodal, 249
Active elements, 78
Assembly of element matrices, 127, 258
Active submatrix, 64
example, 257
Acyclic digraph, 150
numerical algorithm, 261
paths in, 157
symbolic algorithm, 259
Addition
Assignment
of sparse matrices, 219
maximum, 148
of sparse matrices, numerical algorithm., 223
Augmented system, 132
of sparse matrices, symbolic algorithm., 222
Augmenting path, 159, 161
of sparse vectors, 30, 32
shortest, 163
Adjacency level structure, 93
directed, 155 Backsubstitution, 57
Adjacency matrix, 12 Backtracking, 133, 151
Adjacency structure of a graph, 10 Backward error analysis, 67
Adjacent set, 89, 94 Backward substitution, 48, 56
Adjacent vertex, 10 example, 245
Algebra for sparse matrices, 211 Balancing, 84
Algebraic equations Band
linear, 35 generalized eigenproblem, reduction, 188
Algolw, 8 Lanczos algorithm, 206
Algorithm matrix, 12
conjugate gradient, 15 of a matrix, 12
Lanczos, 15 variable, 14
Almost-planar graph, 92, 122 Band matrix
Amalgamation of cliques, 103 band, 12
Ancestor, 90 bandwidth, 12
Arc of tree, 93, 133, 134 diagonal storage, 12, 13
redundant, 154 half-bandwidth, 12

295
296 REFERENCES

semiband, 13 Circular linked list, 7

Bandwidth, 12 Circular storage, 17
reduction algorithm, 96 Clique, 12, 89
Basis, 194 Clique amalgamation, 103
orthonormal, 194 Codiagonal, 13
Bidirectional linked list, 7 Collapsing of vertex, 167
Bigraph, 146, 148 Column
Biorthogonalization Lanczos algorithm, 210 active, 15
Bipartite graph, 146, 148, 163 equilibrated matrix, 83
Bireducible matrix, 145 graph, 148
Bisection method, 184 head, 16
Block-partitioned matrix permutation, 217
storage of, 24 Column-wise representation, 20
Block-within-block-column storage, 25 complete, ordered, 21
Block diagonal pivoting, 82 Column heads, 16
Block Lanczos algorithm, 206 Compact storage, 8, 30
Block lower triangular form, 144, 152, 157 Compatible numbering, 118
Block matrix, 110 Complete pivoting, 78–80
Block methods, 37 Complete representation, 20
Boundary conditions Component partitioning, 90
for scalar problems, 251 Composite vertex, 91, 167
for vector problems, 252 Compression
Boundary element method, 249 Sherman’s, 22
Bounds for eigenvalues, 182 Condensation, 150, 256
Boy, 148, 164 Congruence transformation, 178
Breadth-first search, 93 Congruent pencils, 179
algorithm, 94 Conjugate gradient algorithm, 15
of a digraph, 155 Connected component, 90
of undirected graph, 93 Connected digraph, 147
Bunch’s error formulation, 67 Connected graph, 89
Connected nodes, 251
Cancellation Connection table, 11
accidental, 69, 85 Connectivity level structure, 149
Cardinality, 87 Connectivity matrix, 12, 249, 256
Cell, 6 Consistent numbering, 118
Characteristic polynomial, 183 Control of numerical errors, 77
Chebyshev acceleration, 198 Convenient ordering, 87
Checking a sparse representation, 276 Coordinate over-relaxation, 181
Cholesky factorization, 38, 53 Coordinate relaxation, 181
implementation, 55 Copying a sparse matrix, 279
REFERENCES 297

Cost considerations, 57 Directed graph, 10, 87, 145

CR(C)O, 21 Direct iteration, 190
Cross-link, 93, 154, 155 for generalized eigenproblem, 193
Cuthill and McKee algorithm, 96 Dirichlet nodes, 251
reverse, 98 Disconnected graph, 89
Cutvertex, 90 Disjoint list
Cycle, 89 multiple, 9
directed, 147 Displaying a sparse matrix, 277
Dissection
Data structure, 26 generalized nested, 121
array, 8 height, 114
dynamic, 27 nested, 113
record, 8 Distance, 89, 147
static, 26 Dot product, 194
Data structure set-up, 26 of sparse vectors, 33
Definitions and properties, 37 Dual structure, 150
Deflation, 190 Dynamic storage, 26
Degree, 11, 89 allocation, 27
Depth-first search, 93, 132 Dynamic storage allocation, 27
of a digraph, 151 Dynamic structure, 27
of undirected graph, 132
Descendant, 90 Eccentricity, 89
Desirable forms, 27 Edge, 10, 87
Determinant, 183 Eigenanalysis, 177
Diagonal elementary matrix, 40 of hermitian matrix, 209
Diagonally dominant matrix, 39 of Hessenberg matrix, 189
properly, 39 of tridiagonal matrix, 189
Diagonal pivoting, 79 of tridiagonal unsymmetric matrix, 189
Diagonal storage of band matrix, 12, 13 Eigenpair, 177
Diakoptical system, 131 Eigenproblem
Diameter of graph, 89 generalized, 177
Difference standard, 177
symmetric, 162 unsymmetric, 210
Digraph, 10, 87, 145 Eigenvalue, 177
breadth-first search, 155 economizer, 256
depth-first search, 151 Eigenvalues
Dimension, 194 bounds for, 182
Directed adjacency level structure, 155 Eigenvector, 177
Directed adjacency level substructure, 156 left-hand, 210
Directed cycle, 147 right-hand, 210
298 REFERENCES

Element Exit, 149

generalized, 130 Expanded array of pointers, 33, 34
super, 130 Expanded real accumulator, 31
Elementary matrices, 40 Expanded storage, 9, 30
properties, 41
Elementary matrix, 40 Factorization
Element matrix, 128 Cholesky, 38, 53
assembly, 128 error bounds, 74
Elimination digraph, 147 errors, 68
Elimination form orthogonal, 37
relation with product form, 52 triangular, 22, 35
Elimination form of the inverse, 47, 52 Fast Givens rotation, 186
Elimination graph, 101 Fill, 26, 36
perfect, 136 Fill-in, 85
Elimination ordering for nested dissection, 123
minimal, 138 Filled digraph, 147
minimum, 138 Filled graph, 103
perfect, 136 Finite element, 127
Embedding graph, 92
planar, 88 method, 92, 127, 249
Entrance, 149 ordering, 127
Envelope, 14 Fixed length records, 16
Envelope storage Floating point operations
of symmetric matrices, 14 errors, 65
Equilibrated matrix, 83 Flutter
Equilibration, 83 aircraft analysis, 210
Error Forest
estimator, 82 spanning, 92, 152
Error bounds Format, sparse
for factorization, 74 column-wise, 20
for substitution, 76 row-wise, 19
Error growth Forward and backward substitution
monitoring, 82 algorithm, 246
Errors Forward row-backward column storage, 18
control, 77 Forward substitution, 48, 56
in floating point operations, 65 example, 245
in sparse factorization, 68 Free vertex, 161
in sparse substitution, 73 Frond, 133, 154
Estimator, 82 Front, 129
Euclidean space, 193 Frontal method, 128
REFERENCES 299

for general matrix, 175 Graph theory

Front of queue, 7 and Gauss elimination, 101
Frontwidth, 15 for symmetric matrices, 87
Fundamentals, 5 for unsymmetric matrices, 146–148

Gauss-Jordan elimination Hölder’s inequality, 65

by columns, 50 Half-bandwidth, 12
by rows, 52 Hall’s algorithm, 158
Gauss-Seidel Head, 6
algorithm, 275 of column, 16
method, 274 of row, 16
Gauss elimination Height of dissection, 114
and graph tehory, 101 Hermitian matrix, 40
by columns, 45 eigenanalysis of, 209
by rows, 49 Hessenberg form, 37
Hessenberg matrix, 185
Generalized eigenproblem, 177
eigenanalysis of, 189
Generalized element, 130
Hopcroft and Karp’s algorithm, 161
Generalized nested dissection, 121
Householder’s reflection, 186
Gerschgorin
matrix, 186
interval, 183
Hypermatrix storage, 25
Gerschgorin disc, 183
Girl, 148, 164 Implicit storage, 24
Givens rotation, 186 Incident edge, 89
fast, 186 Indefinite matrix, 37
Gram-Schmidt orthonormalization, 196, 206 Indefinite symmetric system, ordering, 140
Graph, 87 Indegree, 146
adjacency structure, 10 Indistinguishable vertices, 107
bipartite, 146 Inertia theorem, 184
connected, 89 Infinite element, 127
directed, 10, 87 Inner product, 193
disconnected, 89 Integer array of pointers, 32, 34
edge, 10 Integers
labelling, 10 merging of lists, 28
planar, 88, 119 storage of lists, 8
representation, 10 Interval of Gerschgorin, 183
storage, 10 Introduction, 1, 5, 63
undirected, 10 Invariance of a graph, 88, 145, 148
vertex, 10 Invariant subspace, 194
Graphs Inverse iteration, 190
clique, 12 for generalized eigenproblem, 193
300 REFERENCES

simultaneous, 198 directed adjacency, 155, 156

Isostructural rows, 24 length of, 93
rooted, 93
Jacobi rotation, 186 width, 93
Jenning’s storage, 14 Lexicographic search, 136
Jungle, 156 algorithm, 138
Linear algebraic equations, 35
Key’s storage, 20
Line of a matrix, 18
King’s algorithm, 98
Linked list, 6
reverse, 101
bidirectional, 7
Knuth-Rheinboldt-Mesztenyi storage, 17
circular, 7
Knuth sparse storage scheme, 16
Linked sparse storage schemes, 15
Knuth storage
Linked storage, 15
column heads, 16
List
row heads, 16
head, 6
KRM circular storage, 17
linked, 6
Krylov subspace, 199
merging, 28
Label, 10 operations with, 5, 9
Labelled graph, 10, 87 range, 8
Laguerre’s iteration, 183 sparse, 8
Lanczos’ basis, 199 storage of, 5
Lanczos algorithm, 15, 199 terminator, 6
band, 206 Lists of integers, 8
biorthogonalization, 210 compact storage, 8
block, 206 expanded storage, 9
block, for generalized eigenproblems, 208 merging, 28
for generalized eigenproblem, 203 multiple disjoint, 9
in practice, 203 range, 8
with no orthogonalization, 205 sparse, 8
with periodic orthogonalization, 206 storage, 8
with reorthogonalization, 204 Long frond ordering, 134
with selective orthogonalization, 204 Loop, 88
Left-hand eigenvector, 210 Lower column elementary matrix, 40
Left row elementary matrix, 41 Lower triangular matrix, 22, 40
Length block form, 144
of level structure, 93, 155 Lowlink, 168
of partitioning, 150
Level structure, 90, 93 Machine precision, 65
ajacency, 93 Markowitz’s algorithm, 172
connectivity of, 149 Markowitz’s pivoting, 172
REFERENCES 301

Matching, 148, 161 Minimization of trace, 208

Matrices Minimum degree algorithm, 104
elementary, 40 Minimum elimination ordering, 138
triangular, 40 Minor, 38
Matrix principal, 38
adjacency, 12 Minrow-within-mincolumn pivoting, 174
bireducible, 145 Modularity, 27
connectivity, 12 Monitoring error growth, 82
diagonally dominant, 39 Monotone ordering, 90
elementary, 40 Multi-frontal method, 130
hermitian, 40 Multiple disjoint lists, 9
indefinite, 37 Multiple switch technique, 29
lower triangular, 22, 40 Multiplication
minor, 38 of diagonal matrix by matrix, 279
nondefinite, 37 of general matrix by vector, 224
nullity, 39 of sparse matrices, 230
of rank one, 39 of symmetric matrix by vector, 228, 229
orthogonal, 38 of triangular matrix by matrix, 268–270
permutation, 44 of triangular matrix by vector, 272
positive definite, 37 Multiplicity, 177
principal minor, 38
properly diagonally dominant, 39 Nested dissection, 113
rank, 39 algorithm, 116
rank deficiency, 39 fill-in, 123
singular, 39 generalized, 121
structure, 16 ordering, 113
triangular factorization, 38 properties, 118
unit diagonal, 40 tree, 114
unsymmetric, 37 Nodal assembly matrix, 249, 251, 257
upper triangular, 22, 40 Nodes, 127
zero-nonzero pattern, 16 connected, 251
Maximal set, 157 Dirichlet, 251
Maximum assignment, 148 Nonzero, 69, 85
Maximum set, 157 definition, 69
Merging sparse lists of integers, 28 Norm
Mesh, 127 of matrix, 65, 72
Mesh generation, 250 of vector, 64
Minimal elimination ordering, 138 Nullity, 39
Minimal nested dissection partitioning, 115 symbolic, 159
Minimal separator, 90 Nullity of a matrix, 39
302 REFERENCES

Numbered graph, 87 Partitioning, 90

Numbering component, 90
compatible, 118 tree, 91
consistent, 118 Pascal, 8
Numerical assembly algorithm, 261 Path, 89
Numerical errors, 63, 65, see also Errors augmenting, 159, 161
Numerical examples, 59 length, 89
Numerical processing, 26 shortest augmenting, 162
Numerical stability and pivot selection, 78–82 Paths in an acyclic digraph, 157–158
Numerical triangular factorization algorithm, 242– algorithm for finding, 157
245 Pattern
in row-wise format, 238–239 zero-nonzero, 16
Pedigree, 90
Offspring, 90
Pencil, 177
One-way dissection, 122–126
congruent, 179
algorithm, 125–126
Perfect elimination graph, 136
Operations with lists, 5
Perfect elimination ordering, 136
Operations with sparse matrices, 211
Peripheral vertex, 89
Ordered graph, 87
Permutation
Ordered representation, 20
of columns, 217
Ordering
of rows, 217
convenient, 87
Permutation matrix, 44
for pivot selection, 45
storage, 45
monotone, 90
Phase counter, 30
Origin shift, 191
Pivot, 45
Orthogonal factorization, 37
Pivoting, 78
Orthogonal matrix, 38
block diagonal, 82
Orthogonal similarity, 178
complete, 79
Orthonormal basis, 194
Orthonormalization diagonal, 79
Gram-Schmidt, 206 for symmetric matrix, 86, 141
Outdegree, 146 for unsymmetric band matrix, 175
Overhead storage, 15 for unsymmetric matrix, 172–175
Overrelaxation partial, 80
coordinate, 181 threshold, 80
parameter, 181 Pivot selection, 45, 77
and numerical stability, 78–83
Packed storage, 30 Planar embedding, 88
Partial pivoting, 80 Planar graph, 88, 119
Partitioned matrix Plane in n-dimensional space, 185
storage, 24–26 Plane rotation, 186
REFERENCES 303

Pop Rayleigh matrix, 182, 195

item on stack, 7 Rayleigh quotient, 180–182
Positive definite matrix, 37 iteration, 193
Power method, 190–191 iteration, for generalized eigenproblem, 193
for generalized eigenproblem, 193 procedure, 195
Practical Cholesky factorization, 55 procedure, for generalized eigenproblem, 195
Preface, xiii Reachability matrix, 146
Primary candidates, 78 Reachable set, 90
Principal minor, 38 Reachable vertex, 90, 146
Printing a sparse matrix, 277 Real accumulator
Processing expanded, 31
numerical, 26 Rear of queue, 7
symbolic, 26 Receiver, 146
Product, see Multiplication Record data structure, 8
Product form of the inverse, 51–53 Records
Profile, 14 fixed length, 16
reduction, 98–101 Records of variable length, 28
Properties and definitions, 37 Reducible matrix, 145
Properties of elementary matrices, 41 Reduction
Properties of triangular matrices, 42–44 of band generalized eigenproblem, 188
Pseudoperipheral vertex, 89 of band matrix, 188–189
algorithm for finding, 95 of bandwidth, 96–98
Push of general matrix, 185–187
item on stack, 7 of profile, 98–101
Reflection of Householder, 186
Queue, 7
Reflector, 186
front, 7
Relaxation coordinate, 181
rear, 7
Representation, 19–20
storage, 5
Quotient graph, 90 complete, 20
Quotient tree, 91 of graphs, 10–12
algorithm, refined, 110 ordered, 20
ordering, 215–216, 218
Radix sort, simultaneous, 217 transforming, 278
Range of list, 8 unordered, 20
Rank Residual, 75, 84
deficiency, 39 Residual matrix, 200
of a matrix, 39 Residual vector, 181
symbolic, 159 Restriction, 194
Rank one matrix, 39 Reverse
Rayleigh-Ritz procedure, 195 depth-first ordering, 134
304 REFERENCES

monotone ordering, 133 Selection of pivots, 45, 77

Right-hand eigenvector, 210 Selfedge, 88
Right row elementary matrix, 41 Semiband, 13
Ritz values, 195 Separator, 90
Ritz vector, 195 minimal, 90
threshold, 204 Set
Rooted level structure, 93 notation, 162
Rooted substructure, 155 operations, 162
Rooted tree, 90 Shape functions, 250
Root of strong component, 153 Sherman’s compression, 22–24
Rotation Shift of origin, 191
fast Givens, 186 Short frond ordering, 134
Givens, 186 Similarity, orthogonal, 178
Jacobi, 186 Simultaneous iteration, 196–199
plane, 186 for generalized eigenproblem, 198–199
Row inverse, 198
equilibrated matrix, 83 Simultaneous radix sort, 217
graph, 148 Singular matrix, 39
head, 16 Singular values, 208
Row-wise format, 19–20 Sink, 146
Row-wise representation, 19 Skeleton, 92
complete, unordered, 21 Skyline storage, 15
complete and ordered, 20 Software for unsymmetric systems, 175
diagonal and upper, ordered, 21 Source, 146
upper, ordered, 22 Space, 193
Row heads, 16 Span, 90
Row permutation, 217–218 Spanning forest, 92, 152
Rows Spanning set, 194
isostructural, 24 Spanning tree, 92
RR(C)O, 20 Sparse list, 8
RR(C)U, 21 Sparse matrix
RR(DU)O, 21 algebra, 211–247
RR(U)O, 22 operations, 211
Sparse representation
Sargent and Westerberg’s algorithm, 167–168 checking of, 276
Scalar product, 194 Sparse row-wise format, 19–20
of sparse vectors, 33 Sparse substitution
Scaling, 83–84 errors in, 73–77
Search, 93 Sparse tableau, 132
Section graph, 88 Sparse vectors
REFERENCES 305

addition, 30–33 of symmetric matrices, 14

dot product, 33 overhead, 15
scalar product, 33–34 packed, 30
Spectral factorization, 178 skyline, 15
Stability sparse row-wise format, 19
numerical, 78–82 supersparse, 26
Stack, 7 variable band, 14
pop an item, 7 Strong component, 145, 148–151, 167, 168
push an item, 7 root, 153
storage, 5 Strongly connected component, 148
top, 7 Strongly connected digraph, 148
Standard eigenproblem, 177 Structure of a matrix, 16, 20
Static data structure, 26 Sturm sequence, 184
Stiffness matrix, 251 Subgraph, 88
Stodola’s iteration, 190 Subspace, 194
Storage, 5–28 invariant, 190, 194
allocation, 26 iteration, 196
block-partitioned matrix, 24 Krylov, 199
block-within-block-column, 25 Substitution
compact, 30 backward, 48, 56–57
compressed, 22 error bounds for, 76
connection table, 11 errors in, 73–77
dynamic allocation, 27 forward, 48, 56–57
dynamic schemes, 26–28 Substructure
expanded, 30 rooted, 155
forward row-backward column, 18 Substructuring, 130
hypermatrix, 25 Super-element, 130
implicit, 24 Supersparse storage, 26
Knuth-Rheinboldt-Mesztenyi, 17 Switch, 29
Knuth scheme, 16 array, 29
KRM, 17 multiple, 29
linked sparse schemes, 15 phase counter, 30
of arrays, 5 Switch array, 29
of band matrices, 12 Switch technique
of graphs, 10 multiple, 29–30
of integers, 8 Sylvester’s inertia theorem, 184
of lists, 5 Symbolic
of lists of integers, 8 nullity, 159
of queues, 5 Symbolic assembly algorithm, 259–260
of stacks, 5 Symbolic processing, 26
306 REFERENCES

Symbolic rank, 159 redundant, 154

Symbolic section, 26 Tree partitioning, 91
Symbolic singular matrix, 159 Triangular factorization, 22, 35
Symbolic triangular factorization in row-wise format, 235–245
algorithm, 240–242 numerical, 238–239
in row-wise format, 235 numerical algorithm, 242–245
Symmetric symbolic algorithm, 240–242
difference, 162 Triangular matrices, 40
Symmetric indefinite system properties, 42–44
ordering, 140–141 Triangulated graph, 137
Symmetric matrices Triangulation, 137
envelope storage, 14 Tridiagonal matrix, 185
Symmetric matrix, 37 eigenanalysis of, 189
System eigenanalysis of unsymmetric, 189
augmented, 132 Triple, 16
diakoptical, 131
Undirected graph, 10, 87
breadth-first search, 93–95
Table of factors, 43, 47, 51
depth-first search, 132–136
Tarjan’s algorithm, 168–172
Uni-frontal method, 130
Terminal members, 114
Unit diagonal matrix, 40
Terminator, 6
Unordered representation, 20–22
Threshold pivoting, 80
Unsymmetric band matrix pivoting, 175
Threshold vector, 204
Unsymmetric eigenproblem, 210
Tolerance, 80
Unsymmetric matrix, 37
Top of stack, 7
graph theory, 146–148
Trace, 183
pivoting strategies for, 172–175
Trace minimization, 208
Unsymmetric system software, 175
Transforming a representation, 278
Upper almost-triangular matrix, 185
Transmitter, 146
Upper column elementary matrix, 41
Transportability, 212
Upper Hessenberg matrix, 185
Transposition of a matrix, 213–215 Upper triangular matrix, 22, 40
algorithm, 215–216
Transversal, 18, 145, 146, 158 Variable band, 14
Tree, 90 Variable length records, 28
nested dissection, 114 Vectors
partitioning, 109–113 addition, 30–32
quotient, 90 Vertex, 10, 87
rooted, 90 adjacent, 10
spanning, 92 degree, 11
Tree arc, 93, 133, 152, 155 Vertex collapsing, 167
REFERENCES 307

Wavefront, 15
Width
of level structure, 155

Zero-nonzero pattern, 16
Zlatev’s pivoting, 81
improved, 81

View publica