Scribd
Upload
168 views

Positive and Negative Definite Matrices and Optimization

This document provides notes on positive and negative definite matrices and their application to optimization problems. It defines positive and negative definite matrices and discusses how to determine if a symmetric matrix is positive definite based on its principal minors. Examples are given of using the Hessian matrix at a critical point to determine if the point is a local minimum, maximum, or saddle point. The relationship between the definiteness of the Hessian and local optimality is summarized. Exercises from the textbook chapter are listed at the end.

Uploaded by

Anonymous dEfIaU
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
168 views

Positive and Negative Definite Matrices and Optimization

This document provides notes on positive and negative definite matrices and their application to optimization problems. It defines positive and negative definite matrices and discusses how to determine if a symmetric matrix is positive definite based on its principal minors. Examples are given of using the Hessian matrix at a critical point to determine if the point is a local minimum, maximum, or saddle point. The relationship between the definiteness of the Hessian and local optimality is summarized. Exercises from the textbook chapter are listed at the end.

Uploaded by

Anonymous dEfIaU
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Jim Lambers

MAT 419/519
Summer Session 2011-12
Lecture 3 Notes
These notes correspond to Section 1.3 in the text.

Positive and Negative Definite Matrices and Optimization


The following examples illustrate that in general, it cannot easily be determined whether a symmetric matrix is positive definite from inspection of the entries.
Example Consider the matrix

A=

1 4
4 1


.

Then
QA (x, y) = x2 + y 2 + 8xy
and we have
QA (1, 1) = 12 + (1)2 + 8(1)(1) = 1 + 1 8 = 6 < 0.
Therefore, even though all of the entries of A are positive, A is not positive definite. 2
Example Consider the matrix

A=

1 1
1
4


.

Then
QA (x, y) = x2 + 4y 2 2xy = x2 2xy + y 2 + 3y 2 = (x y)2 + 3y 2
which can be seen to be always nonnegative. Furthermore, QA (x, y) = 0 if and only if x = y and
y = 0, so for all nonzero vectors (x, y), QA (x, y) > 0 and A is positive definite, even though A does
not have all positive entries. 2
Example Consider the diagonal matrix

1 0 0
A = 0 3 0 .
0 0 2
Then
QA (x, y, z) = x2 + 3y 2 + 2z 2 ,
which can plainly be seen to be positive except when (x, y, z) = (0, 0, 0). Therefore, A is positive
definite. 2
1

The preceding example can be generalized

d1

0
A=
..
.
0

as follows: if A is an n n diagonal matrix

0 0
..
d2
.
,

..
. 0
0 dn

then A is:
1. positive definite if and only if di > 0 for i = 1, 2, . . . , n,
2. negative definite if and only if di < 0 for i = 1, 2, . . . , n,
3. positive semidefinite if and only if di 0 for i = 1, 2, . . . , n,
4. negative semidefinite if and only if di 0 for i = 1, 2, . . . , n,
5. indefinite if and only if di > 0 for some indices i, 1 i n, and negative for other indices.
We now consider a general 2 2 symmetric matrix


a b
.
A=
b c
This matrix induces the quadratic form
QA (x, y) = ax2 + 2bxy + cy 2 .
If y = 0, then we have QA (x, 0) = ax2 , so we must certainly have a > 0 in order for A to be positive
definite. If y 6= 0, then we use a change of variable x = ty and obtain
QA (x, y) = QA (ty, y) = at2 y 2 + 2bty 2 + cy 2 = (at2 + 2bt + c)y 2 .
Thus QA (x, y) is positive definite if and only if (t) = at2 + 2bt + c is positive.
From
0 (t) = 2at + 2b, 00 (t) = 2a,
we obtain the single critical point t = b/a and determine that it is a strict global minimizer of
(t). The minimum value is
(t ) = a(b/a)2 + 2b(b/a) + c = c

b2
1
= det(A).
a
a

We conclude that A is positive definite if and only if a > 0 and det(A) > 0. This leads to the
following theorem.
Theorem A 2 2 symmetric matrix A is
2

1. positive definite if and only if a > 0 and det(A) > 0


2. negative definite if and only if a < 0 and det(A) > 0
3. indefinite if and only if det(A) < 0
A similar argument, combined with mathematical induction, leads to the following generalization.
Theorem Let A be an n n symmetric matrix, and let Ak be the submatrix of A obtained by
taking the upper left-hand corner k k submatrix of A. Furthermore, let k = det(Ak ), the kth
principal minor of A. Then
1. A is positive definite if and only if k > 0 for k = 1, 2, . . . , n;
2. A is negative definite if and only if (1)k k > 0 for k = 1, 2, . . . , n;
3. A is positive semidefinite if k > 0 for k = 1, 2, . . . , n 1 and n = 0;
4. A is negative semidefinite if (1)k k > 0 for k = 1, 2, . . . , n 1 and n = 0.
Note that the last two statements in the theorem are not if and only if statements. Furthermore,
if A has nonnegative principal minors, then A is not necessarily positive semidefinite. For example,
if

1 1 1
A = 1 1 1 .
1 1 21
All of the principal minors are nonnegative, but (1, 1, 2) A(1, 1, 2) < 0, so A is not positive
semidefinite; it is actually indefinite.
We now show how the preceding discussion can be applied to find minimizers and maximizers
of functions.
Example Let
f (x, y, z) = x2 + y 2 + z 2 xy + yz xz.
We have
f (x, y, z) = (2x y z, 2y x + z, 2z + y x).
Solving f (x, y, z) = (0, 0, 0) yields only the trivial solution (0, 0, 0). We also have

2 1 1
2
1 ,
Hf (x, y, z) = 1
1
1
2

which can be confirmed to be positive definite by examinination of the principal minors: 1 = 2,


2 = 3, 3 = 4. It follows that the critical point (0, 0, 0) is a strict global minimizer of f (x, y, z).
2
Example Let
f (x, y) = exy + eyx .
We have
f (x, y) = (exy eyx , exy + eyx ),
which yields the critical points (x, x) for all x R. We also have


exy + eyx exy eyx
Hf (x, y) =
,
exy eyx
exy + eyx
which yields
1 = exy + eyx > 0,

2 = (exy + eyx )2 (exy eyx )2 = 0.

That is, Hf (x, x) is positive semidefinite, making (x, x) a global minimizer of f (x, y). 2
Example Let

f (x, y, z) = exy + eyx + ex + z 2 .


We have

f (x, y, z) = (exy eyx + 2xex , exy + eyx , 2z)


which yields the conditions z = 0, x = y and x = 0 for a critical point. Therefore, (0, 0, 0) is the
only critical point. We then have
xy

2
2
e
+ eyx + 4x2 ex + 2ex exy eyx 0
Hf (x, y, z) =
exy eyx
exy + eyx 0 ,
0
0 2
and therefore

3 2 0
2 0 .
Hf (0, 0, 0) = 2
0
0 2

which is positive dfinite, in view of 1 = 3, 2 = 2, and 3 = 4. It can also be shown by direct


computation of the minors that Hf (x, y, z) is positive definite on all of R3 , using the fact that
ex > 0 for all x. Therefore, the critical point (0, 0, 0) is a strict global minimizer of f (x, y, z). 2
Example Let
f (x, y) = exy + ex+y .

We have
f (x, y) = (exy + ex+y , exy + ex+y ),
which yields no critical points, as f /x can never be zero. Therefore, there are no global minimizers. 2
We now consider how the Hessian can be used to establish the existence of a local minimizer or
maximizer.
Theorem Suppose that f (x) has continuous first and second partial derivatives on a set D Rn .
Let x be an interior point of D that is a critical point of f (x). Then x is a:
1. strict local minimizer of f (x) if Hf (x ) is positive definite;
2. strict local maximizer of f (x) if Hf (x ) is negative definite.
This theorem can be proved by using the continuity of the second partial derivatives to show
that Hf (x) is positive definite for x sufficiently close to x , and then applying the multi-variable
generalization of Taylors Formula.
We now consider the implications of an indefinite Hessian at a critical point. Suppose that f (x)
has continuous second partial derivatives on a set D Rn . Furthermore, let x be an interior point
of D that is a critical point of f (x). If Hf (x ) is indefinite, then there exist vectors y, w such that
y Hf (x )y > 0,

w Hf (x )w < 0.

By continuity of the second partial derivatives, there exists an  > 0 such that
y Hf (x + ty)y > 0,

w Hf (x + tw)w < 0

for |t| < . If we define


Y (t) = f (x + ty),

W (t) = f (x + tw),

then Y 0 (0) = W 0 (0) = 0, while Y 00 (0) > 0 and W 00 (0) < 0. Therefore, t = 0 is a strict local
minimizer of Y (t) and a strict local maximizer of W (t).
It follows that x is a strict local minimizer along a curve contained in the graph of f that
passes through x in the direction of y, and a strict local maximizer along a curve contained in the
graph of f that passes through x in the direction of w. This gives the graph of f the appearance
of a saddle near x , which leads to the following definition.
Definition A saddle point for f (x) is a critical point x for f (x) such that there are vectors y, w
for which t = 0 is a strict local minimizer for Y (t) = f (x + ty) and a strict local maximizer for
W (t) = f (x + tw).
The result of the preceding discussion is summarized in the following theorem.
5

Theorem If f (x) is a function with continuous second partial derivatives on a set D Rn , if x


is an interior point of D that is also a critical point of f (x), and if Hf (x ) is indefinite, then x is
a saddle point of x .
Example Let
f (x, y) = x3 12xy + 8y 3 .
We have
f (x, y) = (3x2 12y, 12x + 24y 2 ),
which yields the critical points (0, 0) and (2, 1). We also have


6x 12
Hf (x, y) =
,
12 48x
and therefore


Hf (0, 0) =

0 12
12
0


,

Hf (2, 1) =

12 12
12
48


.

We see that Hf (2, 1) is positive definite, because its principal minors are positive, but Hf (0, 0) is
not, as 1 = 0 and 2 = 144. That is, Hf (0, 0) is indefinite, so (0, 0) is a saddle point.
Furthermore,
lim f (x, y) = +,
lim f (x, y) = ,
x

so f (x, y) has no global minimizer on


minimizer. 2

R2 .

We can conclude, however, that (2, 1) is a strict local

It should be emphasized that if the Hessian is positive semidefinite or negative semidefinite


at a critical point, then it cannot be concluded that the critical point is necessarily a minimizer,
maximizer or saddle point of the function.
Example Let f (x, y) = x4 y 4 . We have
f (x, y) = (4x3 , 4y 3 ),
which yields the critical point (0, 0). We then have


12x2
0
Hf (x, y) =
.
0 12y 2
Therefore Hf (0, 0) is the zero matrix, which is positive semidefinite. However, f (x, y) increases
from (0, 0) along the x-direction, and decreases along the y-direction, so (0, 0) is neither a local
minimizer nor maximizer. 2
Example Let f (x, y) = x4 + y 4 . As in the previous example, Hf (0, 0) is the zero matrix, but it
can be seen from a graph that (0, 0) is a strict global minimizer. 2
6

Exercises
1. Chapter 1, Exercise 2
2. Chapter 1, Exercise 6
3. Chapter 1, Exercise 7
4. Chapter 1, Exercise 9
5. Chapter 1, Exercise 11
6. Chapter 1, Exercise 19

You might also like