GEOMETRY OF THE CRAMER-RA0 BOUND

Louis L. Schorf and L. T. McWhorier

Department of Electrical & Computer Engineering

University of Colorado at Boulder
Boulder, Colorado 80309-0425

Abstract: t o partition the parameters that Structure the mean

The Fisher information matrix determines how into disjoint sets. This allows us to study interfer-
much information a measurement brings about the ence between parameters. The corresponding Fisher
parameters that index the underlying probability dis- matrix is similarly partitioned. The inverse of the
tribution for the measurement. In this paper we as- partitioned Fisher matrix may be Cholesky factored
sume that the parameters structure the mean value t o produce a partitioned inverse that brings all of our
vector in a multivariate normal distribution. The geometrical and filtering interpretations. When the
Fisher matrix is. then a Gramian constructed from partition corresponds to linear parameters that de-
the sensitivity vectors that characterize the first-order termine mode weights and nonlinear parameters that
variation in the mean with respect to the parameters. determine mode characteristics, then these interpre-
The inverse of the Fisher matrix has several geomet- tations apply to deterministic modal analysis. For
rical properties that bring insight into the problem example, in time series analysis and array process-
of identifying multiple parameters. For example, it is ing, the linear parameters correspond to mode ampli-
the angle between a given sensitivity vector and the tudes and phases, and the nonlinear parameters de-
linear subspace spanned by all other sensitivity vec- termine mode frequencies, damping coefficients, and
tors that determines the variance bound for identify- directions of arrival [5-6].
ing a given parameter. Similarly, the covariance for
2.0 The Fisher Matrix for Structured Means
identifying the linear influence of two different sub-
sets of parameters depends on the principal angles Let y denote a real N x 1 random vector and e
between the linear subspaces spanned by the sensi- a real p x 1 deterministic but unknown parameter.
tivity vectors for the respective subsets. We say that y E RN and Q E RP. Generally, p 5
N . The probability density function for y is denoted
f(y;e). We shall amume that the distribution of y is
1.0 Introduction
N [x(e),RI. That is, y is multivariate normal, with
Behind the mathematical formulas of the Cramer- the unknown parameters structuring the mean value
Rao bound hides a fascinating geometry. This geom- vector x according t o the function .(e). The function
etry brings a wealth of insight into the way param- x is known, but the parameters are unknown. The
eters interfere with each other and abridge our abil- probability density function for y may be written
ity to identify them. In this paper we consider the
problem of identifying parameters that structure the f(y;e) = [ det(R)]-l12
mean value vector in a multivariate normal distribu-
tion. This is a more general version of the so-called
deterministic modal analysis problem studied in [l-61,
so our results bring insight into frequency estimation
in time series analysis and direction of arrival estima- The Fisher matrix is defined to be
tion in multisensor array processing.
In order to gain our geometrical insight, we study
the Fisher matrix and its inverse. Our procedure is
The partial derivative of tile log-likelihood function
In f(y;B is
This work supported by Bonneville Power Adniin-
istration under contract #DEBI7990BP07346 and by
the Office of Naval Research, Statistics and Probabil-
ity Branch, under contract #N0001489-J-1070.

-5-
Therefore, the Fisher matrix for this problem is the bounds for subsets of parameters. Toward this end,
Gramian we defineeT = ET,& where
J(0J = GTR-'G (1)
where G is the sensiiiviiy mairir QT = [e, ...e,]; = [e,,, .. .ep].
The sensitivity matrix G is then partitioned by C1
and G2:

The vector gi is the variation in the mean with respect

to parameter Bi. It is the fundamental variable in the
analysis that follows. The Fisher matrix for the parameter vector ST =
We shall assume throughout the remainder of this hT,G] is partitioned as follows:
paper that the known covariance matrix R is the di-
agonal matrix u21. This is perfectly general, because
we could as well factor R as R = (R-'/2)TR-'/2and
call R-'I2G the sensitivity matrix.
Invariant.Preprocessors.If the measurement y
is replaced by the linearly transformed measurement
Ty, then the distribution of T y is N [Tx(g), T@] . This is the general formula for partitioning the in-
The sensitivity matrix for Tx(8) is formation that the measurement y brings about the
parrameters El and &. In this formula, the diagonal
term @Cl (or a G 2 ) contains inimcomlalion be-
tween sensitivity vectors that lie tuiihin GI (or C2).
The off-diagonal term q C 2 (or @Cl) contained
and the Fisher matrix for T y is intercomlaiions between sensitivity vectors that lie
in GI and G2 (or G2 and GI). by defining the ma-
J = ~ ( B=
) G~T~(TRT~)-'TG. trix Pc-i = G i ( G G i ) " G , the orthogonal projec-
tion onto the linear subspace (Gi), and the matrix
A transformation Twill be called an invariant pmpro- G n = (GTGi)-'@, the pseudoinverse of Ci, we may
cessor if J T ~ ( =~ J(B.
) For example, when R = I , factor the Fisher matrix as follows:
the Fisher matrix J T ~ ( ~is J

JTy(B) = GT&G,
where & is the projection matrix T ( F T ) - l p .
This means that the linear transformation T y is an
invariant preprocessor whenever PTG = G,meaning
that the sensitivity matrix lies in (T), the column
span of T. This fundamental result, first observed
by Anderson [12], shows that measurements y E RN
may be replaced by a dimension-reducing transforma-
tion T y E RP without changing Fisher information,
provided T is matched to G. As a practical matter, 3.0 The Cramer-Rao Bound
one never knows or, consequently, G, but one may From a measurement y we construct an estimator
well know an approximate range of values for 8, in of e, call it g(y). We shall assume that this estima-
which case T may be designed to be approximately tor is unbiased, meaning that d ( y ) = e.The e m r
invariant. Such invariant transformations are called covariance matriz for any unbiased estimator &y) is
beamformers in array processing and windows in sig- lower bounded by the inverse of the Fisher informa-
nal processing. tion matrix:
Partitioned Formulas. Now let us partition the
parameter vector 8 so that we may study variance E[l(Y) - 4 [&Y> - ElT L J-'(e).

-6-
The e m w variance is bounded by the trace of this and the inverse of the Fisher matrix is
matrix: 1 *
J-'(E) =
E[&Y)-BI~[&Y) -E] 2 tr[J-'(Q)]. [ e ( I - P',)Gi]-'
If the estimator &y) is replaced by f(y), an unbiased The northwest corner lower bounds the variance of
estimator of the function f(&),then these bounds are any unbiased estimator of Bi:
[51
(7)

This result is illustrated in the figure below. There

are two important linear subspaces in the figure: (g),
the one-dimensional subspace spanned by the ithsen-
sitivity vector gi, and (Gi), the p- 1-dimensional sub-
-
space spanned by the remaining (p 1) sensitivity
vectors g j , j # i . The projection of the ith sensitiv-
ity vector gi onto the subspace orthogonal to (Gi) is
This formula holds for f(e) an M x 1 vector, in which (I - PG,)gi. The norm squared of this projection is
case [9
1 is an M x p matrix that is conformable gT(I - Poi)&, and the inverse of this norm squared
is the variance bound. So, it is only when the vari-
with J-'(E). ation of x(e) with respect to ei lies well outside the
Basic Formula for Cramer-Rao Bound. For subspace (Gi) that Bi can be estimated with low vari-
the partition e = [el $1 and the corresponding par- ance. That is, it is only when the variation in x(0)
tition G = [GI G,], the inverse of the partitioned with respect t o Bi cannot be well modeled as a lin-
Fisher matrix of equation (4) is ear combination of variations with respect to other
parameters that we can isolate Bi for low variance es-
J-'(E) = timation. The quadratic form g'(1- pGa)gicharac-
[ IC;T<I - * terizes the interference between parameter Bj and all
u2 :a)~11-'
[Gr(I - PG, )Gz] -'
This is our key finding. From it we may generate the
1.(6)
other parameters. Clearly the addition of any more
parameters to e2 increases the dimension of the sub-
space (Gi) and decreases this norm squared, thereby
making the variance bound larger. The addition of
geometrical interpretations of the following section. extra nuisance parameters can only hurt.

4.0 Geometrical Interpretations

For our geometrical interpretations we return to
the formula for J - ' ( t ) in equation (5) and select the
e
partitions of that best illuminate the result.
Interpretation 1. Let the first partition & be
the single parameter B i , and let the second partition
e, be all remaining parameters except for B i . Then
GIand G2 are

= Gi.
The Fisher matrix is
Interpretation 2. Let's call xi = giei the linear
component of the mean x that is attributable to Bi in

-7-
a Taylor aeries expansion of x in the parameters Ql Expansive de la Vapeur de 1Eau et de la Vapeur
and &. Then the partial derivative of (giBi)T with de lAlcoo1, ir Diffhrentes Tempdratures, J de IEcole
respect to ise Polyiechnique, Paris 1:2 (1795), pp. 24-76.
[2] R. Kumaresan and D. W. Tufts, Estimating
the Parameters of Exponentially Damped Sinusoids
and Pole-Zero Modeling in Noise, IEEE naris ASSP
A S S P - 3 0 : 6 (December 1982), pp. 833-840.
[3] R.Kumaresan, L. L. Scharf, and A. K. Shaw, An
From equation (5), the Cramer-Rao bound on any Algorithm for Pole-Zero Modeling and Spectral Anal-
unbiased estimator of xi is ysis, IEEE l h n s ASSP ASSP-34:3 (June 1986),
pp. 637-640.
-
&[xi xjJ[Xi- 2 J-l(xi),
[4] Y. Bresler and A. Macovski, Exact Maximum
where J-(xi) is Likelihood Parameter Estimation of Superimposed Ex-
ponential Signals in Noise, IEEE lhxns A S S P A S S P -
34:5 (October l986), pp. 1081-1089.
[5] L. L. Scharf, StafisficalSignal Processing (Read-
LoJ ing, MA: Addison-Wesley, 1991), Ch. 6 .
[6] P. Stoica and A. Nehorai, MUSIC, Maximum
Likelihood, and Cramer-Rao Bound, IEEE buns
ASSP ASSP-37:5 (May 1989), pp. 720-741.
This result may be rewritten as [7l L. T. McWhorter and L. L. Scharf, Cramer-
Rao Bounds for Deterministic Modal Analysis, Proc
Asilomar Conf on Signals, Sysiems, and Computers
(November 1991); also accepted for publication in
IEEE R a n s Signal Pmc (December 1991).
where ui and Vi are orthonormal spans of (gi) and
-
(Gi). The value of uT(I ptri)ui is sin*rl,, where [8] D. W.Tufts, F. Giannella, I. Kirsteins, and L. L.
Scharf, Cramer-RBO Bounds on the Accuracy of Au-
rl, is the principal angle [ll] between (gi) and the
subspace (Gi). Therefore, the sole eigenvalue of the toregressive Parameter Estimators, Pmc IEEE Sec-
rank-1 matrix J-(Xi) is & ? / s i n 2rl,. The variance ond Workshop on Spedrum Analysis, Tampa, FL (Nov-
ember 1983).
bound on any unbiased estimator of xi is
[9] T. Wigren and A. Nehorai, Asymptotic Cramer-
. E[?i - xiITIXi - xi] 2 tr J-(x~) Rao Bounds for Estimation of Parameters of Damped
Sine Waves in Noise, IEEE l h n s Signal Pmc 39
--
- U2 (April 1991), pp. 1017-1020.
sin 11, *
[lo] R. T.Behrena and L. L. Scharf, Application
When the subspaces ( g i ) and (Gi) are nearly colinear, of Oblique Projection Operators in Signal Process-
the variance bound is arbitrarily large. The angle rl, ing, submitted t o IEEE Duns Signal Pmc (Novem-
ber 1991).
is illustrated in the figure.
[ll] G. Golub and C. Van Loan, Matriz Compuiafion,
5.0 Conclusion (Baltimore: Johns Hopkins Press, 1990), pp. xx.
Geometrical and filtering interpretations of the [12] S.Anderson, Optimal Dimension Reduction for
Cramer-Rao bound give us a way of interpreting the Sensor Array Processing, Pmc 25th Asilomar Conf
error ellipses generated by the Cramer-Rao bound. on Signals, Sysiems, and Compuiers, Pacific Grove,
These results bring geometrical insights into the re- CA (November 1991), pp. 918-922.
sults on Cramer-Rao bounds published in [6], [8], and
191. In a related paper [7], we have explored these
formulas to obtain finite and asymptotic Cramer-Rao
bounds for deterministic modal analysis.

References:
[l] Baron R. de Prony (Gaspard Riche), Essai Ex-
perimental et Analytique: Sur les Lois de la Dilata-
bilitd de Fluides Elastiques et sur Celle de la Force

-8-