IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 41, NO.

5, MAY 1996 719

[I21 J. F. Zhang, “Comments on robust adaptive regulation with minimal A second example is the identification of the numerator and
prior knowledge,” IEEE Trans. Automat. Cuntr., vol. 39, no. 3, P. 605, denominator coefficients of a rational transfer function model of a
1994. single-input-single-output (SISO) system
[I31 S. M. Naik and P. R. Kumar, “Robust indirect adaptive control of time-
varying plants with unmodeled dynamics and disturbances,” in Pruc.
31th CDC, Tucson, AZ, 1992.
H($, = c;=oa k x k (2)
[ 141 C. Wen, “A robust adaptive controller with minimal modifications for bkxk
discrete time-varying systems,” IEEE Trans. Automat. Contr., vol. 39,
no. 5, pp. 987-991, 1994. where 0 = ( a o , a l , .. . ,a,, bo, b l , . .. , b d ) T and z equals j w or
[I51 G. C. Goodwin and K. S. Sin. AduDtive Filterina. ”. Prediction. and - \“ - , for. remectivelv. continuous and discrete time svstems.
exD(iwT,) , I

Adaptive Control,. Englewood Cliffs,’NJ: Prentice-Hall, 1984. Transfer function model (2) contains a parameter ambiguity (VX E
G. C. Goodwin and D. Q. Mayne, “A parameter estimation perspective R ~ : H ( x ~ , = ~ ( 8w, ) ) which is removed by fixing one
of continuous time model reference adaptive control,” Autumatica, vol.
23, 1987. coefficient of the numerator or denominator (e.g., monic denominator
M. M. Polycqou and P. A. Ioannou, “On the existence and uniqueness polynofial: bd = 1).TO avoid that, a zero coefficient is put equal to
of solution in adaptive control systems,” ZEEE Trans. Automat. Contr., one: it is often numericallv more stable to estimate all the coefficients
vol. 38, no. 3, pp. 474479, 1993. 0 and to scale them afterwards such that I l 8 l l ~= 1. Some examples
R. H. Middleton, G. C. Goodwin, and D. J. Hill, “Design issues in
adaptive control,” IEEE Trans. Automat. Cuntr., vol. 33, pp. 50-58, of estimators using this approach are the total least squares [6],the
1988. generalized total least squares [ 7 ] , the bootstrapped total least squares,
J. K. Hale, Ordinary DifSerential Equations,. New York: Wiley, 1969. and maximum likelihood [ 8 ] . Assuming that the true model order is
P. A. Ioannou and J. Sun, “Theory and design of robust direct and ( n ,d ) , the dimension of the null space of the Fisher infomation
indirect adaptive control schemes,” Int. J. Contr., vol. 47, pP. 775-813, equals one. ~~~~i~~~~of model (2) are, for example, fie roots
1988.
of the numerator and denominator polynomials.

11. MINIMUMVARIANCE BOUNDSFOR

INVARIANTS OF (0VER)PARAMETERIZED MODELS

Minimum Variance Bounds for Overparameterized Models Consider the identification of a particular model out of a model set
M using noisy measurements y E R”. Assume that the model set
R. Pintelon, J. Schoukens, T. McKelvey, and Y. Rolain can be parameterized in an identifiable parameter set $ E Rq with
corresponding regular Fisher information matrix

Abstract-In this paper it is shown that the generalized Cramer-Rao

lower bound on the estimate of invariants of (0ver)parameterized models
is independent of the particular (0ver)parameterizalion chosen and equals
that of the identifiable form. ( f ( y 1 4)is the joint probability function). Assume that it can also be
overparameterized in 8 E RP ( p > q ) with corresponding singular
Fisher information matrix
I. INTRODUCTION
Overparameterized models are models which contain redundant
parameters. They result in singular Fisher information matrices [ 11.
This situation is often encountered in practical parameter estimation such that $(e) is a continuous function of 8 with full rank partial
problems. Consider, for example, the identification of a state-space derivative with respect to 0
model representation of a proper linear time invariant multiple-
input-multiple-output (MIMO) system rank( g G ( 8 ) ) = q.
Xk+i = AXk + Buk (1) Define an invariant of the model set M as any model related quantity
Yk = cxk i-
Duk G which is invariant with respect to all possible parameterizations of
where A E R”’”, B E R n X m , CE R p x n and , D E R p X mfrom , M
measurements of the input uk E R m X 1and output Yk E Rpx’ G ( x )= G ( $ ) vx,$ E M . (6)
signals using subspace identification techniques [2]-[4]. Model (1)
+
contains (n2 n ( p + +
m ) p m ) parameters ( A , B , C , and D Theorem: The Cram&-Rao lower bound on the covariance matrix
of any estimator G ( y ) of an invariant G E C‘ of the model set M
are unknown). Since all possible realizations (1) of a particular
MIMO system are related through a similarity transform T E is independent of the particular (0ver)parameterization chosen.
R n X n (det(T) # 0) [ 5 ] , n2 dependencies exist between these Proofi Thecramer-Rao lower bounds on the covariance matrix
parameters. Assuming that the true model order is n, the dimension of of an estimator G(y) using parameterization 4 or 0 are, respectively,
the null space of the Fisher information matrix equals n 2 . Invariants given by
of model (1) are for example the eigenvalues of A.
Manuscript received August 22, 1995. This work was supported by the
Belgian National Fund for Scientific Research, the Flemish Community
(concerted action IMMI), and the Belgian Government (IUAP 50). and
The author is with Vrije Universiteit Brussel, Department ELEC, 1050
Brussels, Belgium.
Publisher Item Identifier S 0018-9286(96)02806-1.

0018-9286/96$05.00 0 1996 IEEE

 ~

720 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 41, NO 5, MAY 1996

where bG = E{G(y)} - G represents the bias that might be present L7 and V (Lemma 1 of [lo]) which can be written in the complex
in the estimate, and superscript + stands for the Moore-Penrose valued case as
pseudo-inverse [9] (see the Appendix).
Applying the chain rule for the partial derivatives on FO (4) and
G, (8) becomes E { U l i H }2 E { U V H } [ E { V V H } ] + E , { V U H } . (13)

Choosing Lr = G ( g ( y ) ) and V H = $ l n ( f ( y I 4 ( 0 ) ) )in (13),

taking into account that

If we can show that

then the right-hand sides of (7) and (8) are equal which proves the
theorem. and that cov(U) 2 E { U U H } gives
Any matrix M E C a X t ( s 2 t) satisfies M M + M = M [9].
Choosing M = ( g ) T F d (g) gives

= (g>’F+(g). Equation (14) assumes that the necessary regularity conditions to

allow reversal of the order of differentiation and integration are
Left multiplication with (g) and right multiplication with ( g)T fulfilled (Y is the 0-independent range of integration). The suitable
regularity conditions for the existence of the expected values and the
of (11) results in
derivatives can be found in [ll, pp. 300-3021.

REFERENCES
where Q = ($$)(g)T is a regular q by q matrix ((s) has by
A. Shapiro, “Asymptotic theory of overparametrized structural models,”
assumption rank q). Left division by Q F+ and right division by F, Q
J. Amer. StatisficalAssoc., vol. 81, no. 393, pp. 142-149, 1986.
of (12) gives (10). 0 M. Verhaegen, “Identification of the deterministic part of MIMO state
Notes: space models given in innovations form from input-output data,” Aufo-
The theorem has been proven by comparing an overparame- matica, vol. 30, no. 1, pp. 61-74, 1994.
terized parameter set with an identifiable parameter set. By P. Van Overschee and B. De Moor, “N4SID: Subspace algorithms
for the identification of combined deterministic-stochastic systems,”
applying two times the same reasoning, the conclusions are Automatica, vol. 30, no. 1, pp. 75-93, 1994.
also valid when comparing two different overparameterized T. McKelvey and H. A!qay, “An efficient frequency domain state-space
parameter sets. identification algorithm: Robustness and stochastic analysis,” in Proc.
0 Intuitively the theorem can be understood as follows. The 33rd Cont Decision Contr., Lake Buena Vista, FL, Dec. 1994, pp,
3348-3353.
null space of the Fisher information matrix spanned by the
T. Kailath, Linear Systems. New York: Prentice-Hall, 1980.
redundant model parameters is not affected by the noise. Hence S. Van Huffel and J. Vandewalle, The Total Least Squares Problem: Com-
the redundant parameters will not increase the variance of putational Aspects and Analysis, vol. 9. Philadelphia: SIAM, 1991.
identifiable model parameters. J. Swevers, B. De Moor, and H. Van Bmssel, “Stepped sine system
Each identifiable parameter set of a SISO or MIMO model identification errors-in-variables and the quotient singular value de-
composition,” Mechanical Syst. Signal Processing, vol. 6, no. 2, pp.
cannot describe one or more lower dimensional subspaces of the 121-134, 1992.
corresponding full overparameterized parameter set. Example R. Pintelon, P. Guillaume, Y. Rolain, J. Schoukens, and H. Van
model (2) with a monic denominator polynomial cannot describe Hamme, “Parametric identification of transfer functions in the frequency
systems for which b d = 0. Hence the Fisher information matrix domain-A survey,” ZEEE Trans. Automat. Contr., vol. 39, no. 11, pp.
FQ (4) should be constrained, excluding these lower dimensional 2245-2260, 1994.
A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and
subspaces. In [IO] it has been proven under some suitable Applications. London: Wiley, 1974.
regularity conditions that the constrained CramCr-Rao lower J. D. Gorman and A. 0. Hero, “Lower bounds for parametric estimation
bound equals the unconstrained case if the constraints are not with constraints,” IEEE Trans. Inform. Theory, vol. 26, no. 6, pp.
active. We conclude that the proof of the theorem given for the 1285-1301, Nov. 1990.
P. E. Caines, Linear Stochastic Systems. New York: Wiley, 1988.
unconstrained case is still valid for the identification of SISO
and MIMO models if, for the particular identifiable parameter
set chosen, the true system does not lie in a nonreachable lower
dimensional subspace of the full overparameterized set (note that
it is always possible to find such an identifiable set).

APPENDIX:GENERALIZED
CRAMBR-RAOBOUND
The generalized Cram&-Rao bound (8) follows as a special case
of the generalized Cauchy-Schwartz inequality for random vectors