Scribd
Upload
15 views

Selecting A Model and Validation

The document discusses the criteria for selecting and validating models to ensure accurate representation of real systems, emphasizing the balance between bias and variance. It outlines the importance of a priori considerations, data analysis, and a posteriori comparisons for model identification, as well as the necessity of validation against measured data. Additionally, it highlights the significance of experiment design, data processing, and the role of various model structures in system identification.

Uploaded by

Elif
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views

Selecting A Model and Validation

The document discusses the criteria for selecting and validating models to ensure accurate representation of real systems, emphasizing the balance between bias and variance. It outlines the importance of a priori considerations, data analysis, and a posteriori comparisons for model identification, as well as the necessity of validation against measured data. Additionally, it highlights the significance of experiment design, data processing, and the role of various model structures in system identification.

Uploaded by

Elif
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 49

Selecting a model and

validation
What are the criteria to make the above mentioned choices
for the model set in order to obtain a good model for a fair
price?
Depends on the goal for which the model is
obtained.

• Simulate the most important dynamic behavior


• It should be sufficiently accurate to design a (stable) controller
• Related to consistency the model should have a small bias and a
small variance. Conflicting!
• For a model for a real system with a small bias it is necessary to use a
large model set.
☺ Large model set offers the advantage that the fit with the data will
be better.
The variance of the parameters can be large.
 Some of the parameters depends on the specific noise realisation.
overfit.

• For a small variance a small number of parameters is preferred.


A bad fit with the data.
☺ Variance of the identified parameters will be smaller. Better
distinction between stochastic and structured effects in the data.
Models at a fair price?!
• The price may depend on the identification.
- long time to find the optimal parameters from a nonlinear
optimisation criterion.
- Cost a lot of work to investigate many models.
• The price also depends on the intended usage.
- For controller design it may not be necessary a detailed high
frequency behaviour.(too expensive in real time)
Best Model
• A priori considerations, so before experiments are carried out.
• Analysis of the data from an experiment prior to the actual
(parametric) identification.
• A posteriori comparison of the identified models.
The models may be identified in different model sets and/or by
applying different identification criteria (e.g. with and without
prefiltering).
• Validation of the models should give a qualification of the capability
of a model to represent the measured data.
A priori considerations
• Physical insight of the system for minimal model order.

• Nature of noise disturbance.

• The number of data points N should be sufficiently large w.r.t to the


number of number of parameters to be estimated, N ≫ n (N>10n)
Analysis of the data
• Analyse the measurement databefore PEM identification.
-- May be non-parametric identification like spectral analysis.
-- Resonance and anti-resonance peaks for information about the
model order.
• Changes in the phase can also provide information about the
presence of poles and zeros.
• SVD can also be applied using input and output data as in a subspace
identification. (for model order )
A posteriori comparison of the identified
models
• Once a number of parametric models have been identified with e.g.
subspace and/or PEM techniques compare the quality of the models.
!! Recall identification criterion aims at the minimisation of the
(filtered) prediction error in a least squares sense.
--Avoid this overfit by cross-validation. Different data sets for the
identification and for the validation.
√ Use one data set and to split it into two parts.
Validation of the models
• Comparing the identified models with each other isn’t enough.
• Comparison of model with previous results. Use simulated data, to
compare with measured data.
• Model reduction may be necessary. ‘Pole-zero-cancellation’.
• Confidence intervals can be computed. Large confidence intervals
may indicate
(i) too high model order,
(ii) too few data.
• Simulation of the model. If model errors exist large error

For the optimal fit, the simulation error equals only the disturbance.
Again to avoid overfit, cross-validation!!
√ (Residual test) Check that the prediction error is indeed a white
noise signal by,
(a) autocovariance  ( ). (Prediction error small)
(b) evaluating the cross-covariance  ( ). (No correlation btw
prediction error and input signal.
Experiment design
• What are the possible input(s) and output(s) of the system?

• How are these signals being measured?

• At which frequencies do we expect essential dynamic behaviour?

• What are usual and extreme amplitudes of the signals.


Measurement of a step response to
investigate:
• Linearity

• Relevant frequencies

• Static gain

• Selection of input signal.


A “staircase” signal is a sequence of steps that can both increase and
decrease. It can reveal more details of non-linear behaviour and e.g.
hysteresis
Non-parametric identification (correlation
and frequency analysis) to determine
• Duration of the experiment

• Relevant frequencies

• Sample frequency

• Selection of input signal.


Data processing
• Remove outliers or spikes.
• Remove means and drifts, ‘ not included in the (linear) models ’.
• Filtering before and after data acquisition.
→Low-pass filters should be used as an anti-aliasing and/or noise filter
during the measurement.
→The cut-off frequency of an (analog) anti-aliasing filters should match
the original sample frequency.
• Scaling of signals. Very important for MIMO systems.
• A known delay should be removed from the data, ‘ZOH or
measurement equipment ’.
System Identification
Lennart Ljung-Theory for the User
SET 6
– Parametrized model structures
– One-step predictor
– Identifiability

1
Models of LTI Systems

• A complete model

u = input
y = output
e = noise (with PDF).
• A parametrized model

u = input
y = output
e = noise (with PDF of e).
e white noise
One Step Linear Predictor
•

• Want to find that minimizes

• Minimum Prediction Error Paradigm

Data:

Estimate of at time N
One Step Linear Predictor
(Derivation)
The predicted output has the form

Both & have a delay. Past inputs and outputs are mapped
to give the new predicted output.

Notice that:
Now:

has at least has at least


one delay one delay

The lower bound is achieved if z = 0

Equivalently
Result
Examples of Transfer
Function Models

– ARX (Autoregressive with exogenous input)


• Description

• Matched with the model


Examples … ARX
• One step predictor

• Linear Regression

(a function of past data)

• Prediction error
Examples …. ARMAX
– ARMAX (Autoregressive moving average with exogenous input)

• Description

• Standard model

• More general, includes ARX model structure.


Examples …. ARMAX
• One step predictor

or

• Pseudo-linear Regression past predictions

where

or simply
Not linear in
Examples …. OE
• OE (Output Error)

• Description

• One step predictor

• Standard
Examples …. OE
• Nonlinear Regression Vector

Define
Examples …. Box-Jenkins

An even more general model


Examples …. State-Space
– State-Space Models

• Description

disturbance
• Noise: two components
Output noise
Usual assumptions

w, v are white.
• One Step Predictor = Kalman Filter

is a positive semi-definite solution of the steady-state


Riccati equation:

(error covariance)
Innovation Representation

In general
General Notation
Define

It follows

Describes the model structure

Predictor:
For a given

• Notice: can be stable even though is not!

•
Predictor Models
Def: A predictor model is a linear time-invariant stable
filter that defines a predictor

Def: A complete probabilistic model of a linear time-invariant


system is a pair of a predictor model and
the PDF associated with the prediction error (noise).

In most situations, is not complete known. We may


work with means & variances.
Stability Requirements
Example ARX

If has zeros outside the disc, then the map from


is unstable.

is always stable.
Model Sets
Def: A model set is a collection of models

Index set

Examples

all linear models


all models where is FIR of fixed order

a finite set of models

nonlinear fading memory models

Comment: These are “big” sets that are not necessarily


parametrized in a nice way.
Model Structures
Model structures are parametrizations of model sets. We require
this parametrization to be smooth.

Let a model be index by a parameter We require


to be differentiable with respect to
Notice that

Def: A model structure is a differentiable map from a


connected open subset to a model set such that
the gradient, is defined and stable.

is the map
is one particular model
Example: ARX model structure

stable
General Structure
You need

to be differentiable

is stable

(not sufficient).
Model Structures
Proposition: The parametrization

(= set of parameters of A , B , C , D , F )

restricted to the set

has no zeros outside the unit disc

is a model structure
Proof: Follows from the previous general derivative. Notice that
H may be unstable!

Proposition: The Kalman Filter parametrization is a model structure if

(The stability property is equivalent to being detectable).


Independent Parametrization
A model structure m is independently parametrized if

Useful for giving a frequency domain interpretation of the estimate.

• We can define a model set as the range of a model structure:

• We can define unions of different model structures:

Useful for model structure determination!


Identifiability
Central question I: Does the identification procedure yield
a unique parameter ?
Data
Identification procedure
Informative
Identifiability
Question

Def: Model structure is identifiable (globally!) at if


It is locally identifiable at
if there exists an such that
implies

Def: Model structure is strictly identifiable (local or global) if


it is identifiable (local or global) for all
Central question II: Is the identified parameter equal to
the “true parameters” ?

Parametrized structure:
true system

Case I:

Case II:

Define:

Let for some . If is identifiable at ,


then
Identifiability of Model Structures
General:

Identifiable
at

ARX:

If &

&

ARX is strictly identifiable


OE:

Suppose . Then

& iff (B, F ) are coprime.

(To do this cleanly, need to consider where &


are the delay powers in both B & F )
Identifiability
Theorem:

is identifiable at iff

1) There are no common factors of


2) ”
3) ”

You might also like