Advances in Industrial Control

Springer-Verlag London Ltd.

Xue Z. Wang, BEng, MSc, PhD
Department of Chemical Engineering
University of Leeds
Leeds LS2 9JT
UK

British Library Cataloguing in Publication Data

Wang,XueZ.
Data mining and knowledge discovery for process monitoring
and control. - (Advances in industrial control)
1.Process control- Data processing 2.Data mining
1.Title II.McGreavy, C..
629.8'9
ISBN 978-1-4471-1137-5

Library of Congress Cataloging-in-Publication Data

Wang, X. Z. (Xue Zhang), 1963-
Data mining and knowledge discovery for process monitoring and
Control/ Xue Z. Wang.
p. cm. -- (Advances in industrial control)
Inc1udes bibliographical references and index.
ISBN 978-1-4471-1137-5 ISBN 978-1-4471-0421-6 (eBook)
DOI 10.1007/978-1-4471-0421-6
1. Process control-- Data processing. 2. Data mining. 3. Knowledge
acquisition (Expert systems) 1. Title. II. Series.
TSI56.8.W37 1999
670.42'7563--dc21

Advances in Industrial Control series ISSN 1430-9491

ISBN 978-1-4471-1137-5

© Springer-Verlag London 1999

Origina11y published by Springer-Verlag London Berlin Heidelberg in 1999
Softcover reprint of the hardcover 1st edition 1999
Figure 4.3, parts of Sections 4.1.2.1 and 4.1.2.2 and Section 4.2 and reproduced from Kourti T, Process
analysis, monitoring and diagnosis, using projection methods - a tutorial. Chemometrics and
Intelligent Laboratory Systems 28:3-21 (copyright Elsevier 1995).

MATLAB® is the registered trademark ofThe MathW orks, Inc., http://www.mathworks.com

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the
publishers, or in the case of reprographic reproduction in accordance with the terms oflicences issued by
the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent
to the publishers.

The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a
specific statement, that such names are exempt from the relevant laws and regulations and therefore free
for general use.

The publisher makes no representation, express or implied, with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for any errors or
omissions that may be made.

9 8 7 6 543 2 1

springer.com
 XueZ. Wang

Data Mining and Knowledge

Discovery for Process
Monitoring and Control

With 121 Figures

, Springer
Xue Z. Wang, BEng, MSc, PhD
Department of Chemical Engineering, University of Leeds, Leeds. LS2 9JT.

ISBN 978-1-4471-1137-5
British Library Cataloguing in Publication Data
Wang,Xuez.
Data mining and knowledge discovery for process monitoring
and control. - (Advances in industrial control)
1.Process control- Data processing 2.Data mining
LTitle II.McGreavy, C..
629.8'9
ISBN 978-1-4471-1137-5
Library of Congress Cataloging-in-Publication Data
Wang, X. Z. (Xue Zhang), 1963-
Data mining and knowledge discovery for process monitoring and
Control/ Xue Z. Wang.
p. cm. -- (Advances in industrial control)
Includes bibliographical references and index.
ISBN 978-1-4471-1137-5 ISBN 978-1-4471-0421-6 (eBook)
DOI 10.1007/978-1-4471-0421-6
1. Process control-- Data processing. 2. Data mining. 3. Knowledg
acquisition (Expert systems) 1. Title. II. Series.
TSI56.8.W37 1999
670.42'7563--dc2I
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the
publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by
the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent
to the publishers.
© Springer-Verlag London 1999
Originally published by Springer-Verlag London Berlin Heidelberg in 1999
Softcover reprint ofthe hardcover Ist edition 1999
MATLABiI> is the registered trademark ofThe MathWorks,lnc., htţp:llwww.mathworks.com
The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a
specific statement, that such names are exempt from the relevant laws and reguIations and therefore free
for general use.
The publisher makes no representation, express or implied, with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for anyerrors or
omissions that may be made.
Typesetting: Camera ready by author

69/3830-543210 Printed on acid-free paper SPIN 10703341

Advances in Industrial Control

Series Editors

Professor Michael J. Grimble, Professor ofIndustrial Systems and Director

Professor Michael A. Johnson, Professor of Control Systems and Deputy Director
Industrial Control Centre
Department of Electronic and Electrical Engineering
University of Strathc1yde
Graham Hills Building
50 George Street
GlasgowGllQE
United Kingdom

Series Advisory Board

Professor Dr-Ing J. Ackermann

DLR Institut fur Robotik und Systemdynamik
Postfach 1116
D82230 WeBling
Germany

Professor I.D. Landau

Laboratoire d'Automatique de Grenoble
ENSIEG, BP 46
38402 Saint Martin d'Heres
France

Dr D.C. McFarlane
Department of Engineering
University of Cambridge
Cambridge CB2 lQJ
United Kingdom

Professor B. Wittenmark
Department of Automatic Control
Lund Institute of Technology
PO Box 118
S-221 00 Lund
Sweden

Professor D.W. Clarke

Department of Engineering Science
University of Oxford
Parks Road
Oxford OXI 3PJ
United Kingdom
Professor Dr -Ing M. Thoma
Institut fur Regelungstechnik
Universitat Hannover
Appelstr. 11
30167 Hannover
Germany

Professor H. Kimura
Department of Mathematical Engineering and Information Physics
Faculty of Engineering
The University of Tokyo
7-3-1 Hongo
Bunkyo Ku
Tokyo 113
Japan

Professor A.J. Laub

College of Engineering - Dean's Office
University of California
One Shields Avenue
Davis
California 95616-5294
United States of America

Professor J.B. Moore

Department of Systems Engineering
The Australian National University
Research School of Physical Sciences
GPO Box4
Canberra
ACT 2601
Australia

Dr M.K. Masten
Texas Instruments
2309 Northcrest
Plano
TX 75075
United States of America

Professor Ton Backx

AspenTech Europe B.Y.
DeWaal32
NL-5684 PH Best
The Netherlands
 This book is dedicated to
my wife Yanmin for her love
and to
Professor Colin McGreavy, an
influential figure in process control
who died on June 23rd 1999
SERIES EDITORS' FOREWORD

The series Advances in Industrial Control aims to report and encourage

technology transfer in control engineering. The rapid development of control
technology has an impact in all areas of the control discipline. New theory, new
controllers, actuators, sensors, new industrial processes, computer methods,
new applications, new philosophies ... , new challenges. Much of this
development work resides in industrial reports, feasibility study papers and the
reports of advanced collaborative projects. The series offers an opportunity for
researchers to present an extended exposition of such new work in all aspects of
industrial control for wider and rapid dissemination.
How is information technology having an impact on the control,
monitoring and operation of large-scale industrial processes? This monograph
by Xue Wang of Leeds University supplies an in-depth answer to this question.
The text has the starting point that, traditionally, control engineers have
concentrated on system dynamics, measurement selection, control structure
determination and the choice of the controller algorithm to be used. This
ignores the large quantity of data generated by modern plant sensing devices as
a source of information for improved control and enhanced process
monitoring. The difficulty in many cases is the sheer quantity of data arising
and the problem of interrogating, analysing and interpreting this data.
Solutions, some old ... multivariate analysis, and some new ... neural networks
and fuzzy sets, are presented by the author along with illustrative examples
usually based on real industrial processes and data. This very instructive
monograph will be of interest to the practising industrial process engineer for
its insights and to the academic control community for its industrial
perspective.

M.J. Grimble and M.A. Johnson

Industrial Control Centre
Glasgow, Scotland, UK
PREFACE

Being able to collect and display to operators a large amount of information is

regarded as one of the most important advances provided in distributed control
(DCS) over earlier analogue and direct digital control systems. The data are used by
plant operators and supervisors to develop an understanding of plant operations
through interpretation and analysis. It is this understanding which can then be used
to identify problems in current operations and find better operational regions which
result in improved products or in operating efficiency.
It has long been recognised that the information collected by DCS systems tends
to overwhelm operators and so makes it difficult to take quick and correct decisions,
especially in critical occasions. For example, olefin plants typically have more than
5000 measurements to be monitored, with up to 600 trend diagrams. Clearly there is
a need to develop methodologies and tools to automate data interpretation and
analysis, and not simply rely on providing the operators large volumes of
multivariate data. The role of the acquisition system should be to provide the
operators with information, knowledge, assessment of states of the plant and
guidance in how to make adjustments. Operators are more concerned with the
current status of the process and possible future behaviour rather than the current
values of individual variables.
Process monitoring tends to be conducted at two levels. Apart from immediate
safe operation of the plant, there is also the need to deal with the long term
performance which has been the responsibility of supervisors and engineers. The
databases created by automatic data logging provide potentially useful sources of
insight for engineers and supervisors to identify causes of poor performance and
opportunities for improvement. Up to now such data sources have not been
adequately exploited.
XII

The above roles of plant operators and supervisors imply that they are an integral
part of the overall control system. The current approach to designing control
systems has not adequately addressed this point. It is done mainly in terms of
identifying the process dynamics, selecting measurements, defining control
structures and selecting algorithms.
This book introduces development in automatic analysis and interpretation of
process operational data both in real-time and over the operational history, and
describes new concepts and methodologies for developing intelligent, state space
based systems for process monitoring, control and diagnosis. It is known that
processes can have multiple steady and also abnormal states. State space based
monitoring and diagnosis can project multivariate real-time measurements onto a
point in the operational state plane and monitor the trajectory of the point which can
identify previously unknown states and the influence of individual variables. It is
now possible to exploit data mining and knowledge discovery technologies to the
analysis, representation, and feature extraction of real-time and historical
operational data to give deeper insight into the systems behaviour. The emphasis is
on addressing challenges facing interpretation of process plant operational data,
including the multivariate dependencies which determine process dynamics, noise
and uncertainty, diversity of data types, changing conditions, unknown but feasible
conditions, undetected sensor failures and uncalibrated and misplaced sensors,
without being overwhelmed by the volume of data.
To cover the above themes, it is necessary to cover the following topics,
• new ways of approaching process monitoring, control and diagnosis
• specification of a framework for developing intelligent, state space based
monitoring systems
• introduction to data mining and knowledge discovery
• data pre-processing for feature extraction, dimension reduction, noise
removal and concept formation
• multivariate statistical analysis for process monitoring and control
• supervised and unsupervised methods for operational state identification
• variable causal relationship discovery
• software sensor design
The methodologies and concepts are illustrated by considering illustrative examples
and industrial case studies.

Xue Z. Wang
Leeds, England, 1999
ACKNOWLEDGEMENTS

The author wishes to acknowledge the invaluable contribution of the late Professor
Colin McGreavy in the preparation and editing of this text.
The author would like to thank his present and fonner colleagues and students at
the University of Leeds, Drs F.Z. Chen, B.H. Chen, S.H. Yang, Y.C. Huang, Mr.
R.F. Li, Ms B. Yuan, Dr. S.A. Yang, Mr. R. Garcia-Flores, Mr. D. Sampson, and
Mr. A. Watson who have provided much help with this book. He acknowledges
gratefully Dr. M.L. Lu at Aigis Systems USA who has given useful input
continuously. Thanks are also due to Dr. G.G. Goltz for his friendship and help in
many ways.
Particular thanks go to Professors M.A. Johnson and M. 1. Grimble at the
Industrial Control Centre, University of Strathclyde for their encouragement.
Professor M.A. Johnson has read the manuscript and given very useful input. The
author would like to extend his thanks to Mr. O. Jackson, Ms E. Fuoco and Mr. N.
Pinfield at Springer-Verlag London for their successful cooperation and help.
The author is grateful to the University of Leeds for providing the supporting
facilities in the preparation of the book. The book has also benefited from an
EPSRC grant (GRlLl61774).
CONTENTS

1 Introduction ................................................................. 1

1.1 Current Approaches to Process Monitoring, Diagnosis

and Control................................................................. 2
1.2 Monitoring Charts for Statistical Quality Control.. ... 4
1.3 The Operating Window......................... ...... ........... ... 5
1.4 State Space Based Process Monitoring and Control... 6
1.5 Characteristics of Process Operational Data ............... 8
1.6 System Requirement and Architecture ....................... 9
1.7 Outline of the Book ... ...... .......... ..... .............................. 11

2 Data Mining and Knowledge Discovery - an

Overview ........................................................................ 13

2.1 Definition and Development ...................................... 13

2.2 The KDD Process ............................................................ 15
2.3 Data Mining Techniques .............................................. 17
2.4 Feature Selection with Data Mining ............................ 25
2.5 Final Remarks and Additional Resources ................... 27
XVI

3 Data Pre-processing for Feature Extraction,

Dimension Reduction and Concept
Formation ........................................................................ 29

3.1 Data Pre-processing ....................................................... 29

3.2 Use of Principal Component Analysis ........................... 30
3.3 Wavelet Analysis .......................................................... 38
3.4 Episode Approach .......................................................... 57
3.5 Summary ........................................................................ 59

4 Multivariate Statistical Analysis for Data

Analysis and Statistical Control .......................... 61

4.1 PCA for State Identification and Monitoring ..... ...... .... 61
4.2 Partial Least Squares (PLS) ................ ........................... 68
4.3 Variable Contribution Plots .......................................... 68
4.4 Multiblock PCA and PLS .............................................. 72
4.5 Batch Process Monitoring Using Multiway PCA ......... 72
4.6 Nonlinear PCA ............................................................... 74
4.7 Operational Strategy Development and Product Design
- an Industrial Case Study ................................................ 76
4.8 General Observations ........... ........................................... 82

5 Supervised Learning for Operational

Support .............................................................................. 85

5.1 Feedforward Neural Networks ...................................... 85

5.2 Variable Selection and Feature Extraction for
FFNN Inputs .................................................................. 91
5.3 Model Validation and Confidence Bounds .................. 92
5.4 Application ofFFNN to Process Fault Diagnosis ......... 94
5.5 Fuzzy Neural Networks ................................................. 99
 XVII

5.6 Fuzzy Set Covering Method ...................................... 101

5.7 Fuzzy Signed Digraphs ............................................... 102
5.8 Case Studies ...................................................................... 108
5.9 GeneralObservations ...................................................... 116

6 Unsupervised Learning for Operational State

Identification ..................................................................... 119

6.1 Supervised vs. Unsupervised Learning ...................... 119

6.2 Adaptive Resonance Theory ........................................ 120
6.3 A Framework for Integrating Wavelet Feature Extraction
andART2 ..................................................................... 124
6.4 Application of ARTnet to the FCC Process ............... 126
6.5 Bayesian Automatic Classification ........................... 136
6.6 Application of AutoClass to the FCC Process ............. 142
6.7 General Comments ....................................................... 147

7 Inductive Learning for Conceptual Clustering

and Real-time Process Monitoring ................. 149

7.1 Inductive Learning ...................................................... 150

7.2 IL for Knowledge Discovery from Averaged Data .... 153
7.3 IL for Conceptual Clustering and Real-time
Monitoring ..................................................................... 156
7.4 Application to the Refinery MTBE Process ............... 164
7.5 General Review .......................................................... 172

8 Automatic Extraction of Knowledge Rules

from Process Operational Data ............................. 173

8.1 Rules Generation Using Fuzzy Set Operation ............ 173

8.2 Rules Generation from Neural Networks .................. 176
XVIII

8.3 Rules Generation Using Rough Set Method ................ 178

8.4 A Fuzzy Neural Network Method for Rules
Extraction ...................................................................... 180
8.5 Discussion ...................................................................... 190

9 Inferential Models and Software Sensors .... 193

9.1 Feedforward Neural Networks as Software Sensors .. 193

9.2 A Method for Selection of Training I Test Data and
Model Retraining ........................................................ 194
9.3 An Industrial Case Study ............................................ 196
9.4 Dimension Reduction ofInput Variables ................... 204
9.5 Dynamic Neural Networks as Inferential Models ....... 210
9.6 Summary ........................................................................ 212

10 Concluding Remarks ................................................ 215

Appendix A The Continuous Stirred Tank

Reactor (CSTR) .......................................................... 217

Appendix B The Residue Fluid Catalytic

Cracking (R-FCC) Process ................................... 221

Appendix C The Methyl Tertiary Butyl Ether

(MTBE) Process ......................................................... 227

References ................................................................................. 233

Index .......................................................................................... 249

CHAPTER!
INTRODUCTION

Over the last twenty years, it has become increasingly obvious that the perfonnance
of process control systems depends not only on the control algorithms but also on
how these integrate into the operational policy in terms of safety, environmental
protection, equipment protection, as well as general monitoring to identify poor
performance and detect faults. This calls for creating a virtual environment which is
able to provide a comprehensive assessment of performance and can identify the
factors which determine it. The important agents which are responsible for bringing
these together are the plant operators and supervisors. This implies that it is
important to consider them as part of the overall solutions, and if they are part of the
system they must be provided with the means of carrying out the role effectively.
While attention has been given to improving the interface of control systems for
operators and supervisors through the design of information display and alarm
systems, most of this is concerned with awareness, with little concern as to
processing functionality in assessing the large volume of multivariate data more
effectively.
This book addresses these issues by seeking to make use of emerging data mining
and knowledge discovery technology (KDD) to develop approaches for designing
state space based process monitoring and control systems so as to integrate plant
operators and supervisors into the operational strategy. Modem computer control
and automatic data logging systems create large volumes of data, which contain
valuable information about normal and abnormal operations, significant
disturbances and changes in operational and control strategies. The data
unquestionably provides a useful source for supervisors and engineers to monitor

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
2 Data Mining and Knowledge Discovery for Process Monitoring and Control

the perfonnance of the plant and identify opportunities for improvement and causes
of poor perfonnance. The volume of data is generally so large and data structure too
complex for it to be used for characterisation of behaviour by manual analysis. Data
mining and KDD offers the capability of extracting knowledge from such data
which can be used for developing state space based process monitoring systems.
The first chapter reviews the current approaches to process monitoring with
particular reference to distributed control systems (DeS) displays, monitoring charts
for statistical quality control, and the use of a concept of the operating window.
This leads naturally to the concept of state space based process monitoring and
control which is well suited for defining operators and supervisors requirements.
Based on this, it is possible to develop a conceptual architecture for system design.

1.1 Current Approaches to Process Monitoring,

Diagnosis and Control

To meet the goals of process control requires monitoring and control of process
variables such as temperature, pressure, flow, composition and the levels in vessels.
In modem distributed control systems, to assist in process monitoring and fault
diagnosis, the measurements are displayed as groups of variables, dynamic trends
and alanns. To do this effectively requires careful consideration as to the design of
the displays.
A first requirement is to collect and display as much infonnation as possible
which is relevant. In fact, being able to collect and provide access to a large amount
of infonnation of measurement of variables is regarded as one of the most important
advances of DeS when compared with earlier systems [1]. In the case of a typical
olefin plant, for example, over 5000 measurements need to be monitored [2].
Secondly, the display should be arranged in a way that makes it easy for operators
to assimilate the measured values, obtain associated infonnation about key variables
provided by independent sensors and diagnose incipient problems, preferably before
causing major upsets. This requires that the interface display to operators should be
properly designed. The display to operators in a modem Des system has a
hierarchical structure as shown in Figure 1.1 [1]. This arrangement allows engineers
and operators having varied responsibilities to access infonnation at different levels
in a convenient way. For example, supervisors might be mainly interested in the
general perfonnance of the plant in tenns of operating capacity, energy efficiency,
overall mass balances and nonnal and abnonnal status of plant areas or units. On the
 Chapter 1 Introduction 3

other hand, operators are more concerned with one section of a plant and monitoring
the associated variables.
Other features of DCS displays make it possible for operators to quickly
assimilate data using colours to indicate different operational states. For example,
green is used for normal and red for abnormal. Care is also needed in the location of
displays, grouping of variables and grouping and sequencing of alarms.

DISPLAY HIERARCHY TYPES

OF DISPLAYS

Plant Status Display

Process Variable Overview

Deviation Overview
Area Graphics
Alann Summary

WORKING DISPLAYS
Station Mimics
Trend Displays
Control Graphics
Batch Sequences
Operator Guides

X-Y Plots
Tuning Display

Figure 1.1 Typical DCS display hierarchy.

This emphasis again draws attention to the fact that supervisors and operators are
part of the overall control system. In fact, not only are they responsible for many
feedback control tasks which are not automated, such as switching feedbacks, but
also they must undertake supervision of the general strategy and seek to develop an
understanding of the process performance both in the short- and long- term. This
understanding can be used to identify [3] :

• problems in the current operation.

• deteriorating performance of instruments, energy usage, equipment, or catalysts.
• better operating regions leading to improved product or operating efficiency.
 4 Data Mining and Knowledge Discovery for Process Monitoring and Control

Such tasks require operators to be able to not only access the data at the right time
but more importantly to assimilate and assess the data quickly and correctly,
especially when abnormal conditions arise. This is a very challenging task because
the volume of data is generally very large: large scale plants have as many as 20,
000 variables which are measured continuously [4, 5, 6]. Moreover, the data are
multivariate and are interrelated so there is a need to make evaluations
simultaneously. Humans are not able to simultaneously analyse problems involving
more than three variables very effectively and this becomes more difficult when the
data are corrupted by noise and uncertainty. The need to provide computer
assistance in assimilating data has now become a major concern and it is important
that automatic data analysis systems should be developed and integrated with DCS
control.

1.2 Monitoring Charts for Statistical Quality

Control

Automatic process control compensates for the effects of disturbances and maintains
the controlled variable at the desired value. However, it does not eliminate the
course of poor operation. Since the sources of disturbances have not been
eliminated, leaving the process susceptible to future disturbances from the same
source. Statistical process control (SPC) has the goal of detecting and eliminating
disturbance. SPC works in conjunction with automatic control and monitors the
performance of a process over time in order to verify that the process meets the
quality requirements [22]. SPC charts such as Shewhart [7, 8, 9] , cumulative sum
(CUSUM) [10] or exponentially weighted moving average (EWMA) charts [11, 12]
are used to monitor the key product variables in order to detect the occurrence of
any event having a special or assignable cause. In the case of assignable causes,
long-term improvements can be made in process operations which enhance product
quality.
Figure 1.2 is an example of a Shewhart chart. It shows the variation in a single
variable against the statistical mean and the upper and lower limits defining an
acceptable quality band. The mean and the upper and lower limits are obtained by
statistical analysis of historical data. If the measured value falls outside either of the
limits, a diagnostic analysis is carried out to determine the assignable cause so that
appropriate corrective action can be taken.
 Chapter 1 Introduction 5

Most SPC methods record only a small number of variables, usually the final
product quality variables which are examined sequentially. Such an approach is now
not acceptable for modem processes. Modem computer based monitoring collects
massive amounts of data continuously. Variables such as temperatures, pressures,
flowrates etc. are typically measured every seconds although only a few of the
underlying events drive the process at any time: in fact, all measurements are simply
aspects of the same underlying events. Examining one variable at a time, is of
limited value. There is clearly the need to develop methods to examine a number of
variables simultaneously.

Upper control limit

)()(
)()( )()(
)()( )()( Centre line Mean
")O(~ ~

)()(
)()( )()(
)()(

Lower control limit

)()( )()(
)()(
)()(

ommon cause --.

L-----------------------or------------- Time
Special cause

Figure 1.2 An example of Shewhart chart.

1.3 The Operating Window

The use of an operating window enables several variables to be considered together

and is useful in monitoring an allowable region. It covers a region of possible
steady-states for a process limited by constraints such as safety, product quality and
equipment performance [8]. An example of an operating window is shown in Figure
1.3, where the region bounded by the solid curve represents the allowable steady-
states. The dashed locus (or trajectory) shows that during a disturbance the
6 Data Mining and Knowledge Discovery for Process Monitoring and Control

operating point can move outside of the steady-state region. Operating windows
have not been adopted in DeS display design as widely as they ought to have and
there is still the problem of dimension limitation. Systems aiming to replace or help
operators in process status assessment require capabilities for dealing with much
higher dimensionality.

480

_...... '" " ..........

........
380 ~ _ _.c,••••• __ ••• .
...... ,

300~--~----~----~----~----~--~
o 0.1 0.2 0.3 0.4 0.5
Reactantconcenuation

Figure 1.3 An example of operating window and trajectory.

1.4 State Space Based Process Monitoring and

Control

In practice, operators are usually more concerned with the current operational status
and evolving patterns of behaviour, rather than the instant values of specific
variables. The judgement on the operational states requires the simultaneous
assimilation of the multivariate process variables. In doing this, operators are
actually projecting the process to a single point of a state space. It is clear that the
above described monitoring charts and operating windows are still far less powerful
than what is required. The concept of state space based monitoring is helpful in
clarifying the ideas behind this book. Figure 1.4 illustrates how a process can be
operated in various normal and abnormal operational states or modes that have
different characteristics in terms of operability and controllability, safety and flow
 Chapter I Introduction 7

..
Abnormal
operation
region ·4

Figure 1.4 lllustration of state space based operational plane.

patterns among other things. The accumulation of know-how derived from previous
experience and from computer simulation makes it possible to gain better insight
into the operational behaviour of equipment. A state space based monitoring system
can therefore be identified to have the following functions.
It should be able to automatically assimilate the real-time measurements, project
the process to a specific state and indicate the path of the operating point in the state
space in real-time. It is now known that a process or a unit operation can operate at
abnormal and multiple steady states. The best known example is the exothermic
continuous stirred tank reactor (CSTR) with a cooling water jacket [8, 13]. Multiple
steady-state behaviour is also possible in distillation columns [14], reactive
distillation processes [15], and refinery fluid catalytic cracking processes [16].
Changes of product specifications and feedstock properties, which are very common
today [17], as well as large disturbances may also drive a process to a different
operating state. These operational states are not obvious without careful analysis.
Abnormal operations can take various forms and are more difficult to predict.
Nevertheless, with the gradual accumulation of knowledge, more insight is being
generated about operational behaviour patterns. It is also important to be able to
identify various unfamiliar operational states, whether normal or abnormal.
 8 Data Mining and Knowledge Discovery for Process Monitoring and Control

Moreover it should be possible to identify the most important variables which are
responsible for changes in operational states and so provide guidance for operators
in adjusting the process. While the schematic in Figure 1.4 is yet another facet of
system characteristics, it will not replace the traditional DCS display. It provides a
different perspective of system behaviour.

1.5 Characteristics of Process Operational Data

The major challenge in developing the kind of state space based system described
above arises from the characteristics of operational data, which are summarised as
follows:

• Large volume. A DCS automatic data logging system continuously stores data.
The large volume makes manual probing almost impossible. Large volumes of
data also demand large computer memory and high speed.
• High dimensionality. The behaviour of a process is usually defmed by a large
number of correlated variables. As a result it is difficult to visualise the
behaviour without dimension reduction.
• Process uncertainty and noise. Uncertainty and noise emphases the need for
good data pre-processing techniques.
• Dynamics. In operational status identification, it is very important to take
account of the dynamic trends. In other words, the values of variables are
dynamic trends. Many data mining and knowledge discovery tools, such as the
well-known inductive machine learning system C5.0 [17, 18], are mainly
designed to handle categorical values such as a colour being red or green. They
are not effective in dealing with continuous-valued variables. These tools are not
able to handle variables that take values as dynamic trends.
• Difference in the sampling time of variables. On-line measurements and
laboratory analyses have variable sampling periods.
• Incomplete data. Some important data may not be recorded.
• Small and stale data. Sometimes, data analysis is used to identify abnormal
operations. The data corresponding to abnormal operations might be buried in a
huge database. Some tools are not effective in identifying small patterns in a
large database.
• Complex interactions between process variables. Many techniques require that
attributes be independent. However, many process variables are interrelated.
 Chapter 1 Introduction 9

• Redundant measurements. Sometimes several sensors are used to measure the

same variable, which gives rise to redundant measurements.

Current methods only address some of these issues, certainly not all and the
following observations can be made:
(1). Data pre-processing is critical for various reasons including noise removal,
data reconciliation, dimension reduction and concept formation.
(2). Effective integration of the tools is needed. It means combining various tools
for data preparation for other tools or for validation.
(3). Validation of discoveries from the data and presentation of the result is
essential. Many times, because of lack of knowledge about the data, interpretation
becomes a major issue.
(4). Windowing and sampling from a large database for analysis. This is necessary
particularly for analysis of historical operational data.

1.6 System Requirement and Architecture

To develop the kind of state space based monitoring and control environment
described above calls for an integrated data mining and KDD system. The system
should have the following functions: (1) identification of operational states; (2)
projection of the operation to a single point of the operational plane; and (3) giving
explanations on the major variables that are responsible for the projection and
providing guidance on adjustment. A conceptual system architecture and associated
components are shown in Figure 1.5. It is important for a system to be able to
provide some basic functions and be flexible enough to be tailored to meet special
purposes [21]. The basic functions include:

• Pattern discovery. Grouping data records into clusters and then analysing the
similarities and dissimilarities of data between clusters is an important starting
point for analysis. An obvious example is identifying abnormal conditions.
• Trend and deviation analysis. Various technologies for trend and deviation
analysis are available including statistics and calculation of mean and standard
deviation.
• Link and dependency analysis. The link and dependency between performance
metrics is important in understanding process behaviour and improving
performance. Some existing data mining methods such as the inductive learning
10 Data Mining and Knowledge Discovery for Process Monitoring and Control

approach CS.O [18, 19, 20], as well as many graphical tools, are not able to be
applied because of the real-valued dynamic trends and interactions between
variables.
• Summarising. This provides a compact description of a subset of data, such as
the mean and standard deviation of all fields. More sophisticated techniques use
summary rules, multivariate visualisation techniques, and functional
relationships between variables.
• Sequence analysis. Analysing models of sequential patterns (e.g., in data with
time dependence, such as time series analysis) aims at generating the sequence
or extracting report deviations and trends over time. A typical example is in
batch process operations.
• Regression. This is required for predictive model development, as in the case of
software sensor models.

Historical
Database

Supervised Unsupervised
Data "r<~-pl'oclessiinl!:1 classification tools classification tools
- Wavelets - FFNN -ARTI
- Statistics methods - AutoClass
- Fuzzy set covering - PCA
- Episode approac ...
-peA
Dependency modelling Others:
• Dependency discovef) - Visualisation
- Bayesian graph - Regression
Experience - Summarising
- Fuzzy SDU
- Rules extraction
- C5.0
...
Qualitative
models
Integrated Data mining and KDD System

Figure 1.5 The conceptual system architecture.

 Chapter 1 Introduction 11

1.7 Outline of the Book

The rest of the book is organised as follows. In Chapter 2 the methodologies and
tools for data mining and KDD are briefly reviewed. Chapter 3 focuses on data pre-
processing for the purpose of feature extraction, dimension reduction, noise removal
and concept formation. The emphasis is on processing of dynamic trend signals
which are considered as the most important information in operational state
identification. Three approaches are introduced, namely principal component
analysis, wavelet analysis, and episode representation.
Multivariate statistical analysis methods are introduced in Chapter 4 for analysis
of process operational data and developing multivariate statistical control charts.
These include linear and nonlinear principal component analysis (PCA), partial least
squares (PLS), and multiblock PCA. Examples are used to illustrate the approaches
and an industrial case study is described which uses PCA to discover knowledge
from data for operational strategy development and product design.
Supervised machine learning approaches are discussed in Chapter 5 for
identification of process operational states. While the focus is put on feedforward
neural networks (FFNNs), other methods are also introduced and compared with
FFNNs, including fuzzy FFNNs, single layer percetron, fuzzy set covering method
and fuzzy signed digraphs.
Supervised machine learning requires data with known classification as training
data and therefore learns from known to predict unknown, while unsupervised
approaches do not require training data therefore are able to learn from unknown.
Chapter 6 is devoted to unsupervised approaches for identification of operational
states. An integrated framework (ARTnet) combining unsupervised neural network
ART2 with wavelet feature extraction is developed, which uses wavelet as the
substitute of the data pre-processing part of ART2. It is shown that ARTnet is
superior over ART2 in avoiding the adverse effect of noise and is faster and more
robust. A Bayesian automatic classification approach (AutoClass) is also described.
The advantage of the approach is that it does not require the users to give any input:
the system determines the classification scheme automatically. A refinery fluid
catalytic cracking process is used to illustrate the approaches.
Most approaches for operational state identification, whether supervised or
unsupervised, are based on calculating a distance or similarity measure. They give
the classification but not causal explanations. Conceptual clustering which is
introduced in Chapter 7, on the other hand is able to create a conceptual clustering
12 Data Mining and Knowledge Discovery for Process Monitoring and Control

language as well as giving a prediction of operational states. An inductive learning

approach is introduced in this chapter for conceptually clustering operational states.
The methods introduced in Chapter 8 are able to identify cause-effect links
between variables and between variables and operational states, as well as automatic
discovery of operational rules from process operational data. Fuzzy sets, rough sets,
neural networks and fuzzy neural networks are introduced for automatic generation
of rules from data while the emphasis is on fuzzy neural networks.
Software sensor design for real-time monitoring of hard-to-measure variables and
inferential control is examined in Chapter 9. The approach is based mainly on the
use of neural networks. The main concerns are with selection of training and test
data, selection of input variables as well as validation of data and models.
Chapter 10 closes the book with a brief summary of important issues and
suggestions for future work.
CHAPTER 2
DATA MINING AND KNOWLEDGE DISCOVERY
-AN OVERVIEW

2.1 Definition and Development

The emerging of data mining and knowledge discovery in databases (KDD) as a

new technology is due to the fast development and wide application of information
and database technologies. With the increasing use of databases the need to be able
to digest large volumes of data being generated is now critical. It is estimated that
only 5%-10% of commercial databases have ever been analysed [23]. As Massey
and Newing [24] indicated that database technology has been successful in
recording and managing data but failed in the sense of moving from data processing
to making it a key strategic weapon for enhancing business competition. The large
volume and high dimensionality of databases leads to the breakdown of traditional
human analysis. Data mining and KDD is aimed at developing methodologies and
tools to automate the data analysis process and create useful information and
knowledge from data to help in decision making (Figure 2.1). A widely accepted
definition is given by Fayyad et al. [25] in which KDD is defined as the non-trivial
process of identifying valid, novel, potentially useful, and ultimately understandable
patterns in data. The definition regards KDD as a complicated process comprising a
number of steps and data mining is one step in the process.
The goal of data mining and KDD is very broad and can describe a multitude of
fields of study related to data analysis. It is known that statistics has been
preoccupied with this goal for over a century. Other fields related to data analysis

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
 14 Data Mining and Knowledge Discovery for Process Monitoring and Control

Value

Information

Data
Volume

Figure 2.1 From volume to value.

Panerns

I
.. Transformed

r .t.
Data

Prepro~esscd 1
o I
Targe. 0 at. ! I
L...........~ ... ~... "u.................. ~j. . . . ". . . . .". H .. U !....
. . . . . . W• • H T
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . ." . . . . . . . . . . . . . . . . . . . . .

Figure 2.2 An overview of the steps comprising the KDD process.

include statistics [26, 27], data warehousing [28, 29, 30], pattern recognition,
artificial intelligence and computer visualisation. Data mining and KDD draws upon
methods, algorithms and technologies from these diverse fields, and the unifying
goal is extracting knowledge from data.
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 15

50%
Cleaning
cApJotil'l:f
S.Ie: c1lft t Inter'HtU~G
tbe Data and
Evalu tin 9
the ,te n It.
hopi>;
tbt Prepating
Ptl3bl&m E xplomng
the Data
th.
:P.ull&

0%

Figure 2.3 Relative effort of a data mining project.

Over the last ten years data mining and KDD has been developing at a dramatic
speed. In Information Week's 1996 survey of the 500 leading information
technology user organisations in the US, data mining came second only to the
Internet and intranets as having greatest potential for innovation in information
technology. The rapid progress is reflected not only by the establishment of research
groups on data mining and KDD in many international companies, but also by the
investment from banking, telecommunication and marketing sectors.
Figure 2.2 provides an overview of the activities involved in KDD and Figure 2.3
shows the typical distribution of effort.

2.2 The KDD Process

Data mining and KDD is a very complex process, typically involving the following
procedures (Figure 2.4) [23, 34]:

(1). Developing an understanding of the application domain, the relevant prior

knowledge and the goals of the end-user.
(2). Creating a target data set: selecting a data set, or focusing on a subset of
variables or data samples, on which discovery is to be performed.
(3). Data pre-processing and cleaning: basic operations such as the removal of
noise or outliners if appropriate, collecting the necessary information to model and
 16 Data Mining and Knowledge Discovery for Process Monitoring and Control

account for noise, deciding on strategies for handing missing data fields, and
accounting for time sequence information and known changes.

Data organised
by function

Eliminate
noisy data

Transform
values

Transform to
different
re resentation

Figure 2.4 The KDD process.

(4). Data reduction and projection: finding useful features to represent the data
depending on the goal of the task. Using dimensionality reduction or transformation
methods to reduce the effective number of variables under consideration or to find
invariant representations for the data.
(5). Choosing the data-mining task: deciding whether the goal of the KDD process
is logical, summarising, classification, regression, prediction, and clustering etc.
(6). Choosing the data analysis algorithm(s): selecting methodes) to be used for
searching for patterns in the data. This includes deciding which models and
parameters may be appropriate (e.g., models for categorical data are different from
models on vectors over the real domain) and matching a particular data mining
method with the overall criteria ofKDD process.
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 17

(7). Data mining: searching for patterns of interest in a particular representational

form or a set of such representations, including classification rules or trees,
regression, clustering, sequence modelling, dependency, and so forth. The user can
significantly aid the data mining method by correctly performing the preceeding
steps.
(8). Interpreting mined patterns, and possible return to any of the previous steps.
(9). Consolidating the discovered knowledge: incorporating this discovery
knowledge into the performance system, taking actions based on the knowledge, and
reporting it to interested parties. It also includes checking or testing for potential
conflicts with previously believed (or extracted) knowledge.

Data mining and KDD is potentially valuable in virtually any industrial and
business sectors where database and information technology is used. The following
are some reported applications:

• Fraud detection: identify fraudulent transactions

• Loan approval: establish the credit worthiness of a customer requesting a loan
• Investment analysis: predict a portfolio's return on investment [32]
• Portfolio trading: trade a portfolio of fmancial instruments by maximising
returns and minimising risks.
• Marketing and sales data analysis: identify potential customers; establish the
effectiveness ofa sales campaign [32].
• Manufacturing process analysis: identify the causes of manufacturing problems.
• Experiment result analysis: summarise experiment results and predictive models.
• Scientific data analysis [35].
• Intelligent agents and WWW navigation.

2.3 Data Mining Techniques

Data mining methods and tools can be categorised in different ways [25, 33, 28, 37].
According to functions and application purposes, data mining methods can be
classified as clustering, classification, summarisation, dependency modelling, link
analysis and sequence analysis. Some methods are traditional and established and
some are relatively new. In the following a very brief review of the techniques is
given.
 18 Data Mining and Knowledge Discovery for Process Monitoring and Control

2.3.1 Clustering

Given a number of data patterns (sometimes called instances, cases, observations,

samples, objects, or individuals) as shown in Table 2.1, each of which is described
by a set of attributes, clustering (also called unsupervised machine learning) aims to
devise a classification scheme for grouping the objects into a number of classes such
that instances within a class are similar, in some respect, but distinct from those
from other classes. This involves determining the number as well as the descriptions
of classes. The grouping often depends on calculating a similarity or distance
measure. Grouping multivariate data into clusters according to similarity or
dissimilarity measures is the goal of some applications. It is also a useful step to
look at the data before further analysis is carried out. The methods can be further
categorised according to requirement on prior knowledge of the data. Some methods
require the number of classes to be an input though the descriptions of the classes
and assignments of individual data cases are allowed to be unknown. For example,
the Kohonen neural network [36, 38] is designed for this purpose. In some other
methods, neither the number nor descriptions of classes are required to be known.
The task is to determine the number and descriptions of classes as well as the
assignments of data patterns. For example, the Bayesian automatic classification
system - AutoClass [39, 40, 41, 42] and the adaptive resonance theory (ART2) [43]
are designed for this purpose.

.
T abl e 21 Anexamp eo fd ata structure.
Instances Attributes
1 2 ... ... j ...... m

Xl XlI XI2 XIi Xlm

X2 X21 X22 X2i X2m

Xi Xu Xi2 Xii Xim

X. X.I X.2 X." X.m

As a branch of statistics, clustering analysis has been studied extensively for many
years, mainly focused on distance-based clustering analysis, such as using the
Euclidean distance. There are many text books on this topic [44,45,46]. A notable
progress in clustering has been in unsupervised neural networks, including the self-
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 19

organising Kohonen neural network [36, 38] and the adaptive resonance theory
(ART) [47,48,43,49,50,51]. There have been many reports on the application in
operational state identification and fault diagnosis in process industries.

2.3.2 Classification

For a given number of data patterns such as those shown in Table 2.1, if the number
and descriptions of classes as well as the assignments of individual data patterns are
known, the task is to assign unknown data patterns to the established classes, the
task belongs to classification. The most widely currently used classification
approach is based on feedforward neural networks (FFNNs). Classification is also
called supervised machine learning because it always requires data patterns with
known class assignments to train a model which is then used for predicting the class
assignment of new data patterns.

2.3.3 Conceptual Clustering and Classification

Most clustering and classification approaches depend on numerically calculating a

similarity or distance measure and because of this they are often called similarity
based methods. The knowledge used for classification assignment is often an
algorithm which is opaque and essentially a black box. Conceptual clustering and
classification on the other hand develops a qualitative language for describing the
knowledge used for clustering and is basically in the form of production rules or
decision trees which are explicit and transparent. The inductive system C5.0
(previously C4.5) is a typical approach [18, 19, 20], which is able to automatically
generate decision trees and production rules from databases. Decision trees and
rules have a simple representation form, making the inferred model relatively easy
to comprehend by the user. However, the restriction to a particular tree or rule
representation can significantly restrict the representation power. In addition
available approaches were developed mainly for problem domains that variables
only take categorical values, such as colour being green and red. They are not
effective in dealing with variables that take numerical values. Discritisation of
numerical variables to categorical descriptions is a useful approach. However more
powerful discretisition techniques are required.
 20 Data Mining and Knowledge Discovery for Process Monitoring and Control

2.3.4 Dependency Modelling

Dependency modelling describes dependencies among variables. Dependency

models exist at two levels: structural and quantitative. The structural level of the
model specifies (often in graphical form, [52, 53]) which variables are locally
dependent; the quantitative level specifies the strengths of the dependencies using
some numerical scale. Examples of tools for dependency modelling include
probabilistic (or Bayesian) graphs [54, 55, 56, 57, 58] and fuzzy digraph graphs [59,
60].

T a bl e 22
. Ad atabase examp e.
Variable values
case xl x2 x3
1 1 0 0
2 1 1 1
3 0 0 I
4 1 1 1
5 0 0 0
6 0 1 1
7 1 1 1
8 0 0 0
9 1 1 1
10 0 0 0

Table 2.3 The probabilistic table associated with the

P(x\ =1) =0.6 P(x\ = 0) =0.4

P(X2 =11 x\=I)=0.8 P(X2=0 1x\=I)=0.2
P(x2=1 1 x\=0)=0.3 P(X2=0 1 x\=0)=0.7
P(x3=1 1x2=I)=0.9 P(X3=0 1x2=I)=O.l

Take probabilistic networks as an example. Table 2.2 shows a collection of 10

data patterns, each is described by three attributes. The task of dependency
modelling using probabilistic networks is two-fold - learning the network structure
and compiling a conditional probabilistic table. For the data collection of Table 2.2
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 21

it is not possible to know the most probable dependencies directly. Figure 2.5 (a)
and (b) illustrate two such possible dependencies. Theoretically, for a given
database there is a unique structure which has the highest joint probability and can
be found by some algorithm such as those developed by Cooper and Herskovits [54]
and Bouckaert [55, 56]. When a structure is identified, the next step is to find such a
probabilistic table as shown in Table 2.3.

(a)

Figure 2.5 An example of two probabilistic networks.

Probabilistic graphical models are very powerful representation schemes which

allow for fairly efficient inference and for probabilistic reasoning. However, few
methods are available for infering the structure from data, and they are limited to
very small databases. Therefore normally there is the need to find structure by
interviewing domain experts. For a given data structure there are some successful
reports on learning conditional probabilities from data.
Other dependency modelling approaches include statistical analysis (e.g.,
correlation coefficients, principal component and factor analysis) and sensitivity
analysis using neural networks.

2.3.5 Summarisation

Summarisation provides a compact description for a subset of data. Simple

examples would be the mean and standard deviations. More sophisticated functions
involve summary rules, multivariate visualisation techniques, and functional
relationships between variables.
22 Data Mining and Knowledge Discovery for Process Monitoring and Control

T a hI e 24 An example transaction database D.

Data cases Items
100 ACD
200 BCE
300 ABCE
400 B E

C'/
Step-J Itemset No. of supporting Step-2 Itemset N.S.D
datacases (N.S.D) -----+
Scan the
database {A} 2 {A} 2
D {B} 3 {B} 3
{C} 3 {C} 3
{D} 1 {E} 3
{E} 3
Step-3
C L,
Tt,.m",.t "

...
Itpm"pt Nc;:n
{AB} Itemset N.S.D
{AB} 1
{AC} Step-4 {AC} 2 Step-5
{A E} ~
{A E} 1 ~ {AC} 2
{B C} ScanD {BC} 2 {BC} 2
{B E} {B E} 3 {B E} 3
Ic E} I IC E} 2 {C E} 2
Step-6
I
C, C,
L,
Itemset Itemset N.S:C
Step-7 Itemset N.S.D
~
~ {ABC} {ABC} 1 Step-8~
{AB E} ScanD {AB E} 1 {BC E} 2
{B C E} IB C El 2

Figure 2.6 Generation of candidate itemsets and large itemsets.

A notable technique for summarisation is mining association rules [61, 62]. Given
a relational database, mining association rules finds all associations of the form.

IF { set of values } THEN { set of values }

A rule is valid given two parameters T c and T s, such that, the rule holds with
certainty > T c and the rule is supported by at least T s cases. Some commercial
systems have been developed using this approach [61, 62].
A very simple example can be used to illustrate the approach. Table 2.4 shows an
example of the transaction of part of a database. The purpose for mining the
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 23

database is to summarise the number of occurrences of each candidate set of items

(itemset), and then determine large sets of items based on a predetermined minimum
support. Thus for a minimum support of 2 data cases, the mining process is shown
in Figure 2.6.
Step-l involves scanning the database D and summarising the supports for each
item set. In Step- 2, the itemset {C I} whose number of supporting cases is less than 2
is removed, and CI becomes L I . In step-2, binary itemsets are formed and go
through similar procedures. The above procedure repeats until L 3 .
The approach of mining association tules has been quite successful [23]. However
it also has some limitations. For example, it assumes all data is categorical and there
are no good algorithms available for the numeric fields. It also assumes that the set
of associations satisfying thresholds is sparse.

2.3.6 Regression

Linear and non-linear regression is one of the commonest approaches for correlating
data. Statistical regression methods often require the user to specify the function
over which the data is to be fitted. In order to specify the function, it is necessary to
know the forms of the equations governing the correlation for the data. The
advantage of such methods is that from the equation it is possible to gain some
qualitative knowledge about the input - output relationships. However, if prior
knowledge is not available, it is necessary to find out the most probable function by
trial-and-error which may require very time consuming effort. Feedforward neural
networks (FFNNs) do not need functions to be fixed in order to learn and have
shown very remarkable results in representing non-linear functions. However the
resulting function using a FFNN is not easy to understand and is virtually a black
box without any explanations.

2.3.7 Case-based Learning

Case-based learning is based on acquiring knowledge represented by cases and it

employs reasoning by analogy [63, 64]. Case-based learning focuses on the indexing
and retrieval of relevant precedents. Typically the solution sequence is a
parameterised frame or schema where the structure is more or less fixed, rather than
expressed in terms of an arbitrary sequence of problem solving operators. Case-
based reasoning is particularly useful for making use of data which has complex
24 Data Mining and Knowledge Discovery for Process Monitoring and Control

internal structures. Different from other data mining techniques, it does not require
to have a large number of historical data patterns. There are only a few reports of
the application of case-based reasoning in process industries such as case-based
learning for historical equipment failure databases [65, 66] and equipment design
[67].

2.3.8 Mining Time-series Data

Many industrial and business areas deal with time-series or dynamic data. It is
apparent that all statistical and real-time control data in process monitoring and
control are essentially time-series. Figure 2.7 shows the dynamic trend signals of a
variable under two different operational conditions. It is very easy for humans to
visually capture features of each trend and identify their difference. However for
computers to perform the same task is difficult. Most KDD techniques cannot
account for the time series of data. The techniques to deal with time series data are
to carry out pre-processing of the data to use minimum data points to capture the
features and remove noise. These techniques include filters, e.g., Kalman filters,
Fourier and wavelet transforms, statistical approaches, neural networks as well as
various qualitative signal interpretation methods. Chapter 3 will introduce some of
these techniques for pre-processing dynamic data.

31 Ii I 91 121 151 181 211 241

Time

Figure 2.7 The dynamic trend signals of a variable under two different operational
conditions.
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 25

2.3.9 Method Selection

There is no well-developed techniques for selecting data mining and KDD methods.
It is still an art [23]. Apart from the general considerations such as cost, and
support, there are some technical dimensions to the method selection. These include
[23]:
(1) univariate vs. multi-variate data. Most approaches assume independence of
variables or simply consider a single variable at a time.
(2) numerical vs. categorical or mixed data. Some methods are only suitable for
numerical data, others only for categorical data. There are only a few cases which
allow mixed data.
(3) explanation requirements or comprehensibility. Some tools give results which
are implicit to users (black box), while others can give causal and explicit
representations.
(4) fuzzy or precise patterns. There are methods such as decision trees which only
work with clear cut definitions.
(5) sample independence assumptions. Most methods assume independence of data
patterns. If there are dependency on the data patterns, it is necessary to remove or
explore.
(6) availability of prior knowledge. Some tools require prior knowledge which
might be not available. On the other hand, some others do not allow input of prior
knowledge, causing a waste of prior knowledge.
It is important to be aware of the complexity of data which tends to contain noise
and erroneous components and has missing values. Other challanges come from
lack of understanding of the domain problem and assumptions associated with
individual techniques. Therfore, data mining is rarely done in one step. It often
requires using a number of approaches to use some tools to prepare data for other
methods, or for validating purposes. As a result, multifunctional and integrated
systems are required.

2.4 Feature Selection with Data Mining

Data pre-processing may be more time consuming and presents more challenges
than data mining. Process data often contains noise and erroneous components and
has missing values. There is also the possibility that redundant or irrelevant
26 Data Mining and Knowledge Discovery for Process Monitoring and Control

variables are recorded, while important features are missing. Data pre-processing
includes provision for correcting inaccuracies, removing anomalies and eliminating
duplicate records, and filling holes in the data and checking entries for consistency.
It also requires making the necessary transformation of the original to put it in the
format suitable for data mining tools.
The other important requirement with KDD process is feature selection. KDD is a
complicated task and often depends on the proper selection of features. Feature
selection is the process of choosing features which are necessary and sufficient to
represent the data. There are several issues influencing feature selection, such as .
masking variables, the number of variables employed in the analysis and relevancy
of the variables [68].
Masking variables hide or disguise patterns in data. Numerous studies have shown
that inclusion of irrelevant variables can hide real clustering of the data so only
those variables which help discriminate the clustering should be included in the
analysis [68, 69, 70].
The number of variables used in data mining is also an important consideration.
There is generally a tendency to use more variables. However, increased
dimensionality has an adverse effect because, for a fixed number of data patterns,
increased dimensionality makes the multidimensional data space sparse.
However failing to include relevant variables also causes failure in identifying the
clusters. A practical difficulty in mining some industrial data is to know if all
important variables have been included in the data records.
Prior knowledge should be used if it is available. Otherwise, mathematical
approaches need to be employed. Feature extraction shares many approaches with
data mining. For example, principal component analysis (PCA), which is a useful
tool in data mining, is also very useful for reducing the dimension (PCA and its
applications are introduced in Chapters 3 and 4). However, PCA is only suitable for
dealing with real-valued attributes. Mining of association rules mining is also an
effective approach in identifying the links between variables which take only
categoric values [68]. Sensitivity studies using feedforward neural networks
(FFNNs) are also an effective way of identifying important and less important
variables (sensitivity studies using FFNNs are introduced in Chapter 9).
Gnanadeskikan et al. [69] reviewed a number of clustering techniques which
identify discriminating variables in data.
 Chapter 2 Data Mining and Knowledge Discovery - an Overview 27

2.5 Final Remarks and Additional Resources

This Chapter has provided an overview of data mining and KDD. Data mining and
KDD makes use of various technologies including statistics, neural networks,
machine learning, artificial intelligence, pattern recognition and databases. The
unifying goal is to extract useful information and knowledge from massive data. It is
a complex and interative process, starting with data access, continuing with data
cleaning and pre-processing as well as data mining and knowledge discovery, finally
culminating with intepretation and validation of results. It is important to be aware
of the complexity of industrial data and that there are always some assumptions
related to specific KDD techniques. Integration of various methods is necessary, so
that some tools can be used in preparing data for other methods and results obtained
using different methods can be compared.

Table 2.5 Useful data mining and KDD sites.

• The KDD Foundation - a starting point for exploring Internet
resources in knowledge discovery and data mining.
http://www.kdd.org/
• Decision Theory & Adaptive Systems Group, Microsoft Research
http://www.research.microsoft.coml-fayyad
• BT's Data Mining Group homepage
http://www.labs.bt.comlprojects/mining/index.htm
• Birmingham University Data Mine
http://www.cs.bham.ac.uk/-anp/TheDataMine.html
• The University of Ulster Data Mine
http://iserve l.infj .ulst.ac.uk: 8080/
• The Corporate KDD Bookmark
http://www.cs.su.oz.aul-thierry/ckdd. html
• The Machine Learning Database Repository at the University of
California Irvine
http://www.ics. uci.edu/ AIIML/Machine-Learning.html
• ML Net Site - Machine learning publications, data and software
http://www.gmd.de/
28 Data Mining and Knowledge Discovery for Process Monitoring and Control

Despite the rapid growth, the success achieved and a huge predicted market, KDD
is still considered to be in its infancy. There are still many challenges to overcome.
An obvious issue in process monitoring and control is how to deal with dynamics
associated with the data. Another issue is that most data mining tools assume that
the variables are independent of each other, but process variables are often
connected. The challenges posed by operational data have already been summarised
in Section 1.5.
Apart from the need to develop more reliable data mining and KDD tools, there is
also the need to gain more experience in applying them to industrial and business
problems.
For introduction and review of data mining and KDD, readers are referred to
Fayyad et al. [71], Wu [72], Chen et al. [28], Simoudis et al. [73], Wu et al. [74],
and Pyle [75]. There are also some useful web sites which are summarised in Table
2.5 and provide gateways to other resources.
CHAPTER 3
DATA PRE-PROCESSING FOR FEATURE EXTRACTION,
DIMENSION REDUCTION AND CONCEPT FORMATION

This chapter describes data pre-processing for feature extraction, dimension

reduction, noise removal and concept formation from monitored process
measurements. The discussion is concerned with capturing the features in dynamic
trend signals. A dynamic trend representation is the visualisation of the changing
trajectory of a variable over time and consists of many sample values. However, in
order to make effective use of trends in a computer based system, it is necessary to
compress the data to fewer values. One of the earliest examples of dealing with such
trends is based on a real time expert system G2 [76] which uses qualitative
expressions such as temperature increase or decrease as descriptors. Later, various
other approaches were developed including episodes [77, 78, 79], neural networks
[80] and more recently wavelets [81, 82] and principal component analysis [83].
This chapter introduces principal component analysis, wavelet analysis as well as
episode representations.

3.1 Data Pre-processing

Data pre-processing is used to

(1) filter out the noise components otherwise this may result in wrong conclusions
being reached from the data.
(2) extract features, reduce the dimensionality of the original signal and retain as
much relevant information as possible. The main reasons for feature extraction are,

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
30 Data Mining and Knowledge Discovery for Process Monitoring and Control

first of all to minimise the dependencies between attributes and secondly to reduce
dimensionality.
(3) Deal with the problem of variable sampling periods for data, such as on-line
real time signals and laboratory analytical data.
(4) Develop concept formation because some data mining and KDD tools have
been developed only for dealing with discrete-valued attributes and are not effective
in dealing with continuous-valued variables. It is not possible to use variables
represented by a trend without preprocessing the data.
It is worth noting that data pre-processing has many features in common with data
mining, such as principal component analysis, supervised and unsupervised
classification using statistical and neural network algorithms.

3.2 Use of Principal Component Analysis

The method of principal component analysis (PCA) was originally developed in the
1900's [84, 85], and has now re-emerged as an important technique in data analysis.
The central idea is to reduce the dimensionality of a data set consisting of a large
number of interrelated variables, while retaining as much as possible of the variation
present in the data set. Multiple regression and discrimination analysis use variable
selection procedures to reduce the dimension but result in the loss of one or more
important dimensions. The PCA approach uses all of the original variables to obtain
a smaller set of new variables (principal components - PCs) that they can be used to
approximate the original variables. The greater the degree of correlation between
the original variables, the fewer the number of new variables required. PCs are
uncorrelated and are ordered so that the first few retain most of the variation present
in the original set.

3.2.1 Basic Concepts: Mean, Variance, Covariance

It is convenient at this point to gave a brief summary of the basic points. In the
univariate case the mean and variance are used to summarise a data set. The mean or
mean value of a discrete distribution is denoted by j.l and is defined by
J.! = LX,f(X,) (3.1)
,
 Chapter 3 Data Pre-processing 31

where j{Xi) is the probability function of the random variable X considered. The
mean is also known as the mathematical expectation of X and is sometimes denoted
by E(X).
The variance of a distribution is denoted by c/ and is defined by
a 2 = I(Xi-IlY!(Xi) = E(Xi-IlY (3.2)
I

In fact it is an index reflecting the deviation of Xi from the mean Il. In other words,
the variance a 2 describes the linear dependency of all Xi . The bigger the variance,
the less dependent; the smaller the variance and hence the greater the linear
dependency between Xi .
To summarise multivariate data sets, it is necessary to find the mean and variance
of each of the p variables, together with a measure of the way each pair of variables
is related. For the latter, the covariance or correlation of each pair of variables is
used.
The population mean vector is given by J.l = [1lJ, 1l2, ... , IIp], where
Ili = E(x;) (3.3)
An estimate of J.l based on n, p-dimensional observations IS
x = [XI, X2, ... , xp], where Xi is the sample mean of variable Xi
The vector representing the population variance is a· = E(xf) - Il~
An estimate of a based on n, p-dimensional observations is s· = [sf, s~, ... s~],
where s7 is the sample variance of variable Xi.
The covariance of two variables Xi and Xj is defined by
Cov (Xi ,Xj) =E(Xi'Xi) - Ili Ili (3.4)
With P variables, X /, Xl> •.•xP' there are p variances and 1. pep _ 1) covariances. In
2
general these quantities are arranged in the p x p symmetric matrix, called
covariance matrix, I,
cr II cr 12
all 0'22 alp

I= (3.5)

cr pi cr p2 cr pp
where aij = aji. The simple version of I , usually denoted, S, is generally estimated
as
1 "
S = - I ( X i - X)(Xi - x)' (3.6)
n -1 i=1
32 Data Mining and Knowledge Discovery for Process Monitoring and Control

The covariance is often difficult to interpret because it depends on the units in

which the two variables are measured; consequently it is conveniently standardised
by dividing by the product of the standard deviations of the two variables to give a
quantity called the correlation coefficient, Pij , defined by
_ (Ii)
Pij- ~ (3.7)
VO'ii(Jjj

The correlation coefficient lies between -I and I and gives a measure of the linear
relationship between variables Xi and Xj

3.2.2 Principal Component Analysis

Given a data matrix X representing n observations of each of p variables, Xl, X2,

... xp , the purpose of principal component analysis is to determine a new variable Y],
that can be used to account for the variation in the p variables, Xl, X2, ••• xp. The first
principal component is given by a linear combination of the p variables as
YI = WIIXI + W12X2 + ... + WlpX p (3.8)
where the sample variance is greatest for all of the coefficients (also called weights),
WII, W12, ... Wlp' conveniently written as a vector WI.

The WI I , W12, '" Wlp have to satisfy the constraint that the sum-of-squares of
the coefficients, i.e., W'I WI , should be unity.
The second principal component, Y2, is given by the linear combination of the p
variables in the form:
Y2 = WZIXI + W22X2 + ... + W2pXp (3.9)
or y, = w·,x
which has the greatest variance subject to the two conditions,
w',w, = 1 (3.10)
and
W',WI = 0 (so thatYI andY2 are uncorrelated)
Similarly the jth principal component is a linear combination
Yj = w'jX (3.11)

which has greatest variance subject to

W'jWj 1
W'jWi = 0 (i < j)
 Chapter 3 Data Pre-processing 33

To find the coefficients defming the first principal component, the elements of
WI should be chosen so as to maximise the variance of YI subject to the constraint,
w't WI = 1. The variance of Yl is then given by
Var(yl) =Var( WI x) = W'I S
I WI (3.12)
where S is the variance-covariance matrix of the original variables (See Section
3.2.1). The solution of WI =( WI I , W12, ••• Wlp) to maximise the variance Yl is the
eigenvector of S corresponding to the largest eigenvalue. The eigenvalues of S are
roots of the equation,
IIS-A.II =0 (3.13)
If the eigenvalues are A.I, A.2, ••• A. p , then they can be arranged from the largest
to the smallest. The first few eigenvectors are the principal components that can
capture most of the variance of the original data while the remaining PCs mainly
represent noise in the data.
PCA is scale dependent, and so the data must be scaled in some meaningful way
before PCA analysis. The most usual way of scaling is to scale each variable to unit
variance.

3.2.3 Data Pre-processing Using PCA

3.2.3.1 Pre-processing Dynamic Transients/or Compression and Noise Removal

In computer control systems such as DCS, nearly all important process variables are
recorded as dynamic trends. Dynamic trends can be more important than the actual
real time values in evaluating the current operational status of the process and in
anticipating possible future developments. Appendix C describes a data set of one
hundred cases corresponding to various operational modes such as faults,
disturbances and normal operation of a refmery reactive distillation process for
manufacture of methyl tertiary butyl ether (MTBE), a lead-free gasoline additive.
This can be used to illustrate the dimension compression capability of PCA. For
each data case, twenty one variables are recorded as dynamic responses after a
disturbance or fault occurs. Each trend consists of 256 sample points. Figure 3.l
shows the trends of a variable for two different cases. The eigenvalues of the first 20
principal components are summarised in Figure 3.2. It is apparent that the
eigenvalues of the first few principal components can be· used as a concise
representation of the original dynamic trend, and so are used to replace the original
responses for use in pattern recognition.
34 Data Mining and Knowledge Discovery for Process Monitoring and Control

o
31 61 91 121 151 181 211 241

Point

Figure 3.1 The dynamic trends of a variable for two data cases.

250

200

100

50

..
TTTfl
- f f f f I I I I I I I I I I

Points
11 13 15 17 19

Figure 3.2 The first 20 eigenvalues of a variable.

 Chapter 3 Data Pre-processing 35

3.2.3.2 Pre-processing ofDynamic Transient Signals for Concept Formation

Since the first two principal components can capture the main feature of a dynamic
trend, this can be displayed graphically by plotting the eigenvalues on a two-
dimensional plane. Figure 3.3 shows such a plot of the eigenvalues of the first two
principal components of a variable Fo' A point in the two dimensional plane
represents the feature of the variable response trend for one data case. Data points in
region B have response trends which are similar and unlike those in region D.
The fact that a two-dimensional plot is able to capture the features can be seen
from Figures 3.4 and 3.5. Figure 3.4 shows the dynamic responses of the variable
T_MTBE for seven data cases. After being processed using peA (actually the seven
data cases are processed using peA together with another 93 data cases, but here
only the seven are shown for illustrative purpose), the results are shown on the two-
dimensional peA plane" in Figure 3.5. It is clear that the dynamic trends of data
cases 1 and 2 are more alike than with the others in Figure 3.4 and they are grouped
closer in Figure 3.5. Similar observations can be made for data cases 40 and 80, as
well as 14 and 15.

40

30
B
20 ...... ,. .... :,.*..
'"
•
............
"
10

..
... ....... ..
...:.. ~ ""
': .••....•• ::~ C
......... ...........
\ . ...... -.
.
........... .......
.....
.,,:
[" ... )
~ 0
...............]
~
u
=- -10
•
-20 .............
......... '.~
D

A
-30

-40
-40 -20 o 20 40 60 80 100 120
PC-I-Fo

Figure 3.3 The peA two dimensional plane of the variable Fo.
36 Data Mining and Knowledge Discovery for Process Monitoring and Control

12r------------------------------------------------------.
Case 16

o~~~-- __ ~ ____----__--__ ----~---- __----------______ ~

1 13 25 37 49 61 73 85 97 109121 133145 157169181 193205217229241 253

Time

Figure 3.4 The dynamic trends of the temperature T_MTBE for the case study
described in Appendix C.

,•
60

,
50 Case 16

,•
40

30 Case 15

..I
=
f;I;l Case 80 Case I
20
E-
:::!!I
E-I
10

•• II
NI 0
•
I
u
1:1. -10

-20 Case 14
!
Case 2
Case 40
-30

-40
-40 -30 -20 -10 0 10 20 30 40 50 60
PC_LTjlTBE

Figure 3.5 The projection of the dynamic trends of Figure 3.4 on the two-
dimensional PCA plane.
 Chapter 3 Data Pre-processing 37

The plot of the dynamic trends of a variable on a two dimensional plane, as

depicted in Figure 3.3 is referred to as concept formation. Concept formation
transforms a complicated trend to a concept, e.g., "the variable Fa is in region D".
The transformed concept of the trend of a variable can be used to develop
knowledge based systems. A simple example is the following production rule for a
case of a continuous stirred tank reactor where the data refer to historical operating
data:

IF Fo is in region D of Figure 3.3

AND TR is in region C of Figure 3.6
THEN The operation will be in region
ABN-l of Figure 3.7

A detailed discussion on using the concept formation method to develop

conceptual clustering systems will be discussed later in Chapter 7.

40

30

20
.,B
... D
10 ..
'
.'
.,1'1
:~

.. (. \'}
.'.
.. , . I " ' ..

~
M
I

I
0
_,.'"1, •••
..: \ '.
U '"'"""".",
=-. -10
"""" ..,.''''. C
A
-20

-30

-40
-100 -80 -60 -40 -20 o 20 40 60 80

PC-I-TR

Figure 3.6 The PCA two dimensional plane of the variable TR.
38 Data Mining and Knowledge Discovery for Process Monitoring and Control

100
....... ,"1 ••• , •••
~
80
..:......... .
",,,.'

60
ABN2
NOR 2

....,...-...
.•..
.#f••••, ' :••••••••

.
"'"1 1.,11"""'"1,
40
. .... ' ""

"
1 1 '"'",.

.
--'"
.
'. .'
20 ",

. , "... ...
. " • • • • • ' . ' • • 1 • • • • • • • • • 11 . . . .
GI ~ •• I

••••• ,"11111 •••••••••• , . " .

I" ""
~
I 0 ABNI '.
M
U
I •
Q., :..
-20 ·I., '

.
u· ... ,........

. ...• ..
""""1.", •••• , •• 1. 1 "

.,.... ............................. .
NOR 3
-40 ". '.
.:
-60 ,,\~:~.II'I"I., ""1"" 01 , •• , ,I" .1'

NOR I
-80+----r---,.---,---~----.----r----~--~
-180 -140 -100 -60 -20 20 60 100 140

PC-I-State

Figure 3.7 The PCA plane of operational states of a CSTR reactor_

3.2.3.3 Dependency Removal and Clearance ofRedundancy Variables

Studies have found that presence of redundancy and irrelevant variables may
deteriorate pattern recognition or hide the real patterns in the data [68] and so some
data mining and KDD tools require the inputs to be independent. Sometimes it is not
possible to directly identify the dependencies between variables. PCA can be used
to pre-process the data and the first few principal components are then be used by
other data mining and KDD tools.

3.3 Wavelet Analysis

Recently, wavelet analysis has emerged as a promising new approach for signal and
image analysis and has been extended to process monitoring and control. In this
section, it is convenient to start with the well-established approach based on Fourier
 Chapter 3 Data Pre-processing 39

transform for signal processing and how it relates to wavelet transforms. Wavelet
transforms are then introduced, followed by its application for feature extraction.

3.3.1 Signal Processing Using Fourier Transform

Fourier transforms are well known as a useful technique for frequency analysis of a
signal which breaks down a signal into constituent sinusoids of different
frequencies. The transform is defined as
F(oo) = f: f(t) e-iro1dt (3.14)
and the inverse is

j{t)=
1 f-
2n .
-ooF(ro)e1rotdro (3.15)

Here the Fourier transform can be viewed as the decomposition of a functionj{t)

into a sum of frequenc~ components. This transform uses sine and cosine as its
building blocks or basis functions to map a time function into the frequency domain.
Figure 3.8(a) illustrates the Fourier transform process. However, the Fourier
transform does not show how the frequencies vary with time inf(t) so that looking at
the transform of a signal, it is impossible to tell when a particular event took place.
If a signal does not change with time, i.e., it is stationary, this is not important.
However, most signals of interest contain numerous non-stationary or transitory
characteristics: drift trends, abrupt changes related to beginning and ending of
events. These characteristics are often the most important part of the signal, and
Fourier analysis is not suited in detecting them.
The short-time Fourier transform (STFT), or windowed Fourier transform is able
to overcome the limitation, and is now considered as the standard method for
studying time-varying signals. The idea of STFT is to use a window which slides
over the signal in time, and then to compute the Fourier transform within each
window. The general definition of STFT is as follows
F(t, (0) = f:fCt)g("C - t)e-jro'Cd"C (3.16)
where fit) is the original function in time domain and g(t) is the window function.
The simplest window function is

t t
{~
t'~ ~
g(t) = (3.17)
otherwise
 40 Data Mining and Knowledge Discovery for Process Monitoring and Control

W!Wv ...

Signal Constituent sinusoidal ofdifferent frequencies

(a) Fourier transform

+ _._ /"\ 1\ . . -
~ ---vv- +
-rJ'v-+
•••

Signal Constituent wavelets

(b) Wavelet transform

Figure 3.8 Comparison of Fourier and wavelet transforms.

In time-frequency analysis of a non-stationary signal, there are two conflicting

requirements. The window width must be long enough to give the desired frequency
resolution but must also be short enough not to lose track of time dependent events.
While it is possible to design window shapes to optimise, or trade-off time and
frequency resolution. there is a fundamental limitation on what can be done, for a
given fIxed window width. Figure 3.9 depicts a windowed SFTF [87].
 Chapter 3 Data Pre-processing 41

5(t)

~~J=.t
~ t
t'

8(t)=5(t)1(t)

~ t

1
Figure 3.9 The windowing operation.

3.3.2 Signal Transformation Using Wavelets

Wavelet transfonuation is designed to address the problem of non-stationary signals

[99, 100, 101]. It involves representing a time function in tenus of simple, fIxed
building blocks, tenued wavelets. These building blocks are actually a family of
functions which are derived from a single generating function called the mother
wavelet by translation and dilation operations (Figure 3.8(b)). Dilation, also known
as scaling, compresses or stretches the mother wavelet, and translation shifts it along
the time axis.
The mother wavelet satisfIes
(3.18)
and the translation and scaling operations on 'II (t) creates a family of functions,
1 t - b
'I' a,b (t) = -r '1'(-) (3.19)
va a
The parameter a is a scaling factor and stretches (or compresses) the mother
wavelet and is easily understood by considering the sinusoid function in Figure
 42 Data Mining and Knowledge Discovery for Process Monitoring and Control

3.10. The scaling factor works exactly the same way with wavelets, can be seen in
Figure 3.11: the smaller the scale factor, the more compressed the wavelet is.

~ f(t) = sin(t) ; a = 1

= 21
~ : : :J f(t) = sin(2t) a

r\;:
: : : :J
f(t) = sin(4t) a = 41

Figure 3.10 Scaling ofa sinusoid function.

~~ ~ I f(t) = ",(1) a =1

~:::::l f(l) = ",(21) a = 21

~: :::: :I f(t) = "'( 4t) a = 41

Figure 3.11 Scaling of a wavelet function.

The parameter b in Equation 3.19 is a translation along the time axis and simply
shifts a wavelet and so delays or advances the time at which it is activated.
Mathematically, delaying a functionj(t) by td is represented by j(t-td). The factor
1
~ is used to ensure that the energy of the scaled and translated versions is the

same as the mother wavelet. Figure 3.12 shows an example of a particular mother
wavelet '" (t) , known as the Mexican hat function [98],
1 I t2
'V(t) = .Jj1t-"4(l- t 2 )ex p(-Z-)
 Chapter 3 Data Pre-processing 43

The stretched and compressed wavelets through scaling operation are used to
capture the different frequency components of the function being analysed. The
compressed version in Figure 3. 12(b) is used to fit the high frequency needs, and the
stretched version in Figure 3.12(c) is for low frequencies. The translation operation,
on the other hand, involves shifting of the mother wavelet along the time axis to
capture the time information of the function to be analysed at a different position, as
shown in Figure 3.12 (d).

2 ~~~~~~--~~--~~
I I I I I I I

Mother Wavelet Compression

a=1.b=O a=O.25. b=O

0t-------___ o t---------,II,~---------+
\JV
·1 -1 I--t----t---t--t---+--t---t---+
·20 ·15 ·10 -5 0 5 10 15 20 -20 -15 -10 -5 0 5 10 15 20

(a) (b)
2 ~--'---"---"---"---"---"-~--T

Dilation Translation
a=5.b=O a=1.b=8

o r---_ o 1-____________,

\V
-1 -1 I--t---t--t---t--t---+--t---+
-20 -15 -10 -5 0 5 10 15 20 -20 -15 -10 -5 0 5 10 15 20

(c) (d)

Figure 3.12 Mexican hat wavelet at different dilation and translation.

44 Data Mining and Knowledge Discovery for Process Monitoring and Control

In this way, a family of scaled and translated wavelets can be created using
scaling and translation parameters a and b. This allows signals occurring at different
times which have different frequencies to be analysed.
In contrast to the short-time Fourier transform, which uses a single analysis
window function, the wavelet transform can use short windows at high frequencies
or long windows at low frequencies. Thus the wavelet transform is capable of
zooming-in on short-lived high frequency phenomena, and zooming-out for
sustained low frequency phenomena. This is the main advantage of the wavelet over
the short-time Fourier transform.

3.3.3 Continuous Wavelet Transform

Given a mother wavelet function \If (t), the continuous wavelet transform CWTr of a
function.f{t) is defined by
t-b c
CWTfa,
.
b) = <f, \If ah>
.
= \If (--)dt
a
/ va, a, b ER, a"* 0 (3.20)

where \If (t - b )dt /.,fa is sometimes called the baby wavelet. Here time t and the
a
scaling and translation parameters a and b can be changed continuously. In this
case, CWTJa, b) is called the wavelet transform coefficient. If a, b, and t change
continuously, the values of CWTja, b) can be represented by a three dimensional
diagram.
The application of a continuous wavelet transform consists of the following steps.
(1) take a wavelet and compare it to a section at the start of the original signal.

Signal

Wavelet

C=O.OI02

Figure 3.13 (a) Step (2) in continuous wavelet transform.

 Chapter 3 Data Pre-processing 45

(2) Calculate the wavelet coefficient CW1f. representing how closely the wavelet
is related to this section of the signal, as shown in Figure 3. 13(a). The higher CWTf
is, the greater the similarity. The result obviously depends on the shape of the
wavelet chosen.

Signal

Wavelet ¢
Figure 3.13(b) Step (3) in continuous wavelet transform.

(3) Shift the wavelet to the right and repeat steps 1 and 2 until all of the signal has
been examined, as shown n Figure 3.13 (b).

Signal

Wavelet

C=O.2247

Figure 3.13(c) Step (4) in continuous wavelet transform.

(4) Scale (stretch) the wavelet and repeat steps 1 through 3 (Figure 3.l3 (c)).
(5) Repeat steps 1 to 4 for all scales.
 46 Data Mining and Knowledge Discovery for Process Monitoring and Control

Plotting the wavelet coefficients against time and scale generates a three
dimensional diagram, as shown in Figure 3.14. An alternative would be to use the
two dimensional diagram in Figure 3.15, where brightness reflects the magnitude of
the wavelet coefficients.

Figure 3.14 Three dimensional plot of wavelet transform coefficients.

Large
Coefficients

Small
1950 400Q 4050 4100 4150 4200 4:?5O Coefficients
Time

Figure 3.15 Two dimensional plot of wavelet transform coefficients.

Continuous in the context of wavelet transform implies that the scaling and
translation parameters a and b change continuously. In practice, it is necessary to
select a number of scales which is determined by the computational effort. A similar
argument applies to the translation (shifting) parameter. In both cases, however, the
 Chapter 3 Data Pre-processing 47

data of the signal being processed by the continuous wavelet transform using a
computer is discrete.
The wavelet transform coefficients which are generated can be used to reconstruct
the original function.
f( t ) = C;; I J+xJ+x
-x -x CWT r(b)
a, \JI a.h ( / )dadb
-,- (3.21 )
a"
where C. is called the admissibility constant defined by
C N1 (00 )1 doo 2
= C~ < + CX) (3.22)
00

3.3.4 Discrete Wavelet Transform

Calculating wavelet coefficients for every possible scale can represent a

considerable effort and generate a vast amount of data. The discrete parameter
wavelet transform uses scale and position values based on powers of two (so-called
dyadic scales and positions) which makes the analysis much more efficient, while
still being accurate. In practice, a discrete wavelet transform is commonly used. To
do this, the scale and time parameters are discretised as follows,
a = ao, b = n bo ao '
m, n are integers (3.23)
The family of wavelets {'JI "'." (I)} is given by
\JI m,n (I) =ao nJ/2 \JI (aom t - n bo) (3.24)
resulting in a discrete wavelet transform (DWT) having the form
DWTr(m,n) =< f,\JI m,n >= ao m/2 r:f(t}'JI(ao m t-nbo)dl (3.25)
Mallat [89] developed an approach for implementing this using filters. For many
signals, the low frequency content is the most important part. The high frequency
content, on the other hand provides flavour or nuance. In wavelet analysis the low
frequency content is called the approximation and the high frequency content is
called the detail. The filtering process uses lowpass and highpass filters to
decompose an original signal into the approximation and detail parts. It is not
necessary to preserve all the outputs from the filters. Normally they are
downsampled, keeping only the even components of the lowpass and highpass filter
outputs. The decomposition can be iterated, with successive approximations being
decomposed in turn, so that one signal is broken into many lower-resolution
components, as illustrated in Figure 3.16. Here, S refers to the original signal, A is
the approximation and D is the detail.
48 Data Mining and Knowledge Discovery for Process Monitoring and Control

In the case of a discrete wavelet transfonn, reconstruction of the original signal is

not guaranteed. Daubechies [88] developed conditions under which the {\V m," } fonn
an orthononnal basis. Usually, ao = 2 and bo = 1 are used, though any values are
possible. In this case, both the transfonn and reconstruction are complete because
the family of wavelets satisfy the orthogonal condition.

Figure 3.16 Signal multiresolution analysis using wavelets.

3.3.5 Singularity Detection Using Wavelet for Feature

Extraction

Singularities often carry the most important infonnation in signals [82]. For
example, Bakshi and Stephanopoulos [90, 91] used inflexion points as the
connection points of episode segments of a signal, as shown in Figure 3.17 (episode
segments of a signal is discussed in Section 3.4). Singularities of a signal can be
used as the compact representation of the original signal and used as inputs to
pattern recognition systems.
 Chapter 3 Data Pre-processing 49

c d
c d
(I)
b a b

(2) • connection points

b of episodes

(3)
e
b

Figure 3.17 Singularities as connection points of episode segments.

Mathematically, the local singularity of a function is measured by Lipschitz

exponents [92]. Mallat and Hwang [92] proved that the local maxima of the wavelet
transform modulus detect the locations of irregular structures and provided
numerical procedures for computing the Lipschitz exponents. Within the framework
of scale-space filtering, inflexion points of j{t) appear as extrema for af(t) / at and
zero crossing for 0 2 f(t) / a[2 , so Mallat and Zhong [93] suggested using a wavelet
which is the first derivative of a scaling function <J>(t) ,
\jI (t) = d~([)
dt
with a cubic spine being used for the scaling function.
The wavelet modulus maxima and zero-crossing representations were developed
from underlying continuous-time theory. For computer implementation, this has to
be cast in the discrete - time domain. Berman and Baras [94] proved that wavelet
transform extrema / zero-crossing provide stable representations of finite length
discrete-time signals. Cvetkovic and Vetterli [95] have developed a more complete
discrete-time framework for the representation of the wavelet transform. They
designed a non-subsampled multi-resolution analysis filter bank to implement the
wavelet transform for the representation. Using this filter bank, the wavelet function
can be selected from a wider range than the B-spline in Mallat's method.
Non-subsampled multi-resolution analysis can then be used to detect singularities
of a signal. An octave band non-subsampled filter bank with analysis filters H()(z)
and H,(z) is shown in Figure 3.18. In this method, a wavelet transform is defined in
 50 Data Mining and Knowledge Discovery for Process Monitoring and Control

tenns of the bounded linear operators 12 (Z), j = 1, 2, ..1+ 1. The

~: /2 (Z) ~
operators ~ are the convolution operators with the impulse responses of the filters:

••••••••

I
D x

2
D x

3
D x

4
D x

4
Ax
i
D x -- detail on the ith decomposition
i
A x -- approximation on ith decomposition

H (), H 1 -- low-pass and high-pass filters

Figure 3.18 An octave band non-subsampled filter bank.

The multiresolution procedure depicted in Figure 3.18 can be described less

rigorously. Figure 3.18 shows four steps, or four scales analysis. In the fIrst step,
the original signal is split into approximation A~ and detail D~. The detail D~ is
assumed to be mainly the noise components of the original signal and the
approximation A: represents mainly the trend of the original signal. A~ is further
 Chapter 3 Data Pre-processing 51

decomposed into approximation A; and detail D;, A; to A; and D;, and A; to

A~ and D;. In each step we find the extrema of the detail. In the first few steps, the
extrema are due to both the noise and the trend of the noise-free signal. As the scale
increases, the noise extrema are gradually removed while the extrema of the noise-
free signal remain. In this way, using multi-scale analysis and extrema
determination, the extrema of the noise-free signal can be found, which represent
the features of the signal.
To represent the extrema, it is convenient to use a finite impulse response (FIR)
wavelet filter, which has a sequence {a,: k E Z} with only K non-zero terms. A
typical example is the Haar wavelet, having only two non-zero coefficients.
Daubechies's wavelets [88] are also FIR filters and smoother than the Haar wavelet.
Daubechies' wavelets having more coefficients so are smoother and have higher
vanishing moments. They also require less computation effort because they are
constructed by filter convolution.
The Daubechies' scale and wavelet functions are expressed as
~(t) = Lh(k)~(2t - k) (3.26)
k

\If (t) = Lg(k)~(2t

, - k) (3.27)

where {h(k)} is the low-pass filter coefficients and {g(k)} the band-pass filter
coefficients.
Daubechies wavelets have a maximum number of vanishing moments over the
support space. The vanishing moments of the wavelets also have a different number
of coefficients. Using wavelets with more vanishing moments has the advantage of
being able to measure the Lipschitz regularity up to a higher order, which is helpful
in filtering noise, but it also increases the number of maxima lines. The number of
maxima for a given scale often increases linearly with the number of moments of the
wavelet. In order to minimise computational effort, it is necessary to have the
minimum number of maxima to detect the significant irregular behaviour of a signal.
This means choosing a wavelet with as few vanishing moments as possible but with
enough moments to detect the Lipschitz exponents of the highest order components
of interest.
For the cases considered here, an eight coefficient "least-asymmetric" Daubechies
wavelet is used as a filter. The scale and wavelet function for this filter are
illustrated in Figure 3.19.
52 Data Mining and Knowledge Discovery for Process Monitoring and Control

1. 8calirYJ Fm:tion

0.8

o.s
o.
o.
o
-0··2'cO-~~:--~----'4'---'--S'-~--' -1.fI:_3-~_2-_~1-~~1~~~~4--.J
Time Time

Figure 3.19 The "Least-Asymmetric" scale function and wavelet function.

A signal f(t)=sin(t) and its extrema from the wavelet analysis using a non-
subsampled filter bank with Daubechies eight coefficients least asymmetry wavelet
is illustrated in Figure 3.20. It shows that extrema of wavelet analysis correspond to
the singularities of signal. The shape of corresponding extrema of the wavelet
analysis can be maximum or minimum for the same signal singularity and depends
on the wavelet used. In Figure 3.20(b), the wavelet is used as a filter, and the first
singularity of the signal in Figure 3.20(a) corresponds to the minimum in the
wavelet analysis. In Figure 3.21 it is a maximum because a different wavelet is
employed. The former is used here.

-1

-2

-3~~-~-~-r~~-~~
10 12 14 0 10 20 30 40 50 SO 70

(a) (b)
Figure 3.20 Signal (a) and its extrema (b) of the wavelet analysis based on
Daubechies eight coefficients wavelets.
 Chapter 3 Data Pre-processing 53

or - - ' .---

-2

-3
o 10 20 30 40 50 60 70

Figure 3.21 Extrema of wavelet analysis with Daubechies ten coefficients wavelet.

3.3.6 Noise Extrema Removal and Data Compression

The extrema obtained from wavelet multi-resolution analysis correspond to the

singularities of the signal, which may also include those produced by noise,
depending on the analysis scales. Therefore in feature extraction it is necessary to
further filter out noise extrema from the wavelet transform. The most classical
technique of removing noise from a signal is to filter it. Part of the noise is removed
but it may also smooth the signal singularities at the same time. Mallat and Hwang
[92], Mallat and Zhong [93] developed a technique for evaluating noise extrema in
wavelet analysis. Some noise maxima increase on average when the scale decreases
or do not propagate to larger scales. These are the modulus maxima which are
mostly influenced by noise fluctuations.
Figures 3.22 and 3.23 illustrate this. In Figure 3.22, three different noise
frequencies are studied. The wavelet multi-resolution analysis is shown on the left,
and extrema of wavelet analysis are on the right. Clearly, the extrema will decrease
and then disappear as the scale increases.
Figure 3.23 shows a noise signal which is basically the sine in Figure 3.20(a) plus
white noise and the multi-resolution wavelet analysis elements. Noise components
are reduced and then disappear as the scale increases. The results for scales 4 and 5
are similar to that of Figure 3.20(b) which has no noise. This shows that the extrema
of the real trend are retained while noise extrema are filtered on the higher scales.
54 Data Mining and Knowledge Discovery for Process Monitoring and Control

The original signal with noise

1.5 r--T"'""-...----r--~-_._

0.5

o 1---,,-_---lL-_ _---' '--

-0.5 ~
-1 ~-~-~-T_-T_-+
o 20 40 60 80 100

Wavelet analysis Extrema of wavelet analysis

2~-~-~-~--r-~ 2~--r---r---r---..--____._

1.5 1.5
1

1 j
:~~~,l----JI~----~~1
0.5
o 1-....1'-----..lI,.-------..II.---+ Scale I
-0.5 ' ,
-1 -1
-1.5 -1.5

-2~-~-~-~-~~ -2 ~--I______t _ _--_--___+

o 20 40 60 80 100 o 20 40 60 80 100

2~--r-~~~~~--~
2~-~-~-~-_._---.
1.5
1.5

0.5
o 0.5 ~ Scale 2
-0.5 -:5~--------I.~y~~
-1
-1
-1.5
-1.5
-2 ~--_--I___I--____t_-I
-2~--1______t _ ____t--_---I
o 20 40 60 80 100
o 20 40 60 80 100

2r---r-~~~~_.--_.

1.5 1.5
1 1
0.5 0.5
o Scale 3
O~-----------------+
-0.5 -0.5
-1 -1
-1.5 -1.5
-2~--~--~--+_--T_~ -2~_+_-+--~-~-_4
o 20 40 60 80 100 o 20 40 60 80 100

Figure 3.22 Noise signal, its wavelet transfonn and the extrema of wavelet
transfonn_
 Chapter 3 Data Pre-processing 55

The original signal with white noise

Wavelet
mUltiscale1':~
extrema
1
analysis 0.5
t
o
-0.5
-1~--~-----r----T-----r---~~L--r----+
o 10 20 30 40 50 60 70

Extrema ofthe detail ofthe multi-resolution analysis

l ~I- - 1~- -;- - - .,- - -r- - -+1

Scale I

o 10 20 30 40 50 60 70

Scale 2

f!~I------t-----t-~1
Scale 3

o 10 20 30 40 50 60 70

Scale 4

lh: : i ~ : I
Scale 5

A :
o 10 20 30 40 50 60 70

Figure 3.23 Noise signal and its multi-resolution analysis.

A: -approximation of multiresolution analysis, detail. D: -
56 Data Mining and Knowledge Discovery for Process Monitoring and Control

3.3.7 Piece-wise Processing

Two observations can be made from the above discussion. Firstly, extrema analysis
using wavelet multiresolution analysis remains steady with increase in scale. For
example, in Figure 3.23 when the scale is increased from 4 to 5, the four extrema
remain. Secondly, the location of extrema may slightly shift with time as the scale
increases. In Figure 3.23, the extrema representation in scale 4 is a vector of
dimension 70,

Scale-4 = ( .... x5 ................. x23 ............. x37 ............... x53 ................. )

where x5 stands for a non-zero datum.

While in scale 5, it becomes

Scale-5 =(...... x7 .............. x22 ............... x38 ............... x54 ................)

The non-zero datum in the position 5 of scale-4 is shifted to the position 7 of

scale-5. This inconsistency should be avoided. For instance, we should consider
(2,0, ... 0,3) and (2,0 ... ,3,0) to be different. This is necessary especially when
considering the trends of a variable at different operational conditions.
The extrema representation results in very sparse vectors. This is true for dynamic
responses at low frequencies. The method used here is termed piece-wise
processing. The idea is to map a highly sparse vector to a denser vector by
dimension reduction. For example, for the example of scale-4 and scale-5 discussed
above, if the piece-wise sub-region is fixed as four data points, then scale-4 and
scale-5 will be transformed to vectors of dimension 18.

Scale-4' = (.x2 ... x6 ... xlO .. x13 ..... )

Scale-5' = (.x2 ... x6 ... xlO .. x13 ..... )

It is clear that after piece-wise processing, the dimension is reduced and sca1e-4'
and scale-5' are consistent. Therefore using a piece-wise processing technique, it is
possible to achieve consistent feature extraction and reduction in dimension.
 Chapter 3 Data Pre-processing 57

3.4 Episode Approach

This section describes qualitative interpretation of dynamic trends using the episode
approach. It was developed earlier than PCA and wavelets and is straightforward.
However it normally suffers from being week in dealing with noise.
The episode representation approach was originally developed by William [96].
Janusz and Venkatasubramanian [77] adapted it and used nine primitives to
represent any plots of a function, as shown in Figure 3.24. Each primitive consists
of the signs and the first and second derivatives of the function. This means, each
primitive possesses information about whether the function is positive or negative,
increasing, decreasing, or not changing, and the concavity. An episode is an interval
described by only one primitive and the time interval the episode spans. A trend is a
series of episodes that when grouped together can completely describe the
qualitative states of the system. C and D in Figure 3.24 are actually not primitives
because they can be regarded as the combination of A, F and B, E. Therefore they
can be reduced to seven primitives as shown in Figure 3.25.

lc~ ~
A(+,+,-) B (+,-,+) C (+,0,-)

~ ~ b
D (+,0,+) E (+,+,+)

~ ~ l=-
F(+,-,-)

G (+,+,0) H (+,-,0) 1(+,0,0)

Figure 3.24 Nine primitives used in episode approach.

 58 Data Mining and Knowledge Discovery for Process Monitoring and Control

A combination of episodes will fonn a trend over an interval and is described by a

primitive and the associated time. Primitives are different for first and/or second
order derivatives, so the distinguishing points between episode segments are the
extrema and inflexions where

ax
-=0
at
Concave Upward Concave Upward
Monotonic Increase ) Monotonic Decrease
[ax]>o ~ [ax] < 0
[00 x] > 0 [00 x] > 0
(a) (b)
Concave Downward
Monotonic Increase Concave Downward
Monotonic Decrease
[ax] > 0
[00 x] < 0 (~ [ax] < 0
[00 x] < 0
(c) (d)
Linea<ine,ea"~ Constant \ Lin,", Decre"",
[ax] > 0 [ax] < 0
[oox] = 0 [ox] = 0 [oox]=O
(e) (g) (f)

Figure 3.25 The seven primitives episodes.

This is also illustrated in Figure 3.17, where trend 1 consists of primitives c-d-b-
a-c-d-b. The connection points c-d, b-a, are extrema, maximum or minimum points
respectively, and inflexion points are between d and b, a and c, and d and b. Trend 2
in Figure 3.17 illustrates another case.
The task of identifying the episodes from a signal is simply to identify the
inflexions and/or extrema, i.e., singularities in the signal since they correspond to
distinct points of the episode segments. This means that the singularities of a signal
contain the most important infonnation about the trend. Using singularities for
feature representation therefore completely defines the episodes characteristics of a
signal.
 Chapter 3 Data Pre-processing 59

However, the singularities are strongly influenced by noise and this is the major
weakness of this approach. Noise components must be identified and filtered from
the features, otherwise the representation will be misleading. Bakshi and
Stephanopoulos [81, 90] used the wavelet approach developed by Mallat [97],
Mallat and Zhong [93] and Mallat and Hwang [92] for detecting the inflection
points, as described in sections 3.3.5 and 3.3.6.

3.5 Summary

In process operational state identification, dynamic transient signals are probably

more important than the instant values of variables, therefore the discussion in this
chapter has focused on pre-processing of transient signals. The purpose is to remove
noise components, extract the features in a reduced data dimension and concept
formation. The extracted features can then be used by various tools including
multivariate analysis, supervised and unsupervised clustering as well as conceptual
clustering which will be introduced in Chapters 4 to 7.
Signal pre-processing using wavelet multi-scale analysis is developed based on
the fact that irregularities and singularities contain the most important information
of trend signals. Since the extrema of wavelet transform of signals are able to
capture all the irregularities and singularities of a signal when a filter bank and
wavelet function are selected properly, they are regarded as the features of the trend.
The advantage of being able to capture both the frequency and time features of a
transient signal makes the wavelet feature extraction approach suitable not only for
continuous but also batch operations. The approach described in this chapter has a
number of advantages. Firstly, the extrema of wavelet multi-scale analysis can
completely capture the distinguished points of a trend signal, because the original
signals can be reconstructed. Secondly, the method is robust in the sense that the
features captured do not change with the change of scales of analysis. Thirdly, the
episode representation of a trend is primitive, there are no a priori measurements of
compactness for the representation of extrema of wavelet multi-scale
decomposition. In addition, a wavelet-based noise component removal procedure
has been included so that noise effects can be filtered out. Wavelet analysis has also
been used for model identification [98].
Principal component analysis offers an attractive alternative for pre-processing
dynamic trend signals. This approach is particularly useful for concept formation.
60 Data Mining and Knowledge Discovery for Process Monitoring and Control

For a specific variable, its dynamic responses under various disturbances or faults
can be effectively discriminated by inspecting the location of a single data point on
a PCA two dimensional plane. Concept formation is an essential step for conceptual
clustering which, to be introduced in Chapter 7, is an approach that can develop a
descriptive language for clustering operational states.
Episode based approaches are able to convert the signal information to qualitative
descriptions however often suffer from the adverse effect of noise components.
Available episode approaches have not addressed how they could deal with these
effects.
CHAPTER 4
MULTIVARIATE STATISTICAL ANALYSIS FOR DATA
ANALYSIS AND STATISTICAL CONTROL

There has been an increasing interest in applying multivariate statistics to analysis

of historical databases of process operation and designing multivariate statistical
control systems. The methods introduced in this chapter include principal
component analysis (PCA), partial least squares (PLS), and multi-way and nonlinear
PCA. The emphasis will be put on how these approaches can be applied to solving
practical problems and addressing the advantages as well as limitations. The
introduction to relevant mathematical background knowledge is less rigorous
because there are already a large number of text books available.

4.1 peA for State Identification and Monitoring

Principal component analysis (PCA) has been introduced in Chapter 3 as an

approach for feature extraction and concept formation from dynamic transient
signals. In this chapter, PCA is used to develop process monitoring systems through
analysis of historical operational data.

4.1.1 Operational State Identification and Monitoring

Using peA

Process monitoring and diagnosis is conducted at two levels [8]: the immediate
safety and operations of the plant usually monitored by plant operators, and the
long-term performance analysis monitored by supervisors and engineers. Long term

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
62 Data Mining and Knowledge Discovery for Process Monitoring and Control

perfonnance deterioration such as product quality degradation is considered as more

difficult to be diagnosed than a sudden failure of equipment [102]. Trouble-shooting
long tenn perfonnance deterioration often requires examination of historical data
which is large in volume and multivariate in nature. In a number of studies, PCA has
proved to be a useful tool for this purpose. Here we use an example to illustrate the
approach.
MacGregor and Kourti [103] reported an industrial case study of a continuous
recovery process. The process consists of 12 separators to separate a feed of three
major components A, Band C into three products, with the first product #1
comprising mainly component A. The main objectives of operating the plant are to
maintain the concentration of component A in the product # 1 to be at a specified
level, i.e., no below 99.5 %, while achieving a certain minimum recovery, i.e.,
greater than 92%. However in the last three months from when the data was
supplied by the company the recovery dropped significantly below 92%, as depicted
by Figure 4.l.
The data to be analysed is daily averaged and covers 498 days of operation. Each
data pattern has 447 process and 5 product variables. The purpose of the analysis is
to find out why the recovery of product #1 has dropped and what we can do to move
it back.

92 0/0
\---
9J
00
70
~
0 100 200 :m 400 500
Days

Figure 4.1 The recovery history of the product component A.

 Chapter 4 Multivariate Data Analysis and Statistical Control 63

The fIrst step involves PCA analysis of the data matrix of the 442 x 498 (process
variables x number of data patterns). It was found that the fIrst 7 PCs can explain
93% in purity variation and 93% in recovery. Projection of the fIrst two PCs to a
two dimensional plane indicated that for the last three months (data points 401 to
490), where the recovery was low, the process operation had changed to a new
operational state. It clearly explains the reason for dropped recovery over the last
three months, which would have been very difficult to fInd out without this
multivariate analysis.

15

5
T2
-5
...

-15 401 - 490

-20 -10 0 10
Tl

Figure 4.2 Projection of process operational history on a PC two dimensional

plane.

If we regard the area in the circle of Figure 4.2 as normal operation, then this
diagram can also be used for a multivariate statistical monitoring. If the operating
point goes outside the region then the operation can be regarded as abnormal.
Kresta et al. [104] gave other examples using the same approach to design statistical
monitoring systems for a fluidised bed reactor and a binary distillation column.
 64 Data Mining and Knowledge Discovery for Process Monitoring and Control

4.1.2 Multivariate Quality Control Charts Based on PCA

4.1.2.1 Multivariate Shewhart Charts

Traditionally, univariate statistical process control charts, e.g., the Sherwhart chart
shown in Figure 1.2 have been used in industry to separately monitor either a few
process variables, or key measurements on the final product which in some way
defme the quality of the product. The difficulty with this approach is that these
quality variables are not independent of one another nor does any of them
adequately defme product quality by itself. The difficulties with using independent
univariate control charts can be illustrated by reference to Figure 4.3. Here only two
quality variables (y J, Y2) are considered for ease of illustration. Suppose that, when
the process is in a state of statistical control where only common course variation is
present, YJ, and Y2 follow a multivariate nonnal distribution and are correlated
(pYj..l'2 = 0.8) as illustrated in the joint plot ofYJ vs. Y2 in Figure 4.3.

..
.... .- ............
:
........
...
........ . :@.' ."
:~
. .. ...... -.: :
~

.'

l Thue-+

~I ... . : I!
g.! • : · • !

i i
~ I .. . !!
I.

. · ! ·
I .
I
~
. ' '
!

I . . .'· ... @ I
ee

". '
, 'e

I . .....• .:- · I
~

..... . ;
.
!
i II
· '.' • !
!

Figure 4.3 Quality control of two variables illustrating the misleading nature of
univariate charts.
 Chapter 4 Multivariate Data Analysis and Statistical Control 65

The ellipse represents a contour for the in-control process, and the dots represent
a set of observations from this distribution. The same observations are also plotted
in Figure 4.3 as univariate Shewhart charts on YI, and Y2 vs. time with their
corresponding upper and lower limits. Note that by inspection of each of the
individual Shewhart charts the process appears to be clearly in a state of statistical
control. The only indication of any difficulty is that a customer has complained
about the performance of the product corresponding to the EB in Figure 4.3. The
true situation is only revealed in the multivariate YI vs. Y2 plot where it is seen that
the product indicated by the EB is clearly outside the joint confidence region, and is
clearly different from the normal "in-control" population of product.
In multivariate data analysis, the ellipse in Figure 4.3 is determined by calculating
the Mahalanobis distance [106]. In the discussion of bivariate samples the quantity

(4.1)

is often used to describe the locus of an ellipse in two-dimensional space with centre
(YI 'Y2)' This quantity also measures the square of the Mahalanobis distance
between the point (YI, yJ and the centre CY I 'Y2)' All points on this ellipse have the
same distance m2 from (iii 'Y2) .
This can be extended to multivariate situation and used for designing multivariate
Shewhart charts, called x 2 and r charts for statistical process control. This was
originated by Hotelling [107] and several references discussed the charts in more
detail [103, 105, 108, 109, 110].
Given a (k xl) vector of variables y on k normally distributed variables with a
covariance matrix L, the Mahalanobis distance from the centre equivalent to
Equation 4.1 is
(4.2)
L is also called in-control covariance matrix. If L is not known, it must be
estimated from a sample of n past multivariate observations as
S = (n-1)-1 ±(Yi - Y)(Yi-y)T (4.3)
i=1

When new multivariate observations (v) are obtained, then Hotlling's r statistic is
given by
(4.4)
 66 Data Mining and Knowledge Discovery for Process Monitoring and Control

T can be plotted against time. An upper control limit (VCL) on this chart is given
by [105],
2 _ (n -l)(n + l)k (k k
TUCL- n(n-k) Fa ,n- ) (4.5)

where Fa (k,n - k) is the upper lOOa% critical point of the F distribution with k
and n-k degrees offreedom [III].

70 I Hotelling's Tl

•
4 95 0/0

/
10
0 100 200 300 400 500
Days

Figure 4.4 The Hotelling's r chart for the recovery process indicating a deviation
from normal operation around the 400th day

4.1.2.2 Multivariate Quality Control Charts Based on PCA

Since many of the process variables are autocorrelated, only a few underlying events
are driving a process at any time, and all these measurements are simply different
reflections of these same underlying events. Principal component analysis therefore
can be applied to process the data first before the multivariate Shewhart charts are
used. It means the latent variables, i.e., the first few PCs are used rather than the
original variables. If the first A PCs are used, then the Hotlling's r can calculated
by [105],
A 2
T~ = L t~ (4.6)
;=1 Sf,
 Chapter 4 Multivariate Data Analysis and Statistical Control 67

where s~ is the estimated variance of Ii' If A =2, a joint 1/ vs. I] plot can be used.
Note that the traditional Hottelling T in Equation 4.4 is equivalent to
A t2 k t2
T2 = Li+ L i (4.7)
i=1 s~; i=A+1 s~;

For the recovery process, the r chart is shown in Figure 4.4. The 95% confidence
limit was determined based on good operation where the recovery is around 92%.
Had the chart been on-line the deviation from normal could have been detected
based on process data only, immediately when it occurred around observation 400.

2500
SPEy

1500 99 %

o 100 200 300 400 500

Days

Figure 4.5 The SPEy chart for the recovery process.

However, monitoring product quality via T based on the first A pes is not
sufficient. This will only detect whether or not the variation in the quality variables
in the plane of the first A pes is greater than can be explained by common cause. If
a totally new type of special event occurs which was not present in the reference
data used to develop the in-control peA model, the new pes will appear and the
new observation Ynew will move off the plane. Such new events can be detected by
computing the squared prediction error (SPEy) of the residual of a new observation,
k 2
SPEy = L(Ynew,i-Ynew)
A

(4.8)
i=1
68 Data Mining and Knowledge Discovery for Process Monitoring and Control

where Ynew is computed from the reference PCA model. SPEv is also refereed to as
Q statistic or distance to the model. It represents the squared perpendicular distance
of a new multivariate observation from the projection space. Figure 4.5 shows the
SPEv chart for the recovery process. The 99% confidence limit was determined
based on good operation when the recovery is around 92%.
When the process is "in-control", this value of SPEy should be small. Therefore a
very effective set of multivariate statistical control charts is a r chart on the A
dominant orthogonal PCs (t/, t2, ... , tA) plus a SPEychart.

4.2 Partial Least Squares (PLS)

Given two matrices, an (n x m) process data matrix X, and an (n x k) matrix of

corresponding product quality data Y, one would like to extract latent variables that
not only explain the variation in the process data (X), but that variation in X which
is most predictive of the product quality data (Y). PLS is a method which
accomplishes this by working on the sample covariance matrix (XTy)(yTX). In the
most common version of PLS, the first PLS latent variable tl = wi
x is that linear

combination of the x variables that maximises the covariance between it and the Y
space. The first PLS loading vector WI is the first eigenvector of the sample
covariance matrix XTyyTX. Once the scores II = XWI for the first component have
been computed the columns of X are regressed on II to give regression vector PI =
XII! ti
II and the X matrix is deflated to give residuals X2 = X-II pi.
The second
latent variable is then computed as t2 = wJx where W2 is the first eigenvector of
XIyyTX2 and so on. As in PCA, the new latent vectors or scores (II. 12 , •.• ) and the
loading vector (WI. W2, ... ) are orthogonal.

4.3 Variable Contribution Plots

Although the rand SPEy charts are powerful ways for detecting deviations from
normal operations, they do not indicate reasons for such deviations. This can be
achieved by plotting variable contribution plots. There are two alternative ways of
doing this which will be illustrated by reference to a case study described by
MacGregor et al. [113], a low-density polyethylene tubular and autoclave reactor. A
database of dimensionality of 55x 14 (number of observations x number of process
variables) was analysed using PCA and PLS. The PC two dimensional plane of PLS
 Chapter 4 Multivariate Data Analysis and Statistical Control 69

analysis as well as the SPEy plot are shown in Figures 4.6 and 4.7. From Figure 4.6
it can be seen that from data point 53 the operation deviates from normal operation.
The deviation point is detected at point 54 on the SPEy plot. This difference is not
significant and our focus here is on how to find out which variable is the main factor
contributing to the deviation.

4
-- --.- ...........
,,
",I "
",

I '
2 / \

...
\

,,
/

... .. .
I \
\
\

~ 0

_1 .55 . 54 •~~
""T
\
\
52
••
••• 5".II. • ..
:.-
-.

I
I
I
)
\ I
_2
.,
\ /

,,
/
"
.....----
.
_3 /
.... .... "
_4

_5
-10 -5 o 5

Figure 4.6 PLS trt2 plane for the low-density polyethylene reactor indicating
deviation from data pint 54.
70 Data Mining and Knowledge Discovery for Process Monitoring and Control

80

70

60

50

>.. 40
W
a.
(f)
30

20

... .. ....... .. . 51
10
.......- .' • •52
"
""
0
0 10 20 30 40 50 60

# of observations

Figure 4.7 PLS SPEy chart for the low-density polyethylene reactor indicating
deviation from data pint 53.

3.5

3 r-
~

2.S

2 f-
e
~

W 1.5
C
0 r-
r-
U r-

I
~

'C
~ 0.5
a.
I--
0
Tin out' Tcm1 "\,02
" '2 Fi1 Fa
F" F"
P ....

""'" L--

-o.s l...-

L..-

-1
o 2 4 6 8 10 12 14 16
# of X-Variables

Figure 4.8 PLS prediction errors in the individual process variables contributing
to SPEy at time point 54.
 Chapter 4 Multivariate Data Analysis and Statistical Control 71

One way is to plot the SPE prediction error for the deviation point, say point 54,
against the process variables, as shown in Figure 4.8.
To diagnose the event one can examine the contributions of the individual
variables to this larger than normal value of the SPEy at point 54 that is,
k
SPE x ,54 = L(X54,r X54,ji
)=1

Here the predictions X54,) are made from the PLS model developed for the "in-
control" operating data as,
A
X54,) = Lta ,54Pa)
a=1

where the new latent variable projections for the 54th observation are given by,
k
t a ,54 = LWa,)X54,)
)=1

From Figure 4.8 it is clear that the major contributing variable is Z2.

0.5

Tein1 Tcin2
0
mu' ~ mu2 ~ '-- " "

-0.5

~~
-1
-
- '--

i'"
-1.5

'--

-2

'--
-2.5
o 2 4 6 8 10 12 14 16

# of X-Variables

Figure 4.9 PLS variable contributions to the change in t1 from point 51 to 54.
72 Data Mining and Knowledge Discovery for Process Monitoring and Control

An alternative way of diagnosing the event is note that in the latent variable plane
Figure 4.6, the deviation of point 54 is mainly due to a large decrease in tl'
Therefore we can analyse the importance of each process variable to this latent
variable through examining the loading vector WJ. The contribution of each variable
(x;) to this large movement in II can be computed as {(WI.~ Xj); j = 1, 2, ... , k} where
~ Xj = (X54J-X5J,;)' These contributions are shown in Figure 4.9. It confirms that the
main contributing variable is Z2.

4.4 Multiblock PCA and PLS

If the number of variables is not large, generally, the first two or three PCs are
sufficient to capture most of the variance in the data [105, 114]. However if the
number of variables is large, it is necessary to consider more PCs. Increased number
of PCs makes the interpretation of results more difficult. First of all, the two or three
dimensional PC plane, such as Figure 4.2 can not be used. Secondly the variable
contribution analysis becomes difficult. A variation of multiway PCA and PLS
[115] was proposed by MacGregor et al. [113]. The idea is to divide the process
variables (X) into a number of blocks and then perform PCA or PLS analysis for
each block as well as for the entire process. The blocks can be arranged based on
functional and structural analysis of the process to be analysed and the interactions
between blocks should be minimised. MacGregor et al. [113] discussed the
procedures by reference to a low-density polyethylene process.

4.5 Batch Process Monitoring Using Multiway PCA

The above discussed case studies are two-way arrays: the variables and the
observations. In some cases the data takes the form of three-way arrays. Data about
batch process operations is such a case [116]. Batch production consists of batch
runs, one after another. For a typical batch run,j=l, 2, ... , Jvariables are measured
at k= 1,2, ... ,K time intervals throughout the batch. Suppose the data consists of i =
1,2, ... , I such batch runs, then the database will be a three-way array X(IxJxK), as
illustrated in Figure 4.10, where different batch runs are organised along the vertical
side, the measurement variables along the horizontal side, and their time evolution
occupies the third dimension. Each horizontal slice through this array is a (JxK)
data matrix representing the time histories or trajectories for all the variables for
 Chapter 4 Multivariate Data Analysis and Statistical Control 73

.......
........
.-II

]u X
til
CO

•
~\V'LlZ
Measurements
j=l,J

I I

R
I
r=l
n ®
[J +

to
Figure 4.10 Arrangement and decomposition of a three-way array by multi-way
PCA.

single batch, the ith batch. Each vertical slice is an (lxJ) matrix representing the
values of all the variables for all the batches at a common time interval (k).
The multi-way PCA (MPCA) approach proposed by Wold et al. [115] is
statistically and algorithmically consistent with PCA and has the same goals and
benefits. The relation between MPCA and PCA is that MPCA is equivalent to
performing PCA on a large two-dimensional matrix formed by unfolding the three-
way array K in one of the three possible ways. For analysis of batch process
operational data, Nomikos and MacGregor [116] unfolded the data in such a way as
to put each of its vertical slices (lxJ) slide by slide to the right, starting with the one
corresponding to the first time interval. The resulting two-dimensional matrix has
dimensions (lxJK). This unfolding allows us to analyse the variability among the
batches in K by summarising the information in the data with respect both to
variables and their time variation.
 74 Data Mining and Knowledge Discovery for Process Monitoring and Control

4.6 Nonlinear peA

Though peA has been widely studied and applied successfully to many application
areas including chemistry, biology, meteorology and process engineering etc., there
is criticism on this approach being a linear operation. Some researchers have
pointed out that peA can be inadequate in solving some nonlinear problems [117,
118]. Xu et al. [118] has given an example showing that when peA is applied to a
nonlinear problem, minor pes do not always consist of noise or unimportant
variance. If they are discarded, important information is lost; if they are kept, the
large number of pes make the interpretation of results difficult. As a result
nonlinear peA has attracted the interest of some researchers as summarised by
Dong and McAvoy [119].
One way of introducing nonlinearity into peA is the "generalised peA" by
Gnanadesikian [120]. The basic idea of this approach is to extend an m-D variable X
to include nonlinear functions of its elements. For example, for two dimensions
X=(xJ. Xl), three variables can be added: X3= xi, X4= x~, and X5 = X/Xl. Then one can

do the same calculations as linear peA on this 5-D data. Another way of introducing
nonlinearity into peA is "nonlinear factor analysis" [121]. In this method l-D
polynomials are used to approximate m-D data with l<m. A linear least squares
method is used to fmd the coefficients of the polynomials. Dong and McAvoy [119]
commented on the method that for high-dimensional data, it becomes tedious.
Kramer [122] presented a nonlinear principal component analysis method based
on autoassociative neural networks. The architecture of the network is shown in
Figure 4.11. It has five layers, i.e., the input, mapping, bottleneck, de-mapping and
output layers, and its input is used as the desired output. The network is therefore
supervised in nature and can be taught with a backpropagtion learning algorithm.
As no assumptions are needed about the nature of nonlinearity between the
variables, the network can be used in situations where common transformations
(e.g., logarithm, square root) can not be used. The nonlinearity is introduced into the
network by sigmoidal transfer functions in the mapping and de-mapping layers.
The bottleneck layer will perform the dimension reduction, because the number of
neurons in this layer is smaller than that in the input and output layers, so that the
network is forced to develop a computer representation of the input data. The goal
of the network is to minimise the error function
(4.9)
 Chapter 4 Multivariate Data Analysis and Statistical Control 75

where X; is an observation in the data set X and X; , is an output of the network.

Y2

Ym

Input Mapping Bottle- De-mapping Output

Layer Layer neck layer layer
layer

Figure 4.11 An autoassociative nonlinear peA network with five layers. Transfer
functions! are nonlinear and transfer functions I are linear.

Questions can be raised on this approach. An important point is that it is unknown

if the outputs of the bottleneck layer, i.e., the principal components are linearly or
non-linearly independent. It is very clear in linear peA, the pes are linearly
independent. Dong and McAvoy raised other questions. Since there are five layers
training of the network will be difficult. It is also difficult to determine the number
of nodes in the mapping, bottleneck and de-mapping layers. In addition, the
theoretical meaning of the outputs of the bottleneck layer, i.e., the principal
components is not clear. Nevertheless, the approach is still proved to be very
effective in reducing dimension as demonstrated by Kramer [122] in case studies.
76 Data Mining and Knowledge Discovery for Process Monitoring and Control

4.7 Operational Strategy Development and Product

Design - an Industrial Case Study

This section describes an industrial case study of analysing historical data using
PCA for operational strategy development and product design [135].

4.7.1 The FCC Main Fractionator and Product Quality

The fluid catalytic cracking process (FCC) of the refinery converts a mixture of
heavy oils into more valuable products. The relevant section of the process is shown
in Figure 4.12, where the oil gas mixture leaving the reactor goes into the main
fractionator to be separated into various products. The individual side draw products
are further processed by down stream units before being sent to blending units.

FCC Reactor Fractionator

FI Q22

Figure 4.12 The main fractionator of the FCC process.

One of the product is light diesel whose quality is typically characterised by the
temperature of condensation. Traditionally the temperature of condensation has
been monitored by off-line laboratory analysis, which caused time delays because
the interval between two samples is between four to six hours. As a result a software
sensor has been developed using 303 data patterns spanning over nearly a year for
 Chapter 4 Multivariate Data Analysis and Statistical Control 77

predicting the condensation point using fourteen process variables which are
measured on-line (a detail discussion of the software development is given in
Chapter 9). The fourteen variables are listed in Table 4.1.
An interesting problem with the process is that it is required to produce three
product grades according to seasons and market demand, namely -10#,0# and 5#
defined by the ranges of condensation temperature. Because there are more than one
process variable the operators use their experience through trial-and-error to adjust
process variables to move the operation from producing one product grade to
another. There is a clear need to minimise the time of change over because off-
specification product may be produced during transition.

Table 4.1 The fourteen variables used as input to the FFNN model.
Tl-I1 - the temperature on tray 22 where the light diesel is withdrawn
Tl-12 - the temperature on tray 20 where the light diesel is withdrawn
Tl-33 - the temperature on tray 19
Tl-42 - the temperature on tray 16, i.e., the initial temperature of the
pump around
Tl-20 - the return temperature of the pumparound
F215 - the flowrate of the pump around
Tl-09 - column top temperature
Tl-OO - reaction temperature
F205 - fresh feed flowrate to the reactor
F204 - flowrate of the recycle oil
F101 - steam flowrate
FR-1 - steam flowrate
FIQ22 - flowrate of the over-heated steam
F207 - flowrate of the rich-absorbent oil

4.7.2 Knowledge Discovery using peA

The difficulty of the problem comes from the fact that there are fourteen process
variables to consider. Application of PCA to the database of the size 303xl4
(number of data patternsxnumber of process variables) found that the first seven
variables account for about 93% of the variance (Table 4.2). The PCI and PC2 two
dimensional plot is shown in Figure 4.13. It was found that the 303 data patterns are
grouped into four clusters. Three clusters correspond to three products -10#, 5# and
0# and the cluster at the bottom-right comer is found to be a cluster that has a high
probability of product off-specification. Before we analyse how this can be used to
develop operational strategies it is necessary to validate the clustering result since
 78 Data Mining and Knowledge Discovery for Process Monitoring and Control

the fIrst two PCs only account for 53% of the variance. For this purpose, the fIrst
three PCs are plotted in a three dimensional diagram (Figure 4.14). It is found that
the cluster at the centre of Figure 4.13 is further divided into two clusters. Using the
fIrst seven PCs, ART2 gives a similar result as indicated by the dotted curve in
Figure 4.13. This demonstrates that for problems having large dimensions, clusters
may overlap in a two dimensional PC display. Nevertheless, for the current
problem, it is found that the two clusters at the centre of Figure 4.14 both
correspond to product 0#. As a result, in the following discussion, we still use the
result of Figure 4.13.
Therefore the strategy for operation and product design should be to operate the
process in the region of the bottom-left if the desired product is -10#, or the region
at the top if the desire product is 5#, or the region at the middle if the desired
product is 0#, and try to avoid the region at the bottom-right comer. Another point
is that to move from producing -10# to 0#, adjusting PC 1 is more important than
changing PC2. While to switch from producing 0# to 5#, PC2 is more important
than PC 1. Both PC 1 and PC2 are important in avoiding the region at the bottom-
right comer which produces off-specifIcation product.
However, PCl and PC2 are latent variables. To link PCl and PC2 to the original
variables, contribution plots are used. The contribution plot of PC 1 is shown in
Figure 4.15, from which it is found that the most important variables are TI12 (the
temperature on tray 20 where the product is withdrawn) and TI42 (the temperature
on tray 16 close to the flashing zone). Some other variables are not important such
as FR-1. The above discovery is confIrmed by looking at the change of TI-12 over
the 303 data patterns (Figure 4.16). It clearly shows that TI-12 can distinguish
product -10# from 0# and 5#, but can not distinguish 0# and 5#.
The contribution plot of PC2 is shown in Figure 4.17 which indicates that FR-l is
the most important variable. The changing profIle of FR-l for the 303 data patterns
are shown in Figure 4.18. It clearly shows that FR-l can distinguish product 5# from
0# and -10#, but not between 0# and -10#. The fIgure also confIrms that FR-I is not
important to PC 1.
Therefore the operational strategy for product design should be that if we want to
change from producing -10# to 5#, we should increase TI-12 and TR-42 and then
increase FR-1. In order to avoid off-specifIcation product we should carefully
monitor TI-12, TR-42 and FR-l to avoid the region at the bottom-right comer. Of
course it is important to be aware that fIne tuning of all the variables is necessary
but this guidance can help operators to move the process from producing one
 Chapter 4 Multivariate Data Analysis and Statistical Control 79

product quickly to another product. The PC 1 and PC-2 two dimensional plane can
also be used by operators as a monitoring screen as demonstrated by Figure 4.19.

T a bI e.42 Summary 0 f the PC s

PCs Eigenvalues Percentage of Cumulated
Eigenvalues Percentage
1 5.0729 36.2352 36.2352
2 2.3663 16.9017 53.1369
3 1.9453 13.8953 67.0322
4 1.1271 8.0506 75.0828
5 1.0382 7.4155 82.4983
6 0.7757 5.5406 88.0389
7 0.5845 4.1751 92.2140
8 0.4485 3.2032 95.4173
9 0.3034 2.1670 97.5843
10 0.1350 0.9641 98.5484
11 0.0795 0.5679 99.1163
12 0.0682 0.4873 99.6036
13 0.0385 0.2750 99.8786
14 0.170 0.1214 10.0000

PC2
Data patterns
192-210,272-277

Product 5#
...
Product 0#

-0.3

l'
.f/ '
(/
Data patterns
...... -- ....
-0.2
,
1-116,213-242 Data patterns
-0.3
117-124,211,212,
Product -10#
243,244,278-288

PCI
Figure 4.13 The PC 1 and PC2 two dimensional plot.
 80 Data Mining and Knowledge Discovery for Process Monitoring and Control

0.4
PC2
0.3
••
0.2
•• ,1.= ••
•
•• •
0.1 •
••
0 I~ifI'r
....
••••~4f.
-0.1 • • ••

-0.1

PC3
-0.1 0 0.1 0.2 0.3
-0.2
PCI
Figure 4.14 The PCI, PC2 and PC3 three dimensional plot.

PCI

0.6~-----------------------------------,

0.4 ~~~--~~--------------------------~

Original variables

Figure 4.15 The contributing plot of PC 1.

 Chapter 4 Multivariate Data Analysis and Statistical Control 81

T1-12
·C
280 ;--_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _---, High
prob.
270 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . lproduct
260 ..... off-spec.
250
========~~==~=F~~~~O
{J.
240 +......................................................................1 0# 5#
o
230
.=====.J="""'="=====~=~==....,.,.j {!. -10#
220

210

1 19 37 55 73 91 109 127 145 163 181 199217235 253271 289

Figure 4.16 The changing profile ofTI-12 over the 303 data patterns.

pe2
0.6

0.4

0.2

-0.4

f-
,
N

f-
,
M
";"
E=
N
"'f
f-
0
~
f-
on
N
LI.
9'"
E=
0
9
0
f-
0
N
LI.
0
N
LI.
0
LI.
-
LI.
==
N
2'
LI.
.....
0
N
LI.
-0.6
Original variables

Figure 4.17 The contributing plot ofPC2.

82 Data Mining and Knowledge Discovery for Process Monitoring and Control

FR-l
11
5#
10
product
9
4J
8 0# -10#
7
product

6 D
High
5 prob.
product
4 +-r,",,-0,""-0""",,,-0,,,_,-0,""rr",-,rr,_,-o,".,-0",-,rr,~ off-spec.
28 55 82 109 136 163 190 217 244 271 298

Original variables

Figure 4.18 The changing profile ofFR-l over the 303 data patterns.

Transfer
matrix

-0

G
Operatin
point

TII2
-0 TI42

( samPle)

C;) -0

Historical
<::J cases

@data

Figure 4.19 A PCA monitoring plane.

 Chapter 4 Multivariate Data Analysis and Statistical Control 83

4.8 General Observations

PCA and PLS have proved to be powerful tools for operational data analysis and
statistical process control. However they still have limitations. PCA and PLS based
data analysis for statistical process control has the assumption that the first few PCs
can capture most of the variations in a multivariate database. This assumption may
be violated in some cases, e.g., when the dimension of the original variables is very
large. Multiblock PCA and PLS can tackle this problem for some applications,
however, dividing variables into blocks may not always be possible. Sammon [123]
gave an example where data generated to contain five groups in four dimensions are
projected into the space of two principal eigenvectors. Visual examination of this
projection shows only four groups, since two of the clusters overlap completely in
the two dimensional space. In such cases some alternative approaches may have to
be used such as the unsupervised machine learning approaches to be introduced in
Chapter 6, including neural network and Bayesian automatic classification methods.
It has reported that one of the Bayesian automatic classification systems - AutoClass
has successfully clustered data with 1204 attributes [39]. However, PCA and PLS
may still be a useful approach for pre-processing the data to eliminate the linear
dependencies in the data. PCA is also a useful approach for pre-processing the data
for dimension reduction for neural networks [124].
The variable contributing plots such as Figure 4.9 may not be applicable in cases
where the contributions of the original variables to the PCs are not equally
distributed. Use of other approaches to compensate this limit of PC A can be a good
alternative. For example, neural network models can be developed and used as
sensitivity study tools to identifY the contributions of variables.
In the above applications, PCA and PLS are used mainly for statistical process
control for long tenn perfonnance monitoring and the data dealt with are averaged
over hours or days. PCA and PLS are also potentially useful for on-line real time
data analysis. In Chapter 3 we have introduced the application of PCA for feature
extraction and concept fonnation from dynamic trend signals. Bakshi [125]
combined wavelet multiscale analysis with PCA for developing on-line monitoring
systems. Tabe et al. [126] combined Fourier and wavelet analysis and PCA and
developed an approach called dynamic PCA.
PCA can also be categorised as an unsupervised learning approach. However its
learning is not recursive or incremental. For on-line real time use, it is useful for
PCA to be able to learn incrementally, i.e., learn from a single example when it is
84 Data Mining and Knowledge Discovery for Process Monitoring and Control

presented. There has been an report on such on-line learning for principle
component analysis [127].
Only a few case studies were mentioned above mainly for the propose of
illustration the methods. There are many successful applications in using PCA and
PLS approaches to analysing databases about continuous and batch operations.
These include analysis of data of emulsion batch processes [128], product design
[129], inferential process model development [130], reactor analysis [131], fault
diagnosis [132], sensor fault identification [133, 134], normal operational region
identification [136] and monitoring [137]. These applications not only explored the
potential applications, but also provide valuable experience in overcoming some of
the limitations ofthe PCA and PLS in solving practical problems.
CHAPTERS
SUPERVISED LEARNING FOR OPERATIONAL SUPPORT

Studies on machine learning have mainly been concerned with automatic learning
from examples to develop the knowledge describing these examples. This is clearly
different from the kind of learning as learning to ride a bicycle. In supervised
learning, each example used is typically described by a number of attributes. The
attributes are divided into inputs and outputs, and the learning process is to develop
a model mapping the multiple inputs and outputs. The model is gradually refined
during learning to minimise the errors between the predictions and real values of
outputs, i.e., so-called supervised learning. The most widely studied supervised
learning approach is the feedforward neural network (FFNN). The FFNN model and
its application to process operational support will be introduced in this Chapter. The
discussion on FFNN will be focused on many of the practical issues that have to be
considered in applying FFNN. While the focus will be on FFNN, other supervised
models will also be described and compared with FFNN. These include fuzzy
FFNN, fuzzy set covering approach and fuzzy signed digraph.

5.1 Feedforward Neural Networks

5.5.1 FFNN Architecture

There are already a large number of textbooks on FFNNs. Here it is introduced less
technically. Simply speaking, a FFNN neural network is an algorithm or computer
software that can learn to identify the complex nonlinear relationship between
multiple inputs and outputs. The learning process has a number of characteristics.

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
86 Data Mining and Knowledge Discovery for Process Monitoring and Control

Firstly, FFNN does not need fundamental domain problem models and is easy to be
set up and trained. This is different from conventional statistical methods that
usually require the user to specify the functions over which the data is to be
regressed. In order to specify the function, the user has to know the forms of the
equations governing the correlations between the data. If these functions are
incorrectly specified, the data will not be satisfactorily regressed. Furthermore,
considerable mathematics and numerical experience is required to obtain
convergence if these equations are highly nonlinear. FFNN does not need to specifY
the forms of the correlations as well as any mathematical and numerical expertise
requirements. Secondly, data examples used for training are allowed to be imprecise
or noisy, in some cases even incomplete. Thirdly it mimics the human learning
process: learning from examples through repeatedly updating the performance.
A FFNN neural network consists of a number of processing elements called
neurons. These neurons are divided into layers. Figure 5.1 shows a three layered
FFNN architecture including an input, a hidden and an output layer. Typically the
input layer nodes correspond to input variables and the output layer to output
variables. Hidden neurons do not have physical meanings. Neurons between two
adjacent layers are fully connected by branches.

Output layer

Hidden layer

Input layer

Figure 5.1 A three layer feedforward neural network.

Each neuron in the hidden and output layer is described by a transfer function (or
activation function). Usually a sigmoidal function is used,
1
fez) = - 1-07
(5.1)
+e
f(z) transforms an input z to the neuron to the range of [0.0, 1.0] as shown in
Figure 5.2(a). The parameter a in Equation 5.1 is used to change the shape of the
 Chapter 5 Supervised LearningJor Operational Support 87

sigmoidal function. Some other activation functions can also be used as shown in
Figure 5.2 (b) and (c). However for a specific FFNN structure, the neurons in the
hidden and output layers are usually fixed on the same transfer function.

J(z) f(z) J(z)

1.0

z z
z
0.0

(a) Sigmoid (b) Ramp (c) Step

Figure 5.2 Activation functions.

Each connection branch is described by a weight representing the strength of

connection between two linked nodes. The so called learning or training process is
the procedure to adjust the weights. A bias neuron that supplies an invariant output
is connected to each neuron in the hidden and output layers. The bias provides a
threshold force activation of the neuron, and is essential in order to classify
networks input patterns into various subspaces.

5.1.2 FFNN Training Algorithm

Given some arbitrary values for all the connection weights, for a specific data
pattern, the FFNN makes use of the weights and input values to predict the outputs.
The training is intended to gradually update the connection weights to minimise the
mean square error E,
E = f±(tfmLy~m))2 (5.2)
/)/=\ i",l

where M - number of training data patterns

N - number of neurons in the output layer
t: m) - the target value of the ith output neuron for the given mth data
pattern
y;ml _ the prediction for the ith output neuron given the mth data pattern
88 Data Mining and Knowledge Discovery for Process Monitoring and Control

The process involves a forward path calculation to predict the outputs and
backward path calculation to update the weights. For a neuron in the input layer, its
output is equal to the input so there is in fact no activation function for an input
neuron. For a neuron in the hidden and output layers, it receives the values of the
outputs of its front layer nodes and takes the weighted sum as its input. The
weighted sum is then transformed by the activation function to give an output. The
outputs of the output layer neurons are compared with the target values using
Equation 5.2 to calculate an error. The error is used to backwards updating the
weights.
Given the mth data pattern, the weight updating in a supervised learning algorithm
follows the formulation,
(5.3)
where
w;:1) - the weight of the connection between the jth neuron of the upper
layer and the ith neuron of the lower layer, in the mth learning
iteration.
wj:'-ll - the weight of the connection between thejth neuron of the upper
layer and the ith neuron of the lower layer, in the (m-l)th learning
iteration.
~W;:'l - the weight change.
In backpropagation learning approach, the weight change is calculated by,

(5.4)

where
11 - learning rate, providing the step size during gradient descent.
Generally to assure rapid convergence, larger step sizes which
do not lead to oscillation are used
a - coefficient of momentum term, 0< a < 1
eim) - the output value of the ith neuron of the previous layer, in the mth
iteration
8i m ) - the error signal ofthejth neuron in the mth learning iteration.

Ifj belongs to the output layer,

s::lm)
0,
= (t(m)
I
_ yen,)) f'
) J
("
~
(m)
WJ/
(m)
01
+ W,O(m)) (5.5)

and ifj belongs to the hidden layers,

 Chapter 5 Supervised Learning for Operational Support 89

s:(m)
U)
= f' ("
)
i.m)
£- WI' 0,
(m) + WIo( m )£-" Uk
s:(") (m)
Wig (5.6)
i k

where f is the derivative of the transfer function.

Therefore, the error backpropagation approach for adjusting weights computes an
error for each neuron in the output and hidden layers using Equations 5.5 and 5.6,
and recursively updates the weights of all the layers using Equation 5.4, starting
from the output layer and working backwards until the input layer.
Like all first order methods, backpropagation learning is a significantly less
efficient optimisation method than are second order optimisation methods such as
the conjugate gradient or quasi-Newton algorithms. Leonard and Kramer [138]
studied the use of a conjugate gradient approach in order to speed up convergence.
Alternative algorithms have also been proposed by Brent [139], Chen and Billings
[140] and Peel et al. [141]. An attractive approach to the backpropagation learning
algorithm is the quasi-Newton method [142,143].
Since its development, multilayer neural network has shown surprisingly good
performance in solving many complex problems. However there is still a lack of
theoretical explanation but an interesting theorem sheds some light on the
capabilities of multi-layer percetrons [144]. This theorem states that any continuous
function of N variables can be computed using linear summations and nonlinear but
continuously increasing functions of only one variable. It effectively states that a
three layer percetron with N(2N+ 1) nodes using continuously increasing nonlearities
can compute any continuous function of N variables.

5.1.3 Parameter Selection and Training Techniques

From the above discussion, it is clear that FFNN training involves the following
initial decisions to begin: network topology, i.e., number of hidden layers and
hidden neurons; learning rate; momentum factor; error tolerance or number of
iteration; initial values of weights. Learning rate 11 and momentum factor a are not
very difficult to set. We can start with some reasonable values, e.g., 11 = 0.35, a =
0.7 and then find the most appropriate values in training. The error tolerance
apparently depends on the problem to be solved. Understanding how these
parameters affect the learning performance, which is due to discuss next, is useful in
setting the right values. Initial values of weights can be generated by a random
number generator. Therefore here our discussion will focus on network topology
and then some other important issues in learning.
 90 Data Mining and Knowledge Discovery for Process Monitoring and Control

5.1.3.1 Network Topology

It is generally accepted that only one hidden layer is necessary in a network using a
sigmoid activation function, and that no more than two are necessary if a step
activation function is used.
There are no available methods to decide how many hidden neurons are required
in a three layered network. The number of hidden neurons depends on the
nonlinearity of the problem and error tolerance. Empirically the number of neurons
in the hidden layer is of the same order as the number of neurons in the input and
output layers. The number of hidden neurons must be large enough to form a
decision region that is as complex as required by a given problem, too few hidden
neurons hinder the learning process and may not be able to achieve the required
accuracy. However, the number of hidden neurons must not be so large that many
weights required can not be reliably estimated from available training data patterns.
An unnecessary large hidden layer can lead to poor generality. A practical method is
to start with a small number of neurons and gradually increase the number.

5.1.3.2 Local Minima

Chemical process models are multidimensional with peaks and valleys [145], which
can trap the gradient descent process before it reaches the system minimum. There
are several methods of combating the problem of local minima [146, 147]. The
momentum factor a , which tends to keep the weight changes moving in the same
direction, allowed the algorithm to slip over small minimal. Another approach is to
start the learning again with different set of initial weights if it is found that the
network keeps oscillating around a set of weights due to lack of improvement in the
error. Some times adjusting the shape of the activation function (e.g., through
adjusting the constant a in Equation 5.1 can have an effect on the network
susceptibility to local minima. Some new optimising approaches have been applied
to multilayer neural networks which prove to be able to address the local minima
significantly, such as the simulated annealing [147]. In contrast to the comments
made by Crowe and Vassiliadis [145] and Chitra [147], Knight [146] thought that
FFNN rarely slips into local minima. However, it should be dealt with care.
 Chapter 5 Supervised Learningfor Operational Support 91

5.1.3.3 Over fitting or over parameterisation

Overfitting occurs when the network learns the classification of specific training
points but fail to capture the relative probability densities of the classes [148]. This
can be caused by two situations: (1) oversized network, e.g., due to inclusion of
irrelevant inputs in the network structure or too many hidden layers or neurons; and
(2) insufficient number of training data patterns.

5.1.3.4 Generality

Over fitting in training a neural network deteriorates the generality of the network.

5.2 Variable Selection and Feature Extraction for

FFNN Inputs

Inclusion of redundancy or irrelevant variables demands more training data and

causes over fitting and deteriorates generalisation capability. If prior knowledge is
available it should be used to remove irrelevant variables and identify correlated
ones. Otherwise, mathematical techniques are required to solve the problem. In
Chapter 9 by reference to the development of software sensors using FFNN, several
approaches are described in detail. These include principal component analysis,
sensitivity analysis and network weight matrix analysis. Here only a brief
description is given to each method. For detailed discussion, please refer to Chapter
9.
Principal components are the linear combination of all the original variables. They
are orthogonal and therefore linearly uncorrelated. The first few PCs can capture
the variance. Therefore we can use the first few PCs rather than the original
variables as the input variables to develop the FFNN model. Consequently the size
of the network can be reduced, less training data is required and as a result can
improve the generality. A smaller size is also favourable for training speed. A
specific principal component does not represent any specific original input though
relative contribution to individual PCs by the original variables can be analysed.
Sensitivity analysis can also be used to refine a network structure. The idea is to
develop a FFNN model using all original inputs and use this model to carry out
sensitivity studies to find out relative contributions of inputs to an output. Sensitivity
92 Data Mining and Knowledge Discovery for Process Monitoring and Control

studies can be used to develop a simpler network structure. Since the inputs are
possibly correlated, sensitivity study should be carried out with care.
An alternative approach is to analyse the weights of a trained FFNN to find out
the relative importance of inputs. Several approaches have been proposed which
were discussed in Chapter 9. Due to the complex internal structure of FFNN, these
approaches should be used with care.
The difficulty in analysing the relative importance of inputs to an output is that
there is hidden layers and neurons and neurons between two layers are fully
connected. An interesting idea is to develop a model without hidden layer and
neurons, which is traditionally called single layer percetron (PCT). The order of
magnitude of input - output linkage weights is clear. A PCT model may not be
accurate enough but can provide some useful information for analysis. The analysis
result can be used to further develop a FFNN model with hidden neurons. In fact,
as will be demonstrated, for a problem whose nonlinearity is not high, a PCT can
give equally good performance. In this case a PCT clearly should be used rather
thanaFFNN.
For on-line applications, feature extraction also means dimension reduction from
dynamic transients to use minimum data to capture the useful information as well as
noise removal. This has been discussed in detail in Chapter 3.

5.3 Model Validation and Confidence Bounds

The advantage of FFNN not requiring fundamental domain knowledge also brings
a drawback of being a blackbox with poor extrapolation capability. Therefore,
FFNN application always involves training and test procedures: using some data for
training and the rest for testing. However, due to the multivariate nature, when a
new data pattern is given, the confidence of prediction is still unknown, because the
data pattern represents a point in the multidimensional space. The mean squared
error to quantify the accuracy of training does not give this kind of information.
A detailed procedure is introduced in Chapter 9 to address this issue in the course
of developing a software sensor, which uses an automatic clustering approach to
group the multivariate data into clusters, and then training and test data is selected
from each cluster. When new data patterns are available the clustering approach is
also used to test if the data patterns are within the region of previously trained data,
so provides clues of prediction accuracy and information whether the FFNN model
needs to be retrained.
 Chapter 5 Supervised Learningfor Operational Support 93

Techniques used in statistics for error or residual analysis can perfectly be used. A
simple plotting of the error residual can provide useful information. In developing a
software analyser for predicting a toxicity measure, Microtox, it was found that nine
of the 180 data cases used for training and test have abnormally large errors [124].
It was suspicious that these nine data patterns may contain noise components. The
plotting of error distribution supports this. As shown in Figure 5.3 that the error
almost follows a normal distribution and there are irregular structure at the two ends
which correspond to the nine data patterns.

90
c 80
0
'g 70
~ 60
!::
:.a'" 50
g 40
:c«I 30
.J:l
...
0
Q..
20
10
0
-100 -50 0 50 100 150

Relative error

Figure 5.3 Error distribution on predicting a toxicity measure Microtox using

FFNN.

Some researchers [149, 150] tried to develop an accurate procedure to calculate

the confidence bound of FFNN in prediction. The confidence on predicting an
output for a set of given input data depends on a number of factors,
(1) the distribution of residuals of the FFNN model, for training data patterns;
(2) the sensitivity of the output to inputs. For a problem of multiple inputs (xj, Xl>

••. XN) and multiple outputs (Y/. Y2....YM), all the sensitivities form a Jacob matrix,
 94 Data Mining and Knowledge Discovery for Process Monitoring and Control

(5.7)

(3) the density distribution of the data sample in the multidimensional space of
training data.
Shao et al. [149] and Zhang et al. [150] developed a complicated procedure for
calculating the confidence bounds of using a FFNN mode to predict a new data
pattern. Figure 5.4 shows such an example [180].

Model prediction with confidence bounds

4

o mod.1 prediction
3
+ observation
confidence interval
2 - confidence bound

-2

~3~------2~------~1------0~----~------~~--~3
independent varlabl.

Figure 5.4 An example of confidence bounds.

5.4 Application of FFNN to Process Fault Diagnosis

A multilayered neural network can almost solve any multiple input I output
problems. The number of ways in which FFNN can be used in process industries is
limited only by the imagination. In this book our discussion only concerns its
application in designing software sensors which will be dealt with in Chapter 9, and
in process operational state identification and fault diagnosis, which is described in
the following sections. The emphasis on operational state identification and fault
diagnosis will be based on case studies, which are used to reveal the kind of
 Chapter 5 Supervised Learningfor Operational Support 95

challenges that need to be addressed in order to design effective and practical

systems.
Fault identification involves the assimilation of available measurements to
identify malfunction and misbehaviour of the process [151]. Function of a process
equipment is the desired objective. For example the function of a distillation column
is to achieve desired products within specifications, and the function of a heat
exchanger is to increase the temperature of a cold stream or reduce the temperature
of the hot stream to a certain value. The behaviour of a process unit is the status of
the process under operation. Function and behaviour are treated separately because
there is the potential that misbehaviour can be used to predict the malfunction. Both
malfunction and misbehaviour can be described by three types of attributes: process
variables, model equations, and dynamic trends.
Process variables here refer to those which change rapidly over a short time. For
example, a flowrate may change very fast like a step or pulse function. They can be
described simply by high, low, normal, medium high and low. Failure or not failure
of some simple equipment can also be described in this way. For example, we can
describe the failure of a pump as 1 and normal as O.
Model equations. Model equations derived from the material and energy balances,
equilibrium relations and rate equations impose constraints on process variables.
These equations can be written in the form so that they are equal to zero. It is worth
drawing attention to the fact that it is not necessary to list all model equations.
Associated with each model equation are tolerance limits which are the expected
positive and negative values of the residuals for which the constraint equation is to
be satisfied. Violation of model equations may indicate a possible fault [152].
Dynamic trends. Graphically represented dynamic trends provide a direct way for
process operators to make decisions on the operational status of a processes.
However, in order to make efficient use of trends in a computer based system, pre-
processing of the trends, so called feature extraction for the purpose of dimension
reduction and noise removal is needed. The extracted features can then be used for
further analysis by neural networks. Chapter 3 has introduced several approaches.
 96 Data Mining and Knowledge Discovery for Process Monitoring and Control

5.4.1 A Case Study - Fault Diagnosis of a CSTR Reactor

There has been a large number of publications on applying FFNN to process fault
diagnosis, many have used continuous stirred tank reactors as case studies. Here an
earlier case study on a CSTR reactor [153] is described in order to analyse the kinds
of issues that need to be considered. In the CSTR reactor, there is a first-order
exothermic reaction A ~ B. Heat generated by the reaction is taken away by
cooling water. A three layered neural network was developed to use symptom
variables to predict faults, as shown in Figure 5.5.

C(t) High inlet flowrate

T(t) Low inlet flowrate
Vet) High inlet concentration
F(t) Low inlet concentration
T/t) High inlet temperature
F/t) Low inlet temperature

Figure 5.5 A three layer neural network for fault diagnosis of a CSTR reactor.

In Figure 5.5, C(t) is the outlet concentration of component A, T(t) outlet

temperature, Vet), reactor hold up, F(t) outlet flowrate, Tit) outlet temperature of
coolant, Fit) coolant flowrate. The output nodes of the network are possible faults
or operations that can cause the symptom variables to change in certain ways.
Dynamic simulation was used to produce the training data: making a change (5 to
15%) on one or two variables on the output side of Figure 5.5 and recording the
values on all the symptom variables. Because the symptom variables experience
dynamic transients, they were recorded in two ways, either the new steady state
value or an average of four sampling points during the transients. The data was pre-
processed before being used for training the network: each fault variable was given
a boundary to define a variable being fault or not. If it is regarded as being a fault, it
is assigned a value of 1, otherwise, it is assigned O.
The detailed description of the case study can be found in the original publication
[153]. In the following we just want to use this case study as an example to look at
some general issues that need to be considered in applying FFNN to fault diagnosis.
 Chapter 5 Supervised Learning for Operational Support 97

5.4.2 Observations on the CSTR Reactor

Based on the above discussion on applying a FFNN to fault diagnosis of a CSTR,

the following observations can be made:
(1) The study described in the above section considers open-loop situation
because the dynamic simulation was made under open-loop conditions. It is known
that under closed loop control variables will interact even stronger, the symptoms
will change in different ways. Therefore the dynamic transient behaviour of a
variable becomes very important. The time delays, peaks and valleys of a dynamic
trend may contain important information. For example in Figure 5.6, the shapes of
the three response curves for the same variable may tell important information about
the distinction of three disturbances. It is not adequate to use only the new steady
state value or an average of a few sampling points on the dynamic transient signal.
However, it is obviously not suitable to use all the sampling points comprising a
transient signals because the size is too large. Therefore, feature extraction from
dynamic transient signals using appropriate technology for the purpose of dimension
reduction is required. An input variable to the network of Figure 5.5 may need to be
described by several features. Various methods for feature extraction already
described in Chapter 3 can be chosen.

Temperature
250

200

ISO

100
o 100 200 300 400 500 600

Time

Figure 5.6 Dynamic trends of a temperature of a distillation column under closed

loop control.
98 Data Mining and Knowledge Discovery for Process Monitoring and Control

(2) Noise has to be dealt with carefully. Under large noise to signal ratio, the
noise may bury the real trend of a dynamic transient. Methods for noise removal has
also been discussed in Chapter 3.
(3) This CSTR example has six inputs and six outputs. For a larger process there
may be hundred variables to be monitored. It may not be suitable to use a single
neural network. Several neural networks or stacked networks may have to be used as
described by Zhang et al.[154].
(4) Multiple faults also pose a challenge. This has been addressed by many
researchers. However, some researchers have found that networks trained with
single fault data can be used to detect multiple faults. It is rare to have more than
two irrelevant faults occurring at the same time. But it is not unusual that one fault
may cause other faults to occur.
(5) Most discussions on fault diagnosis using multilayerd neural networks are
concerned with failure of equipment, sensors or sudden change of a variable. There
is another type of fault or abnormal operation which is concerned with gradual
degradation of product quality or other performance measures. Though there have
been some discussions on using expert systems to deal with this kind of problems"
little work has been done on how neural networks can be used to deal with this type
of problems.
(6) A necessary step before fault diagnosis is fault recognition or identification. It
clearly depends on effective assimilation of all the measurements. A neural network
may be a useful tool for diagnosis: mapping symptoms to faults, but it is not very
effective in fault recognition. Some researchers used neural networks trained with
normal operational data to identify faults: if the output was not in the normal region
then a fault is expected.
(7) The biggest problem with FFNN is availability of training data. It is
unthinkable that a real plant will initiate some faults to provide training data.
Though dynamic training simulators have been used to generate training data [151,
155], methods for fault diagnosis should not rely on the assumption that a high
flexible customised simulator of high fidelity is available.
The last two points make FFNNs which adopt a supervised learning mechanism
less attractive in fault identification and diagnosis than the multivariate analysis
approaches introduced in Chapter 4 and unsupervised machine learning methods to
be introduced in Chapters 6 and 7.
 Chapter 5 Supervised Learningfor Operational Support 99

5.5 Fuzzy Neural Networks

Descriptions commonly used by engineers in describing a variable being high or

low, or a process being nonnal or abnonnal are inherently fuzzy. These fuzzy
descriptions are a kind of conceptualisation of numerical values, and are more
qualitative and meaningful to operators. Fuzzy mathematics provides a technique for
bridging the gap between qualitative descriptions and numerical values. Figure 5.7
shows how a process variable can be transfonned to fuzzy concepts using fuzzy
membership functions. It states that the variable takes three fuzzy values, high,
nonnal and low. When it takes a fuzzy value, it is also attached with a fuzzy
membership value. For example, (the variable = High, 0.8) is a statement that the
value is high with a fuzzy membership value 0.8.

~ 1.0

t~ 0.0
"'" 0.7
NonnaliztXi value of a process variable

Figure 5.7 Fuzzification of a process variable.

Conventional neural networks have real number inputs and weights. There are
three main types of fuzzy neural networks (FNNs) [156]: FNNs with fuzzy input
signals but real number weights, FNNs with real number input signals but fuzzy
weights, and FNNs with both fuzzy input signals and fuzzy weights. It is the first
type of FNNs, i.e., fuzzy input signals and real number weights that has been studied
for process fault diagnosis [l5I, 155, 154]. In the work of [154] fuzzy membership
functions take part in the learning. In our work, fuzzy membership functions do not
participate in the learning process but are only used for pre-processing the data. The
procedure can be illustrated using Figure 5.8, which illustrates a fuzzy neural
network for two input variables, Xl and X2 and one output y.
100 Data Mining and Knowledge Discovery for Process Monitoring and Control

tU:\ 1- - - - - - - - - -
~I I

~\81 I
:
I
6)1 I-

I
I

/8 1
: r7\_1
\..,L./I-

~
' 6)
~I
I
:
I
I
~
"'----------
I I

Figure 5.8 A fuzzy neural network architecture. L-Iow, H-high, N-normal.

Inside the dashed box of Figure 5.8 is a normal feed forward neural network.
Outside of the dashed box represents fuzzy processing of the data before used for
training. However, Such a FNN increases the size of the network dramatically. Each
nput variable needs to be represented with three nodes. If each input variable is to
Ie expressed by five fuzzy values, i.e., high, medium high, normal, medium low and
ow, it will require five nodes. The FNN used in [154] has a similar problem. An
lternative is to use the structure of Figure 5.9. In this structure each input variable
: always split into a pair of nodes, one is used to describe the fuzzy value being
ther L, M or H, the other to represent the membership value. Case studies of
lplying FNNs for fault diagnosis will be given in Section 5.8

6:)-;:c5 -------:
~~: : I I
~

E0+ :
I :
y

~~
.... _--------
Ire 5.9 An alternative structure of fuzzy neural network.
 Chapter 5 Supervised Learningfor Operational Support 101

5.6 Fuzzy Set Covering Method

If we regard all the symptoms of faults as a symptom fuzzy set M,

M=(ml,m2, ... ,mN) (5.8)
and all faults form a fault fuzzy set D,
D = (dl,d2, ... dT) (5.9)
then fault diagnosis is to develop a mapping between the two fuzzy sets. Based on
this idea, a fuzzy set covering (FSC) method was developed [l51]. It is designated
this name because it was initially inspired by the crisp set covering approach
developed by Reggia et al.[ 157] which was also appreciated by Penalva et al. [158].
In [151] an approach was discussed to convert three types of symptoms, namely
process variables, model equations and dynamic trends to fuzzy symptom concepts.
With the progress in technology for pre-processing measurement signals as
introduced in Chapter 3, it is possible to use better approaches. Therefore here we
do not introduce the way to convert symptoms to fuzzy concepts introduced in
[151], but focus the discussion on the fuzzy set mapping algorithm.
For a given diagnostic task, the extent to which symptom mi exists is called the
grade of membership of mi in M , and is represented as ai. All ai form a fuzzy set
A,
(5.10)
The degree to which mi(i= 1, N) is caused by dj(j= 1, T) represents the fuzzy
relationship between fuzzy sets M and :6. This fuzzy relationship can be
represented as a fuzzy matrix R,
R=(ri,j)NxT, O~fj,j~ I, i=I,N, j=I,T (5.11)
The failure likelihood of fault dj is represented by bj and can be obtained from the
following fuzzy set operation,
Ao R = B =(bl,b2, ... ,bT) (5.12)

bj=(al *fI,i~ (a2 *r2,j)···(aN *fN,j), j=I,T

where operator 0 is represented as M(*, *) and can have different forms, as shown
in Table 5.1. Wang et al. [151] only discussed the M(e, +) operator, comparisons of
different operators were not carried out. A learning approach adapted from
feedforward neural networks were used to learn the fuzzy matrix.
102 Data Mining and Knowledge Discovery for Process Monitoring and Control

Table 5.1 Fuzzy matrix operators.

N
Modell M( 1\, V) bj= 1':1 (ail\ri,j) j=1,2, ... T
N
Model 2 M(e, +) bj= L
;=1
ali,j j=I,2, ... T

N
Model 3 M(e, V) b·- V airi,j
J- 1=1 j=1,2, ... T
N

Model 4 M(I\,Ei3) bj=Ei3 L ;=1

ail\ri,j j=1,2, ... T;

a Ei3 b=min {I ,a+b}

N
Model 5 M(e,Ei3) .t.... a'r"
bJ.=Ei3" 1 1,J J'-I
- , 2 , ... T
;=1

7 Fuzzy Signed Digraphs

recent years a notable development in fault diagnosis has been the signed directed
lph (SDG). Since it was first proposed for fault diagnosis by Iri et a!. [159] it has
:acted much attention [160 - 171, 194]. It is attractive because it provides an
gant and straight forward tool for qualitatively analyse the cause-effect
ltionships between variables. Neural networks are clearly not capable in this
ect. However, there are common limitations in all the SDG models. Firstly, the
ressive capability is very limited since they are crisp graphs - a node or a branch
only take three values, i.e., -, 0 and +. As a result it will give ambiguous
tions in complicated fault diagnosis. The application of fuzzy concepts by Han
'. [172] and Shih and Lee [173] only makes the input nodes to be able to
ert numerical data to qualitative expression but the graph as a whole is still a
one. Secondly the reasoning methods in a SDG model are often dependent on
, over simplified assumptions. As a result large errors can be expected when
ning in a complex structure. Thirdly, the development of the SDG has been
:ation driven. For example, the algorithms have been developed specifically
mIt diagnosis and can't be applied directly to qualitative simulation.
 Chapter 5 Supervised Learning for Operational Support 103

Furthennore, the SDG models are not able to deal with uncertainty in data and
reasoning simultaneously. Finally, they are not able to learn from data.
A fuzzy-SDG method [59, 60, 174] was developed which has far more features to
overcome many of the limitations of SDG models, which was later further improved
by Huang and Wang (1999).
Fuzzy graph is a natural generalisation of the crisp graph using fuzzy set concepts.
A crisp graph is defined by the pair G = (X, E) where X is a finite set of nodes and
E a non fuzzy relation on X x X. A fuzzy graph [179] is a pair (X, E), where X is a
fuzzy set on X and E is a fuzzy relation on X x X such that IlE s: min ( IlX (x), Il
X (x'». Here IlE is the membership function of the binary effect of two adjacent
nodes x and x' over a branch, IlX ' the membership function of the node. However,
in some situations it may be desirable to relax this inequality [178]. Algorithms
about fuzzy graphs can be found in [178]. Obviously, if IlE and IlX only take the
values -1, 0 or 1, then a fuzzy graph becomes crisp. A fuzzy-SDG is defined by
nodes defining variables, branches representing the effects between two variables
and reasoning propagation algorithms associated with the graph.

5.7.1 Nodes

Each node in the fuzzy-SDG is represented by a variable which can take a number
of values from the fuzzy value space. An example of value space of a node is shown
in Figure 5.7, in which the process variable takes three fuzzy values, high, nonnal
and low. Another example is shown in Figure 5.10, in which L is the liquid level in
a tank changing from 0 to 2 meters, v is the nonnalised value of L in the range of
-1.0 to 1.0, and the fuzzy membership value changes from 0 to 1. Each fuzzy value
in Figure 5.10, such as medium low or low, is a fuzzy set Mdefined by Equation
5.13.
M = {x. Il}. Il = [0,1] (5.13)
Mis therefore represented by its membership function, /J., such that the value of
/J. illustrates the degree of membership of the element x belonging to M. The
membership function can have many shapes, such as triangular and trapezoidal. The
fuzzy value medium low in Figure 5.10 is a half-declined trapezoidal fonn which
can be represented by Equation 5.14.
 104 Data Mining and Knowledge Discovery for Process Monitoring and Control

0.6'5 Y < al
a2 -y
m= { al:S; y :s; a2 (5.14)
a2 - al
o v > a or y < 0.6

Medium Medium
0.. Low Low Nonnal High High
:..c
'"....d.l
~
E
E
d.l '"
I:
>,.3
N ....
~ g °1---I-L....iiLI.!......4..l.1i1-=-_-L...L...)_-L..4_+_.
""' c.2 I
I 1.0
V
I
, , . I L
0.0 0.4 0.7 1.0 1.30 1.6 2.0 •
meter

Figure 5.10 The fuzzy value space of a variable.

5.7.2 Branches

Attached to each branch connecting two nodes is an arrow representing the effect
direction and an effect strength. The effect strength is measured by a weight.
Suppose that Xj+ 1 and Xj are two nodes linked by a branch directed from Xj+ 1 to Xj,
then the effect strength of Xj+ 1 on Xj is determined by Equation 5.15,
) S Rx j + 1 (5.15)
e(xj+l~Xj = j,j+l - - -
RXj

in which Rxj+ 1 and Rxj are the value range (i.e., maximum -minimum) of nodes
Xj+ 1 and Xj respectively and Sj, j+ 1 is the sensitivity of Xj to Xj+ 1, determined by
Equation 5.16.
_ OXj
Sj,j+l - - - (5.16)
OXj + 1

The value range for a node consists of positive and negative ranges, corresponding
to fuzzy values, v in the range [0, 1] and [-1, 0] respectively. Obviously, the larger
the value of e(xj+ 1~ Xj), the stronger the effect of Xj+ 1 on Xj. If the relationship
uses a time derivative to account for the dynamics, this can be approximated using a
backward difference. The sensitivity will be derived from the partial derivative
estimated using the partial with respect to the rate of change of the quantity as
shown by Equation 5.17.
 Chapter 5 Supervised Learningfor Operational Support 105

S. '+1 = a(dx j / dt )
J,j a Xj + I
(5.17)

There are three basic connections in a fuzzy-SDG, i.e., serial (Figure 5.11(a»,
divergent (Figure 5. 11 (b» and convergent (Figure 5. 11 (c» connections.
Combination of the three can form any complicated networks such as Figure
5.11(d). Wang et al. [59] described the reasoning strategies for the basic
connections.

(a) Serial connection

(b) Divergent connection

(c) Convergent connection (d) A causal fuzzy network is a

combination of figures (a), (b) and (c)

Figure 5.11 Basic connections in a fuzzy - SDG.

Fuzzy -SDG proved to be able to overcome many of the limitations of SDG

mentioned earlier. More importantly, fuzzy-SDG has the learning capability from
data, therefore can be used as a tool for data mining and knowledge discovery.

5.7.3 The extended fussy-SDG

In many situations, using a single weight (defined by Equation 5.15) is not sufficient
because it implies a linear relationship between two nodes. The extended fuzzy-
SDG developed by Huang and Wang [175] developed a more sophisticated method
describing the connection relationship.
Two adjacent layers of a fuzzy-SDG is shown in Figure 5.12(a), which depicts the
cause - effect relationships between variables [XI. X2 , X3] and [ZI. Zz, Z3],
indicating that ZI is dependent on XI and X3 but independent on X2. Figure 5. 12(b)
shows the detail for training this substructure. The substructure involving XI. X3 and
lO6 Data Mining and Knowledge Discovery for Process Monitoring and Control

ZI can be trained independently. The node XI is converted into two types of nodes:
XI[L, M, H] and Jl. The first type of nodes takes only discrete values such as H
(high), M (medium) and L (low). The second type of nodes takes continuous values
Jl between 0 and 1 representing the fuzzy membership values when the first node
takes the value of H, M or L. The outside of the dashed box of Figure 5.12(b)
represents fuzzy processing of the original data. The arrangement is different from
that of the fuzzy neural network previously studied [151], that requires three nodes
to represent a variable if the variable takes three fuzzy values. However, the present
method always uses two nodes, therefore, the size of the network does not increase
with increased values in the fuzzy space. Inside the box of Figure 5 .12(b) is a single
layer percetron with no hidden layers. But it also allows to have one hidden layer.

G ~0

~
@ .- - - - - - - - - -•
"-+0
(a)

·
~
Xll;r
• •
•
Xl • •
I •I ZI!LM
~ m
• •• Zl

• •
....•

(b)

Figure 5.12 Learning of a convergent fuzzy-SDG.

Figure 5.13 compares a fuzzy - SDG with a fuzzy neural network. Suppose all the
variables in Figure 5.13, Xl to X3, Zl to Z 11 and Yl to Y3 represent variables in a
process. The non-linearity between the input variables [Xl, X2, X3] and the output
variables [Y 1, Y2, Y3] are expected to be high due to their distances. A neural
network (or a fuzzy neural network) with only one hidden layer as shown in Figure
5. 13 (b) can usually be able to deal with the high non-linearity between [Xl, X2, X3]
and [Y1, Y2, Y3].
 Chapter 5 Supervised Learning for Operational Support 107

If we have the knowledge of the cause - effect relationships between [Xl, X2,
X3] and [Yl, Y2, Y3] via a number of intermediate variables, e.g., Zl-Zll, we can
make use of the knowledge to develop a cause-effect diagram like Figure 5.13(a).
The procedure for training the convergent connection in a fuzzy-SDG has been
discussed above (refer to Figure 5.12) using the first layer of Figure 5.13(a). A
single layer percetron (Figure 5.l2(b» can normally give good performance for a
substructure in a fuzzy-SDG. It is because the non-linearity between any directly
connected layers is normally not high. If we view the fuzzy-SDG network in the
horizontal direction, it is a different way of linear summation of a number of small
non-linear (e.g., sigmoid) functions.
There is no doubt that the relationship between two connected nodes becomes
more complex compared with the original fuzzy-SDG because the weight is
replaced by a complicated relationship. However, the relative magnitudes of weights
are not significant, because they depend on the determination of the maximum and
minimum boundary values of variables in normalisation. In fact during reasoning we
are mainly concerned with the values of individual nodes and the propagation of
reasoning in the whole network, not the weight of a branch. Similar observations
can be found in the Bayesian networks in which the branches only mean a link
between two nodes. The reasoning in a Bayesian network is based on the
conditional probability calculation, which requires a complex conditional
probability table.
/Z4 _ _
:SZ:VI
:~~Zlj{~:;;~: ~~:~ ~Y2
X3~Z2~Z9 V3

Z3

(a) a fuzzy-SDG diagram

(b) a neural network

Figure 5.13 Comparison of a fuzzy-SDG and a neural network.

108 Data Mining and Knowledge Discovery for Process Monitoring and Control

5.8 Case Studies

In this section we describe two case studies. The first case study compares fuzzy
neural network, fuzzy single layer percetron and fuzzy set covering approaches to
fault diagnosis of a FCC process. It will demonstrate that three layered fuzzy neural
networks normally give more accurate results than single layer percetron and fuzzy
set covering approaches. However all three can identify significant disturbances or
faults and the difference is at identifying small changes. In terms of qualitative
interpretation of connection weights, FSC and PCT are advantageous compared
with FFNN. In the second case study, it shows that for a problem whose nonlinearity
is not high, a fuzzy - sna gives equally good result as FFNN and has much simpler
structure and clearer causal - effect picture.

5.S.1 Application to Fault Diagnosis of a FCC Process

The refmery residual fluid catalytic cracking process (R-FCC) described in the
appendix B is used as a case study. The data used for this study is summarised in
Table B2 of Appendix B. To avoid confusion in what follows, we use data patterns
to refer to the 67 data patterns, and fault types to refer to the 13 types of faults.

5.8.1.1 Fuzzyjication of1nput- Output Variables

For a process variable that changes rapidly over a short time, violations of
prescribed high and low limits are indications of possible faults. They are described
by two variable pairs: variable_name (high, x) and variable_name (low, x), where x
is the membership value representing the degree of high and low measures.
Model equations impose constraints on the process variables which are derived
from the material and energy balances, equilibrium and rate equations. The
equations are written so that they are equal to zero when satisfied. Associated with
each model equation are tolerance limits which are the expected positive and
negative values of the residual for which the constraint equation is to be satisfied.
Here the residual is defined as the deviation of the constraint equations from zero.
They are also described by the two variable pair: constraint _name (positive, x) and
constraint _name(negative, x). In this case study, no model equations are
considered.
 Chapter 5 Supervised Learningfor Operational Support 109

Chapter 3 has introduced a number of approaches for pre-processing dynamic

trends, such as wavelets, principal component analysis and episode approaches.
Here we use a fuzzy approach. The fuzzy approach may not be superior over
methods introduced in Chapter 3 but can be used directly by fuzzy neural networks.
The approach is an adaptation of the method proposed by Chen and Jong [176],
with the introduction of fuzzy concepts and can be explained by reference to Figure
5.14. The dynamic response is divided into three stages (I, II and III in Figure 5.14).
Each stage corresponds to a particular feature of the system behaviour. Stage I is
associated with the order of the system, II the maximum rate of change of the system
when maximum control input is applied and III relates to the settling stage and is an
indication of the stability of the system.
For stage I, a variable pair: trend_section_l (inc, x) and trend_section_l (dec, x) is
used. The treatment of section II has similar measures: trend_section_2(inc, x) and
trend_section_2(dec, x). However, five values are used to interpret stage III,
namely:

trend_section_3{divergence, x)
trend_section_3(oscillation, x)
trend_section_3{stable, x)
trend_section_3 (mean_value_high, x)
trend_section_3(mean_value_low, x).

In the first three pairs x has a value 0 or 1. While in the last two measures which
represent the deviation of the mean value of this stage from that for normal
operation x has values from 0 to 1. This means that a dynamic trend can be
represented by nine inputs in a network for fault diagnosis, as shown in Figure 5.15
(the part in the box of dashed line) of a diagnostic network.
Figure 5.15 is a fuzzy neural network for diagnosing the FCC process. The output
layer represents faults. The first output node, fresh feed F (high, x), refers to fault of
high fresh feed flowrate. The x is in the range [0, 1] measuring the degree of high.
While the x in node 12, compressor (failure, x) only takes 0 or 1, with 1 meaning
failure and 0 normal.
 110 Data Mining and Knowledge Discovery for Process Monitoring and Control

Regeneration temperature °C
650
:. ············:1
.
...........! "1
.. .

640 .
~ .........................
III
.
II:

630

620
., ........................... ..

610

100 200 300

Time

Figure 5.14 Fuzzy qualitative interpretation of a dynamic trend.

The input layer nodes in Figure 5.15 refer to symptoms. Some are described by
dynamic trends represented by nine nodes, and some as process variables and are
described by two nodes, high and low. By independently changing the parameters of
the outputs and recording the responses of inputs, data corresponding to Table B2
was obtained.
Figure 5.15 shows the fuzzy neural network structure which has one hidden layer.
A fuzzy single layer percetron which has the same input - output structure but no
hidden layers, and a fuzzy set covering model is also used to the same problem in
order to make comparisons. It might be difficult to have a completely fair
comparison, considering the difference in structure and training details. The
comparison is based on the assumption that all three, are in their optimum structure
and training parameters.

5.8.1.2 Discussion of Results

The broad nature of results is summarised in Table 5.2, which shows that the
Fuzzy FFNN model is able to identify 65 of the 67 faults in the samples, compared
with 55 and 53 out of 67 for fuzzy PCT and FSC respectively. The result might be
anticipated since FFNN is the best technique for dealing with non-linear data
followed by PCT and FSC. However there are more interesting observations.
 , Regeneration T trend, section-l ( inc, x) ,
Fresh feed F ( high, x )
. Regeneration T trend, section-l ( dec, x) :
: Regeneration T trend, section-2 ( inc, x) : Fresh feed F ( low, x)
:, Regeneration T trend, section-2 ( dec, x) ,
,
Mixed feed preheated T (high, x)
: Regeneration T trend, section-3 ( con, x) :
: RegenerationT trend, section-3 ( div, x) Mixed feed preheated T ( low, x )
g
Regeneration T trend, section-3 ( osc, x) -§
. Regeneration T ( high, x ) ,CD Residue oil F ( high, x ) ~
v.
Regeneration T ( low, x ) Residue oil F ( low, x )
r?
Reaction T trend, section-l (inc, x)
External heat removal V
••• opening (lzigh, x)
l[
Reaction P trend, section-l (inc, x) External heat removal V
••• opening( low, x) ~
~
Regeneration P trend, section-l ( inc, x) External heat removal water ;:.
()q
pump ifailure, x)
••• '0""I:>
02 content in flue gas ( high, x ) Main air F ( high, x)
02 content in flue gas ( low, x )
~(\)

CO content in flue gas ( high, x ) I n Main air F ( low, x) i:l

~.
CO content in flue gas ( low, x ) Compressor (failure, x) I:l
"-
Catalyst holdup in reactor (high, x) r?
Fractionator top valve opening ( low, x) ~o
Catalyst holdup in reactor (low, x)
~

Figure 5.15 The DIZZy neural network structure for fault diagnosis of FCC. .....
.....
 112 Data Mining and Knowledge Discovery for Process Monitoring and Control

Close inspection of the fault data patterns in Table 5.2 shows that the faults which
are not identified by any of the procedures are for smaller disturbances. For
instance, data pattern 24 represents an increase of 10% in the preheated temperature
of the mixed feed. This is a relatively small change compared with other data
patterns, and the system controllers are able to bring the system back to designed
operating condition fairly quickly. In fact, all the significant disturbances and faults
can be recognised by three models. It is apparent that faults not identified by FFNN
remain unidentified by PCT and FSC as well. In the same way, those not identified
by PCT are missed by FSC. So the order for describing effectiveness in accounting
for small disturbances is FFNN, PCT and FSC.

Table 5.2 Comparison 0 fthre e approach

es·mat
f1 ul d·la~OSlS.
Number of patterns Data patterns that are not identified
identified
Fuzzy 65 out of67 24,51
FFNN
PCT 55 out of67 24,51, 10, 11,23,24,27,28,29,39,44,45
FSC 51 out of 67 24,51, 10, 11,23,24,27,28,29,39,44,45,
33,34

Fe Fe
TCi TCi
FHi Ho FHi Ho
THi TH i
(a) FSC (b}PCT

FC
TC o
TCi
FHi
THi THo

(e) B P

Figure 5.16 The effect weights of three approaches for a heat exchanger and the
cause-effect explanation.
 Chapter 5 Supervised Learning for Operational Support 113

Table 5.3 Comparison of three methods in predicting faultS·,b,e,

1 2 3 4 5 6 7 8 9 10 11 12 13
No 7 0.7778 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
BP 0.7258 0.0093 0.0004 0.0006 0.0000 0.0174 0.0120 0.0000 0.0000 0.0000 0.0155 0.0000 0.0003
PCT 0.5567 0.0409 0.0651 0.0359 0.0322 0.0474 0.0338 0.0196 0.0004 0.0198 0.0069 0.0013 0.0117
FSC 0.5613 0.0141 -0.0111 -0.0097 -0.02790.0083 0.0146 0.0162 -0.0352 -0.0373 0.0157 -0.0307 0.0197

No 32 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
BP 0.0006 0.0024 0.0000 0.0001 0.0001 0.0000 0.8491 0.0001 0.0000 0.0000 0.0035 0.0000 0.0000
PCT 0.1224 0.0476 0.0200 0.0295 0.0305 0.0293 0.7479 0.0085 0.0002 0.0264 0.0061 0.0009 0.0067
FSC 0.21610.0393 -0.0762 -0.0184 0.0309 0.0079 0.6593 -0.0189 -0.0003 0.0171 0.0049 -0.0028 0.0132

No 38 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000
BP 0.0000 0.0022 0.0000 0.0000 0.0000 0.0000 0.00000.0327 0.9871 0.0008 0.0000 0.0000 0.0106
PCT 0.0322 0.0229 0.0199 0.0167 0.0134 0.0161 0.0017 0.3529 0.9018 0.0648 0.0030 0.0675 0.0156
FSC -0.04290.0317 -0.0662 0.0316 0.0007 -0.0205 0.0020 0.2796 0.7681 0.0282 -0.0864 -0.1079 0.0460

No 43 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000
BP 0.0000 0.0128 0.0005 0.0000 0.0000 0.0001 0.0000 0.0000 0.0002 0.9557 0.0006 0.0004 0.0035
PCT 0.00220.14580.02700.00640.00240.0171 0.01120.00300.00780.81700.0041 0.00750.0885
FSC -0.195 0.0313 0.0427 -0.0124 -0.0432 -0.0211 0.0401 -0.1144 0.0695 0.7781 0.0196 0.0316 0.0190

No 60 0.7222 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8454 0.0000 0.0000 0.0000 0.0000 0.0000
BP 0.7347 0.0000 0.0000 0.0002 0.0019 0.0000 0.0013 0.8203 0.0019 0.0000 0.0000 0.0007 0.0000
PCT 0.6892 0.0050 0.Q305 0.0536 0.1205 0.0292 0.0208 0.7579 0.0669 0.0013 0.0058 0.0246 0.0027
FSC 0.5345 0.0153 -0.0145 -0.0559 0.0241 -0.1010 0.1218 0.7848 0.1320 -0.2026 0.0058 0.1114 -0.0328

No 61 0.7778 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000
BP 0.7763 0.0006 0.0000 0.0000 0.0000 0.0000 0.0000 0.0075 0.9908 0.0000 0.0000 0.0033 0.0000
PCT 0.6969 0.0039 0.0082 0.0263 0.0456 0.0181 0.0025 0.1887 0.9362 0.0050 0.0042 0.0586 0.0045
FSC 0.7340 -0.0615 -0.0813 0.0339 0.1096 -0377 -00388 -0.0442 1.1698 0.0665 0.0399 00896 -00729

No 65 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 1.0000 0.0000
BP 0.0000 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0087 0.98\2 0.0015 0.0000 0.9784 0.0037
PCT 0.0022 0.0104 0.0049 0.0097 0.0119 0.0098 0.0010 0.1603 0.8441 0.0433 0.0018 0.8598 0.0679
FSC -.0223 0.0473 -0.0160 -0.0128 -0.0527 0.0374 -0.0055 0.2691 0.6280 0.0932 -0.0255 0.7562 0.0926

No 67 0.0000 0.8333 0.7500 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
BP 0.0001 0.8200 0.7447 0.0000 00000 0.0012 0.0000 0.0000 0.0002 0.0020 0.0021 0.0000 0.01 17
PCT 0.0300 0.7592 0.7144 0.0040 0.0013 0.0077 0.0146 0.0015 0.0100 0.0630 0.0251 0.0003 0.0787
FSC 0.0390 0.7977 0.7038 0.0143 0.0211 -0.0044 -0.0135 0.0201 -0.01760.1017 -0.030 -0.0273 0.0188

a_I to 13 are fault types corresponding the 13 output nodes of Figure 5.15
b _ No 7 is the 7th data pattern

c _ 1.0 means fault; 0 means no fault.

114 Data Mining and Knowledge Discovery for Process Monitoring and Control

Despite the three having significantly different capabilities when monitoring small
disturbances, they are almost equal in isolating major disturbances and faults. Table
5.3 gives the comparison of three methods for a number of fault situations. All three
can identify the faults, even in double fault situations, though in most cases FFNN
predictions are slightly closer to 1.0 than the other two (1 means fault). However,
PCT and FSC are much faster in convergence. A typical example is that FFNN
requires several hours, PCT one hour and FSC 30 minutes on a personal computer.
People are also interested in what FFNN, PCT and FSC can help us to develop a
better understanding of the cause-effect behaviour of the process. It is easier to
illustrate this in a simple counter current flow heat exchanger shown in Figure
5.l6(a). Our interest is to study how the cold stream flowrate (FC) and its inlet
temperature (TC;) and the hot stream flowrate (FH) and its inlet temperature (TH;)
affect the outlet temperature TC o and THo. The weights obtained for the same set of
data for FFNN, PCT and FSC are shown in Figure 5.l6(b). For both PCT and FSC,
it is very clear to see from the weights how an input affects an output. However it is
not so clear from the weights of a FFNN due to the existence of a hidden layer.

5.8.2 Application to a Waste Water Treatment Plant

The extended fuzzy-SDG was applied to a waste water treatment plant and
compared with feedforward neural networks by Huang and Wang [175]. The plant
has three treatment units in series. Through sensitivity study, a fuzzy-SDG is
developed as shown in Figure 5.17. The variables are illustrated in Table 5.4. The
advantage of this fuzzy-SDG network is that compared with neural networks, it is
more intuitive to engineers and supervisors. Neural networks are fully connected
and can give predictions given the inputs. However, it is not straightforward to
qualitatively know the weighted contribution of inputs to the outputs. In contrast,
the causal network is no longer a blackbox because it is a partially connected graph.
Engineers can trace forward and backward the network to analyse problems. For
example, the output suspended solids, SS-S is observed as (High, 0.10), tracing
back the causal graph and the nodes, SS-D (High, 0.68), SS-P(High, 0.28), RD-
SSP(Medium, 1.00), SS-E (High, 0.88) and SSV-E(M, 0.87). So the main reason
causing SS-S(High, 0.10) is due to SS-E(High, 0.88).
 Chapter 5 Supervised Learningfor Operational Support 115

Table 5.4 The attributes of the database a.

I Q-E (input flow to plant) 20 SSV-D (input VSSto secondary settler)
2 ZN-E (input Zinc to plant) 21 SED-D (input sediments to secondary settler)
3 PH-E (input pH to plant) 22 COND-D (input conductivity to secondary settler)
4 BOD-E (input BOD to plant) 23 PH-S (output pH)
5 COD-E (input COD to plant) 24 BOD-S (output BOD)
6 SS-E (input SS to plant) 25 COD-S (output COD)
7 SSV-E (input VSS to plant) 26 SS-S (output suspended solids)
8 SED-E (input sediments to plant) 27 SSV-S (output VSS)
9 COND-E (input conductivity to plant) 28 SED-S (output sediments)
10 PH -P (input pH to primary settler) 29 COND-S (output conductivity)
II BOD-P (input BOD to primary settler) 30 RD-BOD-P (perfonnance input BOD in primary settler)
12 SS-P (input SS to primary settler) 31 RD-SS-P (performance input SS to primary settler)
13 SSV-P (input VSS to primary settler) 32 RD-SED-P (performance input sediments to primary settler)
14 SED-P (input sediments to primary settler) 33 RD-BOD-S (performance input BOD to 2nd settler)
15 COND-P (input conductivity to primary settler) 34 RD-COD-S (performance input COD to 2nd settler)
16 PH-D (input pH to secondary settler) 35 RD-BOD-G (global performance input BOD)
17 BOD-D (input BOD to secondary settler) 36 RD-COD-G (global performance input COD)
18 COD-D (input COD to secondary settler) 37 RD-SS-G (global performance input SS)
19 SS-D (i~ut SS to secondary settler) 38 RD-SED-G (global performance input sediments)
a BOD _ biological oxygen demand, COD - chemical oxygen demand, SS - suspended solids, VSS -
volatile suspended solids

~
~

Figure 5.17 The fuzzy-SDG for the wastewater treatment plant.

M - medium; L - low; H - high.
 116 Data Mining and Knowledge Discovery for Process Monitoring and Control

0.9 • - •••• Measurement

0.8 --Fuzzy-SDG-O
--Fuzzy-SDG-2
0.7 ,.. c-_'c"'_"'" BP Prediction
0.6

0.5

0.4

0.3

0.2

0.1

10 20 30 40 50 60 70 80 90

-o~
/0--
PH-E

-0
PH-P

SSE

Fuzzy-SDG-O Fuzzy-SDG-2

Figure 5.18 Comparisons offuzzy-SDG without hidden neurons (Fuzzy-SDG-

0), fuzzy-SDG with two hidden neurons (Fuzzy-SDG-2) and backpropagation
neural networks (BP).

In terms of accuracy of the fuzzy - SDG, a comparison is made as shown in Figure

5.18. It compares the prediction ofPH-P (the second node on the second layer from
top of the network shown in Figure 5.17) using a fuzzy-SDG with no hidden
neurons (Fuzzy-SDG-O), with two hidden neurons (Fuzzy-SDG-2) and a fully
connected neural network with ten hidden neurons (BP Prediction). The fully
connected neural network has to include all the inputs in the first layer of Figure
5.17. As far as prediction accuracy is concerned, for all data patterns used for
training, three models give equivalent predictions as demonstrated in Figure 5.18.
The average relative errors for the whole data cases using Fuzzy-SDG-O, Fuzzy-
SDG-2 and BP are 11.6%, 12.3% and 11.6% respectively.

5.9 General Observations

Several supervised machine learning approaches are introduced using illustrative

case studies. These include neural and fuzzy neural networks, fuzzy set covering
and fuzzy-SDG. The focus of discussion has been on issues that need to be
 Chapter 5 Supervised Learning for Operational Support 117

addressed in practical applications. These include variable selection and feature

extraction for FFNN inputs, model validation and confidence bounds, accuracy and
generality, feature extraction from dynamic transient signals as well as the ability to
give causal knowledge. There are some other issues which have not or have not
fully addressed in the context.
The backpropagation learning is not a recursive approach, which means that when
new data is to be used to update the knowledge, it has to be mixed with previous
data and the model to be retrained again. Recursive models are particularly useful
for on-line systems because they can continuously update their knowledge.
When applied to fault diagnosis, supervised learning approaches described above
are suitable for mapping the set of symptoms to the set of so-called essential faults
[151]. Faults can be divided into two categories: essential faults and root faults. For
example, the symptom - "the trend of liquid level increases dramatically at the
bottom of the column" can be explained by the following essential faults: (1) high
feedflowrate; (2) low bottom withdrawal flowrate; (3) low flash zone temperature;
(4) low flash zone pressure; (5) change in feed properties. These essential faults are
indications of root faults. Looked at in this way, low bottom withdrawal flow is an
essential fault which can be the result of the root faults of a pump failure or fail to
open a valve. Therefore fault diagnosis is a two step procedure, mapping from the
set of symptoms to the set of essential faults, and explanation of the essential faults.
The unsupervised approaches introduced are suitable for the first step. For the
second step, it is more appropriate to use expert systems or graphical models such
as signed digraph. Similar strategies have been developed by Becraft and Lee [177]
and Wang et al. [151].
Since supervised learning approaches require training data which is difficult to get
for the purpose of fault identification and diagnosis, they are less attractive
compared with multivariate analysis approaches (Chapter 3), and unsupervised
approaches (Chapter 6). However, the good performance in correlating non-linear
inputs-outputs makes them very attractive in developing software sensors or
software analysers (Chapter 9).
The extrapolation problem of feedforward neural networks has been addressed
through input dimension reduction, proper selection of training and test data, model
validation and confidence bound evaluation as well as using single layer percetron,
FSC or fuzzy-SDG if feasible. There are also efforts in incorporating partial domain
knowledge into neural network learning.
CHAPTER 6
UNSUPERVISED LEARNING FOR OPERATIONAL STATE
IDENTIFICATION

6.1 Supervised vs. Unsupervised Learning

This chapter describes some representative unsupervised machine learning

approaches for process operational state identification. Whether a machine learning
approach is regarded as supervised or unsupervised depends on the way it makes
use of prior knowledge of the data. Data encountered can be broadly divided into
the following four categories:
(1) Part of the database is known, i.e., the number and descriptions of classes as
well as the assignments of individual data patterns are known. The task is to assign
unknown data patterns to the established classes.
(2) Both the number and descriptions of classes are known, but the assignment of
individual data patterns is not known. The task is then to assign all data patterns to
the known classes.
(3) The number of classes are known but the descriptions and the assignment of
individual data patterns are not known. The problem is to develop a description for
each class and assign all data patterns to them.
e
4) Both the number and descriptions of classes are not known and it is necessary
to determine the number and descriptions of classes as well as the assignments of
the data patterns.
The methods introduced in Chapter 5 are powerful tools for dealing with the first
type of data where the objective is to assign new data patterns to previously
established classes. Clearly this is not appropriate for the last three types of data,

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
120 Data Mining and Knowledge Discovery for Process Monitoring and Control

since training data is not available. In these cases unsupervised learning approaches
are needed and the goal is to group data into clusters in a way such that intraclass
similarity is high and interclass similarity is low. In other words, supervised
approaches can learn from known to predict unknown while unsupervised
approaches learn from unknown in order to predict unknown. Supervised learning
can generally give more accurate predictions, but can not be extrapolated: when new
data is not in the range of training data, predictions will not generally be reliable.
For process operational state identification and diagnosis, supervised learning needs
both symptoms and faults. Therefore the routine data collected by computer control
systems can not be used directly for training. Faults are unlikely to be deliberately
introduced to an industrial process in order to generate training data.
Grouping of data patterns using unsupervised learning is often based on a
similarity or distance measure, which is then compared with a threshold value. The
degree of autonomy will depend on whether the threshold value is given by the users
or determined automatically by the system. In this chapter three approaches are
studied, the adaptive resonance theory (ART2), a modified version of it named
ARTnet, and Bayesian automatic classification (AutoClass). ART2 and ARTnet
though require a pre-defined threshold value are able to deal with both the third and
fourth types of data. AutoClass is a completely automatic clustering approach
without the need to pre-define a threshold value and the number and descriptions of
classes, so is able to deal with the fourth type of data.

6.2 Adaptive Resonance Theory

The adaptive resonance theory (ART) was developed by Grossberg [47, 48], as a
clustering-based, autonomous learning model. ART has been instantiated in a series
of separate neural network models referred to as ART!, ART2 and ART3 [43,49,
50,51]. ART! is designed for clustering binary vectors and ART2 for continuous-
valued vectors.
A general architecture of the ART neural network is shown in Figure 6.1. The
network consists of three groups of neurons, the input processing layer (F 1), the
cluster layer (F2 ), and a mechanism which determines the degree of similarity of
patterns placed in the same cluster (a reset mechanism). The input layer can be
considered as consisting of two parts: the input and an interface. Some processing
may occur in the both (especially in ART2). The interface combines signals from
the input to the weight vector for the cluster unit which has been selected as a
 Chapter 6 Unsupervised Learningfor Operational State Identification 121

candidate for learning. The input and interface are designated as F, (a) and F, (b) in
Figure 6.1.

(Cluster layer)

Fl (b) Layer
(Interface)

Figure 6.1 The general architecture of ART.

An input data pattern is presented to the network, following some specialised

processing by the network, and is compared to each of the existing cluster
prototypes in the cluster layer (F2)' The "winner" in the cluster layer is the prototype
most similar to the input. Whether or not a cluster unit is allowed to learn an input
data pattern depends on how similar its top-down weight vector is to the input
vector. The decision is made by the reset unit, based on signals it receives from the
input (a) and the interface (b) of the F, layer. If the similarity between the "winner"
and the input exceeds a predetermined value (the vigilance parameter p, in ART
terms), learning is enabled and the "winner" is modified slightly to more closely
reflect the input data pattern. If the similarity between the "winner" and the input is
less than required by the vigilance parameters, a new cluster unit is initialised in the
top layer and learning is enabled to create a prototype similar to the input. Thus
both the number and descriptions of classes are continuously updated during
learning. This is different from the Kohonen network, where the descriptions of
classes (called neurons in a Kohonen network) are also continuously update during
learning, but the number of classes needs to be determined before learning starts.
To control the similarity of patterns placed in the same cluster, there are two sets
of connections (each with its own weights) between each unit in the interface part of
 122 Data Mining and Knowledge Discovery for Process Monitoring and Control

the input layer and the cluster units. The FI (b) layer is connected to the F2 layer by
bottom-up weights; the bottom-up weight on the connection from the ith FI unit to
the jth F2 unit is designated b ij . The F2 layer is connected to the FI (b) layer by top-
down weights; the top-down weight on the connection from the jth F2 unit to the ith
Fl unit is designated tji .

f(x)
Wi l - - - - - - - - + - i

Figure 6.2 Typical ART2 architecture.

ART2 is designed to process continuous-valued data patterns. A typical ART2

architecture [43] is illustrated in Figure 6.2. It has a very sophisticated input data
processing layer (FI ) consisting of six types of units (the W, X U, v, P and Q
units). There are n units of each of these types (where n is the dimension of an input
pattern). Only one unit of each type is shown in Figure 6.2. A supplemental unit
between the Wand X units receives signals from all of the W units, computes the
norm of the vector w, and sends this (inhibitory) signal to each of the X units. Each
of these also receives an excitation signal from the corresponding W unit. Details of
this part of the net are shown in Figure 6.3. A similar supplemental unit performs
the same role between the P and Q units, and another does so between the V and U
units. Each X unit is connected to the corresponding V unit, and each Q unit is
connected to the corresponding V unit also.
 Chapter 6 Unsupervised Learning for Operational State Identification 123

Figure 6.3 Details of connections from W to X units, showing the supplemental

unit N to perform normalisation.

The symbols on the connection paths between the various units in the FJ layer in
Figure 6.2 indicate the transformation that occurs to the signals as it passes from one
type of unit to the next; they do not indicate multiplication by the given quantity.
However, the connections between units Pi (of the Fi layer) and Y; (of the F2 layer)
do show the weights that multiply the signal transmitted over those paths. The
activation of the winning F2 unit is d, where O<d<l. The symbol ---> indicates
normalisation; i.e., the vector q of activations of Q units is just the vector p of
activations of the P units, normalised to be approximately unit length.
The U units perform the role of an input phase of the F J layer. The P units play
the role of the interface to the FJ layer. Units Xi and Qi apply an activation
function to their net input; this function suppresses any components of the vectors of
activations at those levels that fall below the user-selected value 9. The connection
paths from W to U and from Q to V have fixed weights a and b, respectively.
The most complete description of the ART2 network can be found in [43].
Although written for ARTl, the discussions by Caudil [181] and Wasserman [182]
provide a useful but less technical description of many of the principles employed in
ART2.
ART2 has shown great potential for analysis of process operational data and
identification of operational state due to a number of properties [183, 184, 195].
First, it is considered as an unsupervised machine learning system that does not
require training data. It is well known that it is difficult to find training data for the
purpose of process fault identification and diagnosis. Second, the approach is
recursive, or in the terms used in ART2, it is plastic, that is, it is able to acquire new
knowledge and retain stable in the sense that existing knowledge is not corrupted.
 124 Data Mining and Knowledge Discovery for Process Monitoring and Control

This property is apparently very useful for on-line monitoring where information is
received continuously.

6.3 A Framework for Integrating Wavelet Feature

Extraction and ART2

Wang et al. [185] and Chen et al. [82] developed an integrated framework named
ARTnet which combines wavelet for feature extraction from dynamic transient
signals and adaptive resonance theory. In ARTnet the data pre-processing part uses
wavelets for preprocessing the data for feature extraction. In order to introduce
ARTnet it is helpful to first examine the mechanism of ART2 for noise removal.
ART2 has a data pre-processing unit which is very complicated but the mechanism
for removing noise uses a simple activation function A(x),
X x > e
A(x) = { 0 (6.1)
x < e

where e is a threshold value. If an input signal is less than e, it is considered to be

a noise component and set to zero. This has proved to be inappropriate for removing
noise components contained in process dynamic transient signals which are often of
high frequencies and in certain magnitude.

6.3.1 The Integrated Framework ART net

A mechanism was proposed by Pao [186] to replace the data pre-processing part of
ART2 with more efficient noise removal and dimension reduction methods. This
has been followed in this study by using wavelet to replace the data pre-processing
unit of ART2. The integral framework is called ARTnet to distinguish it from
ART2. The conceptual architecture of ART net is shown in Figure 6.4.
In this new architecture, wavelets are used to pre-process the dynamic trend
signals. The extracted features are used as inputs to the kernel of ARTnet for
clustering. A pattern feature vector (XI' x 2 ,.·· ,x N ) is fed to the input layer of the
ARTnet kernel and weighted by bij' bottom-up weights. In Chapter 3, the extrema of
wavelet multiscale analysis should be regarded as the features of dynamic transient
signals.
 Chapter 6 Unsupervised Learningfor Operational State Identification 125

The weighted input vector is then compared with existing clusters in the top layer
by calculating the distance between the input and existing clusters. The existing
cluster prototype, which has a smaller distance than the input is called the winner.
By considering this input the description or knowledge of the wining cluster is
updated. Whether or not a winning cluster prototype is allowed to learn from an
input data pattern depends on how similar the input is to the cluster. If the similarity
measure exceeds a predetermined value, called the vigilance parameter, learning is
enabled. If the similarity measure is less than the required vigilance parameter, a
new cluster unit is then created which reflects the input. Clearly this is unsupervised
and recursive learning process.

Updating the

f-G ... . .......

C C
~ ~
ITTdescription
~ f1'l../ofac1uster prototype
~____+.-==~ Top layer

·{~j?<l
•
t •t •t Input to
ARTnet
Features Kernel
~
( Wavelet Feature extraction)
~
Dynamic trend signals

Figure 6.4 The conceptual architecture of ARTnet.

6.3.2 The Similarity Measure in ART net

It is apparent that the learning process is concerned with the extent to which how
similar two vectors are. There are several ways to measure the distance between two
pairs of observations, such as the Hamming or Euclidean distance. For continuous
data, the Euclidean distance is the most commonly used [187]. Formally, the
Euclidean distance between two vectors x and y is defined as the root sum-squared
error,
126 Data Mining and Knowledge Discovery for Process Monitoring and Control

(6.2)

Suppose there are K existing cluster prototypes. The kth cluster prototype consists
of a number of data patterns and is also described by a vector, denoted as Z(k) ,
which has considered all data patterns belonging to it. Clearly, if there is only one
data pattern in the cluster, Z(k) , it is equal to that data pattern. When a new input
data pattern x is received, a distance between x and z(k) is calculated according to
the expression,

(6.3)

Since the distance between x and all existing cluster prototypes is calculated, the
cluster prototype with the smallest distance is the winner. If the distance measure for
the winner is smaller than a pre-set distance threshold, p, then the input x is assigned
to the winning cluster and the description of the cluster is then updated,

1
Z(k) = Z(k) + --x.b .. i= 1...N, j = 1.. .K (6.4)
I I NF I Y

where Z,(k) refers to the ith attribute of the vector z for the cluster k. h'l is the weight
between the ith attribute of the input and the jth existing cluster prototype. NF is the
number of features.

6.4 Application of ARTnet to the FCC Process

The FCC process and the data used is described in appendix B. To demonstrate
the procedure, 64 data patterns are used. Discussion is limited to 64 data patterns in
order to keep the discussion manageable and to assist in presentation of the results.
The data sets include the following faults or disturbances:

• fresh flow rate is increased or decreased

• preheat temperature for the mixed feed increases or decreases
• recycle slurry flow rate increases or decreases
• opening of the hand valve V20 increases or decreases
• air flow rate increases or decreases
• the opening of the fully open valve 40 I-ST decreases
• cooling water pump fails
 Chapter 6 Unsupervised Learning/or Operational State Identification 127

• compressor fails
• double faults occur

The sixty four data patterns were obtained from a customised dynamic training
simulator, to which random noise was added using a zero-mean noise generator
(MATLAB®). In the following discussions, the term "data patterns" refers to these
sixty four data patterns and "identified patterns" to the patterns estimated by
ARTnet.
Figure 6.5(a) shows a reactor temperature transient when the fresh feed flowrate
increases by 70%. Figure 6.5(b) is the same transient with random noise. The
corresponding four scales from multiresolution analysis for this transient are shown
in Figure 6.6, together with the corresponding extrema on the right hand side of
Figure 6.6.

2
0
·2
·4
·6
·8
·10
·12
·14
·16
0 20 40 60 80 100
(a)

-2

-4

·6

-8

-10

-12

-14

-16

(b)

Figure 6.S A signal from the simulator (a) and the signal with random noise (b).
128 Data Mining and Knowledge Discovery for Process Monitoring and Control

5 5
4 4
3 3
2 2
1
0 • 1
0 ~A
-1 -1 Scale 1
-2 -2
-3 -3
-4 -4
-5 -5
0 20 40 60 80 100 0 5 10 15 20 25

5 5
4 4
3 3
2
l
2
1 1
A 0
0
-1 Scale 2
-1
-2 -2
-3 -3
-4 -4
-5 -5
0 20 40 60 80 100 0 5 10 15 20 25

20 20
15 15
10 10

'1 ~~
5 5
0 -y 0
Scale 3
-5 -5
-10 -10
-15 -15
-20 -20
0 20 40 60 80 100 0 5 10 15 20 25

20
20
15
15 10
10
5
5 0 Scale 4
0
-5 ~ -5
-10
-10 -15
-15 -20
-20 '0 20 40 60 80 00 0 5 10 15 20 25

Extrema representation Extrema representation

after noise removal (a) after piece-wise analysis (b)

Figure 6.6 Multiresolution analysis (left) and extrema (right),

 Chapter 6 Unsupervised Learningfor Operational State Identification 129

As stated in Chapter 3, the extrema that are mostly influenced by noise

fluctuations are those (1) where the amplitude decreases on average as the
decomposition scale increases and (2) do not propagate to large scales. Using these
criteria, noise extrema are removed.
The extrema representation after the noise extrema are removed is a sparse vector,
so a piece-wise technique is employed to reduce the dimensionality of the signal.
The extrema after removing noise and carrying out piece-wise processing are shown
in Figure 6.7.
Figure 6.8 shows the result after noise removal and compares it with the
multiresolution analysis of the original noise-free signal. The extrema are the same
in positions but are slightly different in value. The noise removal algorithm is
therefore suitable in this case and the 4th scale extrema are selected as input to the
ARTnet for pattern identification
It is important that a suitable threshold for pattern recognition is used when
applying ARTnet. For a threshold p = 0.8, all 64 data patterns are identified as
individual patterns. A more suitable threshold is obtained by analysing clustering
results for increased threshold values as shown in Table 6.1.

Table 6.1 ARTnet clustering result using different distance thresholda .

Threshold p Number of
patterns Identified grouping of data samples
identified
0.8 64
1.0 63 r56571
2.0 60 r5 71 r25 261 r27 281 r56 571
3.0 57 r5 71 r19 2023241 r25 261 m 281 r56 571
4.0 54 r5 6 7 81 r19 20 21 23 241 r25 26] r27 281 r56 571
4.5 49 [3456789] [192021222324] [25 26] [2728]
r35 361 r56 57]
5.0 48 r34 5 6 7 8 91 r 19 20 21 22 23 24 291
r25 261 r27 281 r35 361 r56 57]
6.0 47 r34 5 6 7 8 91 r 19 20 21 22 23 24 29 611
r25 261 r27 281 [35 361 r56 571
a - [56 57] means that data patterns 56 and 57 are identified in the same cluster.
 130 Data Mining and Knowledge Discovery for Process Monitoring and Control

With the threshold p increased to 1.0, data patterns 56 and 57, which represent
cases where the opening of valve 401-ST is decreased from 100% by 80% and 90%
are grouped together. When p is 2.0, further groupings are [5, 7], representing the
fresh feed flowrate increasing by 50% and 70%, [25, 26] recycle oil flowrate
increasing by 70% and 90%, and [27, 28] recycle oil flowrate decreasing by 70%
and 90%. It is obvious that these are all reasonable groupings.
When the threshold value is 4.5, the groupings are [3,4,5,6,7,8,9], [192021 22
2324], [2526], [27,28], [35,36] and [56 57]. The pairing of identified patterns and
original data patterns are shown in Table 6.2. The clustering is justified by
inspecting the results in detail. Figure 6.9 shows the trends of three measurements
for data pattern 5. It shows that regenerator temperature and concentration of
oxygen in regenerator flue gas drop sharply while catalyst hold-up in reactor
increases dramatically. All of which mean abnormal operations. Very similar
scenarios can be found for data patterns 3, 4, 5, 6, 7, 8, and 9, so the result of
regarding them as a single pattern is acceptable. The grouping [35, 36] can also be
justified by inspecting the dynamic responses (Figure 6.10). In both cases, the
dynamic responses of catalyst recycle rate lead to a steady state with the process
remaining under control.

Table 6.2 ATRnet identified clusters when the distance threshold is 4.5 and the
corresponding data patternsa .
Identified Corresponding Identified Corresponding Identified Corresponding
clusters data patterns clusters data natterns clusters data patterns
1 I 19 32 37 51
2 2 20 33 38 52
3 [34 5 6 7 8 91 21 34 39 53
4 10 22 [35361 40 54
5 11 23 37 41 55
6 12 24 38 42 [56571
7 13 25 39 43 58
8 14 26 40 44 59
9 15 27 41 45 60
10 16 28 42 46 61
11 17 29 43 47 62
12 18 30 44 48 63
13 [ 19 20 21 22 23 31 45 49 64
241
14 f25261 32 46
15 [27281 33 47
16 29 34 48
17 30 35 49
18 31 36 50
a - [3 4 5 6 7 8 9] means data patterns 3 to 9 are identified in the same cluster
 Chapter 6 Unsupervised Learning for Operational State Identification 131

4 4
3 3
2 2
1
0
, ~A......
-1 -1 Salle 1
-2 -2
-3 -3
-4 -4
-5 -5
0 20 40 00 00 100 0 5 10 15 20 25

,
5
4 4
3 3
2 2
~
1
0 A
-1 -1 Salle 2
-2 -2
-3 -3
-4 -4
-5 -5
A;, 100
20 40 00 00 0 5 10 15 20 25

20 5
15 4
3
10
2
~ 5
1, 1
0
-1
:~ Salle 3
-5
-2
-10
-3
-15 -4
-20 -5
20 40 00 00 100 0 10 15 20 25

20
15 4
3
10
5
1, 1
0
-1
~ , ~ Salle 4
-5
-2
-10
-3
-15 -4
-20 -5
20 40 00 00 100 10 15 20 25

Exrrerra rerre;<rtatirn Exrrerra rqre;enatirn

aft'nriseremMII (a) after pece-\\ise (b)

Figure 6.7 Extrema after removing noise (left) and piece wise analysis (right).
Di - detail of multiscale wavelet analysis; Ai - approximation.
132 Data Mining and Knowledge Discovery for Process Monitoring and Control

However, any further increase in threshold is not useful because some data
patterns that are significantly different are grouped in the same cluster. For instance,
when the threshold value is 5, data pattern 29 (opening ratio of the hand-valve V20
increasing by 5%) is merged with the clusters representing increase and decrease in
the preheat temperature of the mixed feed. Therefore, the threshold p = 4.5 is
considered as the most appropriate value for this case.

10

10

200~-~--'-:::0----:!15:----:::2o----!2·5

(a)

·10

~200L-~--,-:::0-~,5:-----:::20------:!25

(b)

Figure 6.8 Comparison of the result after noise removal (b) with the
multiresolution analysis of the original simulation signal (a).
 Chapter 6 Unsupervised Learning for Operational State Identification 133

o ,........,--,---....---,----,----, 0.05 I-:;::X::::::::=;;:--""--'---'

Regenemtor
Temperature

-10 -0.15

Oxygen in outlet of
Regenerator

_201---_ _-+-_---<_ _+-_-1 -0.35'--_-'-_--..._ _"--_-'-_--'

Time Time

300

200

100 Catalyst
holdup

Time

Figure 6.9 Variable dynamic trends of data pattern 5.

' .. ~-~----~------.

Catalyst rElC)de rate #36

.,

Tine Tim!

Figure 6.10 Dynamic trends of catalyst recycle rate for data patterns 35 and 36.
134 Data Mining and Knowledge Discovery for Process Monitoring and Control

6.4.1 Comparison between ARTnet and ART2

It is apparent that the data pre-processing part of ARTnet is able to effectively

reduce the dimension of the dynamic trend signals using wavelet feature extraction
and piece wise processing. ARTnet has also shown other advantages over ART2 in
operational data analysis. These include the determination of threshold values, the
ability to deal with noise and computational speed. In the comparison followed only
the fIrst fIfty seven data patterns were used.

6.4.1.1 Threshold Determination

In this case, only 57 data patterns are used to compare the distance threshold for
using ARTnet and the vigilance value in ART2 using noise-free data. For noise free
data, ARTnet and ART2 give the same results if the ARTnet distance threshold and
the ART2 vigilance are appropriately adjusted, as shown in Table 6.3. To
understand the table, consider the last row, which shows that when the distance
threshold of ARTnet is 4.5 it gives the same grouping result as ART2 with a
vigilance value of 0.9985. From Table 6.3, for the same groupings, the ARTnet
distance threshold changes from 0.8 to 4.5 while the vigilance of ART2 varies from
0.9998 down to 0.9985. So the distance threshold for ARTnet is less sensitive than
the vigilance of ART2. The ART2 clustering is too sensitive to the vigilance value,
making it difficult to set a value.

6.4.1.2 Robustness with Respect to Noise

The following demonstrates that ARTnet gives consistent clustering result

regardless of the magnitude of noise to signal ratio, providing it is in a reasonable
range. ART2 gives fewer clusters at a low noise to signal ratio and more clusters at
a larger ratio. 57 data patterns are considered with white noise added. A constant
Cnoise is introduced to control the magnitude of noise defIned by
The magnitude of noise from
the noise generator
The magnitude of noise = -----=------ (6.5)
c'lOi.\'e
 Chapter 6 Unsupervised Learningfor Operational State Identification 135

Table 6.3 Comparison of the value ranges of the distance threshold of ARTnet
and the vigilance value of ART2, for the same grouping schemes'abc, .
ARTnet ART2
distance vigilance Grouping of data samples
threshold value
0.8 0.9998
1.0 0.9996 [5657]
2.0 0.9992 [5 7] [2526] [2728] [5657]
3.0 0.9990 [57] [19202324] [25 26] [2728] [5657]
4.0 0.9987 [5 67 8] [192021 2324] [2526] [2728] [56 57]
4.5 0.9985 [3456789] [192021222324] [25 26] [2728]
[35 36] [5657]
a [56 57] means that data patterns 56 and 57 are grouped in the same cluster, b Only the
first 57 data patterns are considered and the data is noise free, cThe ARTnet distance
threshold changes in a wider range while ART2 vigilance is too sensitive making it
difficult to set a value.

Table 6.4 Clusters predicted by ARTnet when the distance threshold is 4.5 and
Cnoise varies over a Wi'de range, firom 0 .00 1 to 100 a
Identified Corresponding Identified Corresponding Identified Corresponding
patterns data patterns patterns data patterns patterns data patterns
1 I 15 [27,28] 29 43
2 2 16 29 30 44
3 [34 5 6 7 8 9] 17 30 31 45
4 10 18 31 32 46
5 II 19 32 33 47
6 12 20 33 34 48
7 13 21 34 35 49
8 14 22 [35 36] 36 50
9 15 23 37 37 51
10 16 24 38 38 52
11 17 25 39 39 53
12 18 26 40 40 54
13 [19202122 27 41 41 55
23241
14 [25 26] 28 42 42 [5657]

a[3 4 5 6 7 8 9] means that data patterns 3 to 9 are grouped in the same cluster.
 136 Data Mining and Knowledge Discovery for Process Monitoring and Control

In Equation 6.5, Cnoise changes ranging from 0.001 to 100 are examined in what
follows where the smaller the Cnoise, the larger the noise to signal ratio.
The best clustering results are obtained when the distance threshold of ARTnet is
4.5. This result is not affected by changing Cnoise from 0.001 to 100, as can be seen
in Table 6.4. For ART2, the best value of the vigilance is 0.9985 and Cnoise = 100,
and is the same result as ARTnet (Table 6.4). However, as Cnoise decreases to 10,
i.e., larger noise to signal ratio, ART2 splits the cluster [3 4 5 6 7 8 9] into two [3 4
5 6 7] and [8 9]. As Cnoise decreases to 0.001, i.e., a much larger noise to signal
ratio, there are further new groupings, [20 42] and [29 51]. The new groups are not
able to be satisfactorily explained. Although the inappropriate groupings [20 42]
and [29 51] can be avoided by changing the vigilance value, other unreasonable
groupings are generated.

6.3.1.3 Computational Speed

It is found that ARTnet is faster than ART2. After optimum values of the distance
threshold of ARTnet and the vigilance of ART2 are found, for the same data,
ARTnet is typically two times faster than ART2.

6.5 Bayesian Automatic Classification

6.5.1 The Bayesian Automatic Classification System - AutoClass

The approach used is based on an unsupervised Bayesian classification scheme

developed by NASA [39, 40, 41, 42]. For a given number of data patterns (some
times called cases, observations, samples, instances, objects or individuals), each of
which is described by a set of attributes, AutoClass can devise an automatic
procedure for grouping the data patterns into a number of classes such that instances
within a class are similar, in some respect, but distinct from those in other classes.
The approach has several advantages over other clustering methods.

• The number of classes is determined automatically. Deciding when to stop

forming classes is a fundamental problem in classification [188]. More classes
can often explain the data better, so it is necessary to limit the number of
classes. Many systems rely on an ad hoc stopping criterion. For example,
ART2 (Adaptive Resonance Theory) is strongly influenced by a vigilance or
 Chapter 6 Unsupervised Learningfor Operational State Identification 137

threshold value which is set by users based on trial and error. The Kohonen
network requires the number of classes to be detennined beforehand. The
Bayesian solution to the problem is based on the use of prior knowledge. It
assumes that simpler class hypotheses (e.g., those with fewer classes) are more
likely than complex ones, in advance of acquiring any data, and the prior
probability of the hypothesis reflects this preference. The prior probability
tenn prefers fewer classes, while the likelihood of the data prefers more, so
both effects balance at the most probable number of classes. Because of this,
AutoClass fmds only one class in random data.
• Objects are not assigned to a class absolutely. AutoClass calculates the
probability of membership of an object in each class, providing a more intuitive
classification than absolute partitioning techniques. An object described equally
well by two class descriptions should not be assigned to either class with
certainty, because the evidence cannot support such an assertion.
• All attributes are potentially significant. Classification can be based on any or
all attributes simultaneously, not just the most important one. This represents an
advantage of the Bayesian method over human classification. In many
applications, classes are distinguished not by one or even by several attributes,
but by many small differences. Humans often have difficulty in taking more
than few attributes into account. The Bayesian approach utilises all attributes
simultaneously, pennitting unifonn consideration of all the data. At the end of
learning, AutoClass gives the contributing factors to class fonnation.
• Data can be real or discrete. Many methods have difficulty in analysing mixed
data. Some methods insist on real valued data, while others accept only discrete
data. The Bayesian approach can utilise the data exactly as they are given.
• It allows missing attribute values.

6.5.2 Overview of Bayesian Classification

AutoClass is based on Bayes's theorem, for combining probabilities. Given

observed data D and a hypothesis H, it states that the probability that the
hypothesis explains the data p(H I D), (called the posterior probability of the
hypothesis given the data) is proportional to the probability of observing the data if
the hypothesis were known to be true p(D I H) (the likelihood of the data) times the
inherent probability of the hypothesis regardless of the data (P(H), the prior
probability of the hypothesis). Bayes's theorem is commonly expressed as,
 138 Data Mining and Knowledge Discovery for Process Monitoring and Control

(HI D) = p(H)p(D / H) (6.6)

p p(D)
For classification, the hypothesis H is the number and descriptions of the classes
from which the data D is believed to have been drawn. Given D, the goal is to
determine H so as to maximise the posterior p(HjD). For a particular classification
hypothesis, calculation of the likelihood of the data p(D/H) involves a
straightforward application of statistics. The prior probability of the hypothesis p(H)
is less transparent and is taken up later. Finally, the prior probability of the data,
p(D) in the denominator above, need not be calculated directly. It can be derived as
a normalising constant or ignored so long as only the relative probability of
hypotheses is considered.

6.5.3 Application to Classification.

The fundamental model of AutoClass is the classical finite mixture model of Everitt
and Hand [188] and Titterington et al. [189], made up of two parts. The first is the
probability of an instance being drawn from a class Cs (s = 1, k), denoted As. Each
class Cs then is modelled by a class distribution function, P(Xi I Xi E Cs, es), giving
the probability distribution of attributes conditional on the assumption that instance
Xi belongs to class C. These class distributions are described by a class parameter
vector, es, which for single attribute normal distribution consists of the class mean,
·
J.ls. and vanance cr s2•
Thus, the probability of a given datum coming from a set of classes is the sum of
the probabilities that it came from each class separately, weighted by the class
probabilities.

L
k
P(Xi Ie,A, k) = As P(Xi I Xi E Cs, es) (6.7)

It is assumed that the data is unordered and independent, given the model. Thus
the likelihood of measuring an entire database is the product of the probabilities of
measuring each object

p(x Ie,A, k) = n
i=!
n
p( Xi Ie,A, k) (6.8)

For a given value of the class parameters, the probability that instance i belongs
to a class using Bayes's theorem is calculated using
 Chapter 6 Unsupervised Learningfor Operational State Identification 139

p( Xi E Cs I Xi, a, A., k) = (6.9)

P(Xi Ia ,A., k)
These classes are "fuzzy" in the sense that even with perfect knowledge of object
attributes, it will be possible to detennine only the probability that it is a member of
a given class.
The problem of identifying a mixture is done in parts: determining the
classification parameters for a given number of classes and the number of classes.
Rather than seeking an estimator of the classification parameters (i.e., the class
parameter vectors, a, and the class probabilities, A.), the full posterior probability
distribution is sought. The posterior distribution is proportional to the product of the
prior distribution of the parameters p(a ,1..1 k) and the likelihood function p(x Ia ,A.,
k) .
p(a ,1..1 k) p(x Ia ,A., k)
p (a ,1..1 x, k) = - - - - - - (6.10)
p (x Ik)
I
The pseudo-likelihood p(x k) is simply the nonnalising constant of the posterior
distribution, obtained by nonnalising (integrating) out the classification parameters -
in effect, treating them as "nuisance" parameters:
p (x Ik) = If p(a ,1..1 k) p(x Ia ,A., k) dO dA. • (6.11)

To solve the second half of the classification problem (i.e., detennining the
number of classes k) the posterior distribution of the number classes k has to be
calculated. This is proportional to the product of the prior distribution p(k) and the
pseudo- likelihood function p(x Ik).
p(k) P (x Ik)
p(k Ix) = (6.12)
p(x)
In principle, the most probable number of classes are detennined by evaluating
p(k Ix) over the range of k for which the prior p(k) is significant. In practice, the
multi-dimensional integrals of Equation 6.6 are computationally intractable, and the
maximum of the function has to be found so that it can be approximated at about
that point.
 140 Data Mining and Knowledge Discovery for Process Monitoring and Control

6.5.4 The AutoClass Attribute Model

In AutoClass, it assumed that attributes are independent for each class. This permits
an extremely simple form for the class distributions used in Equation 6.2.
m

p(Xj I xjEC s,8s) = I1p(x ij lx i EC s ,8 s) (6.13)

j=!

where 8sj is the parameter vector describing the j th attribute in the sth class C s.
AutoClass models for real valued attributes are Gaussian normal distributions
parameterised by a mean and a standard deviation, and thus 8 sj takes the form
!-l s}
(8 sj ) =
cr s}
The class distribution is thus

p(XijlX i ECs,!-lS}'cr S}) = ~exp

1
2ncr s}
[-12 (xij cr- J2]
-
!-lsi

s}
(6.14)

6.5.5 Search Algorithm

As mentioned earlier AutoClass breaks the classification problem into two parts:
determining the number of classes and determining the parameters defining them. It
uses a Bayesian variant of Dempster and Laird's EM (expectation and
maximisation) algorithm [190] to find the best class parameters for a given number
of classes. To derive the algorithm, the posterior distribution is differentiated with
respect to the class parameters and equate with zero. This yields a system of non-
linear equations which hold at the maximum of the posterior:
Ws + w'-1
AS = s = l...k ( 6.15)
n+kCw' -1)
a a
- I n p(
as s
e +L
s)
n

i=!
W is as
s
In p( Xi
~
Ie, ) = 0 (6.16)

where Wi, is the probability that the datum, Xi' was drawn from class s (given by
Equation 6.4) and Ws is the total weight for class Cs :
Wis = P(Xi EC s I Xi,e,~)
 Chapter 6 Unsupervised Learningfor Operational State Identification 141

Ws = LW;s
;=1

To find a solution to this system of equations is found to be iterative based on

Equations 6.10 and 6.11 (treating w as a constant) and Equation 6.4 (treating "-
and e as constants).
For any given iteration, the membership probabilities are constant, so Equation
6.11 can be simplified by using Wi s via the derivative, so
a
II
A n A

-[pees) p(x;les,XiECsfis ]=0 (6.17)

ae s i=1

Thus far, the discussion of the search algorithm has related to general class model
with an arbitrary eSj • The Equation 6.12 is now applied to the specific AutoClass -
model of Equation 6.8 through 6.9.
For real valued attributes, the equations for the updated ~sj and cJSj are a function
- 2
of the prior information and the empirical mean, X,j and cr sj of the jth attribute in

class C" weighted by Wis :

n

L W;sXij
;=1
X sj =

L W;sX2ij
n

i=1 -2
X sj

The update formulas are then:

w'Xj '+U;:XSj
(6.18)
A

J.lsj = s =l...k, j =l...m

w'+U;:

""2
,
W' (cr j)
2
+ U;:cr Sj
2
W' u;: , _ 2
cr = + (cr .- x .) S = l...k
s} W'+Ws + 1 (W'+W)(W'+W
s S
+ 1) J SJ

(6.19)
Equations 6.10, 6.13 and 6.14 do not, of course, give the estimators explicitly;
instead they must be solved using an iterative procedure [191]. The simplest way of
estimating parameters using maximum likelihood estimate method is that suggested
by Wolfe [192] which is essentially an application of EM algorithm [190]. Whereas,
by the Bayesian parameter estimation method, AutoClass uses a Bayesian variant of
 142 Data Mining and Knowledge Discovery for Process Monitoring and Control

Dempster and Laird EM algorithm. Initial estimates of the A" 11" 0" s~ are obtained
by one of the variety of methods [188], and these are then used to obtain first
estimates of the p(sl Xj) i.e., the weights Wj s and hence Ws, - the E-step; these are
then inserted into Equations 6.10, 6.13 and 6.14 to give revised parameter estimates,
which is essentially the M-step. The process is continued until some convergence
criterion is satisfied [188].

6.5.6 The EM Algorithm

The steps of EM algorithm are:

E- STEP: The starting values of Jl, cr, A are obtained using a variety of cluster
analysis methods and are then used to obtain first estimates of the weights W(which
is an estimate of w) using Equation 6.4.
For instance assuming that the data has 3 classes (k=3), i.e., A= Al ,A2 and A3, two
attributes j=1,2, then there are 3* 2 = 611 's and 6 cr's. In total there are 6 +6+3=9
parameters to estimate, yielding 3 weights w i,s= 1 , wi' s=2 and w i,s=3 for observation
i.
M-STEP: This step requires the calculation of Jl, cr, A using Equations 6.10,
6.13 and 6.14.
The iteration of the two steps is then continued until the parameters are
maximised.

6.6 Application of AutoClass to the FCC Process

Using the same FCC process as the case study, 42 data patterns are studied, which
are summarised in Table 6.5. For every variable in each of the data patterns, 60
sampling points are recorded. For example, the dynamic trend represented by 15
(reaction temperature) in Figure 6.11 is composed of 60 data points when the valve
opening on the top of the distillation column changes from 100% to 90%. Six
process parameters are recorded including reaction and regeneration temperatures
(TRA and TRG), reactor and regenerator pressures (PRA and PRG), oxygen and
carbon monoxide volumetric contents in the flue gas from the regenerator (PT02
and PTCO). These are known to be the major variables for the FCC process,
although more precise characterisation would be expected if more parameters were
included.
 Chapter 6 Unsupervised Learningfor Operational State Identification 143

Table 6.5 Summary of the forty two data patterns.

Cases Description of cases
I - II Nonnal operation
12 Fresh feed pump (PI321) failure
13 A step increase of 15% in fresh feed flowrate
14 A step decrease of 50% in fresh feed flowrate
15 The valve opening on top of the distillation column changes from 100% to
90%
16 100% to 80%
17 100% to 60%
18 100% to 40%
19 100% to 30%
20 100% to 20%
21 Manual valve (V20) controlling catalyst to heat removal system, 75%~80%
22 ......................................................................... 75% ~ 90%
23 ......................................................................... 75% ~ 100%
24 75%~60%
25 75%~50%
26 75%~40%
27 75%~ 35%
28 ......................................................................... 75% ~ 30%
29 Recycle sludge oil flowrate increased to 300%
30 Fresh feed flowrate decreased to 10% of that ofnonnal operation
31 Recycle sludge oil pump (P1329) failure
32 Fresh feed flowrate increased by 15%
33 - 42 Nonnal operation

.·,.
510 660
."'-'",

._
650

..--- .... 640

·
j
u 6'"
u

a"' 470
620

J 610
• • • .. 1 5 - 1 7 - 1 9

450
600

S90
430
10 20 30 40 50 60
0 10 20 11mo 30 40 50 60 11. .

Figure 6.11 Reaction temperature response Figure 6.12 Regeneration temperature

for cases 15, 17 and 19. response for cases 15, 17 and 19.

Clearly prior knowledge about the data makes it possible to test the automatic
classification capabilities. Since each data instance involves six variables and each
variable is represented by 60 data points, the data base is a 360 x 42 matrix. In
 144 Data Mining and Knowledge Discovery for Process Monitoring and Control

Chapter 3, various methods have been introduced for reducing the dimensionality of
a dynamic trend without signal losing its important features. These include wavelets,
principal components and episode representations. Here the crude data are used
since the concern is the identification of operational states.

6.6.1 Result of Classification

Although the classification using AutoClass is an automatic process, it still has

many options which allow users to exercise control over the learning and the way in
which the results are output [193]. For example, choices can be made of the
attribute probability models including
(i) the single multinomial model which implements a single multinomial
likelihood model term for symbolic or integer attributes, which is conditionally
independent of other attributes given the class.
(ii) the single normal CN model which uses real valued attributes with a
conditionally independent Gaussian normal distribution, assuming no missing
values.
(iii) the single normal CM Model which also models real valued attributes but
allows missing values.
(iv) the multi normal CN covariant model, also for real valued attributes expresses
mutual dependencies within the class. The probability that the class will produce
any particular instance is then the product of any independent and covariant
probability terms.
In this case the single normal CN model is used. The approach is based on
generation of alternative classification schemes which are ranked (the first being the
best). It then remains to analyse the classification schemes to determine which is the
most acceptable. The best for the current problem is shown in Table 6.6 .

..
. T able 65
Table 6.6 The AutoClass clustenng resu ts 0 f the 42 cases III
Classes Cases
I 1,2,3,4,5,6, 7, 8,9, 10, 11,
33,34,35,36,37,38,39,40,41,42
2 21,22,2331,3213,29
3 15,16,17,18,19,20
4 24,25,26,27,28
5 12, 14,30
 Chapter 6 Unsupervised Learningfor Operational State Identification 145

6.6.3 Analysis of the Classification Results

The classified results are analysed by comparing Table 6.6 and Table 6.5. Class I
includes cases 1 - 11 and 33 - 42, which correspond to normal operations as can be
seen from Table 6.5 so it is reasonable to assign them into a single class. The
significance of being able to automatically distinguish between normal and
abnormal operational data, which represent moderate to significant disturbances as
well as faults is that process upsets can be seen.
Class III includes cases 15 - 20 which correspond to decreases in the opening of
the valve 401-ST opening from 100% to 90%, 80%, 60%, 40%, 30% and 20%
which cause the differential pressure (PRG - PRA) between the regenerator and
reactor to decrease. Consequently, the regenerated catalyst circulation rate falls and
the reaction and regeneration temperatures will be influenced. The closed - loop
dynamic responses of reaction and regeneration temperatures for cases 15, 17 and
19 are shown in Figures 6.11 and 6.12. It can be seen that although they are not
identical, they show greater similarity than those in class I, i.e., normal operation.
Class IV includes cases 24 - 28 which correspond to the changes in the opening
of the hand-operated valve V20, which first cause the regeneration temperature to
change due to changes in heat transfer and then affect all other parameters. Cases 24
- 28 refer to a reduction in the opening of V20 from its normal operation value
(75%) to 60%, 50%, 40%, 35% and 30%. All these operations cause the
regeneration temperature to increase, so is quite reasonable that they should be
grouped into one class. The regeneration temperature TRG and oxygen volumetric
percentage in flue gas PT02 change are shown in Figures 6.13 and 6.14. It is
important that cases 21, 22 and 23 are grouped in a different class (class II)
although they also represent V20 opening changes, because they represent increases
in opening which have different effects on associated variables. Figure 6.15 shows
the PT02 changes for cases 21, 22 and 23. It is clear that they are different from
cases 24,26 and 28 shown in Figure 6.14.
 146 Data Mining and Knowledge Discovery for Process Monitoring and Control

M0r---------------------____ ~
"r-----------------------~
-24 -26 ·-"-28

.,.~.....................-..,..-*.........- ... - .....-......--..

,,•
2.5

.,,
I

05 -24-26·""-28

10 20 TI~ 40 50 60

Figure 6.13 Regeneration T responses Figure 6.14 Dynamic responses of the 02

for cases 24, 26 and 28. content in flue gas for cases 24, 26 and 28.

4~r-----------------------~
0.9
"",,
0.' ,,
,
.. ,-.._-".. ---........_----_._..
0.7
i .90

..i
0.6
U

'"
i
0.5

0.'
.... .. -21 ~
--~~
--23
of .85
0.3

!l 0.2 - - 3 1 · _ .. -32

0.1

.90~--+_--~--~---+--~--~
10 20 30 .0 50 60 o 10 20 30 40 50 60
Tio. Time

Figure 6.15 Dynamic responses of the Figure 6.16 Reaction temperature

02 content in flue gas for cases 21, 22 and 23. responses for cases 13,29,31 and 32.

Class V includes 12, 14 and 30 which are clearly in one class and represent feed
pump P1329 failure, feed flowrate decreases by 50% and 90%, all mean a sharp
decrease in feed flowrate.
Class II includes 21, 22, 23, 31, 32, 13 and 29. As discussed above, it is
reasonable that 21, 22 and 23 should be grouped together since they represent
increases in the opening of V20 which cause decrease in regeneration temperature.
Both 13 and 32 represent slight increases (15% and 9%) in fresh feed flowrate,
while 29 a three times increase in sludge oil flowrate and case 31 sludge oil pump
failure. Since sludge oil flow is only very small compared with the fresh feed at
normal operations (8000 kg/hr sludge oil vs 150,000 kg/hr fresh feed), it is not
surprising that a factor of three time increase in sludge oil has a similar effect on
process operation as slight fresh feed increases by 15% and 9%. The major
difference between the four cases, 12, 32, 29 and 31 is that 12, 29 and 32 represent
increases in feed while 31 indicates a decrease, as indicated in the early stage of
reaction temperature responses (Figure 6.16). Because the process is under closed
 Chapter 6 Unsupervised Learningfor Operational State Identification 147

loop control, such a difference is not significant enough to regard 31 as a different

class. However it is interesting that 21, 22, 23 and 31, 32, 13, 29 are grouped
together which would not have been expected. The explanation is that they are not
very significant disturbances and all affect in similar ways the operation of the
process mainly by influencing the heat balance of the two reactors which is the
dominant factor in FCC operations.

6.7 General Comments

The above discussion has introduced unsupervised machine learning as a powerful

method for process operational state identification. The data pre-processing
methods described in Chapter 3 have been used to reduce the dimensionality of the
data and remove noise before analysis of data using unsupervised machine learning.
There are several issues that are important but have not been fully addressed. First,
for on-line process monitoring, it is important for the approach to be recursive.
ART2 is an recursive method, but AutoClass is not. Second, although unsupervised
procedures do not need training data, they are usually not as accurate as supervised
methods, therefore interpretation and validation of results becomes an important
issue. Furthermore when adapted to on-line monitoring the speed is obviously
critical as is the selection of variables used for classification. Process variables tend
to be interrelated so it is necessary to remove redundant variables without losing the
important ones.
CHAPTER 7
INDUCTIVE LEARNING FOR CONCEPTUAL
CLUSTERING AND REAL-TIME PROCESS MONITORING

Multivariate statistics, supervised and unsupervised machine learning approaches

have been introduced in previous chapters as a basis for developing process
monitoring systems. The approaches often depend on calculating a similarity or
distance measure for identifying clusters in data. Apart from giving predictions,
however they are not able to provide causal explanations on why a specific set of
data is assigned to a particular cluster. In this chapter, conceptual clustering based
on an inductive machine learning approach is introduced for use in designing state
space based process monitoring systems. As distinguished from similarity or
distance based clustering, such conceptual clustering is able to generate conceptual
knowledge about the major variables which are responsible for clustering, as well as
predicting operational states. The resulting knowledge is expressed in the form of
production rules or decision trees.
The first stage is to introduce inductive learning. Following this, an example is
given to demonstrate how inductive learning can be used to analyse process
operational data which has been averaged over days or weeks so that the embedded
information can be exploited to improve process performance. After this a section
on application of inductive learning to conceptual clustering and real-time process
monitoring is followed, by reference to a relatively simple case study based on a
continuous stirred tank reactor (CSTR). A more complicated case study based on a
refinery methyl tertiary butyl ether (MTBE) example is then presented.

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
150 Data Mining and Knowledge Discovery for Process Monitoring and Control

7.1 Inductive Learning

Inductive learning is probably the most widely studied method based on symbolic
learning [202-204]. It attempts to acquire a conceptual language for describing an
object by drawing inductive inference from observations. The focus is on deriving
rules or decision trees from unordered sets of examples, especially attributes based
induction, a formalism where examples are described in terms of a fixed collection
of attributes. The discussion excludes such learning methods as feedforward neural
networks which learn to develop an implicit rather than explicit and transparent
rules or decision trees. An obvious motivation for inductive learning is that it
provides a method for solving the bottleneck arising from knowledge acquisition
which is necessary to develop expert systems. It is relatively easy for human experts
to document cases rather than for them to articulate the expertise explicitly and
clearly. Several approaches to inductive learning have been proposed, such as AQ 11
[196], VersionSpaces [197] and C5.0 [18, 19, 20]. Here the focus is on C5.0, a
system that is designed to learn to develop rules and decision trees from examples.

7.1.1 The Inductive Learning System

The conceptual clustering approach used in C5.0 was developed by Quinlan [18, 19,
20]. Given a database of objects (or in other words data sets) which are described in
terms of a collection of attributes, which measure some important feature of an
object. Each object belongs to one of a set of mutually exclusive classes, the task is
to develop a classification rule that can determine the class of any object from its
values of the attributes. The decision tree generated can be used for conceptual
clustering. The procedure is iterative and can be summarised as follows [18, 19]:

(1) Select a random subset of the given training examples (called the window)
(2) Repeat (a) to (c)
(a) Develop a decision tree which correctly classifies all objects in the
window
(b) Find exceptions to this decision tree in the remaining examples
(c) Form a new window by adding incorrectly classified objects to the
window
 Chapter 7 Inductive Learning and Conceptual Clustering 151

Until there are no exceptions to the decision tree.

The crux of the problem is how to develop a decision tree for an arbitrary
collection of objects in the window. To form a decision tree requires selecting the
root attribute. To do this, assume that there are only two classes representing all the
data, P and N (extension to any number of classes is not difficult). The method of
finding the root attribute is adopted from an information based method that depends
on two assumptions. Suppose the window C contains p objects of class P and n
objects of class N. The assumptions are:
(1) Any correct decision tree for the window C will classify objects in the same
proportion as their representation in C. An arbitrary object will be determined as
belonging to class P with probability p/(p+n) and to class N with probability
n/(p+n).

(2) When a decision tree is used to classify an object, it returns a class. A decision
tree can then be regarded as the source of a message 'P' or 'N'. with the expected
information needed to generate this message given by
I(p,n)= _ _ P- log _P_ _ _ n_ log _n_ (7.1)
p+n 2 p+n p+n 2 p+n

If attribute, A, having values {A I, A 2, ..• A v }, is used for the root of the decision
tree, it will partition the window C into { C I, C2 , ... Cv} where C i contains those
objects in C that have values Ai of A. Suppose C i contains Pi objects of class P and
ni of class N. The expected information required for the subtree for C i is I(pi, ni) and
for the tree with A as root is then obtained as the weighted average given by
E(A) = ±Pi+niI(p,n) (7.2)
i=l p+n I

where the weight for the ith branch is the proportion of the objects in C that belong
to C i. The information gained by branching on A is therefore

gain(A) = I(p, n) - E(A) (7.3)

The approach calculates the gain for all attributes and chooses the attribute having
the biggest gain as the root node. The root node will have as many branches as its
values. The branches divide the database into a number of subsets. For each subset,
the root node is obtained following the same procedure.
 152 Data Mining and Knowledge Discovery for Process Monitoring and Control

The approach has been used in the commercial software C5.0 [20], which has
evolved from the earlier versions C4.5 [18] and ID3 [19]. A major limitation of ID3
was that it assumed that the values of all attributes are discrete, for instance a colour
being red or green. C4.5 claimed to be able to deal with continuous-valued
attributes, is still weak compared with the way it deals with discrete-valued
attributes, as noted by Quinlan [198]. Though Quinlan [198] made a further effort to
improve the method so that it could deal with continuous-valued attributes, the
outcome is still not very satisfactory. Nevertheless, C5.0 has become one of the
most well known tools for use in data mining and knowledge discovery, especially
in domains involving only discrete values.

7.1.2 Dealing with Continuous-valued Attributes in

Inductive Learning

Like most of the available inductive learning methods, C5.0 was developed for
problem domains where attributes only take discrete values. The methods have
proved to perform remarkably well with discrete valued attributes. However, when
the problem domains contain real numbers, the performance usually decreases in
terms of accuracy. Using inductive learning based on continuous-valued attributes
requires discretisation of the values into a number of intervals. To deal with this, a
number of approaches have been proposed.
The class-separating method [199] is based on the assumption that if the
assignments of individual examples are known, then such knowledge can be used to
discretise the attributes. An example of this is to use the range of boiling points of
petroleum fractions into product grades. This approach however sometimes
produces too many intervals which are not very informative. Moreover, if the
assignment of individual cases is not known, the approach can't be used.
It is possible to use an equal distance approach to divide the values of the
continuous variable between the minimum and maximum values into a fixed number
of intervals. Alternatively equal n1lmhers can be used to make each interval contain
equal number of examples.
Another approach is the k-nearest neighbours approach which tries to estimate to
which class the value of a specified attribute most likely belongs. It places a border
between two values Xi and Xi+ I if the estimates for them are different. The estimates
 Chapter 7 Inductive Learning and Conceptual Clustering 153

are based on the assumption that the most probable class is the most common class
among the k nearest examples, where k is normally a user specified parameter.
Wu [199] developed a Bayesian discretizer. The approach is shown to be more
accurate than some other approaches but can only be used for integer values and
requires the assignments for the examples to be known. The method used by C4.5
[18] is based on information gain. But this approach only provides binary
discretisation.

7.1.3 Attributes That Take Values Related to Dynamic Trends

In process monitoring and control, dynamic trends of variables might be more

important than the instantaneous values. The differences for the seven dynamic
trends for a variable shown in Figure 3.4 can have important implications. The issue
of dealing with this kind of problem has not previously been considered. An
approach using principal component analysis to extract qualitative concepts from
dynamic trend signals was given in Section 3.2.3.2, so will not be repeated here. In
this chapter it will be shown how the concept formation can be used in inductive
learning to develop conceptual clustering systems for process monitoring and
diagnosis.

7.2 IL for Knowledge Discovery from Averaged Data

Saraiva [5] and Saraiva and Stephanopoulos [4] divided process plant data and
monitoring and control strategies into layers as shown in Figure 7.1 and investigated
the application of inductive learning to the highlighted layer in the figure based on
analysis of daily/weekly averaged data as a basis to continuous process
improvement. The goal is to develop a conceptual language describing the
contributions of various operational parameters to one or a number of performance
metrics which are product quality related.
154 Data Mining and Knowledge Discovery for Process Monitoring and Control

Level of decision making Scope of Application

Strategic Decisions
Overall Plant

Plant Departments

Plant Area

SecondslMinutes

SecondslMinutes

Time Scales for Decision-Making

Figure 7.1 Levels, time scales, and application scopes of decision making
activities.

Saraiva [5] presented four industrial case studies using this approach. A simple
example is used here to illustrate the methodology. The case study is concerned with
records of operating data from a refinery unit [200] shown in Table 7.1 which has
five variables, Xl, X2, X3, X4 and y. The latter is the octane number of the gasoline
product and is discretised as very low if the value is less or equal to 91, low if it is
between 91 and 92, and good if greater than 92. Xl, X2 and X3 are different measures
of the feed composition and X4 is the log of a combination of process conditions.
Figure 7.2 shows the induced tree as well as the partition of the (Xl, X4) plane
defined by the leaves, together with a projection of all the available (x, y) pairs on
the same plane. These two decision variables clearly influence the current
performance of the refinery unit, and the decision tree leaves give a reasonable
partition of the plane. To achieve better performance, operating zones that will
result in obtaining mostly y = 3 values need to be found. Terminal nodes 2 and 7
identify two such zones. The corresponding solutions are

If Xl E[44.6, 55.4], then conditional probability ofy = 3 is 0.9

Ifx4 E[1.8, 2.3], then the conditional probability ofy =3 is 1.0
 Chapter 7 Inductive Learning and Conceptual Clustering 155

T able 7.1 Data 0 btained III a refinery unit.

x1 x2 x3 x4 Obsv. x1 x2 x3 x4 Obsv.
55.33 1.72 54 1.66219 92.19 71.31 3.44 55 1.60325 90.51
59.13 1.2 53 1.58399 92.74 72.3 4.02 55 1.66783 90.24
57.39 1.42 55 1.61731 91.88 68.81 6.88 55 1.69836 91.01
56.43 1.78 55 1.66228 92.8 66.61 2.31 52 1.77967 91.9
55.98 1.58 54 1.63195 92.56 63.66 2.99 52 1.81271 91.92
56.16 2.12 56 1.68034 92.61 63.85 0.24 50 1.81485 92.16
54.85 1.17 54 1.58206 92.33 67.25 0 53 1.72526 91.36
52.83 1.5 58 1.54998 92.22 67.19 0 52 1.86782 92.16
54.52 0.87 57 1.5523 91.96 62.34 0 48 2.00677 92.68
54.12 0.88 57 1.57818 92.17 62.98 0 47 1.95366 92.88
51.72 0 56 1.60401 92.75 69.89 0 55 1.89387 92.59
51.29 0 58 1.59594 92.89 73.13 0 57 1.81651 91.35
53.22 1.31 58 1.54814 92.79 65.09 1.01 57 1.45939 90.29
54.76 1.67 58 1.63134 92.55 64.71 0.61 55 1.38934 90.71
53.34 1.81 59 1.60228 92.42 64.05 1.64 57 1.33945 90.41
54.84 2.87 60 1.54949 92.43 63.97 2.8 60 1.42094 90.43
54.03 1.19 60 1.57841 92.77 70.48 4.64 60 1.5768 89.87
51.44 0.42 59 1.61183 92.6 71.11 3.56 60 1.41229 89.98
53.54 1.39 59 1.51081 92.3 69.05 2.51 60 1.54605 90
57.88 1.28 62 1.56443 92.3 71.99 1.28 55 1.55182 89.66
60.93 1.22 62 1.53995 92.48 72.03 1.28 56 1.6039 90.08
59.59 1.13 61 1.56949 91.61 69.9 2.19 56 1.67265 90.67
61.42 1.49 62 1.4133 91.3 72.16 0.51 56 1.55242 90.59
56.6 2.1 62 1.54777 91.37 70.97 0.09 55 1.45728 91.06
59.94 2.29 61 1.65523 91.25 70.55 0.05 52 1.26174 90.69
58.3 3.11 62 1.29994 90.76 69.73 0.05 54 1.28802 91.11
58.25 3.1 63 1.19975 90.9 69.93 0.05 55 1.36399 90.32
55.53 2.88 64 1.20817 90.43 70.6 0 55 1.4221 90.36
59.79 1.48 62 1.30621 90.83 75.54 0 55 1.67219 90.57
57.51 0.87 60 1.29842 92.18 49.14 0 40 2.1714 94.17
62.82 0.88 59 1.40483 91.73 49.1 0 42 2.31909 94.39
62.57 0.42 60 1.45056 91.1 44.66 4.99 42 2.14314 93.42
60.23 0.12 59 1.54357 91.74 44.64 3.73 44 2.08081 94.65
65.08 0.1 60 1.6894 91.46 4.23 10.76 41 2.1707 97.61
65.58 0.05 59 1.74695 91.44 5.53 7.99 40 1.99418 97.08
65.64 0.05 60 1.74915 91.56 17.11 5.06 47 1.61437 95.12
65.28 0.42 60 1.78053 91.9 67.6 1.84 55 1.64758 91.86
65.03 0.65 59 1.78104 91.61 64.81 2.24 54 1.69592 91.61
67.84 0.49 54 1.72387 92.09 63.13 1.6 52 1.65118 92.17
73.74 0 54 1.73496 90.64 63.48 3.46 52 1.48216 91.56
72.66 0 55 1.71966 91.09 62.25 3.56 50 1.49734 92.16

It effectively says that one should expect to get almost only "good" y values while
operating inside these zones of the decision space, as opposed to the current
operating conditions, which lead to just 40% "good" y values.
Saraiva [5] presented more complex case studies and methods for defining
performance metrics. He used the binary discretisation mechanism for continuous-
valued variables embedded in C4.5. In addition, the approach is only suitable for
 156 Data Mining and Knowledge Discovery for Process Monitoring and Control

analysing data of daily/weekly averaged data, not on-line data from computer
control systems, based on seconds.

(b)
• .,=1
1=3 01,,2
•
'II
• .1=3
2.0
••
.
..
.-
1=3 • • n

x
~C ':~CIIo o·

~ o 0 •
..-:
0 •
..
o·~
I.S • ~o • 0

...
o c ••
• 1=2 1=1

• 0 •
50 eo 10
XI

Figure 7.2 (a) Induced decision tree; (b) partition of the plane defmed by its leaves.

7.3 Inductive Learning for Conceptual Clustering

and Real-time Monitoring

In this section, we present our work on application of inductive learning to the

analysis of data collected by computer based control systems which is on second
basis. A conceptual clustering approach is thus developed for designing state space
based on-line process monitoring systems. The approach is illustrated by reference
to a simple case study based on a CSTR reactor (see Appendix A). It is concerned
with analysis of an operational database consisting of eighty five data patterns
obtained in operating the CSTR. For each data pattern seven variables are recorded
including, reaction temperature TR, reaction mixture flow out of the reactor F0'
cooling water flowrate Fw, feed flowrate F j , feed inlet temperature T j , feed
concentration Cj, and cooling water temperature Tw. Each variable is recorded as a
dynamic trend comprising 150 sample points. The goal is to identify operational
 Chapter 7 Inductive Learning and Conceptual Clustering 157

states using a conceptual clustering approach. This will then be extended to a more
complicated case study of a refinery MTBE process in Section 7.4.
The approach basically comprises the following procedures: (1) concept
extraction from dynamic trend signals using PCA. (2) identification of operational
states using an unsupervised machine learning approach, and (3) application of the
inductive machine learning system to develop decision trees and rules for process
monitoring.

7.3.1 Concept Extraction from Dynamic Trend Signals

This has been discussed in detail in Chapter 3, so only a brief review is presented
here. For a specific set of data, the value of a variable represents a dynamic trend,
consisting of tens to hundreds of sampled points. In inductive learning, it is the
shape of the trend that matters so for a specific variable, when the trends of all the
data sets are considered and processed using PCA, the first two principal
components (PCs) can be plotted in a two dimensional plane, as shown in Figure
7.3. Figure 7.3 showing PC-I-TR and PC-2-TR corresponds to the first two PCs of
the reaction temperature TR. The data sets are grouped into clusters in this two
dimensional plane. This pennits a dynamic trend to be abstracted as a concept as
typically by PC-l-TR in region A. The following sections will show how this
process can be used for conceptual clustering using inductive learning.

7.3.2 Identification of Operational States

The next step is identification of operational states. In this case, this can be done
using PCA because there are only eight variables. For more complex processes,
more sophisticated approaches need to be used, which will be described later in the
case study of MTBE. The first two PCs of the eight variables (TR, Fo, Fw, Fi, Ti,
Ci, Twi, L) are plotted in Figure 7.7. The five groups which are identified represent
the 85 data cases as five clusters corresponding to five distinct operational modes.
Detailed examination of the clusters shows that these groups are reasonable.
 158 Data Mining and Knowledge Discovery for Process Monitoring and Control

~~------------------------~ 40
Cases 22.23,45.46.67.68.70
30
B

Om; 25-46
Om; 1-24
D 20 .. , ...... ;.~. C<lSe 19
B .... C
10 '.
....... ::. ....
., .. \
10 ", ".,,, "

~ . ,'"''
'.
:
......
f .:

~ 0 &: 0
,........ j .....
: ", " •••• II.:
,.!,
:::~::.;:'
'"
-10' ~ -10
D
-2) A -2C ".,', .....:.;.. C<lSeS 71-80
Om;8HlS A
-3C C<lSes H R,20.21 ,24-44,
_m
4Nl6,69.81-85
~I+--r--r-~_.--.--'--.--r~
~m ~ ~ ~ ~ 0 2) ~ ro ro -40 -20 0 20 40 60 60 100 120
K:-I-lR PC-I-F.

Figure 7.3 PCA two dimensional plot ofTR. Figure 7.4 PCA two dimensional plot ofF o
rn~----------------------~
1m

l.,
8J 65.67.6'\8Hl5 Cases 1-15,17,20-39,41,
ffi
41
Om;~~~ 43-60,62,64,66-85
B
B ,-...1 '--__
~
~
J)
0
-J)
:-"'\
".:,f,"
Y
,,::,'\1',":
~~/\~.i.~,~
ff \
"tit.., .... -:~ t)
, c .........._____.. __.-.....
'-"\
:~.,,;:(t~yr.'.~·".';,:;
'••.: C
-4l A .". Cases 18.
-6l Om; 55, 7HD 42,65
-8l
-1m
-rn +-,..-,..-,..-,..-,..-,..-,-,..-,..-,..-.,.--i
-rn -1m -8l -6l -4l .J) 0 J) 41 ffi 8J 1m rn
K;.1-FW PC-l-Fi

Figure 7.5 PCA two dimensional plot ofF w • Figure 7.6(a) PCA two dimensional plot ofF;.
"
I'
10 Cases 14,39,59
Cases 1,25.27,47,49 IC
Cases 4,28,51 C
5 B D 5

~ 0 :::~.:~:. .::>::. C() ;'1.: ~)':'~. ~O :f

: .:
:::~::::
'B
-5 Cases 6,30 C D
~-" : .:
: .:
Cases 12,37,57
: .:
Cases 3,5.7-24,29,31--46, Cases 2,26. 8 -Ie .....~ A
50,52-85
-lC _I' Cases H 1,13,15-36,
38,40--56,58,60-85
~Ur_--~--~--~--.---r--4
-15I---~----.---_,_--_.---r_---I
-20 0 20 40 60 80 100
-15 -10 -5 o 5 10 15
PC-l-TI PC-I-O

Figure 7.6(b) PCA two dimensional plot ofTi. Figure 7.6(c) PCA two dimensional plot ofC.
 Chapter 7 Inductive Learning and Conceptual Clustering 159

15r---------------.
ClIses 1-7.9.1 1-31,34.
10
.l6-12.54.56-l!5
Ca'6.21,44,69 Caso; 23,46.67.68
B
.'
'J
0 :,:
:'t'.'....
: a:
.', .'.
~
'.'
~ ~. '.,,0'

·5 A C D
c
Caso; 71-M
ClIses35.55 ClIses8,32,53 ClIsesI(),33 -20 Caso; 1-20.
·10 22.24-43,45,
-30 47-<;6.70.81-85

·15 ~~-r--~~~~-~-~-_4
·15 -10 ·5 o 10 15 -100 -M .(i) ~ -20 20 40
PCI·Thi PC-I-L

Figure 7.6(d) PCA two dimensional plot ofTwi. Figure 7 .6(e) PCA two dimensional plot of L.

100r-----------------------------~

80
ABN 2
60 NOR 2 Cases 81-85
Cases 47-70
" ~:: :'.' ..................... ,.
40
.... :.
.. ..,.-.-... .......
20 'I,
.................... ....

.........................

. ..' ....:
ABN I
o Cases 71-80 .......... .J .....

-20 .'0 ...... , ..: . . . . • ......

. .-. -:-
NOR 3 ..•. :::::::::: .. ::::::::::. '"
-40 Cases 25-4~ •.. ·· • • ".
., .'
':,
-60 NOR I _.:::.....~ ....................... .
Cases 1-24
-80 +----.---.------.---...---...,.----,..---.---1
-180 -140 -100 -60 -20 20 60 100 140

PC-I-State

Figure 7.7 PCA two dimensional plot of the CSTR operational states.

7.3.3 Conceptual Clustering

Having characterised the dynamic trend signals and identified the operational states,
it is necessary to find out how to generate knowledge which correlates the variables
and operational states. To do this requires generating a file as shown in Table 7.2. In
fact, each data set in Table 7.2 can be interpreted as a production rule. Thus, the
first case is equivalent to the following rule,
160 Data Mining and Knowledge Discovery for Process Monitoring and Control

IF PC-L = C in Figure 7.6(e)

AND PC-TR =D in Figure 7.3

AND PC-Fo = A in Figure 7.4

AND PC-Fw = D in Figure 7.5

AND PC-Twi =B in Figure 7.6(d)

AND PC-Ci = A in Figure 7.6(c)

AND PC-Ti =D in Figure 7.6(b)

AND PC-Fi = B in Figure 7.6(a)

THEN States =NORI in Figure 7.7

Table 7.2 The data structure used by C5.0 for conceptual clustering.

PCL PCTR PC]o PC_Fw PC_Tw PCCi PCTi PC]i States

i
C D A D B A D B NORI
C D A D B A E B NORI
... ... ... ... ... ... ... ... .. .
A C D A B A C B ABNI
A C D A B A C B ABNI
... ... ... ... ... ... ... ... .. .
Variable name, Value space, Ref. figure Variable name, Value space, Ref. figure
PCL, [A, B,C, D], Figure 7.6(e) PCCi, [A, B, C], Figure 7.6(c)
PC]o, [A, B,C, D], Figure 7.4 PCTi, [A, B, C, D, E], Figure 7.6(b)
PC]w, [A, B, C, D], Figure 7.5 PCFi, [A,B,C,D], Figure 7.6(a)
PCTwi, [A, B, C, D], Figure 7.6(d) States, [NORI, NOR2, NOR3, ABNI, ABN2]
PC TR, [A, B, C, D], Figure 7.3 Figure 7.7

Obviously this is simply an explanation of the database and the decision tree
developed will be very complex. C5.0 makes it possible to develop a simpler tree.
Simple tree is preferable because it can usually perform better than a complex tree
for data cases outside the training data set.
 Chapter 7 Inductive Learning and Conceptual Clustering 161

The decision tree developed for the CSTR case study is shown in Figure 7.8 and
can be converted to production rules, as shown in Table 7.3. CS.O identifies the
reactor temperature as the root node and states that if TR is in the region of A, B or
D of Figure 7.3, then the operation will be in regions ABN2 (abnormal mode 2),
NOR2 (Normal operation mode 2), or NORl (Normal operation mode 1) of Figure
7.7. If TR is in the region C in Figure 7.3, then there are three possible situations
depending on Fo. If Fo is in the region D of Figure 7.4, then the operation will
correspond to ABNl (Abnormal operation 1): ifFo is in A or B of Figure 7.4, then
the operation will be NOR3 (Normal operation 3). The result effectively states that
it is possible to focus on monitoring TR in Figure 7.3. IfTR is in the region C, then
Fo in Figure 7.4 should be examined. It also shows the variables responsible for the
location placing the operation in a specific region of Figure 7.7.

-<c c c
0 0 0
'50 '50 '50
....
<l)
~ ~
.5 .5

ABN2 NOR2 NOR3 ABN 1 NORl

(Fig7. 7) (Fig7.7) (Fig7.7) (Fig7.7) (Fig7.7)

Figure 7.8 The decision tree developed for the CSTR after conceptualization of the
variable trends.
 162 Data Mining and Knowledge Discovery for Process Monitoring and Control

Table 7.3 The production rules converted from the decision

tree in Figure 7.8.
Rule 1: IF TR =A in Figure 7.3
THEN Operational state =ABN 2 in Figure 7.7

Rule 2: IF TR = B in Figure 7.3

THEN Operational state = NOR 2 in Figure 7.7

Rule 3: IF TR =C in Figure 7.3

AND Fo =A or B in Figure 7.4
THEN Operational state = NOR 3 in Figure 7.7

Rule 4: IF TR =C in Figure 7.3

AND Fo = D in Figure 7.4
THEN Operational state = ABN I in Figure 7.7

Rule 5: IF TR =D in Figure 7.3

THEN Operational state = NOR I in Figure 7.7

yes

: ••••••••••• 1

Figure 7.9. The decision tree developed for the CSTR using the eigenvalues of the
first two principal components of each variable.
 Chapter 7 Inductive Learning and Conceptual Clustering 163

Table 7.4 The numerical values of the first two principal components of each
. F19ure 7 9
. bl e are use d b)y C5 0 to deve Iop the tree III
vana
Fi Ti C, Twi Fw Fo TR L State

pC·t PC·2 pC·t PC-2 PC·1 PC-2 PC·1 PC·2 PC·1 PC·2 PC-t PC-2 PC·1 PC·2 PC·1 PC-2

·107 0.47 5.66 -0.45 -4.47 0.19 -0.10 ·0.08 -41.9 -5.57 -11.5 ·0.92 47.83 ·2.48 119 0.04 NOR1

-0.92 0.47 ·0.29 0.01 -4.72 0.28 ·0.11 -0.06 86.7 0.64 -11.7 -0.98 -88.9 -6.19 11.96 0.30 ABN2

Table 7.S The production rules converted from the decision

tree of Figure 7.9.
Rule 1: IF PC-l-TR =< -33.8
AND PC-I-TR =< -85.9
THEN Operational state = ABN 2
Rule 2: IF PC-I-TR =< -33.8
AND PC-I-TR > -85.9
THEN Operational state = NOR 2
Rule 3: IF PC-I-TR > 4.2
THEN Operational state =NOR 1
Rule 4: IF PC-I-TR =< 4.2
AND PC-I-Fo =< 28.7
THEN Operational state =NOR 3
Rule 5: IFPC-I-TR> 4.2
AND PC-I-Fo> 28.7
THEN Operational state =ABN 1

The decision tree shown in Figure 7.8 and the rules in Table 7.3 provide guidance
for operation clustering which is transparent.
In the above discussion, the dynamic trends in the two-dimensional PCA planes
have been abstracted. An alternative way is to use the numerical values of the first
two PCs directly. The data structure to be processed by C5.0 is then put in the
format of Table 7.4. A decision tree thus developed is shown in Figure 7.9 and the
rules shown in Table 7.5. Inspection of Figure 7.9 reveals that it is very similar to
 164 Data Mining and Knowledge Discovery for Process Monitoring and Control

the tree in Figure 7.8. For example, the conditions leading to ABN2 in Figure 7.9
are

IF PC-I-TR < -33.8

AND PC-I-TR < -85.9

THEN ABN2

From Figure 7.3 it can be seen that the two preconditions of the rule are
equivalent to IF TR = region A and so gives the same result as Figure 7.8. The only
difference is the last branch leading to NOR3 (the dashed line box of Figure 7.9). In
Figure 7.9 the rule is,

IF PC-I-TR>-33.8

AND PC-I-TR < 4.2

AND PC-I-Fo < 28.7

THEN NOR3

By making reference to Figures 7.3 and 7.4, the rule condition is equivalent to IF
TR = C in Figure 7.3 and Fo = A or B or C in Figure 7.4. But in Figure 7.8, the
corresponding rule condition is TR = C in Figure 7.3 and Fo = A or B. This slight
difference does not say which approach is better. However, in the next section
involving a larger case study, conceptualisation of dynamic trends first gives better
results.

7.4 Application to the Refinery MTBE Process

The above description of conceptual clustering is based on a CSTR. Here we apply

the approach to a more complicated case study, the refinery methyl tertiary butyl
ether (MTBE), which is described in more detail in Appendix C. A database of 100
sets is obtained using a process simulator and the output is summarised in Table C 1
of Appendix C. Thirty six data sets correspond to various operations which are
regarded as abnormal or subject to significant disturbances. The rest are considered
 Chapter 7 Inductive Learning and Conceptual Clustering 165

as operating normally. The study is restricted to 100 data sets since increasing the
size of the data set by including more normal operational data does not make
difference to the result and it makes it easy to understand the analysis and
presentation of the result. Each data set consists of twenty one variables, which are
listed in appendix C. For each variable, a dynamic trend consisting of 256 points is
used to record the response to changes. Therefore the size of the data to be analysed
is 100x21x256.

.
T a bie 76The c usters 0 f operatlOna states usmg ART2
Clusters Cluster Cases Clusters Cluster Cases
Name Name

I ABNI 1,2,3 7 ABN7 14,15

2 ABN2 4,5,7,8, II 8 ABN8 16

3 ABN3 6 9 ABN9 19,20,21,22

4 ABN4 9 10 ABNIO 23,24,25,26,27

5 ABN5 10,35,36 II ABNII 29,30,31,32

6 Normal 12,13,17,18 12 Normal 28,33,34,37-100

7.4.1 Concept Formation from Dynamic Trends Using PCA

The projections of the dynamic trends of some variables onto the PCA plane are
shown in Figures 7.10 (a) to (I). Only those variables that appear later in the
decision trees are shown in Figure 7.10. In Figure 7.1O(a), the regions A, C and D
are clearly distinguished. However, region B is fuzzy. This simply means that the
cases in region B (4-13, 17, 18 and 23-100) can not be distinguished in Figure
7.10(a). Their differences can only be identified by reference to other variables. A
similar explanation applies to Figure 7.1O(j) which classifies the responses of the
variable F_D202_in I into two classes for all data sets. This requires other variables
to discriminate the data sets. When all the variables are considered together, some
variables might be more important than others to the classification. For most of the
twenty one variables, the grouping based on visual examination of the two
dimensional PCA plane is straightforward. Even for a few variables, the groupings
may not be very clear, it does not affect the final result significantly. When grouping
for one variable is not clear, other variables will play more important roles in the
operational state clustering.
166 Data Mining and Knowledge Discovery for Process Monitoring and Control

6 0 . - - - -_ _ _ _ _ _ _ _ _ _---,
4 0 . - - - - -_ _ _ _ _ _ _ _ _ _----,
50 (Case 16)
C 30
40 (Cases4-13,17,18,23-100)
'-'-~
:,J..... _.............. ",.,...-;"
.
.....
B 20
~
.."....
(Cases4-24,28-100)
30 { ....
D
~ 20 ,'" \
~I
f
:=: , , 10 . . -r'
!-oliO'
I ',-.--":,~ ,:
\
.'. .... ,
..l,
N'
0 \ ( Ca 6)
... - . .
;,':',:--,
"'"\!'
~I
l' ' . . . __
"' __ "' , _• • • • ,
.!- ... ., \ ·f I ~I -10 (Case27 ),r"" . . .. .,
.,.,. ... ' 1
'. .:...1
-10
-20 (Case~ 14, 15)
J~.' -20
C
~
(Case~
E
1-3)
D (Case25 )
(Cases 1-3,19-22)
-30 -30

404~0~---_'2-0-----r----~2~0-----,41~)----~60 40~--._--~---r---r--_,---.~--4
-100 -80 -60 -40 -20 20 40
PC_CL_C201
(b)
30.--_ _ _ _ _ _ _ _ _ _ _ _- - ,
5 0 , - - - -_ _ _ _ _ _ _ _ _-----,

,"' \
20
...... - ............
.: .. ,,
30

,,,
".
(Case 31) (Ca~e32)
'.
I... _. .,B, . , ....
10

..... ,,
\

.
(Case29 ) (Case30) D " 10 \ \
~I ~ \'-.... (\~ .... = \ \
... ...
\

.....,, ,_......'
\. \
~I
..' ... ,
\
[J 0 • •• ~ I
~
~ .... ,
..~:
...... , ... I. , I "---) '--' -10 \
[J
NI -10
1
\~ ""--' ""--' '
\ ,~~
I;:: A
-30 ',-' ""A
.J (Cases 25-27)
,I
(Cases 1-28,33-1(0) f
0-
-20 B
-50 (Cases 1-24.28-100)

-301+----r-_r---.----._..-----._,---.------,--j -70 t-------.---.-.---r-.--r:----r-r---+

1

-30 -10 10 30 50 70 90 110 130 150 170 -UO -I]() -1)0 -70 -50 -30 -10 ]() 30 50
PC_LF_ClOI_ref
PC_CF_C20Louf
(e)
(d)
100.--_ _ _ _ _ _ _ _ _ _ _ _ _-----,
50r--------------,
80 40
(Cases 10,35,)6)
..,.F
(Case 9)
, ... -.. . -t
-
60
C
~l ..
':, ...,,,,~-:,. ~ 30
40 .... : ' (Case 16) 1
(Ca~ I)

.
Q
' 20
Case;; 12-15,17-34,
37-100) :
....
'i'
~

_"H (Case 6)
~ 20
...' (Ca C 20)
F
\"
,.,
~: o
~
,-t ':".. .
V 1.1
l'-'
1

~I 10 (C'ases4-18.23-IOO) \ (Case21)"

'-.. '- E,
(Cases 1-5,11 A (Ca:;,e 19) , D (Cases 2,),22)
....... __ .-. ::.;,; "._ ......G..;'-.... \ .~
-20 I. • '.(\ ... ,.... ''''\
B
(Case 8)
'--"""" \ •• ,
'" D ...... -;,.
(Case 7)
'." " ';" ':) \.:., ..'
40'+---.-~r------,--_.~_.--_r--_+ -10)+----.---_.----.---0:-----0::-----+
-40 -20 20 40 60 80 100 -10 10 20 30 40 50

(f)
Ie)
2001.---------------, 2001 r--------------,
150 150

=
J 100 100
(Case 1)
~

~: (Case" 1-3) (Cases 4-1(0)

C
.""'\
....,!
(Cases4-100)
50
50 B A t \
\,
A
}O, ,'-\ 2-3)\~"
'.,
(Ca'>es
,...., ~; ''I'':) C .... ..
--r-:-.... __
-50-!----__-,-____-,--____-,-____--,____ ---+ -50't-----,------,--_,-----r----+
-50 50 IOc) 150 200 -50 50 100 150 200

PC_CF_D20Cin PC_I_F_D201_out
(0) (II)
 Chapter 7 Inductive Learning and Conceptual Clustering 167

200,--_ _ _ _ _ _ _ _ _ _ _---.
140

120
100 ISO

80 ;;
~I
iii0, 60
100

... 40 ( Cases 2-3) (Cases 4- 100 ) (Case I)

~,
( Cases 1-5,7-100 )
(Cas.6)
N'
,,'.. ..
C 1 50 B
A B N A
20
.-.,(, I._
' ....,F' \--. u'
•.... ,
'J \ ..l
I .:
,~, 1.
\
1
I
-20 .'
40~~-~_,-,_-~_r-._~-~ -50~----'r--_,--~--...- - - t
~ ~ M ~ 60 W ~ rn ~ -50 50 100 ISO 200
PC_CL_D201

0) (j)
40,-_ _ _ _ _ _ _ _ _ _ _ _ _•
60 ,
I.' (Cases 1-13,17-100)
'.I'A
D

.,
30 ~

..
(Cases29-31)
20
B ( Cases 1-3,20-22,25-27 )
'\ D
20
(Case 16) .--
, ~·-·,t ,
,:, N
;;
,
\ , • I

:,," ~ .... , , .... -:~...,,\ ":- ...., '", '",

,., '----"""
'.-Ito
--- ,-~,'
• ~,. _,I • • I ,
' .... ::~ ....,
S
~I
-20
,",., 'iC
40 '0( (Case 14)

..,,'
't N' B
(Case 32) C -60 (Case 15)
-20 (Cases4-19,23,24,28,33-100)
-80
-30
-100
40~-r-~r--'-~--r--r-...--t -100 -&0 -60 40 -10 0 to 40
40 -30 -20 -10 10 20 30 40 PCI1_DZ02_ouI2
PC_I_T_C20Uop
0)
(k)

Figure 7.10 (Including preceding page) PCA two dimensional plots of variables for
MTBE,

300

250

200

150 , ,. ; '
I
...
.,
Cases 19-22

...::;;r:;s 100
, .'
I .,

-~:-
~
.... ...
50 , . . . . , . ... ...,... Cases 4,5

.....:.,. ... ,
: ••• •.:.,.t
, .. -....
·~
""
N
V ~~- ..
, .....
;---'
",'
Ilo -50 Cases 6-18. , __.....- -

.
23,28-100 I • '\
'\ Cases 1-3
,
-100
'\
.~
-150 "" Cases 25-27

-200
-150 -50 50 150 250 350

PC-l-State-MTBE

Figure 7.11 PCA two dimensional plot of operational states for the MTBE process.
168 Data Mining and Knowledge Discovery for Process Monitoring and Control

7.4.2 Identification of Operational States

The CSTR case study showed that PCA can classify the operational states
satisfactorily. Here, both PCA and the adaptive resonance theory (ART2, introduced
in Chapter 6) are used. The PCA analysis shown in Figure 7.11 results in five
clusters and the ART2 result is shown in Table 7.6 which predicts twelve clusters.
Both results are reasonable, but ART2 gives a more detailed picture and appears to
be more accurate. Sammon [123] indicated that since PCA is a linear method, it may
not give an adequate representation in two or three dimensions when the original
number of attributes is large, visual examination may not be possible. He also gave
an example where data generated giving five groups in four dimensions are
projected into the space of the two principal eigenvectors. Visual examination of
this projection shows only four groups, since two of the clusters overlap completely
in the two dimensional space. In analysis of process operational data, similar
observations have been made by other researchers [122, 119, 201]. In the following
discussion, only the ART2 clustering result will be used.
It is apparent that clusters with only one data set are correct. These are clusters
three, four and eight. Data sets 1, 2 and 3 in cluster one all cause the flowrate of the
C4 hydrocarbons feed to fall to zero and so should be in the same class. Data sets in
cluster two comprising 4, 5, 7, 8 and 11 cause the methanol flow to the mixer M201
to be either completely cut off or greatly reduced to cause them to be in the same
class. Data sets in class five, which include cases lO, 35 and 36 corresponding to
changes of the output of the controller FC202D from 33% to 59%, 33% to 50% and
33% to 55% respectively. Cluster six has cases 12, 13, 17 and 18 representing
reduction or cut in methanol flow to the tank D211 and column C20l. The two
cases in class seven, 14 and 15 represent the changes in the opening of the valve
HC211D from 18% to 40% and 60% respectively. Cases 19 to 22 in cluster nine
refer to changes of the output of the controller TC201R from 39% to 30%, 20%,
lO% and 0%. 23 to 27 are cases corresponding to changes in the output of the
controller FC203E from 37% to 33%, 29%, 16%, 6% and 0%, and are classified as
cluster ten. Cluster eleven has four cases, 29-32 corresponding to changes of the
output of the controller FC201D from 40% to 30%, 20%, lO% and 0%. The last
cluster, cluster twelve, has the normal operational data sets 37 to lOO. The
assignment of cases 28, 33 and 34 to this cluster is not apparent but is nevertheless
not unreasonable given that they represent insignificant changes.
 Chapter 7 Inductive Learning and Conceptual Clustering 169

7.4.3 Decision Tree Development

Conceptual clustering not only predicts the operational states but also interprets the
prediction using causal knowledge in tenns of decision trees or production rules.
Here, a variable takes discrete values from a region of the two dimensional PCA
plane of the variable. For example, the liquid level at the bottom of column C20l,
L_C201, takes values from its PCA plane in Figure 7.10(b) including A, B, C, D,
and E. For the data case number 24, L_C201 takes the value ofD. For each data set,
the operational state takes values from Table 7.6. For example, data set 24 has the
value of ABNlO.
The decision tree developed is shown in Figure 7.12. The decision tree can be
easily translated into rules. For example, the rules that lead to ABN4 and ABNll
are,

Rule7: IF T_MTBE = B in Figure 7.1O(a)

AND LD202_outl = C in Figure 7.1O(e)
THEN Operational state = ABN4
Rule II: IF T_MTBE=BinFigure7.10(a)
AND F_D202_outl = A in Figure 7.1 O( e)
AND T_C20 I_top = A or B in Figure 7.1 O(k)
THEN Operational state = ABNII

The root node is T_MTBE, the bottom temperature of the reactive distillation
column C201. It indicates that it is the most important variable that distinguishes
operational modes representing the one hundred data sets. Detailed examination of
the decision tree and all the dynamic responses in conjunction with the MTBE
process flowsheet revealed that the tree is reasonably good. An example illustrating
this is the rules leading to ABNII. From Table 7.6, it is known that ABNll covers
data cases 29, 30, 31 and 32, corresponding to changes of the output of the
controller FC20lC (reflux flowrate control) in manual mode from 40% to 30%,
40% to 20%, 40% to 10% and 40% to 0%. Clearly the most important variable
which discriminates between these data sets from others is the column top
temperature T_C20 I_top. This is continned by Figure 7.12, in which the nearest
node to ABNII is T_C20Ltop.
 170 Data Mining and Knowledge Discovery for Process Monitoring and Control

EJ
ABN7 ABN 5 ABN2 ABNII ABNIO NORMAL
4,12,13,17, /8
16 14,15 10,35,36 4,5,7,8.11 29-32 25-27 28,33.34.37-100

Figure 7.12 Decision tree developed for the MTBE process.

In Figure 7.12, the numbers at the bottom of the nodes indicates the data sets, For
instance, the node ABN8 has only one data set, namely 16, Comparing Figure 7,12
and Table 7.6, it is found that the decision tree in Figure 7.12 gives correct
predictions except for the nodes ABNlO and NORMAL in Figure 7.12.
Data sets 23 and 24 (underlined in Figure 7.12) are assigned to ABNlO by ART2,
as shown in Table 7.6, but to the node NORMAL in Figure 7.12 by CS.O. Data sets
23 and 24 represent the cases where the output of the steam flowrate controller
FC203E at the bottom of the reactive distillation column C201 is changed from 37%
to 33% and 37% to 29%. In fact these are insignificant changes so assigning them to
the NORMAL operational state is acceptable. This inconsistency with ART2 can be
attributed to two factors. Firstly ART2 is based on numerical values which is more
precise than the conceptual clustering using CS.O. Secondly, the conceptualisation
of variables by visual examination of the PCA two dimensional plane may give rise
to some inaccuracies.
Data sets 12, 13, 17 and 18 (in italics in Figure 7.12) are clustered in a separate
class by ART2, as shown in Table 7.6. They are mixed with other cases in the node
labelled NORMAL in Figure 7.12. Referring to Table Cl and Figure Cl, all the four
 Chapter 7 Inductive Learning and Conceptual Clustering 171

cases cause flowrate changes on the methanol stream to tank D211 and then to the
column C201. In reality, this flowrate is very small compared to the total methanol
and C4 hydrocarbons flows to the mixer M201, (about 1112 or 11100 respectively).
As a result, the changes of the methanol flow to D211 are insignificant.
Consequently it is reasonable to regard cases 12, 13, 17 and 18 as being in the
NORMAL operation class.

ye~

Figure 7.13 Decision tree developed if variables are not conceptualised.

In the above discussion, the dynamic trend signals of a variable are converted to
qualitative concepts in a PCA two dimensional plane and then the inductive learning
approach is used. An alternative way is using the eigenvalues of the first two PCs of
a variable directly. The resulting decision tree derived in this way is shown in Figure
7.13. The tree is found to be completely unreasonable because most clusters are
overlapped therefore it cannot be used for predictions. Figure 7.14 shows the
discretisation of two variables T_MTBE and F_D220_out2. The dashed lines are
the boundary values for the discretisatiom. Since for each variable the discretisation
is always binary, this is obviously not satisfactory. For example, Figure 7.1O(i)
shows that the variables L_D201 can clearly take three values. A two-valued
discretisation is not able to capture all the features which causes the inaccuracy of
the tree.
 172 Data Mining and Knowledge Discovery for Process Monitoring and Control

_
60r-----------------~------------------------------__.
50
40

OJ 30
al
.... 20
~I
..
10
....,
"',
U -10 ·c
0.
·20
·30
~o~ ____- -____- - - - - - - -________- -__- - - -__- - - -__- -__ --~

~o ·30 ·20 ·10 10 20 40 50 60

(a)

(b)

Figure 7.14 Binary discretisation of variables in C5.0.

7.5 General Review

Inductive learning has been introduced as a method for analysis of data records
averaged over days or weeks and a conceptual clustering tool for developing on-line
operational monitoring systems. It can learn from a large number of examples to
develop explicit and transparent knowledge in the form of decision trees and
production rules. It is also able to identify the most important variables that
contribute to clustering, which is clearly valuable for analysing process operational
data and process monitoring. There are several issues that need to be addressed.
Like most inductive learning systems, C5.0 is not recursive, it means it can only
deals with data as a batch, not be able to learn as an example is presented. In
addition though PCA has proved to be an effective way of concept extraction from
dynamic trend signals, it is expected that the combination of PCA and Wavelet will
deliver more effective pre-processing methods. Compared with similarity or
distance based methods which have been widely studied, conceptual clustering
clearly needs more research attention.
CHAPTERS
AUTOMATIC EXTRACTION OF KNOWLEDGE RULES
FROM PROCESS OPERATIONAL DATA

An alternative technique for developing intelligent process monitoring and control

systems is expert systems (ESs). ESs use logic rules to carry out heuristic reasoning.
Knowledge used to reach a conclusion is transparent because ESs have explanation
capabilities. In this respect, ESs are superior to neural networks because the
knowledge embedded in neural networks is opaque. It has long been recognised that
a critical issue in developing knowledge based ESs is the bottleneck of knowledge
acquisition. Traditionally knowledge acquisition has been dependent on consulting
domain experts. A disadvantage of this approach is that experts are usually better in
collecting and archiving cases than in expressing the experience and cases explicitly
into production rules. There are more difficulties in compiling knowledge about
process operations due to the large number of interacting variables.
A promising approach for acquiring heuristic knowledge rules is to extract
knowledge automatically from data. In this chapter we describe methods for this
purpose which include fuzzy sets, neural networks, rough sets and fuzzy neural
networks.

8.1 Rules Generation Using Fuzzy Set Operation

Generating fuzzy rules based on fuzzy set method developed by Wang and Mendel
[205-208] is concerned with the following context. Suppose we are given a set of
input-output data pairs,

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
 174 Data Mining and Knowledge Discovery for Process Monitoring and Control

(XII), X~I), ... , XWl) (y\ll, y~l), ... , yyl)

(xF), X~2), ... , X~» (yF), yi
2 ), .•• , y~l)

(8.1)
where XI. , X2 , ",XN are inputs and yJ, Y2, ...Yr are outputs. M is the total number of
given data patterns. The procedure for generating fuzzy rules is divided into five
steps which are explained based on a problem of two inputs and one output which is
a special case of Expression 8.1.
(XII), X~l») (y(l»)
(xI2), x~2») (y(2»)

(8.2)

Step I: Divide the input and output variables into fuzzy regions of fuzzy concepts.
Each region of a fuzzy concept is represented by a membership function. Figure
8.I(a) shows that the input variable Xl is divided into five regions of fuzzy concepts,
i.e., CE (Centre), BI (Big I), B2 (Big 2), Sl (Small 1) and S2 (Small 2). The shape
of the membership function, m(xl) is triangular. X2 is divided into seven regions and
Y into five regions.
Step 2: Generate fuzzy rules from given data pairs. For given xli), x~i)and y(i) find
corresponding membership values. For example, in Figure 8.1(a), since m(xj!) in B1
= 0.8, m(x\,l) in B2 = 0.2 and 0.8 > 0.2, we take m(x\I) = 0.8 and consider xP)
belonging to Bl. Following the same consideration, from Figure 8.I(b), we have
x~ll = 0.7 in Sl. Then we can obtain one rule from one pair of desired input-output

data, e.g.,
.: (xPl, x~ll) (y(l»)
=> [xPl(0.8 in BI), xill(0.7 in SI)] [y(l) (0.9 in CE)]
:. we generate rule 1 as
IF XI is B1 andx2 is Sl, THEN Y is CEo
Similarly,
.: (X\2l, xS2») (y(2))
=> [X\2 l (0.6 in B1), xS2l(1 in CE] [y(2) (0.7 in B1)]
:. we generate rule 2 as
IFxl isBl andx2is CE, THENyisBI.
 Chapter 8 Automatic Extraction of Knowledge Rules 175

The rules generated in this way are only "and" rules. However, if two rules
generated have the same THEN part then they represent "or" relation.

m(x l )

10 t l)<)()(N S2 SI CE 81 82

0.0 ~
Xl
+
X l~ x:21 X:ll Xl
(a)

1"1') S3 S2 SI CE 81 82 83

0.0 JXXXl>XxJ
X2 X(ll
2
X~2)
X;
(b)
~ X2

m(y)

10 t
S2 SI CE 81 82

0.0 ~ill y yl21 y+

~ y
(c)

Figure 8.1 Divisions of the input and output spaces into fuzzy regions
and the corresponding membership functions. (a) m(xI), (b) m(x;), (c) m(y).

Step 3: Assign a degree to each rule. Since there are normally lots of data pairs
and each data pair generates one rule, it is highly probable that there are conflicting
rules, i.e., rules have the same IF part but a different THEN part. This conflict is
solved by assigning a degree to each rule and accept only the rule from a conflict
group that has the maximum. The equation to calculate the degree for a rule is, for
example rule 1,
Degree of Rule 1 = mBI(xI)msl(x2)mcE(Y)m(1)
= 0.8*0.7*0.9* mOl
=0.504* mOl (8.3)
Where m O) is a degree given by human expert about the importance and trustiness
of the data pair.
Step 4: Create the combined fuzzy rule base. The combined fuzzy rule base is
created by filling the box of Figure 8.2 with the rule produced in the above steps. If
there is more than one in one box, use the rule with maximum degree.
176 Data Mining and Knowledge Discovery for Process Monitoring and Control

B3

B2

Bl

Sl

S2

S3

S2 Sl CE Bl B2
X,

Figure 8.2 The form of a fuzzy rule base.

The approach uses algorithms of fuzzy set and is simple and straightforward. The
rules generated by this method may be repeating and the set of rules may be large
[209]. Srinivasan et a1. [210] improved this method in this aspect using an inductive
learning to determine the fuzzy rules.
Other methods of generating fuzzy rules from numerical data using fuzzy sets
have slightly different procedures and are not discussed here [211-216].

8.2 Rules Generation from Neural Networks

The rules generation procedure from neural networks by Fu [217] can be illustrated
by reference to Figure 8.3.
A rule generated has the form of
IF the premise, THEN the action (conclusion).
Specifically for the case in Figure 8.3,
lFAi, ... Ai, ... -,Aj, ... -,Aj, ... , THENC(or-,C)
where Ai is a positive antecedent (an attribute in the positive form), -,Aj is a
negative antecedent (an attribute in the negative form), C the concept (conclusion),
and -, reads "not". Each node in the hidden or output layer is designated by a
symbol which represents a concept to be confirmed or disconfirmed, such as node C
in Figure 8.3. Confirmation (or disconfrrmation) of a node concept is measured by
the activation of the node which is calculated by

OJ = F(I;wjiXi -9 j ) (8.4)
 Chapter 8 Automatic Extraction of Knowledge Rules 177

where,
OJ - the activation of node j (a hidden or output node);
Wji - the weight on the connection from unit i to unit j;
OJ - the threshold on unit j, which is adjustable;
F - the activation function which represents the nonlinearity on the hidden or
output nodes and is defined as the sigmoid form.
1
F(a) = (8.5)

where A determines the steepness of the function.

In the case of a hard-limiting activation function, the output of a node is given by
o. = {I if t Wji Xi > e
j (activation> threshold) (8.6)
J 0 else
The method needs to train the neural network first and then based on the above
concepts a detailed procedure is developed to extract rules. Although it shows
advantages compared with some other methods of extracting rules (e.g., Gallant
[218]) from a trained neural network, the method still has its weakness: it produces
rules with concepts that have no physical meaning since it uses hidden neurones.

A- [ A-
j
I I

Figure 8.3 The post-attrs and neg-atts relative to a concept. An attribute can be
either a post-att or a neg-att, depending on the concept.
 178 Data Mining and Knowledge Discovery for Process Monitoring and Control

8.3 Rules Generation Using Rough Set Method

Learning production rules by rough set approach has been studied by Chan [219],
Srinivasan and Chan [210], Chmielewski et al. [220], Quafafou and Chan [209] and
Ziarko [226]. The approach makes use of the rough set theory introduced by Pawlak
[221, 222, 223]. Let U be a nonempty set called universe, and R be an equivalence
relation on U called indiscernibility relation. An ordered pair A = (U, R) is called an
approximation space. For each subset in U, X is characterised by a pair of sets, the
lower and upper approximation of X in A. The lower approximation of X in A is
defined as
BX = {x E V![xlR<;:;X} and
the upper approximation of X in A is defined as
RX= {XE UI[xlRnX;t0}
in which [xJR denotes the equivalence class of R containing x. A subset X of U is
said definable in A iff (if and only if) Jix= Fix.
The rules generating procedure based on the above concepts is better to be
illustrated by reference to an example as shown in Table 8.1.
The universe U is the set of example objects,
U= {el, e2, ... e8}
and each object is characterised by the attributes
Density, Colour, Boiling range and Class.
The attributes are classified as condition attributes and decision attributes. (The
production rule for a single example object is IF condition attributes THEN decision
attributes. For el as an example, IF Density = Low, Colour = Light and Boiling
range = Narrow, THEN Class = I). The automatic rules generating method is
dependent on the calculation of the lower and upper approximations of the class 1
and 2 (denoted asXl andX2) as well as the definability.
The concept "class = I" is the subset Xl = {el, e2, e4, e5, e7} and the concept
"class = 2" is the subsetX2 = {e3, e6, e8}.
These define the relations on U.
The relations on U can also be defined on the basis of single attributes, then we
have the following attributes on U induced by those equivalence relations on U.
{Density} * = {{el, e5, e8}, {e2, e3, e4, e6, e7}},
{Colour} * = {{el, e3, e7, e8}, {e2, e4, e5}, {e6}},
{Boiling range}*={ {el, e2, e7}, {e3, e4, e5, e6, e8}}, and
{Class}*= {{el, e2, e4, e5, e7}, {e3, e6, e8}}
 Chapter 8 Automatic Extraction of Knowledge Rules 179

The above partitions will also be called classification of U generated by single

attribute sets.

Table 8.1 A training set for the rough set approach represented
b)ya d · · tabl e.
eClSlOn
Attributes

Examples Density Colour Boiling range Class

el Low Light Narrow I

e2 High Very light Narrow I

e3 High Light Wide 2

e4 High Very light Wide I

e5 Low Very light Wide 1

e6 High Dark Wide 2

e7 High Light Narrow 1

e8 Low Light Wide 2

The lower approximations of the class Xl by each condition attribute are:

{Density }XI = 0
{Colour}XI = {e2, e4, e5}, and
{Boiling range }XI = {el, e2, e7} = U
The upper approximation of the class X 1 by each condition attribute are
{Density)XI = {el, e2, e3, e4, e5, e6, e7, eS} = U,
{COlOUr)XI = {el, e2, e3, e4, e5, e7, eS}, and
{Boiling range)XI = {el, e2, e3, e4, e5, e6, e7, eS} = U.
Given a decision table S, the concepts to be learned are subsets of the universe U
in S. One question is how do we know that a concept X can be described by
condition attributes in S. This is solved by the introduction of definability of sets.
Let P be a nonempty subset of the set of condition attributes in S, then a subset X of
U is said to be P-definable in S iff f..X = P X , and X is P-undefinable in S if
PX*PX.
Still consider the decision table in Table S.l. Let P be the set {Density, Colour}
of attributes, then the classification of U generated by P is
p* = {Density, Colour} * = {Density}*.{Colour}*
= {{el, eS}, {e3, e7}, {e5}, {e2, e4}, {e6}}.
The lower approximation of Xl by Pis
 180 Data Mining and Knowledge Discovery for Process Monitoring and Control

EXl = {el, e4, e5}, and

the upper approximation of Xl by P is
P Xl = {el, e2, e3, e4, e5, e7, e8}.
Therefore, the concept Xl is P-undefinable in S because E.Xl "" PXl
Let R = {Density, Colour, Boiling range}, then,
R*= {{el}, {e2}, {e3}, {e4}, {e5}, {e6}, {e7}, {e8}}, and
EXl = {el, e2, e4, e5, e7} =Xl = RX1.
Thus, the concept Xl is R-definable in S.
Using the concepts of lower and upper approximations and definability of sets,
programs LEM2 [219] and LERS [220] have been developed for extracting
production rules from data which are able to deal with noise.

8.4 A Fuzzy Neural Network Method for Rules Extraction

We have developed a fuzzy neural network method for generating production rules
from data [174]. A conventional NN has real number inputs and weights. There are
three main types of fuzzy neural networks (fuzzy NNs) [156]: fuzzy NNs with fuzzy
input signals but real number weights, fuzzy NNs with real number input signals but
fuzzy weights and fuzzy NNs with both fuzzy input signals and fuzzy weights. The
first type is used here. The method involves the following steps:
(1) The input and output variables are divided into fuzzy regions. Figure 8.4
shows examples of such regions, such as (Normal, High and Low) or extended to
five values (Normal, Medium High, High, Medium Low and Low).
(2) All input and output data patterns are processed using the membership
functions in step (1). For example, value of 600°C for a temperature may be
regarded as High ( I-lH = 0.7) when expressed as a fuzzy variable.
(3) A fuzzy NN is constructed. To illustrate the fuzzy NN structure, consider a
fuzzy NN with two input variables (T Cl and FCI ) and a single output variable (T Hll),
as shown in Figure 8.5. The box contains a normal NN which has one input and one
output layer together with a simple hidden layer. The activation functions of the
neurons for both the hidden and output layers are sigmoidal functions. Thus, if the
normalised value for TCl is 0.7 and it has the fuzzy membership function of the type
shown in Figure 8.4( c), then the outputs of the three )l neurons corresponding to T Cl
would be
 Chapter 8 Automatic Extraction of Knowledge Rules 181

Il L, Tel
=0 Il N, Tn
=0 II
rH, Tel
=1

If the nonnalised value for Tel is 0,7 and it has the fuzzy membership function of
Figure 8.4(d), then the outputs of the three I-l neurons corresponding to Tel would
be

Il L, Tel = °Il N, TCi

= 0 11 H, TCi = 0,8

"
.~
u
: : lL-_L_---L__ M_L_.L-_N_---L_M_H_...l-_H_-L-_~
-=:"
(a)
c. 1.0

~~
"O,O~
..0
E (b)
"
E 1.0
»
N
N

::>
u...
0.0 1 I I
L N H

(c)

t ~
1.0
L H
0.0
0,7 (d)
Normalized values of process variables

Figure 8.4 Divisions of input and output variables into fuzzy regions and the
corresponding membership functions.

(4) Use the data produced in Steps 2 and 3 to train the fuzzy neural network
shown in Figure 8.5, The parts outside the dashed-line box are the fuzzy
representation of input and output data. The part inside the dashed-line box is a
nonnal three layer back propagation neural network. The training procedure uses
error backpropagation which adjusts the weights between the input and output
layers,
The fuzzy rules obtained by training the network are of two types. One is of the
following fonn:
IF Tel is High AND FCI is Low
THEN THlI is high (8.7)
182 Data Mining and Knowledge Discovery for Process Monitoring and Control

This means that if the value of a variable belongs to a fuzzy class it belongs to it
unambiguously. Another type of rule which is obtained from training the network is
of the following form:
IF TCl is High ( !..l H, T CJ ) AND FCl is Low (J.l L. F CJ )

THEN THll is high (!..l H, T HI!) (8.8)

The next section examines each of these rules.

8.4.1 Generating Rules Without Fuzzy Membership Values

If the membership functions of the variables of a fuzzy NN have the form of Figure
8.4(a) or (c), then the rules generated take the form of Expression 8.7, i.e., rules
without fuzzy membership values. The maximum number of rules is determined by
the network topology. If there are NOY input variables and each variable takes NFY
fuzzy values (i.e., each input variable has NFY corresponding nodes, assuming all
input variables take the same number of fuzzy values), then the maximum number of
rules will be equal to NOyNFV. This represents the maximum number of rules
although it is likely that in practise it will be less since it depends on the range of
data used to generate rules.

--
--
--

Figure 8.5 The architecture of the fuzzy neural network.

 Chapter 8 Automatic Extraction of Knowledge Rules 183

For the fuzzy NN structure shown in Figure 8.5 having two input variables and
one output variable with each variable taking three fuzzy values, the maximum
number of rules will be 32=9. If each variable takes five values (High, Medium
High, Normal, Medium Low, and Low), then the maximum number of rules is
52=25;
In the case of a shell-and-tube heat exchanger of the type illustrated in Figure
8.6(a) [224]. The steady state values are shown in Figure 8.6(a). Assuming no leaks,
then the cause-effect diagram for variables is shown in Figure 8.6(b). For simplicity,
attention is restricted to fixed values of T HI and FHI so only Fel and TCI change.
Then if each input variable takes three fuzzy values, i.e., Normal, High and Low, the
situation is as depicted by Figure 8.4(c). A fuzzy NN structure for the heat
exchanger is shown in Figure 8.5 and there will be a maximum number of 32 =9
rules.
The boundary values of the variables for low, normal and high are indicated in
Table 8.2. 87 input-output patterns are obtained by changing FCI and/or Tel based
on Table 8.3. Details of simulation results are not given to save space but the
resulting data patterns (as shown in Table 8.4) are given which are obtained after
fuzzification of simulation results using the fuzzy membership function of Figure
8.4(c) and the boundary values of Table 8.2. Each pattern in Table 8.4 is a different
form of a production rule, for example, data pattern I represents the following rule:

IF Fel is Normal and Tel is Normal

THEN T HII is Normal

If there are no conflicts in any of the data patterns , such as in the case of data
patterns I, 2, and 3 which give the same rule then Table 8.4 is the final result.
However, more often than not, there are conflicts in data patterns, i.e., rules have the
same IF part but different THEN part. Thus data pattern 7 and 7* are examples
which take the following form:

7: IF Fel is High and TCI is Normal

THEN T HII is Low
7*: IF Fel is High and Tel is Normal
THEN T HII is Normal
 184 Data Mining and Knowledge Discovery for Process Monitoring and Control

Here, 7 and 7* have the same IF part but different THEN part, the conflict has to
be resolved. The approach being proposed here is to use the neural networks to train
the data patterns in Table 8.4. Since neural networks are capable of dealing with
noise, the noise data patterns can be identified and neglected. To demonstrate this,
Table 8.4 contains 10 data patterns having noise which illustrate the conflicts. The
10 data patterns are shown with a "*,, e.g., 7*. The result is an increase in the
number of data patterns from 87 to 97.
The 97 data patterns are fed to a back propagation neural network (Figure 8.5)
having 6 inputs (FCl Low, Fel Normal, Fel High; Tel Low, Tel Normal and TCl
High) and 3 outputs (THII Low, THII Normal and TH11 High) with 6 hidden neurons.
The activation functions of output and hidden layers are sigmoidal. The trained
results after 5000 iterations are shown in Table 8.5.
Table 8.5 gives the targets and predictions of T HII for all the 97 data patterns of
Table 8.4. For example, for data pattern I, the target values for T HII are L = 0, N =
I and H = 0 and the predictions for T HII are L =0.1, N = 0.85 and H = 0.06.
It was found that for all 87 data patterns obtained from simulation, very good
predictions are obtained, but for all 10 of the noisy data the errors are very large.
The conflict can be resolved by comparing errors between the target values and
predictions, keeping the rules with small errors and abandoning those with large
errors.
The conflicting data patterns 7 and 7* can be used as an example. By comparing
the results in Table 8.5, it can be seen that the error for 7* is very large and for 7 is
small. The square root of the sum of error squares for data patterns 7 and 7* are
0.26 and 1.15 respectively. So 7* should be rejected. The same conclusion can be
reached by looking at the other 9 noise data patterns. The rules generated have been
summarised in Table 8.6.

Table 8.2 The boundary values of low, normal and high for T HI ], Fel and Tel.
L(Low) N (Normal) H(High)

THII OF < 285 [285,320] >320

FCI lh/sec <0.627*Fcl. "eady [0.627*Fcl 'teadv , 1.42* FCI steadY 1 >1.42* FCI steady

TCI OF <0.55*Tcl "eadv rO.55*Tcl '!ead ,1.6* TCI steadyl > 1.6* TCI. steadv
 Chapter 8 Automatic Extraction ofKnowledge Rules 185

Stream Hot-I
1 Stream cold - I
TCflOooF
FC f 20000 Ibisec

TH 1= 400 of
FH 1=20000 Ibisec Stream Hot - II
•

! Stream cold -I I
TClf 200 of

(a)

(b)

Figure 8.6 The shell-and-tube heat exchanger and its cause-effect diagram_

. . db'yc h angmg FCI and/ or TCI-

Ta ble 83 87 data patterns are 0 btame
Data patterns Coefficient (Fcl =FC\* Coefficient (T CI =T CI *
Coefficient) Coefficient)
(1) - (5) LO, U, 12, 1.3, 1.4 LO
(6) - (9) 1.45, 150, L60, L65 LO
(10) - (13) 0.9,0.8,0.7,0.65 LO
(14) - (16) 0.6, 0.5, 0.45 LO
(17) - (20) LO L2, 1.3, 1.4, L5
(21) - (24) LO L65,L7, L8,L9
(25) - (27) LO 0.8,0.7,0.6
(28) - (31) LO 0.5, 0.4, 0.35, 0.3
(32) - (47) 1.45, 1.5, 1.6, 1.65 1.65, 1.7, 1.8, 1.9
(48) - (59) 0.6,0.5, 0.45 1.65, 1.7, 1.8, 1.9
(60) - (75) 1.45, 1.5, 1.6, 1.65 0.5, 0.4, 0.35, 0.3
(76) - (87) 0.6, 0.5, 0.45 0.5, 0.4, 0.35, 0.3
186 Data Mining and Knowledge Discovery for Process Monitoring and Control

Table 8.4 Fuzzification results of data Table 8.5 Target and predicted fuzzy
patterns obtained through simulation using membership values for THII.
the membership function of Figure 8.4( c).
FCI TCI THII FCI TCI THII
No. L NH L NH L NH No. (T, P) (T, P) (T, P)

I 0 I 0 0 I 0 0 I 0 I 0 .10 I .85 0 .06

2 0 I 0 0 I 0 0 I 0 2 0 .10 I .85 0 .06
3 0 I 0 0 I 0 0 I 0 3 0 .10 I .85 0 .06
3" 0 I 0 0 I 0 I 0 0 3* I .10 0 .85 0 .06
7 0 0 I 0 I 0 I 0 0 7 I .82 0 .19 0 .01
7" 0 0 I 0 I 0 0 I 0 7* 0 .82 I .19 0 .01
II 0 I 0 0 I 0 0 I 0 II 0 .10 I .85 0 .06
11* 0 I 0 0 I 0 0 0 I 11* 0 .10 0 .85 I .06
18 0 I 0 0 I 0 0 I 0 18 0 .10 I .85 0 .06
18" 0 I 0 0 I 0 I 0 0 18" I .10 0 .85 0 .06
22 0 I 0 0 0 I 0 0 I 22 0 .20 0 .02 I .81
22* 0 I 0 0 0 I I 0 0 22* I .20 0 .02 0 .81
29 0 I 0 I 0 0 I 0 0 29 I .80 0 .03 0 .06
29* 0 I 0 I 0 0 0 0 I 29" 0 .81 0 .03 I .06
40 0 0 I 0 0 I 0 I 0 40 0 .06 I .95 0 .03
40* 0 0 I 0 0 I I 0 0 40" I .06 0 .95 0 .03
50 I 0 0 0 0 I 0 0 I 50 0 .07 0 .01 I .94
50* I 0 0 0 0 I I 0 0 50" I .07 0 .01 0 .94
70 0 0 I I 0 0 I 0 0 70 I .94 0 .05 0 .01
70* 0 0 I I 0 0 0 I 0 70* 0 .94 I .05 0 .01
80 I 0 0 I 0 0 0 I 0 80 .0 .01 I .78 0 .24
80* I 0 0 I 0 0 0 0 I 80" .0 .01 0 .78 I .24
81 I 0 0 I 0 0 0 I 0 81 .0 .01 I .77 0 .24

Table 8.6 Rules generated.

IF FCI and TCI THEN THll
Nonnal Nonnal Nonnal
High Nonnal Low
Low Nonnal High
Nonnal High High
Nonnal Low Low
High High Nonnal
Low High High
High Low Low
Low Low Nonnal
 Chapter 8 Automatic Extraction of Knowledge Rules 187

8.4.2 Generating Rules With Fuzzy Membership Values

If fuzzy membership functions take the fonn shown in Figure 8.4(b) or (d), then
rules of the fonn of Expression 8.8 are obtained. Translating all the data into rules
of the fonn of Expression 8.8, there are as many rules as the number of data
patterns. Therefore an important step is to deal with number of rules. Two
approaches are proposed and described next.

8.4.2.1 A-Cut approach

One way to reduce the number of rules is to reject the rules that are less reliable.
There are two situations which make rules less reliable. One is that in Expression
8.8, IlH. Tel, ~L, F cl , and ~H,THll are small because small membership values

mean it is ambiguous in identifying which fuzzy values should be used. The other
situation is that the error, ER, for the data pattern is large when training is
completed. It is therefore necessary to define a confidence factor, CF, representing
reliability:
CF = IlH. TCl Ill. Fel IlH. TCHll / ER (8.9)
A A. -Cut value can then be defined as the threshold for the rule to be worth
keeping. If the confidence factor of the rule is smaller than the A -Cut value then the
rule should be abandoned. Adjusting the A -Cut value can change the number of
rules generated.
Referring to the heat exchanger the simulation results of the heat exchanger shown
in Table 8.3 are first converted to fuzzy patterns using corresponding fuzzy
membership functions. The function used in this case is represented in Figure 8.4(b)
and (d) and described by the following expression:
y = e- 1xl ' (8.10)

Here, x changes in the range [-2, 2] and I changes from 0.51 to 5.0. In this example,
1= 1.0. The boundary values for Low, Nonnal and High for the three variables (Fel'
Tel and THII) are shown in Table 8.7. There is a little difference from Table 8.2 but
it does not affect the principle to be demonstrated.
By applying the fuzzy membership functions in Equation 8.10 to the 87
simulation data patterns corresponding to Table 8.3, data patterns can be converted
to fuzzy rules of the following fonn,
IF Fel is Nonnal (0.61) and Tel is Nonnal (1.0)
THEN T HII is Nonnal (0.67).
188 Data Mining and Knowledge Discovery for Process Monitoring and Control

All the 87 data patterns after fuzzification are input to the neural network having 6
hidden neurons, as shown in Figure 8.5. The network has six input neurons (FCI
Low, Normal and High as well as T CI Low, Normal and High) and three outputs
(T HII Low, Normal and High).
After training, the confidence factors CF are shown in Table 8.8. A large value of
CF means that it is more reliable. So a A-cut value is used to select the desired
number of rules. By changing the A -cut values it is possible to change the number
of rules, as is demonstrated in Table 8.9. For example, with a A -cut value xlO is
6.0, there are 17 rules and for 5.0 are 21 rules. The rules for the former case are
shown in Table 8.10.
The A -cut approach can effectively select the desired number of rules which then
can be used by a fuzzy expert system shell to develop an expert system. In
applications, when the input data does not exactly match the IF parts, interpolation
and extrapolation are required.

Table 8.7 Boundary values the three variables or Low, Norma an dH"19h.
L N H
THII <280 r280, 320] >320
FCI 12000 [12000,28000] >28000
TCI <40 r40, 1601 >160

Table 8.8 Confid 1 data patterns.

1 ence actors or some example

Data patterns 1 2 3 4 5 ... 48 49 50

Confidence 24.01 32.23 3.07 .89 .41 ... .21 .35 .79
Factor x 10

51 52 53 54 55 56 57 58 59 ...

3.28 .59 1.54 7.99 9.82 .34 4.20 5.03 ...

Table 8 9 Number of rules generated can change with changing A - cut values
A -Cut value x 10 1.0 1.5 2.0 3.0 5.0 6.0 8.0 9.0 10.
Number of rules 48 42 32 25 21 17 11 8 6
 Chapter 8 Automatic Extraction of Knowledge Rules 189

Table 8.10 Fuzzy rules generated (A -cut value x 10 = 6..

)
No. IF FCI and TCI THEN THII
(1) (N, 1.00) (N, 1.00) (N,0.99)
(2) (10) (N,0.61) (N, 1.00) (N,0.67)
(17)(25) (N, 1.00) (N,0.51) (N,0.51)
(18)(26) (N,1.00) (N,0.37) (N,0.37)
(40) (H,0.67) (H,0.19) (N,0.55)
(47) (H, 1.00) (H, 1.00) (N,0.22)
(54) (L, 0.51) (H,0.51) (H,0.82)
(55) (L, 0.51) (H, 1.00) (H,0.96)
(57) (L, 1.00) (H,0.26) (H,0.87)
(70) (H, 0.67) (L, 0.37) (L, 0.79)
(71) (H, 0.67) (L, 1.00) (L, 0.90)
(74) (H, 1.00) (L, 0.37) (L, 0.87)
(75) (H, 1.00) (L, 1.00) (L, 1.00)
(83) (L, 0.51) (L, 1.00) (N,0.28)

8.4.2.2 Neuro-expert approach

Although the A-cut can effectively select the desired number of rules, the rules
obtained do not keep all the infonnation inherent in the original data. Some
infonnation is lost. The neuro-expert system approach is different in that it can
make use of all the infonnation in the data. It requires writing the basic number of
rules (i.e., NOVNFv). For two inputs, where each variable takes three values, the
basic number of rules are 32 =9. These rules are of the fonn of expression (8.11) in
which IlH , Xl ' ilL X2 and IlMH ' yare variables that can be any values in the range
I

of [0, 1].
IF Xl is High ( Il H• Xl) AND X2 is Low ( ilL. X2)
THEN y is medium high ( Il MH. y ) (8.11)

When an expert system is used, it receives fuzzy values of Xl and X2 and the
corresponding Il values. The expert system gives the fuzzy values for y and uses a
trained neural network algorithm (based on the weights) to calculate the Il for y.
In conclusion, it is essentially the same as the neural network but it has the
following important features. Firstly, it is able to give explanations about how
conclusions are reached. In the case of the heat exchanger, it can also give the
following explanations:
 190 Data Mining and Knowledge Discovery for Process Monitoring and Control

Conclusion: THII is (Normal, 0.24), CF xlO is 2.95.

HOW (user input asking how conclusion was reached.)
IF FCI is (Low, 0.51) and TCI is (Low, 0.37)
THEN T Hil is (Normal, 0.24)
Confidence factor xlO = 2.95

Secondly, the basic rules are represented as cause-effect knowledge in contrast to

the black box feature of a neural network.

8.S Discussion

Both neural networks and expert systems are now starting to realise their potential in
developing intelligent operational decision support systems. Knowledge used by an
expert system to reach conclusions is transparent and can be displayed through the
HOW and WHY explanation facilities. However, a critical problem with expert
systems is knowledge acquisition. The methods introduced in this Chapter are able
to extract production rules automatically from data therefore are potentially very
useful.
The fuzzy neural network method is similar to the method of rule generation by
fuzzy set operation [205, 207] in the first two steps. The major difference is in
dealing with conflicts. The fuzzy set operation method calculates the extent to which
a rule depends on a factor derived from human experts. This is not practical in many
applications and so efficiency can be a serious problem when large amounts of data
are involved. The fuzzy NN described in this chapter is not an incremental
approach. Frayman and Wang [225] developed a recurrent fuzzy NN which is able
to learn incrementally.
The rough set method [219] initially was not able to generate fuzzy rules.
Quafafou and Chan [209] have improved the method so as to be able to generate
fuzzy rules but there are still limitations, as indicated by Quafafou and Chan [209].
The neural network method by Fu [217] depends on the explanation of a properly
trained structured neural network. The nodes represent concepts and the branches
describe cause-effect relations. However, since hidden neurons are used, the
knowledge obtained includes concepts which do not have physical significance.
Another problem is that the number of rules increases dramatically with increase in
the size of the problem.
 Chapter 8 Automatic Extraction of Knowledge Rules 191

Chan [219] has classified the various methods into two categories: incremental
and non-incremental learning. The former works on one example at one time and
the latter on a mass of training examples. The methods of the fuzzy set operation
and the rough set belong to the incremental method group. The neural network
method by Fu [217] is non-incremental for generating the trained neural network but
becomes incremental when the weights are explained. The method proposed in this
paper belongs to incremental. A disadvantage is that some information contained in
the numerical data may be lost during conflict filtering, which is an essential step in
incremental methods. This is unlike the NN which takes account of all data used for
training to obtain the weight. However, incremental learning has its advantage that it
is a recursive method, so that as new data becomes available, it updates the system
and changes the existing knowledge base. This is particularly helpful in on-line
application, as new data may be continuously made available for updating.
Another very effective technique for automatic generation of rules from data is
inductive learning which has been introduced in Chapter 7 therefore is not repeated
here.
CHAPTER 9
INFERENTIAL MODELS AND SOFTWARE SENSORS

9.1 Feedforward Neural Networks as Software Sensors

Process control contributes to process operational perfonnance in safety,

environmental protection and product quality through proper monitoring and control
of process variables. Some of the variables including flowrate, temperatures,
pressures and levels can be easily monitored on-line with cost effective and reliable
measuring devices. Some variables, however, are often analysed off-line in
laboratories because they are either too expensive or technically unreliable to install
on-line instrument. These variables often relate to product quality and
environmentally significant parameters such as composition and physical properties.
A significant delay is nonnally incurred in laboratory testing, which is in the range
of half an hour to a few hours. Such a delay makes it too late to make timely
adjustments. An inferential or software sensor can predict the primary output
variable from process inputs and other measured variables without incurring the
measurement delay. Such prediction is necessary for implementation of inferential
control.
Software sensors assume that there is a relationship between some easily
measured and continuously available variables and the variable to be inferred.
Traditionally mathematical models have been used which are obtained from
fundamental domain knowledge. However, for many chemical and biochemical
processes such fundamental models are not available due to lack of knowledge
about these processes. A promising approach is to develop systems that can learn to
develop inferential models from operational data. A major advantage of machine

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
 194 Data Mining and Knowledge Discovery for Process Monitoring and Control

learning based approaches is that the model is able to be continuously improved as

more and more data become available. Among the various approaches studied,
feedforward neural networks (FFNN) have attracted wide attention and shown great
potential as a basis for reliable software sensor design.
This chapter describes inferential models and software sensor design using
FFNNs, with emphasis being put on addressing a number of critical issues. Firstly,
FFNN is recognised to have the danger of extrapolation beyond the parameter space
used for the training data. It is therefore important to select the data for model
development and test with some care. This also means that there is a need to know
when the FFNN model needs to be retrained with new data during use. In Section
9.2, a method for addressing the extrapolation issue is introduced. An industrial case
study describing the development of a software sensor using the approach is
presented in Section 9.3. Secondly, how to select input variables is also an
important issue. Because of inadequate knowledge of domain problem, people tend
to use more variables than necessary while some may be redundant. Redundant
variables deteriorate the performance. Methods are introduced in Section 9.4 to
select input variables. Finally, Section 9.5 gives a brief introduction to dynamic
neural networks for software sensor development.

9.2 A Method for Selection of Training / Test Data and

Model Retraining

A limitation of FFNN models is that they are data intensive because they are trained
and not programmed [145]. Therefore when such a model is being developed, it is
important to divide the data into training and test sets so that the model developed
can be validated. In addition, the trained model needs to be periodically updated to
maintain the accuracy of the model. The traditional way of dividing a database into
training and test sets is through random sampling. Random sampling of test data has
major shortcomings. If more sampled patterns belong to the sparse region of the
high dimensional parameter space, the results will not be reliable, because the
training patterns from that region are not adequate. On the other hand if more
sampled patterns for testing come from the dense space, the model will give over
optimistic estimation. Another problem is to decide when the model needs to be
retrained. It is generally assumed that if the values of all input variables are within
the boundaries of the training data, the model is reliable, but this assumption is
questionable. Given a new data pattern, with input attributes within the boundaries
 Chapter 9 Inferential Models and Software Sensors 195

of the training data, it is still difficult to tell if it is covered by the training data since
the new data represents a point in a very high dimensional non-linear parameter
space.
To address this issue, we have proposed a method which combines unsupervised
clustering approach and FFNN, using unsupervised clustering to automatically
cluster the data into classes so that data patterns within a class can be distinguished
from those in other classes and so form a basis for sampling test data patterns [227].
Application of the method to the development of a software sensor for the main
fractionator of a refinery fluid catalytic cracking process is described to illustrate the
approach.
Figure 9.1 schematically depicts the method. It is divided into two stages, model
development and model maintenance. In model development, data are first grouped
into classes using the unsupervised clustering system AutoClass, a Bayesian
clustering approach that has been introduced in Chapter 6 and can automatically
group multivariate data patterns into clusters. Test patterns for FFNN are sampled
from each class according to the size of the class. During model maintenance, it is
necessary to decide when the model requires to be retrained. Because of the high
dimensionality and large volume of data, it is difficult to carry out the analysis
manually. The approach used here mixes new data with older data and then analyses
it using AutoClass. If any of the following three situations arises, the model needs to
be retrained.
• New classes are formed that cover mainly new data. This means that the new
data are located in new parameter spaces.
• Some new data are assigned to small sized existing classes. This implies that the
parameter spaces covered by small sized classes are insufficiently trained. When
new data are available, the model should be retrained to improve its performance.
• New data are assigned to large classes and the degree of confidence estimated
using the old model for the new patterns is low.
Other unsupervised machine learning approaches introduced in Chapter 6 could
also be used. AutoClass is used here because its degree of autonomy is high. For
example, users are not required to give a threshold for the similarity measure. The
algorithm was discussed in detail in Chapter 6 so will not be repeated here.
 196 Data Mining and Knowledge Discovery for Process Monitoring and Control

Model development

Model maintenance

Figure 9.1 A combined framework for software sensor design

using automatic classification and FFNN.

9.3 An Industrial Case Study

This section describes the application of the above method to an industrial process
for developing a software sensor predicting product quality.

9.3.1 The Main Fractionator and Product Quality Prediction

The fluid catalytic cracking process of the refinery has been described in Chapter
4 (Figure 4.12). One of the products is light diesel which is typically characterised
by temperature of condensation. Previously the temperature of condensation was
monitored by off-line laboratory analysis. The sampling interval is between four to
six hours and it is not practical to sample it more frequently since the procedure is
time consuming. This deficiency of off-line analysis is obvious and the time delay is
a cause of concern because control action is delayed. Moreover, laboratory analysis
requires sophisticated equipment facilities and analytical technicians. Clearly there
is great scope for a software sensor to carry out such on-line monitoring.
Fourteen variables are used as inputs to the FFNN model as shown in Figure 9.2.
These variables are also summarised in Table 4.1.
 Chapter 9 Inferential Models and Software Sensors 197

Initially 146 data patterns (Dataset-I) from the refinery have been used to develop
the model (Model-I). Later, three different sets of data became available having
data patterns of 84 (Dataset-2), 48 (Dataset-3) and 25 (Dataset-4) respectively.
Model-I has therefore been modified three times with the versions being denoted by
Model-2, Model-3 and Model-4. There are interesting and different results for every
model improvement, as described below. In all cases, the accuracy of the model is
required to be within ±2°C.

Input layer Output layer

~
Tl-Il
T1-l2
T1-33
TI-42
T1-20
F2l5 Condensation
T1-09 ---=::::::~K:::>+ point of light
TO-OO diesel
F205
F204
FlOI
FR-I
FIQ22
F207

Figure 9.2 The feedforward neural network structure.

9.3.2 Model Development (Model-I) Using Dataset-l

The data set (Dataset-I) used for model development comprises 146 patterns. As
illustrated in Figure 9.1, the first step is to process the data using AutoClass. It
requires selecting a density distribution model for all attributes (i.e., all the inputs to
the FFNN, Figure 9.2). All variables have real values and no data are missing so a
Gaussian model is used.
AutoClass predicts seven classes (numbered 0, 1, ... , 6) as shown in Table 9.1.
°
For example, class has 32 members (i.e., 32 data patterns). Test data are sampled
from each class and they are indicated in bold and underlined in Table 9.1. More
data patterns are sampled from larger classes and fewer from smaller classes.
 198 Data Mining and Knowledge Discovery for Process Monitoring and Control

Altogether there are 30 data patterns used for testing and 116 for training. The
procedure is summarised in Table 9.2.
The assignment of a data pattern to a class is fuzzy in the sense that there is a
membership probability. Examples of membership probabilities are shown in Table
9.3. For instance, data pattern 17 (the third row in Table 9.3) has a membership
probability of 0.914 to class 0,0.055 to class 1 and 0.030 to class 4. It therefore is
assigned to class O.
The FFNN software sensor model (Model -1) is obtained when the training error
reaches 0.424. With normalized [0, 1] training data, the error is calculated using
1~,--. \2 '
- L.i \Vi - Yi J ' where Y; is the prediction by the model for the ith training
2 ;=1
pattern and Yi the target value. There are three training patterns and two test
patterns with absolute errors exceeding the required ±2oC. Therefore the degree of
confidence for training data is 100% - (3/116)% =97.4%, and that for test data is
100% - (2/30)% = 93.3%. A degree of confidence of 90% is considered acceptable
by the refmery.

Table 9.1 Classification of Dataset-l and test data selection for FFNNa•
Class 0 Weir:.ht 32 Class 1 Weir:.ht 31 Class 2 Weir:.ht 29
5 ~ 16 1731 32 3435 36 1 10 25 26 27 44 45 46 2 J 4 15 18 19 64 65 69
37-39 40 41-43 81 103 104 47-50 51 52 76 79 80 70 72 73 74 75 83 84
105 108 109 110 112 114 82 120 121 122-24 125 85 86-88 89 90 91 93 94
115 116 118 136 137 138 139 126 128 129 132 133 95 96 97 146
134 135
Class 3 Weir:.ht 26 'lass 4 Weir:.ht II Class 5 Weir:.ht 9 Class 6 Weir:.ht 8
11 12 13 14 53 54 55 7 9 28 29 33 20 21 30 101 102 8 22 23 24 107
56-59 60 61-63 66 67 68 III 117 127 130 106 140 141 142 113 119 144
71 77 78 92 98 99 100 131 145
143
a data in bold and underlined are test data; weight - number of members

Table 9.2 Test and training data from the 146 patterns in Dataset-I.
Classes 0 1 2 3 4 5 6
Class weight 32 31 29 26 11 9 8
Number of patterns for training 25 24 23 21 9 7 7
Number of patterns for test 7 7 6 5 2 2 1
 Chapter 9 Inferential Models and Software Sensors 199

Table 9.3 Examples of class membership probability.

Data Pattern (Class, Membership (Class, Membership (Class, Membership
probability ) probability) probability )
16 (0,0.803) (1,0.197)
17 (0,0.914) (1,0.055) (4,0.030)
31 (0,0.984) (1,0.014) (6,0.002)
32 (0, 1.000)
34 (0,0.995) (6,0.005)
35 (0, 1.000)
36 (0,0.967) (1,0.003) (6,0.001)
37 (0,0.996) (1,0.002)

9.3.3 Model-l Improvement Using Dataset-2

The production strategy changes according to season. The accuracy of Model-l

originally developed was later found to be inadequate and 84 more data patterns
(Dataset-2) were provioded in order to improve model performance. Initial
application of Model-l to the 84 data patterns indicated that the degree of
confidence is only 100% - (63/84)% = 25.0%.

Table 9.4 Classification result of dataset-l plus dataset-2a.

Class 0 Class 1 Class 1 Class 3 Class 4
Weif:.ht 38 Weight 31 Weight 16 Weight 19 Weight 17
2 J 4 14 15 18 5 6 17 29 32 160 161 175 I 10 16 44 II 12 13 25
19 21 64 65 66- 35 36-39 40 41- 177-179 182 186 45-48 49 56 57 58-60
69 70 71-75 82- 43 81 103 104 187 191 193-195 50-52 53 54 61 62 63 77
84 86 87 88-90 105 108 109 196 197 209 55 79 80 78 85 92
91 93 94 95 96- ill III 112 210 111 212 121 122 143
98 99 100 146 114 115 116 215 116 217-2/9
117 136 137 110 221
138 139
Class 5 Class 6 Class 7 Class 8 Class 9
Weight 15 Weight 14 Weight 12 Weight 11 Weight 10
157 176 180 181 7 8 22 23 24 31 162 163 164 158 159 198 147 148 149
183 184 185 188 33 34 76 107 113 222 223124 199 200-202 150-152 153
189 190 191 205 118 119 144 225-227 118 103 204 213 154-156
106 207 208 229 230 214
Class 10 Class 11 Class 11 Class 13 Class 14
Weight 10 Weight 9 Weight 8 Weight 7 Weight 3
9 26 28 120 123 27 125 126 128 20 30 101 166 167 165 173 174
124 127 130 ill 129 132 133 134 102 106 168-170 171
145 135 140 141 142 172
a In ItalIc refers to data patters from dataset-2; In bold and underlmed are used for test.
 200 Data Mining and Knowledge Discovery for Process Monitoring and Control

The 84 new data patterns were combined with dataset-l and processed by
AutoClass. It was found that the 230 data patterns (84 + 146) were classified into 15
classes. Interestingly, all the 84 new data patterns in dataset-2 were classified into
seven new classes (classes 2,5, 7, 8, 9, 13, 14), while all the 146 patterns in dataset-
1 are in eight classes (classes 0, 1,3,4, 6, 10, 11, 12). This is consistent with the
poor predictions for the dataset-2 using Model-I. A summary of the classification is
given in Table 9.4 where the data in italic are from dataset-2. The data patterns in
bold and underlined are chosen for testing and the rest for retraining to generate
model-2. All together 49 data patterns are chosen for test, of which 19 are from
Dataset-2 and 30 from Dataset-I. The degree of confidence for the training data is
100% - (111181)% =92.8% and for test data 100%-(3/49)% =93.9%.

9.3.4 Improvement Using Dataset-3

Later further 48 data patterns (Dataset-3) were provided. Model-2 was applied to
dataset-3 and the degree of confidence was found to be 100% - (27/48)% = 43.8%.
This is low but better than the prediction to dataset-2 using Model-l (25.0%). The
48 new data patterns are then mixed with dataset-l and dataset-2 and processed by
AutoClass to give sixteen classes (a detailed classification is given in Table 9.5). It
is found that twenty-five of the forty-eight data patterns form a new class (class 1 in
Table 9.5, data patterns 251 - 275) but the rest are assigned to the classes combined
with data from dataset-l and dataset-2. The degree of confidence using model-2 to
predict the classes 1, 2, 3, 8 and 11 are summarised in Table 9.6. It can be seen that
class I contains only new data patterns and has the lowest degree of confidence for
prediction using model-2. Nineteen of the twenty-five patterns in dataset-l have
deviations bigger than ±2oC. Classes with fewer new data patterns have a higher
degree of confidence in the predictions. This result further demonstrates the
advantage of using AutoClass for clustering data before training a FFNN model. It
is also found that a confidence in a model for predicting new data is lower if more
data are grouped into new classes. For example, all patterns in dataset-2 are grouped
into new classes and the confidence of predicting dataset-2 using Model-l is 25.0%;
while some patterns in dataset-3 are classified into old classes, with a confidence in
predicting dataset-3 using Model-2 of 43.8%.
 Chapter 9 Inferential Models and Software Sensors 201

Table 9.5 Classification result of dataset-I, 2 and 3°.

Class 0 Weight 50 Class 1 Class 2 Class 3 Class 4
I 6 7.2 10 16 17 Weight 25 Weight 22 Weight 22 Weight 9
22 ll... 26 28 29 31 251 252 253- 160 161 175 162 176 182 157 158 180
33 34 36-39 40 41 255 256 257- 177 178 179 183 184 185 181 188 189
47 48 49 50 51 52 59 260 261- 186 187 192 222 223 190 191 198
76 81 82 111 113 264 265 266- 193 209 212 224-227 228 199 200
117118 120 21 122 268 269 270- 216 217 218 229-231 232 201-203 204
123 124 126 127 128 273 274 275 219 220 221 233 238 239 205 206 207
129 130 131 133 134 234 235 236 240 241 208
135 138 145 237
Class 5 Weight 18 Class 6 Class 7 Weight Class 8 Class 9
5 32 35 42 43 Weight 18 1!L8 20 24 27 Weight 13147 Weight 12
103 104 105 108 11 12 13 53 30 44 45-46 80 148 149-151 21 25 73 83
109110112114 54 55 56-69 101 102 119 152 153 154 87 92 93 94
115 116 136 137 60 61-63 77 125 132 140 155 156 276 100 106 107
139 78 79 85 141 142 144 277 278 143
Class 10 Class 11 Class 12 Class 13 Class 14 Class 15
Weight 12 Weight 11 Weight 10 Weight 10 Weight 10 Weight 8
2 14 15 163 164 242 3181972 165 166 167 159 194 195 4 69 74
64 65 66 243-245 246 84 86 88 89 168-171 172 196 197 210 75 95 96
67 68 70 247-249 250 90 91 173 174 211 213 214 97146
71 98 99 215
a Italic - data from dataset-3; in bold and underlined - test data

The 48 new data patterns in dataset-3 have been combined with datasets-1 and -2
to develop model-3. The sampling of test data patterns follows the same procedure
as before. A total of 68 data patterns has been used for testing with the remaining
2lO patterns for training. Model-3 has a degree of confidence of 92.6% (=lOO% -
(5/68)%) for testing data and 93.8% (=100%-{l3/2lO)%) for the training data.

Table 96 Degree 0 f confid l2 topre d'lct dataset-3

I ence to usmg mo de-
Number of data Number of Total data Number of Degree of confidence
patterns from data patterns patterns predictions with an using model-2 to
Class dataset-3 from in the class absolute error greater predict the class
es datasets-l than ±2°C
and 2
I 25 0 25 19 24.0%(-100%-(19/25)%)
2 4 18 22 0 100.0%
3 7 15 22 4 57.1%
8 3 10 13 0 100.0%
II 9 2 11 4 44.4%
202 Data Mining and Knowledge Discovery for Process Monitoring and Control

9.3.5 Improvement Using Dataset-4

Dataset-4 was subsequently obtained from the refinery and has 25 new data patterns.
The confidence of applying model-3 to dataset-4 is 100%-(8/25)% = 68.0. Dataset-4
was then combined with previous data sets to give a database of 303 patterns. These
are classified by AutoClass into fifteen classes. 77 data patterns have been selected
as test data and the rest for training to develop the FFNN model-4. The model has a
confidence of 92.5% (=100%-(17/226)%) for training data and 90.9% (=100%-
(7177)%) for test data. In order to make a comparison with the conventional
sampling approach, 77 data patterns have selected using random sampling as test
patterns. The training is terminated using the same criterion, i.e., a training error of
5.75e- l . On this basis, 11 test data patterns have errors exceeding ±2°C. So the
confidence for the test data is 100%-(11177)% = 85.7%. This is lower than that of
the test data used in this study, which is 90.9% (100%-(7177)%). The confidence of
the training data for both approaches is the same, 92.5%.

Table 9.7 The changing ranges of input /output variables ofmodel-4.

Attribute Unit Input Output Max. Min. Range
Tl-ll °c -J 205.40 181.50 23.90
Tl-12 °c -J 262.30 215.10 47.20
Tl-33 °c -J 228.00 186.10 41.90
Tl-42 °c -J 285.00 246.80 38.20
Tl-20 °c -J 187.30 129.20 58.10
F215 Ton/hr -J 170.00 106.00 64.00
Tl-09 °c -J 113.00 102.30 10.70
Tl-OO °c -J 511.00 493.00 18.00
F205 Ton/hr -J 118.40 80.00 38.40
F204 Ton/hr -J 39.50 10.00 29.50
F101 Ton/hr -J 3.46 2.39 1.07
FR-l Ton/hr -J 10.00 6.40 3.60
FIQ22 Ton/hr -J 7.63 3.30 4.33
F207 Ton/hr -J 14.00 6.80 7.20
TS3 °c -J 7.00 -13.00 20.00
 Chapter 9 Inferential Models and Software Sensors 203

,-
I
-------------- ------ ------------~

10 1
I
Prediction
5~ Target

o~-- --

-15 - I
I
------- --~

Figure 9.3 Comparison between predictions using Model-4 and the target values for
the first 151 data patterns.

10 -

-10 ~ - Prediction
Target
-15 -

Figure 9.4 Comparison between predictions using Model-4 and the target values for
the last 152 data patterns.

Model-4 (i.e., all data sets) covers all the operational seasons and is being used
very satisfactorily. It has proved to be robust and reliable over a wide range of
operational conditions. Figures 9.3 and 9.4 show the differences between
predictions using Mode-4 and the target values of the 303 patterns. Variations in the
input and output variables are summarised in Table 9.7. The plant has reduced the
sampling frequency to twice a day compared with four to six times used previously
and the intention is to reduce it to once a day in the future.
204 Data Mining and Knowledge Discovery for Process Monitoring and Control

9.4 Dimension Reduction of Input Variables

In applying neural networks for software sensor design, researchers tend to include
as many variables as possible to ensure that no relevant variables are omitted. This
may sometimes result in a very large dimension of input. An unnecessary large
dimension of input variables may have adverse effects. For a fIxed number of
training data patterns, with the increase of input variables it becomes more sparse in
the multi-dimensional space, and therefore degrades the learning performance. The
generality of the learned model may also be reduced due to inclusion of irrelevant or
not important input variables. Increased number of inputs also means more
connection weights to be determined and therefore leads to longer training time.
Apart from irrelevant and not important variables that cause large dimension of
input variables, there may be correlations between input variables. Correlated inputs
make the model more sensitive to the statistical peculiarities of the particular
sample; they accentuate the ovemtting problem and limit generalisation - a common
occurrence in linear regression.
Use of prior knowledge, if available to identify irrelevant or correlated variables
is clearly useful. Otherwise mathematical techniques are required to solve the
problem. In the following, we will be looking at several techniques.

9.4.1 Use of peA for Input Dimension Reduction

Since the fIrst few PCs are able to capture most of the variance of the original
explanatory variables, it is possible to use only the fIrst few PCs to replace the
explanatory variables as the inputs to a FFNN model. Since normally not all PCs are
required, the dimension of inputs is reduced. In addition, these PCs are linearly
uncorrelated and orthogonal.
McAvoy et al. [228] studied the use of FFNNs to fluorescent spectral analysis
data for predicting composition of a binary mixture. Thirty data points were
sampled from each fluorescent spectrum which were used as inputs of a FFNN
mode whose output is the composition of one of the components. Baughman and
Liu [229] used the same data and developed a 30-20-1 FFNN for this purpose.
Because there are only 46 data patterns available, the size of the network is
relatively large.
 Chapter 9 Inferential Models and Software Sensors 205

Table 9.S Principal components for the fluorescent spectral analysis data.
Original variables PC I PCz PC3 PC30
XI 8.74E-03 0.975 0.223 -2.42E-05
X2 6.20E-02 0.995 7.37E-02 1.13E-04
X3 9.50E-02 0.995 -7.16E-03 -1.42E-04
X4 0.123 0.991 -4.39E-02 -8.74E-05
X5 0.172 0.984 -4.90E-02 1.06E-04
X6 0.28 0.958 -5.70E-02 -9.44E-05
X7 0.472 0.88 -5.77E-02 2.92E-04
X8 0.715 0.697 -5.06E-02 -5.28E-05
X9 0.898 0.437 -3.94E-02 -2.27E-04
Xo 0.975 0.219 -2.8 I E-02 -1.30E-OS
XII 0.996 7.98E-02 -1.98E-02 1.26E-04
X/2 1.000 -6.04E-03 -1.32E-02 -3.17E-04
Xi3 0.998 -S.80E-02 -7.49E-03 7.07E-04
X/4 0.996 -8.84E-02 -5.41 E-03 -2.87E-04
XI5 0.994 -0.109 -9.89E-04 7.49E-OS
XI6 0.993 -0.122 1.34E-04 -1.82E-04
XI7 0.991 -0.131 2.36E-03 1.0IE-OS
XJ8 0.991 -0.137 3.7SE-03 1.6SE-OS
XJ9 . 0.99 -0.141 6.07E-03 S.93E-OS
X20 0.989 -0.144 7.51 E-03 -1.72E-OS
X21 0.989 -0.145 8.06E-03 S.70E-05
X22 0.989 -0.146 1.0IE-02 -8.84E-OS
X23 0.989 -0.145 l.38E-02 7.60E-OS
X24 0.989 -0.145 1.62E-02 S.66E-06
X25 0.989 -0.145 1.87E-02 -9.03E-05
X26 0.99 -0.142 1.81 E-02 9.29E-06
X27 0.99 -0.139 1.69E-02 2.64E-OS
X28 0.99 -0.137 2.2 I E-02 -6.22E-OS
X29 0.99 -0.135 2.72E-02 -1.62E-OS
X30 0.991 -0.13 2.79E-02 3.38E-OS

% of variance 74.331% 25.398% 0.252% 3.241E-06%

% ofcummulative
variance 74.331% 99.729% 99.980% 100%

We have carried out PCA analysis of the data and the result is shown in Table 9.8.
Two observations are made. Firstly, it was found that the fIrst two PCs can capture
99.729% of the variance. Secondly, Xl - X7 are mainly associated with PC 2 and X9-
X30with PCI.XS is nearly equally associated both PCs'
Because the first two PCs can capture 99.729% of all the variance, it is possible to
use the two PCs rather than the 30 x, as the inputs to develop a FFNN model. As a
result we have developed a 2-4-1 model. Comparison was made between the 30-20-
1 model developed by Baughman and Liu [229] and the 2-4-1 model. For both
models, 32 data patterns were used for training and 9 for test. The training and test
errors for the 30-20-1 model are 3.02% and 3.87%, while for the 2-4-1 model are
206 Data Mining and Knowledge Discovery for Process Monitoring and Control

2.29% and 3.55%. Because the difference in configuration of the two networks it is
not appropriate to give a definite conclusion on which model is more accurate, only
based on the above errors. Nevertheless it is obvious that the 2-4-1 model can give
more accurate or equivalent result.
An alternative way to use PCA to reduce the input variable dimension is to
analyse the contribution of individual explanatory variables to the first few PCs.
Those which are not important can be excluded from the FFNN model.

9.4.2 Sensitivity Analysis

Sensitivity analysis is a simple method of fmding the effect of an input on the output
of the network. A FFNN can be developed first using all possible input variables
and then be used to carry out sensitivity studies. Irrelevant variables can be
excluded as a new FFNN is developed. Suppose there are a number of inputs Xi and
only one output z, then the relationship of an input variable Xi and the output z is
found by determining the impact of a small change in Xi on z. If large change occurs
on z, then Xi is considered to be one of the key factors in producing the current
activation value of z. The sensitivity can be deduced given a FFNN structure with
multiple inputs [231], a number of hidden neurons and only single output. The input
to a hidden neuron is
(9.1)

where aoj is the bias weight and Xi denotes an input with associated weight aij. The
output of a hidden neuron is
YJ.= ---
1 (9.2)
1+ e -u,

Then the output of the output node is

v=b o+ ~ bjY j (9.3)

where bo is the bias weight and bj are the respective weights for the hidden nodes.
Clearly the output z = _ 1 _ .
1+ e- v
The sensitivity of the output z to the input variable Xi is dzldx i . Using the chain
rule, the sensitivity may be related to the output z, the output from each of the
hidden nodes and the different set of weights aij and bj as follows,
dzldxi = (dz/dv)( dvldYj)( dyjduj)( dujdx;)
= z(1-z) I bjYj(I - Yj)aij
 Chapter 9 Inferential Models and Software Sensors 207

For a FFNN with multiple outputs, the same procedure can be applied to study the
impact of each input on each output. Table 9.9 is an example of sensitivity study for
a waste water treatment process [230]. It shows sensitivity of the standard deviation
of suspended solid to a number of operational variables. It indicates that the
classifier interface level is the most important variable and inlet oil concentration
and inlet cyanide act have positive impact, while an increase in ammonia flow has a
positive effect in reducing the standard deviation of suspended solids.

Table 9.9 The sensitivity of the standard deviation of suspended

solids to various operational variables.
Variable Average sensitivity
Interface levOel 0.52
Sludge age 0.48
Inlet oil concentration 0.46
Inlet cyanide concentration 0.44
Aerated tank temperature -0.42
Inlet ammonia flow -0.33
Treated effiuent pH -0.33

Sensitivity can also be carried out by plotting the output variable against an input
variable as shown in Figure 9.5 [175]. Figure 9.5 shows the relationships of
variables of a waste water processing unit. PH-P is the measured PH value of the
effiuent from the unit and PH-E, SS-E and Q-E are the PH value, suspended solid
and flowrate of the influent to the unit.

8.4
8.3
8.2
PH-E
8.1
8.0
"-
:i: 7.9
"-
7.8
7.7
7.6
7.5
7.4
7.3
0 0.2 0.4 0.6 0.8 1.0
Normalised values of PH-E, SS-E and Q-E

Figure 9.5 The sensitivity of the effect ofPH-P to influent SS-E, Q-E and PH-E.
208 Data Mining and Knowledge Discovery for Process Monitoring and Control

However the above sensitivity studies should be conducted with great care. If the
inputs are strongly correlated then the sensitivity of the output to an input variable
may be dependent on what values the other input variables are fixed at. If PCs rather
than the explanatory variables are used, this problem can be resolved since all PCs
are linearly uncorrelated. Although it may be able to estimate the impact of each PC
on the output, to relate to the explanatory variables is not always possible. For
example, Table 9.10 shows the coefficients of the first four PCs and the six
explanatory variables Xl - X6. Since PCl is nearly equally weighted on four (i.e., XJ,
Xl> X4, xs)of the six variables and the PCl only accounts about half of the total
variance, even we can find the impact of PC I on the output, it is difficult to
determine the impact on the output by Xl - X6 [231]. A satisfactory situation would
be that the first PC accounts for most of the total variance and/or is heavily
weighted on a few explanatory variables.

Table 9.10 The coefficients of the first four PCs and

the explanatory variables.

Variable PCl PC2 PC3 PC4

Xl 0.449 0.249 0.349 -0.447
X2 0.530 -0.058 0.041 0.197
X3 0.280 -0.537 -0.575 -0.542
X4 0.457 -0.253 -0.149 0.669
Xs 0.467 0.223 0.267 -0.134
X6 0.110 0.729 -0.673 0.042

9.4.3 Weight Matrix Analysis

Another approach to analyse the input impact on an output is to analyse the weight
matrix. Since the nodes between two adjacent layers are fully connected, such an
analysis is not straightforward. Garson [232] proposed the following measure for the
proportional contribution of an input to a particular output,
 Chapter 9 Inferential Models and Software Sensors 209

(9.5)

In the above equation, p and q refer to the input layer, p = 1, nj, j refers to hidden
layer, j = 1, nh, i, j, k are the ith, jth and kth input, hidden and output node. A
disadvantage of the approach is that during the summation process, positive and
negative weights can cancel their contribution, which leads to inconsistent results.
Milne [233] commented that the sign of the contribution is lost, and proposed the
following measure,

(9.6)

Wong et al. [234] and Gedeon [235] defmed measures for the contribution of an
input to a neuron in the hidden layer, and a hidden layer neuron to an output layer
neuron,
(9.7)

Iw},1 (9.8)
P}k =-nh--
Llwrkl
r""l

Then the contribution of an input neuron to an output neuron is,

nh
Q., = L(Pir X Prk) (9.9)
r=!

The approaches using weight matrix to examine the impact of input nodes on output
nodes are interesting but they have not been widely tested.
210 Data Mining and Knowledge Discovery for Process Monitoring and Control

9.5 Dynamic Neural Networks as Inferential Models

The data used in the industrial case study described in Section 9.3 was collected on
steady state basis. Consequently the model developed will only be accurate if the
operation is around steady states. A feedforward neural network is static in nature
and is not designed to deal with dynamics, e.g., time delays. Recently several
approaches have been proposed to introduce dynamics to FFNNs, in order to
capture the dynamic nonlinear feature of processes. Such neural networks can
essentially be classified according to implementation of the dynamic character as:
(1) NNs with lumped dynamics; (2) recurrent NNs; and (3) dynamic multilayer
percetron [236].
NNs with lumped dynamics are static networks with delayed measurements of the
process inputs and outputs. Accordingly, the network approximate the current
process outputy(k) as a function of the delayed measurements:

y(k) = j(u/(k1), u/((k-l)1), ... u/((k-m)1),

u2(k1), ui(k-l)1), ... u2((k-m)1),

up(k1), uP((k-l)1), ... up((k-m)1)

y((k-l)1), ...y((k-n)1) (9.10)

where y - output variable

u/, UJ, ... up - in total there are p input variables
T - period or time step for sampling training data
k - the current sampling point
m - the (k-m)th sampling point for inputs
n - the (k-n)th sampling point for the output

It is clear that NNs with lumped dynamics do not change the internal structure of
a static NN. Dynamics is captured by sampling data from input-output dynamic
trends. Consequently, the dimension of the NN input space increases according to
the number of delayed measurements used.
Bhat and McAvoy [237] described such a case study using a stirred tank reactor
as shown in Figure 9.6. The dynamics of the system can be described by two linear
differential equations and three nonlinear algebraic equations. Training data were
 Chapter 9 Inferential Models and Software Sensors 211

generated using dynamic simulation by forcing the F2 stream with a 2% PRBS

signal super imposed upon its steady state value. The F2 and pH responses used for
training are shown in Figure 9.7. A FFNN neural network was developed to use the
past five pH and F2 values as well as five future F2 values to predict the future pH
values. The time step T is 0.4 min. Since the data is actually historical data
generated through simulation, the future F2 values are available. The size of the
network is therefore fifteen inputs and five outputs. The number of hidden neurons
was five.

Figure 9.6 pH CSTR.

~
0
0 550
z
"0
~
e
~
0
LA:
450

4 ~ ________ ~ ________- L_ _ _ _ _ _ _ _ ~ ________ ~

o .00 200 300 400

Time (min)

Figure 9.7 Data used for training the CSTR neural network.
212 Data Mining and Knowledge Discovery for Process Monitoring and Control

A moving window approach is used. At the beginning of the training process the
window is placed at the beginning of the database. The fIrst fIve pH and ten F2 are
used as inputs and the next fIve pH are used as the outputs. After the fIrst
presentation of the data, the window moves T down the database. Again the fIrst
fIve pH and ten F2 in the window are inputs to the net and the next fIve pH in the
window are outputs. This process is continued until the end of the database is
reached and, the process is repeated by starting at the beginning of the database,
until the training is terminated through either reaching the required error or the
maximum number of iteration.
Recurrent neural networks (RNN) (e.g., Shaw et al. [238]) dispense with the
explicit use of lagged measurements at the network input. Instead, RNN posses a
long-term memory through recurrent connections within the network. However,
according to Ayoubi [236], the application of RNN was minimised to sequence
processing, yet there are no signifIcant applications to model identifIcation of real-
world systems.
An alternative method is the multilayer dynamic percetron [236, 239] which
modifIes the neuron processing and interconnections in order to incorporate
dynamics inherently within the network.

9.6 Summary

It is important to have an appropriate mechanism to test the reliability of software

sensor models developed using FFNN in both model development and maintenance
stages. This is complicated by the high dimensionality and large volume of data.
The integral framework described in Sections 9.2 and 9.3 uses Bayesian automatic
classifIcation to cluster the multivariate data into classes having similar features.
Test data for FFNN model development are then selected from each of the classes.
When new data become available, it is possible to tell if they represent new
information which makes it possible to determine when the neural network needs to
be retrained. Through the progressive improvement of the model, it is found that the
confIdence of the model for predicting new data is lower if more data are grouped
into new classes. For example, all patterns in dataset-2 have been grouped into new
classes, so the confidence in predicting dataset-2 using Model-l is 25.0%; while
some patterns in dataset-3 are classifIed into old classes, where the confIdence of
predicting dataset-3 using Model-2 is 43.8%.
 Chapter 9 Inferential Models and Software Sensors 213

Due to inadequate knowledge of the domain problem, people tend to use many
input variables. Inclusion of irrelevant and redundant variables may deteriorate the
model performance because it increases the size of the net, leading to longer
training time, and requires more training data, and reduces model generalisation
capability. The several approaches introduced in this chapter, including PCA,
sensitivity analysis and weight matrix analysis can be used to combat the problem.
However there is still the need to carry out further investigation in these approaches,
especially in solving practical problems.
Only a very brief introduction has been given to dynamic FFNNs for software
sensor design. Due to its great application potential, there has been a growing
interest in dynamic neural networks. Apart from the need to develop the most
appropriate network structure and learning approaches, there is clearly a practical
difficulty in applying dynamic neural networks, that is, how to get the large number
of data patterns needed. Experimenting on industrial processes is time consuming
and expensive because it may cause the process to deviate from desired conditions.
There is another problem associated with doing experiment on industrial processes,
which is that the processes are often under closed loop control using linear models.
Kim et al. [240] addressed the need to consider the reality that most processes are
under linearized control scheme.
There are many other issues associated with determination of FFNN network
topology and training. A detailed discussion on these has been given in Chapter 5
therefore they are not repeated in this chapter.
CHAPTER 10
CONCLUDING REMARKS

In this monograph we have sought to provide an introduction to automatic analysis

and interpretation of process operational data both in real time and over the
operating history. Methods are developed for designing intelligent, state space based
systems for process monitoring, control and diagnosis. Such a system is able to
identify known and new operational states, either normal or abnormal and project
the operation of the process to a single point of the operational state space by
simultaneously considering all measurements and giving causal explanations to
operators and plant managers. The techniques have also proved useful in
discovering operational states for product design. In developing the methods, we
have attempted to address the point that plant operators and supervisors are part of
the overall control system responsible for data interpretation and critical decision
making, and therefore should be integrated into control systems in a way to provide
them with necessary computer based, automatic processing tools.
It is believed that the problems addressed in this book which have not been fully
studied before are important in process monitoring and control, and we hope readers
frod the methods of approaching the problems both interesting and practical. Many
of the methods introduced in this book are new, such as feature extraction using
wavelets and conceptual clustering using inductive learning. Some techniques are
applications of traditional methods, such as principal component analysis. Even for
well developed methods such as feedforward neural networks we have tried not to
repeat the algorithms. Instead we present the methods in a novel way that
summarises experience and addresses the various necessary considerations, such as
training and test data selection and sensitivity study in using neural networks.

X. Z. Wang, Data Mining and Knowledge Discovery for Process Monitoring and Control
© Springer-Verlag London Limited 1999
216 Data Mining and Knowledge Discovery for Process Monitoring and Control

Examples and industrial case studies in varying degrees of complexity have been
used to illustrate these methods. They include a continuous stirred tank reactor, the
reaction regeneration system of a fluid catalytic cracking process, a methyl tertiary
butyl ether unit, the reaction fractionation system of fluid catalytic cracking and a
waste water treatment plant. Many more examples could have been added, but we
feel these should be sufficient to demonstrate the effectiveness of the methods and
aid in clarifYing the ideas and concepts.
We have intended to provide a new way of thinking in the approach to designing
and operating process control systems and introduce practical methodologies for
implementation. We also hope that the book will provide a stimulus to researchers
since the field is still in its infancy. Many more challenges than those addressed in
this book need to be considered in order to develop effective, robust and practical
systems for knowledge discovery and for designing intelligent and state space based
monitoring and control environment.
Suggestions for future work have been summarised at the end of each chapter and
will therefore not be repeated here. Most of the case studies in the book are with
respect to continuos processes save a case study described in Chapter 4. Batch
processes are expected to provide more challenges due to the distinctive features of
batch operation mode. Batch processes do not have steady state operations and are
more flexible. They also provide great opportunities for application of the
techniques discussed in this book. For example, wavelets are potentially a powerful
technique for dealing with signals of batch operations because they often change at
higher frequencies than in continuous processes. In addition, feedstock disturbances
occur frequently and on-line measurements of product quality variables are not
available. As a result, most batch processes have not been able to achieve tight
quality control. Knowledge discovery approaches are attractive for dealing with
these kind of problems because of difficulties associated with developing accurate
process models from first principles [241]. At present, industrialised countries are
shifting the focus from large-scale commodity production to smaller scale fine
chemicals, pharmaceutical and food production, therefore, there is a clear need for
advances in this field.
APPENDIX A
THE CONTINUOUS STIRRED TANK REACTOR (CSTR)

A single, non-isothennal continuous stirred-tank chemical reactor (CSTR) is shown

in Figure AI.

. . .(0;. . ....
·
····· .....
Feed ·

. . . . . . .Q
o
......... -_ ..... .

Cooling !
Wdter~

Figure Al The CSTR reactor.

A single reaction A ~ B takes place in the reactor. It is assumed that the tank is
very well mixed therefore the concentration of Co of component A and the
temperature of the product stream leaving the tank are equal to that in the tank. The
reaction is exothermic and reaction temperature T R is automatically controlled by a
PID controller through manipulating the cooling water flowrate. There are also feed
218 Data Mining and Knowledge Discovery for Process Monitoring and Control

flowrate and liquid level controllers as shown in Figure AI. Apart from the three
controllers, the dynamic behaviour of the process is described by the following
equations:
Component mass balance

d Co
V ----;;t=FiC;-FoCo- V -EIRT
Koe RCa (AI)

Energy balance

dTR
VPCp----;;t = PCpFi Ti - PCpFo TR
(A2)

Total mass balance

dV
-dt= F - Fa (A3)
I

V - holdup of the reaction mixture, m 3 . The volume of the tank is 1 m 3 .

L - Liquid level in the tank, 0 - 100.

Co - Concentration of component A in the product stream leaving the tank,

kmol/m 3, which is equal to that in the tank.

Ci - Concentration of component A in the inlet feed stream, kmol/m 3 •

Fi - Feed flowrate, m 3/min.

Fa - Product stream flowrate, m3/min.

Fe - Cooling water flowrate, m 3/min.

Ko - Reaction coefficient, Ko = 1.0 x 10 10 min· l .

EIR - 8330.1 K"I.
. 0 f reactIOn
P - DenSIty .. mIxture, P = 10 6 gIm 3 .

Pc - Density of cooling water, Pc = 10 6 glm3 •

Cp - Specific heat capacity of the reaction mixture, Cp = 1 cal/m3 .

Cpc - Specific heat capacity of cooling water, Cp = 1 cal/m3.
-!J.Hrxn = 130 x 106 (cal/kmol)
 Appendix A The Continuous Stirred Tank Reactor (CSTR) 219

a - Constant, a = 1.678 x 106 (cal/min)/K.

b - Constant, b =0.5

1';- Temperature of the inlet feed, K.

TR - Temperature of the reaction mixture, K.

Tcin - Inlet temperature of cooling water, K.

More detailed description of the process can be found in [8]. A dynamic simulator
was developed for the CSTR which has included three controllers as shown in
Figure AI. The simulator has a MS Windows based graphical interface which
allows users to make any changes and faults and disturbances can be easily
introduced. To generate a data set or data case, run the simulator at steady state and
introduce a disturbance or fault and at the same time start to record the dynamic
responses. Eighty five data sets were generated which are summarised in Table AI.
For each data set, eight variables were recorded, including F;, T;, C;, T w;, Fw, TR, Co
and L. In each data set, each variable was recorded as a dynamic trend consisting of
150 sampling points. Therefore for each variable the data size is a matrix 85 (the
number of data sets) x 150 (the number data points representing a dynamic trend).

Table At Data cases generated.

Data sets Data detail
1-11 All control loops are at AUTO mode and S.P. ofT R= 350K. Change Ti (K)
from 343 ~ 333, 343 ~ 323, 323 ~ 343, 343 ~ 353, 353 ~ 343, 343 ~
363, 363 ~ 343, 310 ~ 303, 303 ~ 313, 310 ~ 293, 293 ~ 313
12-15 AIl control loops are at AUTO mode and S.P. ofTR = 350K. Change Ci
(kmolfm3 ) from 2.0 ~ 1.6, 1.6 ~ 2.0, 2.0 ~ 1.2, 1.2 ~ 2.0
16-20 AIl control loops are at AUTO mode and S.P. ofT R = 350K. Change Fi (m 3
fmin) from 1.00 ~ 1.06, 1.06 ~ 1.02, 1.02 ~ 0.94, 1.00 ~ 0.62, 0.62 ~
1.00
21-24 All control loops are at AUTO mode and S.P. ofT R =350K. Change L ( % )
from 50.0 ~ 60.0, 60.0 ~ 50.0, 50.0 ~ 40.0, 40.0 ~ 50.0
25-31 AIl control loops are at AUTO mode and S.P. ofT R = 405 K. Change Ti (K)
from 343 ~ 333, 333 ~ 323, 323 ~ 333, 343 ~ 353, 353 ~ 343, 343 ~
363, 363 ~ 343
32-36 All control loops are at AUTO mode and S.P. ofT R= 405 K. Change Twi (K)
from 310 ~ 303, 310 ~ 293, 293 ~ 313, 3 13 ~ 323, 323 ~ 313
220 Data Mining and Knowledge Discovery for Process Monitoring and Control

Table Al Data cases generated (continued).

Data sets Data detail

37-39 All control loops are at AUTO mode and S.P. ofT R = 40S K. Change Ci
(lanol/m3) from 2.0 ~ 1.6, 1.6 ~ 2.0, 2.0 ~ 1.2
40-43 All control loops are at AUTO mode and S.P. of TR =40S K. Change Fi (m 3
/min) from I.OO~ 1.06, 1.06~ 0.98, 0.98~ 0.86, 0.86~ 1.02
44-46 All control loops are at AUTO mode and S.P. ofT R =40S K. Change L (%)
from SO.O~ 60.0, 60.0~ SO.O, SO.O~ 40.0
46-52 All control loops are at AUTO mode and S.P. ofTR =380 K. Change Ti (K)
from 343~333, 333 ~ 323, 323 ~ 333, 333 ~ 343, 343 ~ 3S3, 3S3 ~ 343
53-56 All control loops are at AUTO mode and S.P. of TR = 380 K. Change Twi (K)
from 310 ~ 303, 303 ~ 313, 310 ~ 323, 323 ~ 313
57-60 All control loops are at AUTO mode and S.P. ofT R = 380 K. Change Ci
(kmollm3) from 2.0 ~ 1.6, 1.6 ~ 2.0, 2.0 ~ 1.2, 1.2 ~ 2.0
61-66 All control loops are at AUTO mode and S.P. of TR = 380 K. Change Fi (m 3
/min) from 1.00~ 1.06, 1.06~ 1.00, 1.00~ 1.10, 1.I0~ 1.00, 1.00~ 0.90,
0.90~ 1.00
67-70 All control loops are at AUTO mode and S.P. of TR = 380 K. Change the S.P.
of L ( % ) from SO.O~ 40.0, 40.0~ SO.O, SO.O~ 60.0, 60.0~ SO.O
71-80 All control loops are at AUTO mode and S.P. ofTR =380 K. Change the
output of the CSTR level controller from (S) SO.O~ 10.0, SO.O~ 8.0,
SO.O~ 6.0, SO.O~ 12.0, SO.O~ 14.0, SO.O~IS.O, SO.O~ S.O, SO.O~ 2.0,
SO.O~ 3.0, SO.O~ 0.0
81-85 All control loops are at AUTO mode. Change the output of the controller T R
from (%) 71.4~ 20.0, 71.4~ 21.0, 71.4~ 22.0, 71.4~ 19.0, 71.4~18.0
APPENDIXB
THE RESDIUE FLUID CATALYTIC CRACKING (R-FCC)
PROCESS

Fluid catalytic cracking process (FCC) is a dominant feature of most refmery

operations, representing the order of thirty percent in product value. FCC is also a
very complicated process featured by highly non-linear dynamics due to the strong
interactions between the riser tube reactor and fluidised bed regenerator through
heat, mass and pressure balances.
The residual fluid catalytic cracking (R-FCC), as shown in Figure B 1 is used as
case studies in Chapters 5 and 6. It converts heavy bottoms of the crude and vacuum
distillation columns into more valuable gasoline and lighter products. Preheated
feed is mixed with high temperature slurry recycle, which comes from the bottom of
the main fractionator, and injected into the riser tube reactor, where it is mixed with
high temperature regenerated catalyst and totally vaporises. The high temperature
regenerated catalyst provides the sensible heat, heat of vaporisation and heat of
reaction necessary for the endothermic cracking reactions which complete in a few
seconds. As a result of the cracking reactions, a carbonaceous material - coke is
deposited on the surface of the catalyst. As a result continuous regeneration is
required.
Separation of catalyst and gas occurs in the disengaging zone of the reactor.
Entrained catalyst is stopped by the cyclones. Catalyst is returned to the stripping
section of the reactor where steam is injected to remove entrained hydrocarbons.
Reactor product gas is channelled to the main fractionator for heat recovery and
separation into various product streams. Wet gas from the overheads of the main
 222 Data Mining and Knowledge Discovery for Process Monitoring and Control

fractionator (C 6 and lighter) is compressed for further separation in downstream

fractionators.
Spent catalyst is transported from the reactor to the regenerator through the spent
catalyst pipe. Air is injected into the bottom of the regenerator lift pipe to assist the
circulation of catalyst.
Catalyst in the regenerator is fluidised with air which is provided by the lift and
combustion air blowers. Carbon and hydrogen on the catalyst react with oxygen to
produce carbon monoxide, carbon dioxide and water. While most of the reactions
occur in the fluidised bed, some reaction does occur in the disengaging section
above the bed, where some catalyst is still present. Gas travels up the regenerator
into the cyclones where entrained catalyst is removed and returned to the bed.
The regenerator is run at a high temperature and an excess of oxygen to ensure
that virtually all carbon monoxide produced in the bed is converted to carbon
dioxide before entering the cyclones. Because this is a residual FCC process
processing heavy feed, the amount of heat generated through burning coke in the
regenerator is more than that required by the endorthemic cracking reactions.
Therefore it has internal and external heat exchangers to remove the excess heat.
Regenerated catalyst flows over a weir into the regenerator standpipe. The head
produced by catalyst in the standpipe provides the driving force for catalyst through
the regenerated catalyst pipe to the riser tube reactor.
The major control loops for the process are summarised in Table Bl. They
include reaction temperature, pressures of the two reactor vessels as well as
flowrates of feed, air and steam as well as catalyst hold-up. A very important feature
of the process is the very complicated and dynamic interactions between the heat,
mass and pressure balances between the two reactor vessels, and fluidisation
conditions. On top of the control loops, safety guard systems playa critical role in
preventing disastrous situations. There are four major safety guard systems as shown
in Table Bl. They all need the authorisation of operators by pressing one or two
buttons to prevent system over reactions. When a safety guard system is activated,
in total fourteen valves will act to close, open or maintain at the original position
according to predefined logic.
A dynamic simulator for training operators was developed in 1992 by one of the
author, Wang for the refinery. The simulator has a number of features compared
with some other FCC simulators. First, it is able to simulate continuously and
smoothly the start-up and shutdown procedures and normal operation. In addition, it
has a very good operability. All the operating variables and equipment units are
 Appendix B The Residue Fluid Catalytic Cracking (R-FCC) Process 223

operable within a wide range, including conditions regarded as abnonnal.

Furthennore apart from. faults that can be initiated randomly, there are also extra
twenty faults that are common to FCC processes. More importantly, it has high
fidelity: the simulator has been tested in the refinery for operator training for several
years and during the process the simulator has been continuously improved. There
are other features including intelligent on-line support, scoring and snapshot.
In this study, sixty four data patterns generated from the simulator was used which
are summarised in Table B2. We limit our discussion to 64 data patterns in order to
simplify the discussion and result presentation. The data sets include faults or
disturbances:

• fresh flow rate increased and decreased

• preheat temperature of mixed feed increased and decreased
• recycle slurry flow rate increased and decreased
• opening rate of hand valve V20 increased and decreased
• air flow rate increased and decreased
• valve 401-ST opening from 100% decreased
• cooling water pump failure
• compressor failure
• double faults

The sixty four data patterns were first generated by the customised dynamic
training simulator, then random noises were added to the data using a noise
generator ofMATLAB®, before it is fed to the system.
 Flue gas fj
~
401-ST
LC302
----T
~
~ ~
Water -_J ~
~.
:..... ~ ....... . Steam §
!:l..
~ ~
<::l
~
E ~
~
G)
~
~

~ b
G) I-<
~.
::t:: £
'"
~ ~
S
o ~
""'C
(,)
'"tl
~
r)

~
'"
air feed ~
;:s
~.
fresh
combustion P1321 ~.
feed
oil I:l
furnace ;:s
8- !:l..
~
;:s
Figure Bl The simplified flow sheet of the R-FCC process. ~
-.
 Appendix B The Residue Fluid Catalytic Cracking (R-FCC) Process 225

Table Bl The major control loops and safety guard systems of the R-FCC
rocess.
Major control loops
Reaction TC302 Temperature control at the exit of the riser tube
temperature reactor. Set pint, 5 \30c. Manipulated variable is
regenerated catalyst flow to the riser tube reactor

Regenerator PC30 I Pressure control of the regenerator at 1.6 kg/cm 2•

Pressure Manipulated variable is the flue gas flowrate

Inlet pressure PC551 Pressure control. This is to maintain a pressure

of the difference of> 0.3 kg/ cm 2 between the regenerator
compressor and the reactor. The manipulated variable is
normally the compressor speed, but can also be
switched to value on top ofthe fractionator,
especially during start-up

Other controllers Feed flowrates FC309 and FC305, Steam flowrate

F302, water level LC302, catalyst holdup
controller LC301
Safeguards
Low feed flowrate At low feed flowrate, reactions will deteriorate with
increased reaction depth and secondary reactions.

Low flow of Low flow of main air supply causes increase of

main air supply regenerator temperature, poor fluidisation and
even cause reverse flow of catalyst or catalyst
flowing to the air compressor

Low pressure difference A pressure difference of 0.3 kg/ cm 2 between the

between the regenerator regenerator and the reactor is normally required

and the reactor to prevent gas oil flowing into the regenerator
because this may cause possible very high
temperature in the regenerator and damage. This
safety guard will induce the low feed safety guard

External heat When the low main air supply safety guard is
exchanging system activated the external heat exchanging system is also
needed to be cut off
226 Data Mining and Knowledge Discovery for Process Monitoring and Control

Table B2 Summary of the data patterns studied.

Data Fault when time t < 0 operation is
patterns mode at steady state, at t = 0 make
the following step change

1-9 1 Fresh feed increased by 10 20 30

40 50 60 70 80 90%
10-18 2 Fresh feed decreased by 10 20 30
40 50 60 70 80 90%
19-22 3 Preheat Temperature T of mixed
feed increased by 5 10 15 20 C Q

23-24 4 Preheat Temperature of mixed feed

decreased by 15 10 C
Q

25-26 5 Recycle oil Flow rate F increased 70 90%

27-28 6 Recycle oil Flow rate F decreased 70 90%
29-32 7 Opening ratio of hand-valve V20
increased by 5 10 15 24%
33-37 8 Opening ratio of hand-valve V20
decreased by 15253555 10%
38 9 Cooling water pump P-02 failure
39-43 10 Air flow rate increased by 6.5 11.5 15
31.540.5%
44-49 11 Air flow rate decreased by 3.5 8.5 28.5
38.5 48.5 53.5%
50 12 Compressor failure
51-57 13 Valve 401-ST opening from 100%
decreased by 10 20 40 45 60 80 90%
58 1 Fresh feed Flow rate F increased by 65%
59 2 Fresh feed Flow rate F decreased by 85%
60 1&8 Fresh feed Flow rate F increased 65%
and V20 opening rate decreased 55%
61 1&9 Air flow rate increased 9.5% and
compressor failure
62 2&9 Air flow rate increased 9.5% and valve
401-ST opening decreased 35%
63 10&12 Pump P-02 failure and compressor failure
64 10&13 Fresh feed decreased 70% and valve
40 I-ST opening decreased 20%
APPENDIXC
THE METHYL TERTIARY BUTYL ETHER (MTBE)
PROCESS

The refinery methyl tertiary butyl ether (MTBE) process, shown in Figure Cl is
used in Chapter 7 as a case study of applying conceptual clustering to developing
process monitoring systems. MTBE is an important industrial chemical because
large quantities of MTBE are now required as an octane booster in gasoline to
replace tetra ethyl lead. MTBE is produced from the reaction between isobutene and
methanol in a reactor and a catalytic distillation column packed with catalysts. The
reaction takes place at a temperature of 35-75 0C and a pressure of 0.7-0.9 MPa.
The main chemical reaction is:
methanol + isobutene === methy tertiary butyl ether
CH30H + (CH 3hC=CH 2 === (CH3)3COCH3
The reaction occurs in the liquid phase and is reversible and exothermic. Proper
selection of the reaction pressure allows part of the reaction product to be
vaporised, absorbing part of the heat of reaction. Thus the temperature in the reactor
could be controlled.
An unavoidable side reaction is:
isobutene + isobutene === diisobutene
(CH 3hC=CH 2 + (CH 3hC=CH 2 === {(CH 3hC=CH 2 h
However, the selectivity of the primary reaction is about 98-99%. Since the product
of the side of reaction is very limited it can be ignored.
MTBE is mostly produced in the reactor, and a small amount is produced in the
catalytic distillation column which combines reaction and distillation. The rest of
the isobutene and methanol from the reactor passes to the catalytic distillation
 228 Data Mining and Knowledge Discovery for Process Monitoring and Control

column and is reacted. The MTBE produced is then separated simultaneously,

which ensures a high conversion level of isobutene in the catalytic distillation
column.
The rest of methanol after the reaction is recovered from the distillate of the top of
the column by water extraction and a conventional column. The solvability of
methanol in C4 hydrocarbon and in water is quite different so C4 hydrocarbon can be
easily separated from methanol water solution. The recovered methanol and C4
hydrocarbon are then recycled.
The process used in this study comprises a catalytic reactor R201, catalytic
distillation column C20 1, water extraction column C202, methanol distillation
column C203 and several other vessels together with pumps. The overview of the
flowsheet of the process is shown in Figure C 1. Methanol from D202 and C4 from
D201 are mixed in M201 and then pass to the reactor R201. An on-line analyser
measures the ratio of isobutene and methanol and the inlet temperature of the
reactor is controlled by means of the heat exchanger E20 1. Maintaining the pressure
in the reactor is important so as to control the amount of product vaporised to
absorb the heat of reaction. To achieve a high conversion level of isobutene an
additional methanol feed stream is introduced into the catalytic distillation column
at the top of the column. The vessel D211 is filled with catalysts and is used to filter
methanol before it enters the column. The MTBE is taken from the bottom of the
column and the distillate is the mixture of methanol and the C4 hydrocarbon. The
pressure controller at the top of the column is also used to control the pressure of
the upstream reactor. Part of the methanol and C4 hydrocarbon is returned to he
catalytic distillation column as reflux. The rest of the mixture passes to the water
extraction column C202 to be separated into unreacted C4 hydrocarbon which is
taken off from the top and the methanol water solution from the bottom.
The case study is based on data generated from a customised dynamic simulator
used for training operators in start-up and shutdown procedures, emergency
management as well as normal operation monitoring and control of a commercial
MTBE process. This simulator has also been used to study the start-up procedures
and has been validated over several years of usage for various normal and abnormal
conditions. Disturbances as well as faults concerned with various equipment items
can be easily introduced to the simulator and corresponding responses of variables
displayed in trend groups. This study comprises 100 tests, 64 of which are normal or
with small disturbances and 36 have significant upsets or faults, as summarised in
Table Cl. For each test, 21 variables were recorded as indicated in Table C2. For
 Appendix C The Methyl Tertiary Butyl Ether (MTBE) Process 229

each variable, 256 data points are recorded following a disturbance or fault. So the
data has a dimension of 21 x 256. Random noise was added to the data. Figure C2
shows such an example.
Only the sections of feed, reactor and reactive distillation column of the process
have been considered, which means that columns C202 and C203 have not been
included in the study apart from the fact that there is a small recycle stream of the
recovered methanol which affects the feed tank D202. This recycle stream was fixed
at the design flowrate.

Table C2 The variables recorded as dynamic responses.

No. Variable Description
1 F_D201_in Inlet Flow ofD201
2 F_D20Lout Outlet Flow ofD201
3 L_D201 Liquid Level of D20 1
4 LD202_in1 Adding methanol Flow of D202
5 F_D202_out1 Methanol flow from D202 to R201
6 F_D202_out2 Methanol flow from D202 to D211
7 L_D202 Liquid Level of D202
8 T_E201_out Outlet Temperature ofE201
9 F_E20 I_steam Steam flow through E201
10 T_R201_top Top Temperature of reactor R201
11 T_R201_mid Middle Temperature of reactor R201
12 T_R201_bot Bottom Temperature of reactor R201
13 F_C20Lref Reflex flow ofC201
14 F_C201_out Product Flow ofMTBE
15 T_C20Ltop Top Temperature ofC201
16 T_C20Lmid Middle Temperature ofC201
17 T_C201_bot Bottom Temperature ofC201
18 T_MTBE MTBE Temperature
19 L_C201 Liquid Level ofC201
20 LD203_out Liquid Flow from D203 to C202
21 L D203 Liquid Level ofD203
 tv
Table Cl Data cases used for the study, W
o
Cases Case Detail Case Descriotion
I Control valve of LC20 I 0 50%~ 0% Liquid level control of C4 hydrocarbons tank 020 I
2 Pump P201 Failure Pump after the 020 I ~
l:)
3 Switch valve SW3 On~ Off Switch valve after P201, on the feed stream ofC4 hydrocarbons
4 Pump P202 Failure Pump after 0202, on the feed stream of methanol
5 Switch valve SW2 On~ Off Switch valve after P202, on the feed stream of methanol ~
6 Manual valve HC2020 20%~ 0%, Manual valve on the feed of methanol to tank 0202 ~'
7 Control valve of FC2020 33%~ 13% Feed flowrate controller of methanol
8 Control valve of FC2020 33%~ I:l
26% ... '" ~
9 Control valve of FC202D 33%~ 46% ...... I:l..
10 Control valve of FC2020 33%~ 59% ......
.. , , ..
II Control valve of FC2020 33%~ 3% a
~
12 Pump P208 Failure
13 Switch valve SWI On~ Off Switch valve on the feed stream of methanol directlJ, to column ~
14 Manual valve HC211D 18%~40% Manual valve on the feed stream of methanol direct y to column
15 Manual valve HC211D 18%~60% ...... 0%-
(1)
16 Manual valve HC2110 18%~ 100% ••• ,0'

17 Manual valve HC211D 18%~0% - .....

i::1
1;;'
18 Manual valve HC2110 18%~ 10% ~
19 Control valve ofTC20lR 39% ~ 30% Bottom T controller of the reactive distillation column C201 a
20 Control valve of TC20 I R 39% ~ 20% ......
21 Control valve of TC20 I R 39%~ 10% - .. '"
j
22 Control valve ofTC20lR 39%~ 0% ~
23 Control valve of FC203E 37% ~ 33% Ste~m flowrate controller at the bottom of the column C201 ....
24 Control valve ofFC203E 37% ~ 29% ... ,-, I
25 Control valve ofFC203E 37% ~ 16% ...... ~
~
26 Control valve of FC203E 37%~ 6% ... , ..
27 Control valve ofFC203E 37%~ 0% ... ... ~
28 Control valve ofFC203E 37% ~ 57%
29 Control valve ofFC20lC 40% ~ 30% Re'flux F controller of the reactive distillation column C20 I ~
• • • • 0. ~
30 Control valve ofFC20lC 40%~ 20%
31 Control valve ofFC20lC 40% ~ 10% ... ,., ~.
32 Control valve ofFC20lC 40%~ 0% ....
33 Control valve of FC2020 33% ~ 37% F~~d flowrate controller of methanol ~'
34 Control valve of FC2020 33%~ 40% Feed flowrate controller of methanol
35 Control valve of FC2020 33% ~ 50% Feed flowrate controller of methanol I:l
~
36 Control valve of FC2020 33%~ 55% Feed flowrate controller of methanol I:l..
37 - 100 Nonnal Nonnal Ope@tioll g;::;
~
-...
C 4 Hydrocarbons

~
~
§
l:l...
~.

()

FC20 I C 1-'1
''-I~~t-I- - - f :j
~

~
~
6
~
-
:::t
: .1··6 5·
P201 \.!:9 ~
Methanol e .. :- .... LC203D b::I
_ . • • . • • • r-J.---.--...r l::
~ ~
@@ • Steam
.~ @ ~
~
<$).; ~
: FC203E ~
~
... ·cw ~
n
P203
MTBE ~
P208

tv
Figure Cl The simplified flowsheet of the MTBE process. w
232 Data Mining and Knowledge Discovery for Process Monitoring and Control

o 3
;::s
<;j
> 2

o
4

o
31 61 91 121 151 181 211 241
Data point

Figure C2 Two dynamic trends with and without noise.

REFERENCES

Lukas MP. Distributed control systems: their evaluation and design. Van Nostrand
Reinhold Company, New York Wokingham, 1986.
2 Yamanaka F and Nishiya T. Application of the intelligent alarm system for the plant
operation. Comput Chern Engng 1997; 21: s625-s630.
3 Howat CS. Analysis of plant performance. In: Perry RH, Green DW (eds.) Perry's
chemical engineers' handbook, Section 30. McGraw-Hill, 1997,30-1 - 30-36.
4 Saraiva PM and Stephanopoulos G. Continuous process improvement through
inductive and analogical learning. AIChE 1. 1992; 38: 161-183.
5 Saraiva, PM. Inductive and analogical learning: data-driven improvement of process
operations. In: Stephanopoulos G and Han C (Eds) Intelligent Systems in Process
Engineering: Paradigms from Design and Operations. Academic Press, Inc., San
Diego, California, 1996,377-435.
6 Taylor W. What every engineer should know about artificial intelligence. MIT Press,
Cambridge, MA, 1989.
7 Shewhart WA. Economic control of quality of manufactured product. Van Nostrand,
Princeton, N.J., 1931.
8 Marlin TE. Process control - designing processes and control systems for dynamic
performance. McGraw-Hill, 1995.
9 Oakland JS. Statistical process control: a practical guide. Heinemann, London, 1986.
10 Woodward RH, Goldsmith PL. Cumulative sum techniques. Oliver and Boyd"
London, 1964.
II Roberts SW. Control charts tests based on geometric moving averages.
Technometrics 1959; I: 239-250.
12 Hunter JS. The exponentially weighted moving average. J of Quality Technology
1986; 18:203-210.
234 Data Mining and Knowledge Discovery for Process Monitoring and Control

13 Stephanopoulos G. Chemical process control: an introduction to theory and practice.

Prentice-Hall, Inc., New Jersey, 1984.

14 Bekiaris N, Morari M. Multiple steady states in distillation: infinity/infinity

predictions, extensions, and implications for design, synthesis, and simulation. Ind.
Eng. Chern. Res. 1996; 35: 4264-4280.

15 Schrans S, DeWolfS, Baur R. Dynamic simulation of reactive distillation: an MTBE

case study. Comput. Chern Engng, 1996; 20: S1619- S1624.

16 Arandes JM, Delasa HI. Simulation and multiplicity of steady-states in fluidised

FCCUs. Chern. Eng. Sci., 1992; 47: 2535- 2540.

17 Hangos KM, Perkins JD. Structural stability of chemical process plants. AIChE 1.,
1997; 43: 1511-1518.

18 Quinlan JR. C4.5: programs for machine learning. Morgan Kauffman, 1993.

19 Quinlan JR. Induction of decision trees. Machine learning 1986; 1: 81-106.

20 Quinlan JR. http://www.rulequest.com/. 1999

21 Chen FZ, Wang XZ. An integrated data mining system and its application to process
operational data analysis. Comput. Chern. Engng 1999; 23(suppl.): s777-s780.

22 Clarke-Pringle T, MacGregor JF. Product quality control in reduced dimensional

spaces. Ind. Eng. Chern. Res. 1998; 37:3992-4002.

23 Fayyad UM, Simoudis E. Data mining and knowledge discovery. Tutorial Notes at
PADD'97 - 1st Int. Conf. Prac. App. KDD & Data Mining, London, 1997.

24 Massey J, Newing R. Trouble in mind. Computing 1994; May: 44-45.

25 Fayyad UM, Piatetsky-Shapiro G, Smyth P. From data mining to knowledge

discovery: an overview. In: Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy
R (eds.) Advances in Knowledge Discovery and Data Mining. AAAI Press/MIT
Press, 1996, 1-36.

26 Glymour C, Madigan D, Pregibon D, Smyth P. Statistical themes and lessons for data
mining. Data Mining and Knowledge Discovery 1997; 1: 11-28.

27 Elder IV J, Pregibon D, A statistical perspective on knowledge discovery in

databases. In: Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy R (eds.)
Advances in Knowledge Discovery and Data Mining. AAAI Press/MIT Press, 1996,
83-116.

28 Chen MS, Han JW, Yu PS. Data mining: an overview from a database perspective.
IEEE Trans. Know. Data Eng. 1996,8: 866-883.

29 Imielinski T, Mannila H. A database perspective on knowledge discovery. Comm. of

ACM 1996; 39(11): 58-64.
 References 235

30 Inmon WHo The data warehouse and data mining. Comm. of the ACM 1996; 39( II):
49-50.

31 Shortland R, Scarfe R. Data mining applications in BT. BT Technology J. 1994;

12(4): 17-22.

32 Brachman RJ, Khabaza T, Kloesgen W, Piatesky-shapiro G, Simoudis E. Mining

business databases. Comm. of ACM 1996; 39(11 ):42-48.

33 Fayyad U, Piatetsky-Shapiro G, Smyth P. The KDD process for extracting useful

knowledge from volumes of data. Comm. of the ACM 1996; 39(11): 27-34.

34 Meggs G, Fagan M. Knowledge discovery - practical methodology and case studies.

Tutorial Notes in PADD98 - 2 nd Int. Conf. Prac. App!. Know!. Disc. Data Mining.
London, 1998.

35 Fayyad U, Haussler D, Stolorz P. Mining scientific data. Comm. of the ACM 1996;
39(11): 51-57.

36 Kohonen T. Self-organisation and associative memory. Springer-Verlag, Berlin,

1988.

37 Apte, C. Data Mining: an industrial research perspective. IEEE Computational Sci. &
Eng. 1997; 4(2): 6-9.

38 Kohonen T. Self-organising maps. Springer-Verlag, Berlin, 1995.

39 Cheeseman P and Stutz J. Bayesian classification (AutoClass): theory and results. In:
Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy R (eds.) Advances in
Knowledge Discovery and Data Mining. AAAI Press/MIT Press, 1996, 153-180.

http://fi-www.arc.nasa.gov/fi/projects/bayes-group/group/autoclass/autoclass-c-
program.html

40 Cheeseman P, Freeman D, Kelly J, SelfM, Stutz, J, Taylor W. AutoClass: a Bayesian

classification system. In: Proc Fifth Inter Conf Machine Learning, 1988.

41 Cheeseman P, Stutz J, Self M, Taylor W, Goebel J, Volk K, Walker H. Automatic

classification of spectrum from the infrared astronomical satellite (IRAS), NASA
reference publication # 1217, National technical Information Service, Springfield,
Virginia, 1989.

42 Hanson R, Stutz J, Cheeseman P. Bayesian classification theory, hup://fi-

www.arc.nasa.gov/fi/projects/bayes-group/group/autoclass/autoclass-c-program.html.
1997.
43 Carpenter GA and Grossberg S. ART2: self-organisation of stable category
recognition codes for analogue input patterns. Appl Opt 1987b; 26: 4919-4930.

44 Duran BS and Odell PL. Cluster analysis: a survey. Springer-Verlag, 1974.

45 Everitt BS and Dunn G. Applied multivariate data analysis. Edward Arnold. 1991.
236 Data Mining and Knowledge Discovery for Process Monitoring and Control

46 Everitt BS. Cluster analysis. 3rd edition, Edward Arnold, London, 1993.

47 Grossberg S. Adaptive pattern classification and universal recording: I. parallel

development and coding of neural feature detector. Bio Cybernet 1976; 23: 121-134.
48 Grossberg S. Adaptive pattern classification and universal recording: II feedback,
expectation, olfaction, illusions. Cybernet 1976; 23: 187-202.

49 Carpenter GA and Grossberg S. A massive parallel architecture for a self-organising

neural pattern recognition machine. Computer Vision Graphics and Image Processing
1987; 37: 54-115.

50 Carpenter GA and Grossberg S. ART3: hierarchical search using chemical

transmitters in self-organising pattern recognition architectures. Neural Networks
1990; 3: 129-152.

51 Carpenter GA and Grossberg S. The ART of adaptive pattern recognition by a self-

organising neural network. Computer 1988; 21:77-88.

52 Buntine W. A guide to the literature on learning probabilistic networks from data.

IEEE Trans. Know!. Data Engng 1996; 8(2): 195-21 O.

53 Buntine W. Graphical models for discovering knowledge. In: Fayyad UM, Piatetsky-
Shapiro G, Smyth P, Uthurusamy R (eds.) Advances in Knowledge Discovery and
Data Mining. AAAI Press/MIT Press, 1996, 59-82.
54 Cooper GF and Herskovits E. A Bayesian method for the induction of probabilistic
networks from data. Machine Learning 1992; 9: 309-347.
55 Bouckaert RB. Properties of Bayesian network learning algorithms. In: Lopez R,
Poole, D (Eds.), Uncertainty in Artificial Intelligence: Proc. Ninth Conf., de
Mantaras, 1993, 102.
56 Bouckaert RB. Belief network construction using the minimum description length
principle. Proc. ECSQARU 1993; 41.
57 Wang XZ, Chen BH, McGreavy C. Data mining for failure diagnosis of process units
by learning probabilistic networks. Trans. IChemE 1997; 75B: 210-216.
58 Jensen FV. An introduction to Bayesian networks. UCL Press, 1996.
59 Wang XZ, Yang SA, Veloso E, Lu ML, McGreavy C. Qualitative process modelling
- a fuzzy signed directed graph method. Comput. & Chern. Engng 1995; 19(suppl.):
s735-s740.
60 Wang XZ, Yang SA, Yang SH, McGreavy C. Fuzzy qualitative simulation in safety
and operability studies of process plants. Comput. & Chern. Engng 1996; 20: s671-
s676.
61 Agrawal R, Imielinski T, Swami A. Mining association rules between sets of items in
large databases. In: Proc. ACM SIG-MOD Conf. on Management of Data,
Washington, D.C., 1993,207-216.
 References 237

62 Agrawal R, Mannila H, Srikant R, Toivonen H, Verkamo AI. Fast discovery of

association rules. In: Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy R
(eds.) Advances in Knowledge Discovery and Data Mining. AAAI Press/MIT Press,
1996,307-328.
63 Kolodner JL. An introduction to case-based reasoning. Artificial Intelligence 1992;
6: 3-34.
64 Watson I, Marir F. Case-based reasoning: a review. The Knowl. Engng Review 1994;
9: 327-354.
65 Caruana CM. Case-based reasoning gets CPI foothold. Chern. Eng. Progr 1999;
95(4): 9-10.
66 Jefferson M, Chung PW, Kletz TA. Learning the lessons from past accidents.
Hazards XIII process safety - the future, symposium series No. 141, The Institution
of Chemical Engineers, Manchester, 1997,217-226.
67 Kraslawski A, Koiranen T, Nystrom L. Case-based reasoning system for mixing
equipment selection. Comput. Chern. Engng 1995; 19(suppl.): s821-s826.
68 Graco W, Cooksey RW. Feature selection with data mining. Proc. PADD98 - The
second int. conf. on the practical application of knowl. discovery and data mining,
London, March 1998, 111-130.
69 Gnanadesikian R, Kettenring JR, Tsao SL. Weighting and selection of variables for
cluster analysis. J. of Classification 1995; 12: 113-136.
70 Milligan GW. Clustering validation: results and implications for applied analysis. In:
Arabie P, Hubert LJ, De Sote G (eds.), clustering and classification. River Edge NJ:
World Scientific, 1996,341-375.
71 Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy R (eds.) Advances in
Knowledge Discovery and Data Mining. AAAI Press/MIT Press, 1996.
72 Wu, X. Knowledge acquisition from databases. Ablex, USA, 1995.
73 Simoudis E, Han, J, Fayyad UM (eds.) KDD-96: proceedings-second international
conference on knowledge discovery & data mining. AAAI Press, 1996.
74 Wu X, Kotagiri R, Korb K (eds.) Research and development in knowledge discovery
and data mining: proceedings of the second Pacific-Asia conference, PAKDD-98.
Springer, 1998.
75 Pyle D. Data preparation for data mining. Morgan Kaufmand Publishers, 1999.
76 Moore RL, Kramer MA. Expert systems in on-line process control. Proc. third int.
conf. chemical process control, in Chemical Process Control - CPCIII (Edited by
Morari M and McAvoy TJ), Asilomar, California, January 1986, Elsevier
Amsterdam. p.839.
77 Janusz ME and Venkatasubramanian V. Automatic generation of qualitative
238 Data Mining and Knowledge Discovery for Process Monitoring and Control

descriptions of process trends for fault detection and diagnosis. Engng Applic. Artif.
Intel!., 1991; 4: 329-339.

78 Cheung lTY and Stephanopoulos G. Representation of process trends - I. a formal

representation framework. Comput. & Chern Engng 1990; 14: 495-510.

79 Cheung lTY and Stephanopoulos G. Representation of process trends - 2. the

problem of scale and qualitative scaling. Comput. & Chern Engng 1990; 14: 511-
539.

80 Whiteley JR and Davis JF. Knowledge-based interpretation of sensor patterns.

Com put. Chern. Engng 1992; 16: 329-346.

81 Bakshi BR and Stephanopoulos G. Representation of process trends. Part III. Multi-

scale extraction of trends from process data. Comput. & Chern Engng 1994; 18: 267-
320.

82 Chen BH, Wang XZ, Yang SH, McGreavy C. Application of wavelets and neural
networks to diagnostic system development - I. feature extraction. Comput. Chern.
Engng. 1999; 23: 899-906.

83 Wang XZ, Li RF. Combining conceptual clustering and principal component analysis
for state space based process monitoring. Internal report, University of Leeds, 1999.

84 Pearson K. On lines and planes of closest fit to systems of points in space. Phi.1 Mag.
1901; 2: 559-572.

85 Hotelling H. Analysis of a complex of statistical variables into principal components.

J Educ. Psycho!. 1933; 24: 417-441,498-520.

86 Misiti M, Misiti Y, Oppenheim G, Poggi, JM. Wavelet toolbox User's Guide,

Version I. The Math Works Inc.1997.

87 Chan YT. Wavelet basics. Kluwer Academic Publishers, Boston London, 1995.

88 Daubechies I. Ten lectures on wavelets. Society of Industrial and Applied

mathematics, Philadelphia, Pennsylvania, 1992.

89 Mallat S. A theory for multiresolution signal decomposition: the wavelet

representation. IEEE Pattern Ana!. and Machine Intel!. 1989; 11: 674-693.

90 Bakshi BR and Stephanopoulos G. Reasoning in time: modelling, analysis, and

pattern recognition of temporal process trends. In: Stephanopoulos G, Han C (eds.)
Intelligent systems in process engineering - paradigms from design to operations,
Academic Press, 1996,487-549.

91 Bakshi BR and Stephanopoulos G. Representation of process trends. Part IV.

Induction of real-time patterns from operating data for diagnosis and supervisory
contro!. Comput. & Chern Engng 1994; 18: 303-332.

92 Mallat S, Hwang WL. Singularity detection and processing with wavelets. IEEE
Trans. on Inf. Theory 1992; 38: 617-643.
 References 239

93 Mallat S, Zhong S. Characterisation of signals from multi-scale edges. IEEE Trans.

Pattern Analysis and Machine Intelligence 1992; 14,710-732.
94 Berman Z, Baras JS. Properties of the multi-scale maxima and zero-crossing
representations. IEEE Trans. Signal Processing 1993; 41: 3216-3231.
95 Cvetkovic Z, Vetterli M. Discrete-time wavelet extrema representation: design and
consistent reconstruction. IEEE Trans. Signal Processing 1995; 43: 681-693.
96 William Be. Doing time: putting qualitative reasoning on firmer ground. National
conf. on artif. intell., Philadelphia, 1986.
97 Mallat S. Zero-crossing of a wavelet transform. IEEE Trans. on Information Theory
1991; 37: 1019-1033.

98 Carrier JF, Stephanopoulos G. Wavelet-based modulation in control-relevant process

identification. AIChE J. 1998; 44: 341-360.
99 Dai X, Joseph B, Motard RL. Introduction to wavelet transformation and time
frequency analysis. In: Motard RL, Joseph B (eds.) Wavelet applications in chemical
engineering. Kluwer Academic Publishers, 1994, 1-32.
100 Joshi A, Kumar A, Motard RL. Trend analysis using the Frazier-Jawerth transform.
In: Motard RL, Joseph B (eds.) Wavelet applications in chemical engineering.
Kluwer Academic Publishers, 1994, 85-113.
101 Dai X, Joseph B, Motard RL. Process signal feature analysis. In: Motard RL, Joseph
B (eds.) Wavelet applications in chemical engineering. Kluwer Academic Publishers,
1994, 115-137.
102 Ramesh TS, Shum SK, Davis JF. A structured framework for efficient problem-
solving in diagnostic expert systems. Comput. Chern. Engng 1988; 12: 891-902
103 MacGregor JF, Kourti T. Multivariate statistical treatment of historical data for
productivity and quality improvements. In: Pekny J, Blau G. (eds.) Foundation of
computer aided process operations, University of Michigan, Ann Arbor Press ,
1998, in print.
104 Kresta JV, MacGregor JF, Marlin TE. Multivariate statistical monitoring of process
operating performance. Cana. 1. Chern. Eng. 1991; 69: 35-47.
105 Kourti T, MacGregor JF. Process analysis, monitoring and diagnosis, using
projection methods - a tutorial. Chemometrics and Intell. Lab. Systems 1995; 28:3-
21.
106 Jobson JD. Applied multivariate data analysis. Vol II: categorical and multivariate
methods. Springer-Verlag New York, Inc., 1992.
107 Hotelling H. Multivariate quality control, illustrated by the air testing of sample
bombsights. In: Eisenhart C, Hastay MW, Wallis WA (eds.) Techniques of Statistical
Analysis, McGraw-Hill, New York, 1947, 113-184 ..
240 Data Mining and Knowledge Discovery for Process Monitoring and Control

108 Alt FB, Smith ND. Multivariate process control. In: Krishnaiah PR, Rao CR (eds.)
Handbook of statistics, Vol. 7, North-Holland, Amsterdam, 1988,333-351.

109 Ryan TP. Statistical methods for quality improvement. Wiley, New York, 1989.

110 Jackson lE. A user's guide to principal components. John Wiley and Sons, Inc., New
York,1991.

I II Tracy ND, Young JC, Mason RL. Multivariate control charts for individual
observations. 1. Quality Tech. 1992; 24: 88-95.

112 MacGregor JF, Kourti T. Statistical process control of multi variable processes.
Control Engng Practice 1995; 3: 403-414.

I I3 MacGregor JF, Jaeckle C, Kiparissides C, Koutoudi M. Process monitoring and

diagnosis by multiblock PLS methods. AIChE J. 1994; 40: 826-838.

114 Martin EB, Morris J, Papazoglou MC. Confidence bounds for multivariate process
performance monitoring charts. Preprints of the IF AC workshop on on-line fault
detection and supervision in the chemical process industries, Newcastle, June 1995;
33-42.

115 Wold S, Geladi P, Esbensen K, Ohman J. Multi-way principal components- and PLS-
analysis. 1. Chemometrics 1987; 1:41-56.

116 Nomikos P, MacGregor JF. Monitoring batch process using multi way principal
component analysis. AIChE 1. 1994; 401361-1375.

117 Palus M, Dvorak I. Singular-value decomposition in attractor reconstruction: pitfalls

and precautions. PhysicaD 1992; 55: 221-234.

118 Xu L, Oja E, Suen CY. Modified hebbian learning for curve and surface fitting.
Neural Networks 1992; 5: 441-457.

119 Dong D, McAvoy TJ. Nonlinear principal component analysis - based on principal
curves and neural networks. Comput Chern Engng 1996; 20: 65-78.

120 Gnanadesikian R. Methods for statistical data analysis of multivariate observations.

Wiley, New York, 1977.

121 Etezadi-Amoli J, McDonald RP. A second generation nonlinear factor analysis.

Psychometrika 1983; 48(3): 315-327.
122 Kramer MA. Nonlinear principal component analysis using autoassociative neural
networks. AIChE J 1991; 37: 233-243.
123 Sammon JW. A nonlinear mapping for data structure analysis. IEEE Trans. Comput.
1969; C18: 401.

124 Yuan B, Wang XZ, Chen FZ, Morris T. Software analyser design using data mining
technology for toxicity prediction of aqueous effluents. Proc. 2 nd Conf. on Process
Integration, Modelling and Optimisation for Energy Saving and Pollution Reduction
 References 241

(Pres'99). Budapest, Hungary, May 31 - June 2,1999.

125 Bakshi BR. Multiscale PCA with application to multivariate statistical process
monitoring. AIChE J. 1998; 44:1596-1610.
126 Tabe H, Chow KC, Tan KJ, Zhang J, Thornhill N. Dynamic principal component
analysis using integral transforms. AIChE Annual Meeting, November 1998.
127 Biehl M, Schlosser E. The dynamics of on-line principal component analysis. J Phys.
A: Math. Gen. 1998; 31: L97-Ll03.
128 Neogi D, Schlags CEo Multivariate statistical analysis of an emulsion batch process.
Ind. Eng. Chern. Res. 1998; 37: 3971-3979.
129 Jaeckle CM, MacGregor JF. Product design through multivariate statistical analysis
of process data. AIChE J. 1998; 44: 1105-1118.
130 Kresta JV, Marlin TE, MacGregor JF. Development of inferential process models
using PLS. Comput. Chern. Engng. 1994; 18: 597-611.
131 Santen A, Koot GLM and Zullo LC. Statistical data analysis of a chemical plant.
Comput Chern Engng 1997; 21(suppl.): sI123-s1129.
132 Zhang J, Martin EB, Morris AJ. Fault detection and diagnosis using multivariate
statistical techniques. Trans. IChemE 1996; 74A: 89-96.
133 Dunia R, Qin SJ, Edgar TF, McAvoy T1. Identification of faulty sensors using
principal component analysis. AIChE 1. 1996; 42: 2797-2812.
134 Qin SJ, Vue HY, Dunia R. Self-validating inferential sensors with application to air
emission monitoring. Ind. Eng. Chern. Res. 1997; 36: 1675-1685.
135 Chen FZ, Wang xz. Knowledge discovery using PCA for operational strategy
development and product design. Submitted to Trans. IChemE Part A, 1999.
136 Chen JG, Bandoni JA, Romagnoli JA. Robust PCA and normal region in
multivariate statistical process monitoring. AIChE J. 1996; 42: 3563-3566.
137 Negiz A, Cinar A. Statistical monitoring of multivariable dynamic processes with
state space models. AIChE J. 1997; 43: 2002-2020.
138 Leonard J, Kramer MA. Improvement of the backpropagation algorithm for training
neural networks. Comput Chern Engng 1990; 14: 337-341.
139 Brent RP. Fast training algorithms for multilayer neural nets. IEEE Trans. Neural
Nets 1991; 2:346-354.
140 Chen S, Billings SA. Neural networks for nonlinear dynamic system modelling and
identification. Int. J. Control 1992; 56: 319-346.
141 Peel C, Willis MJ, Tham MT. A fast procedure for the training of neural networks. J.
Proc. Cont. 1992; 2: 205-211.
242 Data Mining and Knowledge Discovery for Process Monitoring and Control

142 Powell MJD. Some global convergence properties of a variable metric algorithm for
minimisation without exact line searches. In: Cottle R, Lemke CE (eds.) SIAM-AMS
Proc. Symposium on non-linear programming, 1975, IX: 53-72.

143 Press WH, Flannery, BP, Teukolsky, SA, Vetterling, WT. Numerical recipes, the art
of scientific computing (Fortran version). Cambridge University Press, Cambridge,
1989.

144 Lorentz, GG. The 13 th problem of Hilbert. In: Browder FE (Ed.) Mathematical
developments from Hilbert problems. America Mathematical Society, Providence,
R.I., 1976.

145 Crowe ER, Vassiliadis CA. Artificial intelligence: staring to realise its practical
promise. Chern. Eng. Progr. 1995; 91(1): 22-31.

146 Knight K. Connectionist ideas and algorithms. Comm. of the ACM 1990; 33 (II):
59-74.

147 Chitra SP. Use of neural networks for problem solving. Chern. Eng. Progr. 1993;
89(4): 44-52.

148 Kramer MA, Leonard, JA. Diagnosis using backpropagation neural networks -
analysis and criticism. Comput. Chern. Engng 1990; 14: 1323-1338.

149 Shao R, Martin EB, Zhang J, Morris AJ. Confidence bounds for neural network
representations. Comput. Chern Engng 1997; 21: sI173-s1178.

150 Zhang J, Martin EB, Morris AJ, Kiparissides C. Inferential estimation of polymer
quality using stacked neural networks. Comput. Chern. Engng 1997; 21: s I 025-
s1030.

151 Wang XZ, Lu ML, McGreavy C. Learning dynamic fault models based on a fuzzy set
covering method. Comput. Chern Engng 1997; 21: 621-630.

152 Kramer MA. Malfunction diagnosis using quantitative models with non-Boolean
reasoning in expert systems. AIChE J. 1987; 33: 130-140.

153 Venkatasubramanian V, Vaidyanathan R, Yamamoto Y. Process fault-detection and

diagnosis using neural networks, 1. steady-state processes. Comput. Chern. Engng
1990,14: 699-712.

154 Zhang J, Morris J. Process modelling and fault diagnosis using fuzzy neural
networks. Fuzzy Sets and Systems 1996; 79: 127-140.

155 Wang XZ, Chen BH, Yang SH, McGreavy C. Neural nets, fuzzy sets and digraphs in
safety and operability studies of refinery reaction processes. Chern Engng Sci 1996;
51: 2169-2178.

156 Buckley JJ, Hayashi Y. Fuzzy neural networks: a survey. Fuzzy Sets and Systems
1994; 66: 1-13.

157 Reggia JA, Nau DS, Wang PY. Diagnostic expert system based on a set covering
 References 243

model. Int J Man-machine Studies 1983; 19: 437-460.

158 Penalva JM, Coudouneau L, Leyval L, Montmain J. A supervision support system for
industrial processes. IEEE Expert 1993; 8(5): 57-65.

159 Iri M, Aoki E, O'Shima E, Matsuyama H. An algorithm for diagnosis of system

failures in the chemical process. Comput. Chern. Engng 1979; 3: 489-493.

160 Oyeleye 00, Kramer MA. Qualitative simulation of chemical process systems:
steady-state analysis. AIChE J. 1988; 34: 1441-1454.

161 Yu CC, Lee C. Fault diagnosis based on qualitative/quantitative process knowledge.

AIChE J. 1991; 37: 617-628.

162 Gujima F, Shibata B, Tsuge Y, Shiozaki J, Matsuyama H, O'shima E. Improvements

of the accuracy of fault diagnosis systems, using signed directed graphs. Int. Chern.
Engng. 1993; 33: 671-679.

163 Mohindra S, Clark P A. A distributed fault diagnosis method based on graph models:
steady-state analysis. Comput. Chern. Engng 1993; 17: 193-209.
164 Wilcox NA, Himmelblau DM. The possible cause-effect graph (PCEG) model for
fault diagnosis. Comput. Chern. Engng 1994; 18: 103-127.

165 Ouassir M, Melin C. Causal graphs and rule generation: application to fault diagnosis
of dynamic processes. Proceedings of the 3rd IF AC symposium on fault detection,
supervision and safety for technical processes 1997 (SAFEPROCESS 97), Kingston
Hull, England, Aug. 1997,1087-1092.
167 Vedam H, Venkatasubramanian V. Signed digraph based multiple fault diagnosis.
Comput. Chern. Engng 1997; 21: s655-s660.
168 Umeda T, Kuriyama T, O'shima E. Matsuyama H. A graphical approach to cause and
effect analysis of. chemical processing systems. Chern. Eng. Sci. 1980; 35: 2379-
2388.
169 Vaidhyanathan R, Venkatasubramanian V. Digraph-based models for automated
HAZOP analysis. Reliability Engineering and System Safety 1995; 50: 33-49.
170 Srinivasan R, Venkatasubramanian V. Petri Net-Digraph models for automating
HAZOP analysis of batch process plants. Comput. Chern. Engng 1996; 20: s719-
s725.
171 Kuo DH, Hsu DS, Chang CT, Chen DH. Prototype for integrated hazard analysis.
AIChE J. 1997; 43: 1494-1510.
172 Han CC, Shih RF, Lee LS. Quantifying signed directed-graphs with the fuzzy set for
fault diagnosis resolution improvement. Ind. Eng. Chern. Res. 1994; 33: 1943-1954.
173 Shih RF, Lee LS. Use of fuzzy cause-effect digraph for resolution fault diagnosis for
process plants. Ind. Eng. Chern. Res. 1995; 34: 1688-1717.
244 Data Mining and Knowledge Discovery for Process Monitoring and Control

174 Wang XZ, Chen BH, Yang SH, McGreavy C. Fuzzy rule generation from data for
process operational decision support. Comput. Chern. Engng 1997; 21: s661-s666.

175 Huang YC, Wang XZ. Application of fuzzy causal networks to wastewater treatment
plants. Chern. Eng. Sci. 1999; 54: 2731-2738.

176 Chen CL, Jong MJ. Fuzzy predictive control for the time-delay system. Proc. 2nd
IEEE into conf. fuzzy systems 1993; 236-240.

177 Becraft WR, Lee PL. An integrated neural network/expert system approach for fault
diagnosis. Comput. Chern. Engng 1993; 17: 1001-1014.

178 Dubois D, Prade H. Fuzzy sets and systems: theory and applications. Academic
Press, 1980

179 Rosenfield A. Fuzzy graphs. In: Zadeh LA, Fu KS, Tanaka K, Shimura M (eds.)
Fuzzy sets and their applications to cognitive and decision processes. Academic
Press, New York, 1975,77-95.

180 Zhang J, Morris AJ, Martin EB, Kiparissides C. Prediction of polymer quality in
batch polymerisation reactors using robust neural networks. Chern. Eng. J. 1998; 69:
135-143.

181 Caudil M. Neural network primer, Part VIII, AI Expert 1989.

182 Wasserman PD. Neural computing: theory and practice. Van Nostrand Reinhold,
New York, 1989.

183 Whiteley JR, Davis JF. A similarity-based approach to interpretation of sensor data
using adaptive resonance theory. Comput. & Chern. Engng 1994; 18: 637-661.

184 Whiteley JR, Davis JF Mehrotra A and Ahalt SC. Observations and problems
applying ART2 for dynamic sensor pattern interpretation. IEEE Trans. on Sys. Man
Cybernetics Part A: Sys. and Humans 1996; 26: 423-437.

185 Wang XZ, Chen BH, Yang SH, McGreavy C. Application of wavelets and neural
networks to diagnostic system development - 2. an integrated framework and its
application. Comput. Chern. Engng. 1999; 23: 945-954.

186 Pao YH. Adaptive pattern recognition and neural networks. Addison-Wesley, 1989.

187 Looney CG. Pattern recognition using neural networks: theory and algorithms for
engineers and scientists. Oxford University Press, 1997.
188 Everitt BS, Hand DJ. Finite mixture distributions. London Chapman & Hall, 1981.
189 Titterington DM, Smith AFM, Makov UE. Statistical analysis of finite mixture
distributions. John Wiley & Sons, New York, 1985.
190 Dempster AP, Laird NM, Rubin DB. Maximum likelihood from incomplete data via
the EM algorithm. Journal of the Royal Statistical Society 1977; 39 (series B), No.
1: 1.
 References 245

191 Ayoubi M, Leonhardt S. Methods of fault diagnosis. Control Eng. Practice 1977; 5:
683.

192 Wolfe JH. Pattern clustering by multivariate mixture analysis. Multivariate

behavioural research 1970; 5: 329-350.

193 Wang XZ, McGreavy C. Automatic classification for mining process operational
data. Ind. Eng. Chern. Res. 1998; 37: 2215-2222.

194 Nam DS, Han C, Jeong CW, Yoon ES. Automatic construction of extended
symptom-fault associations from the signed digraph. Comput. Chern. Engng. 1996;
20(supp\.): s605-s610.

195 Wang XZ, Chen BH. Clustering of infrared spectra of lubricating base oils using
adaptive resonance theory. 1. Chern. Inf. Comput. Sci. 1998; 38: 457-462.

196 Michalski, R.S. and Larson, J.B. Selection of most representative training examples
and incremental generation of VLI hypotheses: the underlying method and desc. of
prog. ESEL and AQll, 867, Computer Science Department, University of Illinois,
1978.

197 Mitchell, T.M. Version Spaces: a candidate elimination approach to rule learning, in
Proc. ofIJCAI-87, Cambridge, Mass., 1977

198 Quinlan JR. Improved use of continuous attributes in C4.5. J. Artif. Intel\. Res. 1996;
4: 77-90.

199 Wu, X. A Bayesian discretiser for real-valued attributes. The Comput 1., 1996; 39:
688-69l.

200 Daniel C, Wood FS. Fitting equations to data, 2nd Edition. John Wiley & Sons, Inc.,
1980.

201 Raich A, Cinar A. Statistical process monitoring and disturbance diagnosis in

muItivariable continuous processes. AIChE J. 1996; 42: 995-1009.

202 Shi ZZ. Principles of machine learning. International Academic Publishers, Beijing,
1992.

203 Dutton DM, Conroy GV. A review of machine learning. The Know\. Engng Review
1996; 12: 341-367.

204 Kocabas S. A review oflearning. The Know\. Engng Review 1991; 6: 195-222.

205 Wang LX, Mendel JM. Generating fuzzy rules by learning from examples. IEEE
Trans. on Systems, Man, and Cybernetics 1992; 22: 1414-1427.

206 Wang LX, Mendel 1M. Fuzzy basis functions, universal approximation, and
orthogonal least-squares learning. IEEE Trans. on Neural Networks 1992; 3: 807-
814.

207 Wang LX. Adaptive fuzzy systems and control: design and stability analysis.
246 Data Mining and Knowledge Discovery for Process Monitoring and Control

Prentice-Hall, Inc., 1994.

208 Wang LX. Fuzzy systems are universal approximators. Proc. IEEE Int. Conf. on
Fuzzy Systems, San Diego, 1992, 1163-1170.

209 Quafafou M. and Chan CC. An incremental approach for learning fuzzy rules from
examples. EUFIT' 95, Aachen, Germany, 1995,520-523.

210 Srinivasan ACB, Chan CC. Using inductive learning to determine fuzzy rules for
dynamic system. Engng Applic. of Artif. Intell. 1993; 6: 257-264.

211 Abe S, Lan MS. A method for fuzzy rules extraction directly from numerical data
and its application to pattern classification. IEEE Trans. on Fuzzy Systems 1995; 3:
18-28.

212 Foulloy L, Galichet S, Nakoula Y. Learning a fuzzy symbolic rulebase. EUFIT'95,

Aachen, Germany, 1995,594-598.

213 Schretter N, Hollatz J. Consistency and completeness in fuzzy rulebases generated

from numerical data. EUFIT '95, Aachen, Germany, 1995,599-604.

214 Levrat E, Rondeau L, Ruelas R, Lamotte M. Fuzzy rules learning method. EUFIT
'95, Aachen, Germany, 1995,515-519.

215 Sudkamp T, Hammell RJ. Interpolation, completion, and learning fuzzy rules. IEEE
Trans. on Systems, Man, and Cybernetics 1994; 24: 332-342.

216 Kosko, B. Neural networks and fuzzy systems: a dynamic systems approach to
machine intelligence. Prentice Hall, Englewood Cliffts, NJ, 1992.

217 Fu LM. Rule generation from neural networks. IEEE Trans. on Systems, Man, and
Cybernetics 1994; 24: 1114-1124.

218 Gallant SI. Connectionist expert systems. Comm. of The ACM 1988; 31(2): 152-169.

219 Chan CC. Incremental learning of production rules from examples under uncertainty:
a rough set approach. Int. J. of Software Engng and Know!. Engng 1991; I: 439-461.

220 Chmielewski MR, Grzymala-Busse JW, Peterson NW, Than S. The rule induction
system LERS - a version for personal computers. Foundations of Computing and
Decision Sciences 1993; 18: 181-212.

221 Pawlak Z. Rough sets. Int. J. Computer and Information Sci. 1982; 11: 341-356.

222 Pawlak Z. Rough classification. Int. 1. man-machine Studies 1984; 20: 469-483.

223 Pawlak Z. Rough sets and fuzzy sets. Fuzzy Sets and Systems 1985; 17: 99-102.

224 Fusillo RH, Powers GJ. Operating procedure synthesis using local models and
distributed goals. Comput. Chern Engng 1988; 12: 1023-1034.

225 Frayman Y, Wang L. Data mining using dynamically constructed recurrent fuzzy
neural networks. In: Wu X, Kotagiri R, Korb KB (eds.) Research and development in
 References 247

knowledge discovery and data mining, Springer, 1998, 122-13\.

226 Ziarko WP (ed.) Rough sets, fuzzy sets and knowledge discovery. Springer-Verlag,
1994.

227 Chen FZ, Wang XZ. Software sensor design using Bayesian automatic classification
and backpropagation neural networks. Ind. Eng. Chern. Res. 1998; 37: 3985-399\.

228 McAvoy TJ, Su HT, Wang NS, He M, Horvath J, Semerjian H. A comparison of

neural networks and partial least squares for deconvoluting fluorescence spectra.
Biotech. and Bioeng 1992; 40: 53-62.

229 Baughman DR, Liu YA. Neural networks in bioprocessing and chemical engineering.
Academic Press, 1995.

230 Miller RM, Itoyama K, Uda A, Takada H, Bhat N. Modeling and control of a
chemical waste water treatment plant. Comput. Chern. Engng 1997; 21(suppl.): s947-
s952.

231 Gillespie ES, Wilson RN. Application of sensitivity analysis to neural network
determination of financial variable relationships. Applied Stochastic Models and
Data Analysis 1998; 13: 409-414.

232 Garson GD. Interpreting neural-network connection weights. AI Expert 1991; April:
47-5\.

233 Milne LK, Gedeon TD, Skidmore AK. Classifying dry sclerophyll forest from
augmented satellite data: comparing neural networks, decision tree & maximum
likelihood. Proc. Australian Conf. Neural Networks, Sydney, 1995, 160-163.

234 Wong PM, Gedeon TD, Taggart 11. An improved technique in porosity prediction: a
neural network approach. IEEE Trans. Geoscience and Remote Sensing 1995; 33:
971-980.

235 Gedeon TD. Data mining of inputs: analysing magnitude and functional measures.
Int. J. Neural Systems 1997; 8: 209-218.

236 Ayoubi M. Comparison between the dynamic multi-layered percetron and the
generalised Hammerstein model for experimental identification of the loading
process in diesel engines. Control Engng Practice 1998,6: 271-279.

237 Bhat N, McAvoy TJ. Use of neural nets for dynamic modelling and control of
chemical process systems. Comput. Chern. Engng 1990; 14: 573-583.
238 Shaw AM, Doyle FJ, Schwaber JS. A dynamic neural network approach to nonlinear
process modelling. Comput. Chern. Engng 1997; 21: 371-385.

239 Morris AJ, Montague GA, Willis MJ. Artificial neural networks: studies in process
modelling and control. Trans. IChemE. 1994; 72A: 3-19.

240 Kim SJ, Lee MH, Park SW, Lee SY, Park CH. A neural linearizing control scheme
for nonlinear chemical processes. Comput. Chern. Engng 1997; 21: 187-200
248 Data Mining and Knowledge Discovery for Process Monitoring and Control

241 Martinez EC, Pulley RA, Wilson JA. Learning to control the performance of batch
processes. Trans IChemE 1998; 76A: 711-722.
INDEX

Adaptive resonance theory (ART2) 11, Data

18-19,34,120-125 dynamic trends 24, 11,29, 153-172
ARTnet 11,120,126-135 multivariate 25
comparison with ART2 134-136 pre-processing 24, 8-11,
Association rules 22-23 univariate 25
Autoassociative neural networks 74-75 categorical 25
Automatic classification system characteristics 8
(AutoClass) 11,18,136-147,195, Data mining 1, 10, 13-28
197,200,202 definition 13
Backpropagation neural networks see development 13-14
Feedforward neural networks preprocessing 15-17,
Batch process 72-73, 216 techniques 13-28
Bayesian graphs 20, dimension reduction 29
Case-based learning 23 noise removal 29
Causal explanation see explanation and Decision trees 19,149-172
Cause-effect links Dependency modelling see graphical
Cause-effect links 1,4, II, 12, 15,25, models
112, 183, 185, 190 Diagnosis 2,13,33,35,61,85,94-117,
Classification 11,16-19,33,34,82,91, 123,153,177-179
120,136-150,165,168,173,179, Display hierarchy 2,3
196,198-201,212 Distance measure II, 18
Clustering 11, 12, 16-19, 78, 92, 120, Distillation 76, 196-197
124, 129, 130, 134, 136, 144, 149- Distributed control systems (DCS) 2,4,
172, 195, 200 14,33,38
Concept formation 29-37, 153 Dynamic multilayer percetron 210
Conceptual clustering 11, 19,37, 149- Dynamic simulation see Simulator
172 Dynamic trend signals 2,8-12,24,29-
Continuous stirred tank reactor (CSTR) 60,124-135
7,14,37-38,96-98,156-164,168, qualitative representation 57-59
210-212,217-220 episode 49,57-59
250 Data Mining and Knowledge Discovery for Process Monitoring and Control

dimension reduction 29-60,126-135 C4.5, C5.0, 103 150-152

feature extraction 29-60,126-135 conceptual clustering 156-172
concept formation 30-38, 157-172 continuous valued attributes 152
Error distribution 93 decision tree development 149-172
Essential faults 117 identification of states 156-172
Expert systems 29, 173, 190 using averaged data 153-156
Explanation 9, II. 23, 25, 112, 117, process monitoring 156-172
173,189,190 rules generation 149-152, 190
Exponentially weighted moving average Inferential models see software sensors
(EWMA) 4 Inflection points 48-49
FCC main fractionator see distillation Interface 1-2
Feature selection 25-26, 206-208 Knowledge acquisition 173
Feedforward neural networks II, 23, 92, Knowledge discovery I, 10, 13-28
204-209 definition 13
algorithm 87-89 development 13-14
architecture 85-86 process 15-17
as software sensors 193-203 techniques 13-28
confidence bounds 92, 94 Latent variables 67,68,71,72,79 see
dynamic neural networks 210-213 also Principal component analysis
extrapolation 91,117,194 Link and dependency analysis 9, also
for fault diagnosis 94-117 see Graphical models, Fuzzy
for sensitivity studies 206-208 digraph and Bayesian graphs
fuzzy neural networks 99-100, 180- Methyl tertiary butyl ether (MTBE) 149,
191 157,164,167,169-171,177,227-
Fluid catalytic cracking (FCC) II 14, 232
76,108,142-147,195-196,221- Microtox 93
226 Mixture model 139-140
Fourier Transform 39-41 Monitoring charts 2, 4, 6, also see
short-time Fourier transform 39-41 Statistical process control
windowed Fourier transform 39-41 Multiple steady states 7,14,15
comparison with wavelets 40-42 Multivariate statistical analysis 11, also
Fuzzy digraphs 20, 105-116 see Statistical process control
Fuzzy neural networks 99-100, 180-191 Noise to signal ratio 98
Fuzzy set covering model 10 I-I 02 Operating window 2, 5, 6
Fuzzy sets 11,99-116,180-190 Operational plane 7,9,82
Fuzzy signed digraph 102-108, 114-116 Operational state identification 11
Graphical models 20, also see fuzzy Partial least squares (PLS) 68, 82-83
digraphs and Bayesian graphs Principal component analysis (PCA)
Heat exchanger 112, 114, 183, 185 11,30-38,61-83,156-172,204-206,
Hotling's T2 chart 65-72 208
Incremental learning 191 multiblock II
Inductive learning 12,19,149-172,190 nonlinear 74-75
algorithm 150-152 for product design II
 Index 251

algorithm 30-38 State space, monitoring, control,

processing trends 33-38, 156-172 diagnosis 1-2, 6-9
operational data analysis 61-83 Statistical process control (SPC) 4, II
dimension reduction 204-208 Shewhart charts 4
concept formation 30-38, 157-172 Cumulative sum charts 4
variable contribution plots 68 Exponentially weighted moving
Probabilistic graphs see Bayesian graphs average (EWMA) 4
Process data, characteristics 8 Statistical quality control 2, 4
Product quality 62, 64, 66-69, 153, 156, Summarising 10
196-204 Supervised machine learning II, 85-
Product recovery 62-63 II 7, 119, see also feedforward
Production rules 19,160,169,173-191 neural networks
Reactor 63,68-70, 77, 83 also see System architecture 10
Continuous stirred tank reactor, Time series data 24, also see Data
fluid catalytic cracking, methyl Toxicity 93
tertiary butyl ether Training simulator see Simulator
Recurrent neural networks 2 \0 Trend, analysis 9
Recursive learning 117,191 Unsupervised machine learning II, 18,
Refinery 7, II, 14,76, \08, 164, 149, II 9-147 see also Adaptive
154, 157, 195-198, 202, 221-223, resonance theory, Self organising
227 Kohonen neural networks,
Regression 10, 23 Automatic classification system
Root faults 117 (AutoClass), Conceptual clustering
Rough sets 11, 178-180 Unsupervised neural networks see
Rulesgeneration 15,173-191 unsupervised machine learning
fuzzy set method 173-176 Waste water treatment plant 114-116,
rough set method 178-180 206-207
neural network approach 176-177 Wavelets 29, 38-56, 124-136
fuzzy neural network 180-190 feature extraction 38-56, 124-136
inductive learning 149,157-172, comparison with Fourier transform
191 40-44
Self organising Kohonen 18-19 Mexican hat 43
Sensitivity analysis 206-209 continuous 44-47
Sequence analysis \0 discrete 47-48
Set covering model \01 for singularity detection 48
Signed directed graph 102 extrema 38-56, 124, 127-131
Similarity measure II Daubechies 48, 51-52
Simulator 99, 165, 219, 222, 223, 228, for data compression 53
229 noise removal 53-55
Software analyser 93 piece-wise processing 56
Software sensor II, 193-203 multi-scale analysis 47-55, 126-130
Squared prediction error SPEy 67, 70 multiresolution 47-55, 126-130