Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models

Front Cover
Springer Science & Business Media, 2001 - Computers - 785 pages
The goal of this book is to provide engineers and scientIsts in academia and industry with a thorough understanding of the underlying principles of nonlinear system identification. The reader will be able to apply the discussed models and methods to real problems with the necessary confidence and the awareness of potential difficulties that may arise in practice. This book is self-contained in the sense that it requires merely basic knowledge of matrix algebra, signals and systems, and statistics. Therefore, it also serves as an introduction to linear system identification and gives a practical overview on the major optimization methods used in engineering. The emphasis of this book is on an intuitive understanding of the subject and the practical application of the discussed techniques. It is not written in a theorem/proof style; rather the mathematics is kept to a minimum and the pursued ideas are illustrated by numerous figures, examples, and real-world applications. Fifteen years ago, nonlinear system identification was a field of several ad-hoc approaches, each applicable only to a very restricted class of systems. With the advent of neural networks, fuzzy models, and modern structure opti mization techniques a much wider class of systems can be handled. Although one major characteristic of nonlinear systems is that almost every nonlinear system is unique, tools have been developed that allow the use of the same ap proach for a broad variety of systems.
 

Contents

1 Introduction
1
112 Prediction
2
113 Simulation
3
114 Optimization
4
117 Fault Detection
5
12 Tasks in Nonlinear System Identification
6
121 Choice of the Model Inputs
8
122 Choice of the Excitation Signals
9
1331 Local Linear Model Tree LOLIMOT Algorithm
365
1332 Different Objectives for Structure and Parameter Optimization
372
1333 Smoothness Optimization
374
1334 Splitting Ratio Optimization
376
1335 Merging of Local Models
378
1336 Flat and Hierarchical Model Structures
380
1337 Principal Component Analysis for Preprocessing
383
1338 Models with Multiple Outputs
385

123 Choice of the Model Architecture
10
124 Choice of the Dynamics Representation
11
127 Choice of the Model Parameters
12
128 Model Validation
13
13 White Box Black Box and Gray Box Models
15
14 Outline of the Book and Some Reading Suggestions
16
15 Terminology
18
Optimization Techniques
21
2 Introduction to Optimization
23
21 Overview of Optimization Techniques
25
23 Loss Functions for Supervised Methods
28
231 Maximum Likelihood Method
30
232 Maximum APosteriori and Bayes Method
32
24 Loss Functions for Unsupervised Methods
34
3 Linear Optimization
35
31 Least Squares LS
36
311 Covariance Matrix of the Parameter Estimate
44
312 Errorbars
45
313 Orthogonal Regressors
48
314 Regularization Ridge Regression
49
315 Noise Assumptions
54
316 Weighted Least Squares WLS
55
317 Least Squares with Equality Constraints
57
318 Smoothing Kernels
58
32 Recursive Least Squares RLS
60
321 Reducing the Computational Complexity
63
322 Tracking TimeVariant Processes
64
323 Relationship between the RLS and the Kalman Filter
65
33 Linear Optimization with Inequality Constraints
66
34 Subset Selection
67
341 Methods for Subset Selection
68
342 Orthogonal Least Squares OLS for Forward Selection
72
343 Ridge Regression or Subset Selection?
75
35 Summary
77
4 Nonlinear Local Optimization
79
41 Batch and Sample Adaptation
81
42 Initial Parameters
83
43 Direct Search Algorithms
86
432 HookeJeeves Method
88
44 General GradientBased Algorithms
90
441 Line Search
91
442 Finite Difference Techniques
92
443 Steepest Descent
93
444 Newtons Method
96
445 QuasiNewton Methods
98
446 Conjugate Gradient Methods
100
45 Nonlinear Least Squares Problems
102
451 GaussNewton Method
104
452 LevenbergMarquardt Method
105
46 Constrained Nonlinear Optimization
107
47 Summary
110
5 Nonlinear Global Optimization
113
51 Simulated Annealing SA
116
52 Evolutionary Algorithms EA
120
521 Evolution Strategies ES
123
522 Genetic Algorithms GA
126
523 Genetic Programming GP
132
53 Branch and Bound BB
133
54 Tabu Search TS
135
6 Unsupervised Learning Techniques
137
61 Principal Component Analysis PCA
139
62 Clustering Techniques
142
621 KMeans Algorithm
143
622 Fuzzy CMeans FCM Algorithm
146
623 GustafsonKessel Algorithm
148
624 Kohonens SelfOrganizing Map SOM
149
625 Neural Gas Network
152
626 Adaptive Resonance Theory ART Network
153
627 Incorporating Information about the Output
154
63 Summary
155
7 Model Complexity Optimization
157
72 BiasVariance Tradeoff
158
721 Bias Error
160
722 Variance Error
161
723 Tradeoff
164
73 Evaluating the Test Error and Alternatives
167
731 Training Validation and Test Data
168
732 Cross Validation
169
733 Information Criteria
171
734 MultiObjective Optimization
172
735 Statistical Tests
174
736 CorrelationBased Methods
176
Implicit Structure Optimization
179
752 Regularization by NonSmoothness Penalties
180
753 Regularization by Early Stopping
182
754 Regularization by Constraints
184
755 Regularization by Staggered Optimization
186
756 Regularization by Local Optimization
187
76 Structured Models for Complexity Reduction
189
761 Curse of Dimensionality
190
762 Hybrid Structures
192
763 ProjectionBased Structures
195
764 Additive Structures
196
765 Hierarchical Structures
197
766 Input Space Decomposition with Tree Structures
198
77 Summary
200
8 Summary of Part I
203
Static Models
207
9 Introduction to Static Models
209
92 Basis Function Formulation
210
921 Global and Local Basis Functions
211
922 Linear and Nonlinear Parameters
212
93 Extended Basis Function Formulation
215
94 Static Test Process
216
10 Linear Polynomial and LookUp Table Models
219
102 Polynomial Models
221
103 LookUp Table Models
224
1031 OneDimensional LookUp Tables
225
1032 TwoDimensional LookUp Tables
227
1033 Optimization of the Heights
229
1034 Optimization of the Grid
231
1035 Optimization of the Complete LookUp Table
232
1037 Properties of LookUp Table Models
235
104 Summary
237
11 Neural Networks
239
111 Construction Mechanisms
242
1112 Radial Construction
244
1113 Tensor Product Construction
245
112 Multilayer Perceptron MLP Network
246
1121 MLP Neuron
247
1122 Network Structure
249
1123 Backpropagation
252
1124 MLP Training
253
1125 Simulation Examples
256
1126 MLP Properties
260
1127 Multiple Hidden Layers
261
1128 Projection Pursuit Regression PPR
262
113 Radial Basis Function RBF Networks
264
1132 Network Structure
267
1133 RBF Training
269
1134 Simulation Examples
277
1135 RBF Properties
279
1136 Regularization Theory
281
1137 Normalized Radial Basis Function NRBF Networks
283
114 Other Neural Networks
286
1142 Cerebellar Model Articulation Controller CMAC
288
1143 Delaunay Networks
292
1144 JustinTime Models
293
115 Summary
296
12 Fuzzy and NeuroFuzzy Models
299
1211 Membership Functions
300
1212 Logic Operators
302
1213 Rule Fulfillment
303
122 Types of Fuzzy Systems
304
1222 Singleton Fuzzy Systems
307
1223 TakagiSugeno Fuzzy Systems
309
123 NeuroFuzzy NF Networks
310
1231 Fuzzy Basis Functions
311
1232 Equivalence between RBF Networks and Fuzzy Models
312
1233 What to Optimize?
313
1234 Interpretation of NeuroFuzzy Networks
316
1235 Incorporating and Preserving Prior Knowledge
320
1236 Simulation Examples
321
124 NeuroFuzzy Learning Schemes
323
1242 Nonlinear Global Optimization
325
1244 Fuzzy Rule Extraction by a Genetic Algorithm FUREGA
327
1245 Adaptive Spline Modeling of Observation Data ASMOD
337
125 Summary
339
Fundamentals
341
131 Basic Ideas
342
1311 Illustration of Local Linear NeuroFuzzy Models
343
1312 Interpretation of the Local Linear Model Offsets
346
1313 Interpretation as TakagiSugeno Fuzzy System
347
1314 Interpretation as Extended NRBF Network
349
132 Parameter Optimization of the Rule Consequents
351
1322 Local Estimation
352
1323 Global Versus Local Estimation
356
1324 Data Weighting
361
133 Structure Optimization of the Rule Premises
362
134 Summary
389
Advanced Aspects
391
1411 Identification of Processes with Direction Dependent Behavior
395
142 More Complex Local Models
397
1422 Local Quadratic Models for Input Optimization
400
1423 Different Types of Local Models
402
143 Structure Optimization of the Rule Consequents
404
144 Interpolation and Extrapolation Behavior
408
1442 Extrapolation Behavior
411
145 Global and Local Linearization
416
146 Online Learning
420
1461 Online Adaptation of the Rule Consequents
421
1462 Online Construction of the Rule Premise Structure
428
147 Errorbars Design of Excitation Signals and Active Learning
430
1471 Errorbars
431
1472 Detecting Extrapolation
434
1473 Design of Excitation Signals
435
1474 Active Learning
436
148 From Local Linear NeuroFuzzy Models to Hinging Hyperplanes
437
1481 Hinging Hyperplanes
438
1482 Smooth Hinging Hyperplanes
439
1483 Hinging Hyperplane Trees HHT
441
1484 Local Linear NeuroFuzzy Models Versus Hinging Hyperplane Trees
443
149 Summary and Conclusions
444
15 Summary of Part II
451
Dynamic Models
455
16 Linear Dynamic System Identification
457
161 Overview of Linear System Identification
458
162 Excitation Signals
459
163 General Model Structure
462
1631 Terminology and Classification
465
1632 Optimal Predictor
471
1633 Some Remarks on the Optimal Predictor
474
1634 Prediction Error Methods
476
164 Time Series Models
478
1641 Autoregressive AR
479
1642 Moving Average MA
480
1643 Autoregressive Moving Average ARMA
481
165 Models with Output Feedback
482
1652 Autoregressive Moving Average with Exogenous Input ARMAX
492
1653 Autoregressive Autoregressive with Exogenous Input ARARX
496
1654 Output Error OE
499
1655 BoxJenkins BJ
503
1656 State Space Models
505
1657 Simulation Example
506
166 Models without Output Feedback
509
1661 Finite Impulse Response FIR
510
1662 Orthonormal Basis Functions OBF
512
1663 Simulation Example
520
167 Some Advanced Aspects
524
1672 Consistency
526
1674 Relationship between Noise Model and Filtering
528
1675 Offsets
529
168 Recursive Algorithms
531
1681 Recursive Least Squares RLS Method
532
1683 Recursive Extended Least Squares RELS Method
533
1684 Recursive Prediction Error Methods RPEM
534
169 Determination of Dynamic Orders
536
1610 Multivariable Systems
537
16101 PCanonical Model
539
16102 Matrix Polynomial Model
540
16103 Subspace Methods
541
16111 Direct Methods
542
16112 Indirect Methods
544
16113 Identification for Control
545
1612 Summary
546
17 Nonlinear Dynamic System Identification
547
172 External Dynamics
549
1721 Illustration of the External Dynamics Approach
550
1722 SeriesParallel and Parallel Models
555
1723 Nonlinear Dynamic InputOutput Model Classes
557
1724 Restrictions of Nonlinear Dynamic InputOutput Models
562
173 Internal Dynamics
563
174 Parameter Scheduling Approach
564
1751 BackpropagationThroughTime BPTT Algorithm
565
1752 Real Time Recurrent Learning
567
176 Multivariable Systems
568
177 Excitation Signals
569
178 Determination of Dynamic Orders
574
179 Summary
576
18 Classical Polynomial Approaches
579
181 Properties of Dynamic Polynomial Models
580
182 KolmogorovGabor Polynomial Models
581
183 VolterraSeries Models
582
184 Parametric VolterraSeries Models
583
186 Hammerstein Models
584
187 Wiener Models
585
19 Dynamic Neural and Fuzzy Models
587
1911 MLP Networks
588
192 Interpolation and Extrapolation Behavior
589
193 Training
591
1931 MLP Networks
592
194 Integration of a Linear Model
593
195 Simulation Examples
594
1951 MLP Networks
595
1952 RBF Networks
597
1953 Singleton Fuzzy and NRBF Models
599
196 Summary
600
20 Dynamic Local Linear NeuroFuzzy Models
601
201 OneStep Prediction Error Versus Simulation Error
604
202 Determination of the Rule Premises
606
203 Linearization
608
2032 Dynamics of the Linearized Model
610
2033 Different Rule Consequent Structures
612
204 Model Stability
613
2041 Influence of Rule Premise Inputs on Stability
614
2042 Lyapunov Stability and Linear Matrix Inequalities LMIs
616
2043 Ensuring Stable Extrapolation
617
205 Dynamic LOLIMOT Simulation Studies
618
2052 Hammerstein Process
620
2053 Wiener Process
624
2054 NDE Process
625
206 Advanced Local Linear Methods and Models
626
2061 Local Linear Instrumental Variables IV Method
628
2062 Local Linear Output Error OE Models
630
2063 Local Linear ARMAX Models
631
208 Structure Optimization of the Rule Consequents
636
209 Summary and Conclusions
640
21 Neural Networks with Internal Dynamics
645
212 Partially Recurrent Networks
646
213 State Recurrent Networks
647
214 Locally Recurrent Globally Feedforward Networks
648
215 Internal Versus External Dynamics
650
Applications
653
22 Applications of Static Models
655
2211 Process Description
656
2212 Smoothing of a Driving Cycle
657
2213 Improvements and Extensions
658
2214 Differentiation
659
2221 The Role of LookUp Tables in Automotive Electronics
660
2222 Modeling of Exhaust Gases
663
2223 Optimization of Exhaust Gases
666
Dynamic Models
672
223 Summary
674
23 Applications of Dynamic Models
677
2312 Experimental Results
679
232 Diesel Engine Turbocharger
683
2321 Process Description
684
2322 Experimental Results
685
233 Thermal Plant
691
2331 Process Description
692
2332 Transport Process
693
2333 Tubular Heat Exchanger
698
2334 CrossFlow Heat Exchanger
702
234 Summary
707
24 Applications of Advanced Methods
709
242 Online Adaptation
713
2421 Variable Forgetting Factor
714
2422 Control and Adaptation Models
715
2423 Parameter Transfer
717
2424 Systems with Multiple Inputs
718
2425 Experimental Results
719
243 Fault Detection
723
2432 Experimental Results
726
244 Fault Diagnosis
729
2442 Experimental Results
731
245 Reconfiguration
732
A Vectors and Matrices
735
A2 Gradient Hessian and Jacobian
737
B Statistics
739
B2 Probability Density Function pdf
741
B3 Stochastic Processes and Ergodicity
743
B4 Expectation
745
B5 Variance
748
B6 Correlation and Covariance
749
B7 Properties of Estimators
753
References
757
Index
779
Copyright

Other editions - View all

Common terms and phrases

Popular passages

Page 770 - D. Park, A. Kandel, and G Langholz, "Genetic-based new fuzzy reasoning models with application to fuzzy control," IEEE Transactions on Systems, Man, and Cybernetics, Vol.
Page 774 - Locally recurrent globally feedforward networks: a critical review of architectures", IEEE Transactions on Neural Networks, vol. 5, no. 2, 229239, March 1994.
Page 776 - Performance Evaluation of a Sequential Minimal Radial Basis Function (RBF) Neural Network Learning Algorithm", IEEE Transactions on Neural Networks, Vol.
Page 758 - L. Davis, Handbook of Genetic Algorithms, Van Nostrand Reinhold, New York, 1991.