Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy ModelsThe goal of this book is to provide engineers and scientIsts in academia and industry with a thorough understanding of the underlying principles of nonlinear system identification. The reader will be able to apply the discussed models and methods to real problems with the necessary confidence and the awareness of potential difficulties that may arise in practice. This book is self-contained in the sense that it requires merely basic knowledge of matrix algebra, signals and systems, and statistics. Therefore, it also serves as an introduction to linear system identification and gives a practical overview on the major optimization methods used in engineering. The emphasis of this book is on an intuitive understanding of the subject and the practical application of the discussed techniques. It is not written in a theorem/proof style; rather the mathematics is kept to a minimum and the pursued ideas are illustrated by numerous figures, examples, and real-world applications. Fifteen years ago, nonlinear system identification was a field of several ad-hoc approaches, each applicable only to a very restricted class of systems. With the advent of neural networks, fuzzy models, and modern structure opti mization techniques a much wider class of systems can be handled. Although one major characteristic of nonlinear systems is that almost every nonlinear system is unique, tools have been developed that allow the use of the same ap proach for a broad variety of systems. |
Contents
1 Introduction | 1 |
112 Prediction | 2 |
113 Simulation | 3 |
114 Optimization | 4 |
117 Fault Detection | 5 |
12 Tasks in Nonlinear System Identification | 6 |
121 Choice of the Model Inputs | 8 |
122 Choice of the Excitation Signals | 9 |
1331 Local Linear Model Tree LOLIMOT Algorithm | 365 |
1332 Different Objectives for Structure and Parameter Optimization | 372 |
1333 Smoothness Optimization | 374 |
1334 Splitting Ratio Optimization | 376 |
1335 Merging of Local Models | 378 |
1336 Flat and Hierarchical Model Structures | 380 |
1337 Principal Component Analysis for Preprocessing | 383 |
1338 Models with Multiple Outputs | 385 |
123 Choice of the Model Architecture | 10 |
124 Choice of the Dynamics Representation | 11 |
127 Choice of the Model Parameters | 12 |
128 Model Validation | 13 |
13 White Box Black Box and Gray Box Models | 15 |
14 Outline of the Book and Some Reading Suggestions | 16 |
15 Terminology | 18 |
Optimization Techniques | 21 |
2 Introduction to Optimization | 23 |
21 Overview of Optimization Techniques | 25 |
23 Loss Functions for Supervised Methods | 28 |
231 Maximum Likelihood Method | 30 |
232 Maximum APosteriori and Bayes Method | 32 |
24 Loss Functions for Unsupervised Methods | 34 |
3 Linear Optimization | 35 |
31 Least Squares LS | 36 |
311 Covariance Matrix of the Parameter Estimate | 44 |
312 Errorbars | 45 |
313 Orthogonal Regressors | 48 |
314 Regularization Ridge Regression | 49 |
315 Noise Assumptions | 54 |
316 Weighted Least Squares WLS | 55 |
317 Least Squares with Equality Constraints | 57 |
318 Smoothing Kernels | 58 |
32 Recursive Least Squares RLS | 60 |
321 Reducing the Computational Complexity | 63 |
322 Tracking TimeVariant Processes | 64 |
323 Relationship between the RLS and the Kalman Filter | 65 |
33 Linear Optimization with Inequality Constraints | 66 |
34 Subset Selection | 67 |
341 Methods for Subset Selection | 68 |
342 Orthogonal Least Squares OLS for Forward Selection | 72 |
343 Ridge Regression or Subset Selection? | 75 |
35 Summary | 77 |
4 Nonlinear Local Optimization | 79 |
41 Batch and Sample Adaptation | 81 |
42 Initial Parameters | 83 |
43 Direct Search Algorithms | 86 |
432 HookeJeeves Method | 88 |
44 General GradientBased Algorithms | 90 |
441 Line Search | 91 |
442 Finite Difference Techniques | 92 |
443 Steepest Descent | 93 |
444 Newtons Method | 96 |
445 QuasiNewton Methods | 98 |
446 Conjugate Gradient Methods | 100 |
45 Nonlinear Least Squares Problems | 102 |
451 GaussNewton Method | 104 |
452 LevenbergMarquardt Method | 105 |
46 Constrained Nonlinear Optimization | 107 |
47 Summary | 110 |
5 Nonlinear Global Optimization | 113 |
51 Simulated Annealing SA | 116 |
52 Evolutionary Algorithms EA | 120 |
521 Evolution Strategies ES | 123 |
522 Genetic Algorithms GA | 126 |
523 Genetic Programming GP | 132 |
53 Branch and Bound BB | 133 |
54 Tabu Search TS | 135 |
6 Unsupervised Learning Techniques | 137 |
61 Principal Component Analysis PCA | 139 |
62 Clustering Techniques | 142 |
621 KMeans Algorithm | 143 |
622 Fuzzy CMeans FCM Algorithm | 146 |
623 GustafsonKessel Algorithm | 148 |
624 Kohonens SelfOrganizing Map SOM | 149 |
625 Neural Gas Network | 152 |
626 Adaptive Resonance Theory ART Network | 153 |
627 Incorporating Information about the Output | 154 |
63 Summary | 155 |
7 Model Complexity Optimization | 157 |
72 BiasVariance Tradeoff | 158 |
721 Bias Error | 160 |
722 Variance Error | 161 |
723 Tradeoff | 164 |
73 Evaluating the Test Error and Alternatives | 167 |
731 Training Validation and Test Data | 168 |
732 Cross Validation | 169 |
733 Information Criteria | 171 |
734 MultiObjective Optimization | 172 |
735 Statistical Tests | 174 |
736 CorrelationBased Methods | 176 |
Implicit Structure Optimization | 179 |
752 Regularization by NonSmoothness Penalties | 180 |
753 Regularization by Early Stopping | 182 |
754 Regularization by Constraints | 184 |
755 Regularization by Staggered Optimization | 186 |
756 Regularization by Local Optimization | 187 |
76 Structured Models for Complexity Reduction | 189 |
761 Curse of Dimensionality | 190 |
762 Hybrid Structures | 192 |
763 ProjectionBased Structures | 195 |
764 Additive Structures | 196 |
765 Hierarchical Structures | 197 |
766 Input Space Decomposition with Tree Structures | 198 |
77 Summary | 200 |
8 Summary of Part I | 203 |
Static Models | 207 |
9 Introduction to Static Models | 209 |
92 Basis Function Formulation | 210 |
921 Global and Local Basis Functions | 211 |
922 Linear and Nonlinear Parameters | 212 |
93 Extended Basis Function Formulation | 215 |
94 Static Test Process | 216 |
10 Linear Polynomial and LookUp Table Models | 219 |
102 Polynomial Models | 221 |
103 LookUp Table Models | 224 |
1031 OneDimensional LookUp Tables | 225 |
1032 TwoDimensional LookUp Tables | 227 |
1033 Optimization of the Heights | 229 |
1034 Optimization of the Grid | 231 |
1035 Optimization of the Complete LookUp Table | 232 |
1037 Properties of LookUp Table Models | 235 |
104 Summary | 237 |
11 Neural Networks | 239 |
111 Construction Mechanisms | 242 |
1112 Radial Construction | 244 |
1113 Tensor Product Construction | 245 |
112 Multilayer Perceptron MLP Network | 246 |
1121 MLP Neuron | 247 |
1122 Network Structure | 249 |
1123 Backpropagation | 252 |
1124 MLP Training | 253 |
1125 Simulation Examples | 256 |
1126 MLP Properties | 260 |
1127 Multiple Hidden Layers | 261 |
1128 Projection Pursuit Regression PPR | 262 |
113 Radial Basis Function RBF Networks | 264 |
1132 Network Structure | 267 |
1133 RBF Training | 269 |
1134 Simulation Examples | 277 |
1135 RBF Properties | 279 |
1136 Regularization Theory | 281 |
1137 Normalized Radial Basis Function NRBF Networks | 283 |
114 Other Neural Networks | 286 |
1142 Cerebellar Model Articulation Controller CMAC | 288 |
1143 Delaunay Networks | 292 |
1144 JustinTime Models | 293 |
115 Summary | 296 |
12 Fuzzy and NeuroFuzzy Models | 299 |
1211 Membership Functions | 300 |
1212 Logic Operators | 302 |
1213 Rule Fulfillment | 303 |
122 Types of Fuzzy Systems | 304 |
1222 Singleton Fuzzy Systems | 307 |
1223 TakagiSugeno Fuzzy Systems | 309 |
123 NeuroFuzzy NF Networks | 310 |
1231 Fuzzy Basis Functions | 311 |
1232 Equivalence between RBF Networks and Fuzzy Models | 312 |
1233 What to Optimize? | 313 |
1234 Interpretation of NeuroFuzzy Networks | 316 |
1235 Incorporating and Preserving Prior Knowledge | 320 |
1236 Simulation Examples | 321 |
124 NeuroFuzzy Learning Schemes | 323 |
1242 Nonlinear Global Optimization | 325 |
1244 Fuzzy Rule Extraction by a Genetic Algorithm FUREGA | 327 |
1245 Adaptive Spline Modeling of Observation Data ASMOD | 337 |
125 Summary | 339 |
Fundamentals | 341 |
131 Basic Ideas | 342 |
1311 Illustration of Local Linear NeuroFuzzy Models | 343 |
1312 Interpretation of the Local Linear Model Offsets | 346 |
1313 Interpretation as TakagiSugeno Fuzzy System | 347 |
1314 Interpretation as Extended NRBF Network | 349 |
132 Parameter Optimization of the Rule Consequents | 351 |
1322 Local Estimation | 352 |
1323 Global Versus Local Estimation | 356 |
1324 Data Weighting | 361 |
133 Structure Optimization of the Rule Premises | 362 |
134 Summary | 389 |
Advanced Aspects | 391 |
1411 Identification of Processes with Direction Dependent Behavior | 395 |
142 More Complex Local Models | 397 |
1422 Local Quadratic Models for Input Optimization | 400 |
1423 Different Types of Local Models | 402 |
143 Structure Optimization of the Rule Consequents | 404 |
144 Interpolation and Extrapolation Behavior | 408 |
1442 Extrapolation Behavior | 411 |
145 Global and Local Linearization | 416 |
146 Online Learning | 420 |
1461 Online Adaptation of the Rule Consequents | 421 |
1462 Online Construction of the Rule Premise Structure | 428 |
147 Errorbars Design of Excitation Signals and Active Learning | 430 |
1471 Errorbars | 431 |
1472 Detecting Extrapolation | 434 |
1473 Design of Excitation Signals | 435 |
1474 Active Learning | 436 |
148 From Local Linear NeuroFuzzy Models to Hinging Hyperplanes | 437 |
1481 Hinging Hyperplanes | 438 |
1482 Smooth Hinging Hyperplanes | 439 |
1483 Hinging Hyperplane Trees HHT | 441 |
1484 Local Linear NeuroFuzzy Models Versus Hinging Hyperplane Trees | 443 |
149 Summary and Conclusions | 444 |
15 Summary of Part II | 451 |
Dynamic Models | 455 |
16 Linear Dynamic System Identification | 457 |
161 Overview of Linear System Identification | 458 |
162 Excitation Signals | 459 |
163 General Model Structure | 462 |
1631 Terminology and Classification | 465 |
1632 Optimal Predictor | 471 |
1633 Some Remarks on the Optimal Predictor | 474 |
1634 Prediction Error Methods | 476 |
164 Time Series Models | 478 |
1641 Autoregressive AR | 479 |
1642 Moving Average MA | 480 |
1643 Autoregressive Moving Average ARMA | 481 |
165 Models with Output Feedback | 482 |
1652 Autoregressive Moving Average with Exogenous Input ARMAX | 492 |
1653 Autoregressive Autoregressive with Exogenous Input ARARX | 496 |
1654 Output Error OE | 499 |
1655 BoxJenkins BJ | 503 |
1656 State Space Models | 505 |
1657 Simulation Example | 506 |
166 Models without Output Feedback | 509 |
1661 Finite Impulse Response FIR | 510 |
1662 Orthonormal Basis Functions OBF | 512 |
1663 Simulation Example | 520 |
167 Some Advanced Aspects | 524 |
1672 Consistency | 526 |
1674 Relationship between Noise Model and Filtering | 528 |
1675 Offsets | 529 |
168 Recursive Algorithms | 531 |
1681 Recursive Least Squares RLS Method | 532 |
1683 Recursive Extended Least Squares RELS Method | 533 |
1684 Recursive Prediction Error Methods RPEM | 534 |
169 Determination of Dynamic Orders | 536 |
1610 Multivariable Systems | 537 |
16101 PCanonical Model | 539 |
16102 Matrix Polynomial Model | 540 |
16103 Subspace Methods | 541 |
16111 Direct Methods | 542 |
16112 Indirect Methods | 544 |
16113 Identification for Control | 545 |
1612 Summary | 546 |
17 Nonlinear Dynamic System Identification | 547 |
172 External Dynamics | 549 |
1721 Illustration of the External Dynamics Approach | 550 |
1722 SeriesParallel and Parallel Models | 555 |
1723 Nonlinear Dynamic InputOutput Model Classes | 557 |
1724 Restrictions of Nonlinear Dynamic InputOutput Models | 562 |
173 Internal Dynamics | 563 |
174 Parameter Scheduling Approach | 564 |
1751 BackpropagationThroughTime BPTT Algorithm | 565 |
1752 Real Time Recurrent Learning | 567 |
176 Multivariable Systems | 568 |
177 Excitation Signals | 569 |
178 Determination of Dynamic Orders | 574 |
179 Summary | 576 |
18 Classical Polynomial Approaches | 579 |
181 Properties of Dynamic Polynomial Models | 580 |
182 KolmogorovGabor Polynomial Models | 581 |
183 VolterraSeries Models | 582 |
184 Parametric VolterraSeries Models | 583 |
186 Hammerstein Models | 584 |
187 Wiener Models | 585 |
19 Dynamic Neural and Fuzzy Models | 587 |
1911 MLP Networks | 588 |
192 Interpolation and Extrapolation Behavior | 589 |
193 Training | 591 |
1931 MLP Networks | 592 |
194 Integration of a Linear Model | 593 |
195 Simulation Examples | 594 |
1951 MLP Networks | 595 |
1952 RBF Networks | 597 |
1953 Singleton Fuzzy and NRBF Models | 599 |
196 Summary | 600 |
20 Dynamic Local Linear NeuroFuzzy Models | 601 |
201 OneStep Prediction Error Versus Simulation Error | 604 |
202 Determination of the Rule Premises | 606 |
203 Linearization | 608 |
2032 Dynamics of the Linearized Model | 610 |
2033 Different Rule Consequent Structures | 612 |
204 Model Stability | 613 |
2041 Influence of Rule Premise Inputs on Stability | 614 |
2042 Lyapunov Stability and Linear Matrix Inequalities LMIs | 616 |
2043 Ensuring Stable Extrapolation | 617 |
205 Dynamic LOLIMOT Simulation Studies | 618 |
2052 Hammerstein Process | 620 |
2053 Wiener Process | 624 |
2054 NDE Process | 625 |
206 Advanced Local Linear Methods and Models | 626 |
2061 Local Linear Instrumental Variables IV Method | 628 |
2062 Local Linear Output Error OE Models | 630 |
2063 Local Linear ARMAX Models | 631 |
208 Structure Optimization of the Rule Consequents | 636 |
209 Summary and Conclusions | 640 |
21 Neural Networks with Internal Dynamics | 645 |
212 Partially Recurrent Networks | 646 |
213 State Recurrent Networks | 647 |
214 Locally Recurrent Globally Feedforward Networks | 648 |
215 Internal Versus External Dynamics | 650 |
Applications | 653 |
22 Applications of Static Models | 655 |
2211 Process Description | 656 |
2212 Smoothing of a Driving Cycle | 657 |
2213 Improvements and Extensions | 658 |
2214 Differentiation | 659 |
2221 The Role of LookUp Tables in Automotive Electronics | 660 |
2222 Modeling of Exhaust Gases | 663 |
2223 Optimization of Exhaust Gases | 666 |
Dynamic Models | 672 |
223 Summary | 674 |
23 Applications of Dynamic Models | 677 |
2312 Experimental Results | 679 |
232 Diesel Engine Turbocharger | 683 |
2321 Process Description | 684 |
2322 Experimental Results | 685 |
233 Thermal Plant | 691 |
2331 Process Description | 692 |
2332 Transport Process | 693 |
2333 Tubular Heat Exchanger | 698 |
2334 CrossFlow Heat Exchanger | 702 |
234 Summary | 707 |
24 Applications of Advanced Methods | 709 |
242 Online Adaptation | 713 |
2421 Variable Forgetting Factor | 714 |
2422 Control and Adaptation Models | 715 |
2423 Parameter Transfer | 717 |
2424 Systems with Multiple Inputs | 718 |
2425 Experimental Results | 719 |
243 Fault Detection | 723 |
2432 Experimental Results | 726 |
244 Fault Diagnosis | 729 |
2442 Experimental Results | 731 |
245 Reconfiguration | 732 |
A Vectors and Matrices | 735 |
A2 Gradient Hessian and Jacobian | 737 |
B Statistics | 739 |
B2 Probability Density Function pdf | 741 |
B3 Stochastic Processes and Ergodicity | 743 |
B4 Expectation | 745 |
B5 Variance | 748 |
B6 Correlation and Covariance | 749 |
B7 Properties of Estimators | 753 |
757 | |
779 | |
Other editions - View all
Nonlinear System Identification: From Classical Approaches to Neural ... Oliver Nelles Limited preview - 2013 |
Nonlinear System Identification: From Classical Approaches to Neural ... Oliver Nelles No preview available - 2010 |
Nonlinear System Identification: From Classical Approaches to Neural ... Oliver Nelles No preview available - 2014 |
Common terms and phrases
adaptation algorithm applied approximation ARMAX ARX model basis functions chosen clustering computational constraints convergence curse of dimensionality data samples data set errorbars evaluation evolution strategies example extrapolation behavior filter finite impulse response fuzzy model fuzzy systems Gaussian genetic algorithms global global optimization gradient hidden layer input space interpolation iteration least squares linear models linear neuro-fuzzy models LOLIMOT look-up table loss function matrix membership functions methods MLP network model architectures model complexity model error model output model structure MSFs NARX neural network neurons nonlinear optimization number of parameters OE model one-step prediction operating point optimization techniques parameter estimation performance polynomial prediction error prior knowledge process output RBF networks regression regressors rule consequents Sect shown in Fig signal simulation static step responses strategy structure optimization system identification tion training data typically u₁ unsupervised learning utilized validity functions variables variance error vector นา
Popular passages
Page 770 - D. Park, A. Kandel, and G Langholz, "Genetic-based new fuzzy reasoning models with application to fuzzy control," IEEE Transactions on Systems, Man, and Cybernetics, Vol.
Page 774 - Locally recurrent globally feedforward networks: a critical review of architectures", IEEE Transactions on Neural Networks, vol. 5, no. 2, 229239, March 1994.