Neural Fuzzy Systems A Neuro-Fuzzy Synergism to Intelligent Systems Chin-Teng Lin Department of Control Engineering National Chiao-Tung University Hsinchu, Taiwan C.S.George Lee School of Electrical and Computer Engineering Purdue University West Lafayette, Indiana For book and bookstore information http://www.prenhall.com Prentice Hall PTR, Upper Saddle River, NJ 07458
PREFACE xiii 1 INTRODUCTION 1 1.1 Motivation 1 1.2 Fuzzy Systems 3 1.3 Neural Networks 4 1.4 Fuzzy Neural Integrated Systems 7 1.5 Organization of the Book 8 1.6 References 9 PART I FUZZY SYSTEMS 2 BASICS OF FUZZY SETS 10 2.1 Fuzzy Sets and Operation on Fuzzy Sets 10 2.2 Extensions of Fuzzy Set Concepts 22 2.2.1 Other Kinds of Fuzzy Sets, 22 ". 2.2.2 Further Operations on Fuzzy Sets, 23 2.3 Extension Principle and Its Applications 29 2.3.1 Operations oftype-2 Fuzzy Sets, 31 2.3.2 Consistency Degree of Two Fuzzy Sets, 32 2.4 Concluding Remarks 33 2.5 Problems 34
3 FUZZY RELATIONS 37 3.1 Basics of Fuzzy Relations 37 3.2 Operations on Fuzzy Relations 41 3.3 Various Types of Binary Fuzzy Relations 47 3.3.1 Similarity Relations, 49 3.3.2 Resemblance Relations, 51 3.3.3 Fuzzy Partial Ordering, 52 3.4 Fuzzy Relation Equations 54 3.5 Concluding Remarks 59 3.6 Problems 60 4 FUZZY MEASURES 63 4.1 Fuzzy Measures 64 4.1.1 Belief and Plausibility Measures, 65 4.1.2 Probability Measures, 72 4.1.3 Possibility arid Necessity Measures, 74 4.2 Fuzzy Integrals 80 4.3 Measures of Fuzziness 83 4.4 Concluding Remarks 85 4.5 Problems 86 5 POSSIBILITY THEORY AND FUZZY ARITHMETIC 89 5.1 Basics of Possibility Theory 89 5.2 Fuzzy Arithmetic 93 5.2.1 Interval Representation of Uncertain Values, 94 5.2.2 Operations and Properties of Fuzzy Numbers, 97 5.2.3 Ordering of Fuzzy Numbers, 108 5.3 Concluding Remarks 111 5.4 Problems 111 6 FUZZY LOGIC AND APPROXIMATE REASONING 114 6.1 Linguistic Variables 114 6.2 Fuzzy Logic 118 6.2.1 Truth Values and Truth Tables in Fuzzy Logic, 119 6.2.2 Fuzzy Propositions, 121 6.3 Approximate Reasoning 123 6.3.1 Categorical Reasoning, 123 6.3.2 Qualitative Reasoning, 126 6.3.3 Syllogistic Reasoning, 127 6.3.4 Dispositional Reasoning, 129 6.4 Fuzzy Expert Systems 131 6.4.1 MILORD, 132 6.4.2 Z-II, 136 6.5 Concluding Remarks 139 6.6 Problems 140 vi
7 FUZZY LOGIC CONTROL SYSTEMS 142 7.1 Basic Structure and Operation of Fuzzy Logic Control Systems 142 7.1.1 Input-Output Spaces, 143 7.1.2 Fuzzifier, 145 7.1.3 Fuzzy Rule Base, 145 7.1.4 Inference Engine, 145 7.1.5 Defuzzifier, 156 7.2 Design Methodology of Fuzzy Control Systems 159 7.3 Stability Analysis of Fuzzy Control Systems 166 7.4 Applications of Fuzzy Controllers 172 7.5 Concluding Remarks 175 7.6 Problems, 177 8 APPLICATIONS OF FUZZY THEORY 180 8.1 Fuzzy Pattern Recognition 180 8.1.1 Classification Methods Based on Fuzzy Relations, 182 8.1.2 Fuzzy Clustering, 186 8.2 Fuzzy Mathematical Programming 190 8.3 Fuzzy Databases 193 8.3.1 Fuzzy Relational Databases, 194 8.3.2 Fuzzy Object-Oriented Databases, 196 8.4 Human-Machine Interactions 199 8.5 Concluding Remarks 202 8.6 Problems 203 ARTIFICIAL NEURAL NETWORKS INTRODUCTION TO ARTIFICIAL NEURAL NETWORKS 205 9.1 Fundamental Concepts of Artificial Neural Networks 205 9.2 Basic Models and Learning Rules of ANNs 207 9.2.1 Processing Elements, 207 5 9.2.2 Connections, 211 9.2.3 Learning Rules, 212 9.3 Distributed Representations 217 9.4 Concluding Remarks 221 9.5 Problems 221 10 FEEDFORWARD NETWORKS AND SUPERVISED LEARNING 224 10.1 Single-Layer Perceptron Networks 224 10.1.1 Perceptron Learning Rule, 225-10.1.2 Adaline,231 10.2 Multilayer Feedforward Networks 235 10.2.1 Back Propagation, 236 10.2.2 Learning Factors of Back Propagation, 244 10.2.3 Time-Delay Neural Networks, 250 vii
10.3 Other Feedforward Networks 253 10.3.1 Functional-Link Networks, 253 10.3.2 Tree Neural Networks, 254 10.3.3 Wavelet Neural Networks, 255 10.4 Concluding Remarks 256 10.5 Problems 257 11 SINGLE-LAYER FEEDBACK NETWORKS AND ASSOCIATIVE MEMORIES 263 11.1 Hopfield Networks 263 11.1.1 Discrete Hopfield Networks, 263 11.1.2 Continuous Hopfield Networks, 267 11.2 Associative Memories 272 11.2.1 Recurrent Autoassociative Memory Hopfield Memory, 273 11.2.2 Bidirectional Associative Memory, 277 11.2.3 Temporal Associative Memory, 282 11.3 Optimization Problems 284 11.3.1 Hopfield Networks for Optimization Problems, 284 11.3.2 Boltzmann Machines, 291 11.4 Concluding Remarks 296 11.5 Problems 298 12 UNSUPERVISED LEARNING NETWORKS 301 12.1 Unsupervised Learning Rules 301 12.1.1 Signal Hebbian Learning Rule, 302 12.1.2 Competitive Learning Rule, 304 12.1.3 Differential Hebbian Learning Rule, 308 12.1.4 Differential Competitive Learning Rule, 309 12.2 Hamming Networks 309 12.3 Self-Organizing Feature Maps 311 12.4 Adaptive Resonance Theory 314 12.4.1 The Instar-Outstar Model Shunting Activation Equations, 315 12.4.2 Adaptive Resonance Theory, 321 12.5 Counterpropagation Networks 326 12.6 Radial Basis Function Networks 328 12.7 Adaptive Bidirectional Associative Memories 331 ) 12.8 Hierarchical Networks Neocognitron 332 12.9 Concluding Remarks 336 12.10 Problems 337 13 RECURRENT NEURAL NETWORKS 340 13.1 Feedback Backpropagation Networks 341 13.1.1 Recurrent Backpropagation Networks, 341 13.1.2 Partially Recurrent Networks, 345 13.2 Fully Recurrent Networks 349 13.2.1 Real-Time Recurrent Learning, 350 viii
13.2.2 Time-Dependent Recurrent Backpropagation, 353 13.2.3 Second-Order Recurrent Networks, 357 13.2.4 The Extended Kalman Filter, 363 13.3 Reinforcement Learning 367 13.3.1 Associative Reward-Penalty, 368 13.3.2 REINFORCE Algorithms, 371 13.3.3 Temporal Difference Methods, 373 13.4 Concluding Remarks 379 13.5 Problems 380 14 GENETIC ALGORITHMS 382 14.1 Basics of Genetic Algorithms 382 14.2 Further Evolution of Genetic Algorithms 393 14.2.1 Improved Selection Schemes, 393 14.2.2 Advanced Operators, 394 14.3 Hybrid Genetic Algorithms 398 14.4 Applications of Genetic Algorithms 399 14.4.1 Genetic Algorithms for Neural Network Parameter Learning, 399 14.4.2 Genetic Algorithms for Path Planning, 404 14.4.3 Genetic Algorithms for System Identification and Controls, 405 14.5 Genetic Programming 406 14.6 Concluding Remarks 411 14.7 Problems 411 15 STRUCTURE-ADAPTIVE NEURAL NETWORKS 414 15.1 Simulated Evolution for Neural Network Structure Learning 414 15.1.1 Genetic Algorithms for Designing Neural Networks, 414 15.1.2 Evolutionary Programming for Designing Neural Networks, 420 15.2 Pruning Neural Networks 424 15.2.1 Weight Decay, 424 15.2.2 Connection and Node Pruning, 425 15.3 Growing Neural Networks 427 ; ; 15.3.1 Input Space Partitioning, 427 15.3.2 Prototype Selection, 433 15.4 Growing and Pruning Neural Networks 437 15.4.1 Activity-Based Structural Adaptation, 437 15.4.2 Function Networks, 439 15.5 Concluding Remarks 442 15.6 Problems 443. 16 APPLICATIONS OF NEURAL NETWORKS 445 16.1 NeuraLNetworks in Control Systems 445 16.1.1 Dynamic Backpropagation for System Identification and Control, 448 16.1.2 Cerebellar Model Articulation Controller, 457 16.2 Neural Networks in Sensor Processing 464 16.3 Neural Networks in Communications 468 ix
16.4 Neural Knowledge-Based Systems 470 16.5 Concluding Remarks 474 16.6 Problems 475 PART III FUZZY NEURAL INTEGRATED SYSTEMS 17 INTEGRATING FUZZY SYSTEMS AND NEURAL NETWORKS 478 17.1 Basic Concept of Integrating Fuzzy Systems and Neural Networks 478 17.1.1 General Comparisons of Fuzzy Systems and Neural Networks, 478 17.1.2 Choice of Fuzzy Systems or Neural Networks, 480 17.1.3 Reasons for Integrating Fuzzy Systems and Neural Networks, 481 17.2 The Equivalence of Fuzzy Inference Systems and Neural Networks 482 17.2.1 Fuzzy Inference Systems as Universal Approximators, 483 17.2.2 Equivalence of Simplified Fuzzy Inference Systems and Radial Basis Function Networks, 487 17.2.3 Stability Analysis of Neural Networks Using Stability Conditions of Fuzzy Systems, 489 17.3 Concluding Remarks 494 17.4 Problems 494 18 NEURAL-NETWORK-BASED FUZZY SYSTEMS 496 18.1 Neural Realization of Basic Fuzzy Logic Operations 496 18.2 Neural Network-Based Fuzzy Logic Inference 498 18.2.1 Fuzzy Inference Networks, 498 18.2.2 Fuzzy Aggregation Networks, 504 18.2.3 Neural Network Driven Fuzzy Reasoning, 507 18.3 Neural Network-Based Fuzzy Modeling 511 18.3.1 Rule-Based Neural Fuzzy Modeling, 511 18.3.2 Neural Fuzzy Regression Models, 517 18.3.3 Neural Fuzzy Relational Systems, 523 18.4 Concluding Remarks 530 18.5 Problems 531 19 NEURAL FUZZY CONTROLLERS 533 19.1 Types of Neural Fuzzy Controllers 534 19.2 Neural Fuzzy Controllers with Hybrid Structure-Parameter Learning 535 19.2.1 Fuzzy Adaptive Learning Control Network, 535 19.2.2 Fuzzy Basis Function Network with Orthogonal Least Squares Learning, 545 19.3 Parameter Learning for Neural Fuzzy Controllers 551 19.3.1 Neural Fuzzy Controllers with Fuzzy Singleton Rules, 551 19.3.2 Neural Fuzzy Controllers with TSK Fuzzy Rules, 556 19.3.3 Fuzzy Operator Tuning, 559
19.4 Structure Learning for Neural Fuzzy Controllers 561 19.4.1 Fuzzy Logic Rule Extraction from Numerical Training Data, 562 19.4.2 Genetic Algorithms for Fuzzy Partition of Input Space, 567 19.5 On-Line Structure Adaptive Neural Fuzzy Controllers 573 19.5.1 FALCON with On-Line Supervised Structure and Parameter Learning, 573 19.5.2 FALCON with ART Neural Learning, 579 19.6 Neural Fuzzy Controllers with Reinforcement Learning 592 19.6.1 FALCON with Reinforcement Learning, 592 19.6.2 Generalized Approximate Reasoning-Based Intelligent Controller, 600 19.7 Concluding Remarks 604 19.8 Problems 605 20 FUZZY LOGIC-BASED NEURAL NETWORK MODELS 609 20.1 Fuzzy Neurons 609 20.1.1 Fuzzy Neuron of Type I, 610 20.1.2 Fuzzy Neuron of Type II, 612 20.1.3 Fuzzy Neuron of Type III, 613 20.2 Fuzzification of Neural Network Models 614 20.2.1 Fuzzy Perceptron, 614 20.2.2 Fuzzy Classification with the Back-propagation Network, 618 20.2.3 Fuzzy Associative Memories, 620 20.2.4 Fuzzy ART Models, 626 20.2.5 Fuzzy Kohonen Clustering Network, 635 20.2.6 Fuzzy RCE Neural Network, 639 20.2.7 Fuzzy Cerebellar Model Articulation Controller, 641 20.3 Neural Networks with Fuzzy Training 643 20.3.1 Neural Networks with Fuzzy Teaching Input, 643 20.3.2 Neural Networks with Fuzzy Parameters, 648 20.3.3 Fuzzy Control for Learning Parameter Adaptation, 654 20.4 Concluding Remarks 657 20.5 Problems 658 ) 21 FUZZY NEURAL SYSTEMS FOR PATTERN RECOGNITION 661 21.1 Fuzzy Neural Classification 661 21.1.1 Uncertainties with Two-Class Fuzzy Neural Classification Boundaries, 661 21.1.2 Multilayer Fuzzy Neural Classification Networks, 667 21.1.3 Genetic Algorithms for Fuzzy Classification Using Fuzzy Rules, 674 21.2 Fuzzy Neural Clustering 678 21.2.1 Fuzzy Competitive Learning for Fuzzy Clustering, 678 21.2.2 Adaptive Fuzzy Leader Clustering, 680 21.3 Fuzzy Neural Models for Image Processing 684 21.3.1 Fuzzy Self-Supervised Multilayer Network for Object Extraction, 684 "^ 21.3:2 Genetic Algorithms with Fuzzy Fitness Function for Image Enhancement, 690 21A Fuzzy Neural Networks for Speech Recognition 692 xi
21.5 Fuzzy-Neural Hybrid Systems for System Diagnosis 696 21.6 Concluding Remarks 700 21.7 Problems 702 A MATLAB FUZZY LOGIC TOOLBOX 704 A. 1 Demonstration Cases 706 A. 1.1 A Simple Fuzzy Logic Controller, 706 A. 1.2 A Neural Fuzzy System ANFIS, 707 A. 1.3 Fuzzy c-means Clustering, 708 A.2 Demonstration of Fuzzy Logic Applications 709 A. 2.1 A Fuzzy Controller for Water Bath Temperature Control, 709 A.2.2 A Neural Fuzzy Controller for Water Bath Temperature Control, 712 B MATLAB NEURAL NETWORK TOOLBOX 715 B. 1 Demonstration of Various Neural Network Models 716 B.I.I Hebbian Learning Rule, 716 B.I.2 Perceptron Learning Rule, 717 B.I.3 Adaline, 718 B.I.4 Back Propagation, 719 B.I.5 Hopfield Network, 722 B.I.6 Instar Learning Rule, 723 B.I.7 Competitive Learning Rule, 724 B.I.8 LVQ Learning Rule (Supervised Competitive Learning Rule), 726 B.1.9 Self-Organizing Feature Map, 726 B.I.10 Radial Basis Function Network, 727 B.I.11 ElmanNetwork, 729 B.2 Demonstration of Neural Network Applications 730 5.2.7 Adaptive Noise Cancelation Using theadaline Network, 730 B.2.2 Nonlinear System Identification, 731 BIBLIOGRAPHY 734 INDEX 783 xii,