Advanced Information Processing

Transcription

1 Advanced Information Processing Series Editor Lakhmi C. Jain Advisory Board Members Endre Boros Clarence W. de Silva Stephen Grossberg Robert J. Hewlett Michael N. Huhns Paul B. Kantor Charles L. Karr Nadia Magenat-Thalmann Dinesh P.Mital Toyoaki Nishida Klaus Obermayer Manfred Schmitt

2 Hisao Ishibuchi Tomoharu Nakashima Manabu Nii Classification and Modeling with Linguistic Information Granules Advanced Approaches to Linguistic Data Mining With 217 Figures and 72 Tables ^ Spri rineer

3 Hisao Ishibuchi Department of Computer Science and Intelligent Systems Osaka Prefecture University 1-1 Gakuen-cho, Sakai Osaka , Japan Tomoharu Nakashima Department of Computer Science and Intelligent Systems Osaka Prefecture University 1-1 Gakuen-cho, Sakai Osaka , Japan Manabu Nii Department of Electrical Engineering and Computer Sciences Graduate School of Engineering University of Hyogo 2167Shosha, Himeji Hyogo , Japan Library of Congress Control Number: ACM Subject Classification (1998): 1.2 ISBN Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in databanks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9,1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springeronhne.com Springer-Verlag BerHn Heidelberg 2005 Printed in Germany The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: by the Authors Cover design: KiinkelLopka, Heidelberg Production: LE-TeX Jelonek, Schmidt & Vockler GbR, Leipzig Printed on acid-free paper 45/3142/YL

4 Preface Many approaches have already been proposed for classification and modeling in the literature. These approaches are usually based on mathematical models. Computer systems can easily handle mathematical models even when they are complicated and nonlinear (e.g., neural networks). On the other hand, it is not always easy for human users to intuitively understand mathematical models even when they are simple and linear. This is because human information processing is based mainly on linguistic knowledge while computer systems are designed to handle symbolic and numerical information. A large part of our daily communication is based on words. We learn from various media such as books, newspapers, magazines, TV, and the Internet through words. We also communicate with others through words. While words play a central role in human information processing, linguistic models are not often used in the fields of classification and modeling. If there is no goal other than the maximization of accuracy in classification and modeling, mathematical models may always be preferred to linguistic models. On the other hand, linguistic models may be chosen if emphasis is placed on interpretability. The main purpose in writing this book is to clearly explain how classification and modeling can be handled in a human understandable manner. In this book, we only use simple linguistic rules such as "// the 1st input is large and the 2nd input is small then the output is large^^ and "// the 1st attribute is small and the 2nd attribute is medium then the pattern is Class ^". These linguistic rules are extracted from numerical data. In this sense, our approaches to classification and modeling can be viewed as linguistic knowledge extraction from numerical data (i.e., linguistic data mining). There are many issues to be discussed in linguistic approaches to classification and modeling. The first issue is how to determine the linguistic terms used in linguistic rules. For example, we have some linguistic terms such as young, middle-aged, and old for describing our ages. In the case of weight, we might use light, middle, and heavy. Two problems are involved in the determination of linguistic terms. One is to choose linguistic terms for each variable, and the other is to define the meaning of each linguistic term. The choice of linguistic terms is related to linguistic discretization (i.e., granulation) of each variable. The definition of the meaning of each linguistic term is performed using fuzzy logic. That is, the meaning of each linguistic term is specified by its membership function. Linguistic rules can be viewed as combinations of linguistic terms for each

5 VI Preface variable. The main focus of this book is to find good combinations of linguistic terms for generating linguistic rules. Interpret ability as well as accuracy are taken into account when we extract linguistic rules from numerical data. Various aspects are related to the interpretability of linguistic models. In this book, the following aspects are discussed: Granulation of each variable (i.e., the number of linguistic terms). Overlap between adjacent linguistic terms. Length of each linguistic rule (i.e., the number of antecedent conditions). Number of linguistic rules. The first two aspects are related to the determination of linguistic terms. We examine the effect of these aspects on the performance of linguistic models. The other two aspects are related to the complexity of linguistic models. We examine a tradeoff between the accuracy and the complexity of linguistic models. We mainly use genetic algorithms for designing linguistic models. Genetic algorithms are used as machine learning tools as well as optimization tools. We also describe the handling of linguistic rules in neural networks. Linguistic rules and numerical data are simultaneously used as training data in the learning of neural networks. Trained neural networks are used to extract linguistic rules. While this book includes many state-of-the-art techniques in soft computing such as multi-objective genetic algorithms, genetics-based machine learning, and fuzzified neural networks, undergraduate students in computer science and related fields may be able to understand almost all parts of this book without any particular background knowledge. We make the book as simple as possible by using many examples and figures. We explain fuzzy logic, genetic algorithms, and neural networks in an easily understandable manner when they are used in the book. This book can be used as a textbook in a one-semester course. In this case, the last four chapters can be omitted because they include somewhat advanced topics on fuzzified neural networks. The first ten chapters clearly explain linguistic models for classification and modeling. I would like to thank Prof. Lakhmi C. Jain for giving me the opportunity to write this book. We would also like to thank Prof. Witold Pedrycz and Prof. Francisco Herrera for their useful comments on the draft version of this book. Special thanks are extended to people who kindly assisted us in publishing this book. For example, Mr. Ronan Nugent worked hard for the copy-editing of this book. Ms. Ulrike Strieker gave us helpful comments on the layout and production. And general comments are given by Mr. Ralf Gerstner, who patiently and kindly contacted us. Some simulation results in this book were checked by my students. It is a pleasure to acknowledge the help of Takashi Yamamoto, Gaku Nakai, Teppei Seguchi, Yohei Shibata, Masayo Udo, Shiori Kaige, and Satoshi Namba. Sakai, Osaka, March 2003 Hisao Ishibuchi

6 Contents 1. Linguistic Information Granules Mathematical Handling of Linguistic Terms Linguistic Discretization of Continuous Attributes 4 2. Pattern Classification with Linguistic Rules Problem Description Linguistic Rule Extraction for Classification Problems Specification of the Consequent Class Specification of the Rule Weight Classification of New Patterns by Linguistic Rules Single Winner-Based Method Voting-Based Method Computer Simulations Comparison of Four Definitions of Rule Weights Simulation Results on Iris Data Simulation Results on Wine Data Discussions on Simulation Results Learning of Linguistic Rules Reward-Punishment Learning Learning Algorithm Illustration of the Learning Algorithm Using Artificial Test Problems Computer Simulations on Iris Data Computer Simulations on Wine Data Analytical Learning Learning Algorithm Illustration of the Learning Algorithm Using Artificial Test Problems Computer Simulations on Iris Data Computer Simulations on Wine Data Related Issues Further Adjustment of Classification Boundaries Adjustment of Membership Functions 62

7 VIII Contents 4. Input Selection and Rule Selection Curse of Dimensionality Input Selection Examination of Subsets of Attributes Simulation Results Genetic Algorithm-Based Rule Selection Basic Idea Generation of Candidate Rules Genetic Algorithms for Rule Selection Computer Simulations Some Extensions to Rule Selection Heuristics in Genetic Algorithms Prescreening of Candidate Rules Computer Simulations Genetics-Based Machine Learning Two Approaches in Genetics-Based Machine Learning Michigan-Style Algorithm Coding of Linguistic Rules Genetic Operations Algorithm Computer Simulations Extensions to the Michigan-Style Algorithm Ill 5.3 Pittsburgh-Style Algorithm Coding of Rule Sets Genetic Operations Algorithm Computer Simulations Hybridization of the Two Approaches Advantages of Each Algorithm Hybrid Algorithm Computer Simulations Minimization of the Number of Linguistic Rules Multi-Objective Design of Linguistic Models Formulation of Three-Objective Problem Multi-Objective Genetic Algorithms Fitness Function Elitist Strategy Basic Framework of Multi-Objective Genetic Algorithms Multi-Objective Rule Selection Algorithm Computer Simulations Multi-Objective Genetics-Based Machine Learning Algorithm 139

8 Contents IX Computer Simulations Comparison of Linguistic Discretization with Interval Discretization Effects of Linguistic Discretization Effect in the Rule Generation Phase Effect in the Classification Phase Summary of Effects of Linguistic Discretization Specification of Linguistic Discretization from Interval Discretization Specification of Fully Fuzzified Linguistic Discretization Specification of Partially Fuzzified Linguistic Discretization Comparison Using Homogeneous Discretization Simulation Results on Iris Data Simulation Results on Wine Data Comparison Using Inhomogeneous Discretization Entropy-Based Inhomogeneous Interval Discretization Simulation Results on Iris Data Simulation Results on Wine Data Modeling with Linguistic Rules Problem Description Linguistic Rule Extraction for Modeling Problems Linguistic Association Rules for Modeling Problems Specification of the Consequent Part Other Approaches to Linguistic Rule Generations Estimation of Output Values by Linguistic Rules Standard Fuzzy Reasoning Limitations and Extensions Non-Standard Fuzzy Reasoning Based on the Specificity of Each Linguistic Rule Modeling of Nonlinear Fuzzy Functions Design of Compact Linguistic Models Single-Objective and Multi-Objective Formulations Three Objectives in the Design of Linguistic Models Handling as a Single-Objective Optimization Problem Handling as a Three-Objective Optimization Problem Multi-Objective Rule Selection Candidate Rule Generation Candidate Rule Prescreening Three-Objective Genetic Algorithm for Rule Selection Simple Numerical Example Fuzzy Genetics-Based Machine Learning 190

9 X Contents Coding of Rule Sets Three-Objective Fuzzy GBML Algorithm Simple Numerical Example Some Heuristic Procedures Comparison of Two Schemes Linguistic Rules with Consequent Real Numbers Consequent Real Numbers Local Learning of Consequent Real Numbers Heuristic Specification Method Incremental Learning Algorithm Global Learning Incremental Learning Algorithm Comparison Between Two Learning Schemes Effect of the Use of Consequent Real Numbers Resolution of Adjustment Simulation Results Twin-Table Approach Basic Idea Determination of Consequent Linguistic Terms Numerical Example Handling of Linguistic Rules in Neural Networks Problem Formulation Approximation of Linguistic Rules Multi-Layer Feedforward Neural Networks Handling of Linguistic Rules Using Membership Values Basic Idea Network Architecture Computer Simulation Handling of Linguistic Rules Using Level Sets Basic Idea Network Architecture Computer Simulation Handling of Linguistic Rules Using Fuzzy Arithmetic Basic Idea Fuzzy Arithmetic Network Architecture Computer Simulation Learning of Neural Networks from Linguistic Rules Back-Propagation Algorithm Learning from Linguistic Rules for Classification Problems Linguistic Training Data Cost Function 237

10 Contents XI Extended Back-Propagation Algorithm Learning from Linguistic Rules and Numerical Data Learning from Linguistic Rules for Modeling Problems Linguistic Data Cost Function Extended Back-Propagation Algorithm Learning from Linguistic Rules and Numerical Data Linguistic Rule Extraction from Neural Networks Neural Networks and Linguistic Rules Linguistic Rule Extraction for Modeling Problems Basic Idea Extraction of Linguistic Rules Computer Simulations Linguistic Rule Extraction for Classification Problems Basic Idea Extraction of Linguistic Rules Computer Simulations Rule Extraction Algorithm Decreasing the Measurement Cost Difficulties and Extensions Scalability to High-Dimensional Problems Increase of Excess Fuzziness in Fuzzy Outputs Modeling of Fuzzy Input-Output Relations Modeling of Fuzzy Number-Valued Functions Linear Fuzzy Regression Models Fuzzy Rule-Based Systems Fuzzified Takagi-Sugeno Models Fuzzified Neural Networks Modeling of Fuzzy Mappings Linear Fuzzy Regression Models Fuzzy Rule-Based Systems Fuzzified Takagi-Sugeno Models Fuzzified Neural Networks Fuzzy Classification Fuzzy Classification of Non-Fuzzy Patterns Fuzzy Classification of Interval Patterns Fuzzy Classification of Fuzzy Patterns Effect of Fuzzification of Input Patterns 292 Index 304