SHOUHONG WANG. A Thesis. Submitted to the School of Graduate Studies. in Partial Fulfilment of the Requirements. for the Degree. Doctor of Philosophy

Transcription

1 NEURAL NETWORK TECHJlIQUES IN MANAGERIAL PATTERN RECOl.""NITION By SHOUHONG WANG A Thesis Submitted to the School of Graduate Studies in Partial Fulfilment of the Requirements for the Degree Doctor of Philosophy McMaster University (c) Copyright by Shouhong Wang, June 1990

2 NEURAL NETWORK TECHNIQUES IN KAHAGERIAL PA1TERN RECOGNITION

3 DOCTOR OF PHILOSOPHY (1990) McMASTER UNIVERSITY (Management Science / Systems) Hamilton, Ontario TITLE: Neural Network Techniques in Managerial Pattern Recognition AUTHOR: Shouhong Wang, B.Eng. MBA (Tsinghua University, China) (Tsinghua University, China) SUPERVISOR: Dr. Norman P. Archer NUMBER OF PAGES: xiii, 228 ii

4 ABSTRACl lhe management area includes a large class of pattern recognition (classification) problems. Traditionally, these problems have been solved by using statistical methods or expert system~. In practice, however, statistical assumptions about the probability distributions of the pattern variables are often not verifiable, and expertise concerning th~ correct classification is often not explicitly available. These obstacles may make statistical methods and expert system techniques difficult to apply. Since the early 1980s neural network techniques have been widely used in pattern recognition, especially after Rumelhart's back propagation learning algorithm was ~dapted to the solution of these problems. The standard neural network, using the back propagation learning algorithm, requires no statistical assumptions but uses training sample data to generate classification boundaries, allowing it to perform pattern recognition. In this dissertation the neural ~etwork's behavior in classification boundary generation is analyzed. Based on this analysis, three models are developed. The first model improves the classification performance of neural networks in managerial pattern recognition by modifying the training algorithm through the use of monoton~city. Using simulated and real data, the developed model is tested anrl verified. The second model solves bias problems caused by small sample size in neural network classification results. The third model develops multi-architecture neural networks to supply iii

5 1 decision makers with more natural pattern recognition information, based on fuzzy theory. iv

6 This dissertation owes its existence to many people whose help I deeply appreciate and ~ould like to acknowledge here. I first would like to thank my supervisor, Dr. N. P. Archer for his invaluable advice and guidance throughout this research. His help has gone beyond the call of supervisory duties. I shall always be grateful to him for his patience, and his time and energy spent wor~ing with me. I also wish to thank Dr. G. W. Torrance and Dr. D. P. Taylor for serving on my supervisory committee. More generally I would like to thank other people at McMaster University who have assisted me in completing my graduate studies, especially including Dr. W. G. Truscott, Dr. M. W. L. Chan, Dr. G. O. Wesolowsky, and Dr. M. Parlar. Finally, my sincere appreciation and love go to my parents, my wife, and my son for their support and sacrifices while I have been apart from them during my years of graduate studies. v

7 Notation TABLE OF CONTENTS Chapter One: Introduction General Description of Neural Networks Background A Class of Problems in Manag~ria1 Decision Making - Pat:ern Recognition / Problem Definitions Classification Managerial Pattern Recognition Characteristics of Cla$sification Problems Pattern Recognition Techniques Review of Pattern Recognition Techniques Bayes Rule Discriminant Functions Sequential Classification Hierarchical Classification Expert System Production Rules Systems for Pattern Recognition Frame-Based Systems for Pattern Rec\Jgnition Properties of Expert Systems in Pattern Recognition Neural Network Techniques Comparison of Classification Techniques Primary Comparison of Discriminant Analysis and Neural Networks Limitations of Discriminant Analysis An Example for Comparison of Discriminant Analysis and Neural Network Page ix Chapter Two : The Neural Network Model 2.1. Overview Computational Network Elements 2.3. Learning The Learning Paradigm and The Hecbian Rule Back-Propagation Least Mean Square Error Learning Algorithm Issues Relevant to Learning Learning Rate and Momentum Convergence Global Minima vs. Local Minima Symmetry Breaking Current Status of Neural Network Applications Acceptance Criteria for The Neural Network Model Ne~ra1 Network Development Cycle Chapt~r Three : Classification Boundaries in Neural Networks Boundary Generation Discussion of Boundary Behavior 73 vi

8 Special Cases Positive vi's UiX s ' tends to -GO U.X' tends to +< s Sma11~. small UiX'. and moderate vi's Error Feedback Neural Network Yeight Mat=ices Are Not Statistic of Sample Random Variable X Unpredictability of Neural Network Classification Unpredictable Boundary Generation Behavior Do&~atic Learning Mechanism Prospective Methods for Controlling Neural 83 Net~ork Unpredictability Prior Knowledge about Managerial Pattern Recognition Problems Utility Functions and Monotonicity A Proposed Heuristic for Improving The Neural Network BPLMS Training Algorithm The Monotonic Function Neural Network Model Monotonic Condition for Neural Networks Proper Training Data Set 3.7. Example Applications of The Monotonic Function 100 Neural Network Model An Example of Simulated Data An Example of Real Data Discussion Efficiency of The MF Model Robustness of The MF Model Chapter Four : Classification Bias in The Neural Networks 4.1. and An Approach to Reduce Its Effect Monotonically Separable Problem Learning Bias Effect of Learning Bias A Type of Ambiguity in Managerial Pattern Recognition A Model to Reduce The Effect of Learning Bias Monotonically Separable Problem Model The Integrated Neural Network to Supply Assurance Information The Relationship Between The Monotonic Function (MF) Model and The Monotonically Separable Problem (MSP) Model Minimum Number of Hidden Nodes 134 Chapter Five : Fuzzy Set Representations of Neural Network 5.1. Classification Boundaries... A Problem in Statistical Classification Fuzzy Set Concepts The Fuzzy Set Model A-Complementation in Two Class Classification.. 14H vii

9 Fuzzy Representation of the Typical Neural Network The Fuzzy Membership Model An Example Application of the Fuzzy Membership Model Discussion Practical Fuzzy Membership Function Values The Relationship between MFM, MSPS and FMM 164 Chapter Six : Generalization to More Than Two Classes Classifiers for k>2 Classification Bayes Rule Fisher's Approach Linear Function Classifier Other Classifiers A Possible Extension of The k-2 MF Model The GenerallzE!d MF Model Typical Neural Network Topologies for k>2 Classification A General Algorithm for Neural Networks in k>2 Classification An Example of k>2 Classification Discussion Chapter Seven : Conclusions and Discussion General Conclusions Remarks Future Research 187 Appendix I. Generating Random Sample Appendix II. Back Propagation Algorithm Appendix I II. Monotonic Condition Appendix IV. An Experiment on Green's Data Appendix V. An Experiment on Fisher's Data References [Green 1978] [Fisher 1936] viii

10 NOTATION B B(X) b. 1 C C S C o d a E e F f fz(x) G g h i j k L M m n net a x discriminane funceicn coefficiene vector in Fisher method discriminant function of observation X coefficient of xi in discriminant function set of goal classes goal class, ctc class to which s belongs number of Fisher discriminant functions for more than two group case desired output of neural network node a error measured in learning pcwp.r of the test in sequential classification activation function in neural network distribution function fuzzy function of X number of clusters in the MF model algorithm subscript the number of hidden nodes in two-layer neural network subscript subscript number of goal classes in the set of classes C output of a hidden node in Kolmogorov's theorem momentum dimension of pattern vector dimension of feature vector net input for node a output value of node a output value of node a after learning sample s probability testing data set subscript training data set training data sample point temperature time weight vector of the inputs of i-th hidden node in two layer neural network model constant in Kolmogorov's theorem weight value of the output of i-th hidden node in two layer and single output dim~nsion neural network weight matrix between hidden layer and output layer weight matrix between input layer and hidden layer weight on the ith input to perceptron weight on the connection from node P to Q in neural network pattern vector ix

11 ith component of X feature vector ith component of Y Q fj e 1 general expression of the neural network node general expression of the neural ne~work node preceding to node Q impurity function of r Fisher ratio error signal in neural network learning algorithm constant in Kolreogorav's theorem classification score (from linear discriminant analysis) learning rate threshold in F one-di.mensional feature vector value for T subscript parameter of ~-complementation in fuzzy set mean of a random variable sample size of S transfer function in Ko~mogorov's theorem small number for the training scheme model subscript covariance matrix of multivariate variable covariance tree node in hiernrchical classifier desired value of the feature vector of s cumulative norm~l distribution function mapping function from X to Y transfer function in Kolmogorov's theorem transfer function in Kolmogorov's theorem pattern space angle between pattern vector X and its i-th component xi error tolerance in neural network learning algorithm split in hierarchical classifier x

12 Figure 1.1. Figure 1.2. Figure 1.3. Figure 1.4. Figure 1.5. Figure Figure 1.7. Figure 1.8. Figure 1.9. Figure Figure Figure Figure Figure Figure Figure Figure Figure 2.1. Figure 2.2. Figure 2.3. Figure 2.4. Figure 2.5. Figure 2.6. Figure 2.7. Figure 3.1. Figure 3.2. Figure 3.3. Figure 3.4. Figure 3.5. Figure 3.6. Figure 3.7. Figure 3.8. Figure 3.9. Figure Figure Figure 4.1. Figure 4.2. Figure 4.3. Figure 4.4. Figure 4.5. Figure 4.6. Figure 4.7. Figure 4.8. Figure 4.9. Figure 5.1. FIGURES General Neural Network Model Stock Mean-Variance Chart Pattern Recognition Machine Decision Regions and Decision Boundary Hierarchical Classifier A Split of The Classification Tree Production Rule System for Pattern Recognition Frame Representation of Pattern Recognition Layered Neural Network Model Perceptron Model Linear Decision Boundary XOR (Exclusive OR) Problem A Multi-layer Neural Network Can Generate Arbitrarily Complex Decision Regions Synthesis Problem The Position of Various Classification Techniques in The "Glass Box - Black Box" Spectrum Coordinate Transformation Neural Network Results for the Exponential Boundary Example Neural Network ~ith One Hidden Layer Computational Elements in a Neu~al Network Activation Functions Sigmoid Logic Hebbian Learning Rule Global Minima, Local Minima, and Possible Solutions Neural Network Prototype Development Neural Network Hypercube Two Layer, Single Output Neural Network Boundary Movement Error Feedback Signal Example of Boundary Classification Results Complex Boundaries Violate Monotonicity Clusters of Misclassified Points Vector Analysis of Clusters of Misclassified Points Vector Analysis of toe Trade-off Algorithm Main Steps of the Algorithm to Find a Proper Training Data Set Experimental Result For The Monotonic Function Neural Network Classifier A Monotonically Separable Classification Problem Effect of Small Learning Rate ~ Frontiers of Decision Consequences Monotonically Separable Problem Model Determining the Frontiers of Sample Sets An Integrated Neural Network to Supply Assurance Information "Unbiased" Boundary Example of Uncertain Boundary Generation Experimental Result With Combined MF Linear Discriminant Analysis xi and MSP Models Page Sl

13 Figure 5.2. Figure 5.3. Figure 5.4. Figure 5.5. Figure 5.6. Figure 6.1. Figure 6.2. Figure 6.3. Figure 6.4. Figure 6.5. Figure 6.6. Figure 6.7. Figure 6.8. Figure 6.9. Figure A.l. Graphs Representing Fuzzy Relationships ~-ComplementationRelationship Fuzzy Membership Functions in the Two Class Classification Case Fuzzy Membership Functions Implemented In Conjunction With Neural Network Classification Fuzzy Membership Functions for the Example in Section Linear Functions in the k>2 Case (Scheme 1) Linear Functions in che k>2 Case (Scheme 2) Linear Functions in the k>2 Case (Scheme 3) Split Complex Decision Region Into Subclasses Minimum Distance Classifier in Piecewise Learning Machine Decision Region Complexity Spectrum Decomposition of a Pattern Dimension Two Typical Neural Network Topologies Classifying a New Observation Falling in an Undecided Region m The Behavior of R as a Function of i L wi x r-o rr xii

14 TABLES Table 1.1. Table 1.2. Table 3.1. Table 3.2. Table 5.1. Page Comparison of Classification Techniques 33 Discriminant Analysis for the Exponential Boundary Example 40 Example of the Heuristic Approach to Obtain a Proper Training Data Set 91 A Comparison of the Percentage Correctly Classified by the MF Model and the LOA Method on the Alpha TV Commercial Study Data [Green 1978] 108 A Fuzzy Relationship Between Assets and Creditworthiness 143 xiii

15 CHAPTER ONE INTRODUCTION 1.1. GENERAL DESCRIPTION OF NEURAL NElVORXS For our purposes, artificial neural networks are defined as massively parallel interconnected networks of simple (usually adaptive) elements and their hierarchical organizations, which are intended to react to information o~ the objects of the real world in a manner analogous to biological nervous systems [Kohonen 1987). Neural networks or simply "neural nets" may also be referred to as connectionist models, parallel distributed processing models, and neuromorphic systems [Lippmann 1987J. Neural network architecture may be described in various ways, depending upon its desired function [Lippmann 1987, Hecht Nielsen 1987}. The most general topology of a neural network is shown in Figure 1.1. The neural network system carries ou~ the information processing operation as a mathematical mapping ~ of vector X to vector Y so that Y- (X), where X is the vector of external inputs to the network, and Y is the vector of outputs. Units ("neurons") within the network may receive input signals and/or lateral feedbacks 1

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242