Performance Comparison of RBF networks and MLPs for Classification

Transcription

1 Performance Comparison of RBF networks and MLPs for Classification HYONTAI SUG Division of Computer and Information Engineering Dongseo University Busan, REPUBLIC OF KOREA Abstract: - The two well-known neural network algorithms, multilayer perceptrons and radial basis function networks, have different structures and characteristics, so that they have different performance in classification tasks depending on the available training data sets. This paper comparares the performance of the two neural networks with respect to training data set size in classification tasks. Experiments using two real world data sets with the two neural network algorithms show that multilayer perceptrons have relatively better performance for larger data sets and radial basis function networks have relatively better performance for smaller data sets. Key-Words: - multilayer perceptrons, radial basis function networks, training data sets, performance 1 Introduction For the task of machine learning and data mining in various application field artificial neural networks have been very successful [1], and there has been a variety of successful neural network algorithms [2]. For example, multilayer perceptrons (MLPs) for various prediction tasks, Hopfield networks for associative memory and optimization problems, ART networks for autonomous learning systems, boltzman machine for optimization problems, etc. Recent development in radial basis function (RBF) networks draws many researchers attention, because many researchers have reported their good performance in various application areas [3, 4, 5, 6, 7]. One of important application area of machine learing and data mining fields is classification. So, developing neural networks with the smallest error rates for a given data set have been a major concern for their success. MLPs and RBF networks are two major neural networks that have been applied successfully for classification tasks. Even though the two networks are similar in shape, their training mechanisms are very different. MLPs use backpropagation algorithms to train the connection weights so that it takes long time to train, because the backpropagation algorithms rely on some greedy search algorithms like gradient decent. Even though the gradient descent works well in most cases, there is still some possibility of considering local optima as global optima [8]. In structural aspect, RBF networks have three layers including the input layer, hidden layer, and output layer, they differ from a MLP, because in RBF networks the hidden layer performs some computation. [9, 10]. Even though there are many algorithms to determine the structure of the neural networks, basically the structure of the networks is usually determined by the knowledge of human experts with many experiments. As a result, built neural networks may not represent the best knowledge models that are best for some collection of the target data set. It is also true that the performance of neural networks is dependent upon the available data sets. So, in this paper we want to see any relationship between the performance of neural networks, especially for MLPs and RBF networks, and the data set size. In section 2, we provide the related work to our research, and in sections 3 we present our method of experiment, and experiments were run to see the property of the two neural networks in section 4. Finally section 5 provides some conclusions. 2 Related work Artificial neural networks are widely used for machine learning or data mining tasks since the first neural network algorithm, the perceptron [11]. In artificial neural networks there are two kinds of networks based on how the networks are interconnected feed-forward neural networks and recurrent neural networks [12]. RBF networks are one of the most popular feed-forward networks. Papers like [3, 4, 5, 6, 7] show better performance of RBF networks than other neural network algorithms. In RBF networks, local optima problem may occur due to the feed-forward nature of the network and chosen radial functions in the hidden layer. In order to overcome this problem many evolutionary search algorithms were suggested [13, 14, 15]. Evolutionary search algorithms try to find global optimal solutions so ISSN: ISBN:

2 that it is possible to find better RBF networks. But the algorithms require more extensive computing time as well as more elaborate techniques related to the evolutionray computation like the representation technique of network structures and weights. Because the behavior of trained knowledge models also dependent on the taining data set, there is research on sample size as well as the property of samples and sampling scheme. In paper [16] the authors discussed the effect of sample size for parameter estimates in a family of functions for classifiers. In paper [17] the authors prefer small sized samples for feature selection and error estimation for several classifiers of pattern recognition. In [18] the authors showed that class imbalance in training data has effects in neural network development especially for medical domain. In paper [19] the authors found that the accuracy of predictors increases as the sample size increases and the curve of accuracy is logarithmic, so they used the rate of increase in accuracy as stopping criteria for their progressive sampling method. They showed their theory is good with a decision tree algorithm, C The method of experiment Because we want to check the performance of the given neural network algorithms based on the size of training data set, we want to use some large data sets, and we do random progressive sampling to simulate the situations where the available training data set sizes are small to large, and because we want to have large enough test data also in our simulation, we do the geometrical sampling until the sample size becomes almost the half of the target data set. The following is a brief description of the procedure of the experiment for sampling INPUT: a data set s: initial sample size. OUTPUT: Ar, Am /* Ar: the set of accuracy of RBF networks, Am: the set of accuracy of MLPs */ j := 1; Do While s target data set / 2 Do random sampling of size s; Train and test RBF network and MLP; r j := the accuracy of the RBF network; m j := the accuracy of the MLP; A r := A r {r j }; A m := A m {m j }; s := s 2; j++; End while; In the above algorithm we double the sample size until the sample size reaches to about half of the data set size. The used RBF networks and MLPs should have appropriate network structures and appropriate number of iterations for back propagation besed on the characteristics of the given data sets. 4 Experimentation Experiments were run using two data sets in UCI machine learning repository [20] called adult and forest cover types to see the effect of the method. The adult data set [21] is a refined version of census income data set. The census income data set is census in The census income data set is originated from the census bureau database. The number of instances in the adult data set is 48,842. The total number of attributes in the adult data set is forteen, and among them six attributes are continuous attributes and one attribute is a class attribute where it has two classes, yearly income being greater than or equal to 50,000 and less than 50,000. The forst cover types data set [22] includes forest information in four wilderness areas found in the Roosevelt National Forest of northern Colorado. It has twelve continuous attributes as independent variables, while seven major forest cover types were used as a dependent variable. The two data sets were selected, because they are relatively very large and contain lots of values so that they are appropriate for the simulation. We used RBF network using K-means clustering to train for various sample sizes. The average class value distribution of the forest cover types data set in each sample size is (38%, 48%, 16%) for classes (1, 2, 3 to 7) respectively, and the adult data set has two classes. So, we choose two as the number of clusters for K-means clustering for both data sets. The used radial function is Gaussian. In order to train MLPs the given number of hidden layers is the half of the number of attributes plus the number of classes, and the traing time is 500 for the forest cover types data set, because the forest cover types data set contains continuous values only for dependent variables. For the adult data set the given number of hidden layers is the number of class values which is two, and the traing time is 10,000, because the adult data set contains many nominal values in dependent variables. Table 1 and 2 show the result of training for the two neural network algorithms. The initial sample size for training is 200, and the size of samples is doubled as the while loop runs, and we stop sampling when the sample size reaches to about half of the data set size. The rest of the data set after sampling ISSN: ISBN:

3 is used for testing, so we have bigger test set data when sample size is small. Table 1. RBF networks and MLPs for adult data set with different sizes of training data sets Samp. Accuracy of RBF(%) Accuracy of MLP(%) size , , , , , If we look at table 1, we can notice the fact that when sample sizes are small, the accuracy of RBF networks are better. But when sample sizes are large, the accuracy of MLPs are better. Fig. 1 displays the trend of prediction accuracy of RBF networks (dotted line) and MLPs (solid line) for the adult data set more clearly as the training data set size grows. In the figure X axis represents the sample size and Y axis represents prediction accuracy. 3, , , , , , , If we look at table 2, we can notice also the fact that when sample sizes are small, the accuracy of RBF networks are also better, and when sample sizes are large, the accuracy of MLPs are better. Fig. 2 displays the trend of prediction accuracy of RBF networks (dotted line) and MLPs (solid line) for forest cover types data set more clearly as the training data set size grows. In the figure X axis represents the sample size and Y axis represents prediction accuracy. Fig. 2 The accuracy values of RBF networks and MLPs for forest cover types data set with different sizes of training data sets Fig. 1 The accuracy values of RBF networks and MLPs for adult data set with different sizes of training data sets Table 2. RBF networks and MLPs for forest cover types data set with different sizes of training data set Samp. Accuracy of RBF(%) Accuracy of MLP(%) size , Conclusion Artificial neural networks are widely accepted for data mining or machine learning tasks so that it is known that artificial neural networks are one of the most successful tools for prediction. Among many artificial neural networks multilayer perceptrons and radial basis function networks are two representative neural network algorithms that are widely used for classification. Many researchers reported that the performance of radial basis function networks are better than that of multilayer perceptrons for their applications, but some other researchers reported the opposite results. This somewhat conflicting reports may be due to the fact that whatever neural networks are used, the neural networks may not always be the best predictors due to the fact that they are ISSN: ISBN:

4 trained based on some greedy algorithms and the knowledge of human experts. And, more importantly, the performance of the algorithms also dependent on the available training data set. Because the target data sets in machine learning or data mining tasks may not contain enough data that represent the target domain well, the trained neural networks might act poorly in the future. So we want to find out any relationship between training data set size and the performance of neural network algorithms. We experimented the two representive neural network algorithms, RBF networks and MLPs, for classification tasks. We applied a repeated progressive sampling method with various sample sizes to find out if there is any dependency between data set size and the performance of the neural network algorithms. As a conclusion, we found out that the performance of RBF networks are good when the size of training data set is relatively small, but the performance of MLPs are good when the size of training data set size is relatively large. The conclusion was drawn by experiments with two real world data sets. References: [1] D.T. Larose, Data Mining Methods and Models, Wiley-Interscience, [2] C.M. Bishop, Neural networks for pattern recognition, Oxford University press, [3] M. Qu, F.Y. Shih, J. Jing, H. Wang, Automatic solar flare detection using MLP, RBF, and SVM, Solar Physics, vol. 27, no. 1, 2003, pp [4] F.Y. Shih, S.S. Chen, Adaptive document block segmentation and classification, IEEE Transactions on Systems, Man, and Cybernetics, vol. 27, no. 5, 1996, pp [5] S. Marinai, M. Gori, G. Soda, Artificial neural networks for document analysis and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 1, 2005, pp [6] C. Baroglio, A. Giordana, M. Kaiser, M. Nuttin, R. Piola, Learning controllers for industrial robots, Machine learning, vol. 23, no. 2-3, 1996, pp [7] K.Z. Mao, K.C. Tan, W. Ser, Probabilistic Neural Network Structure Determination for Pattern Classification, IEEE Transactions on Neural Networks, Vol. 11, issue 4, 2000, pp [8] S. Russel, P. Novig, Artificial Intelligence: a Modern Approach, 2 nd ed., Prentice Hall, [9] J. Stastny, V. Skorpil, Analysis of Algorithms for Radial Basis Function Neural Network, IFIP International Federation for Information Processing, Vol. 245, Personal Wireless Communications, eds. B. Simak, R. Bestak, E. Kozowska, Springer, 2007, pp [10] R.J. Howlett, L.C. Jain, Radial Basis Function Networks I: recent developments in theory and applications, Physics-Verlag, [11] M.L. Minsky, S.A. Papert, Perceptrons extended edition: an introduction to computational geometry, MIT press,1987. [12] P. Tan, M. Steinbach, V. Kumar, Introduction to Data Mining, Addison Wesley, [13] A. Esposito, M. Marinaro, D. Oricchio, S. Scarpetta, Approximation of Continuous and Discontinuous Mappings by a Growing Neural RBF-based Algorithm, Neural Networks, Vol. 13, No. 6, 2000, pp [14] O. Buchtala, M. Klimek, B. Sick, Evolutionary Optimazation of Radial Basis Function Classifiers for Data Mining Applications, IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, Vol. 35, No. 5, 2005, pp [15] A. Hofmann, B. Sick, Evolutionary Optimazation of Radial Basis Function Networks for Intrusion Detection, Proceedings of the International Joint Conference on Neural Networks, Vol. 1, 2003, pp [16] K. Fukunaga, R.R. Hayes, Effects of Sample Size in Classifier Design, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11, issue 8, 1989, pp [17] S.J. Raudys, A.K. Jain, Small Sample Size Effects in Statistical Pattern recognition: Recommendations for Practitioners, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 13, No. 3, 1991, pp [18] M.A. Mazuro, P.A. Habas, J.M. Zurada, J.Y. Lo, J.A. Baker, G.D. Tourassi, Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance, Neural Networks, Vol. 21, Issues 2-3, 2008, pp [19] T. Oatesm, D. Jensen, Efficient progressive sampling, Proceedings of the Fifth International Conference on Knowledge Discovery and data Mining, 1999, pp [20] D. Newman, UCI KDD Archive [ Irvine, CA: University of California, Department of Information and Computer Science, [21] R. Kohavi, Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid, Proceedings of the scond international conference on knowledge discovery and data mining, 1996, pp [22] J.A. Blackard, J.D. Denis J, Comparative Accuracies of Artificial Neural Networks and ISSN: ISBN:

5 Discriminant Analysis in Predicting Forest Cover Types from Cartographic Variables, Computers and Electronics in Agriculture, vol. 24, no. 3, 2000, pp ISSN: ISBN: