Ž. Pattern Recognition Letters 18 1997 1385 1390 Knowlege reuse in multiple classifier systems 1 Kurt Dewitt Bollacker ), Joyeep Ghosh Department of Electrical an Computer Engineering, UniÕersity of Texas, Austin, TX 78712, USA Abstract We introuce a framework for the reuse of knowlege from previously traine classifiers to improve performance in a current an possibly relate classification task. The approach use is flexible in the type an relevance of reuse classifiers an is also scalable. Experiments on public omain ata sets emonstrate the usefulness of this approach when one is face with very few training examples or very noisy training ata. q 1997 Publishe by Elsevier Science B.V. Keywors: Knowlege transfer; Multiple classifier; Mutual information 1. Introuction Artificial classifiers epen heavily on the set of training samples to make classification ecisions. If the training set insufficiently represents the essence of a classification task, then creation of a well generalizing classifier for that task may not be possible. In the construction of artificial classifiers, the inclusion of previously learne knowlege emboie in existing classifiers is a potential approach to the problem of inaequate training ata. However, both a suitable representation of the knowlege to be reuse, an a mechanism for ientification of pertinent knowlege an its incorporation using that representation must be esigne. We use attributes of human knowlege reuse as a guie to this esign. One of the most impressive traits of human knowlege reuse is the ability to raw simultane- ously from a large number of previous experiences quickly an easily. Each bit of learne knowlege may not help much, but as a whole, the knowlege gaine from experience can paint a very clear picture of the problem omain. Analogously, a practical artificial knowlege reuse system shoul be able to have goo performance scalability with the amount of knowlege reuse. Human flexibility in knowlege reuse is also quite notable. Humans can use knowlege learne from a variety of types of experiences without consiering how that knowlege was gaine. Also, humans are capable of quickly an efficiently picking out learne knowlege that is relevant to the current classification task from their immense boy of experience. A flexible knowlege reuse system shoul be able to take avantage of a iversity of knowlege sources for reuse an have a means to juge the relevance of such knowlege. 1.1. PreÕious work ) Corresponing author. 1 This work was supporte by the Army Research Office contracts DAAH-94-G-0417 an DAAH 049510494 an The National Science Founation. The most common approach to obtaining a ecent generalization given inaequate training sets is to severely constrain the solution space using prior 0167-8655r97r$17.00 q 1997 Publishe by Elsevier Science B.V. All rights reserve. Ž. PII S0167-8655 97 00087-1
1386 ( ) K.D. Bollacker, J. GhoshrPattern Recognition Letters 18 1997 1385 1390 omain knowlege. For example, in Bayesian approaches to classification, such knowlege exists in the form of prior istributions assume for the moel parameters an the choice of prior class probabilities. In the machine learning community, the esign choice is calle inuctive bias of the classifier. For example, in a ecision tree, the bias is inicate by the size of the tree an the variables Žor combinations thereof. consiere for making the branches. In fee-forwar neural networks, the type an number of hien units, amount an form of regularization Ž e.g. weight ecay. serve to constrain the solution. In all of these approaches, knowlege reuse is inirect. More importantly, they work well only if the inuctive bias is a goo match to the current problem. This is often ifficult to attain in practice. Some recent work in knowlege reuse has focuse on the automate extraction an reuse of knowlege from the ata sets of other relevant classifiers, incluing reuse of the traine classifiers themselves. Uner the belief that relate classification tasks may benefit from common internal features, Caruana Ž 1995. has create a multilayer perceptron Ž MLP. base multiple classifier system that is traine simultaneously to perform several relate classification tasks. The first layer of the MLP is common to all tasks an the secon layer is specific to iniviual tasks. The first layer is expecte to learn common features that are useful to all of the relate tasks. Baxter Ž 1994. has evelope a rigorous analysis of a similar type of architecture, showing that as the number of simultaneously traine tasks increases, the number of examples neee per task for goo generalization ecreases. Pratt Ž 1994. has explore a similar knowlege reuse metho in which some of the traine weights from one MLP network are use to initialize weights in an MLP to be traine for a later, relate task. A ifferent approach is taken by Thrun an O Sullivan Ž 1996., who propose a metho to estimate classifier relevance by measuring how much better a classifier performs with a reuse scaling vector for nearest neighbor classifiers. Tasks with mutually helpful scaling vectors can be clustere into relate groups. Recently, popular approaches such as committees, ensembles, an mixture of experts also use multiple classifiers. However, since all these classifiers try to Ž solve the same task though they may specialize in. ifferent input regions, they are not germane to the work presente here. 2. Methos We escribe here an architecture for knowlege reuse from previously traine classifiers. Classifiers traine for the current classification task are calle target classifiers while classifiers previously traine to perform other classification tasks are terme support classifiers as inicate in Fig. 1. Our reuse strategy is to apply the input values of each of the training samples available for the target task to all available releõant classifiers. The output class labels of the target an support classifiers are observe by a secon stage supra-classifier which makes the ultimate classification Žc Ž P. in Fig. 1. A. Since no inter- nal information is being use, the support classifiers can be of any type. 2.1. A few efinitions Let the target classification task be A, an let A have a iscrete range SA an -imensional input 4 omain space R. Let x, y A : xgr, ygsa be the set of training examples for task A. We assume that x, y4a is sample from the true istribution for task A with associate ranom variable Ž X,Y. A Ag Ž R, S. A. Our goal is to fin the most likely value of the conitional marginal Y <Ž X s x. A A an efine this maximum likelihoo function to be t Ž x. A s argmax PYsy Ž < X sx.. Thus, t Ž P. : t Ž P. gs y A A A A A Fig. 1. A supra-classifier reuse architecture.
( ) K.D. Bollacker, J. GhoshrPattern Recognition Letters 18 1997 1385 1390 1387 is the target function that we woul like to approximate using the information in x, y 4. Let c Ž P. A A be a function mapping R SA which is esigne to perform classification task A. Let B be a set of support classification tasks which have the same input omain space R as task A. Let c Ž P.4 B : BgB be the corresponing set of classifiers where each c Ž P. maps R S : BgB. Let Xˆ B B A be the ranom variable associate with the input values of training sample set x, y 4. Let T : T st Ž X. A A A A Aˆ be efine as the ranom variable associate with the target function of X ˆ. Similarly, let C : C sc Ž X ˆ. A A A A A an C : C sc Ž X. B B B Aˆ be the ranom variables resulting from the application of Xˆ A to classifiers c Ž P. an c Ž P. A B, respectively. An ieal supra-classifier c ) Ž x. A will always choose the most likely class ygsa given the class labels c Ž x. an c Ž x.4 A B : B g B. More specifically, for any given k : k gs, k : k gs 4 A A A B B B : BgB we can efine the maximum probability function mž P. as mk, Ž k 4 : bgb, A, B. sargmax PT Ž A B y A sy< C sk, C sk 4 : BgB. A A B B. We can then efine an ieal classifier base on this maximum probability function as c A ) Ž x. smž caž x., cbž x. 4:BgB, A, B., Ž 1. ) Ž. ) where ca P has an associate ranom variable C A : ) ) C sc Ž X ˆ. A A A. In practice, if the number of support classifiers is quite large, Eq. Ž. 1 becomes impractical ue to the curse of imensionality Ž Frieman, 1994.. Therefore, we introuce two approximating approaches to Eq. Ž. 1 in Section 2.3. 2.2. Classifier releõance measure A measure of relevance of each support classifier to the target classification task woul be helpful in the construction of a supra-classifier. We have chosen to use mutual information IŽ P;P. Ža measure of the amount of share information between two ranom variables. with the target istribution as a classifier s relevance to that classification task. If IT;C Ž.)IŽ T;C. A B A B, then we say that c B knows 1 2 1 more about t Ž P. than oes c Ž P. A B 2. We have empirically emonstrate that mutual information can be use effectively as a relevance measure in our knowlege reuse framework ŽBollacker an Ghosh, 1997.. 2.3. Practical supra-classifier methos The problem of esigning a practical supra-classifier can be thought of as esigning a classifier for a task with a large number of iscrete features, many or most of which may only be barely useful. We introuce two supra-classifier approaches that are esigne to scale linearly in their computational requirements with the number of reuse support classifiers in orer to satisfy our esign goal of scalability. 2.3.1. Cascae maximum posterior probabilities Let us relabel the support classifiers c Ž P.4 B : BgB in an orere fashion in the form c Ž P.4 B i : is0... < B< y1 an then revisit Eq. Ž. 1 consiering only c Ž P. an the first support classifier c Ž P. A B 0 to compute cˆ 1 Ž x. : c 1 Ž x. smžcž x,c. Ž x.,a, B. A ˆA A B 0.We 0 then progressively upate our approximation of Eq. Ž. 1 by aing each of the remaining support classinq1 fiers in B one at a time using the form c Ž x. ˆA s mc Ž ˆ n Ž x,c. Ž x.,a, B.. ŽWe efine c Ž P. sc 0 Ž P. A B n A ˆA for n consistency.. As n increases, classification performance on the training example set is strictly non-ecreasing Ž a simple proof omitte for brevity.. In a variation to CMAP, we orer the cascae of support classifiers by ecreasing relevance as a heuristic base on the belief that it woul be beneficial to have the most relevant classifiers be earlier in the cascae. 2.3.2. Hamming nearest neighbor If Ž P. is the inicator function, then the istance measure between two samples xtrn an x tst can be calculate as D Ž x, x. Hamming trn tst s S Žc Ž x./ c Ž x.. B :is0...< B <y1 B trn B tst. For each test i i i example, the Hamming Nearest Neighbor Ž HNN. supra-classifier will choose the class label of the training example with the smallest Hamming istance from it. In a weighte variation of this supraclassifier metho Ž WHNN., the istance contribution of each support classifier Ž 0 or 1. to the total Hamming istance is multiplie by its relevance Žmutual information.. 3. Experiments If there are too few training examples, or if the examples are too noisy, then goo generalization
1388 ( ) K.D. Bollacker, J. GhoshrPattern Recognition Letters 18 1997 1385 1390 may not be possible with the information from the target problem s training examples alone. It is these two cases that we have investigate. In orer to test an compare the supra-classifier methos with unaie target classifiers, we took two public omain ata sets from the U.C Irvine Machine Learning atabase an partitione the examples into two isjoint an unequal size subsets base on their class labels. The subset with fewer classes became the target task. The other subset was use to create several two-class problems using all combinations of two classes. First, a 20 000 example capital English letter ata set Ž LR. onate by Davi Slate was ivie into the target ata set consisting of the five classes H, L, O, R an S an 210 support classifiers consisting of two-class classifiers of the other 21 classes. Secon, a spoken vowel ata set Ž VOW. contribute by Peter Turney consiste of 990 examples evenly istribute among 11 spoken vowels. The two classes hu an he were chosen to form the target classifier task an examples from the remaining 9 classes were use to construct 36 support classifiers. 3.1. Case of few examples The LR ata set of 20 000 training examples was ranomly partitione into equal size base training an test sets. Both target an support classifier training an test sets were create by taking only examples of the target or support classes respectively from the base training an test sets. The target training set was use to create MLP, single nearest neighbor Ž 1-NN., an C4.5 target classifiers for each target problem. The 210 LR support classifiers were traine MLPs. In orer to consier the case of few available target training examples, only a fraction of the available training examples was actually use for training of the target classifier an supra-classifiers. Target training sets of sizes 5, 20, 40, 80, 160, 320 an 480 examples were applie to the MLP an 1-NN target classifiers an all of the support classifiers. The outputs of these target an support classifiers were then use as the input vector for each of the supra-classifiers. Average results over 20 trials can be seen in Fig. 2. The WHNN followe by the unweighte HNN supra-classifiers showe better classification performance than the unaie MLP, Fig. 2. Classification performance of supra-classifiers an unaie classifiers versus number of training examples on the LR ata set. 1-NN an C4.5 classifiers, especially when the number of training examples was very low. The sorte CMAP supra-classifier performe ientically to the unaie 1-NN an the unsorte CMAP performe worse. This gives evience that the information provie by the support classifiers can compensate somewhat for a lack of sufficient training set size. 3.2. Noisy examples A similar experimental setup to the above was mae but for the VOW ata set in the case of noisy examples. For the target classification problems, Gaussian noise was ae to each input vector of the target training set. ŽThe noise covariance matrix was 2 s I.. An MLP target classifier an 36 1-NN support classifiers were use in 100 experimental trials per- Ž 2 forme over a range s s 0 to 16. of noise variances. Average results for the vowel problem are shown in Fig. 3. The performance boost from knowlege reuse in the HNN is quite prominent, but as Fig. 3. Classification performance of supra-classifiers an unaie classifiers versus Gaussian noise variance on the vowel ata set.
( ) K.D. Bollacker, J. GhoshrPattern Recognition Letters 18 1997 1385 1390 1389 expecte, the avantage isappears as the noise level is lowere. 4. Conclusions In both the case of high noise an of few training examples, knowlege reuse from relevant classifiers via an appropriate supra-classifier improve performance while ahering to the flexibility an scalability esign constraints. This happene even though the previously traine support classifiers ha no output classes in common with the target classifier. Thus, we have evience that the knowlege reuse framework presente here can be a practical means for the reuse of knowlege from classifiers that are iverse in form an purpose. We also use a mutual information base relevance measure to guie the construction of some of the supra-classifier methos. This ha mixe results for both the CMAP an HNN supra-classifiers, inicating that a relevance measure may help if use carefully. Although we have shown some encouraging empirical results, there are several irections in which this work can be extene. One of the most important extensions to this work will be application to a truly complex problem omain. We envision the eventual construction of a powerful an broaly applicable warehouse of previously constructe reusable classifiers for a large omain of interest Ž e.g. image atabases., where the set of support classifiers will serve as an efficient representation of the problem omain knowlege. Discussion Rhagavan: I was wonering what the relationship is between the classifier that you re trying to esign an the classifiers that you use as support? Bollacker: I am simply using the probabilities, trying to see if there is some correlation. In the letter recognition ataset you have for instance the two classes H an O. You might expect that a classifier for N an Q might be use as a support classifier. You might expect that the support classifier for N an Q woul be able to say something useful about ifferentiating H an O, since the features for that ataset were statistics on the shapes of the letters an since N is similar to H an Q is similar in shape to O. So, you woul hope that there woul be some useful information that you coul erive from that N, Q classifier. Van Dyck: I want to point out an analogy between what you i an the infotree metho which I presente. There we consiere the pixel as the most primitive classifier. We combine pixels using mutual entropy, to get a higher level classifier. This is repeate in a kin of tree fashion until the final classifier gives the whole pattern. Bollacker: When I first starte looking at this, I thought of something like that. The first omain I consiere was images. But I ecie that there was too much overhea builing a set of classifiers for an image omain. So we use these simpler atasets. In the long term, towars builing a warehouse of classifiers, you woul start to buil classifiers that say very simple things about the image omain. Then using those, you might be able to buil classifiers that say slightly more complex things, an so on, in some sort of hierarchy like you presente. But that seems to be a much longer-term project than what we have one so far. Loew: When you showe the curves of performance, it woul have been helpful if we woul have been able to see some error bars on those points, so we coul have a feel for whether the ifferences between them were significant or not. But perhaps more importantly, I am wonering about your nearest neighbour classifier, which seems to be almost the best or secon in the two cases. I woner whether, if you ha gone to some K-nearest neighbour classifier, if the performance woul have come very close. Do you have any feel for that? Bollacker: About the error bars: the reason that I i not put these on is that the graphs are alreay relatively cluttere. So I just mae sure that I performe a large enough number of trials to make sure that the error bars woul be small. Regaring the other question: are you talking about a single nearest neighbour supra-classifier? Loew: No, I think at the lower level. Bollacker: You are talking about the unaie single nearest neighbour classifier. Actually, I gave these
1390 ( ) K.D. Bollacker, J. GhoshrPattern Recognition Letters 18 1997 1385 1390 unaie classifiers the benefit of the oubt. An it turne out in all the ata sets that I use, that for all K-nearest neighbour classifiers, the single nearest neighbour classifier worke better than those with larger K. Loew: Do you have any feel for why that was so? Bollacker: Well, in fact that was not true in all the ata sets that I use, but it was true for these two. So I just ecie that I woul choose the best one an use that for comparison. Roli: I woul like to have a clarification. Is it correct to say that the concept of knowlege reuse can be regare as a problem of ientifying in a library of classification algorithms, the most inepenent ones, that is the algorithms that make uncorrelate errors? Because, of course, classifiers that make uncorrelate errors are the classifiers most promising to be combine. In your opinion, is this true? Bollacker: We haven t one an analysis to look at error correlation. An certainly what you are suggesting might result in a better relevance measure. So that is certainly something to look at. It coul be ae to the list of future things to o. References Baxter, J., 1994. Learning internal representations. Ph.D. Thesis, The Fliners University of South Australia. Bollacker, K.D., Ghosh, J., 1997. A scalable metho for classifier knowlege reuse. In: Proc. 1997 Internat. Conf. on Neural Networks. Caruana, R., 1995. Learning many relate tasks at the same time with backpropagation. Av. Neural Inform. Process. Systems 7, 657 664. Frieman, J.H., 1994. An overview of preictive learning an function approximation. In: Cherkassky, V., Frieman, J.H., Wechsler, H. Ž Es.., From Statistics to Neural Networks, Proc. NATOrASI Workshop. Springer, Berlin, pp. 1 61. Pratt, L.Y., 1994. Experiments on the transfer of knowlege between neural networks. In: Hanson, S., Drastal, G., Rivest, R. Ž Es.., Computational Learning Theory an Natural Learning Systems, Constraints an Prospects, Chapter 19. MIT Press, pp. 523 560. Thrun, S., O Sullivan, J., 1996. Discovering structure in multiple learning tasks: The TC algorithm. In: Proc. 13th Internat. Conf. on Machine Learning.