IMPACT OF FUZZY ARTMAP MATCH TRACKING STRATEGIES ON THE RECOGNITION OF HANDWRITTEN DIGITS

IMPACT OF FUZZY ARTMAP MATCH TRACKING STRATEGIES ON THE RECOGNITION OF HANDWRITTEN DIGITS PHILIPPE HENNIGES 1, ERIC GRANGER 1, ROBERT SABOURIN 1 AND LUIZ S. OLIVEIRA 2 1 Laboratoire d'imagerie, de vision et d'intelligence artificielle, École de technologie supérieure, Montreal, Canada, philippe.henniges @livia.etsmtl.ca, eric.granger@etsmtl.ca, robert.sabourin@etsmtl.ca 2 Computer Science Departement, Pontifical Catholic University of Parana, Curitiba, Brazil, soares@ppgia.pucpr.br ABSTRACT Applying fuzzy ARTMAP to complex real-world problems such as handwritten character recognition may lead to poor performance and a convergence problem whenever the training set contains very similar or identical patterns that belong to different classes. To circumvent this problem, some alternatives to the network s original match tracking (MT) process have been proposed in literature, such as using negative MT, and removing MT altogether. In this paper, the impact on fuzzy ARTMAP performance of different MT strategies is assessed using different patterns recognition problems two types of synthetic data as well as a real-world handwritten digit data. Fuzzy ARTMAP is trained with the original match tracking (MT+), with negative match tracking (MT-), and without MT algorithm (WMT), and the performance of these networks is compared to the case where the MT parameter is optimized using a PSO-based strategy, denoted PSO(MT). Through a comprehensive set of computer simulations, it has been observed that by training with MT-, fuzzy ARTMAP expends fewer resources than other MT strategies, but can achieve a significantly higher generalization error, especially for data with overlapping class distributions. Generalization error achieved using WMT is significantly higher than other strategies on non overlapping data with complex non-linear decision bounds. Furthermore, the number of internal categories increases significantly. PSO(MT) yields the best overall generalization error, and a number of internal categories that is smaller than WMT, but generally higher than MT+ and MT-. However, this strategy requires a large number of training epochs to convergence. I. INTRODUCTION Automatic reading of numerical fields has been attempted in several domains of application areas such as bank cheque processing, postal code recognition, and form processing. Such applications have been very popular in handwriting recognition research, due to the availability of relatively inexpensive CPU power, and the possibility of considerably reducing the manual effort involved in these tasks (Oliveira et al., 2002). The fuzzy ARTMAP neural network architecture (Carpenter et al., 1991; Carpenter et al., 1992) is capable of self-organizing stable recognition categories in response to arbitrary sequences of analog or binary input patterns. It has been successfully applied in complex real-world pattern recognition tasks such as handwritten recognition (Granger et al., 2006; Oliveira et al., 2002; Milgram et al., 2005). 1

When learning such complex data, a well known convergence problem occurs when learning inconsistent cases, whenever the training subset contains very similar or identical patterns that belong to different classes (Carpenter and Markuzon, 1998). The consequence is a failure to converge, as identical prototypes linked to inconsistent case proliferate. This anomalous situation occurs as a result of the original match tracking (MT+) process. This convergence problem may be circumvented by using the feature of ARTMAP-IC (Carpenter and Markuzon, 1998) called negative match tracking (MT-). This allows fuzzy ARTMAP to converge but may lead to a higher generalization error. In addition, some authors have studied the impact on performance of removing the MT altogether (Anagnostopoulos and Georgiopoulos, 2003; Marriot and Harrison, 1995) and concluded that the usefulness of MT is questionable. However, training without MT may lead to a network with a greater number of internal categories. In this paper, the impact on fuzzy ARTMAP performance of adopting different MT strategies original MT (MT+), negative MT (MT-) and without MT (WMT) is assessed. As an alternative, the value of the MT parameter has been optimized along with network weight using a strategy based on Particle Swarm Optimization (PSO(MT)). An experimental protocol has been defined such that the generalization error and resource requirements of fuzzy ARTMAP trained with different MT strategies may be compared on different types of synthetic data, and a real-world handwritten numerical character (NIST SD19 data) pattern recognition problem. In the next section, the MT strategies for fuzzy ARTMAP training are briefly reviewed. Section III presents the experimental methodology, e.g., protocol, data sets and performance measures employed for proof of concept computer simulations. Sections IV and V present and discuss experiment results obtains with synthetic and handwritten digits data, respectively. II. MATCH TRACKING STRATEGIES FOR THE FUZZY ARTMAP The fuzzy ARTMAP neural network architecture consists of two fully connected layers of nodes: an M node input layer, F 1, and an N node competitive layer, F 2. A set of real-valued weights W = {w ij [0,1]: i = 1, 2,, M; j = 1, 2,, N} is associated with the F 1 -to-f 2 layer connections. If complement coding is used, M = 2m, where m is the dimensionality of input patterns. Each F 2 node j represents a recognition category that learns a prototype vector w j = (w 1j, w 2j,, w Mj ). The F 2 layer of fuzzy ARTMAP is connected, through learned associative links, to an L node map field F ab, where L is the number of classes in the output space. A set of binary weights W ab = { w ab jk {0,1}: j = 1, 2,, N; k = 1, 2,, L} is associated with the F 2 -to-f ab connections. The vector w ab ( ab, ab,, ab j = wj1 wj2 wjl) links F 2 node j to one of the L output classes. During training, when a mismatch occurs between a predicted response and a desired response for a complement coded input pattern A, original MT+ process raises the internal vigilance parameter ρ just enough to induce another search among F 2 category nodes: ρ = A w J A -1 + ε (1) where J is the index of the F 2 node selected through winner-take-all competition. The original MT+ is parameterized by MT parameter ε, which was originally introduced as a small positive value, 0 < ε << 1. With negative match tracking (MT-) (Carpenter and Markuzon, 1998) ρ is also initially raised but is allowed to decay slightly before a different node J is selected. Then, the MT parameter is set to a small negative value, ε 0, which allows for identical inputs 2

that predict different classes to establish distinct recognition categories. Training without MT (WMT) (Anagnostopoulos and Georgiopoulos, 2003; Marriot & Harrison, 1995) implies creating a new category each time that a predictive response does not match a desired response. Note that training fuzzy ARTMAP WMT is equivalent to performing MT by setting ε = 1. Finally, another strategy consists in using Particle Swarm Optimization (PSO) to determine fuzzy ARTMAP weights and parameter ε such as the generalization error is minimized, denoted PSO(MT). It is based on the PSO training strategy proposed by Granger et al. (2006), but focused only on ε, in a one-dimensional optimization space. PSO is a population based stochastic optimization technique that was inspired by social behavior of bird flocking or fish schooling (Kennedy and Eberhart, 2001). III. EXPERIMENTAL METHODOLOGY Four different MT strategies are considered for training fuzzy ARTMAP neural network : MT+ (ε = 0.001), MT- (ε = -0.001), WMT (ε = 1) and PSO(MT). Training is performed by setting the other three parameters such that the resources (number of categories, training epochs, etc.) are minimized : α = 0.001, β = 1, ρ a = 0. In all cases, training is performed using a hold-out validation strategy (Stone, 1974). That is the learning phase ends following the epoch for which the generalization error is minimized on an independent validation set. The PSO(MT) strategy uses the hold-out validation technique on fuzzy ARTMAP network to calculate the fitness of each particle, and therefore find the network and ε value that minimize generalization error. Other fuzzy ARTMAP parameters are left unchanged. In all simulations involving PSO, the search space of the MT parameter was set to the following range of ε [ 1, 1]. Each simulation trial was performed with 15 particles, and ended after a maximum of 100 iterations (although none of our simulations have ever attained that limit). The training also ends if the global best fitness is constant for 10 consecutive iterations. All but one of the particle vectors were initialized randomly, according to a uniform distribution in the search space. The initial position of the first particle was set to ε = -0.001. After each simulation trial, the performance of fuzzy ARTMAP is assessed in terms of resources required during training, and generalization error on the test sets. The amount of resources required during training is measured by compression rate and convergence time. Compression refers to the average number of training patterns per F 2 node created in fuzzy ARTMAP network. Convergence time is the number of epochs required to complete learning for a training strategy. Generalization error is estimated as the ratio of correctly classified test subset patterns over all test set patterns. Average results, with corresponding standard error, are always obtained, as a result of the 10 independent simulation trials, for randomly generated data sets (synthetic data) and randomly generated data presentation orders (NIST SD19 data). The Quadratic Bayes Classifier (QBC) and k-nearest-neighbors (knn) classifiers were included for reference with generalization error results. For each computer simulation, the value for the k parameter in knn is selected among k = {1, 3, 5, 7, 9} based on hold-out validation using the validation set. In order to observe the effects of the MT parameter value from a perspective of different data structure, several data sets were selected for computer simulations. Two types of synthetic data sets are used to represent pattern recognition problems that involve either overlapping class distributions with linear boundaries, or involve complex decision boundaries, were class distributions do not overlap on decision boundaries. In addition, one real-world data set is representative of a handwritten digits recognition 3

problem. All synthetic data sets are composed of a total of 30,000 randomly-generated patterns, with 10,000 patterns for the training, validation, and test subsets. They correspond to 2 class problems, with a 2 dimensional input feature space. Each data subset is composed of an equal number of patterns per class. In addition, the area occupied by each class is equal. During simulation trials, the number of training subset patterns used for supervised learning was progressively increased from 10 to 10,000 patterns according to a logarithmic rule. The synthetic data sets are call D µ (e tot ) and D CiS. D µ (e tot ) have linear decision boundaries with overlapping class distributions. The degree of overlap is varied from a total theoretical probability of error, e tot = 1% to e tot = 25%, with 2% increments, by adjusting the mean vector µ of class 2. A complete description of this data set and the specific parameters employed to create the 13 data set of D µ are presented in (Henniges et al., 2005). The Circle-in-Square problem D CiS (Carpenter et al., 1991) consists of one non-linear decision boundary where classes do not overlap. The real-world data representing handwritten numerical character is the NIST SD19 data set consisting of images organized into eight series, hsf {0,1,2,3,4,6,7,8}. For our simulations, the data in hsf {0,1,2,3} has been further divided into training subset (52,760 samples) and validation subset (15,000 samples). The training and validation subsets contain an equal number of samples per class. Data in hsf 7 have been used as a standard test subset (60,089 samples). The distribution of samples per class in test set is approximately equal. During pre-processing the set of features extracted from samples consists of a mixture of concavity, contour, and surface characteristics (Oliveira et al., 2002). Accordingly, 78 features are used to describe concavity, 48 features are used to describe contour, and 6 features are used to describe surface. Each sample is therefore composed of 132 features that are normalized between 0 and 1 by summing up their respective feature values, and then dividing each one by its summation. With this feature set, the NIST SD19 data base exhibits complex decision boundaries, with moderate overlap between digit classes. During simulations, the number of training patterns used for supervised learning was progressively increased from 100 to 52,760 patterns, according to a logarithmic rule. IV. RESULTS ON SYNTHETIC DATA Figure 1 presents performance obtained when fuzzy ARTMAP is trained with MT-, MT+, WMT and PSO(MT) on Dµ(13%). Very similar tendencies are found in simulations results where fuzzy ARTMAP is trained with other D µ data set. With more than 20 training patterns per class, the generalization error of MT- and MT+ algorithms tends to increase in a manner that is indicative of fuzzy ARTMAP overtraining (Henniges et al., 2005). However, with more than about 500 training patterns per class, the generalization error grows more rapidly with the training set size for MT- than MT+, WMT and PSO(MT). With a training set of 5000 patterns per class, a generalization error of about 21.22% is obtain with MT+, 26.17% with MT-, 16.22% with WMT, and 15.26% with PSO(MT). The degradation in performance of MT- is accompanied by a higher compression rate and a lower convergence time than other MT strategies. Although it allows to converge for inconsistent cases, MT- produces networks with fewer but larger categories than other MT strategies because of the ε polarity. Those large categories contribute to a lower resolution of the decision boundary, and thus a greater generalization error. By training with WMT, the generalization error is significantly lower than both MT- and MT+ especially with a large amount of training patterns, but the compression rate is the lowest of all training strategies. Therefore, the usefulness of the 4

MT algorithm can be questioned with overlapping data when compared with MT- and MT+, especially for application in which resource are not critical. By using PSO(MT) one creates a network that yields a significantly lower generalization error than all other strategies, and a compression rate that falls between that of WMT and MT- or MT+. With a training set of 5000 patterns per class, a compression rate of about 8.0 is obtain with MT+, 26.4 with MT-, 4.8 with WMT, and 5.3 with PSO(MT). The convergence time is longer with WMT than with MT- and MT+. However, PSO(MT) requires a large amount of training epochs to complete the optimization process. With a training set of 5000 patterns per class, a convergence time of about 8.2 epochs is obtained with MT+, 3.6 with MT-, 12.3 with WMT, and 2534 with PSO(MT). The optimized value of ε tends to grows with the size of the training set from about 0.06 to 0.74. A preference for high positive ε value with overlapping data indicates why WMT yields lower generalization error than MT- and MT+. Recall that the behavior of the fuzzy ARTMAP WMT is equivalent to performing MT by setting ε = 1. (a) (b) Figure 1 Average performance of fuzzy ARTMAP as a function of the training subset size for D µ (13% ). (a) generalization error, (b) compression ratio. Error bar are the standard error of the sample mean. (a) (b) Figure 2 Average performance of fuzzy ARTMAP as a function of the overlap degree with D µ. (a) net generalization error, (b) compression ratio. Let us define the net generalization error as the difference between the generalization error obtained by using all the training data (5,000 patterns per class) and the theoretical error e tot of the database. Figure 2 presents the net generalization error and 5

the compression rate with all training pattern obtained as a function of e tot for all D µ data sets when fuzzy ARTMAP is trained using MT-, MT+, WMT and PSO(MT). Using PSO(MT) always leads to the lowest generalization error overall e tot values, followed by WMT, MT+ and MT-. However, MT- obtains the best compression rate, while PSO(MT) obtains compression between WMT and MT+. Figure 3 presents performance obtained when fuzzy ARTMAP is trained with MT-, MT+, WMT and PSO(MT) on D CiS. In this case, MT+, MT- and PSO(MT) tends towards similar generalization errors across training set sizes, while WMT yields a generalization error that is significantly higher than the others strategies for larger training set sizes. With a training set of 5000 patterns per class, a generalization error of about 1.51% is obtain with MT+, 1.64% with MT-, 4.36% with WMT, and 1.47% with PSO(MT). Compression rate grows as a functions of training set size in a similar way for MT-, MT+ and PSO(MT). With a training set of 5000 patterns per class, a compression rate of 107 is obtain with MT+, 108 with MT-, 14 with WMT, and 109 with PSO(MT). WMT cannot create a network with high compression rate because the non-overlapping data structure creates many categories that overlap on the decision boundaries. However, WMT provides the fastest strategy to convergence, while PSO(MT) is the slowest. With a training set of 5000 patterns per class, a convergence time of about 18.4 epochs is obtain with MT+, 14.4 with MT-, 6.6 with WMT, and 4186 with PSO(MT). PSO(MT) required a large amount of training epochs to optimize the MT parameter. When the training set grows, the optimized value of ε decrease from 0.39 to a small positive value of about 0,01. This is indicative of the similar result obtains with MT+ and PSO(MT) on this data set. Furthermore, all training strategies tested on this data with non linear decision boundaries generate no overtraining. (Henniges et al., 2005). (a) (b) Figure 3 Average performance of fuzzy ARTMAP as a function of the training subset size for D CiS. (a) generalization error, (b) compression ratio. V. RESULTS ON NIST SD19 Figure 4 presents performance obtained when fuzzy ARTMAP is trained with MT-, MT+, WMT and PSO(MT) on NIST SD19. As shown in this figure, MT- and MT+ obtain similar average generalization error across training set sizes. With a training set of 52760 patterns, a generalization error of about 5.81% is obtain with MT+, 6.02% with MT-, 32.84% with WMT, and 5.57% with PSO(MT). When optimizing the MT parameter with PSO(MT), generalization error is lower then other MT strategies with a small number of training pattern, and similar to MT- and MT+ with greater number of training pattern. WMT is unable to create fuzzy ARTMAP network with low 6

generalization error on NIST SD19. Since NIST database possesses complex decision boundaries with a small degree of overlap, WMT cannot generate a good representation of the decision boundaries because it generates too many categories that overlap between classes. With all training data, MT- obtains the best compression rate, followed by MT+, PSO(MT) and WMT. However, with small amount of training patterns, PSO(MT) generates the best compression ratio. With a training set of 52760 patterns, a compression rate of about 237.4 is obtain with MT+, 281.9 with MT-, 2.7 with WMT, and 141.6 with PSO(MT). WMT obtains the lowest compression rate because of the absence of the MT process. With a training set of 52760 patterns, a convergence time of about 15.7 epochs is obtain with MT+, 6.8 with MT-, 1.0 with WMT, and 381 with PSO(MT). WMT still posses the fastest convergence time. The low generalization error of PSO(MT) requires a high convergence time (about 24.3 time higher than MT+ with all training pattern). As with D CiS, when the training set size grows, the optimized value of ε tends toward a small positive value with all training patterns. The optimized value of ε grows with the size of the training set from -0.18 to 0.03. Despite promising results training fuzzy ARTMAP with PSO(MT), other pattern classifiers (such as SVM) have achieved significantly lower generalization error. (Oliveira et al., 2002; Milgram et al., 2005). (a) (b) Figure 4 Average performance of fuzzy ARTMAP in function of the training subset for NIST SD19. (a) generalization error, (b) compression ratio. VI. CONCLUSION A fuzzy ARTMAP applied to complex real-world problems such as handwritten character recognition may achieve poor performance and encounter a convergence problem whenever the training set contains very similar or identical patterns that belong to different classes. In this paper, the impact on fuzzy ARTMAP performance of adopting different MT strategies - original MT (MT+), negative MT (MT-) and without MT (WMT) - is assessed. As an alternative, the value of the MT parameter has been optimized along with network weight using a Particle Swarm Optimization based strategy (PSO(MT)). An experimental protocol has been defined such that the generalization error and resource requirements of fuzzy ARTMAP trained with different MT strategies may be assessed on different types of synthetic and a real-world handwritten numerical character pattern recognition problem. A polarity change in the MT parameter can significantly impact generalization error and the compression on fuzzy ARTMAP neural network, especially with overlapping class distribution. Training with MT- tends to produce fewer categories than the other MT strategies. However, this advantage coincides with a higher generalization error for data with overlapping classes. With overlapping data, the need for a MT process is 7

questionable as WMT yield lower generalization error than MT- and MT+. However, PSO(MT) shows that the MT algorithm is useful by creating fuzzy ARTMAP network with lower generalization error than WMT. To represents data with overlapping class distributions PSO(MT) find ε values that tends toward large positive values, and thereby favors the creation of new categories. With complex decision bounds and no overlap, MT-, MT+ and PSO(MT) obtain a similar generalization error and compression rate. In fact PSO(MT) tends to determine a MT parameter that tends forwards a value similar to that of MT+. WMT yield a higher generalization error than the other MT strategies. Finally, with the NIST SD19 data set, when using all training pattern the generalization error obtain with PSO(MT) is about 0.84% lower than MT-, but comes at the expense of lower compression and a convergence time that can be two order of magnitude grater than other strategies. Multi-objective optimization of both generalization error and compression may mitigate the low compression problem. In addition light weight version of PSO may reduce the convergence time. In this paper, training fuzzy ARTMAP with PSO(MT) has been shown to produce a significantly lower generalization error than with other strategies. Overall results obtained with PSO(MT) underline the importance of optimizing the MT parameter for different problems. The ε values found using this strategy vary significantly according to, training set size and data set structure, and differ considerably from the popular choice with overlapping data. In fact, since all four network parameter (α, β, ρ a and ε) are interdependant, they should all be optimized along with weights, using a PSO-based strategy, for each specific application in mind. Acknowledgement: This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). REFERENCES Anagnostopoulos C.G., Georgiopoulos M., 2003, "Putting the Utility of Match Tracking in Fuzzy ARTMAP Training to the Test", Lecture Notes in Computer Science, 2774, 1-6. Bartfai, G., 1995, "An improved learning algorithm for the fuzzy ARTMAP neural network", Conference on Artificial Neural Networks and Expert Systems, 34-37. Carpenter, G. A., Grossberg, S., and Reynolds, J. H., 1991, "ARTMAP: Supervised Real-Time Learning and Classification of Nonstationary Data by a Self-Organizing Neural Network", Neural Networks, 4, 565-588. Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., and Rosen, D. B., 1992, "Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analog Multidimensional Maps", IEEE Trans. on Neural Networks, 3:5, 698-713. Carpenter, G. A. and Markuzon N., 1998, "ARTMAP-IC and medical diagnosis: instance counting and inconsistent cases", Neural Networks, 11:2,323-336. Granger, E., Henniges, P., Oliveira, S.L. and Sabourin, R., 2006, "Particle Swarm Optimization of Fuzzy ARTMAP Parameters", Int. Joint Conf. on Neural Networks, in press. Henniges, P., Granger, E., and Sabourin, R., 2005, "Factors of Overtraining with Fuzzy ARTMAP Neural Networks", Int. Joint Conf. on Neural Networks, 1075-1080. Kennedy, J., and Eberhart, R. C., 2001, Swarm Intelligence, Morgan Kaufmann Edition. Marriott, S. and Harrison R.F., 1995, "A modified fuzzy ARTMAP architecture for the approximation of noisy mappings", Neural Networks, 8:4, 619-41. Milgram, J., Chériet, M. and Sabourin, R., 2005, "Estimating Accurate Multi-class Probabilities with Support Vector Machines", Int. Joint Conf. on Neural Networks, 1906-1911. Oliveira, L. S., Sabourin, R., Bortolozzi, F., and Suen, C. Y., 2002, "Automatic Recognition of Handwritten Numerical Strings: A Recognition and Verification Strategy", IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:11, 1438-1454. Stone, M., 1974, "Cross-Validatory Choice and Assessment of Statistical Predictions", Journal of the Royal Statistical Society, 111-47. 8