Recognition of phonemes in continuous speech using a modified LVQ2 method

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) Recognition of phonemes in continuous speech using a modified LVQ2 method Shozo Makino,* Mitsuru Endo,** Toshio Sone,*** and Ken'iti Kido**** *Research Center for Applied Information. Sciences, Tohoku University, 211 Katahira, Aobaku, Sendai, 980 Japan **Matsushita Research Institute Tokyo Inc., Higashimita, Tamaku, Kawasaki, 214 Japan ***Research Institute of Electrical Communication, Tohoku University, 211 Katahira, Aobaku, Sendai, 980 Japan ****Department of Information Engineering, Chiba Institute of Technology, 2171 Tsudanurna, Narashino, 275 Japan (Received 19 February 1992) This paper proposes a new phoneme recognition method based on the Learning Vector Quantization (LVQ2) algorithm proposed by Kohonen. We propose three versions of a modified training algorithm to overcome a shortcoming of the LVQ2 method. In the modified LVQ2 algorithm, p reference vectors are modified at the same time if the correct class is within the Nth rank where N is set to some constant. Using this al gorithm, the phoneme recognition scores obtained by the modified LVQ2 algorithm were higher than those obtained by the original LVQ2 algorithm. Furthermore, we propose a segmentation and recognition method for phonemes in continuous speech. At first a likelihood matrix is computed using the reference vectors, where each row indicates the likelihood sequence of each phoneme and each column indicates the likeli hood of all phonemes for each 10ms unit. The optimum phoneme sequence is com puted from the likelihood matrix using the DP with duration constraints. We applied this method to a multispeakerdependent phoneme recognition task for continuous speech uttered Bunsetsu by Bunsetsu. The phoneme recognition score was 85.5% for the speech samples in continuous speech. Keywords:Phoneme recognition, Learning vector quantization, Phoneme segmenta tion, Neural network, Discriminative training PACS number:43. 72. Ne 1. INTRODUCTION The neural network approach is one of the most promising approaches to a phoneme recognition task. We proposed phoneme recognition methods based on the multilayer perceptrons, 1,2) which produced a hypothesis of a phoneme sequence from continuous speech. However, there is a problem in robustness for variations in duration within a phoneme and for speaker differences, because these systems used a single phoneme reference vector with the length set to the average duration of each phoneme. The Learning Vector Quantization (LVQ, LVQ2) methods were proposed by Kohonen et al. 3) They showed that the LVQ2 was superior to the LVQ. However, in the LVQ2 method, two reference vec tors are modified at the same time if the first nearest class to an input vector is incorrect while the second nearest class to the input vector is correct. If the given vector is recognized as the third rank, the modification is not performed. McDermott et al. 4) 351

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) developed a shifttolerant phoneme recognition system based on the LVQ2 method. This system uses multiple reference vectors with a duration of 70 ms for each phoneme. By adopting multiple reference vectors with lengths shorter than the average duration of the phoneme, the system ob tains a robustness for the variation in duration with in the phoneme and for the variation in speaker characteristics. However, this system does not perform segmentation and recognition of phoneme in continuous speech. The system only performs discrimination of a single phoneme from a given phoneme group for a given segment. Iwamida et al.5) developed an LVQHMM phoneme recogni tion system. In this system, speech is first trans formed to a vectorcode sequence using a code book obtained by the LVQ2 method and then the discretetype HMM is applied to the vectorcode sequence. This phoneme recognition system also only performs discrimination. As noted above, the following problems in con structing a phoneme recognition system using the LVQ2 method still remain: (1) No training algorithm if the rank of the given vector is greater than 2, and (2) No segmentation and recognition method for continuous speech. This paper describes a phoneme recognition sys tem based on the Learning Vector Quantization algorithm and the DP. The reference vectors are trained using the modified Learning Vector Quanti zation algorithm (MLVQ2) which we propose in this paper. Each phoneme has multiple reference vectors with a duration of 70 ms. Accordingly, the system can also deal with the variation in dura tion within a phoneme and the speaker variation. We then investigate the optimum dimension for representing the reference vectors. Finally we construct a phoneme recognition system which produces a hypothesis of a phoneme sequence from continuous speech using the DP with phoneme duration constraints. 2. MODIFIED LEARNING VECTOR QUANTIZATION ALGORITHM (MLVQ2) In the LVQ2 algorithm, the reference vector is modified when a given training vector x satisfies the following three conditions:1) the nearest class to the given vector must be incorrect, 2) the next nearest class to the given vector must be correct, and 3) the training vector must fall inside a small, sym metric window defined around the midpoint of the incorrect reference vector and the correct reference vector. In our preliminary phoneme recognition experi ments using the LVQ2 algorithm, we found that the given training vector hardly contributed to the learning when the rank of the given vector was greater than 2. In this paper, we propose three versions of a modified training algorithm for the LVQ2 method. Figure 1 shows a flowchart of the modified LVQ2 algorithm. In the modified LVQ2 (MLVQ2) al gorithm, p reference vectors are modified at the same time if the correct class is within the Nth rank where N is set to some constant. The modified LVQ2 algorithm consists of the following 6 steps. In step 1, reference vectors are chosen using the K Means clustering method from each class. In step 2, the reference vector of each class nearest to an input vector is selected. In step 3, the rank of the correct class is computed. When the rank of the correct class is n, we assume that the reference vector of the correct class is mn. In step 4, n is checked to see whether or not n falls in the range 2 n N. In step 5, a check is made to see whether or not the input vector falls within a small window, where the window is defined around the midpoint of m1 and mn. In step 6, the ith reference vector is modified according to one of the following three versions of the modified LVQ2 algorithm. MLVQ2.a6) step 6: MLVQ2.b step 6: MLVQ2.c step 6: where, (1) (2) (3) (4) (5) t is the sample number, T=No. of iterations ~No. of (6) (7) 352

S. MAKINO et al.:recognition OF PHONEMES USING A MODIFIED LVQ2 MLVQ2.a MLVQ2.b NILVQ2.c LVQ3[McDermott] Fig. 1 Modified LVQ2 algorithm. samples, and ƒ 0 is the initial learning coefficient equal to 0.02 After we proposed MLVQ2.a, McDermott pro posed the LVQ3 7) and Kohonen also proposed another LVQ3. 8) Recently McDermott and Kata giri 9) proposed several generalized training algo rithms for various speech units based on the gener alized probabilistic decent method. 10) The LVQ3 proposed by Kohonen is almost the same as the LVQ2, while one more modification equation is added. That is, if an input and the two closest classes belong to the same class, the modification is performed so that the mean vector mi can approxi mate the class distribution. McDermott and Katagiri found that the generalized training algo rithm for the segment unit corresponded to the LVQ3 proposed by McDermott when the approximation was extremely simplified. Until the present we can not make clear the relation between the MLVQ2 and the generalized training algorithms. Hereafter, we compare the MLVQ2 with the LVQ3 proposed by McDermott. Figure 2 shows examples of the three versions together with the LVQ3 proposed by McDermott.7) In the MLVQ2.a, if the correct training vector is recognized as the nth rank, the top (n1) reference vectors are moved away by ƒ n while the nth refer ence vector is moved nearer by ƒ n. In the MLVQ2.b, Fig. 2 Examples of various LVQ algo rithms(rank of correct vector=4). Num bersindicate ranks. the (n1)th reference vector is moved away by ƒ 1 while the nth reference vector is moved nearer by In the MLVQ2.c, the top n1 reference vectors are moved away by ƒ 1/(n1) while the nth refer ence vector is moved nearer by ƒ 1. In the LVQ3 proposed by McDermott, the first reference vector is moved away by ƒ while the nth reference vector is moved nearer by ƒ. 3. COMPARISON AMONG THE VARIOUS LVQ ALGORITHMS Recognition experiments were carried out to compare the various LVQ algorithms. The recogni tion experiments were carried out for a phoneme segment where the beginning and final frames of each input phoneme were given. Training was carried out for the phoneme samples in a 212word vocabulary uttered by 7 male and 8 female speakers. The recognition experiments for the 30 phonemes shown in Table 1 were carried out for the phoneme samples in the 212word vocabulary uttered by another set of 3 male and 2 female speakers. Speech was analyzed by a 29channel bandpass 353

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) Table 1 Thirty phonemes to be recognized. an activation value aw as defined by (8) filter bank. The speech was represented by a se quence of logarithmic spectra with 10ms frame shift. The phoneme recognition system, for the com parison was similar to the shifttolerant model pro posed by McDermott et al. 4): (1) Eight melcepstrum coefficients and 8 4 mel cepstrum coefficients were computed for every frame from the logarithmic spectrum. The value of each coefficient was normalized by the maximum magnitude of the coefficient. Each reference vector was represented by 112 co efficients (7 frames ~16 coefficients). Each class was assigned 15 reference vectors chosen by the KMeans clustering method. (2) A 7frame window was moved over the given phoneme segment and yields a 112dimensional input vector every frame. (3) In the training stage the various LVQ2 al gorithms were applied to the input vector as previously described. (4) In the recognition stage we computed the dis tances between the input vector of each frame and the nearest reference vector within each class. (5) From these distances, each class was assigned where d, c and t are distance, class, and frame number, respectively. (6) The activation value of each frame was summed over the given phoneme segment. (7) The class with the maximum activation value was regarded as the recognized output. Figure 3 shows the relation between the recogni tionscores for the consonants /b/, /d/ and /g/ and the number of iterations. In the LVQ2 method, the top2 recognition scores reached a plateau after a few iterations and was lower compared to those obtained by the MLVQ2.a. That is, the modified LVQ2 methods gave higher accumulated recognition scores because the modification was carried out even when the rank of the given vector was greater than 2. Thus, the recognition scores obtained by the MLVQ2 methods were higher than those obtained by the LVQ2 method. Table 2 shows the phoneme recognition scores obtained by the various algorithms. The recogni tionscores obtained by the modified LVQ2 algo rithms were higher by about 5% than those obtained by the LVQ2 algorithm. However, the MLVQ2.a gave a lower recognition score compared to the LVQ2 when the number of iterations was increased and N was set to 30. In the MLVQ2.a algorithm, there remains a problem in choosing the value of N. The recognition scores obtained by the MLVQ2.b and MLVQ2.c increased as the number of iterations and N were increased. Accordingly we used the MLVQ2.b for phoneme recognition hereafter. LVQ2 MLVQ2.a Fig. 3 Relation between number of iterations and percent correct, 354

S. MAKINO et al.:recognition OF PHONEMES USING A MODIFIED LVQ2 No. of iteration=10. Table 2 Comparison among various LVQ algorithms. 4. INVESTIGATION ON OPTIMUM DIMENSION OF REFERENCE VECTORS In the experiments described in section 3, we used the 112dimensional vector computed from the 7frame window, where each frame was, represented by the 8 melcepstrum coefficients and the 8 mel cepstrum coefficients computed by a regression analysis over 5 frames. In this section, we investi gate the optimum dimension for representing a reference vector. We should investigate the follow ing dimensions: (1) Number (Nc) of the melcepstrum coefficients computed in every frame (2) Number (Nd) of the melcepstrum coefficients computed in every frame (3) Number (Ns) of frames of the time span for computing melcepstrum (4) Number (Nw) of frames of the time window of the reference vector We defined the optimum dimension by phoneme recognition experiments for /b/, /d/ and /g/ samples in the 212word vocabulary uttered by 3 male and 2 female speakers, where the reference vectors of each phoneme were constructed using speech samples uttered by another set of 7 male and 8 female speakers. At first, N, was set to 7 and then the optimum Nc and Nd were examined for. Ns=3, 5, 7. Figure 4(a) shows the results for Ns=5. Two parameter sets gave the best recognition scores for the test set: the combination of 8 melcepstrum coefficients and 8 melcepstrum coefficients, and the 16 melcepstrum coefficients. A similar tendency was observed for Ns=3 and Ns=7. Next, the optimum Ns was examined when using the parameters pre viously defined. Figure 4(b) shows the results. The two sets of parameters showed the best recogni tion scores at Ns=5. There was no significant difference between the two sets of parameters in the recognition scores. We used the 8 melcepstrum coefficients and the 8 melcepstrum coefficients obtained from the 5frame span hereafter. Under the conditions previously described, we investigated the optimum Nw. Figure 4(c) shows the results. The recognition scores show a fluctuation. We used 7 frames for Nw since the recognition score reached a plateau for the training set and a relatively higher recognition score was obtained for the test set. 5. SELECTION METHOD OF INITIAL REFERENCE VECTORS As previously described, we used 15 reference vectors for each phoneme obtained by the KMeans clustering method. However, it still remains whether or not the 15 reference vectors is sufficient and which method is appropriate for selecting the initial reference vectors. In this section, we in vestigate on these questions. At first, we changed the number of reference vectors selected by the KMeans clustering method. Figure 5 shows the result for the 30 phonemes. As can be seen from the figure, the recognition score does not change much when the number of reference 355

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) (a) Effect of number of coefficients of melcepstrum (b) Effect of number of frames of time span for computing melcepstrum (c) Effect of number of frames of time window Fig. 4 Investigations on optimum dimensions for reference vectors. recognizing the phoneme. Next we investigated the following four methods for selecting the initial reference vectors. (1) Method 1:each phoneme is assigned 15 refer ence vectors selected by the KMeans clustering method. (2) Method 2:the number of reference vectors of each phoneme is changed based on the average distortion D defined by (9) where, mi, is a reference vector and x is an input vector. x mi denotes that mi is the nearest Fig. 5 Relation between number of refer ence vectors of each phoneme and percent correct. vectors is greater than 10. Accordingly when each phoneme was assigned the same number of reference vectors, the 15 reference vectors were sufficient for reference vector to the input vector x. For comparison with method 1, when the total number of reference vectors is set to 450 and the maximum number of the reference vectors of each class is limited to 30, the number of ref erence vectors is so defined that the averagedistortion of each class is almost the same. 356

S. MAKINO et al.:recognition OF PHONEMES USING A MODIFIED LVQ2 Table 3 Number of reference vectors ob tained by method 2 and method 3. Method 2 Table 4 Recognition scores for four methods for selecting the initial reference vectors. Method 3 Table 3 shows the number of reference vectors. The obtained reference vectors are used as the initial reference vectors for the training. (3) Method 3:the number of reference vectors of each phoneme is automatically defined in the training process of the MLVQ2. m is defined as the average vector of the whole training speech data. Each phoneme class always has the vector m in addition to the reference vec tors. At the first step each phoneme only has the single vector m. When the average vector m satisfies the conditions of the modification and the average vector is moved nearer to the input vector, the modification is carried out. After the modification is over, the modified average vector is reassigned to the phoneme class and the average vector m is again added to the phoneme class. When the average vector is moved away, the modification and the assignment are not carried out. The training is carried out until the total number of true reference vectors reaches 450. Table 3 also shows the number of reference vectors of each phoneme obtained by method 3. (4) Method 4:the KMeans clustering method is applied to each class using the number of reference vectors defined by method 3. Table 4 shows the recognition scores obtained by the four methods mentioned previously. Method 2 gave the worst recognition scores among the four methods and needed the longest computation time. Method 3 gave the best recognition scores for the training set and needed the shortest computation time. However, the recognition score for the test set was not so high. Method 4 showed the best recognition score for the test set. This indicates that method 3 gave a nearly optimum number of reference vectors. Method 1 gave a moderate recognition score and a moderate computation time. Hereafter we used method 1 for selecting the initial vectors for the training. 6. RECOGNITION OF PHONEMES IN SPOKEN WORDS The recognition scores described in the previous sections were obtained for given segments. It is necessary to carry out segmentation of speech for an acoustic processor in a continuous speech rec ognition system. It is desirable to carry out simultaneous recognition and segmentation of phonemes. In this paper, we used the DP with phoneme duration constraints for the recognition and segmentation of phonemes in continuous speech, where the beginning and final frames of the input speech were given. The phoneme recognition system is as follows: (1) A 7frame window is moved over the input speech and yields a 112dimensional input vector per frame. (2) The distances between the input vector and the nearest reference vector within each class are computed every frame. (3) An optimum hypothesis for a phoneme se quence is made using the DP by taking into account phoneme duration constraints. The DP equation with phoneme duration con straints is as follows: Initialization:G(0)=0 DP equation: for j:=1 to N do begin 357

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) for i:=1 to j do begin classes where (ji+1) satisfy their duration constraints) Table 5 Recognition scores for four kinds of duration constraints. where distc(k) is the distance between the kth frame vector and phoneme class c end end This DP equation is similar to the 2level DP. 11) However, the first stage DP is substituted with linear matching. The following four kinds of constraints were examined for integrating to the DP. (a) Minimum and maximum duration constraints of a phoneme independent of the context. (b) Phoneme connection constraints between suc cessive two phonemes in addition to constraint (a). (c) Minimum duration constraints of a phoneme dependent on the preceding phoneme, where the maximum duration constraints of a phoneme is defined independent of the context. (d) Minimum duration constraints of a phoneme dependent on the preceding phoneme, where no constraints are used for the maximum dura tion constraints of a phoneme. The recognition experiments were carried out for evaluating the effectiveness of the duration con straints mentioned above. The training data and the test data were the same as those described in section 3. Table 5 shows the recognition scores for the various constraints. As can been seen from Table 5, the minimum duration constraints of a phoneme dependent on the preceding phoneme are very effective. On the contrary, the maximum duration constraints of a phoneme are not necessary. Next we investigated the effectiveness of the following methods, where dc is the minimum Euclid distance within a phoneme class c. (a) Method using the square of the Euclid distance (b) (c) (d) Table 6 Recognition scores for four methods. and the DP (10) Method using the activation value defined by Ep. (8) and the DP Method using the activation value and the DP for selecting an optimum phoneme se quence 12,13) Method using the logarithmic activation value and the DP Table 6 shows recognition scores for the test set using the various methods. By comparing method (b) to method (c), the DP is superior to the DP for selecting an optimum phoneme sequence. All distances or activation values gave similar per formances. 7. RECOGNITION OF PHONEMES IN CONTINUOUS SPEECH Recognition experiments were carried out for continuous speech uttered Bunsetsu by Bunsetsu. Each of two adult male uttered 148 sentences. The sentence speech were analyzed in the same fashion as described in section 3. Additional training was carried out for phoneme samples in 70 sentences 358

S. MAKINO et al.:recognition OF PHONEMES Table 7 Relation between recognition, scores and number of iterations in addi tional training. USING A MODIFIED 6) 7) 8) 9) uttered by the two male speakers, where the ref erence vectors of each phoneme obtained from the spoken words were used as the initial values. A recognition experiment for 30 phonemes was carried out for the phoneme samples in the remaining 226 sentences uttered by the same two speakers. Table 7 shows the relation between the recognition scores and the number of iterations in the additional training. The recognition scores reached a plateau after ten iterations. 8. CONCLUSION We proposed a modified LVQ2 algorithm and showed its superiority to the original LVQ2 algo rithm. We also showed that the DP using the Euclid distance obtained by the MLVQ2.b gave the best performance. The minimum duration of a phoneme dependent on the preceding phoneme was the most effective constraint in achieving a high recognition score. In order to apply the reference vectors ob tained from spoken words to continuous speech, ten iterations in the additional training were sufficient for the adaptation. REFERENCES 1) S. Makino, T. Kawabata, and K. Kido,"Recogni tionof consonant based on the perceptron model," Proc. ICASSP83, 738741 (1983). 2) S. Moriai, S. Makino, and K. Kido,"Phoneme recognition in continuous speech using phoneme discriminant filters," Proc. ICASSP86, 22512254 (1986). 3) T. Kohonen, G. Barna, and R. Chrisley,"Statistical pattern recognition with neural networks:bench marking studies," IEEE Proc. ICNN, Vol.1, 6168 (1988). 4) E. McDermott and S. Katagiri,"Shiftinvariant phoneme recognition using Kohonen networks," Proc. Autumn Meet. Acoust. Soc. Jpn., 217218 (1988). 5) H. Iwamida, S. Katagiri, E. McDermott, and Y. Tohkura,"A hybrid speech recognition system 10) 11) 12) 13) LVQ2 using HMMs with an LVQtrained codebooks," Proc. ICASSP90, 489492 (1990). M. Endo, S. Makino, and K. Kido,"Phoneme recognition using a LVQ2 method," Trans. IEICEJ SP 8950 (1989)(in Japanese). E. McDermott,"LVQ3 for phoneme recognition," Proc. Spring Meet. Acoust. Soc. Jpn., 151152 (1990). T. Kohonen,"The selforganizing map," Proc. IEEE 78, 14641480 (1990). E. McDermott and S. Katagiri,"Prototypebased minimum error classification for various speech units," Proc. Autumn Meet. Acoust. Soc. Jpn., 117118 (1991). S. Amari,"A theory of adaptive pattern classifiers," IEEE Trans. Electron. Comput. 16, 299307 (1967). H. Sakoe,"Twolevel DPmatchingA dynamic programming based pattern matching algorithm for connected word recognition" IEEE Trans. Acoust. Speech Signal Process. ASSP27, 588595 (1979). S. Makino, S. Moriai, and K. Kido,"A method for selecting an optimum phoneme sequence using a posteriori probabilities of phonemes," J. Acoust. Soc. Am. Suppl. No.1, PPP5 (1988). S. Makino, A. Ito, M. Endo, and K. Kido,"A Japanese text dictation system based on phoneme recognition and a dependency grammar," Trans. IEICEJ E74, 17731782 (1991). Shozo Makino was born in Osaka, Japan, on January 3, 1947. He received the B.E., M.E. and Dr. en gineeringdegrees from Tohoku Uni versity, Sendai, Japan, in 1969, 1971 and 1974 respectively. In 1974, he worked with the Research Institute of Electrical Communication, Tohoku University as a Research Associate. In 1980, he worked with the Research Center for Applied Information Sciences, Tohoku University as a Research Associate. From 1984 to 1986 he worked with the Speech Technol ogy Laboratory, Santa Barbara, California, USA, as a visiting research scientist. Since 1987 he has been an Associate Professor of Information Science at the Research Center for Applied Information Sciences, Tohoku University. His present research interest is in spoken language processing, speech database, image processing and digital signal processing. Mitsuru Endo was born in 1966. He received the B.E. and M.E. degrees from Tohoku University, in 1989 and 1991. He went on to join the Matsu shita Research Institute Tokyo, Inc., where he has been engaged in speech recognition research and develop ment. He is a member of the Acousti 359

J. Acoust. Soc. Jpn.(E) 13, 6 (1992) cal Society in Japan. Toshio Sone was born on 14 May 1935. A graduate in electrical en gineering at Tohoku University, Japan in 1958, Sone did his graduate work at the same university, where he was awarded his Ph. D in electrical and communication engineering in 1963. He joined the Faculty of Engineering, Tohoku University as a Research Associate in 1963, be coming an Associate Professor in 1964 and a Full Pro fessor in 1979. He is now a Professor of the Research Institute of Electrical Communication, Tohoku Uni versity. He has been engaged in researches on psycho logical acoustics, electroacoustics, room acoustics and noise control for over thirty years, and is a fellow mem ber of the Institute of Noise Control Engineering of the USA, a member of the Board of Directors of INCE/ Japan, and a member of some other academic societies. He was a member of the Board of Directors of the Inter national Institute of Noise Control Engineering from 1985 to 1990 and also acted as SecretaryGeneral of the Western Pacific Commission for Acoustics from 1989 to 360 1991. Ken'iti Kido was born in Hamhun, Korea, on April 15, 1926. He received the B.E. and Dr. engineering degrees form Tohoku University, Sendai Japan, in 1948 and 1962 respectively. In 1948, he worked with the Research Institute of Electrical Communication, Tohoku University as a Research Associate. In 1957 he was an Associate Professor in the Faculty of Engineering. In 1963 he was a Professor of Acoustics at the Research Institute of Electrical Com munication, and from 1976 to 1990 he was a Professorof Information Sciences at the Research Center for Applied Information Sciences, Tohoku University, and the Direc tor of the Center. Since 1990 he has been a Professor of Department of Information Engineering at the Chiba Institute of Technology, Narashino, Japan. His present research interest is in digital signal processing and its application to acoustics and speech recognition. Dr. Kido is a Fellow of the Acoustical Society of America. From 1983 to 1985 he was President of the Acoustical Society of Japan.