Unvoiced Landmark Detection for Segment-based Mandarin Continuous Speech Recognition Hua Zhang, Yun Tang, Wenju Liu and Bo Xu National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences, {hzhang,tangyun,lwj}@nlpr.ia.ac.cn, xubo@hitic.ia.ac.cn Abstract. This paper presents an attempt to introduce unvoiced landmarks into statistical continuous speech recognition system. The unvoiced landmark detection algorithm proposed here locates the points in speech where the vocal folds stop or begin freely vibrating. In our experiments, 87.47% of stops and 98.94% of fricatives are segmented from speech after the unvoiced landmark detection, with a very low insertion error rate of 0.13%. Then these landmarks are incorporated into decoding process of segment model based recognizer as search beginning indicators. The effectiveness of landmark detection algorithm is verified in our landmark-guided recognition system with 240 sentences in 863- Test database. Key words: Speech recognition, segment model, landmark detection 1 Introduction The study of integrating linguistic and phonetic knowledge into current probabilistic speech recognition system is being put increasing importance on recently. Although current dominant statistical model-based recognition systems have obtained rapid development because of elaborate modeling ability and data-driven methods, the blind search in large data without heuristic guide still needs lots of data and compute resource. To find a new development for automatic speech recognition technique after introducing the statistical model into speech recognition domain, the methods of combining phonetic knowledge and statistical models are considered to be a potential way[1][3]. Human speech is pronounced by the movements of articulators under the control of brain. It is believed that acoustic cues for speech perception are most obvious around those points with rapid acoustic variation, just as the close and release of consonants. These points act as landmarks in human speech perception[2], so should they do in machine speech recognition. Landmarks have the ability to indicate the beginning of abrupt acoustic changes, such as close and release of consonants[3], or the maximal point of acoustic energy for sonorants[4]. Speech recognition system utilizing landmarks were also explored[5] [6] [7]. An unvoiced landmark detection system is built in this paper, and is incorporated into a Segment Model(SM) based large vocabulary continuous speech
recognition system[8] [9]. Our approach presented here is quite different from previous works in several regards. Firstly, at landmark detection stage, our goal is unvoiced landmarks, +unvoiced and -unvoiced, which describe the beginning and ending of articulators constricting when pronouncing consonants. The - unvoiced landmarks correspond to the articulators closure and the vocal folds stop freely vibrating, which appear in the beginning of stops closure, or the beginning of friction in fricatives and affricates; +unvoiced landmarks indicate articulators release and the vocal folds start to freely vibrating, as the beginning of vowel after a fricative, or the release of stops. Not all consonants have these unvoiced landmarks. Without salient narrowing and release in articulators when pronounced, there is no unvoiced piece in sonorant consonants. So the unvoiced landmarks could appear for three kinds of speech: stops, fricatives and affricates. Those speech is our goal to be segmented from continuous speech at landmark detection stage. Secondly and importantly, information about these unvoiced segments given at the detection stage is merged into a SM-based speech recognition system[14], rather than traditional HMM-based system. In HMM-based system, knowledge information is integrated before model decoding in the form of knowledge-based parameters[10][11][12], or after decoding process by candidates rescoring[13], but hardly integrated in decoding process directly. Modeling and decoding using the units of segment, instead of frame in HMM, the SM system has inherent structure for integrating linguistic and phonetic knowledge in speech recognition. In our system, location information about unvoiced segments is directly used in decoding process to seek the start points for decoding segments given end points, rather than by using defined segment length. The remainder of this paper is organized as follows. Background knowledge about consonants in Mandarin is introduced in section 2. The unvoiced landmark detection method is presented in section 3. How the landmark information is applied in SM system is described in section 4, then the experiments and results in section 5. Finally, we conclude this paper and give remarks on future work. 2 Characteristics of Consonants in Mandarin An important difference between consonants in Mandarin and in English is that all stops, fricatives and affricates in Mandarin are unvoiced, and only sonorant consonants are voiced. This phonetic knowledge provides a larger scope to using unvoiced landmark detection in Mandarin speech than in English, whose stops, fricatives and affricates include both voiced and unvoiced ones. Consonants in Mandarin contain 6 stops, 5 fricatives, 6 affricates and 6 sonorant consonants. Their corresponding phones and occur frequencies in 863-Train database(introduced later) are listed in table 1. The stops in Mandarin have similar close and burst actions as the unvoiced counterparts in English, and these actions are used as cues to detect unvoiced landmarks. Although affricates have burst like stops, their fricative character are more obvious to detect. So we group affricates into fricatives in our system. As shown in table 1, another evident dif-
ference of consonants in Mandarin to those in English is that there are more fricatives in Mandarin, and those fricatives take a large part in Chinese spoken language. In conclusion, we believe that effective and efficient detection of unvoiced segments in Mandarin continuous speech, which include stops and fricatives, is a worthwhile study, and speech recognition system would benefit greatly from this work. Table 1. Three categories of Mandarin consonants, and their occur frequencies in 863- Train male database, in which the occurrence of all consonants is 564,812. Affricates are included in fricatives. Type Phones Freq.% Stops b d g p t k 25.55 Fricatives f s sh x h z zh j c ch q 47.43 Sonorant cons. y w l r m n 27.02 3 Detection of Unvoiced Landmark The fricatives have obvious high and middle frequency energy because of articulators friction, but no low energy as the vocal folds isn t freely vibrating; the stops have a short silence before burst because of articulators closure. Thus a piece of speech without low frequency energy is a decisive acoustic cue for these sounds, which makes them the most easy segments to be detected in continuous speech and be used as landmarks in subsequence recognition system. We adopt S. Liu s basic framework of general processing for landmark detection in [3] to find the candidates of unvoiced landmarks. Then the final unvoiced landmarks are picked out at a selecting stage according to acoustic rules, where energy and duration criteria is applied. 3.1 General Processing In general processing, a spectrogram is computed with 6-ms Hanning window every 1 ms, and two frequency bands are defined as shown in Table 2. Then smoothing energy and energy ROR(Rate Of Rise) data for each band are calculated in a coarse-processing(cf) and a fine-processing(fp) pass. In next step, the peaks for each CP and FP ROR waveform with local maximum or minimum value are found. The peaks have same polarity with its ROR value. In the end, the final peaks are picked out from FP peaks using CP peaks as guides. A new peak-picking algorithm is proposed in our approach. After ROR data Ror i [t] for frame t in band i is calculated, where i = 1, 2, and t = 0, 1,...T, the
frames whose absolute ROR value is greater than 9dB and satisfy the requirement below are picked out first: [Ror i (t 1) Ror i (t)][ror i (t + 1) Ror i (t)] > 0.5. (1) Then we look at two segments [t 8, t] and [t, t + 8] for frame t; if both pieces have more than half of points except t whose absolute ROR values are all greater or smaller than t s, we keep the frame t on. Last, just one peak with maximum absolute ROR value is hold and others are deleted in [t 10, t]. This algorithm is executed in both coarse and fine processing. Fig.1 shows the spectrogram, fine-processing energy and ROR waveform for band 1 of a Mandarin Chinese speech sentence in three panels. Locations of final negative peaks in band 1 are shown in the form of dotted vertical lines, and positive peaks in real lines. Table 2. Two energy bands and their frequency range. Band Frequency(kHz) 1 0.0 0.8 2 0.8 1.5 Fig. 1. Demonstration of peaks. The top panel is the spectrogram, then FP energy and ROR waveform in band 1 for a Mandarin Chinese sentence. Dotted vertical lines are the negative peaks in band 1, and real lines are positive peaks in band 1 3.2 Unvoiced Landmark Localization The output of general processing, peaks, represent time points of abrupt energy changes in two bands. To locate unvoiced landmarks, the peaks in band 1 are
candidates because the energy in band 1 monitors the presence or absence of glottal vibration. The ROR data in band 2 is used as reference when selecting the one with maximum ROR from two peaks of band 1. Using 863-Train database as development data, several energy and duration rules, designed from observation and acoustic knowledge, are used to select reliable unvoiced landmarks from peaks in band 1 and remove others. First of all, for two adjacent peaks within 20ms with same sign, the one with the maximum absolute ROR value is saved. After this processing, those peaks of small energy fluctuate caused by a bigger one are removed. Secondly, a sentence must begin with a +peak which labels the point the vocal folds start vibrating, and end with a -peak which locates stop of vibration. The first valid +peak is found that the total energy of the segment between +peak and next -peak is larger than 40dB, which make sure this region is a speech segment. At the end of speech, if the last peak is +peak, we delete it until a valid -peak is the last one. The +peaks and -peaks must be paired in speech, corresponding to the begin and stop of vocal folds freely vibration. For each +peak, the -peak before it and itself locate the unvoiced speech segment; the -peak behind it labels the end of voiced segment started at this +peak and a new unvoiced segment s start. In our system, for adjacent two +peaks, if they are within 200ms which is the maximum duration of syllable, the latter one is deleted because it is a energy jump for a new syllable s sonorant consonant; otherwise, the one with minimum absolute ROR value is deleted. For more than one -peaks behind a +peak, the duration between + and -peak must be longer than 80ms which is the minimum voiced segment duration. And for connected -peaks, only the one with maximum ROR value is kept. Energy rule, that the average total energy in +/-peak duration must be larger than 40dB, is used in here to remove creaky voiced segments. The +peak and -peak before it are removed if energy rule is not satisfied. Another energy criteria is imposed to unvoiced segments that the average band 1 energy must less than 45dB, making sure that the vocal folds do not freely vibrate. After these selecting, about 20% of the peaks are deleted, and the left peaks are the unvoiced landmarks we look for. Shown in Fig. 2, the unvoiced speech segments located by the -unvoiced and +unvoiced landmarks are colored in black in the horizon bar at the bottom of spectrogram. All the unvoiced segments for stops and fricatives in this sentence are found using our approach. 4 Landmark-guided SM Decoding 4.1 SM-based System Segment models adopt segmental distribution rather than frame-based features in HMMs to represent the underlying trajectory of the observation sequence, so the SM-based system can resolve some limitations of HMM system, and achieve more delicate accuracy but with high complexity and computation. To speed up
Fig. 2. Demonstration of unvoiced landmarks. Dotted vertical lines are the -unvoiced landmarks, and real lines are +unvoiced landmarks. The unvoiced speech segments located by the -unvoiced and +unvoiced landmarks are colored in black in the horizon bar at the bottom of spectrogram. SMs is a crucial work for SM s applications to large vocabulary continue speech recognition[15]. 4.2 SM Decoding The decoding of SM is completed by two stages. For a segment of speech x m τ, starting at τ, ending at m, the segment model α with highest likelihood score is found first: D m (τ)=max α {ln[p(xm τ α)](m τ) + ln[p (α)] + ln[p s (x m τ α)]}, 0 τ < m < T. (2) where D m (τ) is the highest likelihood score, P (α) is the language score, and P s (x m τ α) is the segmental score(duration, etc.). Then, for each end points from 0 to T, the most suitable start point t is selected to construct a segment with highest probability: where, J (m) = max{j (τ) + D m (τ) + C}, J (0) = 0. (3) τ max{m L ext, 0} τ < m. (4) J (m) is the accumulated score of the best acoustic model sequence at point m, and C is the insertion factor for each segment; L ext is the allowed maximum segment duration.
In our landmark-guided SM decoding system, the unvoiced landmarks at l are used to determine the start points for segments search, in the case that: max{m L ext, 0} l d < m. (5) The offset value d, 7ms for +unvoiced landmark and 0ms for -unvoiced, is used to compensate the distance between landmarks and the SM boundaries. The points at (l 4)ms are the new search start points, in which 4ms is added before the landmark location for a robust consideration. 5 Experiments and Results The database provided by Chinese National Hi-Tech Project 863 for Chinese LVCSR system development is used in our experiments. There are 83 male speakers 48373 sentences(55.6 hours) and 6 male speakers 240 sentences(17.1 minutes) in the training and testing sets respectively, all sampled at 16kHz. Both training and testing data are high quality standard Mandarin with the accent well controlled. The acoustic features used in our experiments are 12 dimensions MFCC plus 1 dimension normalized energy and their 1st and 2nd order derivatives, using 25.6ms length and 10ms step Hamming window. The training set are used to design the landmark detection algorithm, and to train the segment models used in recognition system. Both the results of landmark detection and SM recognition are given on the testing set. 5.1 Landmark Detection Results The unvoiced landmarks should exist for all stops and fricatives in Mandarin continuous speech. We also detected unvoiced landmarks for sonorant consonants in our experiments, because there may be short pauses before sonorant consonants in speech which are actually unvoice pieces. The consonant h is a velar fricative with acoustic character more similar to sonorant consonants, so we group it to sonorant consonants. To evaluate the detection algorithm, deletion rate and insertion error rate of unvoiced landmarks for stops, fricatives and other sonorant consonants are given in Table 3. Insertion error occurs mostly because of creaky voice, and gives more serious impact on recognition than landmark deletion error. So a hard threshold is taken in our experiment to reduce insertion error rate. As a result, among all 3145 consonants tokens, there are only 4 insertions in our unvoiced landmarks. Shown in Table 3, landmarks for fricatives have the best detection rate, owning to their regular and steady acoustic character that there is obvious energy in high frequency range and very low energy in band 1. Not all of the stops landmarks are easy to detect, because very short duration of articulator s closure when pronounced in some cases would not cause the disconnecting of energy in band 1. We compare the landmarks locations with the boundaries of corresponding consonants given by the SM recognition system, shown as the Dist1 for -unvoiced
landmarks and Dist2 for +unvoiced landmarks in Table 3. All the distance are negative, indicating that the landmarks are behind the boundaries in average. The distance of -landmarks and start point of SM models is smaller than that of +landmarks and end point of SM. That is not surprising because the +landmark is the beginning of the freely vibrating for vocal folds, not the end of consonants especially for stops. Table 3. Results of unvoiced landmark detection for stops, fricatives and sonorant consonants. Cons. Del% Ins% Dist1(ms) Dist2(ms) Stops 12.53 0.26-23.62-56.34 Fricatives 1.06 0.15-13.07-49.80 Sonorant cons. 67.73 0-22.63-50.27 Total 26.42 0.13 5.2 SM System Experiments The baseline SM system in our experiment is a context-dependent triphone SSM recognition system, in which there are 15 regions modeled by 12 Gaussian mixtures for each segment model. Region models are tied through phone based decision trees. Endpoint detection is processed for all speech, and no language model in system. The recognition results and decoding time of our landmark-guided SM system and of the baseline system for 863-Test are listed in Table 4. We add the +unvoiced landmarks into SM system first, as the system 1; then both + and -unvoiced in system 2. The decoding time of recognition system is reduced in system 1 and more in system 2, as well as the average search length for acoustic model decoding. The performance is slightly better than baseline in system 1, but slightly worse in system 2. Such results can be forecasted from Table 3, in the fact that the -unvoiced landmarks give a precise location of the beginning of stops, fricatives and some sonorant consonants, but +unvoiced landmarks do not give the ending location of consonants for all of them. Another important reason for worse performance in system 2 is that the landmark information hasn t been included in segment models training, which will be our future work. And it is expected that segment models adapted to unvoiced landmarks would be able to bring better recognition performance. 6 Conclusions This paper presents an unvoiced landmark detection algorithm for Mandarin continuous speech, and demonstrates our preliminary study of introducing landmarks to an SM-based LVCSR system. Our algorithm of unvoiced landmark
Table 4. Recognition results of landmark-guided SM system. System 1 utilizes -unvoiced landmarks; system 2 utilizes all + and - unvoiced landmarks. Average search length, word error rate and run-time for three systems are listed. System Landmark Search length WER% Time(min.) baseline 54.6 12.21 30.1 system1 -u 40.69 12.11 29.4 system2 +/-u 26.32 12.69 28.4 detection provides an effective methods to segment continuous speech to voiced and unvoiced subsegments, which utilizes only convenient energy and duration rules. The unvoice landmarks provide important information about segmentation for speech recognition. Although only location information of unvoice landmarks were utilized, our experiment to incorporate unvoiced landmarks into SM system illustrated the feasibility to use landmarks as a guide in acoustic model decoding. For future work, other landmarks will be introduced into our SM system, such as sonorant landmarks, fricative landmarks and stop landmarks separated from unvoiced landmarks. And subsequent processing about acoustic cues around landmarks would give speech decoding additional information than only given by landmarks location. On the other hand, training segment models adapted to the defined landmarks, which means that they have consistent beginning and ending time points with landmarks, is expected to provide better recognition performance. Acknowledgements. This research is supported in part by the China National Nature Science Foundation (No. 60172055, No. 60121302), the Beijing Nature Science Foundation (No.4042025) and the National Fundamental Research Program (973) of China (2004CB318105). References 1. Lee, C. H., Juang, B. H.: A new detection paradigm for collaborative automatic speech recognition and understanding. Jan. 12-14, SWIM (2004) 2. Stevens, K. N., Manuel, S. Y., Hufnagel, S. Shattuck, Liu, S.: Implementation of a model for lexical access based on features. Proc. ICSLP, Vol. 1, Banff, Alberta (1992) 499-502 3. Liu, S. A.: Landmark detection for distinctive feature-based speech recognition. J. Acoust. Soc. Am., 100(5) (1996) 3417-3430 4. Schutte, K., Glass, J.: Robust detection of sonorant landmarks. Proc. Interspeech (2005) 1005-1008 5. Stevens, K. N.: Toward a model for lexical access based on acoustic landmarks and distinctive features. J. Acoust. Soc. Am., 111(4) (2002) 1872-1891 6. Johnson, M. Hasegawa et. al: Landmark-based speech recognition: Report of the 2004 Johns Hopkins summer workshop. Proc. ICASSP, Vol. 1 (2005) 213-216 7. Tang, M., Seneff, S., Zue, V.: Two-stage continuous speech recognition using feature-based models: a preliminary study. Proc. ASRU (2003) 49-54
8. Ostendorf, M., Digalakis, V., Kimball, O.: From HMM s to segment models: a unified view of stochastic modeling for speech recognition. IEEE Trans. Speech and Audio Proc. (1996) 360-378 9. Ostendorf, M., Roukos, S.: A stochastic segment model for phoneme based continuous speech recognition. IEEE Trans. Speech and Audio Proc. (1989) 1857-1869 10. Bitar, N. N., Wilson, C. Y. Espy: Knowledge-based parameters for HMM speech recognition. Proc. ICASSP (1996) 29-32 11. Launay, B., Siohan, O., Surendran, A., Lee, C. H.: Towards knowledge-based features for HMM based large vocabulary automatic speech recognition. Proc. ICASSP (2002) 817-8202 12. Kirchhoff, K., Fink, G. A., Sagerer, G.: Combining acoustic and articulatory feature information for robust speech recognition. Speech Communication, 37 (2002) 303-319 13. Li, J. Y., Tsao, Y., Lee, C. H.: A study on knowledge source integration for candidate rescoring in automatic speech recognition. Proc. ICASSP (2005) 837-840 14. Tang, Y., Liu, W. J., Zhang, H., Xu, B., Ding, G. H.: One-pass coarse-to-fine segmental speech decoding algorithm. To be appeared Proc. ICASSP (2006) 15. Tang, Y., Zhang, H., Liu, W. J., Xu, B.: Coloring the speech utterance to accelerate the SM based LVCSR decoding. NLP-KE 05 (2005)