MULTI-STREAM FRONT-END PROCESSING FOR ROBUST DISTRIBUTED SPEECH RECOGNITION Kaoukeb Kifaya 1, Atta Nourozian 2, Sid-Ahmed Selouani 3, Habib Hamam 1, 4, Hesham Tolba 2 1 Department of Electrical Engineering, Université de Moncton, New Brunswick, Canada 2 INRS - EMT, Université du Québec, Québec, Canada 3 LARIHS Lab., Université de Moncton, Shippagan campus, New Brunswick, Canada 4 Canadian University of Dubai, Dubai, UAE ekk4162@umoncton.ca, noroua@emt.inrs.ca, selouani@umcs.ca, hamamh@umoncton.ca, htol@link.net Abstract This paper investigates a multi-stream-based front-end in Distributed Speech Recognition (DSR). It aims at improving the performance of Hidden Markov Model (HMM)-based systems by combining features based on conventional MFCCs and formant-like features to constitute a new multivariate feature vector. The approach presented in this paper constitutes an alternative to the DSR-XAFE (XAFE: extended Audio Front-End) available in mobile communications. Our results showed that for highly noisy speech, using the paradigm that combines formant-like features with MFCCs, leads to a significant improvement in recognition accuracy on the Aurora 2 task. 1 INTRODUCTION Extraction of reliable parameters remains one of the most important issues in Automatic Speech Recognition (ASR). This parameterization process serves to maintain the relevant part of the information within a speech signal while eliminating the irrelevant part for the ASR process. A wide range of possibilities exists for parametrically representing the speech signal. The cepstrum is one popular choice, but it is not the only one. When the speech spectrum is modeled by an all-pole spectrum, many other parametric representations are possible, such as the set of p-coefficients αi obtained using Linear Predictive Coding (LPC) analysis and the set of line spectral frequencies (LSF). This latter possesses properties similar to those of the formant frequencies and bandwidths, based upon the LPC inverse filter. Another important transformation of the predictor coefficients is the set of partial correlation coefficients or reflection coefficients. In previous papers [1-3], we introduced a multistream paradigm for ASR in which, we merge different sources of information about the speech signal that could be lost when using only the MFCCs to recognize uttered speech. Our experiments in [1] showed that the use of some auditory-based features and formant cues via a multistream paradigm approach leads to an improvement of the recognition performance. This proved that the MFCCs loose some information relevant to the recognition process despite the popularity of such coefficients in all current ASR systems. In our experiments, we used a 3-stream feature vector. The First stream vector consists of the classical MFCCs and their first derivatives, whereas the second stream vector consists of acoustic cues derived from hearing phenomena studies. Finally, the magnitudes of the main resonances of the spectrum of the speech signal were used as the elements of the third stream vector. The above-mentioned work has been extended in [2] by the use of the formant frequencies instead of their magnitudes for ASR within the same multi-stream paradigm. In these experiments, the recognition of speech is performed using a 3-stream feature vector, which uses the formant frequencies of the speech signal obtained through an LPC analysis as the element of the third stream vector combined with the auditory-based acoustic distinctive features and the MFCCs. The obtained results [2] showed that the use of the formant frequencies for ASR in a multi-stream paradigm improves the ASR performance. Then in [3], we extended our work to evaluate the robustness of the above mentioned proposed features using a multi-stream paradigm for ASR in noisy car environments. The obtained results showed that the use of such features renders the recognition process more robust in noisy car environments. In this paper, we investigate the potential of the multi-stream front-end to improve the robustness of a Distributed Speech Recognition (DSR) system. The DSR concept has been initiated by the Aurora proect in the European Telecommunications Standards Institute (ETSI) [4]. In DSR, the process of extracting features from speech signal, also called the front-end process, is implemented on the terminal. The extracted features are transmitted over a data channel to a remote back-end recognizer where the remaining parts of the recognition process takes place. In this way the transmission channel does not affect the recognition system performance. ETSI has developed the DSR ex-tended Advanced Front-end (DSR-XAFE). This front-end is mainly based on MFCC features.
Our approach aims at improving the DSR-XAFE performance by combining MFCCs with formant-like features. In DSR systems the feature extraction process takes place on a mobile set with limited processing power. On the other hand, there is a certain amount of bandwidth available for each user for sending data. Among the features mentioned above, formant-like features are more suitable for this application, because extracting them can be done as part of the process of extracting MFCC which saves a lot of computational process. Besides this, they represent the speech spectrum with relatively few features (usually 4) and hence can be coded with fewer bytes. However, due to some problems related to their inability to provide information about all parts of speech such as silence and weak fricatives, formants have not been widely adopted. In [5], it has been shown that shortcomings of formant representation can be compensated to some extent by combining them with features containing signal level and general spectrum information, such as cepstrum features. The outline of this paper is as follows. In Section 2, we briefly describe the formant-like extraction procedure. Then, in Section 3, we describe the statistical framework of the multi-stream paradigm. Then, in Section 4, we proceed with a description of the database, the parameters of our experiments and the evaluation of our proposed approach for DSR. Finally, in Section 5 we conclude and discuss our results. 2 FORMANT-LIKE FEATURES Formant frequencies are defined as the resonance frequencies of the vocal tract. Formants are considered to be representative of the underlying phonetic knowledge of speech and relatively robust in the particular case of ASR in noisy or band-limited environments. It is well established that the first two or three formant frequencies are sufficient for perceptually identifying vowels. Two main reasons motivated our choice to consider the formant in noisy mobile communications. The first reason is related to the fact that formant regions of the spectrum may stay above the noise level even in very low signal-to-noise ratio (SNR), while the lower energy regions will tend to be masked by the noise energy. The second reason is the formant ability to represent speech with very few parameters. This is particularly important for the systems with limited coding rate such as DSR systems. It is worth noting that many problems are associated with the extraction of formants from speech signals. For example, in the case of fricative or nasalized sounds, formants are not well defined. Several methods have been suggested in the literature provide a solution to the problem of determining formant frequencies. However, accurate determination of formants remains a challenging task. There are basically three mechanisms for tracking formant frequencies in a given sonorant frame: computing the complex roots of a linear predictor polynomial; analysis by synthesis; peak picking of a short-time spectral representation [6]. In the LPC analysis, speech can be estimated in terms of a ratio of z polynomials which is the transfer function of a linear filter. The poles of this transfer function include the poles of the vocal tract as well as those of the voice source. Solving for roots of the denominator of the transfer function gives both the formant frequencies and the poles corresponding to the voice source. Formants can be distinguished recognized by their property of having relatively larger amplitude and narrow bandwidth. While this method is of nature to be precise, it turns out to be expensive since for representing 4-5 formants often the order of the polynomial goes beyond 10. Analysis by synthesis is a term referred to as a method in which the speech spectrum is compared to a series of spectra that are synthesized within the analyzer. In such systems, a measure of error is computed based on the differences between the synthesized signal and the signal to be analyzed. The process of synthesizing signals continues until the smallest value of error is obtained then the properties of the signal which caused the less error is extracted from the synthesizer. In the case of formant tracking this information contains the formant frequencies and bandwidth. The third approach which is more typical than the other two approaches consists of estimating formants by the peaks in the spectral representation from short-time Fourier transform, filter bank outputs, or linear prediction. The accuracy of such peak-picking methods is approximately 60 Hz for the first and the second formants and about 110 Hz for the third formant. In spite of the fact that this approach provides less accurate results comparing to the other two approaches, it is simple. This can be very beneficial for real time recognizers and those with limited processing power. In DSR, using algorithms with minimum amount of computation and with minimum delay is crucial. Hence, the peak picking algorithm is more suitable for this purpose. In our experiments an LPC analyzer with the order of 12 was used to estimate the smoothed spectral peak and then four spectral peaks were selected using a peak picking algorithm which merely compares each sample with the two neighbouring samples. The process of extracting formant-like features based on the LPC analysis is illustrated in Figure 1. Figure 1: Block diagram of a formant extractor based on the LPC analysis.
3 MULTI-STREAM STATISTICAL FRAMEWORK In this approach multiple acoustic feature streams obtained from different sources are concatenated to form a multi-stream feature set which is then used to train multistream HMMs. Consider S information sources that provide time synchronous observation vectors O st, s = 1,, S ; s indicates the information source, and t, the time index. The dimensionality of the observation vectors can vary from one source to another. Each observation vector time sequence provides information about a sequence of hidden states. In a multi-stream system, instead of generating S state sequences from S observation sequences, only one state sequence is generated. This is actually done by introducing a new output distribution function for states. The output distribution of state is defined as: It can be seen that the output distributions of multiple observation vectors are merged to form a single output distribution for state. The exponent specifies the contribution of each stream to the overall distribution by scaling its output distribution. The value of γ is normally assumed to satisfy the constraints: (1) and γ = 1. (2) In HMMs, Gaussian mixture models are used to represent the output distribution of states. The probability of vector O t at each time instance t in state can be determined from the following formula: b S M ( Ot ) = [ C N ( Ost ; µ ; Σ ) where M is the number of mixture components, c is the m th mixture weight of state for the source s. N denotes a multivariate Gaussian with µ as the mean vector and Σ as the covariance matrix: N( o st ; µ, Σ s = 1 b m = 1 ) = S ( Ot ) = [ b ( O st )] 0 γ 1 1 n s=1 (2π ) Σ exp 1 ' ( ost µ ) Σ 1 ( ost µ ) 2 The choice of the exponents plays an important role. The performance of the system is significantly affected by the values of γ. S s = 1 γ ], γ, (3) (4) Recently, exponent training has received attention and the search for an efficient method is still undergoing. Most of the exponent training techniques in the literature have been developed in logarithmic domain [8]. By taking the log of the distribution function the exponents appear as scale factors of the log terms. log b ( O ) γ = γ log b ( O ). The multi-stream HMMs presented in this work have three streams and therefore s is equal to three. Obtaining an estimate for the exponent s parameters is a difficult task. Due to the very nature of the signal, there is no predictable underlying structure we can exploit. We expect γ s to be a function of SNR. To extract this function from data, an experiment based on a cross-validation set, was performed by studying the recognition performance obtained across all SNRs. In our experiment all states are assumed to have three streams. The first two streams are assigned to MFCCs and their first derivatives and the third stream is dedicated to formant-like features. Several crossvalidation experiments were carried out to find the optimum weight for formants relative to MFCCs with respect to eq. (2). It should be noted that in order to avoid complexity, the stream exponents are generalized to all states for all models. It can be seen from Figure 2 that the best results are obtained when one fourth of the weight of MFCC features is given to the formant-like features. 4 EXPERIMENTS & RESULTS 4.1 AURORA Database t In our experiments the AURORA database was used. It is a noisy speech database that was released by the Evaluations and Language resources Distribution Agency (ELDA) for the purpose of performance evaluation of DSR systems under noisy conditions. The source speech for this database is the TIdigits downsampled from 20 khz to 8 khz, and consists of a connected digits task spoken by American English talkers. The AURORA training set which is selected from the training part of the TIDigits, includes 8440 utterances from 55 male and 55 female adults and filtered with the G.712 (GSM standard) characteristic. Three test sets (A, B and C) from 55 male and 55 female adults collected from the testing part of the TIDigits form the AURORA testing set. Each set includes subsets with 1001 utterances. One noise signal is artificially added to every subset at SNRs ranging from 20 db to -5 db in decreasing steps of 5 db. The experiments presented in this paper are based on test set A, and two subsets with two different noises namely Babble and Car. Both noises and speech signals are filtered with the G.712 characteristic. In total, this set consists of 2*7*1001=14014 utterances. t (5)
provide similar information and, depending on the application, using one of them is sufficient. 4.3 Tests & Results Figure 2: Word recognition accuracy as percentage averaged over all noise conditions obtained from all four subsets of test set B of the AURORA database as a function of the stream exponent of the formant-like features. 4.2 Training & Recognition Parameters In the AURORA proect, whole-word HMMs were used to model the digits. Each word model consists of 16 states with 3 Gaussian mixtures per state. Two silence models were also considered. One of the silence models has relatively longer duration, modeling the pauses before and after the utterances with 3 states and 6 Gaussian mixtures per state. The other one is a single state HMM tied to the middle state of the first silence model, representing the short pauses between words. In DSR- XAFE, 14 coefficients including the log-energy coefficient and the 13 cepstral coefficients are extracted from 25 msec frames with 10 msec frame shift intervals. However, the first cepstral coefficient and the log-energy coefficient In our experiment, the baseline system is defined over 39-dimensional observation vectors that consist of 12 cepstral and the log-energy coefficients plus the corresponding delta and acceleration vectors. It is noted MFCC-E-D-A. The front-end presented in the ETSI standard DSR-XAFE was used throughout our experiments to extract 12 cepstral coefficients (without the zeroth coefficient) and the logarithmic frame energy. Training and recognition were carried out by the HMMbased tooklit (HTK) [7]. In some special cases HTK toolkit automatically divides the feature vector into multiple equally-weighted streams in a way similar to the multistream paradigm. The idea of this separation is based on the lack of correlation between features. For example, it is known that the correlation between static coefficients and dynamic coefficients is small; therefore putting them in different streams, as if they were produced by independent sources, results in better statistical models. The feature vector used in the baseline experiment is automatically split into three streams by HTK with the following order. The first stream consists of static coefficients plus an energy coefficient. The second stream includes delta coefficients plus the delta energy and the third stream contains acceleration coefficients plus the acceleration energy. In our proposed method, at first 12 cepstral coefficients and the logarithmic frame energy were extracted, then a 12 pole LPC filter and a simple peak picking algorithm were used to extract the frequencies of formant-like features. Finally, MFCCs and their first derivatives plus the 4 frequencies of the formant-like features were combined to generate a 30-dimensional feature set. This vector is referred to as MFCC-E-D-4F. Table 1. Percentage of word accuracy of multi-stream-based DSR systems trained with clean speech and tested on set A of the AURORA database. Babble noise Car noise Signal to Noise Ratio 20dB 15dB 10dB 5dB 0dB -5dB MFCC-E-D-A (39) 90.15 73.76 49.43 26.81 9.28 1.57 MFCC-E-D-4F (30) 87.45 71.74 52.90 30.14 12.67 6.92 MFCC-E-D-3F (29) 87.24 71.49 52.12 28.84 12.27 5.32 MFCC-E-D-2F (28) 87.18 71.80 52.03 28.96 12.30 4.96 MFCC-E-D-1F (27) 87.42 71.58 51.81 28.51 12.33 6.17 MFCC-E-D-A (39) MFCC-E-D-4F (30) MFCC-E-D-3F (29) MFCC-E-D-2F (28) MFCC-E-D-1F (27) 97.41 90.07 89.47 89.59 89.17 90.04 79.81 78.91 79.12 78.80 67.01 60.69 59.44 59.44 57.38 33.34 36.44 34.63 34.18 32.06 14.46 17.21 17.51 17.33 15.72 9.39 10.05 10.38 10.23 10.05
The multi-stream paradigm, through which the features are assigned to multiple streams with proper weights discussed in section 3, was used to merge the features into HMMs. In order to be consistent with the baseline, the MFCCs and their derivatives were put into the first and second streams respectively and the third stream was reserved for the formant-like features. In order to evaluate the impact of the number of included formantlike features, we carried out additional experiments where the third stream is composed of 1, 2, and 3 formants-like features. The resulted systems are respectively noted MFCC-E-D-1F, MFCC-E-D-2F and MFCC-E-D-3F, and then, their dimensions are respectively 27, 28 and 29. Table 1 indicates that for the babble noise, when the SNR decreases less than 10 db, the use of our proposed front-end with 30-dimensional feature vector generated from both MFCCs and formant-like features (MFCC-E-D-4F) leads to a significant improvement in word recognition accuracy. This improvement can reach 5% relative to the word recognition accuracy obtained for the MFCC-based 39-dimensional feature vector (MFCC-E- D-A). In the case of the car noise when the SNR decreases below than 5 db, the multi-stream front-end performs better with fewer parameters. It should be noted that under high-snr conditions, the 39-dimensional system performs better. These results suggest that it could be interesting to concomitantly use the two front-ends: the multi-streambased front-end under severely degraded noise conditions, and the current DSR-XAFE for relatively noise-free conditions. In this case, the estimation of SNR is required in order to switch from one front-end to another. 5 CONCLUSION We have proposed a new front-end for the ETSI DSR XAFE codec. The results obtained from the experiments we carried out on AURORA task 2 showed that combining cepstral coefficients with the main frequencies of the spectrum of a speech signal using the multi-stream paradigm leads to a recognition improvement in noisy speech. This improvement is noticeable for very low SNRs, compared to the recognition performance of the DSR-XAFE, which uses cepstral features alone. On the other hand, although extracting the new features may add some level of complexity to the front-end process, the use of a 29-dimensional feature vector instead of a 39- dimensional feature vector will save time and computational power for the back-end process. We are currently continuing the effort towards the optimization of stream weights with respect to the noise source, speaker gender and phonetic contents of the speech. 6 REFERENCES [1] H. Tolba, S.-A. Selouani and D. O Shaughnessy. Comparative Experiments to Evaluate the Use of Auditory-based Acoustic Distinctive Features and Formant Cues for Automatic Speech Recognition Using a Multi-Stream Paradigm, ICSLP 02, September 2002. [2] H. Tolba, S.-A. Selouani & D. O Shaughnessy, Auditory-based acoustic distinctive features and spectral cues for automatic speech recognition using a multi-stream paradigm, Proc. of the ICASSP, pp. 837-840, Orlando, USA, 2002. [3] S.-A. Selouani, H. Tolba & D.OShaughnessy, Auditory-based acoustic distinctive features and spectral cues for robust automatic speech recognition in low-snr car environments, Proc. of Human Language Technology Conference of the North American Association for Computational Linguistics, CP volume, 91-94, Edmonton, 2003. [4] ETSI standard document, Speech processing, transmission and quality aspects (stq); distributed speech recognition; front-end feature extraction algorithm; compression algorithm, ETSI ES 201 108, vol. 1.1.3., 2001. [5] P. Garner and W. Holmes, On the robust incorporation of formant features into Hidden Markov Models for automatic speech recognition, Proc. IEEE ICASSP, pp. 1 4, 1998. [6] P. Schmid and E. Barnard, Robust, n-best formant tracking, Proc. of EUROSPEECH, pp. 737 740, 1995. [7] S.J. Young et al., HTK version 3.4: Reference Manual and User Manual, Cambridge University Engineering Department Speech Group, 2006. [8] R. Rose and P. Momayyez, Integration of multiple feature sets for reducing ambiguity in automatic speech recognition, Proc. IEEE-ICASSP, pp. 325-328, 2007.