Review Article A REVIEW ON LANDMARK DETECTION METHODOLOGIES OF STOP CONSONANTS

Transcription

1 , pp Available online at Review Article A REVIEW ON LANDMARK DETECTION METHODOLOGIES OF STOP CONSONANTS NIRMALA S. R.* AND GOSWAMI UPASHANA Department of Electronics and Communication Engineering, Gauhati University Institute of Science and Technology, Guwahati, , India. *Corresponding Author: -nirmalasr3@gmail.com,upashanagoswami15@gmail.com Received: May 28, 2017; Revised: October 10, 2017; Accepted: December 21; Published: December 30, 2017 Abstract- Human can produce different sounds such vowels, semi-vowels, nasals, fricatives, stops, murmurs etc. Some sounds are produced by vocal fold vibration and others by making constriction in the vocal cord. Stop consonants fall under the second category. They are associated with low energy, high var iability and highly random in nature. The extraction of useful features from speech signal is a very challenging task. They can be extracted from certain locations, where there are sudden and significant articulatory changes known as landmarks. Landmarks associated with stops are Voicing Onset Time and burst rel ease. Analysis of Burst and Voicing Onset Time can give the place of articulation during the production of sounds. Therefore, detection of these landmarks is studied by a number of researchers. Landmark based processing is required for analysis of events associated with stops. In this paper, we reviewed some of the ex isting methodologies for landmark detection of stop consonants. Keywords- Landmark, Burst, Voicing Onset Time, Stop, Detection. Citation: Nirmala S. R. and Goswami Upashana, (2017) A Review on Landmark Detection Methodologies of Stop Consonants., ISSN: & E-ISSN: , Volume 8, Issue 1, pp Copyright: Copyright 2017 Nirmala S. R. and Goswami Upashana, This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution and reproduction in any medium, provided the original author and source are credited. Academic Editor / Reviewer: Introduction Speech is produced by the convolution of the excitation signal from the vocal folds (source) and the impulse response of the vocal tract filter [1]. One of the most important classes of sound is stop consonants. They are produced by making constriction at some point in the vocal tract and then releasing the air. In English phonemes /b/, /p/, /t/, /d/, /k/, /g/ are called stops or plosives [2]. In the past few decades, stops have been studied by a number of researchers. They particularly studied the salient regions known as `landmarks'. Acoustic landmarks or events contain important cues for speech perception. There are mainly two important landmarks associated with any stop. They are burst onset and voicing onset time (VOT) by which the nature of stops is completely characterized Detection of these events plays a major role in many applications such as automatic speech recognition, phoneme recognition etc. [3, 4, 5, 12]. Detection of these landmarks in various other languages is also a wide area of research [22, 23]. Stops are classified based on presence of vocal fold vibration, manner of articulation and place of articulation. In production of stop consonant, the changing articulators produce multiple events or landmarks. In this paper, we have discussed the landmark detection methodologies of stop consonants. Classification of Stops Phonemes are the smallest unit, which distinguish one word from other in a particular language. One of the main sub-classes of phoneme is stop consonants. They are mainly classified into two types, voiced and unvoiced depending on vocal fold state. A. Voiced Stop If the vocal fold vibration is present during the production of stops then they are called voiced stops, for example phonemes /b/, /d/, /g/ are voiced. B. Unvoiced Stop In case of unvoiced stops this vibration is absent and phonemes /p/, /t/, /k/ fall under this category. Manner of Articulation It describes how the airstream is affected when it flows from lung and out of the mouth. From this view point, stop production leads to multiple articulatory events. They are dependent on type of stop consonants, speaker variability and the context in which it appears. The events are mainly divided into five components such as-closure-interval, transient, frication, aspiration and transition [2,3] and described below. The [Fig-1] shows the five components of stops. Fig-1 Events occurred during a stop production [6]. Closure-interval: In stop production, first a closure is formed within the oral cavity of the vocal tract. In case of unvoiced stops, silence region is present during closure interval. A low frequency dominant periodic energy is present in this phase of voiced stops which can be seen as 'voice bar' in the spectrogram. Transient: The release of closure is termed as transient. It is characterized by a brief pulse of intense energy in the spectrogram. Bioinfo Publications 316

2 A Review on Landmark Detection Methodologies of Stop Consonants Frication: The component resulting from the combination of high intra-oral pressure being released through a narrow opening at the point of release. Aspiration: The component resulting of the vocal tract opening creating turbulence through the glottis rather than the oral constriction. Formants can often be present during this phase. Transition: The component where formants are present and the oral tract is moving to the position for the following vowel target. The main landmarks or events of stops are burst and VOT [shown in [Fig-2] and [Fig-3]. The interval between the transient or burst onset and the onset of voicing is known as VOT [2]. It is an important temporal cue to distinguish between voiced and voiceless stop especially when stops are in word initial position [16]. This parameter can range from 0-30ms for voiced and ms for unvoiced stops [6]. On average, the VOT range are as follows [14]: VOT (/b d g/) < VOT (/p t k/). VOT (/b/) <VOT (/d/) < VOT (/g/). VOT (/p/) < VOT (/t/) <VOT (/k/). Place of Articulation In the production of stops, the point of contact at which the constriction occurs in the vocal tract is known as place of articulation. The distinctive sound of consonants is given by the place of articulation along with manner of articulation. According to the place of articulation, stops are classified into following types [7]: Bilabial Bilabial stops are produced by making maximum constriction using both lips and the phonemes /b/ (voiced) and /p/ (unvoiced) fall under this category. The point of constriction [in Fig-4] of bilabial stops is shown by red line. Fig-2 Illustration of the events/landmarks related to the voiceless or unvoiced stop consonant /k/ producing sound ka. Fig-4 Point of constriction (red line) of bilabial stop [7]. Alveolar: Alveolar stops are produced when the tongue tip articulates with the frontal part of the alveolar ridge such as /d/ (voiced) and /t/ (unvoiced). Alveolar stop place of constriction [in Fig-5] is shown by red line. Fig-2 Illustration of the events/landmarks related to the voiced stop consonant /b/ producing sound ba. Burst onset Burst onset or burst or closure-burst-transition (CBT) is a brief pulse of acoustic energy produced by the initial releases of constriction in stop production. Automatic detection of burst onset is one of the major problem considered in several studies [4,5,8]. The typical value of this may range for 5-10ms. There are high degree of variability in burst such as- the burst release may be weak (voiced stops) and sharp (unvoiced stops); multiple burst may be present in a single stop (Velar stop). Voicing onset time (VOT) Fig-5 Point of constriction (red line) of alveolar stop [7]. Velar Making constriction at velum produces velar stops. The examples are /g/ (voiced) and /k/ (unvoiced). The red line [in Fig-6] shows the constriction of velar stops. Bioinfo Publications 317

3 Nirmala S. R. and Goswami Upashana Fig-6 Point of constriction (red line) of velar stop [7]. Burst Detection Methodologies This section provides a systematic study on various existing methods for burst onset detection. Liu et al. [3], proposed a method for landmark detection, which involves 6-energy band, rate of rise (ROR) measure and threshold logic. In their proposed method, they computed the broad spectrogram of the input speech signal using 512-point DFT, 6 ms hamming window. They divided the resulting spectrogram into six frequency bands. These bands were used to monitor different events. The energy changes in the six bands were computed by using two pass strategies. The first pass used coarse information to find the general vicinity of a spectral change and in the second pass some parameter values were modified in order to localize the energy changes in time. The ROR was measured by computing overlapping db first difference of energy in each of the six bands. The abrupt spectral change in six bands was found by the positive and negative peaks in ROR waveform. Burst onset landmarks were selected from these peaks using a threshold method. Detection rates were 41%, 68%, 85%, and 88%, for temporal accuracies of 5, 10, 20, and 30 ms respectively for sentences from TIMIT database. Salomon et al. [4] proposed a method for landmark detection based on temporal parameters. In their experiment, they took band pass components of the speech signal as the temporal parameter. Initially, the signal is passed through an auditory filter-bank, which divides the signal into 60 band-pass frequency channels. Then they computed envelope analysis and feature extraction in each channel. They include energy onset and offset measures, periodicity and aperiodicity content. By combining all features in the individual channels, summary feature was produced to detect the abrupt change in energy. Detection rate was 96% with 50ms temporal accuracy for sentences from TIMIT database. A gross comparison of the method with that of [3] was also reported in [4]. Based on the experiment performed on TIMIT database, Liu s method [3] had an error rate of 15% for deletions, 6% for substitutions, and 25% for insertions. On the other hand Salomon method [4], had an overall detection rate of 80.2% with 15% error for deletion, 4.8% for substitutions and 8.7% for insertions. The study shows that summary measures across all frequency channels may be better for stop landmark detection compared to the selection of broader frequency band used by Liu. A stop burst release exists only for 5-10 ms and temporal accuracy less than this affect the systems which require this landmark detection. Temporal accuracy can be improved by using parameter characterization approach such as parameters from Gaussian Mixture Model (GMM). Pandey et al. [5], [8] used rate of change (ROC) measure defined on Gaussian Mixture Model (GMM) of log magnitude using 4 Gaussian components, along with onset-offset detector and spectral flatness measures to detect stop landmarks. Approximate error in modeling log magnitude spectra is less than other modeling such as magnitude spectra or squared magnitude spectra. Another advantage is that, gain normalization is not necessary in this type of spectrum. They computed the log magnitude spectra of the speech signal by taking 512-point DFT and 6-ms hanning window. The high frame rate is useful for capturing fast spectral variations. A median filter with 50 points was used to smooth the magnitude spectrum. The resulting spectrum was approximated by a weighted sum of the Gaussian functions. The expectation maximization algorithm was used to estimate GMM parameters. ROC was calculated on the smoothed parameters to capture the abrupt change. Detection rates are 98%, 97%, 95%, 90% and 73% at temporal accuracies of 30, 20, 15, 10, and 5 ms respectively for sentences from TIMIT database. The iterative process of estimation of the Gaussian parameters is computation intensive and the method is not suited for a real-time implementation. Lin et al. [9] used a method based on spectral moment in addition to energy band parameters for burst onset detection. They used spectral moments as parameters for classification of Mandarin stops. Considering a fixed weight for a parameter affects the detection rate because it desensitizes the parameter variation. Jayan et. al. [10] proposed a method by combining peak energy from fixed frequency band and first four spectral moments from the speech spectrum for burst onset detection. In order to calculate the energy band parameters, the speech was sampled at 10kHz and spectrogram was computed by taking 512-point DFT and 6ms hanning window. A 20-point moving average was taken along the time index to smooth the resulting spectrum for each frame. The peaks in three different frequency bands ( , , and khz) were taken from the smoothed spectrum. Again to compute the spectral moments, normalized speech spectrum was considered as probability density function. The first four spectral moments; centroid, variance, skewness and kurtosis was computed from the spectrum. A combined rate of change measure based on Mahalanobis distance, referred to as ROC-MD. Their results showed that energy parameters were highly reliable and contribute more towards detection rate. Spectral moments were useful as additional parameters for improving detection rates of burst onset landmark, but for reliable and accurate detection of landmarks combined parameters were required. Rate of change obtained by Mahalanobis distance based first difference (ROC-MD) operation was more effective in combining parameters and deriving a single parameter indicative of the overall variation. It was less sensitive to the variations in time steps and it is effective for time-localizing the burst onsets. The detection rates of the combined system were 90% and 96% for temporal deviation of 5 ms and 10 ms respectively (time step 3ms). Niyogi et. al. [11], proposed a method for stop detection using three energy measures namely total log energy, log energy above 3 khz and spectral flatness measure based on wiener entropy as feature vector. Then these are used as input to support vector machine (SVM) to detect stop consonant. Lin et. al. [12], used two dimensional cepstrum (TDC) as feature and a random forest (RF) detector to detect burst onset. RF creates lots of decision trees and used them to make the classification. Each tree in a forest judges the input test sample to make a local decision. The plurality votes determine the final decision on that input test sample. The stop sounds from TIMIT database were used to train the classifier. In order to increase the detection accuracy the stops were classified as voiceless stop burst, voiced stop burst and stop aspiration. They implemented the asymmetric boot-strapping to avoid imbalance in the training set. TDC was derived by the two dimensional discrete cosine transform (2D-DCT). They compared their technique with other machine learning techniques such as SVM and GMM and showed that detection was better in their method. The temporal accuracies were 64%, 86%, 99% with tolerances 5 ms, 20 ms and 30 ms respectively. Prathosh et al. [13], proposed two new temporal features plosion index (PI), maximum normalized cross-correlation (MNCC) and a rule based classifier to detect closure burst transition (CBT). In this algorithm first, Hilbert Envelop and zero crossing rate was computed on high pass filtered speech signal. Then PI was computed at local maxima of Hilbert envelop between successive zero crossing and if it exceeds a particular threshold it was considered as potential CBT. They assumed that within 20ms interval two burst releases couldn t occur and the very first potential CBT within 20 ms was considered representative burst candidate (RBC). MNCC was computed over three successive epochs and based on threshold criteria decision was taken whether RBC is CBT or not. The detection rates for CBT using TIMIT database sentences were 64%, 84%, 97%, and 100% Bioinfo Publications 318

4 A Review on Landmark Detection Methodologies of Stop Consonants for deviation of 5, 10, 15, and 20 ms respectively. VOT detection methodologies Various existing methods for estimating VOT fall under two categories: Identification of the locations of the burst and voicing onsets through a set of customized acoustic-phonetic rules (knowledge-based) and Those which train a learning machine (such as random forest, support vector machine) to estimate the VOT using some acoustic features corresponding to the stop-to-voiced-phone transition event. Stouten et al. [14], used a reassigned time-frequency representation (RTFR) to automatically estimate the VOT of stops. RTFR is a high resolution signal analysis method with better time information preserving capacity than Mel frequency cepstral coefficients (MFCC). This method involves three steps. In the first step, HMM based speech recognizer was used to select plosive segment. It searched burst onset 2.5ms or 4 frames prior to the burst segment start found by the recognizer and extended the end segment to 10ms or 16 frames, in order to minimize the error. Second step was burst onset detection, for which they retained only the frequency range 3.2 to 8 KHz. The burst power p(n) for frame n was estimated by summing all frequency bins in RTFR power and the first local maxima of p(n) with sufficient strong and sharp change was identified as burst onset. A missing burst or weak burst could lead to absence in local maxima. In those cases, the start of the segment was identified as burst onset. The third step was to find the starting of periodicity or voicing. For voicing detection, only frequency components 0-4 KHz were retained. They computed short time autocorrelation of the RTFR frame by multiplying with an asymmetric weighted version of the frame to find short at lag 0 to 40. Then they summed these values over the lag index and over the retained frequency band. They validated their algorithm with manually extracted VOT estimates. The method showed detection rate of 76.10% for temporal accuracy of 10 ms and 91.40% with temporal accuracy of 20 ms for a subset of TIMIT database sentences. Keshet et al. [15] proposed a supervised learning algorithm for VOT detection. First the algorithm was trained on a set of manually marked stop consonants. The trained segments were of arbitrary length and marks indicated the burst onset and vowel onset. They extracted 7 acoustics features order to achieve high accuracy. These features were highly informative about the accurate location of the onset pair. First 4 features were taken from the short time Fourier transform (STFT): log of the total spectral energy, log of the energy between 50 Hz and 1000 Hz, log of the energy above 3000 Hz and wiener entropy (a measure of spectral flatness). The fifth feature was maximum power spectrum computed around 6ms before and 18ms after the frame centre. RAPT-base pitch tracker for voicing detection was accounted as sixth feature. The number of zero crossings were computed around the frame center was considered as seventh feature. At the testing phase, each segment of the input signal was mapped to the same vector space and most probable onset pair and hence VOT was detected. A kernel machine was used to map the input signal and target onset pair to a vector space, which included all possible onset pairs. They used four databases for their experiment namely TIMIT database, Big Brother database which contain spontaneous speech samples, switchboard database which contain spontaneous telephone conversation and Paterson / Goldrick database which contain data from laboratory study. Performance of the algorithm was evaluated based following criteria - difference between the automatic and manual VOT estimation. They detected VOT with 99% accuracy with 50ms temporal accuracy. Lin et al. [16], proposed a random forest (RF) classifier for onset detection. They used HMM based force alignment to align a speech signal with its accompanying text transcription. Two-dimensional cepstral coefficients (TDCC) were used to capture voicing onsets. They applied Fourier Transform of the on the log magnitude spectra computed over successive group frames. Using these features RF classifier was applied on segments obtained using HMM based force alignment to detect the voicing and burst onsets. Detection rates of VOT were 57.20%, 83.40%, 93.40% and 96.50% with temporal accuracies of 5 ms 10 ms 15 ms and 20 ms respectively with TIMT database. Prathosh et al. [17], used temporal measure to estimate VOT estimation in continuous speech. Burst onset was detected as described in paper [13] discussed above. For voicing detection Maximum weighted inner-product (MWIP) and zero crossing difference (ZCD) were used. MWIP measures the similarity between two vectors and hence for periodic segment two successive epochs will have high degree of similarity. They selected the closest epoch to the burst onset and computed MWIP between two successive epochs starting from the closest epoch to the burst onset and selected a threshold to check whether computed MWIP was greater than t1 or not. If it was greater they checked whether ZCD over both of the two successive inter-epoch intervals starting from closest epoch to the burst onset is less than another threshold. If this condition was also satisfied they termed the 1st epoch as voicing onset. Conclusion The various existing methodologies for landmark detection rely on spectral and temporal features. Generally, these methods for burst and VOT detection are validated against a manually marked database. The automatic detection of events of stops is used in many applications. The analysis of extracted features around the useful landmarks or events can represent the speech information for various applications like speech recognition, phoneme recognition [13, 24]. Accurate detection of landmarks can be used to determine the appropriate place of articulation of stops [21]. Again, VOT is significant in discriminating voiced and unvoiced stops. The parameters around these two landmarks are also useful in diagnosis of pathological condition like dysarthria which causes disruption in the speech production system [18]. Due to the various difficulties associated with sound production of dysarthric speakers the landmark detection play an important role in acoustic analysis of dysarthric speech. Application of research: It gives an idea of various existing methodologies for landmark detection of stop consonants. These landmarks can be used for various application like speech recognition, phoneme recognition etc. Research Category: Speech Processing *Abbreviations: VOT- Voicing onset time GMM- Gaussian Mixture Model ROR- Rate of rise ROC- Rate of change DFT- Discrete Fourier Transform TDC- Two dimensional cepstrum RF- Random forest DCT- Discrete cosine transform SVM- Support vector machine MNCC- Maximum normalized cross correlation CBT- Closure burst transition RBC- Representative burst candidate PI- Plosion Index RTFR- Reassigned time-frequency representation HMM- Hidden Markov model MFCC- Mel frequency cepstral coefficients STFT- Short time Fourier transform ZCD- Zero crossing difference MWIP- Maximum weighted inner-product Author Contributions: All author equally contributed Author statement: All authors read, reviewed, agree and approved the final manuscript Conflict of Interest: None declared Acknowledgement / Funding: Author thankful to Gauhati University Institute of Science and Technology, Guwahati, , India Bioinfo Publications 319

5 References [1] Rabiner L. and Juang B., Fundamentals of Speech Recognition, (1993) Prentice-Hall, Englewood Cliffs [2] Fry D. B., Acoustic Phonetics (2009) Cambridge Universtiy Press. [3] Liu S. A. (1996) Journal of Acoustical Society of America, 100(5), pp [4] Salomon A, Wilson C.V. and Deshmukh O. (2004) Journal of Acoustical Society of America, 115(3), [5] Jayan A.R. and Pandey P. C. (2009) IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), [6] Prathosh A. P. (2015) PhD Thesis, Indian Institute of Science, Bangalore. [7] ``Places of Articulation - The Complete List (with Examples)''.[ONLINE]. Available at: (Accessed on 3 rd May, 2017.) [8] Jayan A. R. and Pandey P. C. (2008) Proceedings of Int. Symposium on Frontiers of Research on Speech and Music, [9] Lin C.Y. and Wang H.C. (2008) 6th International Symposium on Chinese Spoken Language Processing, 1-4. [10] Jayan A.R., RajathBhat P.S. and Pandey P.C. (2011) 17 th National Conference on Communications, 1-5. [11] Niyogi P., Burges C. and Ramesh P. (1999) IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1, [12] Lin C. and Wang H. (2010) IEEE International Conference on Acoustics, Speech and Signal Processing, [13] Prathosh A.P., Ramakrishnan A.G. and Ananthapadmanabha T.V. (2014) Journal of Acoustical Society of America, 135 (1), [14] Stouten V., Hamme H. V. (2011) Speech Communication, 51 (12), [15] Sonderegger M. and Keshet J. (2012) Journal of Acoustical Society of America, 132 (6), [16] Lin C.Y. and Wang H.C. (2011) Journal of Acoustical Society of America, 130 (1), [17] Prathosh A.P., Ramakrishnan A.G. and Ananthapadmanabha T.V. (2014) Journal of Acoustical Society of America, 136 (2), [18] Kent R.D., Weismer G., Kent J.F., Vorperian H.K. and Duffy J.R. (1999) Journal of Communication Disorders, 32, [19] Mengistu K. and Rudzicz F. (2011) IEEE International Conference on Acoustics, Speech and Signal Processing, [20] Kim H., Martin K., Hasegawa-johnson M., and Perlman A. (2015) Clinical Linguistics and Phonetics, 24, [21] Prathosh A. P., Ramakrishnan A. G., and Ananthapadmanabha T.V. (2015) Interspeech, [22] Potisuk S. (2016) International Journal of Signal Processing Systems, 4 (3), [23] Singh K. and Tiwari N. (2016) Journal of Acoustical Society of America, 140 (5), [24] Arjun P. and Jayan A. R. (2017) International Conference on Inventive Systems and Control, 1-6. Nirmala S. R. and Goswami Upashana Bioinfo Publications 320