Toolkits for ASR; Sphinx

Transcription

1 Toolkits for ASR; Sphinx Samudravijaya K samudravijaya@gmail.com 08-MAR-2011 Workshop on Fundamentals of Automatic Speech Recognition CDAC Noida, 08-MAR-2011 Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 1/31

2 A Block Diagram of an ASR System Signal Feature Extraction Training Acoustic Model Testing Matching (acoustic domain) Symbol sequence Language Model Matching (symbolic domain) Sentence Hypothesis Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 2/31

3 Hierachy of Units in an Utterance source: state of art ASR by Steve Young, 2000 Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 3/31

4 Sentence HMM is composed of Phone HMMs Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 4/31

5 Toolkits for Automatic Speech Recognition (1) Training, (2) Testing, (3) Performance Evaluation There are several public domain toolkits that help to build an ASR system: HTK: Hidden Markov Model ToolKit [1]. Public domain, but decoder cannot be distributed (C). Sphinxes [2]: Open source: (C, C++, java) Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 5/31

6 Toolkits for Automatic Speech Recognition (1) Training, (2) Testing, (3) Performance Evaluation There are several public domain toolkits that help to build an ASR system: HTK: Hidden Markov Model ToolKit [1]. Public domain, but decoder cannot be distributed (C). Sphinxes [2]: Open source: (C, C++, java) ISIP Production system [3]. Public domain ( without any restrictions) (C++) Julius Open-Source Large Vocabulary CSR Engine [4]. It uses Acoustic Models in HTK format, and Grammar files in its own format. Open license (no limitations on distribution) (C++). HMM toolbox for Matlab Useful for Isolated Word Recognition [5]. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 5/31

7 What is CMU Sphinx? According to Arthur Chan (the editor of Hieroglyphs[6], the sphinx manual in a book form), there are two definitions of Sphinx: A large vocabulary speech recognizer with high accuracy and speed performance. A collection of tools and resources that enables developers/researchers to build successful speech recognizers Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 6/31

8 Pocketsphinx source: SphinxLunch ppt by Arthur Chan, 2004 Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 7/31

9 A Block Diagram of an ASR System Signal Feature Extraction Training Acoustic Model Testing Matching (acoustic domain) Symbol sequence Language Model Matching (symbolic domain) Sentence Hypothesis Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 8/31

10 Language model training source: state of art ASR by Steve Young, 2000 Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 9/31

11 CMU-Cambridge SLM toolkit Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 10/31

12 Lexicon (Pronunciation Dictionary) source: Ph.D. thesis of Ravi Shankar M., CMU [7] Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 11/31

13 source: Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 12/31

14 source: erwin/sr2003/sphinx.ppt Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 13/31

15 Feature Extraction (Frontend processing) * wave2feat program computes 13 MFCCs from speech files stored in any of wav,nist,raw format. * Caution: use -dither yes option. Excise long silences. * cepview s0001.cep prints the cepstral coefficients. source: Ph.D. thesis of Ravi Shankar M., CMU [7]. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 14/31

16 SphinxTrain Training sub-word HMMs Stages of training (Reference: 1 Training context Independent phone HMMs 2 Training context Dependent phone HMMs 3 Decision tree building 4 Training context Dependent tied phone HMMs 5 Recursive Gaussian splitting Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 15/31

17 Training Context Independent phone HMMs 2 steps: Initialization and Embedding re-estimation. Inputs: * Feature vector sequences * Word-level transcriptions * Pronunciation dictionary Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 16/31

18 Training Context Independent phone HMMs 2 steps: Initialization and Embedding re-estimation. Inputs: * Feature vector sequences * Word-level transcriptions * Pronunciation dictionary (I) Initialization: 1 Make a proto-type HMM (5-state, left-to-right, skipping 1 state permitted); copy to all phone HMMs. 2 Compute means and variance of all training feature vectors 3 Initialise Gaussians of all states of phone HMMs with global means and variance. 4 For each and every utterance, generate phone-level transcriptions from word-level transcriptions using the pronunciation dictionary. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 16/31

19 Training subword HMMs An iterative algorithm (Baum-Welch, also known as Forward-Backward) is used. The Maximum Likelihood approach guarantees increase of the likelihood of the trained model matching with training data with each iteration. To begin with, an initial estimation of parameters of HMMs (A,B,π) is required. Q: How to get an initial estimation of (λ = {A,B,π}? A: We can estimate parameters if we know the boundaries of every subword HMM in training utterances. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 17/31

20 Training subword HMMs An iterative algorithm (Baum-Welch, also known as Forward-Backward) is used. The Maximum Likelihood approach guarantees increase of the likelihood of the trained model matching with training data with each iteration. To begin with, an initial estimation of parameters of HMMs (A,B,π) is required. Q: How to get an initial estimation of (λ = {A,B,π}? A: We can estimate parameters if we know the boundaries of every subword HMM in training utterances. Practical solution: Assume that the durations of all units (phones) are equal. If there are N phones in a training utterance, divide the feature vector sequence into N equal parts. Assign each part, to a phoneme in the phoneme sequence corresponding to the transcription of the utterance. Repeat for all training utterances. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 17/31

21 Initial estimation of HMM parameters: an illustration Let the transcription of the 1st wave file be the following sequence of words: mera bhaarat mahaan Let the relevant lines in the dictionary be as follows: bhaarata bh aa r a t mahaana m a h aa n mera m e r aa The phonemehmm sequence (of length 16) corresponding to this sentence is sil m e r aa bh aa r a t m a h aa n sil Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 18/31

22 Initial estimation of HMM parameters: an illustration Let the transcription of the 1st wave file be the following sequence of words: mera bhaarat mahaan Let the relevant lines in the dictionary be as follows: bhaarata bh aa r a t mahaana m a h aa n mera m e r aa The phonemehmm sequence (of length 16) corresponding to this sentence is sil m e r aa bh aa r a t m a h aa n sil If the duration of the wavefile is 1.0sec, there will 98 feature vectors (frame shift = 10msec and frame size = 25msec). Assign the first 6 feature vectors to sil HMM; the next 6 (7 through 12) to m ; the next 6 (13 through 18) to e ;... ; the last 8 feature vectors to sil. If HMM has 3 states, assign 2 feature vector to each state; compute mean,sd. Assume a i,j =0.5 if j=i or j=i+1; else assign 0. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 18/31

23 Embedded Re-estimation (II) Embedding re-estimation: 1 For each utterance, do the following: Using the phone-level transcriptions, compose a sentence HMM out of phone HMMs. Forward-Backward algorithm: compute the likelihood of each feature vector being generated by each state of each phone HMM in the sentence HMM Accumulate likelihoods of feature vectors being generated by each state. 2 For each state: re-estimate HMM parameters using the accumulated likelihoods. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 19/31

24 Embedded Re-estimation (II) Embedding re-estimation: 1 For each utterance, do the following: Using the phone-level transcriptions, compose a sentence HMM out of phone HMMs. Forward-Backward algorithm: compute the likelihood of each feature vector being generated by each state of each phone HMM in the sentence HMM Accumulate likelihoods of feature vectors being generated by each state. 2 For each state: re-estimate HMM parameters using the accumulated likelihoods. Repeat the Embedded Re-estimation a few times. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 19/31

25 Training Context Dependent phone HMMs 1 Initialise N 3 triphone models, where N is the number of phones. 2 Compose sentence HMM out of triphone (CD) models instead of monophone (CI) models. 3 Carry out the Embedded Re-estimation for a few iterations. The sequence of CI HMMs was sil m e r aa bh aa r a t m a h aa n sil The sequence of CD HMMs (triphones) is sil sil-m+e m-e+r e-r+aa r-aa+bh... Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 20/31

26 Training Context Dependent phone HMMs 1 Initialise N 3 triphone models, where N is the number of phones. 2 Compose sentence HMM out of triphone (CD) models instead of monophone (CI) models. 3 Carry out the Embedded Re-estimation for a few iterations. The sequence of CI HMMs was sil m e r aa bh aa r a t m a h aa n sil The sequence of CD HMMs (triphones) is sil sil-m+e m-e+r e-r+aa r-aa+bh... If N = 50, each HMM has 3 states, there may be upto 375,000 states. Each state is associated with one Gaussian. Huge amount of speech data is needed for robust estimation of the parameters (µ,σ) of 375,000 Gaussians! Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 20/31

27 Training Context Dependent phone HMMs 1 Initialise N 3 triphone models, where N is the number of phones. 2 Compose sentence HMM out of triphone (CD) models instead of monophone (CI) models. 3 Carry out the Embedded Re-estimation for a few iterations. The sequence of CI HMMs was sil m e r aa bh aa r a t m a h aa n sil The sequence of CD HMMs (triphones) is sil sil-m+e m-e+r e-r+aa r-aa+bh... If N = 50, each HMM has 3 states, there may be upto 375,000 states. Each state is associated with one Gaussian. Huge amount of speech data is needed for robust estimation of the parameters (µ,σ) of 375,000 Gaussians! Reduce the number of states by state-tying; use Decision Trees. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 20/31

28 Training Context Dependent tied phone HMMs * Build Decision Trees for parameter sharing. * One decision tree is built for each state position (5 decision trees if there are 5 emitting states of HMMs). Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 21/31

29 Training Context Dependent tied phone HMMs * Build Decision Trees for parameter sharing. * One decision tree is built for each state position (5 decision trees if there are 5 emitting states of HMMs). The first step is to generate Linguistic Questions. Two methods: 1 Manually create linguistic questions using phonetic knowledge. 2 Run make quests program to automatically form phone groups. First few lines of a linguistic-questions file may look like this. SIL sil h s sh VOWELS a aa i ii u uu e ee o oo NASAL m n ng LABPLO p ph b bh Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 21/31

30 Decision trees are used to decide which of the HMM states of all the triphones (seen and unseen) are similar to each other, so that data from all these states are collected together and used to train one global state, which is called a senone (also called a tied state). Example: Left states of 1st and 3rd triphones above would be similar. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 22/31

31 Training Context Dependent tied phone HMMs 1 Prune the Decision trees so that the number of senones (tied states) is commensurate with the amount of training data. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 23/31

32 Training Context Dependent tied phone HMMs 1 Prune the Decision trees so that the number of senones (tied states) is commensurate with the amount of training data. 2 Create CD tied model definition file that has (a) all triphones which are seen during training, and (b) has the states corresponding to these triphones identified with senones from the pruned trees (state-senone mapping). 3 Carry out the Embedded Re-estimation (tied CD models) for a few iterations. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 23/31

33 Training Context Dependent tied phone HMMs 1 Prune the Decision trees so that the number of senones (tied states) is commensurate with the amount of training data. 2 Create CD tied model definition file that has (a) all triphones which are seen during training, and (b) has the states corresponding to these triphones identified with senones from the pruned trees (state-senone mapping). 3 Carry out the Embedded Re-estimation (tied CD models) for a few iterations. 4 Generate Gaussian mixtures for each senone (tied state) and re-train. Repeat this step till the desired number (say 8) of mixtures are created for each GMM (senone). Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 23/31

34 Training Context Dependent tied phone HMMs 1 Prune the Decision trees so that the number of senones (tied states) is commensurate with the amount of training data. 2 Create CD tied model definition file that has (a) all triphones which are seen during training, and (b) has the states corresponding to these triphones identified with senones from the pruned trees (state-senone mapping). 3 Carry out the Embedded Re-estimation (tied CD models) for a few iterations. 4 Generate Gaussian mixtures for each senone (tied state) and re-train. Repeat this step till the desired number (say 8) of mixtures are created for each GMM (senone). 5 One can carry out discriminative training following the Maximum Mutual Information Estimation scheme (maximises the posterior probability of the correct word sequence given all possible word sequences) [9]. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 23/31

35 source: erwin/sr2003/sphinx.ppt Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 24/31

36 Inputs to sphinx3 decoder source: erwin/sr2003/sphinx.ppt Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 25/31

37 Sphinx3 decoders source: erwin/sr2003/sphinx.ppt Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 26/31

38 Output of recogniser source: erwin/sr2003/sphinx.ppt Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 27/31

39 Samudravijaya source: K SphinxLunch ppt samudravijaya@gmail.com bytoolkits Arthur for ASR; Chan, Sphinx /31

40 Sphinx4 Sphinx-4 is a state-of-the-art speech recognition system written entirely in the Java programming language [10]. Generalized pluggable front end architecture: MFCC, CMN Generalized pluggable language model architecture: trigram, JSGF and ARPA-format FST grammars. Generalized acoustic model architecture: Sphinx-3 acoustic models. Generalized search management: breadth first and word pruning Post-processing recognition results: obtaining confidence scores, generating lattices. Standalone tools: displaying waveforms and spectrograms; generating features from audio. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 29/31

41 Comparison of Performance of Sphinxes source: [10]. PocketSphinx[11]: It is a small-footprint continuous speech recognition system, suitable for handheld and desktop applications. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 30/31

42 Sphinx, the eternal mystery source: [10]. Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 31/31

43 Bibliography Cambridge University, UK; Entropic; Microsoft HTK, Hidden Markov Model ToolKit Project by Carnegie Mellon University The CMU Sphinx group open source speech recognition engines Joe Picone et al. ISIP Production system (r02 n02) (23-JUL-2009) Japanese Universities and Laboratories Open-Source Large Vocabulary CSR Engine: Julius Kevin Murphy HMM toolbox for Matlab Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 31/31

44 murphyk/software/hmm/hmm.html Arthur Chan Hieroglyphs: Building Speech Application Using Sphinx and Related Resources, (3rd Draft) 11-MAR Ravishankar M., Efficient Algorithms for Speech Recognition Ph.D Thesis, Carnegie Mellon University, May 1996, Tech Report. CMU-CS rkm/th/th.pdf Cambridge University, UK; Entropic; Microsoft HTK Book, Documentation of HTK L Qin and A Rudnicky Implementing and Improving MMIE Training in SphinxTrain CMU Sphinx Workshop 2010, 13 March 2010, Dallas, USA sphinx/sphinx2010/papers/107.unblinded.p Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 31/31

45 Bhiksharaj et al. A speech recognizer written entirely in the Java programming language A small-footprint continuous speech recognition system release/ Samudravijaya K samudravijaya@gmail.com Toolkits for ASR; Sphinx 31/31