# Word Recognition with Conditional Random Fields. Jeremy Morris 2/05/2010

Save this PDF as:
Size: px
Start display at page:

## Transcription

1 ord Recognition with Conditional Random Fields Jeremy Morris 2/05/2010 1

2 Outline Background ord Recognition CRF Model Pilot System - TIDIGITS Larger Vocabulary - SJ Future ork 2

3 Background Conditional Random Fields (CRFs Discriminative probabilistic sequence model Directly defines a posterior probability YX of a label sequence Y given a set of observations X 3

4 Background Y X exp ( isi ( x, yk jt j ( x, yk, yk k i j Z( x 1 The form of the CRF model includes weighted state feature functions and weighted transition feature functions Both types of functions can be defined to incorporate observed inputs 4

5 Background Our previous work compared CRF models for phone recognition to HMM models Model Accuracy CRF (phone classes 69.92%* HMM Tandem16mix (phone classes 69.34% CRF (phone classes +phonological features 70.63%* HMM Tandem16mix (phone classes+ phonological features 69.40% *Signficantly (p<0.05 better than comparable Tandem 16mix triphone system (Morris & Fosler-Lussier 08 5

6 Background Problem: How do we make use of CRF classification for word recognition? Attempt to fit CRFs into current state-of-the-art models for speech recognition? Attempt to use CRFs directly? Each approach has its benefits Fitting CRFs into a standard framework lets us reuse existing code and ideas (Crandem system A model that uses CRFs directly opens up new directions for investigation Requires some rethinking of the standard model for ASR 6

7 Review - ord Recognition argmax X Problem: For a given input signal X, find the word string that maximizes X 7

8 Review - ord Recognition arg max X arg max X X Problem: For a given input signal X, find the word string that maximizes X In an HMM, we would make this a generative problem 8

9 Review - ord Recognition argmax X argmax X Problem: For a given input signal X, find the word string that maximizes X In an HMM, we would make this a generative problem e can drop the X because it does not affect the choice of 9

10 Review - ord Recognition argmax X argmax X e want to build phone models, not whole word models 10

11 Review - ord Recognition argmax X argmax X argmax X e want to build phone models, not whole word models so we marginalize over the phones 11

12 Review - ord Recognition argmax X argmax X e want to build phone models, not whole word models so we marginalize over the phones argmax X arg max X, and look for the best sequence that fits these constraints 12

13 Review - ord Recognition X Acoustic Model Language Model Lexicon 13

14 ord Recognition X Acoustic Model However - our CRFs model ΦX rather than XΦ This makes the formulation of the problem somewhat different 14

15 ord Recognition argmax X e want a formulation that makes use of ΦX 15

16 ord Recognition argmax X argmax, X argmax, X X e want a formulation that makes use of ΦX e can get that by marginalizing over the phone strings But the CRF as we formulate it doesn t give ΦX directly 16

17 ord Recognition, X X Φ here is a phone level assignment of phone labels CRF gives related quantity QX where Q is the frame level assignment of phone labels 17

18 ord Recognition Frame level vs. Phone level Mapping from frame level to phone level may not be deterministic Example: The word OH with pronunciation /ow/ Consider this sequence of frame labels: ow ow ow ow ow ow ow This sequence can possibly be expanded many different ways for the word OH ( OH, OH OH, etc. 18

19 ord Recognition Frame level vs. Phone segment level This problem occurs because we re using a single state to represent the phone /ow/ Phone either transitions to itself or transitions out to another phone e can change our model to a multi-state model and make this decision deterministic This brings us closer to a standard ASR HMM topology ow1 ow2 ow2 ow2 ow2 ow3 ow3 Now we can see a single OH in this utterance 19

20 ord Recognition X Multi-state model gives us a deterministic mapping of Q -> Φ P (, Q X Q P ( Q, X Q X Q ( Q P Q Q X Each frame-level assignment Q has exactly one segment level assignment associated with it Potential pitfalls if the multi-state model is inappropriate for the features we are using 20

21 ord Recognition Replacing ΦX we now have a model with our CRF in it hat about Φ,X? Conditional independence assumption gives Φ 21 ( max arg X P (, ( max arg X P X P Q X Q P Q P X P, ( (, ( argmax Q X Q P Q P P, ( ( ( argmax

22 ord Recognition X, Q Q Q X hat about Φ? Non-deterministic across sequences of words Φ = / ah f eh r / =? a fair? affair? The more words in the string, the more possible combinations can arise 22

23 ord Recognition Bayes Rule language model Φ dictionary model Φ prior probability of phone sequences 23

24 ord Recognition hat is Φ? Prior probability over possible phone sequences Essentially acts as a phone fertility/penalty term lower probability sequences get a larger boost in weight than higher probability sequences Approximate this with a standard n-gram model Seed it with phone-level statistics drawn from the same corpus used for our language model 24

25 ord Recognition argmax X arg max,, Q Q Q X Our final model incorporates all of these pieces together Benefit of this approach reuse of standard models Each element can be built as a finite state machine (FSM Evaluation can be performed via FSM composition and best path evaluation as for HMM-based systems (Mohri & Riley,

26 Pilot Experiment: TIDIGITS First word recognition experiment TIDIGITS recognition Both isolated and strings of spoken digits, ZERO (or OH to NINE Male and female speakers Training set 112 speakers total Random selection of 11 speakers held out as development set Remaining 101 speakers used for training as needed 26

27 Pilot Experiment: TIDIGITS argmax X arg max,, Q Q Q X Important characteristics of the DIGITS problem: A given phone sequence maps to a single word sequence A uniform distribution over the words is assumed Φ easy to implement directly as FSM 27

28 Pilot Experiment: TIDIGITS Implementation Created a composed dictionary and language model FST No probabilistic weights applied to these FSTs assumption of uniform probability of any digit sequence Modified CRF code to allow composition of above FST with phone lattice Results scored using standard HTK tools Compared to a baseline HMM system trained on the same features 28

29 Pilot Experiment: TIDIGITS Labels Unlike TIMIT, TIDIGITS is only labeled at the word level Phone labels were generated by force aligning the word labels using an HMM-trained, MFCC based system Features TIMIT-trained MLPs applied to TIDIGITS to create features for CRF and HMM training 29

30 Pilot Experiment: Results Model ER HMM (triphone, 1 Gaussinan, ~4500 parameters 1.26% HMM (triphone, 16 Gaussians ~120,000 paramters 0.57% CRF (monophone, ~4200 parameters 1.11% CRF (monophone, windowed, ~37000 parameters 0.57% HMM (triphone, 16 Gaussians, MFCCs 0.25% Basic CRF performance falls in line with HMM performance for a single Gaussian model Adding more parameters to the CRF enables the CRF to perform as well as the HMM on the same features 30

31 Larger Vocabulary all Street Journal 5K word vocabulary task Bigram language model MLPs trained on 75 speakers, 6488 utterances Cross-validated on 8 speakers, 650 utterances Development set of 10 speakers, 368 utterances for tuning purposes Results compared to HMM-Tandem baseline and HMM-MFCC baseline 31

32 Larger Vocabulary Phone penalty model Φ Constructed using the transcripts and the lexicon Currently implemented as a phone pair (bigram model More complex model might lead to better estimates 32

33 Larger Vocabulary Direct finite-state composition not feasible for this task State space grows too large too quickly Instead Viterbi decoding performed using the weighted finite-state models as constraints Time-synchronous beam pruning used to keep time and space usage reasonable 33

34 Larger Vocabulary Initial Results Model ER HMM MFCC Baseline 9.3% HMM PLP Baseline 9.7% HMM Tandem MLP 9.1% CRF (phone 11.3% CRF (phone windowed 11.7% CRF (phone + phonological 10.9% CRF (3state phone inputs 12.4% CRF (3state phone + phono 11.7% HMM PLP (monophone labels 17.5% Preliminary numbers reported on development set only 34

35 Next Steps Context Exploring ways to put more context into the CRF, either at the label level or at the feature level Feature selection Examine what features will help this model, especially features that may be useful for the CRF that are not useful for HMMs Phone penalty model Results reported with just a bigram phone model A more interesting model leads to more complexity but may lead to better results Currently examining trigram phone model to test the impact 35

36 Discussion 36

37 References J. Lafferty et al, Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data, Proc. ICML, 2001 A. Gunawardana et al, Hidden Conditional Random Fields for phone classification, Proc. Interspeech, 2005 J. Morris and E. Fosler-Lussier. Conditional Random Fields for Integrating Local Discriminative Classifiers, IEEE Transactions on Audio, Speech and Language Processing, 2008 M. Mohri et al, eighted finite-state transducers in speech recognition, Computer Speech and Language,

38 Background Tandem HMM Generative probabilistic sequence model Uses outputs of a discriminative model (e.g. ANN MLPs as input feature vectors for a standard HMM 38

39 Background Tandem HMM ANN MLP classifiers are trained on labeled speech data Classifiers can be phone classifiers, phonological feature classifiers Classifiers output posterior probabilities for each frame of data E.g. Q X, where Q is the phone class label and X is the input speech feature vector 39

40 Background Tandem HMM Posterior feature vectors are used by an HMM as inputs In practice, posteriors are not used directly Log posterior outputs or linear outputs are more frequently used linear here means outputs of the MLP with no application of a softmax function Since HMMs model phones as Gaussian mixtures, the goal is to make these outputs look more Gaussian Additionally, Principle Components Analysis (PCA is applied to features to decorrelate features for diagonal covariance matrices 40

41 Idea: Crandem Use a CRF model to create inputs to a Tandem-style HMM CRF labels provide a better per-frame accuracy than input MLPs e ve shown CRFs to provide better phone recognition than a Tandem system with the same inputs This suggests that we may get some gain from using CRF features in an HMM 41

42 Idea: Crandem Problem: CRF output doesn t match MLP output MLP output is a per-frame vector of posteriors CRF outputs a probability across the entire sequence Solution: Use Forward-Backward algorithm to generate a vector of posterior probabilities 42

43 Forward-Backward Algorithm Similar to HMM forward-backward algorithm Used during CRF training Forward pass collects feature functions for the timesteps prior to the current timestep Backward pass collects feature functions for the timesteps following the current timestep Information from both passes are combined together to determine the probability of being in a given state at a particular timestep 43

44 Forward-Backward Algorithm y i, t X i, t i, t Z( x This form allows us to use the CRF to compute a vector of local posteriors y at any timestep t. e use this to generate features for a Tandem-style system Take log features, decorrelate with PCA 44

45 Phone Recognition Pilot task phone recognition on TIMIT 61 feature MLPs trained on TIMIT, mapped down to 39 features for evaluation Crandem compared to Tandem and a standard PLP HMM baseline model As with previous CRF work, we use the outputs of an ANN MLP as inputs to our CRF Phone class attributes Detector outputs describe the phone label associated with a portion of the speech signal /t/, /d/, /aa/, etc. 45

46 Results (Fosler-Lussier & Morris 08 Model Phone Accuracy PLP HMM reference 68.1% Tandem 70.8% CRF 69.9% Crandem log 71.1% * Significantly (p<0.05 improvement at 0.6% difference between models 46

47 ord Recognition Second task ord recognition on SJ0 Dictionary for word recognition has 54 distinct phones instead of 48 New CRFs and MLPs trained to provide input features MLPs and CRFs trained on SJ0 corpus of read speech No phone level assignments, only word transcripts Initial alignments from HMM forced alignment of MFCC features Compare Crandem baseline to Tandem and original MFCC baselines 47

48 Initial Results Model ER MFCC HMM reference 9.12% Tandem MLP ( % Crandem (19 (1 epoch 8.85% Crandem (19 (10 epochs 9.57% Crandem (19 (20 epochs 9.98% * Significant (p 0.05 improvement at roughly 1% difference between models 48

49 ord Recognition CRF performs about the same as the baseline systems But further training of the CRF tends to degrade the result of the Crandem system hy? First thought maybe the phone recognition results are deteriorating (overtraining 49

50 Initial Results Model Phone Accuracy MFCC HMM reference 70.09% Tandem MLP ( % Crandem (19 (1 epoch 72.77% Crandem (19 (10 epochs 72.81% Crandem (19 (20 epochs 72.93% * Significant (p 0.05 improvement at roughly 0.07% difference between models 50

51 ord Recognition Further training of the CRF tends to degrade the result of the Crandem system hy? First thought maybe the phone recognition results are deteriorating (overtraining Not the case Next thought examine the pattern of errors between iterations 51

52 Initial Results Model Total Errors Insertions Deletions Subs. Crandem (1 epoch Crandem (10 epochs Shared Errors New Errors (1-> * 261** (102 ( * 29 deletions are substitutions in one model and deletions in the other **50 of these subs are different words between the epoch 1 and epoch 10 models 52

53 ord Recognition Training the CRF tends to degrade the result of the Crandem system hy? First thought maybe the phone recognition results are deteriorating (overtraining Not the case Next thought examine the pattern of errors between iterations There doesn t seem to be much of a pattern here, other than a jump in substitutions ord identity doesn t give a clue similar words wrong in both lists 53

54 ord Recognition Further training of the CRF tends to degrade the result of the Crandem system hy? Current thought perhaps the reduction in scores of the correct result is impacting the overall score This appears to be happening in at least some cases, though it is not sufficient to explain everything 54

55 ord Recognition MARCH vs. LARGE Iteration m l en em m em l en m em l hh m em l w m aa em l m aa em w l Iteration m em en n l m em l en m em l en m em l aa m aa em l m aa em l

56 ord Recognition MARCH vs. LARGE - logspace Iteration m l en em m em l en m em l hh m em l w m aa em l m aa em w l Iteration m em en n l m em l en m em l en m em l aa m aa em l m aa em l

57 ord Recognition Additional issues Crandem results sensitive to format of input data Posterior probability inputs to the CRF give very poor results on word recognition. I suspect is related to the same issues described previously Crandem results also require a much smaller vector after PCA MLP uses 39 features Crandem only does well once we reduce to 19 features However, phone recognition results improve if we use 39 features in the Crandem system (72.77% -> 74.22% 57

### Word Recognition with Conditional Random Fields

Outline ord Recognition with Conditional Random Fields Jeremy Morris 2/05/2010 ord Recognition CRF Pilot System - TIDIGITS Larger Vocabulary - SJ Future ork 1 2 Conditional Random Fields (CRFs) Discriminative

### Discriminative Phonetic Recognition with Conditional Random Fields

Discriminative Phonetic Recognition with Conditional Random Fields Jeremy Morris & Eric Fosler-Lussier Dept. of Computer Science and Engineering The Ohio State University Columbus, OH 43210 {morrijer,fosler}@cse.ohio-state.edu

### Automatic Speech Recognition: Introduction

Automatic Speech Recognition: Introduction Steve Renals & Hiroshi Shimodaira Automatic Speech Recognition ASR Lecture 1 15 January 2018 ASR Lecture 1 Automatic Speech Recognition: Introduction 1 Automatic

### L15: Large vocabulary continuous speech recognition

L15: Large vocabulary continuous speech recognition Introduction Acoustic modeling Language modeling Decoding Evaluating LVCSR systems This lecture is based on [Holmes, 2001, ch. 12; Young, 2008, in Benesty

### Segment-Based Speech Recognition

Segment-Based Speech Recognition Introduction Searching graph-based observation spaces Anti-phone modelling Near-miss modelling Modelling landmarks Phonological modelling Lecture # 16 Session 2003 6.345

### Hidden Markov Models use for speech recognition

HMMs 1 Phoneme HMM HMMs 2 Hidden Markov Models use for speech recognition Each phoneme is represented by a left-to-right HMM with 3 states Contents: Viterbi training Acoustic modeling aspects Isolated-word

### A STUDY ON THE USE OF CONDITIONAL RANDOM FIELDS FOR AUTOMATIC SPEECH RECOGNITION

A STUDY ON THE USE OF CONDITIONAL RANDOM FIELDS FOR AUTOMATIC SPEECH RECOGNITION DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School

### Automatic Speech Recognition: Introduction

Automatic Speech Recognition: Introduction Steve Renals & Hiroshi Shimodaira Automatic Speech Recognition ASR Lecture 1 14 January 2019 ASR Lecture 1 Automatic Speech Recognition: Introduction 1 Automatic

### Learning from Mistakes: Expanding Pronunciation Lexicons using Word Recognition Errors

Learning from Mistakes: Expanding Pronunciation Lexicons using Word Recognition Errors Sravana Reddy The University of Chicago Joint Work with Evandro Gouvêa Sang Bissenette SPEECH RECOGNITION Sane visitor

### An Acoustic Model Based on Kullback-Leibler Divergence for Posterior Features

R E S E A R C H R E P O R T I D I A P An Acoustic Model Based on Kullback-Leibler Divergence for Posterior Features Guillermo Aradilla a b Jithendra Vepa b Hervé Bourlard a b IDIAP RR 06-60 January 2007

### Sphinx Benchmark Report

Sphinx Benchmark Report Long Qin Language Technologies Institute School of Computer Science Carnegie Mellon University Overview! uate general training and testing schemes! LDA-MLLT, VTLN, MMI, SAT, MLLR,

### Project #2: Survey of Weighted Finite State Transducers (WFST)

T-61.184 : Speech Recognition and Language Modeling : From Theory to Practice Project Groups / Descriptions Fall 2004 Helsinki University of Technology Project #1: Music Recognition Jukka Parviainen (parvi@james.hut.fi)

### An Overview of the SPRACH System for the Transcription of Broadcast News

An Overview of the SPRACH System for the Transcription of Broadcast News Gary Cook (1), James Christie (1), Dan Ellis (2), Eric Fosler-Lussier (2), Yoshi Gotoh (3), Brian Kingsbury (2), Nelson Morgan (2),

### Discriminative Learning of Feature Functions of Generative Type in Speech Translation

Discriminative Learning of Feature Functions of Generative Type in Speech Translation Xiaodong He Microsoft Research, One Microsoft Way, Redmond, WA 98052 USA Li Deng Microsoft Research, One Microsoft

### Final paper for Course T : Survey Project - Segment-based Speech Recognition

Final paper for Course T-61.184: Survey Project - Segment-based Speech Recognition Petri Korhonen Helsinki University of Technology petri@acoustics.hut.fi Abstract Most speech recognition systems take

### A Hybrid Neural Network/Hidden Markov Model

A Hybrid Neural Network/Hidden Markov Model Method for Automatic Speech Recognition Hongbing Hu Advisor: Stephen A. Zahorian Department of Electrical and Computer Engineering, Binghamton University 03/18/2008

### CS 545 Lecture XI: Speech (some slides courtesy Jurafsky&Martin)

CS 545 Lecture XI: Speech (some slides courtesy Jurafsky&Martin) brownies_choco81@yahoo.com brownies_choco81@yahoo.com Benjamin Snyder Announcements Office hours change for today and next week: 1pm - 1:45pm

### Discriminative Training for Segmental Minimum Bayes Risk Decoding

Discriminative Training for Segmental Minimum Bayes Risk Decoding Vlasios Doumpiotis, Stavros Tsakalidis, Bill Byrne Center for Language and Speech Processing Department of Electrical and Computer Engineering

### Pronunciation Modeling. Te Rutherford

Pronunciation Modeling Te Rutherford Bottom Line Fixing pronunciation is much easier and cheaper than LM and AM. The improvement from the pronunciation model alone can be sizeable. Overview of Speech

### SAiL Speech Recognition or Speech-to-Text conversion: The first block of a virtual character system.

Speech Recognition or Speech-to-Text conversion: The first block of a virtual character system. Panos Georgiou Research Assistant Professor (Electrical Engineering) Signal and Image Processing Institute

### Discriminative Learning of Feature Functions of Generative Type in Speech Translation

Discriminative Learning of Feature Functions of Generative Type in Speech Translation Xiaodong He Microsoft Research, One Microsoft Way, Redmond, WA 98052 USA Li Deng Microsoft Research, One Microsoft

### Statistical pattern matching: Outline

Statistical pattern matching: Outline Introduction Markov processes Hidden Markov Models Basics Applied to speech recognition Training issues Pronunciation lexicon Large vocabulary speech recognition 1

### Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition

Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition Alex Graves 1, Santiago Fernández 1, Jürgen Schmidhuber 1,2 1 IDSIA, Galleria 2, 6928 Manno-Lugano, Switzerland {alex,santiago,juergen}@idsia.ch

### Towards Lower Error Rates in Phoneme Recognition

Towards Lower Error Rates in Phoneme Recognition Petr Schwarz, Pavel Matějka, and Jan Černocký Brno University of Technology, Czech Republic schwarzp matejkap cernocky@fit.vutbr.cz Abstract. We investigate

### Segmental Conditional Random Fields with Deep Neural Networks as Acoustic Models for First-Pass Word Recognition

Segmental Conditional Random Fields with Deep Neural Networks as Acoustic Models for First-Pass Word Recognition Yanzhang He, Eric Fosler-Lussier Department of Computer Science and Engineering The hio

### HMM Speech Recognition. Words: Pronunciations and Language Models. Out-of-vocabulary (OOV) rate. Pronunciation dictionary.

HMM Speech Recognition ords: Pronunciations and Language Models Recorded Speech Decoded Text (Transcription) Steve Renals Signal Analysis Acoustic Model Automatic Speech Recognition ASR Lecture 8 11 February

### VOWEL NORMALIZATIONS WITH THE TIMIT ACOUSTIC PHONETIC SPEECH CORPUS

Institute of Phonetic Sciences, University of Amsterdam, Proceedings 24 (2001), 117 123. VOWEL NORMALIZATIONS WITH THE TIMIT ACOUSTIC PHONETIC SPEECH CORPUS David Weenink Abstract In this paper we present

### CHAPTER 4 IMPROVING THE PERFORMANCE OF A CLASSIFIER USING UNIQUE FEATURES

38 CHAPTER 4 IMPROVING THE PERFORMANCE OF A CLASSIFIER USING UNIQUE FEATURES 4.1 INTRODUCTION In classification tasks, the error rate is proportional to the commonality among classes. Conventional GMM

### Mispronunciation Detection and Diagnosis in L2 English Speech Using Multi-Distribution Deep Neural Networks

Mispronunciation Detection and Diagnosis in L2 English Speech Using Multi-Distribution Deep Neural Networks Kun Li and Helen Meng Human-Computer Communications Laboratory Department of System Engineering

### Joint Decoding for Phoneme-Grapheme Continuous Speech Recognition Mathew Magimai.-Doss a b Samy Bengio a Hervé Bourlard a b IDIAP RR 03-52

R E S E A R C H R E P O R T I D I A P Joint Decoding for Phoneme-Grapheme Continuous Speech Recognition Mathew Magimai.-Doss a b Samy Bengio a Hervé Bourlard a b IDIAP RR 03-52 October 2003 submitted for

FLEXIBLE SPEAKER ADAPTATION USING MAXIMUM LIKELIHOOD LINEAR REGRESSION C.J. Leggetter & P.C. Woodland Cambridge University Engineering Department Trumpington Street, Cambridge CB2 1PZ. UK. ABSTRACT The

### The 1997 CMU Sphinx-3 English Broadcast News Transcription System

The 1997 CMU Sphinx-3 English Broadcast News Transcription System K. Seymore, S. Chen, S. Doh, M. Eskenazi, E. Gouvêa, B. Raj, M. Ravishankar, R. Rosenfeld, M. Siegler, R. Stern, and E. Thayer Carnegie

### Hidden Markov Models (HMMs) - 1. Hidden Markov Models (HMMs) Part 1

Hidden Markov Models (HMMs) - 1 Hidden Markov Models (HMMs) Part 1 May 21, 2013 Hidden Markov Models (HMMs) - 2 References Lawrence R. Rabiner: A Tutorial on Hidden Markov Models and Selected Applications

### Toolkits for ASR; Sphinx

Toolkits for ASR; Sphinx Samudravijaya K samudravijaya@gmail.com 08-MAR-2011 Workshop on Fundamentals of Automatic Speech Recognition CDAC Noida, 08-MAR-2011 Samudravijaya K samudravijaya@gmail.com Toolkits

### IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 19, NO. 7, SEPTEMBER

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 19, NO. 7, SEPTEMBER 2011 1999 Large Margin Discriminative Semi-Markov Model for Phonetic Recognition Sungwoong Kim, Student Member, IEEE,

### Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding

Recurrent Neural Network Structured Output Prediction for Spoken Language Understanding Bing Liu, Ian Lane Department of Electrical and Computer Engineering Carnegie Mellon University {liubing,lane}@cmu.edu

### Speech/Non-Speech Segmentation Based on Phoneme Recognition Features

Hindawi Publishing Corporation EURASIP Journal on Applied Signal Processing Volume 2006, Article ID 90495, Pages 1 13 DOI 10.1155/ASP/2006/90495 Speech/Non-Speech Segmentation Based on Phoneme Recognition

### Using Posterior-Based Features in Template Matching for Speech Recognition Guillermo Aradilla a Jithendra Vepa a Hervé Bourlard a IDIAP RR 06-23

R E S E A R C H R E P O R T I D I A P Using Posterior-Based Features in Template Matching for Speech Recognition Guillermo Aradilla a Jithendra Vepa a Hervé Bourlard a IDIAP RR 06-23 June 2006 published

### Boosting N-gram Coverage for Unsegmented Languages Using Multiple Text Segmentation Approach

Boosting N-gram Coverage for Unsegmented Languages Using Multiple Text Segmentation Approach Solomon Teferra Abate LIG Laboratory, CNRS/UMR-5217 solomon.abate@imag.fr Laurent Besacier LIG Laboratory, CNRS/UMR-5217

### Plasticity in Systems for Automatic Speech Recognition: A Review. Roger K Moore & Stuart P Cunningham. Overview

Plasticity in Systems for Automatic Speech Recognition: A Review Roger K Moore & Stuart P Cunningham Overview Automatic Speech Recognition (ASR) breakthroughs key components training / recognition Practical

### Multilingual Non-Native Speech Recognition using Phonetic Confusion-Based Acoustic Model Modification and Graphemic Constraints

Multilingual Non-Native Speech Recognition using Phonetic Confusion-Based Acoustic Model Modification and Graphemic Constraints Ghazi Bouselmi, Dominique Fohr, Irina Illina, Jean-Paul Haton To cite this

### ADAPTIVE training [1], [2] has become increasingly popular

1932 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 15, NO. 6, AUGUST 2007 Bayesian Adaptive Inference and Adaptive Training Kai Yu, Member, IEEE, and Mark J. F. Gales, Member, IEEE

### Unsupervised Methods for Speaker Diarization: An Integrated and Iterative Approach!

Unsupervised Methods for Speaker Diarization: An Integrated and Iterative Approach! Stephen Shum, Najim Dehak, and Jim Glass!! *With help from Reda Dehak, Ekapol Chuangsuwanich, and Douglas Reynolds November

### Adaptation of HMMS in the presence of additive and convolutional noise

Adaptation of HMMS in the presence of additive and convolutional noise Hans-Gunter Hirsch Ericsson Eurolab Deutschland GmbH, Nordostpark 12, 9041 1 Nuremberg, Germany Email: hans-guenter.hirsch@eedn.ericsson.se

### Phoneme Recognition Using Deep Neural Networks

CS229 Final Project Report, Stanford University Phoneme Recognition Using Deep Neural Networks John Labiak December 16, 2011 1 Introduction Deep architectures, such as multilayer neural networks, can be

### Speech Recognition Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Speech Recognition Deep Speech 2: End-to-End Speech Recognition in English and Mandarin Amnon Drory & Matan Karo 19/12/2017 Deep Speech 1 Overview 19/12/2017 Deep Speech 2 Automatic Speech Recognition

### Statistical Pronunciation Modeling for Non-native Speech

Statistical Pronunciation Modeling for Non-native Speech Dissertation Rainer Gruhn Nov. 14 th, 2008 Institute of Information Technology University of Ulm, Germany In cooperation with Advanced Telecommunication

### Inter-Ing INTERDISCIPLINARITY IN ENGINEERING SCIENTIFIC INTERNATIONAL CONFERENCE, TG. MUREŞ ROMÂNIA, November 2007.

Inter-Ing 2007 INTERDISCIPLINARITY IN ENGINEERING SCIENTIFIC INTERNATIONAL CONFERENCE, TG. MUREŞ ROMÂNIA, 15-16 November 2007. FRAME-BY-FRAME PHONEME CLASSIFICATION USING MLP DOMOKOS JÓZSEF, SAPIENTIA

### Hidden Markov Models (HMMs) - 1. Hidden Markov Models (HMMs) Part 1

Hidden Markov Models (HMMs) - 1 Hidden Markov Models (HMMs) Part 1 May 24, 2012 Hidden Markov Models (HMMs) - 2 References Lawrence R. Rabiner: A Tutorial on Hidden Markov Models and Selected Applications

### Pavel Král and Václav Matoušek University of West Bohemia in Plzeň (Pilsen), Czech Republic pkral

EVALUATION OF AUTOMATIC SPEAKER RECOGNITION APPROACHES Pavel Král and Václav Matoušek University of West Bohemia in Plzeň (Pilsen), Czech Republic pkral matousek@kiv.zcu.cz Abstract: This paper deals with

### REMAP: RECURSIVE ESTIMATION AND MAXIMIZATION OF A POSTERIORI PROBABILITIES Application to Transition-Based Connectionist Speech Recognition

! "\$#\$%'&)(+*,\$-.*/0-354)0567-8*:9;;)4=@*A *B\$C'(EDA 7 FHG'/?,7IDJ#\$%'&;\$%LK@""M#NO4QP8RS"\$;TU9L%WVMK #"R'V)4=XZY\[P8R]"\$;TJ9L%'VZK &9N\$% REMAP: RECURSIVE ESTIMATION AND MAXIMIZATION OF A POSTERIORI

### BENEFIT OF MUMBLE MODEL TO THE CZECH TELEPHONE DIALOGUE SYSTEM

BENEFIT OF MUMBLE MODEL TO THE CZECH TELEPHONE DIALOGUE SYSTEM Luděk Müller, Luboš Šmídl, Filip Jurčíček, and Josef V. Psutka University of West Bohemia, Department of Cybernetics, Univerzitní 22, 306

### A Senone Based Confidence Measure for Speech Recognition

Utah State University DigitalCommons@USU Space Dynamics Lab Publications Space Dynamics Lab 1-1-1997 A Senone Based Confidence Measure for Speech Recognition Z. Bergen W. Ward Follow this and additional

### We will first consider search methods, as they then will be used in the training algorithms.

Lecture 15: Training and Search for Speech Recognition In earlier lectures we have seen the basic techniques for training and searching HMMs. In speech recognition applications, however, the networks are

### An Improvement of robustness to speech loudness change for an ASR system based on LC-RC features

An Improvement of robustness to speech loudness change for an ASR system based on LC-RC features Pavel Yurkov, Maxim Korenevsky, Kirill Levin Speech Technology Center, St. Petersburg, Russia Abstract This

### Comparison and Combination of Multilayer Perceptrons and Deep Belief Networks in Hybrid Automatic Speech Recognition Systems

APSIPA ASC 2011 Xi an Comparison and Combination of Multilayer Perceptrons and Deep Belief Networks in Hybrid Automatic Speech Recognition Systems Van Hai Do, Xiong Xiao, Eng Siong Chng School of Computer

### Using Word Confusion Networks for Slot Filling in Spoken Language Understanding

INTERSPEECH 2015 Using Word Confusion Networks for Slot Filling in Spoken Language Understanding Xiaohao Yang, Jia Liu Tsinghua National Laboratory for Information Science and Technology Department of

### Midterm practice questions

Midterm practice questions UMass CS 585 October 2017 1 Topics on the midterm Language concepts Parts of speech Regular expressions, text normalization Probability / machine learning Probability theory:

### CS224 Final Project. Re Alignment Improvements for Deep Neural Networks on Speech Recognition Systems. Firas Abuzaid

Abstract CS224 Final Project Re Alignment Improvements for Deep Neural Networks on Speech Recognition Systems Firas Abuzaid The task of automatic speech recognition has traditionally been accomplished

### An Efficient Algorithm for the Transcription of Spontaneous Speech

An Efficient Algorithm for the Transcription of Spontaneous Speech Ami Moyal ACLP Afeka Center for Language Processing Afeka Academic College of Engineering SpeechTEK Europe May 27, 2010 ACLP Afeka Center

### Resource Optimized Speech Recognition using Kullback-Leibler Divergence based HMM

Resource Optimized Speech Recognition using Kullback-Leibler Divergence based HMM Ramya Rasipuram David Imseng, Marzieh Razavi, Mathew Magimai Doss, Herve Bourlard 24 October 2014 1/23 Automatic Speech

### Multi-View Learning of Acoustic Features for Speaker Recognition

Multi-View Learning of Acoustic Features for Speaker Recognition Karen Livescu 1, Mark Stoehr 2 1 TTI-Chicago, 2 University of Chicago Chicago, IL 60637, USA 1 klivescu@uchicago.edu, 2 stoehr@uchicago.edu

### Words: Pronunciations and Language Models

Words: Pronunciations and Language Models Steve Renals Informatics 2B Learning and Data Lecture 9 19 February 2009 Steve Renals Words: Pronunciations and Language Models 1 Overview Words The lexicon Pronunciation

### Joint Discriminative Learning of Acoustic and Language Models on Decoding Graphs

Joint Discriminative Learning of Acoustic and Language Models on Decoding Graphs Abdelaziz A.Abdelhamid and Waleed H.Abdulla Department of Electrical and Computer Engineering, Auckland University, New

### Trees: Themes and Variations

Trees: Themes and Variations Prof. Mari Ostendorf Outline Preface Decision Trees Bagging Boosting BoosTexter 1 Preface: Vector Classifiers Today we again deal with vector classifiers and supervised training:

### Automatic Speech Recognition (CS753)

Automatic Speech Recognition (CS753) Lecture 1: Introduction to Statistical Speech Recognition Instructor: Preethi Jyothi Lecture 1 Course Specifics About the course (I) Main Topics: Introduction to statistical

### I D I A P. Using more informative posterior probabilities for speech recognition R E S E A R C H R E P O R T. Jithendra Vepa a,b Herve Bourlard a,b

R E S E A R C H R E P O R T I D I A P Using more informative posterior probabilities for speech recognition Hamed Ketabdar a,b Samy Bengio a,b IDIAP RR 05-91 December 2005 published in ICASSP 06 Jithendra

### BAUM-WELCH TRAINING FOR SEGMENT-BASED SPEECH RECOGNITION. Han Shu, I. Lee Hetherington, and James Glass

BAUM-WELCH TRAINING FOR SEGMENT-BASED SPEECH RECOGNITION Han Shu, I. Lee Hetherington, and James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge,

### DEGREE FINAL PROJECT. Automatic Speech Recognition with Kaldi toolkit

DEGREE FINAL PROJECT Automatic Speech Recognition with Kaldi toolkit Study branch: Degree in Science and Telecommunication Technologies Engineering Author: Víctor Rosillo Gil Supervisor: Bartosz Ziólko

### Proceedings of NTCIR-9 Workshop Meeting, December 6-9, 2011, Tokyo, Japan

An STD system for OOV query terms using various subword units Hiroyuki Saito g231j018@s.iwate-pu.ac.jp Takuya Nakano g231i027@s.iwate-pu.ac.jp Shirou Narumi g031e133@s.iwate-pu.ac.jp Toshiaki Chiba g031g110@s.iwate-pu.ac.jp

### USE OF SYLLABLE NUCLEI LOCATIONS TO IMPROVE ASR. Chris D. Bartels and Jeff A. Bilmes

USE OF SYLLABLE NUCLEI LOCATIONS TO IMPROVE ASR Chris D. Bartels and Jeff A. Bilmes University of Washington Department of Electrical Engineering Seattle, WA 98195 {bartels,bilmes}@ee.washington.edu ABSTRACT

### PRONUNCIATION MODELLING FOR CONVERSATIONAL SPEECH RECOGNITION: A STATUS REPORT FROM WS97

PRONUNCIATION MODELLING FOR CONVERSATIONAL SPEECH RECOGNITION: A STATUS REPORT FROM WS97 B. Byrne, M. Finke, S. Khudanpur, J. McDonough, H. Nock, M. Riley, M. Saraclar, C. Wooters, G. Zavaliagkos (The

### Convolutional Neural Networks for Speech Recognition

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL 22, NO 10, OCTOBER 2014 1533 Convolutional Neural Networks for Speech Recognition Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang,

### Analysis-by-synthesis for source separation and speech recognition

Analysis-by-synthesis for source separation and speech recognition Michael I Mandel mim@mr-pc.org Brooklyn College (CUNY) Joint work with Young Suk Cho and Arun Narayanan (Ohio State) Columbia Neural Network

### Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. John Lafferty Andrew McCallum Fernando Pereira

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data John Lafferty Andrew McCallum Fernando Pereira Goal: Sequence segmentation and labeling Computational biology Computational

### Edinburgh Research Explorer

Edinburgh Research Explorer Articulatory Feature Classifiers Trained on 2000 hours of Telephone Speech Citation for published version: Frankel, J, Magimai-Doss, M, King, S, Livescu, K & Ãetin, Ã 2007,

### Recurrent Neural Networks for Signal Denoising in Robust ASR

Recurrent Neural Networks for Signal Denoising in Robust ASR Andrew L. Maas 1, Quoc V. Le 1, Tyler M. O Neil 1, Oriol Vinyals 2, Patrick Nguyen 3, Andrew Y. Ng 1 1 Computer Science Department, Stanford

### I D I A P R E S E A R C H R E P O R T. July submitted for publication

R E S E A R C H R E P O R T I D I A P Analysis of Confusion Matrix to Combine Evidence for Phoneme Recognition S. R. Mahadeva Prasanna a B. Yegnanarayana b Joel Praveen Pinto and Hynek Hermansky c d IDIAP

### The ICSI RT-09 Speaker Diarization System. David Sun

The ICSI RT-09 Speaker Diarization System David Sun Papers The ICSI RT-09 Speaker Diarization System, Gerald Friedland, Adam Janin, David Imseng, Xavier Anguera, Luke Gottlieb, Marijn Huijbregts, Mary

### Discriminative training of a phoneme confusion model for a dynamic lexicon in ASR

Discriminative training of a phoneme confusion model for a dynamic lexicon in ASR Penny Karanasou 1,2, François Yvon 1,2, Thomas Lavergne 1,2, Lori Lamel 1 1 LIMSI/CNRS, B.P. 133, 91403 Orsay, France 2

### Speech and Language Processing. Chapter 9 of SLP Automatic Speech Recognition (I)

Speech and Language Processing Chapter 9 of SLP Automatic Speech Recognition (I) Outline for ASR ASR Architecture The Noisy Channel Model Five easy pieces of an ASR system 1) Language Model 2) Lexicon/Pronunciation

### This lecture. Automatic speech recognition (ASR) Applying HMMs to ASR, Practical aspects of ASR, and Levenshtein distance. CSC401/2511 Spring

This lecture Automatic speech recognition (ASR) Applying HMMs to ASR, Practical aspects of ASR, and Levenshtein distance. CSC401/2511 Spring 2017 2 Consider what we want speech to do Buy ticket... AC490...

### Lexicon and Language Model

Lexicon and Language Model Steve Renals Automatic Speech Recognition ASR Lecture 10 15 February 2018 ASR Lecture 10 Lexicon and Language Model 1 Three levels of model Acoustic model P(X Q) Probability

### Large-Scale Speech Recognition

Large-Scale Speech Recognition Madiha Mubin Chinyere Nwabugwu Tyler O Neil Abstract: This project involved getting a sophisticated speech transcription system, SCARF, running on a large corpus of data

### Performance improvement in automatic evaluation system of English pronunciation by using various normalization methods

Proceedings of 20 th International Congress on Acoustics, ICA 2010 23-27 August 2010, Sydney, Australia Performance improvement in automatic evaluation system of English pronunciation by using various

### Joint Sequence Training of Phone and Grapheme Acoustic Model based on Multi-task Learning Deep Neural Networks

Joint Sequence Training of Phone and Grapheme Acoustic Model based on Multi-task Learning Deep Neural Networks Dongpeng Chen 1, Brian Mak 1, Sunil Sivadas 2 1 Department of Computer Science & Engineering

### Robust DNN-based VAD augmented with phone entropy based rejection of background speech

INTERSPEECH 2016 September 8 12, 2016, San Francisco, USA Robust DNN-based VAD augmented with phone entropy based rejection of background speech Yuya Fujita 1, Ken-ichi Iso 1 1 Yahoo Japan Corporation

### FLAT START TRAINING OF CD-CTC-SMBR LSTM RNN ACOUSTIC MODELS. Kanishka Rao, Andrew Senior, Haşim Sak. Google

FLAT START TRAINING OF CD-CTC-SMBR LSTM RNN ACOUSTIC MODELS Kanishka Rao, Andrew Senior, Haşim Sak Google {kanishkarao,andrewsenior,hasim}@google.com ABSTRACT We present a recipe for training acoustic

### Deep Neural Networks for Acoustic Modelling. Bajibabu Bollepalli Hieu Nguyen Rakshith Shetty Pieter Smit (Mentor)

Deep Neural Networks for Acoustic Modelling Bajibabu Bollepalli Hieu Nguyen Rakshith Shetty Pieter Smit (Mentor) Introduction Automatic speech recognition Speech signal Feature Extraction Acoustic Modelling

### Improving Language Models by Learning from Speech Recognition Errors in a Reading Tutor that Listens

Improving Language Models by Learning from Speech Recognition Errors in a Reading Tutor that Listens Satanjeev Banerjee, Jack Mostow, Joseph Beck, and Wilson Tam Project Listen 1, School of Computer Science

### MINIMUM RISK ACOUSTIC CLUSTERING FOR MULTILINGUAL ACOUSTIC MODEL COMBINATION

MINIMUM RISK ACOUSTIC CLUSTERING FOR MULTILINGUAL ACOUSTIC MODEL COMBINATION Dimitra Vergyri Stavros Tsakalidis William Byrne Center for Language and Speech Processing Johns Hopkins University, Baltimore,

### Automatic Speech Recognition (CS753)

Automatic Speech Recognition (CS753) Lecture 20: Pronunciation Modeling Instructor: Preethi Jyothi Oct 16, 2017 Pronunciation Dictionary/Lexicon Pronunciation model/dictionary/lexicon: Lists one or more

### A New Perspective on Combining GMM and DNN Frameworks for Speaker Adaptation

A New Perspective on Combining GMM and DNN Frameworks for Speaker Adaptation SLSP-2016 October 11-12 Natalia Tomashenko 1,2,3 natalia.tomashenko@univ-lemans.fr Yuri Khokhlov 3 khokhlov@speechpro.com Yannick

### Improved recognition by combining different features and different systems

Improved recognition by combining different features and different systems Daniel P.W. Ellis International Computer Science Institute 1947 Center St. #600, Berkeley CA 94704-1198 (510) 666-2940

### Pitch Synchronous Spectral Analysis for a Pitch Dependent Recognition of Voiced Phonemes - PISAR

Pitch Synchronous Spectral Analysis for a Pitch Dependent Recognition of Voiced Phonemes - PISAR Hans-Günter Hirsch Institute for Pattern Recognition, Niederrhein University of Applied Sciences, Krefeld,

### Speech Representation Models for Speech Synthesis and Multimodal Speech Recognition. Felix Sun

Speech Representation Models for Speech Synthesis and Multimodal Speech Recognition by Felix Sun Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the

### Deep Neural Network Training Emphasizing Central Frames

INTERSPEECH 2015 Deep Neural Network Training Emphasizing Central Frames Gakuto Kurata 1, Daniel Willett 2 1 IBM Research 2 Nuance Communications gakuto@jp.ibm.com, Daniel.Willett@nuance.com Abstract It

### Making a Speech Recognizer Tolerate Non-native Speech. through Gaussian Mixture Merging

Proceedings of InSTIL/ICALL2004 NLP and Speech Technologies in Advanced Language Learning Systems Venice 17-19 June, 2004 Making a Speech Recognizer Tolerate Non-native Speech through Gaussian Mixture