Zusammenfassung Vorlesung Mensch-Maschine Kommunikation 19. Juli 2012 Tanja Schultz

Similar documents
Speech Recognition at ICSI: Broadcast News and beyond

AUTOMATIC DETECTION OF PROLONGED FRICATIVE PHONEMES WITH THE HIDDEN MARKOV MODELS APPROACH 1. INTRODUCTION

Speech Recognition using Acoustic Landmarks and Binary Phonetic Feature Classifiers

Learning Methods in Multilingual Speech Recognition

Lecture 9: Speech Recognition

A Comparison of DHMM and DTW for Isolated Digits Recognition System of Arabic Language

Design Of An Automatic Speaker Recognition System Using MFCC, Vector Quantization And LBG Algorithm

Large vocabulary off-line handwriting recognition: A survey

On the Formation of Phoneme Categories in DNN Acoustic Models

BAUM-WELCH TRAINING FOR SEGMENT-BASED SPEECH RECOGNITION. Han Shu, I. Lee Hetherington, and James Glass

Human Emotion Recognition From Speech

The NICT/ATR speech synthesis system for the Blizzard Challenge 2008

Speech Segmentation Using Probabilistic Phonetic Feature Hierarchy and Support Vector Machines

Analysis of Speech Recognition Models for Real Time Captioning and Post Lecture Transcription

Speaker recognition using universal background model on YOHO database

Modeling function word errors in DNN-HMM based LVCSR systems

A study of speaker adaptation for DNN-based speech synthesis

Analysis of Emotion Recognition System through Speech Signal Using KNN & GMM Classifier

Automatic Pronunciation Checker

Modeling function word errors in DNN-HMM based LVCSR systems

Segmental Conditional Random Fields with Deep Neural Networks as Acoustic Models for First-Pass Word Recognition

Investigation on Mandarin Broadcast News Speech Recognition

Learning Structural Correspondences Across Different Linguistic Domains with Synchronous Neural Language Models

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur

Lecture 1: Machine Learning Basics

Speech Emotion Recognition Using Support Vector Machine

have to be modeled) or isolated words. Output of the system is a grapheme-tophoneme conversion system which takes as its input the spelling of words,

ADVANCES IN DEEP NEURAL NETWORK APPROACHES TO SPEAKER RECOGNITION

Role of Pausing in Text-to-Speech Synthesis for Simultaneous Interpretation

Unsupervised Acoustic Model Training for Simultaneous Lecture Translation in Incremental and Batch Mode

Specification and Evaluation of Machine Translation Toy Systems - Criteria for laboratory assignments

Likelihood-Maximizing Beamforming for Robust Hands-Free Speech Recognition

Vimala.C Project Fellow, Department of Computer Science Avinashilingam Institute for Home Science and Higher Education and Women Coimbatore, India

Switchboard Language Model Improvement with Conversational Data from Gigaword

Calibration of Confidence Measures in Speech Recognition

OCR for Arabic using SIFT Descriptors With Online Failure Prediction

A New Perspective on Combining GMM and DNN Frameworks for Speaker Adaptation

CS 598 Natural Language Processing

Speech Recognition by Indexing and Sequencing

Natural Language Processing. George Konidaris

Class-Discriminative Weighted Distortion Measure for VQ-Based Speaker Identification

Phonetic- and Speaker-Discriminant Features for Speaker Recognition. Research Project

Enhancing Unlexicalized Parsing Performance using a Wide Coverage Lexicon, Fuzzy Tag-set Mapping, and EM-HMM-based Lexical Probabilities

Eli Yamamoto, Satoshi Nakamura, Kiyohiro Shikano. Graduate School of Information Science, Nara Institute of Science & Technology

Unvoiced Landmark Detection for Segment-based Mandarin Continuous Speech Recognition

2/15/13. POS Tagging Problem. Part-of-Speech Tagging. Example English Part-of-Speech Tagsets. More Details of the Problem. Typical Problem Cases

Applications of memory-based natural language processing

International Journal of Computational Intelligence and Informatics, Vol. 1 : No. 4, January - March 2012

Speaker Identification by Comparison of Smart Methods. Abstract

Software Maintenance

PREDICTING SPEECH RECOGNITION CONFIDENCE USING DEEP LEARNING WITH WORD IDENTITY AND SCORE FEATURES

Word Segmentation of Off-line Handwritten Documents

English Language and Applied Linguistics. Module Descriptions 2017/18

A Neural Network GUI Tested on Text-To-Phoneme Mapping

Books Effective Literacy Y5-8 Learning Through Talk Y4-8 Switch onto Spelling Spelling Under Scrutiny

Lecture 10: Reinforcement Learning

Semi-Supervised GMM and DNN Acoustic Model Training with Multi-system Combination and Confidence Re-calibration

On Developing Acoustic Models Using HTK. M.A. Spaans BSc.

A Minimalist Approach to Code-Switching. In the field of linguistics, the topic of bilingualism is a broad one. There are many

Florida Reading Endorsement Alignment Matrix Competency 1

CEFR Overall Illustrative English Proficiency Scales

ESSLLI 2010: Resource-light Morpho-syntactic Analysis of Highly

Probabilistic Latent Semantic Analysis

Speech Translation for Triage of Emergency Phonecalls in Minority Languages

STUDIES WITH FABRICATED SWITCHBOARD DATA: EXPLORING SOURCES OF MODEL-DATA MISMATCH

WHEN THERE IS A mismatch between the acoustic

Edinburgh Research Explorer

Language Acquisition Fall 2010/Winter Lexical Categories. Afra Alishahi, Heiner Drenhaus

A NOVEL SCHEME FOR SPEAKER RECOGNITION USING A PHONETICALLY-AWARE DEEP NEURAL NETWORK. Yun Lei Nicolas Scheffer Luciana Ferrer Mitchell McLaren

LEARNING A SEMANTIC PARSER FROM SPOKEN UTTERANCES. Judith Gaspers and Philipp Cimiano

Parsing of part-of-speech tagged Assamese Texts

Effect of Word Complexity on L2 Vocabulary Learning

Semi-supervised methods of text processing, and an application to medical concept extraction. Yacine Jernite Text-as-Data series September 17.

elearning OVERVIEW GFA Consulting Group GmbH 1

On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC

Rachel E. Baker, Ann R. Bradlow. Northwestern University, Evanston, IL, USA

Using Articulatory Features and Inferred Phonological Segments in Zero Resource Speech Processing

Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling

Spoken Language Parsing Using Phrase-Level Grammars and Trainable Classifiers

QuickStroke: An Incremental On-line Chinese Handwriting Recognition System

Language Model and Grammar Extraction Variation in Machine Translation

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 3, MARCH

A Case Study: News Classification Based on Term Frequency

Autoregressive product of multi-frame predictions can improve the accuracy of hybrid models

STA 225: Introductory Statistics (CT)

Beyond the Pipeline: Discrete Optimization in NLP

Rule discovery in Web-based educational systems using Grammar-Based Genetic Programming

Segregation of Unvoiced Speech from Nonspeech Interference

Program Matrix - Reading English 6-12 (DOE Code 398) University of Florida. Reading

Voice conversion through vector quantization

Mandarin Lexical Tone Recognition: The Gating Paradigm

Lip reading: Japanese vowel recognition by tracking temporal changes of lip shape

An Online Handwriting Recognition System For Turkish

International Journal of Advanced Networking Applications (IJANA) ISSN No. :

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS

ENGBG1 ENGBL1 Campus Linguistics. Meeting 2. Chapter 7 (Morphology) and chapter 9 (Syntax) Pia Sundqvist

Evolutive Neural Net Fuzzy Filtering: Basic Description

Corpus Linguistics (L615)

Robust Speech Recognition using DNN-HMM Acoustic Model Combining Noise-aware training with Spectral Subtraction

CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2

Transcription:

Zusammenfassung - 1 Zusammenfassung Vorlesung Mensch-Maschine Kommunikation 19. Juli 2012 Tanja Schultz

Zusammenfassung - 2 Evaluationsergebnisse

Zusammenfassung - 3 Lehrveranstaltung

Zusammenfassung - 4 Lehrveranstaltung

Zusammenfassung - 5

Zusammenfassung - 6 Lehrveranstaltung

Zusammenfassung - 7 DozentInnen

Zusammenfassung - 8 Studium

Zusammenfassung - 9 Monitoring

Zusammenfassung - 10 Gut

Zusammenfassung - 11 Schlecht

Zusammenfassung - 12 Überblick und Zusammenfassung

Zusammenfassung - 13 Spoken Language Systems Input: Speech Hello hi /h//ai/ you /j/u/ we /w//i/ hi you you are I am AM Lex LM NLP / MT TTS Output: Speech & Text Speech Text Meaning Text Speech

Zusammenfassung - 14 Spoken Language Systems A spoken language system needs to have both: Speech Recognition and Speech Synthesis But this is NOT sufficient in order to build a useful spoken language system It also requires an Understanding component Understanding component may include a Dialog component to manage interactions with the user a translation component to transfer between languages Domain knowledge is required to guide the interpretation of speech and to determine the appropriate action All these components have significant challenges such as: + Robustness + Flexibility + Ease of Integration + Engineering Efficiency

Zusammenfassung - 15 Research areas and Foundations Spoken language processing is a diverse field that relies on knowledge of language at the levels of: signal processing acoustics phonology phonetics syntax semantics pragmatics and discourse Foundations of spoken language processing lie in the field of: computer science electrical engineering linguistics psychology

Zusammenfassung - 16 Mensch - Maschine Kommunikation Database Application Discourse Analysis Dialog Manager Dialog Strategy Speech Recognition Sentence Interpretation Response Generation Synthesis Speech Text Meaning Text Speech

Zusammenfassung - 17 Automatic Speech Recognition Input Speech hi??? /h//ai/ you /j/u/ we /w//i/ hi you you are I am AM Lex LM Output Text Hello world

Zusammenfassung - 18 Automatic Speech Recognition The purpose of Signal Preprocessing is: 1) Signal Digitalization (Quantization and Sampling) represent an analog signal in an appropriate form to be processed by the computer 2) Digital Signal Preprocessing (Feature Extraction) Extract features that are suitable for recognition process Input Speech Signal Pre- Processing hi??? /h//ai/ you /j/u/ we /w//i/ hi you you are I am AM Lex LM Output Text Hello world

Zusammenfassung - 19 Representation of Speech Definition: Digital representation of speech Represent speech as a sequences of numbers (as a prerequisite for automatic processing using computers) 1) Direct representation of speech waveform: represent speech waveform as accurate as possible so that an acoustic signal can be reconstructed 2) Parametric representation Represent a set of properties/parameters wrt a certain model Decide the targeted application first: Speech coding Speech synthesis Speech recognition Classical paper: Schafer/Rabiner in Waibel/Lee (paper online)

Zusammenfassung - 20 Quantization & Sampling

Zusammenfassung - 21 Quantization of Signals Given a discrete signal f[i] to be quantized into q[i] Assume that f is between f min and f max Partition y-axis into a fixed number n of (equally sized) intervals Usually n=2 b, in ASR typically b=16 > n=65536 THEN: q[i] can only have values that are centers of the intervals Quantization: assign q[i] the center of the interval in which lies f[i] Quantization makes errors, i.e. adds noise to the signal f[i]=q[i]+e[i] The average quantization error e[i] is (f max -f min )/(2n) Define signal to noise ratio SNR[dB] = power(f[i]) / power(e[i])

Zusammenfassung - 22 The Aliasing Effect Nyquist or sampling theorem: When a f l -band-limited signal is sampled with a sampling rate of at least 2f l then the signal can be exactly reproduced from the samples When the sampling rate is too low, the samples can contain "incorrect" frequencies: Prevention: increase sampling rate anti-aliasing filter (restrict signal bandwith)

Zusammenfassung - 23 Feature Extraction WHY Capture important phonetic information in speech Computational efficiency, Efficiency in storage requirements Optimize generalization WHAT It is hard to infer much from time domain waveform Human hearing is based on frequency analysis Use of frequency analysis simplifies signal processing Use of frequency analysis facilitates understanding

Zusammenfassung - 24 Digital Signal Processing Nature of Speech: Sound Formants Acoustic Features Describe and explain feature extraction: Sampling, Sampling theorem, aliasing Continuous-time and Discrete-time Fourier Transform Short-time Fourier Analysis Effect of windowing Z-Transform (generalization of DTFT) Poles, zeros give insight to frequency response of linear system Features for speech recognition Cepstral coefficients Mel-frequency cepstral coefficients (MFCC)

Zusammenfassung - 25 Automatic Speech Recognition Two sessions Digital Signal Processing Input Speech Signal Pre- Processing hi??? /h//ai/ you /j/u/ we /w//i/ hi you you are I am AM Lex LM Output Text Hello world

Zusammenfassung - 26 Fundamental Equation of Speech Recognition: Observe a sequence of feature vectors X Find the most likely word sequence W arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Input Speech Signal Pre- Processing hi??? /h//ai/ you /j/u/ we /w//i/ hi you you are I am AM Lex LM Output Text Hello world

Zusammenfassung - 27 Speech Recognition arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Input Speech Signal Pre- Processing p(x W) Acoustic Model Output Text Hello world

Zusammenfassung - 28 Speech Recognition arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Input Speech Signal Pre- Processing p(x W) Acoustic Model P(W) Language Model Output Text Hello world

Zusammenfassung - 29 Speech Recognition Search how to efficiently try all W arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Input Speech Signal Pre- Processing p(x W) Acoustic Model P(W) Language Model Output Text Hello world

Zusammenfassung - 30 Automatic Speech Recognition Input Speech Signal Pre- Processing p(x W) P(W) Acoustic Model Output Text Hello world

Zusammenfassung - 31 Automatic Speech Recognition Purpose of Acoustic Model: Given W, what is the likelihood to see feature vector(s) X we need a representation for W in terms of feature vectors Usually a two-part representation / modeling: pronunciation dictionary: describe W as concatenation of phones Phones models that explain phones in terms of feature vectors p(x W) Input Speech Pre- Processing Acoustic Model + Pronunciation Dict I /i/ you /j/ /u/ we /v/ /e/ Output Text Hello world

Zusammenfassung - 32 Why breaking down words into phones Need collection of reference patterns for each word High computational effort (esp. for large vocabularies), proportional to vocabulary size Large vocabulary also means: need huge amount of training data Difficult to train suitable references (or sets of references) Impossible to recognize untrained words Replace whole words by suitable sub units Poor performance when the environment changes Works only well for speaker-dependent recognition (variations) Unsuitable where speaker is unknown and no training is feasible Unsuitable for continuous speech (combinatorial explosion) Difficult to train/recognize subword units Replace the pattern approach by a better modeling process

Zusammenfassung - 33 Automatic Speech Recognition p(x W) P(W) Input Speech Signal Pre- Processing Acoustic Model Output Text Hello world

Zusammenfassung - 34 Speech Production seen as Stochastic Process The same word / phoneme sounds different every time it is uttered Regard words / phonemes as states of a speech production process In a given state we can observe different acoustic sounds Not all sounds are possible / likely in every state We say: in a given state the speech process "emits" sounds according to some probability distribution The production process makes transitions from one state to another Not all transitions are possible, they have different probabilities When we specify the probabilities for sound-emissions (emission probabilities) and for the state transitions, we call this a model.

Zusammenfassung - 35 Generating an Observation of Speech Features Vectors x 1,x 2,,x T The term "hidden" comes from observing observations and drawing conclusions without knowing the hidden sequence of states

Zusammenfassung - 36 Formal Definition of Hidden Markov Models A Hidden Markov Model is a five-tuple consisting of: S The set of States S={s 1,s 2,...,s n } A B V The initial probability distribution (s i ) = probabilty of s i being the first state of a state sequence The matrix of state transition probabilities: A=(a ij ) where a ij is the probability of state s j following s i The set of emission probability distributions/densities, B={b 1,b 2,...,b n } where b i (x) is the probabiltiy of observing x when the system is in state s i The observable feature space can be discrete: V={x 1,x 2,...,x v }, or continuous V=R d

Zusammenfassung - 37 Three Main Problems of Hidden Markov Models The evaluation problem: given an HMM and an observation x 1,x 2,...,x T, compute the probability of the observation p(x 1,x 2,...,x T ) The decoding problem: given an HMM and an observation x 1,x 2,...,x T, compute the most likely state sequence s q1,s q2,...,s qt, i.e. argmax q1,..,qt p(q 1,..,q T x 1,x 2,...,x T, ) The learning / optimization problem: given an HMM and an observation x 1,x 2,...,x T, find an HMM such that p(x 1,x 2,...,x T ) > p(x 1,x 2,...,x T )

Zusammenfassung - 38 Hidden Markov Models and their Recognizers Problems with Pattern Matching Speech Production seen as Stochastic Process Examples for HMMs Typical HMM-Topologies Formal Definition of Hidden Markov Models Some Properties of Hidden Markov Models Three Main Problems of Hidden Markov Models The Evaluation Problem: The Forward Algorithm The Decoding Problem: The Viterbi Algorithm The Learning/Optimization Problem The Forward-Backward Algorithm

Zusammenfassung - 39 Hidden Markov Models in ASR States that correspond to the same acoustic phaenomenon share the same "acoustic model Training data is better used In this HMM: b 1 =b 7 =b g-b Emission prob parameters are estimated more robustly Save computation time: (don't evaluate b(..) for every s i )

Zusammenfassung - 40 From the Sentence to the Sentence-HMM Generate word lattice of possible word sequences: Generate phoneme lattice of possible pronunciations: Generate state lattice (HMM) of possible state sequences:

Zusammenfassung - 41 Context Dependent Acoustic Modeling Consider the Pronunciations of TRUE, TRAIN, TABLE, and TELL. Most common lexicon entries are: Notice that the actual pronunciation sounds a bit like: TRUE TRAIN TABLE TELL TRUE TRAIN TABLE TELL T R UW T R EY N T EY B L T EH L CH R UW CH R EY N T HH EY B L T HH EH L Statement: The phoneme T sounds different depending on whether the following phoneme is an R or a vowel.

Zusammenfassung - 42 Context Dependent Acoustic Modeling First idea: use actual pronunciations in the lexicon: i.e. CH R UW instead of T R UW. Problem: The CH in TRUE does sound different from the CH in CHURCH. Second idea: Introduce new acoustic units such that the lexicon looks like: TRUE TRAIN T(R) R UW T(R) R EY N TABLE TELL T(vowel) EY B L T(vowel) EH L i.e. use context dependent models of the phoneme T

Zusammenfassung - 43 From Sentence to Context Dependent HMM A context independent HMM for the sentence "HELLO WORLD" could look like this: Making the phoneme H dependend on it successor, out of we make Typical improvements of speech recognizers when introducing context dependence: 30% - 50% fewer errors.

Zusammenfassung - 44 Acoustic Modeling Discrete vs Continuous HMMs Parameter Tying Codebook Sizes Pronunciation Variants Context Dependent Acoustic Modeling Speech Units Clustering of Context Bottom-Up vs. Top-Down Clustering Distances Between Model Clusters Clustering with Decision Trees

Zusammenfassung - 45 Automatic Speech Recognition Two lectures on Hidden Markov Modeling Two lectures on Acoustic Modeling (CI, CD) One lecture on Pronunciation Modeling, Variants, Adaptation p(x W) P(W) Input Speech Signal Pre- Processing Acoustic Model + Pronunciation Dict I /i/ you /j/ /u/ we /v/ /e/ Output Text Hello world

Zusammenfassung - 46 Automatic Speech Recognition p(x W) P(W) Signal Pre- Processing Input Speech I /i/ you /j/ /u/ we /v/ /e/ eu sou você é ela é Language Model Output Text Hello world

Zusammenfassung - 47 Motivation Equally important to recognize and understand natural speech: Acoustic pattern matching and knowledge about language Language Knowledge: in SR covered by: Lexical knowledge vocabulary definition vocabulary word pronunciation dictionary Syntax and Semantics, I.e. rules that determine: LM word sequence is grammatically well-formed / Grammar word sequence is meaningful Pragmatics LM structure of extended discourse / Grammar what is likely to be said in particular context / Discourse These different levels of knowledge are tightly integrated!!!

Zusammenfassung - 48 Stochastic Language Models In formal language theory P(W) is regarded either as 1.0 if word sequence W is accepted 0.0 if word sequence W is rejected Inappropriate for spoken language since, grammar has no complete coverage (conversational) spoken language is often ungrammatical Describe P(W) from the probabilistic viewpoint Occurrence of word sequence W is described by a probability P(W) find a good way to accurately estimate P(W) Training problem: reliably estimate probabilities of W Recognition problem: compute probabilities for generating W

Zusammenfassung - 49 What do we expect from Language Models in SR? Improve speech recognizer add another information source Disambiguate homophones find out that "I OWE YOU TOO" is more likely than "EYE O U TWO" Search space reduction when vocabulary is n words, don't consider all n k possible k-word sequences Analysis analyze utterance to understand what has been said disambiguate homonyms (bank: money vs river)

Zusammenfassung - 50 Probabilities of Word Sequences The probability of a word sequence can be decomposed as: P(W) = P(w 1 w 2.. w n ) = P(w 1 ) P(w 2 w 1 ) P(w 3 w 1 w 2 ) P(w n w 1 w 2... w n-1 ) The choice of w n thus depends on the entire history of the input, so when computing P(w history), we have a problem: For a vocabulary of 64,000 words and average sentence lengths of 25 words (typical for Wall Street Journal), we end up with a huge number of possible histories (64,000 25 > 10 120 ). So it is impossible to precompute a special P(w history) for every history. Two possible solutions: compute P(w history) "on the fly" (rarely used, very expensive) replace the history by one out of a limited feasible number of equivalence classes C such that P'(w history) = P(w C(history)) Question: how do we find good equivalence classes C?

Zusammenfassung - 51 Classification of Word Sequence Histories We can use different equivalence classes using information about: Grammatical content (phrases like noun-phrase, etc.) POS = part of speech of previous word(s) (e.g. subject, object,...) Semantic meaning of previous word(s) Context similarity (words that are observed in similar contexts are treated equally, e.g. weekdays, people's names etc.) Apply some kind of automatic clustering (top-down, bottom-up) Classes are simply based on previous words unigram: P'(w k w 1 w 2... w k-1 ) = P(w k ) bigram: P'(w k w 1 w 2... w k-1 ) = P(w k w k-1 ) trigram: P'(w k w 1 w 2... w k-1 ) = P(w k w k-2 w k-1 ) n-gram: P'(w k w 1 w 2... w k-1 ) = P(w k w k-(n-1) w k-n-2... w k-1 )

Zusammenfassung - 52 Estimation of N-grams The standard approach to estimate P(w history) is to use a large amount of training corpus (There's no data like more data) determine the frequency with which the word w occurs given the history simply count how often the word sequence history w occurs in the text normalize the count by the number of times history occurs Count(history w) P(w history) = Count(history) Example: Let our training corpus consists of 3 sentences, use bigram model John read her book. I read a different book. John read a book by Mulan. P(John <s>) = C(<s>,John) / C(<s>) = 2/3 P(read John) = C(John,read) / C(John) = 2/2 P(a read) = C(read,a) / C(read) = 2/3 P(book a) = C(a,book) / C(a) = 1/2 P(</s> book) = C(book, </s>) / C(book) = 2/3 Now calculate the probability of sentence John read a book. P(John read a book) = P(John <s>) P(read John) P(a read) P(book a) P(</s> book) = 0.148 But what about the sentence Mulan read her book. We don t have P(read Mulan)

Zusammenfassung - 53 Language Modeling Language Modeling in Automatic Speech Recognition Deterministic vs. Stochastic Language Modeling Probabilities of Word Sequences Bigrams and Trigrams: The Bag of Words Experiment Interpolation of Language Model Parameters Parameter Smoothing Measuring the Quality of Language Models, Perplexity Different Kinds of Language Models Cache, Trigger, Multilevel, Interleaved, Morpheme-Based, Context-Free Grammars,... Practical Issues Spontaneous Speech Unknown Words Different Languages

Zusammenfassung - 54 Automatic Speech Recognition Two lectures on Language Modeling p(x W) P(W) Signal Pre- Processing Input Speech I /i/ you /j/ /u/ we /v/ /e/ eu sou você é ela é Language Model Output Text Hello world

Zusammenfassung - 55 Automatic Speech Recognition Search how to efficiently try all W arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Signal Pre- Processing Input Speech p(x W) P(W) Output Text Hello world

Zusammenfassung - 56 Search The entire set of possible sequences of pattern is called the search space Typical search spaces have 1,000 time frames (10sec speech) and 500,000 possible sequences of pattern With an average of 25 words per sentence (e.g. WSJ) and a vocabulary of 64,000 words, more possible word sequences than the universe has atoms! It is not feasible to compute the most likely sequence of words by evaluating the scores of all possible sequences We need an intelligent algorithm that scans the search space and finds the best (or at least a very good) hypothesis This problem is referred to search or decoding

Zusammenfassung - 57 Simplified Decoding Speech Speech features Hypotheses (phonemes) Feature extraction Decision (apply trained classifiers) /h/... /h/ /e/ /l/ /o/ /w/ /o/ /r/ /l/ /d/

Zusammenfassung - 58 Compare Complete Utterances What we had so far: Record a sound signal Compute frequency representation Quantize/classify vectors We now have: A sequence of pattern vectors Want we want: The similiarity between two such sequences => Obviously: The order of vectors is important! vs.

Zusammenfassung - 59 Comparing Complete Utterances Comparing speech vector sequences has to overcome three problems: 1) Speaking rate characterizes speakers (speaker dependent!) if the speaker is speaking faster, we get fewer vectors 2) Changing speaking rate by purpose: e.g. talking to a foreign person 3) Changing speaking rate non-purposely: speaking disfluencies vs. So we have to find a way to decide which vectors to compare to another Impose some constraints (compare every vector to all others is too costly)

Zusammenfassung - 60 Alignment of Vector Sequences First idea to overcome the varying length of Utterances, Problem (2): 1. Normalize their length 2. Make a linear alignment Linear alignment can handle the problem of different speaking rates But: it can not handle the problem of varying speaking rates during the same utterance.

Zusammenfassung - 61 One Example Pattern DTW Review Goal: Identify example pattern that is most similar to unknown input compare patterns of different length Note: all patterns are preprocessed 100 vectors / second of speech DTW: Find alignment between unknown input and the example pattern that minimizes the overall distance Find average vector distance, but which frame-pairs? t 1 t 2 t M t 1 t 2? t N Euclidean Distance Input = unknown pattern

Zusammenfassung - 62 Plan 1: Cut Continuous Speech Into Single Words Write magic algorithm that segments speech into 1-word chunks Run DTW/Viterbi on each chunk BUT: Where are the boundaries???? No reliable segmentation algorithm for detecting word boundaries other than doing recognition itself, due to: Co-articulation between words Hesitations within words Hard decisions lead to accumulating errors Integrated approach works better

Zusammenfassung - 63 Dynamic Programming and Single Word Recognizer Comparing Complete Utterances: Problems Endpoint Detection and Speech Detection Approaches to Alignments of Vector Sequences Time Warping Distance Measure between two Utterances The Minimal Editing Distance Problem Dynamic Programming Utterance Comparison by Dynamic Time Warping Constraints for the DTW-Path The DTW Search space DTW with Beam Search The Principles of Building Speech Recognizers Isolated Word Recognition with Template Matching

Zusammenfassung - 64 Search The Search in Automatic Speech Recognition DTW review pattern based recognition, Optimizations Viterbi review model based recognition, Optimizations Continuous speech recognition Reasons against predicting word boundaries Two level DP One stage DP, Search strategies, stack decoder Optimization: How to waste not too much Computation Time Tree-Search, Pruning, Pruning with Beamsearch Search with LM / Grammar Multi-Pass Searches, Problems and Examples Producing more than one Hypothesis, Problems Speeding up the Search Search with Context-Dependent Models

Zusammenfassung - 65 Plan 3: Depth First Search Well known search strategies: Depth first search vs. breath first search: In speech recognition, this corresponds roughly to: time-asynchronous vs. time-synchronous In time-synchronous search, we expand every hypothesis simultaneously frame by frame. In time-asynchronous search, we expand partial hypotheses that have different durations.

Zusammenfassung - 66 Plan 4: One stage Dynamic Programming Within words: Viterbi Between words: For the first state of each words, consider the best last state of all words as predecessor. This last state or the first state in the current word is predecessor. first state means any state that can be the first of a word last state means any state that has a transition out of the word

Zusammenfassung - 67 HMM for word 1 Viterbi Review (3) Compute state transitions Without transition probabilities: -log (P(x,y))= min ( -log (P(x-1,y)) + d(x,y), -log (P(x-1,y-1)) + d(x,y) ) d(x,y) = log(p(frame x s y )) t 1 t 2 Unknown pattern t N a lot like DTW

Zusammenfassung - 68 Search with vs. without Language Model (2) without grammar with grammar Without grammar: The best predecessor state is the same for all word-initial states expand only the word-final state that has the best score in a frame With grammar: The best predecessor state depends also on the word transition probability/penalty

Zusammenfassung - 69 Search Summary (Part 1+2) The Search in Automatic Speech Recognition DTW review pattern based recognition, Optimizations Viterbi review model based recognition, Optimizations Continuous speech recognition Reasons against predicting word boundaries Two level DP One stage DP, Search strategies, stack decoder Optimization: How to waste not too much Computation Time Tree-Search, Pruning, Pruning with Beamsearch Search with LM / Grammar Multi-Pass Searches, Problems and Examples Producing more than one Hypothesis, Problems Speeding up the Search Search with Context-Dependent Models

Zusammenfassung - 70 Automatic Speech Recognition Search how to efficiently try all W Two lectures on Search arg max W P( W X ) arg max W P( W ) p( X P( X ) W ) Signal Pre- Processing Input Speech p(x W) P(W) Output Text Hello world

Zusammenfassung - 71 Vielen Dank für Ihr Interesse!