Analysis-by-synthesis for source separation and speech recognition

Similar documents
ADVANCES IN DEEP NEURAL NETWORK APPROACHES TO SPEAKER RECOGNITION

Robust Speech Recognition using DNN-HMM Acoustic Model Combining Noise-aware training with Spectral Subtraction

Speech Recognition at ICSI: Broadcast News and beyond

A study of speaker adaptation for DNN-based speech synthesis

Eli Yamamoto, Satoshi Nakamura, Kiyohiro Shikano. Graduate School of Information Science, Nara Institute of Science & Technology

Modeling function word errors in DNN-HMM based LVCSR systems

A New Perspective on Combining GMM and DNN Frameworks for Speaker Adaptation

Speech Emotion Recognition Using Support Vector Machine

WHEN THERE IS A mismatch between the acoustic

Semi-Supervised GMM and DNN Acoustic Model Training with Multi-system Combination and Confidence Re-calibration

Segregation of Unvoiced Speech from Nonspeech Interference

Modeling function word errors in DNN-HMM based LVCSR systems

Calibration of Confidence Measures in Speech Recognition

Likelihood-Maximizing Beamforming for Robust Hands-Free Speech Recognition

The NICT/ATR speech synthesis system for the Blizzard Challenge 2008

Learning Methods in Multilingual Speech Recognition

PREDICTING SPEECH RECOGNITION CONFIDENCE USING DEEP LEARNING WITH WORD IDENTITY AND SCORE FEATURES

Speech Segmentation Using Probabilistic Phonetic Feature Hierarchy and Support Vector Machines

Lecture 1: Machine Learning Basics

Human Emotion Recognition From Speech

A Comparison of DHMM and DTW for Isolated Digits Recognition System of Arabic Language

Noise-Adaptive Perceptual Weighting in the AMR-WB Encoder for Increased Speech Loudness in Adverse Far-End Noise Conditions

Autoregressive product of multi-frame predictions can improve the accuracy of hybrid models

Segmental Conditional Random Fields with Deep Neural Networks as Acoustic Models for First-Pass Word Recognition

Speech Synthesis in Noisy Environment by Enhancing Strength of Excitation and Formant Prominence

Unsupervised Acoustic Model Training for Simultaneous Lecture Translation in Incremental and Batch Mode

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 17, NO. 3, MARCH

AUTOMATIC DETECTION OF PROLONGED FRICATIVE PHONEMES WITH THE HIDDEN MARKOV MODELS APPROACH 1. INTRODUCTION

Learning Structural Correspondences Across Different Linguistic Domains with Synchronous Neural Language Models

On the Formation of Phoneme Categories in DNN Acoustic Models

A NOVEL SCHEME FOR SPEAKER RECOGNITION USING A PHONETICALLY-AWARE DEEP NEURAL NETWORK. Yun Lei Nicolas Scheffer Luciana Ferrer Mitchell McLaren

Analysis of Emotion Recognition System through Speech Signal Using KNN & GMM Classifier

STUDIES WITH FABRICATED SWITCHBOARD DATA: EXPLORING SOURCES OF MODEL-DATA MISMATCH

have to be modeled) or isolated words. Output of the system is a grapheme-tophoneme conversion system which takes as its input the spelling of words,

Word Segmentation of Off-line Handwritten Documents

INVESTIGATION OF UNSUPERVISED ADAPTATION OF DNN ACOUSTIC MODELS WITH FILTER BANK INPUT

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur

Deep Neural Network Language Models

UNIDIRECTIONAL LONG SHORT-TERM MEMORY RECURRENT NEURAL NETWORK WITH RECURRENT OUTPUT LAYER FOR LOW-LATENCY SPEECH SYNTHESIS. Heiga Zen, Haşim Sak

BUILDING CONTEXT-DEPENDENT DNN ACOUSTIC MODELS USING KULLBACK-LEIBLER DIVERGENCE-BASED STATE TYING

Python Machine Learning

Design Of An Automatic Speaker Recognition System Using MFCC, Vector Quantization And LBG Algorithm

International Journal of Computational Intelligence and Informatics, Vol. 1 : No. 4, January - March 2012

CSL465/603 - Machine Learning

DIRECT ADAPTATION OF HYBRID DNN/HMM MODEL FOR FAST SPEAKER ADAPTATION IN LVCSR BASED ON SPEAKER CODE

Improvements to the Pruning Behavior of DNN Acoustic Models

A Neural Network GUI Tested on Text-To-Phoneme Mapping

SEMI-SUPERVISED ENSEMBLE DNN ACOUSTIC MODEL TRAINING

Investigation on Mandarin Broadcast News Speech Recognition

Analysis of Speech Recognition Models for Real Time Captioning and Post Lecture Transcription

BAUM-WELCH TRAINING FOR SEGMENT-BASED SPEECH RECOGNITION. Han Shu, I. Lee Hetherington, and James Glass

The Good Judgment Project: A large scale test of different methods of combining expert predictions

OCR for Arabic using SIFT Descriptors With Online Failure Prediction

arxiv: v1 [cs.lg] 7 Apr 2015

THE world surrounding us involves multiple modalities

Phonetic- and Speaker-Discriminant Features for Speaker Recognition. Research Project

QuickStroke: An Incremental On-line Chinese Handwriting Recognition System

Speech Recognition using Acoustic Landmarks and Binary Phonetic Feature Classifiers

Evolution of Symbolisation in Chimpanzees and Neural Nets

Radius STEM Readiness TM

ACOUSTIC EVENT DETECTION IN REAL LIFE RECORDINGS

Automatic Pronunciation Checker

INPE São José dos Campos

Malicious User Suppression for Cooperative Spectrum Sensing in Cognitive Radio Networks using Dixon s Outlier Detection Method

Author's personal copy

The IRISA Text-To-Speech System for the Blizzard Challenge 2017

UTD-CRSS Systems for 2012 NIST Speaker Recognition Evaluation

Distributed Learning of Multilingual DNN Feature Extractors using GPUs

LOW-RANK AND SPARSE SOFT TARGETS TO LEARN BETTER DNN ACOUSTIC MODELS

Lip reading: Japanese vowel recognition by tracking temporal changes of lip shape

Master s Programme in Computer, Communication and Information Sciences, Study guide , ELEC Majors

An Online Handwriting Recognition System For Turkish

Unsupervised Learning of Word Semantic Embedding using the Deep Structured Semantic Model

Lecture 9: Speech Recognition

Generative models and adversarial training

Body-Conducted Speech Recognition and its Application to Speech Support System

Digital Signal Processing: Speaker Recognition Final Report (Complete Version)

Class-Discriminative Weighted Distortion Measure for VQ-Based Speaker Identification

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS

A Reinforcement Learning Variant for Control Scheduling

(Sub)Gradient Descent

Edinburgh Research Explorer

Using Articulatory Features and Inferred Phonological Segments in Zero Resource Speech Processing

Unvoiced Landmark Detection for Segment-based Mandarin Continuous Speech Recognition

A comparison of spectral smoothing methods for segment concatenation based speech synthesis

DNN ACOUSTIC MODELING WITH MODULAR MULTI-LINGUAL FEATURE EXTRACTION NETWORKS

Model Ensemble for Click Prediction in Bing Search Ads

International Journal of Advanced Networking Applications (IJANA) ISSN No. :

Voice conversion through vector quantization

Attributed Social Network Embedding

Voiceless Stop Consonant Modelling and Synthesis Framework Based on MISO Dynamic System

Course Outline. Course Grading. Where to go for help. Academic Integrity. EE-589 Introduction to Neural Networks NN 1 EE

Learning Methods for Fuzzy Systems

Non intrusive multi-biometrics on a mobile device: a comparison of fusion techniques

Atypical Prosodic Structure as an Indicator of Reading Level and Text Difficulty

DOMAIN MISMATCH COMPENSATION FOR SPEAKER RECOGNITION USING A LIBRARY OF WHITENERS. Elliot Singer and Douglas Reynolds

/$ IEEE

Cross Language Information Retrieval

Rule Learning With Negation: Issues Regarding Effectiveness

Speaker recognition using universal background model on YOHO database

Transcription:

Analysis-by-synthesis for source separation and speech recognition Michael I Mandel mim@mr-pc.org Brooklyn College (CUNY) Joint work with Young Suk Cho and Arun Narayanan (Ohio State) Columbia Neural Network Seminar Series September 8, 2015 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 1 / 73

Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 2 / 73

Motivation: need for noise robustness Outline 1 Motivation: need for noise robustness Need for better mobile voice quality Need for noise robust automatic speech recognition (ASR) Main challenge 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 3 / 73

Motivation: need for noise robustness Need for better mobile voice quality Outline 1 Motivation: need for noise robustness Need for better mobile voice quality Need for noise robust automatic speech recognition (ASR) Main challenge 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 4 / 73

Motivation: need for noise robustness Need for better mobile voice quality Need for better mobile voice quality There are now more mobile devices than humans on earth 1 But recording conditions for these devices leave much to be desired Can we recover high quality speech from noisy & degraded recordings? 1 http://www.independent.co.uk/life-style/gadgets-and-tech/news/ there-are-officially-more-mobile-devices-than-people-in-the-world-9780518.html Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 5 / 73

Motivation: need for noise robustness Need for better mobile voice quality Why mobile voice quality stinks 2 2 Je Hecht. Why mobile voice quality still stinks and how to fix it. IEEE Spectrum, September2014 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 6 / 73

Motivation: need for noise robustness Need for better mobile voice quality Why mobile voice quality stinks 2 2 Je Hecht. Why mobile voice quality still stinks and how to fix it. IEEE Spectrum, September2014 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 6 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) Outline 1 Motivation: need for noise robustness Need for better mobile voice quality Need for noise robust automatic speech recognition (ASR) Main challenge 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 7 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) Conversational mobile software agents Source: Tom Vanleenhove Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 8 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) Conversational mobile software agents need to work in Source: Flickr user rickihuang Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 9 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) Conversational mobile software agents need to work in Source: Flickr user retorta net Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 9 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) Conversational mobile software agents need to work in Source: Flickr user Brian Indrelunas Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 9 / 73

Motivation: need for noise robustness Need for noise robust automatic speech recognition (ASR) But automatic speech recognition doesn t work there 3 3 Amit Juneja. A comparison of automatic and human speech recognition in null grammar. The Journal of the Acoustical Society of America, 131(3):EL256 EL261,February2012 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 10 / 73

Motivation: need for noise robustness Main challenge Outline 1 Motivation: need for noise robustness Need for better mobile voice quality Need for noise robust automatic speech recognition (ASR) Main challenge 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 11 / 73

Motivation: need for noise robustness Main challenge Main challenge Speech is a rich signal, it requires rich models Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 12 / 73

Motivation: need for noise robustness Main challenge Main challenge Speech is a rich signal, it requires rich models Synthesis models are rich enough to represent almost all speech Non-parametric synthesis models for high quality DNN as non-linear distance function Parametric synthesis models for e cient representation e cient gradient-based optimization of input (not model) Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 12 / 73

Non-parametric synthesis for speech enhancement Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 13 / 73

Non-parametric synthesis for speech enhancement Overview Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 14 / 73

Non-parametric synthesis for speech enhancement Overview Concatenative resynthesis for speech enhancement 4,5 Standard approaches try to modify noisy recordings We instead resynthesize a clean version of the same speech Should produce infinite suppression and high speech quality 4 Michael I Mandel, Young-Suk Cho, and Yuxuan Wang. Learning a concatenative resynthesis system for noise suppression. In Proc. IEEE GlobalSIP, 2014 5 Michael I Mandel and Young Suk Cho. Audio super-resolution using concatenative resynthesis. In Proc. IEEE WASPAA, 2015. To appear Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 15 / 73

Non-parametric synthesis for speech enhancement Overview Motivating example Your phone records your voice in quiet, close-talk conditions Uses those recordings to replace your voice in noisy, far-talk conditions Resynthesizes your speech from previous high-quality recordings Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 16 / 73

Non-parametric synthesis for speech enhancement Overview Concatenative resynthesis Use a large dictionary of 200 ms chunks of audio Learn DNN-based a nity between dictionary & mixture chunks Perform concatenative synthesis of signal from dictionary General robust supervised nonlinear signal mapping framework Task Map from To Noise suppression Noisy Clean Audio super-resolution Reverberated, compressed Clean Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 17 / 73

Non-parametric synthesis for speech enhancement Deep neural network as nonlinear distance function Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 18 / 73

Non-parametric synthesis for speech enhancement Deep neural network as nonlinear distance function Deep neural network as nonlinear distance function 6 Generative Discriminative Dictionary-based Data-intensive training Moderate training data Data-e cient training Hard to adapt Hard to adapt Very adaptable 6 Michael I Mandel, Young-Suk Cho, and Yuxuan Wang. Learning a concatenative resynthesis system for noise suppression. In Proc. IEEE GlobalSIP, 2014 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 19 / 73

Non-parametric synthesis for speech enhancement Deep neural network as nonlinear distance function Train DNN on correctly and incorrectly paired chunks Noise suppression Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 20 / 73

Non-parametric synthesis for speech enhancement Deep neural network as nonlinear distance function Train DNN on correctly and incorrectly paired chunks Audio super-resolution Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 21 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 22 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Find optimal sequence of clean chunks x = {x t } T t=0 input sequence of noisy chunks ẑ = {z t } T t=0 best sequence of corresponding dictionary chunks Y ẑ = argmax z t Y = argmax z i p(z t = j x t ) p(z t = j z t 1 = i) g(z j, x i ) T ij Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 23 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Find optimal sequence of clean chunks x = {x t } T t=0 input sequence of noisy chunks ẑ = {z t } T t=0 best sequence of corresponding dictionary chunks A nity between clean and noisy chunks Y ẑ = argmax z t Y = argmax z i p(z t = j x t ) p(z t = j z t 1 = i) g(z j, x i ) T ij Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 23 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Find optimal sequence of clean chunks x = {x t } T t=0 input sequence of noisy chunks ẑ = {z t } T t=0 best sequence of corresponding dictionary chunks A nity between clean and noisy chunks Transition a nity between clean chunks Y ẑ = argmax z t Y = argmax z i p(z t = j x t ) p(z t = j z t 1 = i) g(z j, x i ) T ij Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 23 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary M1 M1 M1... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary M2 M2 M2... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary M3 M3 M3... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary M4 M4 M4... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary M5 M5 M5... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Compare all pairs of noisy and clean chunks Observed mixture D1 D2 D3... Clean dictionary MN MN MN... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 24 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Standard Viterbi algorithm for to find optimal sequence Observed mixture D1 D2 D3... Clean dictionary MN MN MN... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 25 / 73

Non-parametric synthesis for speech enhancement Using this DNN for speech enhancement Standard Viterbi algorithm for to find optimal sequence Observed mixture D1 D2 D3... Clean dictionary MN MN MN... Similarity DNN Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 25 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 26 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Original clean speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 27 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Noisy speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 28 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Traditional mask-based separation Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 29 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Concatenative resynthesis output Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 30 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Original clean speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 31 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Subjective quality is high Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 32 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Subjective quality is high Clean Concat Speech Noise Sup Overall IRM NN Noisy 20 40 60 80 100 Quality (higher better) Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 32 / 73

Non-parametric synthesis for speech enhancement Noise suppression experiments Subjective intelligibility is ok Clean Concat IRM NN Noisy Keywords All words 60 70 80 90 100 Words correctly identified (%) Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 33 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 34 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Original clean speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 35 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Reverberated, compressed, 20% packet loss Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 36 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments NMF-based bandwidth expansion output Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 37 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Concatenative resynthesis output Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 38 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Original clean speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 39 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Subjective quality is high Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 40 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Subjective quality is high 100 90 80 70 MUSHRA Score 60 50 40 30 20 10 0 Clean Clean (hid) Rev Rev 8kHz RevOpusL20 RevOpusL20 (hid) Input NMF Concat CleanAmr RevAmr RevOpusL20 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 40 / 73

Non-parametric synthesis for speech enhancement Audio super-resolution experiments Subjective intelligibility is good 100 95 Correct words (%) 90 85 80 75 70 Clean Rev Rev 8kHz RevOpusL20 Input NMF Concat CleanAmr RevAmr RevOpusL20 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 41 / 73

Non-parametric synthesis for speech enhancement Summary Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement Overview Deep neural network as nonlinear distance function Using this DNN for speech enhancement Noise suppression experiments Audio super-resolution experiments Summary 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 42 / 73

Non-parametric synthesis for speech enhancement Summary Summary Concatenative synthesizer, DNN as noise-robust selection function Instead of modifying noisy speech, replace it completely eliminates noise, except for synthesis errors produces high quality, natural-sounding speech General robust supervised nonlinear signal mapping framework Data-e cient to train and adaptable to new talkers Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 43 / 73

Non-parametric synthesis for speech enhancement Summary Future applications Generalize to audio-visual speech recognition Label dictionary elements ahead of time to enable noise-robust non-parametric speech recognition noise-robust pitch tracking noise-robust speaker identification Incorporate language model into transition cost Develop e cient search mechanisms for large-vocabulary dictionaries Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 44 / 73

Parametric synthesis for speech recognition Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition Overview Algorithm Results Summary 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 45 / 73

Parametric synthesis for speech recognition Overview Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition Overview Algorithm Results Summary 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 46 / 73

Parametric synthesis for speech recognition Overview Mask-based source separation: Noisy Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 47 / 73

Parametric synthesis for speech recognition Overview Mask-based source separation: Masked Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 48 / 73

Parametric synthesis for speech recognition Overview Disrupts speech features: Noisy MFCCs He said such products would be marketed by other companies with experience him at this month. Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 49 / 73

Parametric synthesis for speech recognition Overview Disrupts speech features: Masked MFCCs He said such products would be marketed by other companies with experience him at this month. Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 50 / 73

Parametric synthesis for speech recognition Overview Disrupts speech features: Clean MFCCs He said such products would be marketed by other companies with experience in that business. Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 51 / 73

Parametric synthesis for speech recognition Overview Estimate better features using a strong prior model He said such products would be marketed by other companies with experience in that business. Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 52 / 73

Parametric synthesis for speech recognition Overview Our approach: Analysis-by-synthesis Synthesize speech signal so that it looks like the observation looks like speech Itakura-Saito divergence compares prediction with noisy observation Recognizer gives likelihood of speech-ness Both easy to optimize using gradient descent Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 53 / 73

Parametric synthesis for speech recognition Overview Speech recognizer includes lots of information Large vocabulary continuous speech recognizer captures: Acoustics of speech sounds The e ect of neighboring speech sounds Pronunciation of words Order of words Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 54 / 73

Parametric synthesis for speech recognition Algorithm Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition Overview Algorithm Results Summary 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 55 / 73

Parametric synthesis for speech recognition Algorithm Optimization over speech features x: optimization state: MFCCs, 10,000 dimensions y(x): ASR features derived from x M: mask provided a priori by another source separator min x n o L(x; M) =min (1 ) L I (x; M) + L H (y(x)) x Total cost Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 56 / 73

Parametric synthesis for speech recognition Algorithm Optimization over speech features x: optimization state: MFCCs, 10,000 dimensions y(x): ASR features derived from x M: mask provided a priori by another source separator min x n o L(x; M) =min (1 ) L I (x; M) + L H (y(x)) x Total cost Distance to noisy observation Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 56 / 73

Parametric synthesis for speech recognition Algorithm Optimization over speech features x: optimization state: MFCCs, 10,000 dimensions y(x): ASR features derived from x M: mask provided a priori by another source separator min x n o L(x; M) =min (1 ) L I (x; M) + L H (y(x)) x Total cost Distance to noisy observation Negative log likelihood under recognizer Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 56 / 73

Parametric synthesis for speech recognition Algorithm Analysis of audio meets resynthesis of MFCCs at mask Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 57 / 73

Parametric synthesis for speech recognition Algorithm L I (x; M): Distance to noisy observation Resynthesize MFCCs to power spectrum, where mask was computed Do mask-aware comparison in that domain: weighted Itakura-Saito between resynthesis, S!t (x), and noisy observation, S weighted by mask, M L I (x; M) =D M (S k S) = X!,t M!t S!t S!t (x) Does not require modeling speech excitation Numerically di erentiable with respect to x log S!t S!t (x) 1 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 58 / 73

Parametric synthesis for speech recognition Algorithm L H (y(x)): Likelihood under recognizer Large vocabulary continuous speech recognizer big hidden Markov model (HMM) approximated by the lattice of likely paths Closed form gradient with respect to x Serves as a model of clean MFCC sequences Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 59 / 73

Parametric synthesis for speech recognition Algorithm L H (y(x)): Likelihood under recognizer Large vocabulary continuous speech recognizer big hidden Markov model (HMM) approximated by the lattice of likely paths Closed form gradient with respect to x Serves as a model of clean MFCC sequences Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 59 / 73

Parametric synthesis for speech recognition Algorithm Optimization State space of approximately 13 800 10,000 dimensions Quasi-Newton optimization, BFGS gradient plus approximate second-order information Closed form gradient of HMM likelihood using a forward-backward algorithm Numerical gradient of IS divergence independent costs and gradients for each frame Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 60 / 73

Parametric synthesis for speech recognition Results Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition Overview Algorithm Results Summary 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 61 / 73

Parametric synthesis for speech recognition Results Experiment AURORA4 corpus read Wall Street Journal sentences (5000 word vocabulary) six environmental noise types SNRs between 5 and 15 db Masks from ideal binary mask and estimated ratio mask 7 7 Arun Narayanan and DeLiang Wang. Ideal ratio mask estimation using deep neural networks for robust speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages7092 7096. IEEE, May 2013 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 62 / 73

Parametric synthesis for speech recognition Results Recognition results Word error rate (%) averaged across noise type Mask Direct A-by-S Noisy 30.94 Estimated 16.18 15.31 Oracle 14.38 13.62 Clean 9.54 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 63 / 73

Parametric synthesis for speech recognition Results Reconstruction results Itakura-Saito divergence between resynthesized speech and original Mask Direct A-by-S Noisy 272301 Estimated 276497 275224 1273 Oracle 273006 272506 500 Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 64 / 73

Parametric synthesis for speech recognition Results Resynthesis gets closer to reliable regions Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 65 / 73

Parametric synthesis for speech recognition Results Resynthesis gets closer to reliable regions Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 66 / 73

Parametric synthesis for speech recognition Results Resynthesis gets closer to reliable regions Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 67 / 73

Parametric synthesis for speech recognition Results Resynthesis gets closer to reliable regions Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 68 / 73

Parametric synthesis for speech recognition Summary Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition Overview Algorithm Results Summary 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 69 / 73

Parametric synthesis for speech recognition Summary Summary Use a full recognizer as a prior model for clean speech Synthesize from MFCCs to the domain of the mask Adjust synthesis of speech signal so that it looks like the observation looks like speech Reduces recognition errors, distance to clean utterance Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 70 / 73

Parametric synthesis for speech recognition Summary Future directions Apply to DNN-based acoustic models Model speech excitation for full resynthesis of clean speech Model multiple simultaneous speakers and estimate masks jointly Combine with similar binaural model to include spatial clustering Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 71 / 73

Summary Outline 1 Motivation: need for noise robustness 2 Non-parametric synthesis for speech enhancement 3 Parametric synthesis for speech recognition 4 Summary Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 72 / 73

Summary Summary Synthesizers provide strong prior information Non-parametric synthesis models for high quality learned nonlinear matching function for perceptually motivated features Parametric synthesis models for e cient representation strong, di erentiable prior model of speech Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 73 / 73

Summary Summary Synthesizers provide strong prior information Non-parametric synthesis models for high quality learned nonlinear matching function for perceptually motivated features Parametric synthesis models for e cient representation strong, di erentiable prior model of speech Thanks! Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 73 / 73

Summary Summary Synthesizers provide strong prior information Non-parametric synthesis models for high quality learned nonlinear matching function for perceptually motivated features Parametric synthesis models for e cient representation strong, di erentiable prior model of speech Thanks! Any questions? Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 73 / 73

Parametric synthesis for separation Outline 5 Parametric synthesis for separation Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 1 / 3

Parametric synthesis for separation Re-estimate mask using resynthesis: Original Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 2 / 3

Parametric synthesis for separation Re-estimate mask using resynthesis: Re-estimate Michael Mandel (Brooklyn College) Analysis-by-synthesis Sept 8, 2015 3 / 3

2024 © educationdocbox.com Privacy Policy | Terms of Service | Feedback