The CSTR entry to the Blizzard Challenge 2017

Transcription

1 The CSTR etry to the Blizzard Challege 2017 Srikath Roaki, Mauel Sam Ribeiro, Felipe Espic, Oliver Watts The Cetre for Speech Techology Research (CSTR), Uiversity of Ediburgh, UK Abstract The aual Blizzard Challege coducts side-by-side testig of a umber of speech sythesis systems traied o a commo set of speech data. Similar to 2016 Blizzard challege, the task for this year is to trai o expressively-read childre s story-books, ad to sythesise speech i the same domai. The Challege therefore presets a opportuity to ivestigate the effectiveess of several techiques we have developed whe applied to expressive ad prosodically-varied audiobook data. This paper describes the text-to-speech system etered by The Cetre for Speech Techology Research ito the 2017 Blizzard Challege. The curret system is a hybrid sythesis system which drives a uit selectio sythesiser usig the output from a eural etwork based acoustic ad duratio model. We assess the performace of our system by reportig the results from formal listeig tests provided by the challege. Idex Terms: Merli, hybrid speech sythesis, uit selectio, deep eural etworks. 1. Itroductio The CSTR etry to this year s Blizzard Challege builds o the hybrid Multisy [1, 2] system submitted for last year [3]. Hybrid sythesis systems o the basis of target cost fuctio [4, 5, 6, for example] employ statistical models to predict acoustic properties of speech thereby brigs the beefits of extremely atural-soudig uit selectio (which is uaffected by the degradatios itroduced by vocodig [7, 8]). Similar to last year, the data used for this year s Challege was obtaied from professioally-read child-directed audio books ad is therefore much more prosodically rich tha the more stadard prompt-based speech data. The experimet preseted i [5, 6] established that improvig the uderlyig SPSS of a hybrid sythesiser results i improvemets to the cocateated output speech. Therefore, for our previous etry, we have icorporated two major improvemets to the uderlyig SPSS model compared to the system preseted i [6]: the decisio tree duratio model was replaced with a bi-directioal log short-term memory (LSTM) recurret eural etwork, ad the feed-forward DNN acoustic model was replaced with a LSTM etwork. Compared to our previous year etry, the text processig ad speech parameterizatio steps are largely uchaged. Acoustic model predictio is slightly optimised for better predictio of fudametal frequecy by addig supra-segmetal features based o acoustic couts ad is explaied i A otable exceptio is our attempt to smooth the jois: this ew developmet is described i sectio 2.6 below. The eural etworks used i this etry were traied usig our ope-source Merli speech sythesis toolkit [9] Data 2. Descriptio The database provided to the Challege by Usbore Publishig Ltd. cosists of the speech ad text of 56 childre s audiobooks spoke by a British female speaker. We made use of a segmetatio of the audiobooks carried out by two other Challege participats 12 ad kidly made available to other participats. The total duratio of the audio is approximately 6 hours after segmetatio. Three audiobooks from the give corpus were held out to act as a iteral developmet set to gauge system performace before geeratig the fial test data. The held-out data cosists of three full short stories: Goldilocks ad the Three Bears, The Boy Who Cried Wolf ad The Eormous Turip, havig a total combied duratio of approximately 10 miutes Setece selectio For setece selectio, we have followed the same approach as last year. For clarity, we repeat the procedure followed from our previous year etry [3]. Haressig the variety of speakig styles preset i expressively-read audiobooks might eable us to produce less robotic-soudig TTS systems. However, iitial experimets showed that the extreme variatio i parts of the traiig data for the Challege resultig i poor uit selectio. We therefore filtered the data usig the active learig approach described i [10]: 198 utterace-level acoustic features are extracted, ad 15 seteces iitially labelled as keep or too expressive by a expert listeer. Ucertaity samplig [11] usig a esemble of decisio trees was the used to select a further iformative sample to be had-labelled; this process cotiued for 20 miutes (real time). A classifier built o the etire set of hadlabelled data was the used to determie the subset of available seteces to be used for traiig. 20% of the traiig seteces were discarded i this way; iformal compariso suggested this resulted i more stable sythesis with fewer uwarrated prosodic excursios Text processig We have used Festival s Eglish frot-ed with the British Received Prouciatio versio of the Combilex lexico [12]. 163 items were added to cover words appearig i the traiig data but otherwise abset from the dictioary. There were slight differeces i the lexico-lookup procedures used i preparig the aotatio for traiig the SPSS model ad those employed by the Festival frot-ed used for Multisy. The resultig icosistecies were dealt with by aligig the DNN s phoe sequeces to those expected by Multisy i a ad hoc fashio ad is similar to our previous year etry. 1 Ioetics: 2 IIIT-H:

2 Word ad syllable level vector represetatios were icluded, accordig to the method described i [13]. These were leared by takig couts of acoustic evets of f0 ad eergy stylized by clustered vectors ad mea values defied over syllables or words. The traiig data available for the Challege was used to lear these matrices. Experimets usig vectors represetatios leared over a larger database of a differet speaker, but we have observed that results were comparable with speaker-depedet vectors leared o a smaller database Parametric system The parametric system was implemeted usig DNNs i a covetioal two-stage approach. I the first stage, a duratio model is used to predict phoe duratios to form frame-level liguistic features. I the secod stage, a acoustic model is used to geerate parameters from those liguistic features Duratio model The duratio model traied for our etry to the challege made use of a simple ad straightforward approach with feed-forward eural etworks (DNNs) as demostrated i [14, 15]. The duratio model is traied o the aliged data ad geerates statelevel duratios give phoe-level liguistic features. The described approach was used oly to geerated duratios, which were the used to form frame-level liguistic features used as iput i the geeratio of acoustic parameters. The hybrid Multisy uit-selectio system, however, does ot make use of ay duratio-related features i its target cost fuctio. Icludig such features i the uit selectio process is left for future work Acoustic model The liguistic features extracted from the frot-ed were coverted to umerical vectors usig a set of cotiuous ad biary questios [9]. To these, we appeded the syllable ad word level vector represetatios based o acoustic couts [13]. The duratios geerated by the duratio model described above were used to propagate all feature to frame-level. These frame-level feature vectors were the used as iput to a acoustic model. A feedforward eural etwork was traied at the framelevel to map liguistic iputs to vocoder parameters cosistig of static ad dyamic (delta ad delta-delta) features. These acoustic parameters iclude 60 mel-cepstra coefficiets, 25 bad aperiodicities, log-f0, ad a biary voicig decisio. Maximum likelihood parameter geeratio (MLPG) ad postfilterig are the applied to the geerated acoustic parameters. I SPSS these parameter trajectories would the be passed through the vocoder to sythesize a speech waveform. Istead, we use them as targets for selectig waveform uits. Withi each phoe uit, geerated parameters are split uiformly across time ito 4 sectios. A Gaussia distributio is the fitted for each sub-phoe sectio of acoustic parameters. The variaces of these Gaussia distributios are floored at 1% of the global variace per parameter [6]. The distributios associated with each of the 4 sub-phoe sectios are used to costruct a diphoe represetatio for the target utterace. To costruct a diphoe represetatio, we take the first or last 2 sectios associated with its correspodig phoes. Comparable distributios were geerated for the diphoe cadidates i the uit database, based o vocoder parameters extracted from the traiig data ad atural duratios obtaied by forced aligmet Feature extractio Phoe sequeces were obtaied from the text usig Festival [16]. Festvox s ehmm method [17] was used to modify the phoe sequeces by the isertio of acoustically-motivated pauses; A state-level forced aligmet of these phoe sequeces with the setece-segmeted audio was the obtaied usig cotext-idepedet HMMs, similar to [18]. Each phoe was the characterised by a vector of 481 text-derived biary ad umerical features a subset of the features used as decisiotree clusterig questios i the HTS demo [19], adapted for our phoeset. These questios icluded liguistic cotexts such as quiphoe idetity which are added at phoe-level, ad part-ofspeech, positioal iformatio relatig to syllables, words, phrases, etc. All umerical features are give as iput (after appropriate ormalisatio) directly to the etwork, ad ot ecoded as (for example) 1-of-K. For duratio modellig, all these features were used as iput ad ormalised to the rage of [0.01, 0.99]. The output for traiig is a five-dimesioal vector of duratios for every phoe, comprisig five sub-state duratios. For acoustic modellig, the iput uses the same features as duratio predictio, to which 9 umerical features were appeded. These capture frame positio i the HMM state ad phoeme, state positio i phoeme, ad state ad phoeme duratio, similar to [18]. The speech data was aalysed with STRAIGHT [20], ad each 5ms frame was represeted usig 60 mel cepstral coefficiets (MCC), measures of aperiodicity i 25 frequecy bads (BAP), logarithmic F 0 iterpolated through uvoiced regios, ad a biary voicig feature. These 87 static features were supplemeted with delta ad delta-delta features, ad for both the duratio ad acoustic data, a per-compoet mea ad variace ormalisatio was applied prior to model traiig, with the trasformatio reversed as part of sythesis Duratio ad acoustic model traiig For the duratio model, we have used 481-dimesioal biary ad cotiuously valued feature vectors as iput. Its output was a 5-dimesioal feature vector represetig state duratios i terms of frames. The model was defied to be 6 feedforward hidde layers, each with 1024 odes, usig the tah activatio fuctio. Mii batch size was set to 64 ad learig rate was set to 0.002, beig was reduced by 50% with each epoch after the first 10 traiig epochs. For the acoustic model, we have used the same 481- dimesioal feature vector represetig liguistic features. To these, we added syllable ad word level vector represetatios spaig a widow of 3 uits. Nie frame-level features were icluded accordig to [18] ad available from [9]. The iput vector to the acoustic models cosisted of a total of 1900 dimesios. The model cosisted of 6 feedforward hidde layers, each with 1024 odes, usig the tah activatio fuctio. Miibatch was set to 256 ad remaiig parameters were idetical to the duratio model Uit selectio waveform rederer For uit selectio, we have followed the same approach as last year. For clarity, we repeat the procedure followed from our previous year etry [3]. A modified form of Festival s Multisy egie [2] was used for the uit selectio stage of our system. To compare the suit-

3 ability of a give cadidate diphoe i the uit database with the 4 distributios represetig a sythesised diphoe, the symmetrised Kullback Leibler divergece (KLD) [21] is used. The KLD is computed betwee each of the 4 cadidate uit s distributios ad the correspodig target uit distributios idividually. The resultig 4 scores are the summed to produce the fial target score. The stadard Multisy joi cost (sum of distaces betwee 12 MFCCs, f 0 ad eergy from the frame either side of the joi) is retaied, as well as the stadard pre-selectio criterio of cadidate uits (by matchig diphoe idetity). The stadard Multisy Viterbi search (with pruig to reduce the search time) is performed i order to optimise target cost ad joi cost. Also the stadard Multisy back-off rules are used where the target diphoe to be sythesised is ot preset i the traiig data Speech sythesis At sythesis time, duratio is predicted first, ad is used as a iput to the acoustic model to predict the speech parameters. Maximum likelihood parameter geeratio (MLPG) [22] usig variaces computed from the traiig data was applied to the output features for sythesis, ad spectral ehacemet postfilterig was applied to the resultig MCC trajectories. These parameter trajectories are the used to produce diphoe coefficiets. The Festival Multisy egie was used to compute the target ad joit cost betwee target uit ad pre-selected cadidate uits to select the fial cadidate, as explaied above. The fial waveform sythesis was doe by joiig the selected uits. A additioal smoothig ad post-modificatio of prosody was performed durig joiig the uits ad is explaied i below sectio Cocateatio ad joi smoothig The selected waveform uits are parameterised by usig the method proposed i [23]. It extracts pitch sychroous speech features i a frame-by-frame basis, describig the complex spectra ad F0 cotour. The correctio/smoothig operatios are performed over these features to produce seamless cocateatio of uits Cocateatio ad correctio of F0 cotours The F 0 mid poit (F 0 m) betwee two cosecutive uits is give by F 0 m = (F 0 p[n p 1] + F 0 c[0])/2, where p meas precedig uit, c curret uit, ad N is the uit legth i frames. The, the slope of the F0 cotours of both uits are adjusted to reach the F 0 m just i the joi locatio. The corrected F 0 cotours are computed by the Equatios 1 ad 2. ( ) F 0 c c[ c] = F 0 c[ c] + (F 0 m F 0 c[0]) + 1 (1) 1 N c F 0 p[ p] = F 0 p[ p] + (F 0 m F 0 p[n p 1]) p N p 1 (2) Where F 0 is the corrected F 0, ad is the frame idex withi each uit. After havig all the corrected F 0 cotours for all the uits, these are appeded buildig a sigle F 0 cotour for the whole setece Spectral cocateatio ad smoothig Basically, it is doe by overlappig ad crossfadig the complex FFT spectra of two cosecutive uits. Some extra frames are extracted from the sources, so the uits ca be overlapped without Mea Opiio s (aturaless) All listeers Figure 1: Our system(e): Mea opiio score for aturaless of the sythesized speech with ratigs from all listeers. affectig their expected locatios i the sythesised waveform. Three extra frames o each side of the uits are extracted from the sources, thus a overlap of seve frames aroud the jois is produced. The FFT complex spectrum S is derived from the parameters proposed i [23], M, R, ad I, by S = M (R + Ij). The crossfade is liearly applied to mix the FFT complex spectra of two cosecutive uits, progressively. It is seve frames legth, ad i case that a uit is too short, the crossfade is shorteed accordigly. After performig this operatio o every joi, the FFT complex spectra of all the uits are cocateated producig a sigle complex spectra stream, that describes the whole utterace. Fially, the sigal is sythesised by covertig the FFT complex spectra to time domai, ad applyig Pitch Sychroous Overlap-Add as explaied i [23], usig the corrected F 0 cotour Paragraph-level sythesis From the seteces sythesised i this way, files were made cotaiig whole paragraphs, chapters ad books as required by the Challege by simply cocateatig the waveforms. While proper exploitatio of log-distace cotexts ought to improve sythesis quality, o cotexts outside the curret setece were used for the preset submissio. 3. Results The idetifier for our system i the published results is E Naturaless Mea opiio scores for aturaless from all listeers o book seteces are show i Figure 1. I our discussio, we make use of the published statistical aalysis of the results at 1% level with Boferoi corrected alpha) [24]. Our system outperformed all three baselies (systems B, C ad D). Amog the 12 other challege participats, our system is outperformed oly by a sigle system (I). The same tred ca be see across the scores

4 Mea Opiio s (similarity to origial speaker) All listeers Mea Opiio s (audiobook paragraphs overall impressio) All listeers Figure 2: Our system(e): Mea opiio score for speaker similarity with ratigs from all listeers. Figure 3: Our system(e): Mea opiio score for overall impressio with ratigs from all listeers. made by paid listeers, o-lie voluteers ad cosiderig ratigs oly from speech experts, o other system was sigificatly better tha ours. Overall, our system outperformed 11 out of 15 other systems (icludig three baselies) evaluated for listeig test Speaker similarity The mea opiio scores for speaker similarity from all listeers o book seteces are show i Figure 2. Cosiderig ratigs from all listeers (or ay other listeer group), o other system was sigificatly better tha ours ad our system was i tur sigificatly better tha 13 other systems. These results show the effectiveess of waveform cocateatio systems for speaker similarity Evaluatio of audiobook paragraphs We ow cosider the results for evaluatio of audiobook paragraphs that have bee evaluated o several other factors like stress, itoatio, emotio, pleasatess, listeig effort, speech pauses ad overall impressio. Cosiderig ratigs from all listeers o overall impressio, our system showed similar performace as i the case of the isolated setece evaluatio of aturaless ad speaker similarity. Oly oe system (I) outperformed us ad our system was sigificatly better i tur tha 11 other systems (cf. Figure 3). A similar tred ca be see across the scores made by speech experts, olie voluteers ad paid listeers. Cosiderig ratigs for other idividual factors (e.g., itoatio, emotio ad pleasatess) from all listeers, agai oly system I cosistetly outperformed ours. Overall, our system outperforms betwee 7 ad 11 other systems i evaluatio of each of these factors, performig best i emotio ad pleasatess Itelligibility (SUS) We ow cosider the results for itelligibility of sematically upredictable seteces (makig use of the published statisti- cal aalysis of sigificat differece betwee word error rates of the systems). Takig ito accout ratigs from all listeers, there are oly three other systems out of 15 (D, L, ad M) sigificatly better tha ours. Cosiderig oly paid listeers, there are oly two other systems (D ad L) sigificatly better tha ours. Out of 15 other systems evaluated by paid listeers, 10 were ot sigificatly more or less itelligible tha ours, 3 were sigificatly less itelligible, ad oly 2 sigificatly more itelligible. The results show that our system is quite effective o itelligibility as well. Overall, our system has show cosistet performace (stadig i the top four) i all the factors evaluated for the Challege. 4. Coclusios ad future work For this year s CSTR Blizzard Challege etry, the hybrid system submitted for last year [3] was slightly optimized i acoustic modelig for better predictio of F0 ad performed smoothig betwee jois. The results of the evaluatio are o the whole very positive, but there are still a umber of potetial future improvemets which could be made to the hybrid sythesis system described here. These iclude adoptig cosistet lexico-lookup for both the SPSS ad uit selectio systems, makig use of same acoustic features for both joi ad target cost, predictio of phrase breaks, ad the explicit iclusio of predicted duratio i the uit selectio sythesis target cost. Reproducibility: We used the Ope Source Merli toolkit 3 for parameter predictio ad Festival Multisy 4 for uitselectio. Ackowledgemet: Watts was supported i this research by EPSRC Stadard Research Grat EP/P011586/1, Speech Sythesis for Spoke Cotet Productio (SCRIPT)

5 5. Refereces [1] R. A. Clark, K. Richmod, ad S. Kig, Festival 2 build your ow geeral purpose uit selectio speech sythesiser, i Proc. SSW, [2] R. A. Clark, k. Richmod, ad S. Kig, Multisy: Ope-domai uit selectio for the Festival speech sythesis system, Speech Commuicatio, vol. 49, o. 4, pp , [3] T. Merritt, S. Roaki, Z. Wu, ad O. Watts, The CSTR etry to the Blizzard Challege 2016, i Proc. Blizzard Challege workshop, [4] Z.-J. Ya, Y. Qia, ad F. K. Soog, Rich-cotext uit selectio (rus) approach to high quality tts, i Proc. ICASSP, 2010, pp [5] Y. Qia, F. K. Soog, ad Z.-J. Ya, A uified trajectory tilig approach to high quality speech rederig, IEEE Audio, Speech, Laguage Process., vol. 21, o. 2, pp , [6] T. Merritt, R. A. J. Clark, Z. Wu, J. Yamagishi, ad S. Kig, Deep eural etwork-guided uit selectio sythesis, i Proc. ICASSP, [7] H. Ze, A. Seior, ad M. Schuster, Statistical Parametric Speech Sythesis Usig Deep Neural Networks, i Proc. ICASSP, 2013, pp [8] G. E. Heter, T. Merritt, M. Shao, C. Mayo, ad S. Kig, Measurig the perceptual effects of modellig assumptios i speech sythesis usig stimuli costructed from repeated atural speech, i Proc. Iterspeech, 2014, pp [9] Z. Wu, O. Watts, ad S. Kig, Merli: A Ope Source Neural Network Speech Sythesis, i Proc. SSW, Suyvale, USA, [10] O. Watts, A. Sta, R. Clark, Y. Mamiya, M. Giurgiu, J. Yamagishi, ad S. Kig, Usupervised ad lightly-supervised learig for rapid costructio of TTS systems i multiple laguages from foud data: evaluatio ad aalysis, i 8th ISCA Workshop o Speech Sythesis, Barceloa, Spai, Aug. 2013, pp [11] D. D. Lewis ad W. A. Gale, A sequetial algorithm for traiig text classifiers, i Proceedigs of the 17th aual iteratioal ACM SIGIR coferece o Research ad developmet i iformatio retrieval, 1994, pp [12] S. Fitt ad K. Richmod, Redudacy ad productivity i the speech techology lexico - ca we do better? i Proc. Iterspeech 2006, Sep [13] M. S. Ribeiro, O. Watts, ad J. Yamagishi, Learig word vector represetatios based o acoustic couts, i Proc. Iterspeech, Stockholm, Swede, August [14] S. Roaki, Z. Wu, O. Watts, ad S. Kig, A Demostratio of the Merli Ope Source Neural Network Speech Sythesis, i Proc. SSW, Suyvale, USA, [15] S. Roaki, O. Watts, S. Kig, ad G. E. Heter, Media-based geeratio of sythetic speech duratios usig a o-parametric approach, i IEEE workshop o Spoke Laguage Techology, Sa Diego, Califoria, [16] A. Black, P. Taylor, R. Caley, R. Clark, K. Richmod, S. Kig, V. Strom, ad H. Ze, The festival speech sythesis system, versio , Upublished documet available via cstr. ed. ac. uk/projects/festival. html, [17] K. Prahallad, A. W. Black, ad R. Mosur, Sub-phoetic modelig for capturig prouciatio variatios for coversatioal speech sythesis, i Proc. ICASSP, 2006, pp. I 853 I 856. [18] Z. Wu, C. Valetii-Botihao, O. Watts, ad S. Kig, Deep eural etworks employig multi-task learig ad stacked bottleeck features for speech sythesis, i Proc. ICASSP, 2015, pp [19] H. Ze, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. Black, ad K. Tokuda, The HMM-based speech sythesis system (HTS) versio 2.0, i Proc. SSW, vol. 6, 2007, pp [20] H. Kawahara, STRAIGHT, exploitatio of the other aspect of VOCODER: Perceptually isomorphic decompositio of speech souds, Acoust. Sci. Techol., vol. 27, o. 6, pp , [21] J. R. Hershey ad P. A. Olse, Approximatig the Kullback- Leibler divergece betwee Gaussia mixture models, i Proc. ICASSP, [22] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, ad T. Kitamura, Speech parameter geeratio algorithms for HMM-based speech sythesis, i Proc. ICASSP, vol. 3, 2000, pp [23] F. Espic, C. Valetii-Botihao, ad S. Kig, Direct modellig of magitude ad phase spectra for statistical parametric speech sythesis, i Proc. Iterspeech, Stochohlm, Swede, August [24] R. A. Clark, M. Podsiadlo, M. Fraser, C. Mayo, ad S. Kig, Statistical aalysis of the Blizzard Challege 2007 listeig test results, Proc. Blizzard Challege Workshop, 2007.