42 P. ČERVA, J. NOUZA, MAP BASED SPEAKER ADAPAION IN VERY LARGE VOCABULARY SPEECH RECOGNIION MAP Based Speaker Adapaion in Very Large Vocabulary Speech Recogniion of Czech Per ČERVA, Jan NOUZA Dep. of Elecronics and Signal Processing, echnical Universiy of Liberec, Hálkova 6, 46 7 Liberec, Czech Republic per.cerva@vslib.cz, jan.nouza@vslib.cz Absrac. he paper deals wih he problem of efficien adapaion of speech recogniion sysems o individual users. he goal is o achieve beer performance in specific applicaions where one known speaker is expeced. In our approach we adop he MAP (Maximum A Poseriori) mehod for his purpose. he MAP based formulae for he adapaion of he HMM (Hidden Markov Model) parameers are described. Several alernaive versions of his mehod have been implemened and experimenally verified in wo areas, firs in he isolaed-word recogniion (IWR) ask and laer also in he large vocabulary coninuous speech recogniion (LVCSR) sysem, boh developed for he Czech language. he resuls show ha he word error rae (WER) can be reduced by more han 2% for a speaker who provides ens of words (in case of IWR) or ens of senences (in case of LVCSR) for he adapaion. Recenly, we have used he described mehods in he design of wo pracical applicaions: voice dicaion o a PC and auomaic ranscripion of radio and V news. Keywords Speech recogniion, speaker adapaion, maximum a poseriori mehod, hidden Markov models.. Inroducion Modern sysems for auomaic speech recogniion (ASR) are based on he echnique ha uses hidden Markov models (HMM), usually wih coninuous densiy funcion (CDHMM). Saisical parameers of hese probabilisic models mus be esimaed in he phase of he sysem raining by exploiing large daabases of speech recordings. In many pracical applicaions, ASR sysems are used by speakers whose speaking characerisics are differen, depending on heir gender, dialec, ec. o make he ASR robus agains all hese variaions and o allow almos everybody o use he sysems, hese mus operae as speaker independen - SI. For he raining of a successful SI sysem, several ens of hours of annoaed speech recorded by hundreds of differen persons are necessary. Bu here are applicaions, le e.g. voice conrol of a PC or voice dicaion, where only one person is expeced o use hem. In such a case, he HMMs could be rained on he user s speech only and hen he ASR would operae as speaker dependen - SD. Unforunaely, even in his specific case he user would be required o record a leas several hours of his/her speech for raining purposes. Generally his is hardly feasible for he user as well as for he provider of he sysem. A more accepable soluion consiss in he adapaion of he exising SI models for he given speaker and hus making he sysem speaker adaped - SA. he major advanage of his approach is ha he speaker will be asked o record a significanly smaller amoun of daa (less han hour). his recording and reraining is usually done on he user s own compuer, which means ha he ASR sysem also adaps is parameers o he characerisics of he given microphone and o specific noise of he environmen in which he speaker alks. ha is why adapaion mehods are so imporan for pracice and many researchers are sill working on echniques ha lead o furher improvemen in recogniion while asking less adapaion daa from he user. In his paper we describe he mehod ha uilizes he Maximum A Poseriori (MAP) sraegy o he adapaion of phonemically based isolaed-word recogniion (IWR) and coninuous-speech recogniion (CSR) sysems. Boh have been developed for he Czech language and can operae wih very large vocabularies (above, words). his paper is srucured as follows: In he nex secion we describe he heoreical background of he MAP based speaker adapaion echnique. he resuls from quie exensive experimenal evaluaion are presened in secion 3 (for he IWR as and secion 4 (for he CSR as. 2. MAP Based Speaker Adapaion he adapaion (esimaion) mehod MAP (Maximum A Poseriori) [] is based on a differen sraegy han he Maximum Lelihood Esimaion (MLE) [2] ha is widely used during he sandard HMM raining. While he MLE assumes he model parameers o be unknown bu fixed, he MAP assumes hese parameers o be random variables wih known prior disribuions. his informaion abou prior disribuions compensaes he lack of adapaion daa during he adapaion process.
RADIOENGINEERING, VOL. 3, NO. 3, SEPEMBER 24 43 he erm ζ ( i, in (2.3) represens he occupaion lelihood of k-h mixure componen of sae i and indicaes he amoun of he daa used for he adapaion of his mixure. Hence he new esimaes of he SA parameers have form of weighed sum where he original SI parameers are weighed by facor Fig.. Illusraion of he MAP based adapaion process. Le X = {x, x 2,, x } be he sequence of parameerized vecors of adapaion daa and Φ) he prior disribuion of parameer Φ. he parameer Φ can represen means, variances or weighing coefficien of one Gaussian componen (mixure) of he given HMM. he opimal value of parameer Φaccording o MAP esimae can be expressed as Φ MAP = arg max Φ X), (2.) Φ τ / τ + ζ ( i, and mixed wih hose esimaed from he adapaion daa by he MLE echnique. his procedure is illusraed in Fig.. When increasing he amoun of he adapaion daa lim ζ ( i, and he MAP esimae converges o he SD model which would resul from he MLE mehod. he formulae for he MAP esimaion of he variances and mixure weighs can be derived in a similar way. he expressions are raher complex and can be found in [3]. where Φ X) is he poserior probabiliy densiy disribuion of parameer Φ. According o Bayes rule Φ X) can be expressed as 3. Experimenal Evaluaion in IWR ask X Φ) Φ) Φ X) =. (2.2) he evaluaion was performed using he ASR sysem X) developed in our lab [5]. A he momen his is he mos he maximizaion of Φ X) is hen reached by changing powerful IWR sysem designed for Czech. Is sandard he value of Φ o maximize X Φ) Φ). vocabulary conains 8, mos frequen words. he sysem uses hree-sae CDHMMs of Czech phonemes [4] he criical quesion in he MAP based speaker adapaion is: how o deermine he prior parameers. In he pensaed by higher numbers of mixures (32, 64 or even and noises. hese are conex independen, which is com- sandard MAP mehod hese parameers are aken direcly ). he feaure vecor includes 39 MFCC parameers (3 from he available SI models. In secion 4.3 we will show saic coefficiens ogeher wih heir firs and second derivaives) calculaed from he signal sampled a 8 khz rae ha beer resuls can be achieved if gender dependen SI models are used. ino 6 bis. For CDHMMs wih Gaussian sae observaion densiies, where λ = ( µ, Σ i is he k-h Gaussian componen of sae i wih mixure weigh c k, he soluion of he MAP esimaion for he means is: ζ ( i, SA τ SI µ = µ + µ ˆ τ + ζ ( i, τ + ζ ( i,. (2.3) Here τ is he weighing facor (a free parameer), µ SI is he mean vecor of k-h mixure componen of sae i of he given SI model, µ SA is he adaped mean vecor and µ^ is he ML esimae of his vecor compued by Baum-Welch algorihm [2] from all he adapaion daa. In he firs series of experimens, we sudied he influence of he amoun of he adapaion daa on he recogniion accuracy of he adaped sysem (see ab. and Fig.2). Here, only he Gaussian means were adaped using consan value τ = 5 for all he mixures. his value was found opimal in preliminary ess. he se of 43 Czech randomly chosen words was available for he adapaion, from which subses wih variable size were used in he adapaion procedure implemened in accord wih he eq. (2.3). Anoher se of 862 words from he same speaker was used for he ess. he ess were performed wih wo speakers and wo sysem seings (32 or 64 mixures) and he average resuls are given in ab..
44 P. ČERVA, J. NOUZA, MAP BASED SPEAKER ADAPAION IN VERY LARGE VOCABULARY SPEECH RECOGNIION number of adapaion words rae (SI) 2 6 5 2 3 35 43 sysem wih 32 mixures 75 8 8 83 84 85 85 87 87 87 2 24 3 35 4 4 46 45 47 rae sysem wih 64 mixures 74 8 82 83 85 85 85 87 87 88 25 29 35 4 42 44 49 5 52 recogniion rae ab.. Recogniion accuracy as funcion of he number of adapaion words in IWR ask 9 85 8 75 5 32 mixures 64 mixures 5 2 25 3 number of adapaion words Fig. 2. Graph of he recogniion accuracy as funcion of he number of adapaion words in IWR ask. We can noice ha even if only words are used for he adapaion, he relaive word error rae reducion () is abou 2 %. When words are used he reaches 35 %. Wih more adapaion daa he improvemen ges slower. In anoher series of experimens, we sudied he possibiliy o adap also Gaussian variances and mixure weighs. he resuls showed ha he adapaion of hese parameers had only a negligible effec on he recogniion accuracy (see also secion 4.2). We also compared wo ypes of adapaion daa isolaed words vs. coninuously spoken uerances and noiced ha he former were more appropriae for he IWR ask, mos probably because of he voice sress which occurs a iniial pars of isolaed words. 4. Experimenal Evaluaion in CSR ask In his case we used our own large vocabulary CSR sysem [7]. Is vocabulary was made of he 3, mos frequen words. he acousic par of he sysem was same as for he IWR ask (64-mixure HMMs). he linguisic 35 4 par was based on he bigram language model esimaed on a corpus conaining abou 2 GB of Czech (mainly newspaper) ex. Because he ess were ime consuming, he iniial experimens were performed only for one speaker. Afer finding he appropriae values of he free adapaion parameers we run anoher series of ess in which more speakers were involved. 4. Experimens Performed for One Speaker Again, in he firs series of experimens we sudied he impac of he amoun of he adapaion daa here measured by he number of adapaion senences. he es se conained 462 senences recorded by he same speaker. here were 74 words in hese es uerances, from which 225 were no presen in he vocabulary, i.e. he Ouof-vocabulary (OOV) rae was 3.7 %. A varying number of oher o 6 senences were used for he adapaion. he resuls are summarized in ab. 2 and Fig. 3. number of adapaion senences lengh [min] rae (SI) 25 5 75 2 4 955 (SD) 3. 6.4 9.2 2.4 25.3 5. 22. 74.2 76.8 77.7 78.5 78.8 8.2 8.7 8.7.2 3.6 6.9 8. 23.3 25.4 29.3 recogniion rae ab. 2. Recogniion rae as funcion of he number of adapaion senences for he LVCSR sysem. 85 82 79 76 73 7 5 SA model SD model 5 2 25 3 35 4 45 number of senences for adapaion 5 55 6 Fig. 3. Graph of he dependency of he recogniion accuracy on he number of adapaion senences for he LVCSR sysem wih 64-mixure HMMs. For comparison we creaed also SD models by using he oal number of 955 senences recorded by he same speaker. In Fig. 3 we can see he convergence of he SA sysem o he SD one as he amoun of he adapaion daa increases. he very imporan fac is ha senences used in adapaion resuled in 8%. In secion 4.2 his significan improvemen is verified for more speakers. In he second series of experimens (see ab. 3 and Fig. 4) we sudied he impac of weighing coefficien τ
RADIOENGINEERING, VOL. 3, NO. 3, SEPEMBER 24 45 on he recogniion rae. Furhermore, also Gaussian variances and mixure weighs were adaped in he same manner. Here, he adapaion se was fixed o senences. weigh τ 2 3 4 5 8 5 2 recogniion rae adapaion of means 79.3 79.5 79.5 79.7 79.6 79.5 79.2 78.8 78.9 9.7 2.4 2.5 2. 2. 2.5 9.3 7.8 8. adapaion of means and weighs of mixures 8.3 2.9 2.3 2.4 2.9 2.8 2.5 2.6 9.3 Adapaion of means, weighs of mixures and variances 2. 9.9 2.5 2.5 2.7 2.3 2.5 2. 2. ab. 3. Resuls of MAP based adapaion for differen values of adapaion weigh and for differen modes of adapaion. from radio news, he laer using sandard microphone conneced o a PC. he srucure of his es daabase is shown in ab. 4. In oal here were 566 senences (more han 2, words) available for he esing and senences per speaker for he adapaion of HMM means. he weighing facor τ was se o. gender of he speaker Microphone/ Radio lengh of adap. senences [min] SI sysem SA sysem M M M F F F F M M M R R M R 2.4 5.5 2.8 7.5 7.5 7.5 7.5 74.2 68. 77.8 82.5 82. 59. 82. 79.3 73. 8.9 86.2 85.7 68. 84.4 22 means means + weighs means + weighs + variances (SA agains SI) 9.8 3. 8.5 2. 2. 22. 3.3 2 2 9 8 7 ab. 4. Resuls of MAP based adapaion of means for differen speakers by using adapaion senences from each. ab. 4 proves ha he adapaion was successful for all he speakers. he average reducion of word error rae (he value of ) was 2.8 %. 6 5 2 3 4 5 weigh Fig. 4. Resuls of MAP based adapaion for differen values of adapaion weigh and for differen modes of adapaion. he resuls show ha values vary from 7.8 % o 2. % when he weighing coefficien τ changes in range from o 2 and when only means are adaped. his demonsraes he imporance of he proper value of τ on he resuls of he adapaion. he experimen also proves ha he exension of MAP based speaker adapaion from he means only o he complee HMM parameers brings only a small absolue improvemen in he (abou.6 % in average). he opimal value of τ depends on he amoun of he adapaion daa (see [3]) as well as on he ype of adaped parameers. From ab. 3 we can noice ha for he given speaker he opimal value is close o in case adapaion senences are used. 4.2 Experimens Performed for More Speakers he recordings from seven oher speakers were used in his evaluaion. he people were eiher professional (broadcas) speakers or sudens. he former were recorded 8 5 2 4.3 Applicaion of Gender Dependen Models for Speaker Adapaion I is known ha male and female voices are differen. his fac can be used also in speaker adapaion. Hence we performed anoher series of experimens wih he same speakers as above and GD (gender dependen) models. SI models GD models rae rae 74.2 6.8 77.8 82.5 82. 59. 82. 76.6 6.2 79.3 84.4 83.8 6. 84.5 9.3. 6.8.9 9.5 2.4 3.9 ab. 5. Speaker recogniion raes achieved for 7 speakers and eiher general SI models or GD (gender dependen) ones. he simples adapaion echnique only consised in using gender dependen models rained eiher on male or female subses of he general raining daabase. As shown in ab. 5 his simple approach led o small bu consisen improvemen (average was 7.7 %). In he second experimen he parameers of he GD models were used as priors for he MAP based adapaion of he means, i.e. he GD models raher han he SI ones were adaped. From ab. 6 we can observe ha using he prior GD models in he adapaion yielded slighly beer resuls in mos cases.
46 P. ČERVA, J. NOUZA, MAP BASED SPEAKER ADAPAION IN VERY LARGE VOCABULARY SPEECH RECOGNIION prior parameers SI models GD models 9.8 3. 8.5 2. 2. 22. 3.3 2.3 32.7 8. 2. 2. 23.9 5.6.5.6 -.5...9 2.3 Acknowledgmens his work has been parly suppored by he Gran Agency of he Czech Republic (gran no. 2/2/24) and hrough research goal projec MSM 2422. ab. 6. Resuls of speaker adapaion performed by using GD and SI models as priors for he MAP based adapaion. 5. Conclusion his paper deals wih he problem of he efficien speaker adapaion for he very large vocabulary speech recogniion of Czech. We focus on he MAP based mehods whose heoreical background is briefly described. We applied his echnique wih several differen variaions, namely for Gaussian means, variances and mixure weighs, using eiher gender independen or gender dependen models as prior informaion. Our resuls from exensive experimens demonsrae ha he word error rae (WER) in isolaed-word recogniion can be reduced by 2 % or even 35 % if he speaker provides or adapaion words, respecively. In he more complex ask of coninuous speech recogniion, he WER will be reduced by abou 2 % when senences are used for adapaion. Similar resuls have been repored also in [7], where hey were achieved by commercial sofware (HK). Our implemenaion has several advanages: I is more compac and flexible, because i allows furher modificaion and opimizaion. One of hem is he exension owards he adapaion of variances and mixure weighs. In such case he was furher increased in average abou.6 % in he ask of LVCSR. Addiional enhancemen (abou %) was achieved by exploiing gender dependen models as priors wihin he MAP reesimaion. Recenly, all he described mehods are applied in he pracical design of he sysems for voice dicaion ino a PC. One version should serve as a Czech IWR dicaion sysem working wih 5,-word vocabulary, he oher as a CSR dicaion machine wih a 5, word lexicon. hanks o he adapaion ogeher wih furher improvemen of lexical and language model, he recogniion rae will ge above he 9 % level. Anoher benefi from he research can be expeced in he design of he sysem for he auomaic ranscripion of broadcas news. he use of he special models adaped o he speech of key-speakers will improve he overall accuracy of he ranscripion. References: [] GAUVAIN, J.L., LEE, C.H. Maximum a poseriori esimaion for mulivariae Gaussian mixure observaions of Markov chains. IEEE rans. SAP. 994, vol. 2, p. 29 298. [2] HUANG, X.D., ACERO, A., HON, H.W. Spoken language processing. Englewood Cliffs: Prenice Hall, 2. [3] ČERVA, P. Mehods of speaker adapaion for speech recogniion sysem. Diploma hesis (in Czech). U of Liberec. 24. [4] NOUZA, J., PSUKA, J., UHLÍŘ, J. Phoneic alphabe for speech recogniion of Czech. Radioengineering. 997, vol. 6, no. 4, p.6 o 2. [5] NOUZA, J., NOUZA,. A Voice dicaion sysem for a millionword Czech vocabulary. Proc. of Conference on Compuing, Communicaion and Conrol echnologies. Ausin, 24. [6] NOUZA, J., NEJEDLOVA, D., ZDANSKY, J., KOLORENC, J. Very large vocabulary speech recogniion sysem for auomaic ranscripion of Czech broadcas programs. Proc. of In. Conference on Spoken Language Processing (ISCLP 4). Jeju, 24. [7] ŽELEZNÝ, M. Speaker adapaion in coninuous speech recogniion sysem of Czech. PhD hesis (in Czech). ZČU Plzeň 2. Abou Auhors... Jan NOUZA (957) received maser degree (98) and docor degree (986) in elecommunicaions a he Czech echnical Universiy (Faculy of Elecrical Engineering) in Prague. Since 986 he has been a he echnical Universiy of Liberec (UL). In 999 he became professor a he Dep. of Elecronics and Signal Processing. His major research field is speech ineracion beween human and compuer wih he special focus on speech recogniion. He is he head of he Speech Processing Laboraory (Speech- Lab) a he UL founded by him in 993. He is a IEEE member (Signal Processing Sociey) and a member of he Inernaional Speech Communicaion Associaion (ISCA). Per ČERVA was born in Liberec in 98. In 24 he received maser degree a he echnical Universiy of Liberec (UL) and joined he SpeechLab eam as a PhD suden. His research work is focused on speaker adapaion and speech recogniion of Czech.