Dialog Input Ranking in a Multi-Domain Environment Using Transferable Belief Model

Transcription

1 Dialog Iput Rakig i a Multi-Domai Eviromet Usig Trasferable Belief Model Hog-I Ng Departmet of Computer Sciece School of Computig Natioal Uiversity of Sigapore ghi@comp.us.edu.sg Kim-Teg Lua Departmet of Computer Sciece School of Computig Natioal Uiversity of Sigapore luakt@comp.us.edu.sg Abstract This paper presets results of usig belief fuctios to rak the list of cadidate iformatio provided i a oisy dialogue iput. The iformatio uder cosideratio is the iteded to be performed ad the iformatio provided for the completio of the. As a example, we use the of iformatio access i a multi-domai dialogue system. Curretly, the system cotais kowledge of te differet domais. Cers cig i are greeted with a ope-eded How may I help you? prompt (Thomso ad Wisowaty, 1999; Chu-Carroll ad Carpeter, 1999; Gori et al., 1997). After receivig a reply from the cer, we extract word evideces from the recogized utteraces. By usig trasferable belief model (TBM), we i tur determie the that the cer iteds to perform as well as ay iformatio provided. 1 Itroductio Touch-toe meus are prevalet i c ceters for accessig persoal records ad pre-recorded iformatio. However, it ca sometimes be very frustratig whe we eed to liste to a log list of optios. Moreover, the iformatio that we are lookig for may ot seem to be relevat to ay of the give optios. Recetly, systems that ow people to access iformatio based o spoke iputs have bee built. They require a speech recogizer that is traied o a specific set of key words ad speech grammars to uderstad the spoke iputs. Cers are guided through a series of prompts. At each prompt, the cers are supposed to speak out their choices i a way that is easy for the systems to uderstad. However, ew cers may ot kow what should they say at differet prompts ad how should they say it. They might have spoke their choices too early, or the way they say it is ot ecoded i the systems grammar. Thus, we are motivated to work o the problem of accessig iformatio usig atury spoke dialogue. We ow cers to speak i a atural way. Our ultimate aim is to provide the cer with the exact piece of iformatio that s/he is lookig for through a series of dialogue iteractio. The work reported i this paper is our first attempt toward our ultimate aim, i.e., to determie what the cers wat ad fid out the iformatio the cers have provided that are useful for the. To achieve that, we use Smets (1988) TBM. TBM is the cocept used to justify the use of belief fuctios (BFs), Dempster s rule of coditioig ad Dempster s rule of combiatio to model someoe s belief (Smets, 1988). Sice early 19 s, BFs have geerated cosiderable iterest i the Artificial Itelligece commuity. I Smets (1999), Deœux (000) ad Zouhal ad Deœux (1998), BFs are used to provide soud ad elegat solutios to real life problems where some iformatio is missig. As i Bayesia model, give the available evideces, parts of the amout of belief are ocated to each object i our problem domai. However, some evideces might support somethig other

2 tha oly oe of the various domai objects. I this case, Priciple of Isufficiet Reaso (Smets, 1988) is ivoked to decide that the belief mass must be split equy amog the domai objects ivolved. TBM does ot evoke this priciple ad leaves the belief mass ocated to the disjuct of the domai objects ivolved. Examples of the use of BFs iclude discrimiat aalysis usig a learig set where classes are oly partiy kow; determie the umber of sources i a multi-sesor eviromet by studyig the iter-sesors cotradictio ad patter classificatio. As far as we kow, obody has used BFs to solve problems related to huma-computer coversatioal dialogue. However, we belief that BFs ca be applied o problems related to huma-computer coversatioal dialogue, where the recogized utteraces cotai isertio, deletio ad substitutio errors. Curretly, our multi-domai dialogue system cotais kowledge of te differet domais. They are phoe directory service (T ), trai schedule iquiry (T ), flight status iquiry (T ), travel bookig (T ), Bus Service iquiry (T ), fiacial plaig (T ), phoe bakig (T ), checkig of the employee s accout (T ), cocert ticket bookig (T ) ad course registratio (T ). Similar works have bee reported is the past. However, their mai aim is to do c routig istead of iformatio access. Their approaches iclude the use of a vector-based iformatio retrieval techique (Lee et al., 000; Chu-Carroll ad Carpeter, 1999) /bi/bash: lie 1: a: commad ot foud Our domais are more varied, which may results i more recogitio errors. I additio, we do ot have a traiig corpus. However, we have a kowledge base that provides partial iformatio based o word evideces. For examples, the occurrece of word evidece accout idicates that the user wats to access her/his employee s accout or bak accout, the occurrece of a perso ame idicates that the user is ot checkig for a flight status or bus service, the occurrece of word evidece time idicates that the user probably wats to check the trai schedules or flight status. Due to space limitatio, readers are advised to refer to Smets (1988; 1997; 1989) for more detailed discussios o BFs, combiatio of BFs, decisio makig usig BFs ad TBM. Rakig Iformatio from the Recogized Utterace of Natury Spoke Iput Our aim is to use TBM i dialogue maagemet. First, TBM is used to rak the iformatio idetified from the recogized iput. The, the rak list is used i clarificatio dialogues if ecessary. Otherwise, the best result is treated as the user iput. Our experimets are doe usig Sphix II speech recogitio system (Huag et al., 199). Usig a test corpus of 1977 words, we fid that the word recogitio accuracy is 54.5%. I our experimets, we use 139 atury spoke resposes to a ope-eded How may I help you prompt prompt. The cers are told i advace the list of s that the system ca perform. As otatios, let U deotes a recogized utterace, the legth of U i umber of words ad the word evideces from U..1 Idetifyig the Iteded Task I this experimet, we show whether TBM ca be used to idetify the cer s iteded s. First, we eed to idetify our problem domai or frame of discermet, (Smets, 1988). For idetificatio,! #" $%&'(*), i.e., the list of s preseted i Sectio 1. +-,/. 3, i.e., the basic belief mass (bbm) of give to where 5476#8 is calculated based o the occurrece frequecy of word evidece i the kowledge-bases of /9$:'(. Curretly the kowledge base ; of each! <$= '( cosists of (a) a specificatio; (b) iformatio s for / ; ad (c) iformatio for, i.e., the database records, facts ad rules, ad remote tables ad databases used i /. A specificatio specifies the goal of the ad the list of steps required to attai the goal. Each step is liked to either a basic operatio, for examples, fetch some records from a database ad ask the cer for iformatio, or a sub-. Iformatio s specify the highlevel formats of the iformatio used i >. They iclude database s, XML s of facts ad rules, ad format descriptios of some remote tables ad databases used i. We do idexig for each ;$%'( so that it is easy to calculate the bbm s +,/. 0? ad 4B698. We the do the followig adjustmets to make sure that CEDGF DIH 8 +,/. 0KJ %L : if

3 + 8 CEDIF DGH 8 +,. 0KJ, the the BF +,. is scaled to oe; otherwise, +, C DIF DGH 8 +,. 0KJ where 8. +,/. 0 8 is also ced the igorace value relative to (Smets, 1988). implies that it is harder to decide Larger +,. 8 which is the iteded of the cer by lookig at evidece. The BF s +,. are the combied usig Dempster s rule of combiatio, +,., 0KJ C D +,/. 0 +, 0 where J ad L$. +,/.,, is computed by combiig +,/., ad +,. Lastly,! <$ '( are raked i descedig order accordig to their pigistic probability measure "!$#&% 0 C(' H 8 ) '.- ', * +,3,54 '64 with the top of the rak * +0/1 beig the most probable target. Experimet results will be preseted i Sectio Idetifyig the Provided Iformatio I this experimet, we show whether TBM ca be used to idetify the iformatio provided by the cer i U. Here, the frame of discermet 0798;: cosists of the objects i the iformatio s for a specific. As i Sectio.1, we use the idices of ;9$ 4 '( to compute the bbm s of give to each object disjuct <>= @?"A B. Lastly, we combie the BFs +,. ad compute the pigistic probability measures of each object < 4 07C8;:. Experimet results will be preseted i Sectio Experimet Results 3.1 Idetifyig the Iteded Task 0 ifo + +ifo +ifo Figure 1: Percetage of time the correct is icluded whe cosiderig the top raked s. Figure 1 shows the results of selectig top--s i the raked list of / <$ '(. The labels, ad ifo deote that oly kowledge i the specificatios, iformatio s ad basic iformatio respectively are icluded i the calculatio of bbm s. + deotes a combiatio of some ad deotes the combiatio of. The graphs show that we obtai the best rakig of cadidate s whe kowledge from specificatios ad iformatio s are used to calculate the BF s. This is ituitive because cers will ofte say her/his goal ad metio the ame of the piece of iformatio s/he s lookig for, e.g., I wat to buy a movie ticket please. 0 ifo + +ifo +ifo Figure : Percetage of time the correct is icluded whe cosiderig the top raked s, takig similar words ito cosideratios. Next, we examie the result of takig similar words ito cosideratios. This is because cers may use words differet from those occurrig i our kowledge base. Thus, for each word evidece i D, we use WordNet (Fellbaum, 1997) to look for similar words FE $- G i our kowledge base. For each E $%G, we calculate the BF +,H. as discussed i Sectio.1. This time, we also multiply the bbm s i +,H. by the distace measure betwee E ad. The distace measures f i the rage [0:1]. These results are show i Figure. Agai, the results show that we obtai the best rakig of cadidate s whe kowledge from specificatios ad iformatio s are used to calculate the BF s. However, there is a decrease i correct rate whe oly the best (-6.5%) ad -best (-1.58%) s i the raked list are used to ow

4 0 ifo + +ifo +ifo ifo + +ifo +ifo Figure 3: Percetage of time the correct is icluded whe cosiderig the top raked s, takig similar words ad correlatio measures ito cosideratio. the cers to select. The correct rate is icreased oly whe more tha top-rakig s are used for cers selectio, i.e., 4.38%, 1.3%,.66% ad 1.3% whe = 3, 4, 5 ad 6 respectively. From the results, we foud that some words occur commoly across multiple domais. This pheomea is commo i problems related to atural laguage processig. To eviate the problem, we have used words that oly occur commoly i few domais. We use correlatio coefficiet (Ng et al., 1997) to measure the correlatios of words to domais. After that, we scale the correlatio measures to 1. I calculatig the bbm s, we multiply the origial bbm s with the correspodig correlatio measures. Figure 3 shows the results whe similar words ad correlatio measures are cosidered i the calculatio of BF s. This time, the results show that we obtai the best rakig of cadidate s whe kowledge from specificatios ad basic iformatio are used to calculate the BF s. I additio, there is a 67.31% improvemet whe the top i the raked list is take as the cer s iteded s. Whe top- s are used for cers selectio, the improvemets are.5%, 9.53% ad 16.76% for =, 3 ad 4 respectively. For the purpose of compariso, we show the results of idetificatio based o dialogue trascripts, similar words ad correlatio measures i Figure 4. The results show that with the use of oly basic iformatio i the calculatio of BF s, a result of 99.1% ca be achieved by select the top Figure 4: Percetage of time the correct is icluded whe cosiderig the top raked s usig dialogue trascripts, similar words ad correlatio measures. i the raked list. Thus, whe the word accuracy of the speech recogizer is high, basic iformatio is sufficiet to idetify the cers iteded s. Otherwise, kowledge from specificatios ad iformatio are required i target idetificatios. We have show that TBM ca be used for idetificatio i a oisy ad multi-domai eviromet. It would be iterestig to compare these results whe we have eough corpus to trai a vector-based idetifier. 3. Idetifyig the Provided Iformatio Figure 5 shows the percetage of time the correct iformatio is icluded i the top- selected iformatio after they have bee sorted accordig to their pigistic probability measures. SR-best-1 (SR-best-) idicates that the best (respectively, two best) speech recogitio results are used for iformatio idetificatio. The results show that there is a 14.5% (.54%) improvemet whe the best (respectively, two best) speech recogitio results are used for iformatio idetificatio. Trascript idicates that the dialogue trascripts are used for iformatio idetificatio. The results show that there is a average of 63.79% iformatio lost betwee trascript ad SR-best-. 4 Coclusio A ew atury spoke dialogue processig based o the TBM has bee preseted. This approach

5 0 0 SR-best-1 SR-best- trascript Figure 5: Correct idetificatio rate usig the top iformatio i the rak. ca be viewed as lookig for evideces from oisy speech iputs to idetify the s that the cers wat to perform ad the iformatio that they have provided. Our experimets are tested o a multidomai eviromet. The speech recogizer that we use has a word accuracy of aroud 55%. The experimet results show that there is some iitial success i usig TBM to aid i ad iformatio idetificatio whe the recogized iput is oisy. I order to improve users satisfactio, we are lookig ito dialogue processig methods that are able to improve the results of ad iformatio idetificatio. I particular, istead of usig word evideces from the recogized iputs, we are lookig ito the use other evideces such as phoemes. We are also lookig ito dialogue strategies that are able to collaborate with the cers to correct the idetified iformatio. I particular, if the igorace value +E0 8 is high, our system should employ system iitiative strategies to disambiguate the idetified iformatio. If +E0 8 is high, which meas that the evideces do ot poit strogly to ay object i, the our system should employ system iitiative strategies to lear ew -related iformatio. If both +E0 8 ad +E0 8 are low, out system ca employ a mixed iitiative dialogue strategy. Ackowledgmets Our thaks go to the udergraduate studets who have cotributed their valuable time to help us i the recordigs without askig for ay rewards. Refereces Chu-Carroll, Jeiffer ad Bob Carpeter Vectorbased atural laguage c routig. Computatioal Liguistics, 5(3): Deœux, Thierry A eural etwork classifier based o Dempster-Shafer theory. IEEE Trasactios o Systems, Ma, ad Cyberetics Part A: Systems ad Humas, ():131 1, March. Fellbaum, Christiae (Ed) WordNet: A Electroic Lexical Database. Imprit Cambridge, Mass: MIT Press. Gori, Alle L., Giuseppe Riccardi ad Jeremy H. Wright How may I help you? Speech Commuicatio, 3: Huag, Xuedog, Fileo Alleva, Hsiao-Wue Ho, Mei- Yuh Hwag, Roald Rosefeld The SPHINX- II speech recogitio system: a overview. Computer Speech ad Laguage, 7(): Lee, Chi-Hui, Bob Carpeter, Wu Chou, Jeifer Chu- Carroll, Wolfgag Reichl, Atoie Saad ad Qiru Zhou O atural laguage c routig. Speech Commuicatio, 31(4):9-, Aug. Ng, Hwee Tou, Goh Wei Boo ad Low Kok Leog Feature selectio, perceptro learig, ad a usability case study for text categorizatio. I Proceedigs of the 0th Iteratioal ACM SIGIR Coferece o Research ad Developmet i Iformatio Retrieval, Philadelphia, Pesylvaia, USA. Smets, Philippe Practical uses of belief fuctios. Ucertaity i Artificial Itelligece: Proceedigs of the Fifteeth Coferece (UAI-1999), Morga Kaufma Publishers, Sa Fracisco, CA, Smets, Philippe Costructig the pigistic probability fuctio i a cotext of ucertaity. Ucertaity i Artificial Itelligece 5. Herio M., Shachter R. D., Kaal L. N. ad Lemmer J. F. (Eds). North Hollad, Amsterdam, 9. Smets, Philippe Belief fuctios. No-stadard Logic for Automated Reasoig. P. Smets, A. Mamdai, D. Dubois, ad H. Prade (Eds). New York: Academic, Smets, Philippe The axiomatic justificatio of the trasferable belief model. Artificial Itelligece, 9:9 4. Thomso, David L. ad Jack J. Wisowaty User cofusio i atural laguage services. I Proc. ESCA Workshop o Iteractive Dialogue i Multi-Modal Systems, Kloster Irsee, Germay, Jue, , keyote address. Zouhal, La Merieme ad Thierry Deœux A evidece-theoretic k-nn rule with parameter optimizatio. IEEE Trasactios o Systems, Ma ad Cyberetics Part C, 8():63-71.