Fast Multi-task Learning for Query Spelling Correction

Fas Muli-ask Learning for Query Spelling Correcion Xu Sun Dep. of Saisical Science Cornell Universiy Ihaca, NY 14853 xusun@cornell.edu Anshumali Shrivasava Dep. of Compuer Science Cornell Universiy Ihaca, NY 14853 anshu@cs.cornell.edu Ping Li Dep. of Saisical Science Cornell Universiy Ihaca, NY 14853 pingli@cornell.edu ABSTRACT In his paper, we explore he use of a novel online muli-ask learning framework for he ask of search query spelling correcion. In our procedure, correcion candidaes are iniially generaed by a ranker-based sysem and hen re-ranked by our muli-ask learning algorihm. Wih he proposed muliask learning mehod, we are able o effecively ransfer informaion from differen and highly biased raining daases, for improving spelling correcion on all daases. Our experimens are conduced on hree query spelling correcion daases including he well-known benchmark daase. The experimenal resuls demonsrae ha our proposed mehod considerably ouperforms he exising baseline sysems in erms of accuracy. Imporanly, he proposed mehod is abou one order of magniude faser han baseline sysems in erms of raining speed. Compared o he commonly used online learning mehods which ypically require more han (e.g.,) 60 raining passes, our proposed mehod is able o closely reach he empirical opimum in abou 5 passes. Caegories and Subjec Descripors H.3.3 [Informaion Sorage and Rerieval]: Informaion Search and Rerieval Query Aleraion General Terms Algorihms, Performance, Experimenaion Keywords Query Spelling Correcion, Muli-ask Learning 1. INTRODUCTION Search queries presen a paricular challenge for radiional spelling correcion mehods, for a leas hree reasons [3]. Firsly, spelling errors are more common in search queries han in regular wrien exs. For example, [1] Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. To copy oherwise, o republish, o pos on servers or o redisribue o liss, requires prior specific permission and/or a fee. CIKM 1, Ocober 9 November, 01, Maui, HI, USA. Copyrigh 01 ACM 978-1-4503-1156-4/1/10...$15.00. showed ha roughly 10-15% of queries conain misspelled erms. Secondly, mos search queries consis of only a few key words raher han grammaical senences, making a grammar-based approach inappropriae. Mos imporanly, many queries conain search erms, such as proper nouns and names, which are no well esablished in he language. For example, Chen e al. [10] repored ha 16.5% of valid search erms do no occur in heir spelling lexicon, which conains more han 00,000 enries. Due o he pracical imporance, query spelling correcion has received much aenion and a variey of daases have been developed (from differen domains). As discussed in [17], queries in one specific daase can be very biased from anoher daase, and combining such biased daases for query spelling correcion is difficul. In fac, in his paper, we will also demonsrae an empirical phenomenon ha simply merging biased daases for raining a unified speller may no bring improvemen a all. The difficuly in effecively combining biased daases for spelling correcion lies in a leas hree aspecs: (i) heir error-paerns can be high biased; (ii) heir disribuions of misspelled queries can be quie differen; (iii) heir domains can be quie differen (hence, a domain adapaion problem exiss). Thus, a naural quesion arises: how can we effecively inegrae such highly biased daases for improving query spelling correcion? To he bes of our knowledge, here are sill no saisfacory soluions in he lieraure. Our goal in his paper is o solve his well-known difficul problem wih high accuracy as well as high efficiency. The raining speed can be crucial because indusrial query spelling correcion asks may have o deal wih very largescale raining daa (e.g., click-hrough query spelling logs which conain more han millions of enries, see [34]). Our proposed muli-ask learning framework for query spelling correcion is capable of adapively inegraing highly biased raining daases via auomaically learning he askrelaionships (daa-similariies). This allows us o sofly merge biased daases for learning a unified speller. In our procedure, he fis sep is o generae a large number of spelling correcion candidaes using a candidae generaion module. Then he op candidaes are accuraely idenified hrough our novel muli-ask learning algorihm. Encouragingly, wih he abiliy o effecively ransfer informaion among hose biased raining daases, our proposed mehod is capable of improving spelling correcion accuracies on all relaed daases. The prior sandard pracice for combining differen daases is o simply merge hem and feed hem o a single-ask learner. Compared o his 85

baseline, our experimenal resuls will demonsrae ha our new approach will resul in significanly beer performance. Our major conribuions can be summarized as follows: To he bes of our knowledge, his is he firs sudy of muli-ask learning for query spelling correcion. The proposed mehod ouperforms he sae-of-he-ar resuls on he daase. Our novel adapive muli-ask learning framework can effecively uilize nd-order gradien informaion. Our mehod is of fas convergence speed. I is able o approach close o he empirical opimum in only a few (e.g., 5) passes. This is abou one order of magniude faser han radiional raining (e.g., bach or online) mehods. Fas raining speed is crucial for large-scale query spelling correcion asks (e.g., [34]). The res of his paper is srucured as follows. Secion describes he archiecure of our spelling correcion sysem. Secion 3 presens he proposed muli-ask learning framework for query spelling correcion. Secion 4 presens he exensive experimens on query spelling correcion. Secion 5 reviews relaed work, and Secion 6 concludes he paper.. SYSTEM ARCHITECTURE.1 Candidaes Generaion We firs describe our procedure for generaing correcion candidaes. Following Ganjisaffar e al. [17], we implemen hree candidae generaors in our sysem o produce abou,000 candidaes (on average) for each query. Iniially, a characer-based candidae generaor is adoped for producing all possible candidaes wihin an edi disance 1. I considers replacing each characer wih all possible characers in he alphabe, ransposing each pair of adjacen characers, deleing each characer, insering all possible characers afer each characer, and so on. The AOL query logs showed ha 16% of he query correcions are differen from he original query only in adding/removing spaces [17]. For example, ebayaucion is a query which should be correced o ebay aucion. As a more complicaed example, he long query broccoliandcheesebake acually indicaes broccoli and cheese bake. In order o handle his class of queries, we implemen he word segmenaion algorihm based on he Microsof word breaker 1, wih some minor modificaions. For each possible segmenaion of he characer sequence, we use a language model o compue he probabiliy of he segmenaion. The mos probable segmenaion candidaes are hen added o he candidae lis. Neiher of he above wo candidae generaors is able o produce candidaes which have more han 1 edi disance from he original query (in spie of adding or removing muliple spaces). For example, for he query washon universiy, we migh wan o have washingon universiy as a candidae, which however can no be produced by he above wo candidae generaors. To ease his problem, we perform a fuzzy search process based on a lexicon conaining mos frequen words, o quickly find known unigrams wih a small 1 hp://web-ngram.research.microsof.com edi disance o unigrams and bigrams locaed in he original query. For he lexicon conaining frequen-word, we use he op 100K words based on heir frequency on he web. Wih he hree candidae generaors, we can generae,000 candidaes (on average) for each query.. Naive Ranking An imporan facor in selecing and ranking he correcion candidaes is he prior probabiliy of a correcion phrase. I represens our prior belief abou how likely a query will be chosen by he user wihou seeing any inpu from he user. In his work we make use of he Web n-gram service provided by Microsof. Web n-gram model inends o model he n-gram probabiliy of English phrases wih he parameers esimaed from he enire Web daa. I also differeniaes he daa sources o build differen language models from he ile, anchor ex and body of Web pages, as well as he queries from query log. To build our spelling sysem, we make use of he ri-gram language model. Noe ha, even hough he Web n-gram model conains a large amoun of enries, i may sill suffer from daa sparseness in higher-order language models..3 Re-Ranking Re-ranking of op search resuls has shown o improve he qualiy of rankings in query spelling correcion [17, 8]. Since he main focus of spelling correcion is on he op ranked lis of candidaes, we add a re-ranker on op of he naive ranker. This sep significanly improved he resuls of he naive ranker, and he cos is racable. This is because re-ranking only needs o deal wih op-k candidaes, and k is ypically small (in our case, following [8], we se k = 40). Hence, he re-ranker is solving a much easier problem, compared o he original naive ranker which needs o consider abou,000 candidaes for each query. We use a condiional log-linear mehod, he Maximum Enropy model, for re-ranking. The maximum enropy model produces a probabiliy disribuion over muliple classes and have he advanage of handling large numbers of overlapping feaures. Assuming a feaure funcion ha maps a pair of observaion sequence x and a classificaion label y o a feaure vecor f, he probabiliy of y condiioned on he observaion sequence x is modeled as follows [3]: P (y x, w) = exp [ w f(y, x) ] y exp [ w f(y,x) ], (1) where w is a weigh vecor. Given a raining se consising of n labeled samples, (x i,y i), for i = 1...n, weighs are gained via maximizing he objecive funcion, n L(w) = log P (y i x i,w) R(w). () i=1 The firs erm of his equaion represens a condiional loglikelihood of he raining daa. The second erm is a regularizer for reducing overfiing. We employe an L prior. In wha follows, we denoe he condiional log-likelihood of each sample, log P (y i x i,w), as l(i, w). Then, we have L(w) = n i=1 l(i, w) w σ. (3) Since he evaluaion is based on expeced F-score, probabiliy informaion is required in compuing expeced F-score. 86

The original condiional probabiliy produced by maximum enropy models is no proper for compuing expeced F- score, because he sum of condiional probabiliies of op-k candidaes are no 1. To deal wih his problem, we recompue he probabiliy of a candidae as follows: P (y) =exp[cp (y)]/ exp[cp (y k )], k =1,...,k where c is a scalar o conrol he densiy of he disribuion. Since essenially c has a similar funcion like he σ in regularizaion, o avoid inroducing new hyper-parameers, we simply se c based on σ..4 Re-Ranking Feaures In he re-ranking phase we add feaures which are exraced from op-k query-candidae pairs. In addiion, oher han he feaures which based on ransformaions beween query-candidae pairs, we also design feaures which consider he op candidaes of he query (i.e., comparisons wih oher candidaes in he op-k lis). In his way, he feaures can include valuable informaion ha may help in he ranking process. We have abou 30 feaure emplaes for each querycandidae pair. These feaures include: error model feaures (e.g., edi disance), candidae language model feaures, query language model feaures, surface feaures capuring differences of he query and he candidae, frequency of he query and he candidae, he rank of he candidae in he naive ranking phase, and so on. Alhough we have used language model scores for naive ranking for producing op candidaes, we keep using Web-scale n-gram language model feaures for re-ranking he op candidaes, and we find such Web-scale n-gram language model feaures are sill quie useful in he re-ranking phase. In he Web-scale n-gram language model feaures, he log of n-gram language model probabiliies of an original query and is candidae correcions are used for re-ranking. In addiion, he average, max, min, and sandard deviaion of he language model probabiliies of he op-k candidaes are employed as feaures..5 Online Learning There are wo major approaches for raining a log-linear model: bach raining and online raining. Bach raining mehods include, for example, seepes gradien descen, conjugae gradien descen (CG), and limied-memory BFGS (LBFGS) [30]. In such raining mehods, gradiens are compued by using all raining insances. Typically, he raining process is quie slow in pracice. To speed up he raining process, online algorihms have become increasingly popular. A represenaive online learning mehod is he sochasic gradien descen (SGD) [7]. The SGD uses a small randomly-drawn subse of he raining samples o approximae he gradien of he objecive funcion, which allows one o updae he model weighs much more frequenly, and consequenly, o speed up he convergence. Suppose Ŝ is a randomly drawn subse of he full raining se S, he sochasic objecive funcion is hen given by L soch (w, Ŝ) = i S l(i, w) Ŝ w S σ. The exreme case is a bach size of 1, and i gives he maximum frequency of updaes, which we adop in his work. In his case, Ŝ =1and S = n (suppose he full raining se conains n samples). In his case, we have where L soch (w, Ŝ) =l(i, w) 1 w n σ, (4) Ŝ = {i}. Model weighs are updaed like his: w k+1 = w k + γ k wk L soch (w, Ŝ), (5) where k isheupdaecouner,γ k is he learning rae. 3. OUR PROPOSAL In his secion, we inroduce our adapive (and online) muli-ask learning framework ( below). For every posiive ineger q, we define N q = {1,...,q}. Le T be he number of asks which we wan o simulaneously learn. For each ask N T,herearendaa examples {(x,i,y,i) :i N n} available. In pracice, he number of examples per ask may vary bu we keep i consan for simpliciy of noaion. We use D o denoe he n T marix whose -h column is given by he vecor d of daa examples. 3.1 Adapive Muli-Task Learning Model Our goal is o learn he weigh vecors w 1,...,w T from he daa D. For simpliciy of noaion, we assume ha each of he weigh vecors is of he same size f (his is also he feaure dimension), and corresponds o he same ordering of feaures. We use W o denoe he f T marix whose -h column is given by he vecor w.welearnw by maximizing he objecive funcion, Obj(W,D) Likelihood(W,D) R(W ), (6) where Likelihood(W,D) is he accumulaive likelihood over all ineracive asks, namely, Likelihood(W,D) = N T L(w,D), (7) and L(w,D) isdefinedasfollows: L(w,D) [ α, L(w,d ) ]. (8) N T α, is a real-valued ask-similariy, wihα, = α, (symmeric). Inuiively, a ask-similariy α, measures he similariy of paerns beween he -h ask and he -h ask. L(w,d )isdefinedasfollows: L(w,d ) log P (y,i x,i,w ) i N n = (9) l (i, w ), i N n where P ( ) is a prescribed probabiliy funcion. We can flexibly use any prescribed probabiliy funcion. This makes our mehod a flexible and general framework for no maer srucured or non-srucured classificaion asks. In his paper, we will adop he maximum enropy probabiliy funcion (Eq. 1), which works well for spelling correcion. Finally, R(W ) is a regularizaion erm for dealing wih overfiing. In his paper, we simply use L regularizaion: R(W )= w. (10) σ N T 87

wih fixed ask-similariies (-F) Inpu: Iniialize W (0) ;givend, A,β; k 0 for 1oT. Iniialize η (0). Repea unil convergence.. g 1 n w w σ.. for 1oT... Draw i N n a random... g g + A, w l (i, w ).. w (k+1) w (k).. if k +1mod=0... v i w(k+1) w (k) + η (k) g (i) w (k) (i) w (k 1) (i) (i)... Lower-bounds v i wih β... η (k+1) v η (k).. else... η (k+1) η (k).. k k +1 Oupu:, w (k) converges o w : W (k) converges. wih unknown ask-similariies () Inpu: Iniialize W (0),A (0) ;givend; k 0 Repea unil convergence. W (k+1) -F(W (k),a (k),d). for 1oT.. for 1oT... Updae A (k+1), wih Eq.0/1. k k +1 Oupu: A (k) converges o ÂA; W (k) converges o ŴW. Figure 1: algorihms (using bach size of 1). The derivaion of 1 before he regularizaion erm n was explained in Eq. 4. Lower-bounding v i wih β is for sabiliy consideraion in he online seing. To summarize, he overall objecive funcion is as follows: Obj(W,D) =, N T [ α, ] l (i, w ) i N n N T w σ. To simplify denoaions, we inroduce a T T marix A, such ha A, α,. We also inroduce a T T funcional marix Φ, such ha Φ, L(w,d ). Then, he objecive funcion can be compacly expressed as follows: Obj(W,D) =r(aφ ) N T w, σ (11) In he following conen, we will firs discuss a simple case ha he ask-similariy marix A is fixed. Afer ha, we will focus on he case ha A is unknown, because he askrelaionships are unknown among he hree query spelling correcion asks ha we will focus on. 3. wih Fixed Task-Similariies Alhough he ask-similariies are unknown for query spelling correcion, we will presen a learning algorihm ha ieraively reduce he problem o a case of fixed asksimilariies. Hence, i is imporan o discuss he case of fixed ask-similariies. Wih fixed ask-similariies, he opimizaion problem is as follows: [ W =argmax r(a Φ ) ] w. (1) W N T σ I is clear o see ha we can independenly opimize w and w ( ) given fixed ask-similariies. Hence, we can independenly opimize each column of W and derive W : w =argmaxψ(w,d), (13) w where ψ(w,d) has he form as follows: ψ(w,d) = [ ] α, L(w,d ) N T w. (14) σ 3..1 nd-order Gradien Informaion For high convergence speed, an imporan issue of - F is o effecively and efficienly approximae he Hessian marix. Following he work of [0] on single-ask learning, we presen a simple ye effecive mehod o approximae he eigenvalues of he Jacobian marix of a fixed poin ieraive mapping. In -F, he updae formula is as follows: w (k+1) = w (k) + η g, (15) The updae erm g is derived by weighed sampling over differen asks. The weighed sampling is based on fixed ask-similariies, A. g has a form as follows: g = [ ] α, w l (i,w ) 1 w n w, (16) N T σ where α, = A, and i indexes a random sample seleced from d. Then, he expecaion (over disribuion of daa) of he updae erm is as follows: E(g { [ ] } 1 )= α, n w L(w,d ) 1 w n w = 1 n N T { N T = 1 n w ψ(w,d). [ α, w L(w,d ) σ ] } w w In addiion, η R f + is a posiive vecor-valued sep size and denoes componen-wise (Hadamard) produc of wo vecors. As presened in [6], he opimal sep size is he one ha asympoically approaches o H 1, he inverse Hessian marix of ψ(w,d) in our seing. To avoid acually evaluaing H 1, we can approximae H 1 wih is eigenvalues. Following [0], we consider an updae ierae as a fixed-poin ieraive mapping (hough a sochasic one) M. Taking parial derivaive of M wih respec o w,wehave σ J = M = I diag(η )H. (17) By exploiing his linear relaion beween Jacobian and Hessian, we can obain approximae eigenvalues of inverse Hessian using eigenvalues of Jacobian: eigen i(h 1 η (i) )= 1 eigen. (18) i(j ) In addiion, eigen i(j ) can be asympoically approximaed: eigen i(j ) λ i w(k+1) w (k) (i) w (k) (i) (i) w (k 1) (i). (19) 88

When k is sufficienly large, λ i will be sufficienly close o eigen i(j ). Therefore, we can asympoically approximae he inverse of he Hessian marix via efficien esimaion of he Jacobian marix of fixed-poin mapping. In muli-ask seing, his opimizaion problem is a cossensiive opimizaion problem. To summarize our discussion, we presen he nd-order gradien based algorihm wih fixed ask-similariies (-F below) in Figure 1 (upper). As we can see, he algorihm ieraively approximaes he eigenvalues of he inverse Hessian marix via Eq. 19. This approximaion is based on daa poins ha are cossensiively sampled from muliple asks. Then, i updaes he vecor-valued sep size based on he inverse Hessian informaion, and he muli-ask model weighs are updaed accordingly. This ieraive process coninues unil convergence. 3.3 wih Unknown Task-Similariies For our query spelling correcion asks, he asksimilariies are unknown. To solve his problem, we presen an algorihm o learn he ask-similariies and model weighs in an alernaing opimizaion manner. Our alernaing learning algorihm wih unknown ask-similariies, called, is presened in Figure 1 (boom). In he learning, he -F algorihm is employed as a subrouine. In he beginning of he, model weighs W and ask-similariies A are iniialized. W is hen opimized o ŴW by using he -F algorihm, based on he fixed A. Then, in an alernaive way, A is updaed based on he opimized weighs ŴW. Afer ha, W are opimized based on updaed (and fixed) ask-similariies. This ieraive process coninues unil empirical convergence of A and W. In updaing ask-similariies A based on W, a naural idea is o esimae a ask-similariy α, based on he similariy beween weigh vecors, w and w. I is unclear which similariy measure bes fis he asks of query spelling correcion. To sudy his, we propose wo candidae similariy measures for query spelling correcion: Polynomial kernel (Poly): We can use (normalized) polynomial kernel o esimae similariies: α, 1 w,w d C w d w, (0) d where w,w means inner produc beween he wo vecors; d is he degree of he polynomial kernel; w d w d is he normalizer. C is a real-valued consan for uning he magniude of ask-similariies. Inuiively, a big value of C will resul in weak muli-asking and a small value of C will make srong muli-asking. For example, when d = 1, he normalized kernel has exacly he form 1 cos θ, where C θ is he angle beween w and w in he Euclidean space. Correlaion (Cor): Sincehecovariance of ask weigh vecors is a naural way o esimae iner-ask ineracions, we consider using covariance informaion for esimaing ask-similariies. However, we find direcly using a covariance marix (o esimae ask-similariies) faces he problem of sabiliy in our online seing. Hence, we use he correlaion marix via normalizing he covariance marix. α, 1 C cor(w,w )= 1 cov(w,w ) C sd(w )sd(w ), (1) Table 1: Saisics of he hree spelling correcion daases, and he op-1 accuracy of naive ranking (wihou re-ranking). #Candidaes is he number of correcion candidaes generaed for re-ranking. Daa #Queries #Candidaes Top-1 Acc. of NR (%) 1. 10 4 4.8 10 5 45.7 5.0 10 3.0 10 5 35.5 6.0 10 3.4 10 5 31.0 where cov(w,w ) is he covariance beween w and w. sd( ) issandard deviaion. 3.4 Acceleraed Learning The learning algorihm can be furher acceleraed. The naive learning algorihm wais for he convergence of he model weighs W (in he -F sep) before updaing he ask-similariies A. In pracice, we can updae ask-similariies A before he convergence of he model weighs W. For example, we can updae ask-similariies A afer running he -F sep over a small number of raining passes. We will adop his acceleraed version of he learning for experimens so ha he ask-similariies will no be repeaedly updaed. In he experimen secion, we will compare he acceleraed mehod wih a variey of srong baseline mehods. 4. EXPERIMENTS ON TARGET DATA We will es he proposed mehod on hree query spelling correcion asks. We will sudy he mehod wih differen similariy measures for spelling correcion, and compare hem agains a number of srong baselines, including radiional bach and online learning mehods. The resuls have been averaged over 5 runs for random permuaions of he raining daa order, and sandard deviaions are given. 4.1 Daases Since he daase (released by Microsof Speller Challenge 011) is a new and well-known benchmark daase for query spelling correcion, we will use his daase for experimens. The daase is based on he publicly available queries (008 Million Query Track). This daase conains 5,8 queries and correcions annoaed by he Speller Challenge organizers. There could be more han one plausible correcions for a query. In his daase only 5.3% of queries are judged as misspelled. We use he same spli of he raining and esing daa. To improve he performance of query spelling correcion, we use wo auxiliary daases for performing muli-ask learning. One auxiliary daase is he daase [17] colleced from he publicly available AOL query ses, wih a oal of 1,000 samples, and,000 of hem (16.7%) are differen from he original query. The second auxiliary daase is colleced from queries, and for each query here is a mos only one correcion. In his daase, abou 11% of queries are judged as misspelled. Also, we evenly spli he wo auxiliary daases ino raining and esing daa. We show he saisics of he hree daases in Table 1. In he able, we also show he op-1 accuracy (accuracy of he op-1 candidae) of naive ranking using language models (wihou 89

Mehods (1 Pass) Mehods ( Passes) Mehods (3 Passes) Mehods (4 Passes) Mehods (5 Passes) 88 88 88 88 88 86 Mehods (1 Pass) 86 Mehods ( Passes) 86 Mehods (3 Passes) 86 Mehods (4 Passes) 86 Mehods (5 Passes) Mehods (1 Pass) Mehods ( Passes) Mehods (3 Passes) Mehods (4 Passes) Mehods (5 Passes) Mehods (1 Pass) Mehods ( Passes) Mehods (3 Passes) Mehods (4 Passes) Mehods (5 Passes) Figure : F-scores of differen mehods in 5 passes. and Merge are based on SGD raining. re-ranking). As we can see, he naive ranking has low accuracies. This reflecs he imporance of he re-ranking. The wo auxiliary daases are biased from he daase in hree aspecs. Firs, he queries in hose daases are from hree differen domains:, AOL, and domains. Second, he disribuion of misspelled queries are very differen: 5.3% in daase, 16.7% in daase, and 11% in daase. Finally, he number of correc spelling suggesions are differen: For he daase, for each query here is only one correcion. On he oher hand, for he and daases, for each query here can be muliple correcions. Hence, i is expeced o be a big challenge o inegrae hose hree highly biased daases for improving spelling correcion. 4. Seings Four baselines are adoped o make a comparison wih he proposed muli-ask learning mehod, including he limiedmemory BFGS bach raining mehod [30] wih single-ask seing (LBFGS-), he limied-memory BFGS bach raining mehod wih merged seing (LBFGS-Merge), he sochasic gradien descen mehod wih single-ask seing (SGD-), and he SGD mehod wih merged seing (SGD-Merge). For he single-ask seing, i uses only he ask s daa o rain he re-ranker of he speller (i.e., no daa from oher spelling correcion asks). For he merged seing, i merges all of he raining daases of differen asks o rain a unified re-ranker for he speller. For he muli-ask learning mehod, is hyper-parameers of similariy kernels are uned in preliminary experimens, and we find using d = 1 worked well. In pracice, we break he symmeric seing of C (in similariy kernels) and se differen values of C for differen asks. We se β = 0.99 for he proposed mehod, following he prior work on single-ask learning [0]. The proposed mehod and he four baseline mehods use exacly he same feaures, which is presened before. For he LBFGS mehod, we use he OWLQN sofware. The hyper-parameers of LBFGS were lef unchanged from he defaul seings of he OWLQN sofware. 4.3 Resuls We use he expeced F1 score as he evaluaion meric. This is he same evaluaion merics as he Speller Challenge 011. Following he previous work of [8], op 40 correcions are used as he defaul seing. Using his seing, we can compare our resuls wih he resuls of [8]. The performance comparisons from 1-pass o 5-pass seings are highlighed in Figure. In his figure, he represens -Poly; represens he SGD- baseline. Similarly, Merge represens SGD-Merge. Wecan hp://web-ngram.research.microsof.com/ spellerchallenge/rules.aspx

Table : Resuls (expeced F1-score, sandard deviaion, and raining ime) of and baselines in 5 passes. -Poly and -Cor represen he mehod wih polynomial kernel and correlaion kernel, respecively. The sandard deviaions have been derived over 5 runs for random permuaions of he raining daa order. Mehod (#passes) F1 (%) F1 (%) F1 (%) Overall F1 (%) Overall Time (sec) LBFGS- (1) (bach) 19.3 15.8 10.8 15.3 (±0.0) 6.0 LBFGS- (5) (bach).6.1 77.7 87.4 (±0.0) 36.4 LBFGS-Merge (1) (bach) 19.5 15.7 10.7 15.3 (±0.0) 5.5 LBFGS-Merge (5) (bach).5.9.4.6 (±0.0) 35.5 SGD- (1) (online).5 88.9.7.1 (±0.4).3 SGD- (5) (online).5 89.8.7.7 (±0.5) 111.3 SGD-Merge (1) (online).6 87.6.4. (±1.1) 4. SGD-Merge (5) (online).9 87.6.3.3 (±1.6) 116.4 -Poly (1) (new).6.3.9.6 (±0.1) 3.1 -Poly (5) (new).1.6 97..0 (±0.1) 6.7 -Cor (1) (new).7.1.9.6 (±0.1).1 -Cor (5) (new)..6 97.3.0 (±0.1) 6.7 Table 3: Resuls of differen mehods on heir convergence. Mehod (#passes) F1 (%) F1 (%) F1 (%) Overall F1 (%) Overall Time (sec) LBFGS- (00) (bach).0.3 97.0.8 1479.1 LBFGS-Merge (00) (bach).0.9 97..7 1360.9 SGD- (60) (online).1..9.7 1370.9 SGD-Merge (60) (online).0.8 97..7 13.4 -Poly (30) (new).3.5 97.3.1 1463.3 -Cor (30) (new).3.5 97.3.1 1468.1 see ha he proposed mehod achieves much beer F- score han all of he baseline mehods on he 1-pass seing. In a similar way, ouperforms all of he baselines on he -pass, 3-pass, 4-pass, and 5-pass seings, and mos of he differences are saisically significan. In general, we find SGD-Merge does no have considerable advanage over he SGD- mehod. This indicaes ha i is difficul o combine such highly biased daases for spelling correcion, and a simple merge of such biased daases does no give considerable improvemen. The experimenal resuls in 5 passes are summarized in Table wih more deails, including he resuls of bach raining, he raining ime of differen mehods, and he comparisons beween differen similariy kernels. In addiion, he resuls of differen mehods on heir convergence sae are shown in Table 3. As we can see, he bach raining (LBFGS) has weak performance in 5 passes, compared wih SGD and online raining mehods. The wo similariy kernels (polynomial one and correlaion one) have very similar performance on he seing. Comparing he 5-pass resuls of wih he baseline resuls on heir convergence, we find he (wih 5 passes) ouperforms he SGD wih 60 passes and LBFGS wih 00 passes. In addiion, he wih 5 passes is quie close o is empirical opimum on convergence. Compared wih no maer he online or bach baselines, he mehod is abou one-order magniude faser in erms of he empirical convergence speed. In addiion, we summarize all F-score curves by changing he number of passes, so ha we can check he raining process. The curves are shown in Figure 3. We also show he F-score curves based on raining ime. As we can see, he mehod can achieve beer performance han baselines, wih fewer raining passes and fewer raining ime. 5. RELATED WORK 5.1 Spelling Correcion Spelling correcion for regular wrien ex is a long sanding research opic. Previous researches can be roughly grouped ino wo caegories: correcing non-word errors and real-word errors. In non-word error spelling correcion, any word ha is no found in a pre-compiled lexicon is considered o be misspelled. Then, a lis of lexical words ha are similar o he misspelled word are proposed as candidae spelling correcions. Mos radiional sysems use a manually uned similariy funcion (e.g., edi disance funcion) o rank he candidaes, as reviewed by [11, ]. During he las wo decades, saisical error models learned on raining daa (i.e., querycorrecion pairs) have become increasingly popular, and have proven more effecive [1, 37]. Real-word spelling correcion is also referred o as conex sensiive spelling correcion (CSSC). Real-word spelling correcion ries o deec incorrec usages of a valid word based on is conex. A common sraegy in CSSC is as follows. Firs, a pre-defined confusion se is used o generae candidae correcions, hen a scoring model, such as a ri-gram language model or naive Bayes classifier, is used o rank he candidaes according o heir conex (e.g., [19, 9]). When designed o handle regular wrien ex, boh CSSC and non-word error speller sysems rely on a pre-defined vocabulary (i.e., eiher a lexicon or a confusion se). However, in query spelling correcion, i is impossible o compile such a vocabulary, and he boundary beween he non-word

.5.5.5.5.5 98 97.5 97.5.5.5 0 0 40 60 Number of Passes 0 0 40 60 Number of Passes 0 0 40 60 Number of Passes.5 0 0 40 60 Number of Passes.5.5.5.5.5 98 97.5 97.5.5.5 0 500 1000 Time (sec) 0 500 1000 Time (sec) 0 500 1000 Time (sec).5 0 500 1000 Time (sec) Figure 3: F-score curves of differen mehods. Upper panels are based on raining passes. Boom panels are based on raining ime (seconds). and Merge are based on SGD raining. and real-word errors is quie vague. Therefore, recen research on query spelling correcion has focused on exploiing noisy Web daa and query logs o infer knowledge abou misspellings and word usage in search queries. Cucerzan and Brill [1] discuss in deail he challenges of query spelling correcion, and sugges he use of query logs. Ahmad and Kondrak [3] propose a mehod of esimaing an error model from query logs using he EM algorihm. Li e al. [5] exend he error model by capuring word-level similariies learned from query logs. Chen e al. [10] sugges using web search resuls o improve spelling correcion. Whielaw e al. [39] presen a query speller sysem in which boh he error model and he language model are rained using Web daa. Research in his direcion also includes more recen work on uilizing large web corpora and query logs [10, 17, 8], employing large-scale n-gram models [17, 8], raining phrase-based error model from clickhrough daa [34, 18], and so on. Oher relaed work includes [38, 15, 3, 36]. 5. Muli-ask Learning Muli-ask learning has been he focus of much ineres in machine learning socieies over he las decade. Tradiional muli-ask learning mehods include: sharing hidden nodes in neural neworks [8]; feaure augmenaion among ineracive asks [13]; producing a common prior in hierarchical Bayesian models [4, 43]; sharing parameers or common srucures on he learning or predicor space [4, 4]; muliask feaure selecion [41]; and marix regularizaion based mehods [5, 40], among ohers. Recen developmen of muli-ask learning is online muliask learning, sared from [14]. [14] assumes he asks are relaed by a global loss funcion and he goal is o reduce he overall loss via online algorihm. Wih a similar bu somewha differen moivaion, [1] and [] sudied alernae formulaions of online muli-ask learning under radiional exper advice models. This is a formulaion o exploi low dimensional common represenaions [16, 31]. Online muliask learning is also considered via reducing misake bounds [9], and via percepron-based online muli-ask learning [33]. One of our arge is adapive online muli-ask learning. Our adapive (online) muli-ask learning mehods no only learn model weighs, bu also learn ask relaionships simulaneously from daa [35]. More imporanly, he proposed mehod can effecively esimae nd-order informaion, so ha we can achieve very fas convergence of he muli-ask learning. Finally, our proposal is a general framework which allows non-srucured and srucured classificaion. 6. CONCLUSIONS AND FUTURE WORK In his paper, we proposed an adapive (and online) muliask learning mehod o inegrae highly biased daases for query spelling correcion. We performed experimens on hree differen query spelling correcion daases, including he well-known benchmark daase. Experimenal resuls demonsraed ha he proposed mehod considerably ouperformed he exising baseline sysems in erms of accuracy. Imporanly, he proposed mehod was abou one-order magniude faser han baseline sysems in erms of raining speed. In conras o he baseline mehods which require more han 60 passes in raining, he proposed mehod can approach very close o he empirical opimum in five passes. Since query spelling correcion can have very large-scale raining daa in indusrial applicaions (e.g., query spelling logs which conain more han millions of enries, see [34]), he proposed mehod s abiliy o approach empirical opimum in 1-pass or a few passes will be of criical imporance in such large-scale applicaions. The proposed muli-ask learning mehod is a general echnique, and i can be easily applied o oher classificaion asks. As fuure work, we plan o apply his mehod o oher large-scale web search and daa mining asks. Also, we will explore he possibiliy of inegraing online learning wih modern hashing algorihms [6, 7] o solve exremely large-scale problems.

7. ACKNOWLEDGMENTS This work is parially suppored by NSF (DMS-0808864, SES-1131848), ONR (YIP-N00014001), and DARPA (FA-8650-11-1-7149). Xu Sun was a Posdocoral Associae suppored by ONR and NSF. Anshumali Shrivasava is a Ph.D. suden suppored by ONR and NSF. The auhors hank Yanen Li for helpful discussions. 8. REFERENCES [1] Abernehy, J., Barle, P., and Rakhlin, A. Muliask learning wih exper advice. In COLT 07 (007), vol. 4539 of Lecure Noes in Compuer Science, Springer, pp. 484 498. [] Agarwal, A., Rakhlin, A., and Barle, P. Marix regularizaion echniques for online muliask learning. Tech. Rep. UCB/EECS-008-138, Universiy of California, Berkeley, Oc 008. [3] Ahmad, F., and Kondrak, G. Learning a spelling error model from search query logs. In HLT-EMNLP 05 (Ocober 005), pp. 5. [4] Ando, R. K., and Zhang, T. A framework for learning predicive srucures from muliple asks and unlabeled daa. Journal of Machine Learning Research 6 (005), 1817 1853. [5] Argyriou,A.,Micchelli,C.A.,Ponil,M.,and Ying, Y. A specral regularizaion framework for muli-ask srucure learning. In Proceedings of NIPS 07 (007), MIT Press. [6] Benvenise, A., Meivier, M., and Prioure, P. Adapive algorihms and sochasic approximaions. Berlin: Springer (19). [7] Boou, L. Online algorihms and sochasic approximaions. Online Learning and Neural Neworks. Saad, David. Cambridge Universiy Press (1998). [8] Caruana, R. Muliask learning. Machine Learning 8, 1 (1997), 41 75. [9] Cavallani, G., Cesa-Bianchi, N., and Genile, C. Linear algorihms for online muliask classificaion. In COLT 08 (008), Omnipress, pp. 51 6. [10] Chen,Q.,Li,M.,andZhou,M.Improving query spelling correcion using web search resuls. In EMNLP-CoNLL 07 (Prague, Czech Republic, June 007), pp. 181 189. [11] Church, K. W., and Gale, W. A. Probabiliy scoring for spelling correcion. Saisics and Compuing volumn 1 (19), 103. [1] Cucerzan, S., and Brill, E. Spelling correcion as an ieraive process ha explois he collecive knowledge of web users. In EMNLP 04 (Barcelona, Spain, July 004), pp. 300. [13] Daumé III, H. Frusraingly easy domain adapaion. In ACL 07 (Prague, Czech Republic, June 007), pp. 56 63. [14] Dekel, O., Long, P. M., and Singer, Y. Online muliask learning. In COLT 06 (006), vol. 4005 of Lecure Noes in Compuer Science, Springer, pp. 453 467. [15] Duan, H., and Hsu, B.-J. P. Online spelling correcion for query compleion. In WWW 11 (011), ACM, pp. 117 16. [16] Evgeniou, T., Micchelli, C. A., and Ponil, M. Learning muliple asks wih kernel mehods. Journal of Machine Learning Research 6 (005), 615 637. [17] Ganjisaffar, Y., Zilio, A., Javanmardi, S., Ceindil, I., Sikka, M., Kaumalla, S. P., Khaib-Asaneh, N., Li, C., and Lopes, C. qspell: Spelling correcion of web search queries using ranking models and ieraive correcion. In Spelling Aleraion for Web Search Workshop (July 011). [18] Gao, J., Li, X., Micol, D., Quirk, C., and Sun, X. A large scale ranker-based sysem for search query spelling correcion. In COLING 10 (010), pp. 358 366. [19] Golding, A. R., and Roh, D. Applying winnow o conex-sensiive spelling correcion. In ICML (19), Morgan Kaufmann, pp. 18 1. [0] Hsu, C.-N., Huang, H.-S., Chang, Y.-M., and Lee, Y.-J. Periodic sep-size adapaion in second-order gradien descen for single-pass on-line srucured learning. Machine Learning 77, -3 (009), 1 4. [1] Kernighan, M. D., Church, K. W., and Gale, W. A. A spelling correcion program based on a noisy channel model. In COLING (19), pp. 05 10. [] Kukich, K. Techniques for auomaically correcing words in ex. ACM Compu. Surv. 4, 4 (19), 377 439. [3] Laffery, J., McCallum, A., and Pereira, F. Condiional random fields: Probabilisic models for segmening and labeling sequence daa. In ICML 01 (001), pp. 8 89. [4] Lawrence, N. D., and Pla, J. C. Learning o learn wih he informaive vecor machine. In ICML 04 (004), vol. 69, ACM. [5] Li, M., Zhu, M., Zhang, Y., and Zhou, M. Exploring disribuional similariy based models for query spelling correcion. In COLING-ACL 06 (Sydney, Ausralia, July 006), pp. 105 103. [6] Li, P. and Konig, A.C. Theory and Applicaions of b-bi Minwise Hashing. Communicaions of he ACM 54 (011), 101 109. [7] Li, P., Shrivasava, A., Moore, J., and Konig, A.C. Hashing Algorihms for Large-Scale Learning. In NIPS 11 ( Granada, Spain, December 011). [8] Li, Y., Duan, H., and Zhai, C. Cloudspeller: Spelling correcion for search queries by using a unified hidden markov model wih web-scale resources. In Spelling Aleraion for Web Search Workshop (July 011). [9] Mangu, L., and Brill, E. Auomaic rule acquisiion for spelling correcion. In ICML 97 (Nashville, TN, 1997), Morgan Kaufmann, pp. 187 1. [30] Nocedal, J., and Wrigh, S. J. Numerical opimizaion. Springer (1999). [31] Rai, P., and III, H. D. Infinie predicor subspace models for muliask learning. Journal of Machine Learning Research - Proceedings Track 9 (010), 613 60.

[3] Reynaer, M. Characer confusion versus focus word-based correcion of spelling and ocr varians in corpora. IJDAR 14, (011), 173 187. [33] Saha, A., Rai, P., Daumé III, H., and Venkaasubramanian, S. Online learning of muliple asks and heir relaionships. In AISTATS 10 (F. Lauderdale, Florida, 011). [34] Sun, X., Gao, J., Micol, D., and Quirk, C. Learning phrase-based spelling error models from clickhrough daa. In ACL 10 (Uppsala, Sweden, July 010), pp. 66 74. [35] Sun, X., Kashima, H., Tomioka, R., Ueda, N., and Li, P. A new muli-ask learning mehod for personalized aciviy recogniion. In ICDM 11 (011), IEEE. [36] Sun, X., Shrivasava, A., and Li, P. Query spelling correcion using muli-ask learning. In WWW (Companion Volume) (01), A. Mille, F. L. Gandon, J. Misselis, M. Rabinovich, and S. Saab, Eds., ACM, pp. 613 614. [37] Touanova, K., and Moore, R. Pronunciaion modeling for improved spelling correcion. In ACL 0 (Philadelphia, USA, 00), pp. 144 151. [38] Wang, Z., Xu, G., Li, H., and Zhang, M. Afas and accurae mehod for approximae sring search. In ACL 11 (011), pp. 5 61. [39] Whielaw, C., Huchinson, B., Chung, G., and Ellis, G. Using he web for language independen spellchecking and auocorrecion. In EMNLP 09 (009), pp. 8 899. [40] Xue, Y., Dunson, D., and Carin, L. The marix sick-breaking process for flexible muli-ask learning. In ICML 07 (Corvalis, Oregon, 007), ACM, pp. 1063 1070. [41] Yang, H., King, I., and Lyu, M. R. Online learning for muli-ask feaure selecion. In CIKM 10 (010), ACM, pp. 16 16. [4] Yu, K., Tresp, V., and Schwaighofer, A. Learning gaussian processes from muliple asks. In ICML 05 (005), vol. 119, ACM, pp. 101 1019. [43] Zhang, J., Ghahramani, Z., and Yang, Y. Learning muliple relaed asks using laen independen componen analysis. In NIPS 05 (005).