Using Knowledge to Speed Learning: A Comparison of Knowledge-based Cascade-correlation and Multi-task Learning

Transcription

1 Using Knwledge t Seed Learning: A Cmarisn f Knwledge-based Casade-rrelatin and Multi-task Learning Thmas R. Shultz Deartment f Psyhlgy, MGill University, Mntreal, QC H3A 1B1 Canada SHULTZ@PSYCH.MCGILL.CA Franis Rivest FRIVES@PO-BOX.MCGILL.CA Deartment f Cmuter Siene, MGill University, Mntreal, QC H3A 1B1 Canada Abstrat Cgnitive mdeling with neural netwrks unrealistially ignres the rle f knwledge in learning by starting frm randm weights. It is likely that effetive use f knwledge by neural netwrks uld signifiantly seed learning. A new algrithm, knwledge-based asaderrelatin (KBCC), finds and adats its relevant knwledge in new learning. Cmarisn t multi-task learning (MTL) reveals that KBCC uses its knwledge mre effetively t learn faster. 1. Existing Knwledge and New Learning Neural netwrks tyially learn de nv withut the benefit f existing knwledge. Hwever, when ele learn, they rutinely use their knwledge (Pazzani, 1991; Wisniewski, 1995). Suh use f rir knwledge in learning is likely resnsible fr the ease and seed with whih ele learn, and fr interferene with new learning. The tehnial reasn that neural netwrks fail t use knwledge is that they begin learning frm initially randm nnetin weights. This imlements a tabula rasa view f eah distint learning task that very few gnitive syhlgists wuld aet. In this aer, we mare tw algrithms (KBCC and MTL) fr their ability t use knwledge t seed learning. KBCC is an extensin f asade-rrelatin (CC), a generative learning algrithm ften used in the simulatin f gnitive develment (Bukingham & Shultz, in ress; Mareshal & Shultz, 1999; Oshima-Takane, Takane, & Shultz, 1999; Shultz, 1998, 1999; Shultz, Mareshal, & Shmidt, 1994; Siris & Shultz, 1998). CC nstruts its wn netwrk tlgy by reruiting new hidden units int the netwrk as needed in rder t redue errr (Fahlman & Lebiere, 1990). KBCC reruits reviusly learned netwrks in additin t the single hidden units reruited by CC (Shultz & Rivest, 2000). Fllwing terminlgy in the literatures n analgy and transfer, we refer t existing netwrks as tential sure knwledge and t a urrent learning task as a target. Previusly learned sure netwrks mete with eah ther and with single hidden units t be reruited int the target netwrk. Caruana (1993, 1995, 1997) develed multi-task learning (MTL) in whih he trained a netwrk n several tasks taken frm the same dmain in arallel, with a single utut unit fr eah task. Suh netwrks tyially learned a mmn hidden-unit reresentatin, whih rdued better generalizatin than learning the same single tasks ne at a time (STL). MTL an be adated t sequential learning by having a sure netwrk generate resnses t inut values frm a new task. These resnses an then serve as target utut values in arallel MTL f the new task. This aer rerts a marisn f KBCC and MTL n the same sequential learning task. The gals are t determine whether eah algrithm an use sure knwledge t seed learning and t study f the effets f knwledge relevane n learning seed. 2. Previus Wrk n Knwledge and Learning Other revius neural netwrk researh n knwledge and learning has inluded studies f transfer (Pratt, 1993), sequential learning (Silver & Merer, 1996), lifelng learning (Thrun & Mithell, 1993), knwledge insertin (Shavlik, 1994), mdularity (Jrdan & Jabs, 1994), and inut re-ding (Clark & Thrntn, 1997). Pratt (1993) ineered the study f knwledge and learning in neural netwrks with a tehnique alled disriminability-based transfer (DBT). DBT uses the weights frm a reviusly trained netwrk t initialize a new netwrk. This seems the mst straightfrward idea fr using knwledge in new neural learning. Beause it did nt atually wrk very well, Pratt re-saled the

2 revius netwrk's hyer-lanes s that useful nes had large weights and less useful nes had small weights. Silver and Merer (1996) extended MTL t sequential learning in a methd alled task rehearsal (TRM). Here, ld tasks are seud-rehearsed during new learning. In seud-rehearsal, a netwrk generates its wn target vetrs, using its urrent weights, rather than merely aeting them frm the envirnment (Rbins, 1995). In a variatin f MTL, searate learning rates fr eah task are used t ntrl the imat f eah sure task, ensuring that the mst related tasks have the mst imat n learning. Thrun and Mithell (1993) rsed a tehnique they alled lifelng learning, in whih a netwrk meta-learns the sle f the desired funtin at eah training examle. This is the derivative f the funtin at an examle utut with reset t the inut attribute vetr. Then, in new learning, a meta-netwrk redits sles and estimates its auray fr eah new training examle. This tehnique wuld seem t trade nt s muh n knwledge reresentatins as n searh knwledge. Clark and Thrntn (1997) emhasized the imrtane f netwrks being able t re-de their inut in the learning f diffiult, s-alled Tye-2 rblems. Tye-1 rblems are thse that an be slved by samling the riginally ded inut data. Tye-2 rblems need re-ding in rder t use Tye-1 knwledge. Re-ding may require inremental learning, mdularity, and reresentatinal redesritin (Karmilff-Smith, 1992), but n seifi algrithm was rsed. Shavlik (1994) devised an algrithm fr reating knwledge-based artifiial neural netwrks (KBANN). KBANN nverts a set f symbli rules embdying a dmain thery f a rblem int a feed-frward neural netwrk with the final rule nlusins as utut units and intermediate rule nlusins as hidden units. Cnnetin weights and biases are initialized t mimi the njuntive and disjuntive strutures f the riginal rules. Suh knwledge-initialized netwrks are then trained with examles t refine the netwrk's knwledge. Training with KBANN is tyially faster than using standard netwrks with randm weights and leads t better generalizatin. Fllwing training, the mdified rules an be extrated frm the netwrk. Jrdn and Jabs (1994) devised the Hierarhial Mixture f Exerts (HME) mdel t demse rblems int searate netwrk mdules. Distint netwrk mdules beme exert n subtasks, and erate n an verall slutin via gating netwrks that learn t weight the mdular exert ntributins fr artiular arts f a rblem. HME was fund t learn the dynamis f a furdegree-f-freedm rbt arm faster than a multi-layer bak-ragatin netwrk did. Next we desribe in sme detail the tw learning algrithms featured here: KBCC and MTL. 3. Knwledge-based Casade-rrelatin KBCC learns like CC, exet that KBCC treats its reviusly learned netwrks as if they were single andidate hidden units. Bth single units and existing netwrks are andidates fr reruitment int a target netwrk. A andidate unit and a andidate netwrk eah define a funtin that an be differentiated, whih is essential fr weight adjustment by gradient desent. The nnetin sheme fr a samle KBCC netwrk is shwn in Figure 1. This nnetin sheme is the same as in CC exet that a reruited netwrk an have multile weighted sums as inuts and an have multile ututs. In ntrast, a single reruited unit, whether in CC r KBCC, has nly ne weighted sum as inut and ne utut. Outut Inuts Hidden 2 Hidden 1 Figure 1. Third utut hase f a KBCC netwrk in whih the first reruited hidden unit is a reviusly learned sure netwrk with multile inuts and ututs. Dashed lines indiate trainable weights; slid lines indiate frzen weights. Thin lines indiate single weights; thik lines indiate ssible multile weights t and frm the reruited netwrk. Sme ntatinal nventins in ur frmulatin f KBCC: w u, : Weight between utut u f unit u and utut unit. w u, i : Weight between utut u f unit u and inut i f andidate. f, : Derivative f the ativatin funtin f utut unit with reset t its inut at attern. f i, : Partial derivative f andidate utut with reset t its inut i at attern. V, : Ativatin f utut unit at attern. V, : Ativatin f utut f andidate at attern. V, : Ativatin f utut u f unit u at attern. u T, : Target value f utut at attern. KBCC netwrks begin and end their lives in the s-alled utut hase, just as CC netwrks d. In the utut hase, weights entering the utut units are trained with the quikr algrithm (Fahlman, 1988) in rder t redue errr. Weights entering utut units are initialized with unifrm randm numbers within the range f -1 t 1. The

3 funtin t be minimized in the utut hase is the sumsquared errr ver all ututs and all training atterns: ( V T ) F =, 2, Like ther gradient desent algrithms, KBCC requires mutatin f the sle f the funtin t be minimized. The artial derivative f F with reset t the weight w, is F w, u u ( V T ) = 2,, f, V, Outut units an have either sigmid r linear ativatin funtins. As in CC, an utut hase ntinues until sme number f ehs ass withut slutin, errr redutin stagnates fr sme few nseutive ehs, r all utut ativatins are within a seified range f their target values. In eah f the first tw ases, there is a shift t inut hase. In the last ase, learning sts and the system delares vitry. In an inut hase, a new hidden unit is reruited int a netwrk and installed dwnstream f all existing hidden units. The reruited unit is seleted frm a l f andidates. During the reruitment ress, andidates reeive inut frm all existing netwrk units, exet utut units. Inut weights are trained by trying t maximize a rrelatin between ativatin f the andidate and netwrk errr. The andidate that gets reruited is the ne that is best at traking the netwrk's urrent errr. In KBCC, andidates inlude, nt nly single units as in CC, but als reviusly learned sure netwrks. There are N andidates er tye -- single unit and sure netwrk. Weights entering the N single-unit andidates are initialized randmly in a unifrm distributin within the range f -1 t 1. Ativatin funtins f the single units are tyially sigmid, but an be asigmid r Gaussian. Fr eah sure netwrk, inut weights fr N-1 instanes are initialized in the same way. In additin, ne instane f eah sure netwrk has weights f 1 between rresnding inuts f the target and sure netwrks and 0s elsewhere. These identity weights are designed t enable quik use f exat knwledge. The funtin t maximize with quikr in an inut hase is the average variane f the ativatin f eah andidate (indeendently) with the errr at eah utut, nrmalized by the sum squared errr. G = ( V V )( E E ), # O # O E, 2, u In this frmula, E O is the mean errr at utut unit, and V O is the mean ativatin utut f andidate C. G C gets standardized by the number f ututs fr the andidate (#O ) and by the number f ututs fr the main netwrk (#O). Again, the sle f this funtin is required fr weight adjustment. The artial derivative f G with reset t the weight w u, i between utut u f unit u and inut i f andidate is ( E E ) σ,, G = w, i # O # O u E i f 2,, V, Here, σ, is the sign f the variane between the utut f andidate and the ativatin f utut unit. An inut hase ntinues until sme number f ehs asses withut slutin, r at least ne rrelatin reahes a minimum value (default value = 0.2) and rrelatin maximizatin stagnates fr sme few nseutive inut hase ehs. When there is a shift bak t utut hase, weights are reated frm the utut(s) f the best andidate t eah utut unit in the target netwrk. Other andidates are disarded and the new weights are initialized using small randm values with sign site t that in the rrelatin. 4. Multi-task Learning The basi idea underlying MTL is that it an be easier t learn several tasks at ne than t learn them searately (Caruana, 1993). MTL has been demnstrated t imrve generalizatin by biasing netwrks t learn hidden unit reresentatins that are mmn t several related tasks (Caruana, 1993, 1995, 1997). Baxter (1995) rved that the number f examles required fr learning any ne task in a multi-task aradigm dereases as a funtin f ttal number f tasks learned in arallel. Caruana (1997) suggested that MTL uld be alied t sequential learning by using an existing sure netwrk t generate syntheti data that uld be added t the training set f a new, related target task. Assuming that the sure and target tasks have idential inut units and a single utut unit, MTL training an then ur in the usual arallel fashin, as shwn in Figure 2. The new training atterns are assed thrugh the sure netwrk t generate target utut values fr the utut unit reresenting the sure task in the new MTL netwrk. Beause f the requirement that all MTL tasks be learned in arallel, it is nt lear whether MTL wuld inevitably generate faster learning than n knwledge at u

4 all, but Caruana (1995) asserted that "MTB als usually learns in fewer ehs than STB (. 662)." In this ntext, MTB refers t multi-task bak-ragatin and STB refers t single-task bak-ragatin, mre generally alled single-task learning (STL). Oututs Sure task Target task Hiddens Inuts Figure 2. MTL netwrk fr tw tasks. Arrws indiate full layer-t-layer nnetivity. In any ase, Silver and Merer (1996) alied MTL t sequential learning in this fashin (alling it TRM), and fund better generalizatin and faster learning with MTL than with n knwledge using imverished training sets, regardless f whether searate learning rates were used fr eah MTL task. Their rblem dmain was similar t urs, learning t distinguish the inside vs. the utside f a band in varius rientatins. 5. Learning Task T assess the imat f sure knwledge n learning a target task, we varied the relevane f a single sure f knwledge. The idea was t determine whether KBCC and MTL netwrks wuld learn faster if they had sure knwledge that was mre relevant and t assess the degree f seedu. The task invlved learning whether a air f Cartesian rdinates fell inside r utside f a artiular gemetri shae. Sure netwrks varied in terms f shae and translatin. The inut sae was a square entered at the rigin with sides f length 2. Netwrks were trained with a set f 225 atterns frming a regular 15 x 15 grid vering the whle inut sae inluding the bundary. There were 200 randmly determined test atterns distributed unifrmly ver the inut sae but nt used in training. We ran 20 netwrks in eah nditin f eah exeriment in rder t assess the statistial reliability f results. Learning seed was measured by ehs t learn, where an eh is a ass thrugh all f the training atterns. Knwledge relevane was varied by hanging the sitin r shae f the sure knwledge. The target shae in the send hase f knwledge-guided learning was a retangle sized 0.4 x 1.6 entered at (-4/7, 0) in the inut sae (see Figure 3). Figure 3. Outut ativatin diagram shwing the target rblem (left retangle). In the target hase, netwrks had t learn this leftsitined retangle after having reviusly learned a retangle, tw retangles, r a irle entered at the rigin, r t the left, r t the right in the inut sae. The varius exerimental nditins are shwn in Table 1. Table 1. Sure knwledge nditins. Name Centered at Relatin t target Left retangle Left & enter retangles Center retangle Right retangle Center & right retangles -4/7, 0 Exat (-4/7, 0) & (0, 0) Exat/near, verly mlex 0, 0 Near relevant 4/7, 0 Far relevant (0, 0) & (4/7, 0) Near/far, verly mlex Cirle 0, 0 with radius 0.5 Irrelevant Nne N knwledge N relatin Our retangles are similar t the bands used by Silver and Merer (1996) exet that their bands were nt bunded n eah end as ur retangles were. Thus, ur retangles might be exeted t be mre diffiult t learn beause f their greater nn-linearity. In a ntrl nditin, netwrks had n sure knwledge when beginning the target task, essentially similar t rdinary CC netwrks and STL netwrks. Learning sted when all utut unit ativatins were within 0.4 f their target values fr all

5 training atterns. Target values fr ints inside the shae were 0.5; fr ints utside the shae, Predure fr KBCC KBCC netwrks first learned ne f the sure rblems in Table 1 and then learned the target rblem (left retangle). In the ase f the n-knwledge ntrl nditin, there was n sure t learn s the netwrk just learned the target rblem, essentially as a CC netwrk. In inut hases f target learning, the ntrl nditin had 8 single-unit andidates; the ther nditins eah had 4 single-unit andidates and 4 sure-netwrk andidates. In any given KBCC netwrk, the sure netwrks were idential exet fr having fur different sets f initially randm inut weights. This is erfetly analgus t hw several different single andidate units are treated in CC. 5.2 Predure fr MTL Fr MTL, we used the quikr algrithm (Fahlman, 1988) beause rdinary bak-ragatin, ustmarily used in MTL and TRM, uld nt learn ur sure tasks within a reasnable time frame. Tyially, bakragatin netwrks ran fr ver 2000 ehs withut making a ntieable dr in errr, using any f several different mbinatins f learning rate and mmentum. Quikr is a weight training algrithm that uses the sle f errr with reset t weight at the last tw ehs t estimate urvature, the rate at whih sle hanges as a funtin f weight hange. Thus armed with bth sle and an estimate f urvature, quikr mves mre deisively t hange weights in rder t minimize errr than bak-ragatin, whih uses nly sle. Mrever, using quikr t learn weights in MTL ensures a lser marisn t KBCC, whih als uses quikr. We trained a sure netwrk in ne f the knwledge nditins in Table 1, and then nstruted an MTL netwrk t learn bth the sure and target tasks in arallel, using ututs frm the sure netwrk as the target training signal fr the sure utut unit in the MTL netwrk. Training targets fr the ther utut unit in the MTL netwrk were rvided by the target task (left retangle). In the n-knwledge nditin, a netwrk was simly trained n the target task in STL fashin. Twelve hidden units were required t get suessful learning f the sure and target shaes. Learning rate was 0.5. If sure netwrks failed t learn their shae within 2000 netwrks, they were disarded and relaed. The sting riterin was idential fr KBCC, STL, and MTL netwrks -- vitry was delared when all utut units had ativatins within 0.4 f their target values fr all training atterns. We next analyze the learning seed results searately fr eah algrithm t determine whether and t what extent eah algrithm benefits frm knwledge. 6. KBCC Results A fatrial ANOVA f the ehs t vitry in KBCC rdued a main effet f knwledge nditin, F(6, 133) = 33.15, < The mean ehs t vitry, alng with standard deviatin bars and hmgeneus subsets, based n the LSD st h marisn methd, are shwn in Figure 4. The attern f mean differenes reveals that exat knwledge, whether alne r embedded, rdued the fastest learning, fllwed by relevant knwledge, distant and verly mlex knwledge and irrelevant knwledge, and finally the ntrl nditin withut any knwledge. Every knwledge nditin was signifiantly faster than n knwledge at all. Cnditin Left, enter Left Center Right Center, right Cirle Nne Mean ehs t vitry Figure 4. Mean ehs t vitry in the target hase fr KBCC netwrks. Generalizatin tests with the 200 randmly determined test atterns shwed less than 5% mislassifiatin errrs in every nditin. There were n nditin effets n errr, indiating that target rblems were suessfully learned in every nditin. Number f hidden units reruited in the target hase ranged frm 3.95 in the enter retangle nditin t 7.60 in the enter and right retangle nditin, with an verall mean f All but tw f these reruited units were sure netwrks. Outut ativatin diagrams were drawn t understand the knwledge reresentatins ahieved after sure- and target-training hases. Sme examles f these diagrams are shwn in Figures 5-7. In interreting these figures, reall that a netwrk learns a shae by distinguishing ints within it frm ints utside f it. In Figures 5-7, white regins f the inut sae are lassified as being inside the shae, blak regins utside f the shae, and gray areas are unertain, meaning that the netwrk gives a brderline, unlassifiable resnse (in the range -0.1 t 0.1). The hrizntal and vertial lines seen in these figures are the x- and y-axes, resetively, f the inut sae.

6 During target learning, the netwrk an reruit single hidden units r already-learned sure netwrks, suh as thse in Figures 5 and 6. Figure 5 shws a sure netwrk's slutin t the irle rblem. We redited that the irle wuld nstitute an irrelevant sure f knwledge that wuld nt hel target learning very muh beause f the large differene in shae. Figure 6 shws a sure netwrk's slutin t the left and enter retangles rblem. This ntains an exat slutin t the left retangle target rblem, but is embedded in an verly mlex sure. Figure 7 shws a KBCC slutin t the left retangle target rblem based n reruitment f the netwrk whse knwledge reresentatin is shwn in Figure 6. Sures like that in Figure 6 were very effetive in seeding u learning, whereas sures like that in Figure 5 were less effetive, as rerted in Figure 4. in the enter/right nditin, and 3 in the n-knwledge nditin. Eah f these netwrks was given a sre f 2001 ehs. Figure 5. Outut ativatin diagram shwing a sure netwrk's slutin t the irle rblem. The ints in eah f Figures 5-7 are the 225 target training atterns, whih frm a 15 x 15 grid vering the whle inut sae. The learned slutins are irregular beause they result frm testing the netwrk n a fine grid f 220 x 220 inut atterns. Figure 6. Outut ativatin diagram shwing a sure netwrk's slutin t the left and enter retangles rblem. A fatrial ANOVA f the ehs t vitry in MTL netwrks rdued a main effet f knwledge nditin, F(6, 133) = 3.26, <.005. The mean ehs t vitry, alng with standard deviatin bars and hmgeneus subsets, based n the LSD st h marisn methd, are shwn in Figure 8. The attern f mean differenes reveals that the n-knwledge nditin did nt differ signifiantly frm any knwledge nditin, exet exat, verly mlex knwledge (left and enter retangles). In that marisn, netwrks withut any knwledge learned faster than netwrks with exat, but verly mlex knwledge. 7. MTL Results Even with quikr t adjust weights, many netwrks failed t learn within a reasnable number f ehs in bth STL and MTL hases. STL netwrks were disarded and relaed if they did nt learn the sure knwledge task within 2000 ehs. There were 3 disarded STL netwrks in the left retangle nditin, 25 in the enter/right nditin, 16 in the left/enter nditin, and 7 in the irle nditin. All remaining netwrks suessfully learned their sure rblem and were thus arried frward t the target hase. Smaller numbers f MTL netwrks failed t learn the target rblem within 2000 ehs: 1 in the left nditin, 1 in the enter nditin, 5 in the left/enter nditin, 4 Figure 7. Outut ativatin diagram shwing a KBCC slutin t the target rblem in Figure 3 after reruiting the sure rblem in Figure 6.

7 Cnditin Left Cirle Center Nne Right Center, right Left, enter Mean ehs t vitry Figure 8. Mean ehs t vitry in the target hase fr MTL netwrks. 8. Disussin These results shw that nly KBCC netwrks were able t find and adat their existing knwledge in new learning, signifiantly shrtening the learning time. When exat knwledge was resent, KBCC reruited it fr a quik slutin. The mre relevant the sure knwledge, the mre likely it was that KBCC reruited it fr slutin f a target rblem and the faster that new learning was likely t be. Frm any viewint, these are desirable rerties fr a system that effetively uses its knwledge in new learning. On the ther hand, MTL netwrks did nt shw any benefits f knwledge fr learning seed. They had artiular diffiulty extrating exat knwledge frm an verly mlex sure netwrk. Mrever, STL netwrks ften failed t learn their assigned sure rblem and were relaed befre MTL uld be assessed. MTL may nt seed learning f new tasks beause it requires bth ld and new tasks t be freshly learned in arallel. KBCC differs by reruiting, nt relearning, ld knwledge. In ntrast t all revius methds fr using knwledge in learning, KBCC uses established tehniques frm generative learning (Fahlman & Lebiere, 1990). KBCC treats its existing netwrks like single-unit andidates, training weights t the inuts f existing sure netwrks t determine whether their ututs rrelate with the target netwrk's errr. Netwrk reruitment and integratin in KBCC may amlish the inut re-ding needed t nvert diffiult rblems int easier rblems (Clark & Thrntn, 1997). Inuts t a target netwrk are re-ded nt the inuts t a sure netwrk in a way that uld hel t slve the target rblem. In additin, KBCC trains the utut weights frm a reruited netwrk s as t inrrate the reruited netwrk int a slutin f the target rblem. Cnsequently, KBCC is able t use knwledge that is nly artly relevant t the target task. Unlike many f the revius knwledge-based tehniques in whih bth inuts and ututs f the sure and target task must math reisely, KBCC an reruit any srt f funtin t use in a target task. In KBCC, sure netwrk inuts and ututs an be arranged in different rders and numbers and use different ding methds than thse in the target netwrk. The wide range f reruitment bjets ffers mre wer and flexibility than mst knwledgebased learners rvide. In ntrast, MTL requires that the number and rdering f the inuts fr eah task math reisely and that there is a single utut fr eah task. KBCC allws fr a mbinatin f learning by analgy and indutin. It learns by analgy t its urrent knwledge whenever it an and swithes t a mre indutive mde if needed. Reruiting a netwrk an be nsidered as learning by analgy, whereas reruiting a single unit an be regarded as learning by indutin. Bth kinds f learning are seamlessly integrated as KBCC learns a new target task. KBCC derives its wer frm the fat that it an learn t use its existing knwledge t slve a target task rather than having t learn the target task frm srath. In ther exeriments, we studied what sures KBCC selets when it ssesses mre than ne sure f knwledge in its bakgrund (Shultz & Rivest, 2000). When resent, exat knwledge was always referred, even when embedded within verly mlex knwledge. Simle exat knwledge was referred t embedded, verly mlex knwledge. Oasinally, knwledge that we had redited t be irrelevant (irle) was reruited mre ften than knwledge that we had redited t be relevant thugh distant (right retangle). One reasn that s many irrelevant sures were reruited was that they were nt artiularly helful in learning the new rblem (left retangle), and thus rlnged learning. There are still signifiant limitatins t KBCC. One is that single hidden units are rarely reruited. Althugh further study is warranted, ur intuitin is that humans might be smewhat mre likely than this t learn nvel tasks frm srath. One slutin wuld be t enalize the mlexity f the reruited knwledge in sme way. Anther limitatin is that searh thrugh memry fr relevant knwledge raidly bemes mre exensive as knwledge realistially exands. If ther, mre tratable rblems with KBCC an be slved, making memry searh mre effiient will beme a majr gal fr us. Current and future wrk is designed t exlre a wider range f aliatins f KBCC -- t ther kinds f shae transfrmatins suh as rtatin and sizing, syhlgial simulatins f knwledge and learning, and larger-sale realisti rblems. We are als testing whether KBCC an imrve quality f learning with imverished training sets, and whether and when knwledge interferes with learning.

8 Aknwledgements This wrk was surted by a grant frm the Natural Sienes and Engineering Researh Cunil f Canada. We are grateful fr mments n an earlier draft frm David Bukingham, Jaques Katz, Sylvain Siris, Yshi Takane, and annymus reviewers. Referenes Baxter, J. (1995). Learning internal reresentatins. Preedings f the Eighth Internatinal Cnferene n Cmutatinal Learning Thery. Santa Cruz, CA: ACM Press. Bukingham, D., & Shultz, T. R. (in ress). The develmental urse f distane, time, and velity nets: A generative nnetinist mdel. Jurnal f Cgnitin and Develment. Caruana, R. (1993). Multitask learning: A knwledgebased sure f indutive bias. Preedings f the Tenth Internatinal Mahine Learning Cnferene ( ). San Mate, CA: Mrgan Kaufmann. Caruana, R. (1995). Learning many related tasks at the same time with bakragatin. Advanes in neural infrmatin ressing systems 7 ( ). Ls Alts, CA: Mrgan Kaufmann. Caruana, R. (1997). Multitask learning. Mahine Learning, 28, Clark, A., & Thrntn, C. (1997). Trading saes: Cmutatin, reresentatin, and the limits f uninfrmed learning. Behaviral and Brain Sienes, 20, Fahlman, S. E. (1988) Faster-learning variatins n bakragatin: An emirial study. In D. S. Turetzky, G. E. Hintn, & T. J. Sejnwski (Eds.), Preedings f the 1988 Cnnetinist Mdels Summer Shl ( ). Ls Alts, CA: Mrgan Kaufmann. Fahlman, S. E., & Lebiere, C. (1990). The asaderrelatin learning arhiteture. In D. S. Turetzky (Ed.), Advanes in neural infrmatin ressing systems 2 ( ). Ls Alts, CA: Mrgan Kaufmann. Jrdan, M. I., & Jabs, R. A. (1994). Hierarhial mixtures f exerts and the EM algrithm. Neural Cmutatin, 6, Karmilff-Smith, A. (1992). Beynd mdularity: A develmental ersetive n gnitive siene. Cambridge, MA: MIT Press. Mareshal, D., & Shultz, T. R. (1999). Develment f hildren's seriatin: A nnetinist arah. Cnnetin Siene, 11, Oshima-Takane, Y., Takane, Y., & Shultz, T. R. (1999). The learning f first and send rnuns in English: Netwrk mdels and analysis. Jurnal f Child Language, 26, Pazzani, M. J. (1991). Influene f rir knwledge n net aquisitin: Exerimental and mutatinal results. Jurnal f Exerimental Psyhlgy: Learning, Memry, and Cgnitin, 17, Pratt, L. Y. (1993). Disriminability-based transfer between neural netwrks. Advanes in neural infrmatin ressing systems 5 ( ). San Mate, CA: Mrgan Kaufmann. Rbins, A. V. (1995). Catastrhi frgetting, rehearsal, and seudrehearsal. Cnnetin Siene, 7, Shavlik, J. W. (1994). A framewrk fr mbining symbli and neural learning. Mahine Learning, 14, Shultz, T. R. (1998). A mutatinal analysis f nservatin. Develmental Siene, 1, Shultz, T. R. (1999). Rule learning by habituatin an be simulated in neural netwrks. Preedings f the Twenty-first Annual Cnferene f the Cgnitive Siene Siety ( ). Hillsdale, NJ: Erlbaum. Shultz, T. R., Mareshal, D., & Shmidt, W. C. (1994). Mdeling gnitive develment n balane sale henmena. Mahine Learning, 16, Shultz, T. R., & Rivest, F. (2000). Knwledge-based asade-rrelatin. Preedings f the Internatinal Jint Cnferene n Neural Netwrks. IEEE Cmuter Siety Press. Silver, D., & Merer, R. (1996). The arallel transfer f task knwledge using dynami learning rates based n a measure f relatedness. Cnnetin Siene, 8, Siris, S., & Shultz, T. R. (1998). Neural netwrk mdeling f develmental effets in disriminatin shifts. Jurnal f Exerimental Child Psyhlgy, 71, Thrun, S. & Mithell, T. (1993). Integrating indutive neural netwrk learning and exlanatin-based learning. In R. Bajsy (Ed.), Preedings f the Thirteenth Internatinal Jint Cnferene n Artifiial Intelligene. San Mate, CA: Mrgan Kaufmann. Wisniewski, E. J. (1995). Prir knwledge and funtinally relevant features in net learning. Jurnal f Exerimental Psyhlgy: Learning, Memry, and Cgnitin, 21,