Neural Blackboard Architectures. of Combinatorial Structures in Cognition. Frank van der Velde

Transcription

1 Neural Blackboard Archiecures of Combinaorial Srucures in Cogniion Frank van der Velde Uni of Cogniive Psychology Leiden Universiy Wassenaarseweg 52, 2333 AK Leiden The Neherlands

2 Absrac Human cogniion is unique in he way in which i relies on combinaorial (or composiional) srucures. Language provides ample evidence for he exisence of combinaorial srucures, bu hey can also be found in visual cogniion. To undersand he neural basis of human cogniion, i is herefore essenial o undersand how combinaorial srucures can be insaniaed in neural erms. In his recen book on he foundaions of language, Jackendoff formulaed four fundamenal problems for a neural insaniaion of combinaorial srucures: he massiveness of he binding problem, he problem of 2, he problem of variables and he ransformaion of combinaorial srucures from working memory o long-erm memory. This paper aims o show ha hese problems can be solved by means of neural blackboard archiecures. For his purpose, a neural blackboard archiecure for senence srucure is presened. In his archiecure, neural srucures ha encode for words are emporarily bound in a manner ha preserves he srucure of he senence. I is shown ha he archiecure solves he four problems presened by Jackendoff. The abiliy of he archiecure o insaniae senence srucures is illusraed wih examples of senence complexiy observed in human language performance. Similariies exis beween he archiecure for senence srucure and blackboard archiecures for combinaorial srucures in visual cogniion, derived from he srucure of he visual corex. These archiecures are briefly discussed, ogeher wih an example of a combinaorial srucure in which he blackboard archiecures for language and vision are combined. In his way, he archiecure for language is grounded in percepion. 2

3 Conen 1. Inroducion 2. Four challenges for cogniive neuroscience 2.1. The massiveness of he binding problem 2.2. The problem of The problem of 2 and he symbol grounding problem 2.3. The problem of variables 2.4. Binding in working memory versus long-erm memory 2.5. Overview 3. Combinaorial srucures wih synchrony of acivaion 3.1. Nesed srucures wih synchrony of acivaion 3.2. Produciviy wih synchrony of acivaion 4. Processing linguisic srucures wih recurren neural neworks 4.1. Combinaorial produciviy wih RNNs 4.2. RNNs and he massiveness of he binding problem 5. Blackboard archiecures of combinaorial srucures 6. A neural blackboard archiecure of senence srucure 6.1. Gaing and memory circuis 6.2. Overview of he archiecure Connecion srucure for binding in he archiecure 6.3. Muliple insaniaion and binding in he archiecure Answering binding quesions 6.4. Exending he blackboard archiecure The modular naure of he blackboard archiecure 6.5. Consiuen binding in long-erm memory One-rial learning Explici encoding of senence srucure wih synapic modificaion 6.6. Variable binding Neural srucure versus spreading of acivaion 6.7. Srucural dependencies in he blackboard archiecure Embedded clauses in he blackboard archiecure Muliple embedded clauses Dynamics of binding in he blackboard archiecure Dynamics of binding and complexiy 6.8. Furher developmen of he archiecure 7. Neural blackboard archiecures of combinaorial srucures in vision 7.1. Feaure binding 7.2. A neural blackboard archiecure of visual working memory Feaure binding in visual working memory 7.3. Feaure binding in long-erm memory 7.4. Inegraing combinaorial srucures in language and vision 8. Conclusion Noes References 3

4 1. Inroducion Human cogniion is unique in he manner in which i processes and produces complex combinaorial (or composiional) srucures (e.g., Anderson 1983; Newell 1990; Pinker 1998). Therefore, o undersand he neural basis of human cogniion, i is essenial o undersand how combinaorial srucures can be insaniaed in neural erms. However, combinaorial srucures presen paricular challenges o heories of neurocogniion, which have no been widely recognized in he cogniive neuroscience communiy (Jackendoff 2002). A prominen example of hese challenges is given by he neural insaniaion (in heoreical erms) of linguisic srucures. In his recen book on he foundaions of language, Jackendoff (2002; see also Jackendoff in press) analyzed he mos imporan heoreical problems ha he combinaorial and rule-based naure of language presens o heories of neurocogniion. He summarized hese problems under he heading of four challenges for cogniive neuroscience (pp ). As recognized by Jackendoff, hese problems arise no only wih linguisic srucures, bu wih combinaorial cogniive srucures in general. This paper aims o show ha neural blackboard archiecures can provide an adequae heoreical basis for a neural insaniaion of combinaorial cogniive srucures. In paricular, I will discuss how he problems presened by Jackendoff (2002) can be solved in erms of a neural blackboard archiecure of senence srucure. I will also discuss he similariies beween he neural blackboard archiecure of senence srucure and neural blackboard archiecures of combinaorial srucures in visual cogniion and visual working memory (Van der Velde 1997; Van der Velde & de Kamps 2001; 2003a). To begin wih, I will firs ouline he problems described by Jackendoff (2002) in more deail. This presenaion is followed by a discussion of he mos imporan soluions ha have been offered hus far o mee some of hese challenges. These soluions are based on eiher synchrony of acivaion or on recurren neural neworks Four challenges for cogniive neuroscience The four challenges for cogniive neuroscience presened by Jackendoff (2002) consiss of: he massiveness of he binding problem ha occurs in language, he problem of muliple insances (or he problem of 2 ), he problem of variables, and he relaion beween binding in working memory and binding in long-erm memory. I will discuss hese problems in urn The massiveness of he binding problem In neuroscience, he binding problem concerns he way in which neural insaniaions of elemens (consiuens) can be relaed (bound) emporarily in a manner ha preserves he srucural relaions beween he consiuens. Examples of his problem can be found in visual percepion. Colors and shapes of objecs are parly processed in differen brain areas, bu we perceive objecs as a uniy of color and shape. Thus, in a visual scene wih a green apple and a red orange, he neurons ha code for green have o be relaed (emporarily) wih he neurons ha code for apple, so ha he confusion wih a red apple (and a green orange) can be avoided. In he case of language, he problem is illusraed in figure 1. Assume ha words like ca, chases and mouse each acivae specific neural srucures, such as he word 4

5 assemblies discussed by Pulvermüller (1999). The problem is how he neural srucures or word assemblies for ca and mouse can be bound o he neural srucure or word assembly of he verb chases, in line wih he hemaic roles (or argumen srucure) of he verb. Tha is, how ca and mouse can be bound o he role of agen and heme of chases in he senence The ca chases he mouse, and o he role of heme and agen of chases in he senence The mouse chases he ca. (a) chases mouse ca senence neurons senence neurons (b) chases ca chases ca (c) mouse mouse ca chases mouse mouse chases ca Figure 1. (a). Illusraion of he neural srucures ( neural word assemblies ) acivaed by he words ca, chases and mouse. Boom: An aemp o encode senence srucures wih specialized senence neurons. In (b), a senence neuron has he assemblies for he words ca, chases and mouse in is recepive field (as indicaed wih he cone). The neuron is acivaed by a specialized neural circui when he assemblies in is recepive field are acive in he order ca chases mouse. In (c), a similar senence neuron for he senence mouse chases ca. A poenial soluion for his problem is illusraed in figure 1. I consiss of specialized neurons (or populaions of neurons) ha are acivaed when he srings ca chases mouse (figure 1b) or mouse chases ca (figure 1c) are heard or seen. Each neuron has he word assemblies for ca, mouse and chases in is recepive field (illusraed wih he cones in figures 1b and 1c). Specialized neural circuis could acivae one neuron in he case of ca chases mouse and anoher neuron in he case of mouse chases ca, by using he difference in emporal word order in boh srings. Circuis of his kind can be found in he case of moion deecion in visual percepion (e.g., Hubel 1995). For 5

6 insance, he movemen of a verical bar ha sweeps across he reina in he direcion from A o B can be deeced by using he difference in acivaion ime (onse laency) beween he ganglion cells in A and B. A similar specialized circui can deec a verical bar moving from B o A. However, a fundamenal problem wih his soluion in he case of language is is lack of produciviy. Only specific and familiar senences can be deeced in his way. Bu any novel senence of he ype Noun chases Noun or, more generally, Noun Verb Noun will no be deeced because he specific circui (and neuron) for ha senence will be missing. Ye, when we learn ha Dumbledore is headmaser of Hogwars, we immediaely undersand he meaning of Dumbledore chases he mouse, even hough we have never encounered ha senence before. The difference beween language and moion deecion in his respec illusraes a fundamenal difference in naure beween hese wo cogniive processes. In he case of moion deecion here is a limied se of possibiliies, so ha i is possible (and i pays off) o have specialized neurons and neural circuis for each of hese possibiliies. However, his soluion is no feasible in he case of language. Linguiss ypically describe language in erms of is unlimied combinaorial produciviy. Words can be combined ino phrases, which in urn can be combined ino senences, so ha arbirary senence srucures can be filled wih arbirary argumens (e.g., Webelhuh 1995; Sag & Wasow 1999; Chomsky 2000; Pullum & Scholz 2001; Jackendoff 2002; Piaelli-Palmarini 2002). In heory, an unlimied amoun of senences can be produced in his way, which excludes he possibiliy of having specialized neurons and circuis for each of hese senences. One could argue ha many of he senences ha are heoreically possible may be oo complex for humans o undersand (Chrisiansen & Chaer 1999). However, unlimied (recursive) produciviy is no necessary o make a case for he combinaorial naure of language, given he number of senences ha can be produced or undersood. For insance, he average English-speaking 17-year-old knows more han words (Bloom 2000). Wih his lexicon, and wih a limied senence lengh of 20 words or less, one can produce a se of senences in naural language in he order of or more (Pinker 1998). A se of his kind can be characerized as a performance se of naural language, in he sense ha (barring a few seleced examples) any senence from his se can be produced or undersood by a normal language user. Such a performance se is no unlimied, bu i is of asronomical magniude (e.g., exceeds he esimaed lifeime of he universe expressed in seconds). By consequence, mos senences in his se are senences ha we have never heard or seen before. Ye, because of he combinaorial naure of language we have he abiliy o produce or undersand arbirary senences from a se of his kind. Hence, he se of possibiliies ha we can encouner in he case of language is unlimied in any pracical sense. This precludes a soluion of he binding problem in language in erms of specialized neurons and circuis. Insead, a soluion is needed ha depends on he abiliy o bind arbirary argumens o he hemaic roles of arbirary verbs, in agreemen wih he srucural relaions expressed in he senence. Moreover, he soluion has o saisfy he massiveness of he binding problem as i occurs in language, which is due o he ofen complex and hierarchical naure of linguisic srucures. For insance, in he senence The ca ha he dog bies chases he mouse, ca is bound o he 6

7 role of heme of he verb bies, bu i is bound o he role of agen of he verb chases. In fac, he whole phrase The ca ha he dog bies is bound o he role of agen of he verb chases (wih ca as he head of he phrase). Each of hese specific bindings has o be saisfied in an encoding of his senence. Furher examples can be seen in a simple synacic srucure like beside a big sar (Jackendoff 2002). Here, one can idenify relaionships like i is a preposiional phrase, i is a par of a verb phrase, i follows a verb, and i has a preposiion and noun phrase pars. Binding problems occur for each of hese relaionships The problem of 2 The second problem presened by Jackendoff (2002) is he problem of muliple insances, or he problem of 2. Jackendoff illusraes his problem wih he senence The lile sar is beside a big sar 2. The word sar occurs wice in his senence, he firs ime relaed wih he word lile and he second ime relaed wih he word big. The problem is how in neural erms he wo occurrences of he word sar can be disinguished, so ha sar is firs bound wih lile and hen wih big, wihou creaing he erroneous binding of lile big sar. The problem of 2 resuls from he assumpion ha any occurrence of a given word will resul in he acivaion of he same neural srucure (e.g., is word assembly, as illusraed in figure 1). Bu if he second occurrence of a word only resuls in he reacivaion of a neural srucure ha was already acivaed by he firs occurrence of ha word, he wo occurrences of he same word are indisinguishable (Van der Velde 1999). Perhaps he problem could be solved by assuming ha here are muliple neural srucures ha encode for a single word. The word sar could hen acivae one neural srucure in lile sar and a differen one in big sar, so ha he bindings lile sar and big sar can be encoded wihou creaing lile big sar. However, his soluion would enail ha here are muliple neural srucures for all words in he lexicon, perhaps even for all poenial posiions a word could have in a senence (Jackendoff 2002). More imporanly even, his soluion disrups he uniy of word encoding as he basis for he meaning of a word. For insance, he relaion beween he neural srucures for ca and mouse in ca chases mouse could develop ino he neural basis for he long-erm knowledge ( fac ) ha cas chase mice. Similarly, he relaion beween he neural srucures for ca and dog in dog bies ca could form he basis of he fac ha dogs figh wih cas. Bu if he neural srucure for ca (say, ca 1 ) in ca 1 chases mouse is differen from he neural srucure for ca (say, ca 2 ) in dog bies ca 2, hen hese wo facs are abou differen kinds of animals The problem of 2 and he symbol grounding problem I is ineresing o look a he problem of 2 from he perspecive of he symbol grounding problem ha occurs in cogniive symbol sysems. Duplicaing symbols is easy in a symbol sysem. However, in a symbol sysem, one is faced wih he problem ha symbols are arbirary eniies (e.g., srings of bis in a compuer), which herefore have o be inerpreed o provide meaning o he sysem. Tha is, symbols have o be grounded in percepion and acion if symbol sysems are o be viable models of cogniion (Harnad 1991; Barsalou 1999). Grounding in percepion and acion can be achieved wih neural srucures such as he word assemblies illusraed in figure 1. In line wih he idea of neural assemblies 7

8 proposed by Hebb (1949), Pulvermüller (1999) argued ha words acivae neural assemblies, disribued over he brain (as illusraed wih he assemblies for he words ca, mouse and chases in figure 1). One could imagine ha hese word assemblies have developed over ime by means of a process of associaion. Each ime a word was heard or seen, cerain neural circuis would have been acivaed in he corex. Over ime, hese circuis will be associaed, which resuls in an overall cell assembly ha reflecs he meaning of ha word. For insance, assemblies for words wih a specific visual conen would srech ino he visual corex, whereas words ha describe paricular acions (e.g., walking vs alking ) would acivae assemblies ha srech ino specific pars of he moor corex, as observed by Pulvermüller e al. (2001). Bu, as argued above, word assemblies are faced wih he problem of 2. Thus, i seems ha he problem of 2 and he symbol grounding problem are complemenary problems. To provide grounding, he neural srucure ha encodes for a word is embedded in he overall nework srucure of he brain. Bu his makes i difficul o insaniae a duplicaion of he word, and hus o insaniae even relaively simple combinaorial srucures such as The lile sar is beside a big sar. Conversely, duplicaion is easy in symbol sysems (e.g., if 1101 = sar, hen one would have The lile 1101 is beside a big 1101, wih lile and big each relaed o an individual copy of 1101). Bu symbols can be duplicaed easily because hey are no embedded in an overall srucure ha provides he grounding of he symbol The problem of variables The knowledge of specific facs can be insaniaed on he basis of specialized neural circuis, in line wih hose illusraed in figure 1. Bu knowledge of sysemaic facs, such as he fac ha own(y,z) follows from give(x,y,z), canno be insaniaed in his way, ha is, in erms of a lising of all specific insances of he relaion beween he predicaes own and give (e.g., from give(john, Mary, book) i follows ha own(mary, book); from give(mary, John, pen) i follows ha own(john, pen); ec.). Insead, he derivaion ha own(mary, book) follows from give(john, Mary, book) is based on he rule ha own(y,z) follows from give(x,y,z), combined wih he binding of Mary o he variable y and book o he variable z. This raises he quesion of how rulebased derivaion wih variable binding can be insaniaed in he brain. The abiliy of rule-based derivaion wih variable binding provides he basis for he sysemaic naure of cogniion (Fodor & Pylyshyn 1988). Cogniion is sysemaic in he sense ha one can learn from specific examples and apply ha knowledge o all examples of he same kind. A child will indeed encouner only specific examples (e.g., ha when John gives Mary a book, i follows ha Mary owns he book) and ye i will learn ha own(y,z) follows from all insances of he kind give(x,y,z). In his way, he child is able o handle novel siuaions, such as he derivaion ha own(harry, broom) follows from give(dumbledore, Harry, broom) Binding in working memory versus long-erm memory Working memory in he brain is generally assumed o consis of a susained form of acivaion (e.g, Ami 1995; Fuser 1995). Tha is, informaion is sored in working memory as long as he neurons ha encode he informaion remain acive. In conras, long-erm memory resuls from synapic modificaion. In his way, he connecions 8

9 beween neurons are modified (e.g., enhanced) so ha when some of he neurons are reacivaed, hey will reacivae he ohers neurons as well. The neural word assemblies, illusraed in figure 1, are formed by his process. Boh forms of memory are relaed in he sense ha informaion in one form of memory can be ransformed ino informaion in he oher form of memory. Informaion is iniially sored in working memory before i is sored in long-erm memory. Conversely, informaion in long-erm memory can be reacivaed and sored in working memory. This raises he quesion of how he same combinaorial srucure can be insaniaed boh in erms of neural acivaion and in erms of synapic modificaion, and how hese differen insaniaions can be ransformed ino one anoher Overview I is clear ha he four problems presened by Jackendoff (2002) are inerrelaed. For insance, he problem of 2 also occurs in rule-based derivaion wih variable binding, he massiveness of he binding problem is found in combinaorial srucures sored in working memory and in combinaorial srucures sored in long-erm memory. Therefore, a soluion of hese problems has o be an inegraed one ha solves all four problems simulaneously. In his paper, I will discuss how all four problems can be solved in erms of neural blackboard archiecures in which combinaorial srucures can be insaniaed. Firs, however, I will discuss wo alernaives for a neural insaniaion of combinaorial srucures. In he nex secion I will discuss he use of synchrony of acivaion as a mechanism for binding consiuens in combinaorial srucures. In he secion afer ha, I will discuss he view ha combinaorial srucures can be handled wih recurren neural neworks. 3. Combinaorial srucures wih synchrony of acivaion An elaborae example of a neural insaniaion of combinaorial srucures in which synchrony of acivaion is used as a binding mechanism is found in he model of reflexive reasoning presened by Shasri and Ajjanagadde (1993). In heir model, synchrony of acivaion is used o show how a known fac such as John gives Mary a book can resul in an inference such as Mary owns a book. The proposiion John gives Mary a book is encoded by a fac node ha deecs he respecive synchrony of acivaion beween he nodes for John, Mary and book, and he nodes for giver, recipien and give-objec. These nodes encode for he hemaic roles of he predicae give(x,y,z). In a simplified manner, he reasoning process begins wih he query own(mary, book)? (i.e., does Mary own a book?). The query resuls in he respecive synchronous acivaion of he nodes for owner and own-objec of he predicae own(y,z) wih he nodes for Mary and book. In urn, he nodes for recipien and giveobjec of he predicae give(x,y,z) are acivaed by he nodes for owner and own-objec, such ha owner is in synchrony wih recipien and own-objec is in synchrony wih giveobjec. As a resul, he node for Mary is in synchrony wih he node for recipien and he node for book is in synchrony wih he node for give-objec. This allows he fac node for John gives Mary a book o become acive, which produces he affirmaive answer o he query. A firs problem wih a model of his kind is found in a proposiion like John gives Mary a book and Mary gives John a pen. Wih synchrony as a binding mechanism, a 9

10 confusion arises in his proposiion beween John and Mary in heir respecive roles of giver and recipien in his proposiion. In effec, he same paern of acivaion will be found in he proposiion John gives Mary a pen and Mary gives John a book. Thus, wih synchrony of acivaion as a binding mechanism, boh proposiions are indisinguishable. I is no difficul o see he problem of 2 here. John and Mary occur wice in he proposiion, bu in differen hemaic roles. The simulaneous bu disinguishable binding of John and Mary wih differen hemaic roles canno be achieved wih synchrony of acivaion. To solve his problem, Shasri and Ajjanagadde allowed for a duplicaion (or muliplicaion) of he nodes for he predicaes. In his way, he whole proposiion John gives Mary a book and Mary gives John a pen is pariioned ino he wo elemenary proposiions John gives Mary a book and Mary gives John a pen. To disinguish beween he proposiions, he nodes for he predicae give(x,y,z) are duplicaed. Thus, here are specific nodes for, say, give 1 (x,y,z) and give 2 (x,y,z), wih give 1 (x,y,z) acivaed by John gives Mary a book and give 2 (x,y,z) acivaed by Mary gives John a pen. Furhermore, for he reasoning process o work, he associaions beween predicaes have o be duplicaed as well. Thus, he node for give 1 (x,y,z) has o be associaed wih a node for, say, own 1 (y,z) and he node for give 2 (x,y,z) has o be associaed wih a node for own 2 (y,z). This raises he quesion of how hese associaions can be formed simulaneously during learning. During is developmen, a child will learn from specific examples. Thus, i will learn ha, when John gives Mary a book, i follows ha Mary owns he book. In his way, he child will form an associaion beween he nodes for, say, give 1 (x,y,z) and own 1 (y,z). Bu he associaion beween he node for give 2 (x,y,z) and own 2 (y,z) would no be formed in his case, because hese nodes are no acivaed wih John gives Mary a book and Mary owns he book. Thus, when he predicae give(x,y,z) is duplicaed ino give 1 (x,y,z) and give 2 (x,y,z), he sysemaiciy beween John gives Mary a book and Mary gives John a pen is los Nesed srucures wih synchrony of acivaion The duplicaion soluion discussed above fails wih nesed (or hierarchical) proposiions. For insance, he proposiion Mary knows ha John knows Mary canno be pariioned ino wo proposiions Mary knows and John knows Mary, because he enire second proposiion is he y argumen of knows(mary, y). Thus, he fac node for John knows Mary has o be in synchrony wih he node for know-objec of he predicae know(x,y). The fac node for John knows Mary will be acivaed because John is in synchrony wih he node for knower and Mary is in synchrony wih he node for know-objec. However, he fac node for Mary knows Mary will also be acivaed in his case, because Mary is in synchrony wih boh knower and know-objec in he proposiion Mary knows ha John knows Mary. Thus, he proposiion Mary knows ha John knows Mary canno be disinguished from he proposiion Mary knows ha Mary knows Mary. Likewise, he proposiion Mary knows ha John knows Mary canno be disinguished from he proposiions John knows ha John knows Mary and John knows ha Mary knows Mary, because John is in synchrony wih knower in each of hese proposiions. 10

11 3.2. Produciviy wih synchrony of acivaion A furher problem wih he use of synchrony of acivaion as a binding mechanism is is lack of produciviy. The model of Shasri and Ajjanagadde depends on he use of fac nodes, such as he fac node for John gives Mary a book, o deec he synchrony of acivaion beween argumens and hemaic roles. The use of fac nodes is needed because synchrony of acivaion has o be deeced o process he informaion ha i encodes (Denne 1991). Bu fac nodes, and he circuis ha acivae hem, are similar o he specialized neurons and circuis illusraed in figure 1. I is excluded o have such nodes and circuis for all possible verb-argumen bindings ha can occur in language, in paricular for novel insances of verb-argumen binding. As a resul, synchrony of acivaion as a binding mechanism fails o provide he produciviy given by combinaorial srucures. The problems analyzed here, he inabiliy o solve he problem of 2, he inabiliy o deal wih nesed srucures, and he lack of sysemaiciy and produciviy, are also found in oher domains in which synchrony of acivaion is used as a binding mechanism, such as visual cogniion (Van der Velde & de Kamps 2002). 4. Processing linguisic srucures wih recurren neural neworks The argumen ha combinaorial srucures are needed o obain produciviy in cogniion has been quesioned (Elman 1991; Churchland 1995, Por & Van Gelder 1995). In his view, produciviy in cogniion can be obained in a funcional manner ( funcional composiionaliy, Van Gelder 1990), wihou using explici combinaorial srucures. The mos elaborae approach of his kind is found in he processing of linguisic srucures wih recurren neural neworks (Elman 1991; Miikkulainen 1996; Chrisiansen & Chaer 2001; Palmer-Brown e al. 2002). A recurren neural nework (RNN) is a mulilayer (usually hree-layer) feedforward nework, in which he acivaion paern in he hidden (middle) layer is copied back o he inpu layer, where i serves as par of he inpu o he nework in he nex learning sep. In his way, RNNs are capable of processing and memorizing sequenial srucures. Elman (1991) used RNNs o predic wha kind of word would follow nex a a given poin in a senence. For insance, in case of he senence Boys who chase boys feed cas, he nework had o predic ha afer Boys who chase a noun would follow, and ha afer Boys who chase boys a plural verb would occur. To perform his ask, he nework was rained wih senences from a language generaed wih a small lexicon and a basic phrase grammar. The nework succeeded in his ask, boh for he senences ha were used in he raining session and wih oher senences from he same language. A more complex model was presened by Miikkulainen (1996). The model consised of muliple pars (including a parser ), based on RNNs. The purpose of he model was o assign hemaic roles (agen, ac, paien) o he words in a clause. The model succeeded in his ask, even wih embedded clauses (however, clauses were resriced o wo or hree word clauses, which resuled from he fac ha he oupu layer of he parser had hree nodes). Thus, i seems ha RNNs are capable o process linguisic srucures in a noncombinaorial manner. However, as Chrisiansen and Chaer (2001) noed, all RRNs model languages derived from small vocabularies (in he order of 10 o 100 words). In conras, he vocabulary of naural language is huge, which resuls in an asronomical 11

12 produciviy when combined wih even limied senence srucures (e.g., senences wih 20 words or less, see secion 2.1.). Therefore, I will discuss his form of combinaorial produciviy in he case of language processing wih RNNs in more deail Combinaorial produciviy wih RNNs In Elman (1991), he RNN was rained and esed wih a language in he order of 10 5 senences, based on a lexicon of abou 20 words. In conras, he combinaorial produciviy of naural language is in he order of senences or more, based on a lexicon of 10 5 words. A basic aspec of such a combinaorial produciviy is he abiliy o inser words from one familiar senence conex ino anoher. For insance, if one learns ha Dumbledore is headmaser of Hogwars, one can also undersand Dumbledore chases he mouse, or The dog sees Hogwars, even hough hese specific senences have no been encounered before. RNNs should have his capabiliy as well, if hey are o approach he combinaorial produciviy of naural language. Using he predicion ask of Elman (1991), we invesigaed his quesion by esing he abiliy of RNNs o recognize a senence consising of a new combinaion of familiar words in familiar synacic roles (Van der Velde e al. 2003). In one insance, we used senences like dog hears ca, boy sees girl, dog loves girl and boy follows ca o rain he nework on he word predicion ask. The purpose of he raining senences was o familiarize he RNNs wih dog, ca, boy and girl as argumens of verbs. Then, a verb like hears from dog hears ca was insered ino anoher rained senence like boy sees girl o form he es senence boy hears girl, and he neworks were esed on he predicion ask for his senence. To srenghen he relaions beween boy, hears and girl, we also included raining senences like boy who ca hears obeys John and girl who dog hears likes Mary. These senences inroduce boy and hears, and girl and hears, in he same senence conex (wihou using boy hears and hears girl) 4. In fac, girl is he objec of hears in girl who dog hears likes Mary, as in he es senence boy hears girl. However, alhough he RRNs learned he raining senences o perfecion, hey failed wih he es senences. Despie he abiliy o process boy sees girl and dog hears ca, and even girl who dog hears likes Mary, hey could no process boy hears girl. The behavior of he RNNs wih he es senence boy hears girl was in fac similar o he behavior in a word salad condiion, which consised of random word srings, based on he words used in he raining session. Analysis of his word salad condiion showed ha he RNNs prediced he nex word on he basis of direc word-word associaions, based on all woword combinaions found in he raining senences. The similariy beween word salads and he es senence boy hears girl suggess ha RNNs resor o word-word associaions when hey have o process novel senences composed of familiar words in familiar grammaical srucures. The resuls of hese simulaions indicae ha RNNs do no posses a minimal form of he combinaorial produciviy ha underlies human language processing. To pu his in perspecive, i is imporan o realize ha he lack of combinaorial produciviy observed in hese simulaions is no jus a negaive resul, ha could have been avoided by using a beer learning (raining) algorihm. The raining senences were learned o perfecion. The bes ha anoher algorihm could do is o learn hese senences o he same level of perfecion. I is unclear how his could produce a differen resul on he es senences. 12

13 Furhermore, he crucial issue here is no learning, bu he conras in behavior exhibied by he RNNs in hese simulaions. The RRNs were able o process ( undersand ) boy sees girl and dog hears ca, and even girl who dog hears likes Mary, bu no boy hears girl. This conras in behavior is no found in humans, regardless of he learning procedure used. I is no found in human behavior due o he srucure of he human language sysem. This is wha he issue of sysemaiciy is all abou: if you undersand boy sees girl, dog hears ca and girl who dog hears likes Mary, you canno bu undersand boy hears girl. Any failure o do so would be regarded as pahological RNNs and he massiveness of he binding problem The simulaions discussed above again show ha RNNs are capable of processing learned senences like girl who dog hears obeys Mary, and oher complex senence srucures. Thus, even hough RRNs fail in erms of combinaorial produciviy, hey could be used o process senence srucures in absrac erms. Tha is, hey could process a senence srucure in erms of Nouns (N) and Verbs (V), such as N-who-N-V-V-N in he case of senences like girl who dog hears obeys Mary. Senence processing in erms of N-V srings can be relaed wih he word assemblies illusraed in figure 1. Words of a similar caegory, like verbs or nouns, would have a common par in heir cell assemblies ha reflecs ha hey are verbs or nouns. The RRNs could be rained o process senences in erms of hese common pars, hus in erms of N- V srings. However, when used in his way, RRNs can only be a par of a neural model of human language performance. Consider, for insance, he senences ca chases mouse and mouse chases ca. Boh senences are N-V-N senences, and hus indisinguishable for hese RRNs. Ye, he wo senences convey very differen messages, and humans can undersand hese differences. In paricular, hey can produce he correc answers o he who does wha o whom quesions for each of hese senences, which canno be answered on he level of he N-V-N srucure processed by RRNs. This raises wo imporan quesions for he use of RRNs in his manner. Firs, how is he difference beween ca chases mouse and mouse chases ca insaniaed in neural erms? The lack of combinaorial produciviy discussed above shows ha his canno be achieved wih RRNs. Second, given a neural insaniaion of ca chases mouse and mouse chases ca, how can he srucural N-V informaion processed by he RRNs be relaed wih he specific conen of each senence? This is a binding problem, because i requires ha, for insance, he firs N in N-V-N is bound o ca in he firs senence and o mouse in he second senence. However, even if hese problems are solved, senence processing in erms of N-V srings is sill faced wih serious difficulies, as illusraed wih he following senences: The ca ha he dog ha he boy likes bies chases he mouse (1) The fac ha he mouse ha he ca chases roars surprises he boy (2) The absrac (N-V) srucure of boh senences is he same: N-ha-N-ha-N-V-V-V-N. Ye, here is a clear difference in complexiy beween hese senences (Gibson 1998). Senences wih complemen clauses (2) are much easier o process han senences wih cener-embeddings (1). This difference can be explained in erms of he bindings (dependencies) wihin he senence srucures. In (1) he firs noun is relaed wih he 13

14 second verb as is objec (heme) and wih he hird verb as is subjec (agen). In (2), he firs noun is only relaed wih he hird verb (as is subjec). This difference in srucural dependency (binding) is no capured in he sequence N-ha-N-ha-N-V-V-V-N. The srucural dependencies ha consiue he difference beween senences (1) and (2) again illusrae he massiveness of he binding problem ha occurs in linguisic srucures. Words and clauses have o be bound correcly o oher words and clauses in differen pars of he senence, in line wih he hierarchical srucure of a senence. These forms of binding are clearly beyond he capaciy of language processing wih RNNs. Similar limiaions are found wih RNNs in case of he problem of variables (Marcus 2001). 5. Blackboard archiecures of combinaorial srucures A combinaorial srucure consiss of pars (consiuens) and heir relaions. Briefly saed, one could argue ha he lack of combinaorial produciviy wih RNNs, as discussed above, illusraes a failure o encode he individual pars (words) of a combinaorial srucure (senence) in a producive manner. In conras, synchrony of acivaion fails in paricular o insaniae even moderaely complex relaions in he case of variable binding. These examples show ha neural models of combinaorial srucures can only succeed if hey provide a neural insaniaion of boh he pars and he relaions of combinaorial srucures. In compuaional erms, a blackboard archiecure provides a way o insaniae he pars and he relaions of combinaorial srucures. A blackboard archiecure consiss of a se of specialized processors (or demons, Selfridge 1959) ha inerac wih each oher by means of a blackboard (or workbench, or bullein board ). Each processor can process and modify he informaion ha is sored on he blackboard. In his way, he archiecure can process or produce informaion ha exceeds he abiliy of each individual processor. In he case of language, one could have processors for he recogniion of words and (oher) processors for he recogniion of specific grammaical relaions. These processors could hen communicae by using a blackboard in he processing of a senence. Thus, wih he senence The lile sar is beside a big sar, he word processors could sore he symbol for sar on he blackboard, he firs ime in combinaion wih he symbol for lile, and he second ime in combinaion wih he symbol for big. Oher processors could hen deermine he relaion (beside) beween hese wo copies of he symbol for sar. Jackendoff (2002) discusses blackboard archiecures of his kind for phonological, synacic and semanic srucures. In he nex secion, I will propose and discuss a neural blackboard archiecure for senence srucure based on neural assemblies. To address he problems described by Jackendoff (2002), neural word assemblies are no copied in his archiecure. Insead, hey are emporarily bound o he neural blackboard, in a manner ha disinguishes beween differen occurrences of he same word, and ha preserves he relaions beween he words in he senence. For insance, wih he senence The ca chases he mouse, he word assembly for ca is bound o he blackboard as he subjec or agen of chases, and he assembly for mouse is bound as he objec or heme of his verb. Wih he neural srucure of The ca chases he mouse, he archiecure can produce correc answers o quesions like Who chases he mouse? or Whom does he ca chase?. Quesions like hese can be referred o as binding quesions, because hey es 14

15 he abiliy of an archiecure o bind familiar pars in a (poenially novel) combinaorial srucure. A neural insaniaion of a combinaorial srucure such as The ca chases he mouse fails if i canno produce he correc answers o he quesions saed above. In language, binding quesions in fac query who does wha o whom informaion, which is he characerisic form of informaion provided by a senence (e.g., Pinker 1994; Calvin & Bickeron 2000). Aphasic paiens, for insance, are esed on heir language abiliies using non-verbal who does wha o whom quesions (e.g., Caplan 1992). In general, he abiliy o answer binding quesions is of fundamenal imporance for cogniion, because i is relaed wih he abiliy o selec informaion needed for purposive acion (e.g., Van der Heijden & van der Velde 1999). 6. A neural blackboard archiecure of senence srucure In line wih Pulvermüller (1999), words are assumed o be encoded in erms of neural word assemblies, as illusraed in figure 1 (secion 2.1.). I is clear ha he relaions beween he words in a senence canno be encoded in erms of direc associaions beween word assemblies. For insance, he associaion of mouse-chases-ca does no disinguish beween he senences The mouse chases he ca and The ca chases he mouse. However, relaions beween words can be encoded, and he problems discussed by Jackendoff (2002) can be solved, if word assemblies are embedded in a neural archiecure in which srucural relaions can be formed beween he word assemblies. A neural archiecure of his kind can be formed by means of srucure assemblies ha inerac wih he word assemblies. The srucure assemblies provide he possibiliy o encode differen insaniaions of he same word assembly (hereby solving he problem of 2 ), and hey can be used o bind word assemblies in erms of he synacic srucure of he senence. Figure 2 illusraes he neural srucure of he senence The mouse chases he ca in his archiecure. I consiss of word assemblies, srucure assemblies for noun phrases (NPs) and verb phrases (VPs), gaing circuis used for dynamic conrol, and memory circuis used o bind assemblies emporarily. In figure 2, he assemblies for mouse and ca are bound o NP assemblies (N 1 for ca and N 2 for mouse), and he assembly for chases is bound a VP assembly (V 1 ). The srucure assemblies are hen bound o each oher, in a manner ha encodes he verb-argumen srucure of he senence. For his purpose, each srucure assembly in he archiecure is composed of a main assembly (N i for NP assemblies and V i for VP assemblies) and one or more subassemblies. In figure 2, he NP and VP assemblies have subassemblies for he argumens agen (a) and heme () 6. To encode ca as he agen of chases, N 1 is bound wih V 1 by means of heir agen subassemblies. In urn, N 2 and V 1 are bound wih heir heme subassemblies, o provide he neural srucure for mouse as he heme of chases. Main assemblies and subassemblies are assumed o have he abiliy for reverberaing aciviy, in line wih he reverberaing aciviy found in he prefronal corex (e.g., Fuser 1973; Ami 1995; Dursewiz e al. 2000). As a resul, hey will remain acive for a while afer hey have been acivaed, unless hey are inhibied. Subassemblies are conneced o main assemblies by means of gaing circuis, which conrol he flow of acivaion wihin srucure assemblies. For insance, a main assembly can be acive, bu is subassemblies no, or vice versa. The abiliy o conrol he inernal dynamics of srucure assemblies is 15

16 of crucial imporance for he neural archiecure of senence srucure proposed here. Before illusraing his in more deail, I will firs discuss he gaing and memory circuis used in his archiecure. a a N 1 V 1 N 2 ca chases mouse gaing circui a memory circui V 1 srucure assembly (verb phrase) Figure 2. Illusraion of he neural senence srucure of ca chases mouse in he neural blackboard archiecure presened here. The words are encoded wih he word assemblies illusraed in figure 1 (secion 2.1.). Senence srucure is encoded wih srucure assemblies for noun-phrases (NP assemblies) and verb-phrases (VP assemblies). A srucure assembly consiss of a main assembly and a number of subassemblies, conneced o he main assembly by means of gaing circuis. The labeled subassemblies represen he hemaic roles of agen (a), and heme (). Binding beween assemblies is achieved wih acive memory circuis. Here, he assembly for ca is bound o he NP assembly N 1, he assembly for chases is bound o he VP assembly V 1, and he assembly for mouse is bound o he NP assembly N 2. N 1 and V 1 are bound by means of heir agen subassemblies and V 1 and N 2 are bound by means of heir heme subassemblies Gaing and memory circuis A gaing circui in he archiecure consiss of a disinhibiion circui, as described by Gonchar and Burkhaler (1999). Figure 3 (lef) illusraes a gaing circui in he direcion from assembly X o assembly Y. The circui conrols he flow of acivaion beween he wo assemblies by means of an exernal conrol signal. I operaes in he following manner. If he assembly X is acive, i acivaes an inhibiion neuron (or group of neurons) i x, which inhibis he flow of acivaion from X o X ou. When i x is inhibied by 16

17 anoher inhibiion neuron (I x ), ha is acivaed by an exernal conrol signal, X acivaes X ou. In urn, X ou acivaes Y. A gaing circui from Y o X operaes in a similar manner. Conrol of acivaion can be direcion specific. Thus, by producing a conrol signal in he direcion from X o Y, acivaion will flow in his direcion (if X is acive), bu no in he direcion from Y o X. The symbol illusraed in figure 3 (lef) will be used o represen he combinaion of gaing circuis in boh direcions (as in figure 2). Gaing Circui Memory Circui conrol XoY delay I x i x I x i x X X ou Y X X ou Y Symbol: X Y X Y (inacive) (X o Y and Y o X) X Y (acive) (X o Y and Y o X) Figure 3. Lef: A gaing circui in he direcion from assembly X o assembly Y, based on a disinhibiion circui. The large circles depic neural assemblies. The small circles depic (groups of) inhibiory neurons (i). A combinaion of wo gaing circuis in he direcions X o Y and Y o X is depiced in oher figures wih he symbol illusraed a he boom. Righ: A memory (gaing) circui in he direcion from assembly X o assembly Y, based on a gaing circui wih a delay assembly for conrol. A combinaion of wo memory circuis in he direcions X o Y and Y o X is depiced in oher figures wih he symbols illusraed a he boom, one for he inacive sae and one for he acive sae of his combined memory circui. A memory circui in he archiecure consiss of a gaing circui in which he conrol signal resuls from a delay assembly. Figure 3 (righ) illusraes a memory circui in he direcion of X o Y. However, each memory circui in he archiecure in fac consiss of wo such circuis in boh direcions (X o Y and Y o X). The delay assembly (ha conrols he flow of acivaion in boh direcions) is acivaed when X and Y are acive simulaneously (see below), and i remains acive for a while due o he reverberaing 17

18 naure of he acivaion in his assembly. As a resul, a memory circui can be in wo saes: acive and inacive. Each sae will be represened wih he symbol illusraed in figure 3 (righ). If he memory circui is inacive, acivaion canno flow beween he assemblies conneced by he memory circui. On he oher hand, if he memory circui is acive, acivaion will flow beween he assemblies i connecs, if one of hese assemblies is acivaed. In his way, an acive memory circui binds he wo assemblies i connecs. The memory circuis in figure 2 are acive, so ha word assemblies and srucure assemblies are bound in line wih he srucure of he senence Overview of he archiecure Figure 4 illusraes he par of he archiecure in which nouns can be bound as argumens o verbs. This par is illusraive of he archiecure as a whole. Noun assemblies N x N y NP a a V i V j VP Verb assemblies Figure 4. A neural blackboard archiecure for verb-argumen binding. Word assemblies for verbs are conneced o he main assemblies of VP srucure assemblies by means of (iniially) inacive memory circuis. Word assemblies for nouns are conneced o he main assemblies of NP srucure assemblies by means of (iniially) inacive memory circuis. The agen (a) and heme () subassemblies of he VP and NP srucure assemblies are conneced by means of (iniially) inacive memory circuis. Only subassemblies of he same kind are conneced o each oher. The main assemblies of VP assemblies are muually inhibiory. Likewise for NP srucure assemblies. 18

19 Each noun (word) assembly is conneced o he main assembly of each NP assembly by means of a memory circui, which is iniially inacive. Likewise, each verb (word) assembly is conneced o he main assembly of each VP assembly by means of an iniially inacive memory circui. Main assemblies of he same kind are muually inhibiory. Each NP and VP main assembly is conneced o a number of subassemblies by means of gaing circuis. The gaing circuis can be acivaed in a selecive manner by neural conrol circuis (no shown in he figure). For insance, he gaing circuis beween he main assemblies and he agen subassemblies can be acivaed wihou acivaing he gaing circuis for he heme subassemblies. Finally, all subassemblies of he same kind are conneced by means of memory circuis. For insance, each agen subassembly of he NP assemblies is conneced o each agen subassembly of he VP assemblies by means of an (iniially inacive) memory circui. In he processing of a senence i is assumed ha one of he NP assemblies will be acivaed whenever he assembly for a noun is acivaed. I is arbirary which of he NP assemblies is acivaed, provided ha he assembly is free. A srucure assembly is free when i is no already bound o a senence srucure, ha is, when all memory circuis conneced wih ha assembly are inacive 7. As illusraed in figure 4, only one NP main assembly can be acive a he same ime, due o he compeiion beween he NP main assemblies ha resuls from heir muual inhibiion. I is assumed ha he acive NP assembly will remain acive unil a new NP assembly is acivaed by he occurrence of a new noun in he senence 8. The selecion of a VP assembly proceeds in he same manner. In all, in he order of 10 2 VP assemblies and 10 2 NP assemblies would probably be needed in his archiecure. When a sequence of srucure assemblies has been acivaed, he firs assemblies in he sequence will reurn o he inacive sae (i.e., will again be free ), due o he decay of delay aciviy in he memory circuis conneced wih hese assemblies. In his way, only a subse of he srucure assemblies will be concurrenly acive in he blackboard archiecure Connecion srucure for binding in he archiecure Figure 5 (righ) illusraes ha he connecion srucure beween he agen subassemblies in figure 4 basically consiss of a marix-like array of columns. This connecion srucure is illusraive of every connecion beween assemblies by means of memory circuis. Each column conains a (combined) memory circui (figure 3, righ), including he delay assembly ha can acivae he memory circui. Each column also conains a circui ha can acivae he delay assembly (figure 5, lef). This circui is also a disinhibiion circui, in which he delay assembly will be acivaed if he neurons N in and V in are acive a he same ime. The neurons N in and V in, in urn, are acivaed by he respecive agen subassemblies of a NP assembly and a VP assembly. The acivaed agen subassembly of a given NP assembly acivaes he N in neurons in a horizonal row of columns (as illusraed wih N x in figure 5, righ). Likewise, he acivaed agen subassembly of a given VP assembly acivaes he V in neurons in a verical row of columns (as illusraed wih V i in figure 5, righ). The delay assembly in he column on he inersecion of boh rows will be acivaed if he agen subassemblies of N x and V i are acive a he same ime. This resuls in he binding of hese agen subassemblies (illusraed wih he shaded memory circui symbol in figure 5, righ). 19